I have tried to get a list of all of my historical trades from the paper account via the Python api. I have used “check_orders = apiPaper.list_orders(status=‘closed’)”, and I have used status=‘all’. I am only able to get up to 66 orders (trades). There are 100’s that I should be able to get. I have set the limit to limit=1000000 to see if that would work.
How do I get all of my historical orders (trades)? Thank you.
I actually have a simmilar issue and have been working on it for the past 5 days non-stop (same program but hopping around issues), I managed too side step it by doing something kinda silly. I was working In c# as my python skills are lackluster, But I can give you a general Idea of what you need too do.
I ended up looping the API as many times as limit/AmountNeeded, slowly incrementing the date every time it loops, copied the response to a object and then threw that object into a dictionary, when the loop was over the Dictionary<Number, Object> was converted into a JSON file and CSV file and copied into a text file too be used later down the line.
It ended up being really compicated as I was doing it in async but it does work and I do get the data I want within a reasonable amount of time (I am pulling a lot of data, millions of bars).
I would really Like their too be no limit on data as doing what I am doing is increadably time consuming at 200 Requests/Minute, so if your requesting any large amount of data it can be increadably difficult to merge all of the requests into one singular JSON file, but If its what you want prepare for a couple hours of programing, But I whole-Heartedly Support the 1000 bars request limit.
The limit is their for a reason, as trasmitting that much data if very time consuming and takes an entire thread of the server for way too long, But it does become a issue when the API request limit also has too be timed as not too hit over 200 requests per minute, which I feel could be an Issue for programs wanting too get every crumb of data.
Sorry if this is a bit long but it is actually quiet a weird problem too get around. I am not experienced in APIs and as such a lot of my time was spent blind folded walking into walls. Python API seems like it has very good documentation though with multiple youtube videos on it as well as a great amounts of peoples code too refrence, So I wish you luck and hope you can do better then me.
To get all historical orders for an account one typically needs to request multiple ‘chunks’ of orders and then assemble them into a ‘total orders’ list, dataframe, or other conditioner. Much like @Logical proposed above. This can be done rather easily by requesting a sub-set of orders, check the submitted_at time of the last order received, then request another sub-set of orders up to the time of that last order. Repeat until there aren’t any more orders to fetch.
Here is an example (in Python)
import datetime
import pandas as pd
# First set the initial end_time for the orders
# Set to the current time so we fetch all orders up until now
# There's a limit to the number of orders received, but by selecting
# direction='desc' the API will fetch orders from this date BACKWARDS
end_time = datetime.datetime.now(datetime.timezone.utc).isoformat()
# Create an empty list to store our full order list
order_list = []
# Set the 'chunk size' to the 500 max
CHUNK_SIZE = 500
while True:
# Get the max chunk
order_chunk = api.list_orders(status='all',
nested='False',
direction='desc',
until=end_time,
limit=CHUNK_SIZE)
if order_chunk:
# Have orders so add to list
order_list.extend(order_chunk)
# Set end_time for next chunk to earliest fetched order
end_time = order_chunk[-1].submitted_at.isoformat()
else:
# No more orders. Make a dataframe of entire list of orders
# Then exit
order_df = pd.DataFrame([order._raw for order in order_list])
break
This will create both a list (order_list) and dataframe (order_df) of all the orders.
Note this concept of ‘paging’ using an end date and a loop is very common. A similar approach can be used to get larger sets of Polygon data, account activities etc.
First consider if you really need to iterate over rows in a DataFrame. Iterating through pandas dataFrame objects is generally slow. Iteration beats the whole purpose of using DataFrame. It is an anti-pattern and is something you should only do when you have exhausted every other option. It is better look for a List Comprehensions , vectorized solution or DataFrame.apply() method for iterate through DataFrame.
Pandas DataFrame loop using list comprehension
result = [(x, y,z) for x, y,z in zip(df['Name'], df['Promoted'],df['Grade'])]
Pandas DataFrame loop using DataFrame.apply()
result = df.apply(lambda row: row["Name"] + " , " + str(row["TotalMarks"]) + " , " + row["Grade"], axis = 1)