Hi, i am trying to fetching 5 years 1 minute data using python rest_async method but it only return 10000 bar limit and no page_token to continue from that 10000. Is there a way to fetch all 5 years 1 min data with asyncio ?
For problems like this, you need to split the download into small enough chunks so that each chunk downloads in a few seconds. The algorithm is as follows:
- Iterate through days.
- Iterate through blocks of symbols.
- For each day and symbol block, download data, add to a list.
- Finally, write everything to some database or file (e.g. Arrow/Parquet).
- Can parallelize for speed.
- And to add production-quality reliability, handle any exceptions with some intelligent retry logic.
I’ve already written all of the Python code for a number of data providers (FinnHub, IQfeed, Alpaca, etc.). I have a Docker container that auto-downloads the required data on a schedule and puts into a clean form ready for machine learning and my live trading infrastructure.