Historical data using async

Hi, i am trying to fetching 5 years 1 minute data using python rest_async method but it only return 10000 bar limit and no page_token to continue from that 10000. Is there a way to fetch all 5 years 1 min data with asyncio ?

For problems like this, you need to split the download into small enough chunks so that each chunk downloads in a few seconds. The algorithm is as follows:

  • Iterate through days.
  • Iterate through blocks of symbols.
  • For each day and symbol block, download data, add to a list.
  • Finally, write everything to some database or file (e.g. Arrow/Parquet).
  • Can parallelize for speed.
  • And to add production-quality reliability, handle any exceptions with some intelligent retry logic.

I’ve already written all of the Python code for a number of data providers (FinnHub, IQfeed, Alpaca, etc.). I have a Docker container that auto-downloads the required data on a schedule and puts into a clean form ready for machine learning and my live trading infrastructure.