why am I getting too many requests, isn’t it unlimited?
Traceback (most recent call last):
File “/Users/ziyangjiang/Documents/abc/examples/timer.py”, line 205, in analyze_symbol
ret_hour_candles, candle_wrong, last_hour_candle_time, last_hour_candle_open = get_sorted_hour_candles(trade_station_client, symbol, 650, stock, use_polygon, polygon_client, use_alpaca, alpaca_client)
File “/Users/ziyangjiang/Documents/abc/examples/timer.py”, line 129, in get_sorted_hour_candles
hour_candles0 = alpaca_client.get_bars(
File “/Users/ziyangjiang/Documents/abc/venv/lib/python3.8/site-packages/alpaca_trade_api/rest.py”, line 714, in get_bars
bars = list(self.get_bars_iter(symbol,
File “/Users/ziyangjiang/Documents/abc/venv/lib/python3.8/site-packages/alpaca_trade_api/rest.py”, line 698, in get_bars_iter
for bar in bars:
File “/Users/ziyangjiang/Documents/abc/venv/lib/python3.8/site-packages/alpaca_trade_api/rest.py”, line 585, in _data_get
resp = self.data_get(path, data=data, feed=feed,
File “/Users/ziyangjiang/Documents/abc/venv/lib/python3.8/site-packages/alpaca_trade_api/rest.py”, line 270, in data_get
return self._request(
File “/Users/ziyangjiang/Documents/abc/venv/lib/python3.8/site-packages/alpaca_trade_api/rest.py”, line 213, in _request
return self._one_request(method, url, opts, retry)
File “/Users/ziyangjiang/Documents/abc/venv/lib/python3.8/site-packages/alpaca_trade_api/rest.py”, line 234, in _one_request
resp.raise_for_status()
File “/Users/ziyangjiang/Documents/abc/venv/lib/python3.8/site-packages/requests/models.py”, line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://data.alpaca.markets/v2/stocks/KO/bars?timeframe=1Hour&adjustment=raw&start=2022-07-23
@sheep The data APIs, if subscribed to the “Unlimited” data plan is actually limited to 10,000 calls/min. If one only is subscribed to the “Free” plan the limit is 200 calls/min.
Most, if not all the data calls provide for multiple symbols. It’s a ‘best practice’ to, for example, make a single call of 100 symbols rather than 100 calls for one symbol each.
@sheep The API endpoint to fetch multiple symbol bars is GET/v2/stocks/bars (see the docs here). The python SDKs however will take either a single stock or multiple stocks as a parameter. If one is using the alpaca-trade-api SDK then this will fetch about 600 hours of 1 hour bar data for multiple stocks (in this case 3000 random stocks).
# Install and import required packages and classes
!pip install -q alpaca-trade-api==1.5.0
import alpaca_trade_api as alpacaapi
from pandas.tseries.offsets import BDay
# Get 3000 random active symbols
asset_list = api_data.list_assets(status='active', asset_class='us_equity')
asset_df = pd.DataFrame([asset._raw for asset in asset_list])
symbols = asset_df.sample(3000).symbol
# Get approximately the last 600 hr bars
# Set hours per trading day max from 4:00-20:00 = 16 hrs
# Calculate trading days (assume no holidays)
HRS_PER_TRADING_DAY = 16
BUSINESS_DAYS_IN_600_HRS = 600 // HRS_PER_TRADING_DAY
# Set the end date to the current time and the start to previous 600 hours of business days
end_date = pd.to_datetime("now").tz_localize('America/New_York')
start_date = end_date - BDay(BUSINESS_DAYS_IN_600_HRS)
hour_bars = api_data.get_bars(symbols,
'1Hour',
start=start_date.isoformat(),
end=end_date.isoformat(),
).df
# Convert to market time for easier reading
hour_bars = hour_bars.tz_convert('America/New_York')
This takes 2-3 minutes to fetch all the data, but will return a single dataframe with hour bars for 3000 symbols. The dataframe will look like this
The hardest part (believe it or not) is getting exactly 600 hour bars for each symbol. Also, the max number of symbols in a single call is about 3000. There isn’t a fixed limit but the symbols become part of the API URL. With 3-4 characters per symbol, the URL gets to be very long. Some networks don’t handle very long URLs and may give an error. I personally try to stay under 3000 symbols per call.
the other thing, I haven’t tried at market hour yet, say I query at 10:25am, does get_bars return the unfinished 10am hour as a bar? If not, what api should I use?
@sheep You mentioned “single symbol call return length less than 650 bars every 10 symbols roughly”. I assume the issue/question here is why don’t you see 650 bars for each symbol? The simple answer is that bars are only generated if there are trades during a bar. No trades no bar. Many symbols do not trade after market hours, and even during market hours (as you are fetching by using between_time("9:00", "15:00")) many stocks are thinly traded or have ‘trading halts’ or trade but trade in small ‘odd lots’ (odd lot trades are not included in bar calculations).
I took hourly bar data, between 9:00-15:00, for 3000 random stocks for the past 180 days and counted the total bars. While most have 860 bars, there are many with less. Here is a histogram
@sheep You asked “does get_bars return the unfinished 10am hour as a bar”. The short answer is yes. When fetching hour bars, the current bar will be a partial bar and will update every minute (ie hour bars are calculated from minute bars).
You can try this yourself. Simply fetch the latest hour bar for some stock (during market hours). Then fetch the same bar a few minutes later. The values, especially the volume, will update.
