Starting at the beginning of this week (2/6/2023) my service has been encountering a huge amount of timeouts from the Market Data API. I have ruled out rate limiting due to two factors:
- It’s not a rate limiting error and the service’s requests do not exceed 200/minute.
2.The amount of requests the service is making has not gone up and this service has been running for over a year without once encountering this timeout issue.
The timeouts are happening so frequently it has absolutely become an issue and handicapped the processes.
Has anyone else noticed an uptick in timeouts from the Market Data API?
1 Like
Which specific endpoints are you hitting? When you say “timeouts” are you getting 500 errors?
1 Like
HI Dan, thanks for the response.
I was getting the errors for:
https://data.alpaca.markets/v2/stocks/[SYMBOL]/quotes
https://data.alpaca.markets/v2/stocks/[SYMBOL]//bars
Error:
HTTPSConnectionPool(host=‘data.alpaca.markets’, port=443): Max retries exceeded (Caused by NewConnectionError(‘<urllib3.connection.HTTPSConnection object at 0xffff4b77b0d0>: Failed to establish a new connection: [Errno 110] Connection timed out’))
1 Like
I’m having the same issue that started at the same time. I’m attempting to hit the paper-api and I end up at https://data.alpaca.markets
I receive HTTP 200 None when I turn on debugging. However nothing is returned. But I see the same error as @ provonchee if I don’t have debugging enabled. The code that calls alpaca hasn’t changed. I tried resetting my paper api key to see if that did the trick. Also I’m requesting SIP data as a paid user.
1 Like
@Justin_Coffi What is the exact method or endpoint you are calling? Also what is the response you get? You say you get 200 but that wouldn’t happen if you also get the same error as above Connection timed out. I can check the logs and maybe see what’s happening.
1 Like
I’m getting the same error starting last week as well:
ConnectTimeout: HTTPSConnectionPool(host='api.alpaca.markets', port=443): Max retries exceeded with url: /v2/account (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7fe10756d690>, 'Connection to api.alpaca.markets timed out. (connect timeout=None)'))
It’s inconsistent. Half the time the request works, and half the time it times out like this.
1 Like
This is still happening for me, and based on the responses, appears to not be an isolated incident.
@Dan_Whitnable_Alpaca any luck looking into this?
1 Like
@provonchee There hasn’t been any changes on the Alpaca side which should cause this. A HTTPSConnectionPool(host='api.alpaca.markets', port=443) error is a client side error typically caused by the network or network setup issues. I did a google search and came back with a number of individuals seeing similar errors (on different platforms). There didn’t seem to be a common fix but all of them were client side. Maybe check this out. It’s not the same problem but may give some ideas to check.
Thanks @Dan_Whitnable_Alpaca
A recent Docker update could possibly play a part in this. Running locally, I can easily replicate the issue. I’ll try to roll back the update and see if this clears it up. Thanks again for looking into it.
@provonchee I’m always careful when saying ‘it’s not Alpaca’ but one clue that it’s not is that this seems to be happening on both the api.alpaca.markets and the data.alpaca.markets endpoints. Those endpoints are on two separate sets of infrastructure and rather independant and do not share a common update cadence.
Hello. I’m having similar issues today, with the error below. I would appreciate any input you all might have to offer on how to fix this problem.
Traceback (most recent call last):
conn = connection.create_connection(
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
requests.exceptions.ConnectionError: HTTPSConnectionPool(host=‘data.alpaca.markets’, port=443): Max retries exceeded with url: /v2/stocks/APA/snapshot (Caused by NewConnectionError(‘<urllib3.connection.HTTPSConnection object at 0x0000029DFFC191E0>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed’))
Yeah, unfortunately I am still experiencing the issue.
I rolled back my version of Docker, hoping it was a newly introduced bug issue, but no luck. Tried multiple releases of Docker and all of them experience the same timeout issue.
I’ve also noticed that the timeouts are intermittent. There are times they rarely happen and other times they happen every minute. Makes me think it is in fact something on Alpaca’s end of things, especially since I’ve reverted my local env to the state it was when these timeouts did not exist at all (circa January 2023)…but I’m still dealing with them. Very frustrating. My test cycles are four times as long as they should be.
Thanks for your input, @provonchee. The issue is intermittent for me too.
It’s odd that I am able to get snapshots containing bid and ask prices without any problem. The error happens only when my code attempts to get fill activities (eg: fills = api.get_activities(‘FILL’, date=date_today.isoformat()) )
@Dan_Whitnable_Alpaca Can you offer any specific suggestions for how I can resolve this error?
1 Like
I have been running into this same issue recently. After some testing it seems to be related to how quickly I am making requests. I have the pro account but I have found (depending on the day) that I have to significantly throttle my request often as high as 750ms between requests. Other days I can get away with 50 or less. Its very odd… Happens on multiple computers, firewall on and off, multiple networks. I do not think its on my side. When I have a moment I am going to try and implement direct with HTTP client vs using the API and see if there is a difference.
@masoric Are you experiencing a HTTPSConnectionPool error? This seems to be related to a bug in the urllib3 package. This bug was ostensibly fixed in version 1.26.15 however, I have still experienced something similar. First, ensure you are loading the most up to date version of of urllib3 (my pip install loads ver 2.0.7 which isn’t the latest but still >1.26.15). Ultimately, you could put any fetches inside a try-except statement (probably a good practice in any case). My experience is that one can retry immediately and that second retry always succeeds.
@Dan_Whitnable_Alpaca I am using c# and the Historical Bars request functions and Live.GetAlpacaDataClient. everything is inside try statements.
the issue appears to be getting worse. A year ago I did not need to throttle it at all Three months ago I only needed about 15ms between requests. Then it grew to 50 and most recently I am having to increase that to north of 750 some days. Which actually breaks the use case for my app some days.
Hopefully this weekend I will have some time to try removing the usage of the c# api completely and just make direct HTTP requests and use json deserialize to parse the data.
@masoric So you are not getting the HTTPSConnectionPool error (since that is generated from the urllib3 python package and you are using C#). What error are you seeing?
@Dan_Whitnable_Alpaca My apologies I was not clear in my previous response. That is correct. I am not using the python package.
When I can stop the system in its current state and recreate the error this afternoon, I will get the specific error verbiage. If memory serves it is a frustratingly vague connection time out message.
Apologies again I ended up traveling for work out of the blue.
What’s very odd is I actually capture no errors in my code. Everything is wrapped in some kind of try statement and it never captures anything. Its almost like the API just stops, provides a generic message response and that’s it.
I will have to put a packet capture on the nic card to see if there is something unique happening there.
I am also going to revert to an earlier version of the API and see if that works as well.