V1 is way better than V2

Alpaca Market Data
1

Transition to V2 is a struggle V2 has problems regarding fetching data,

  • in V1 when i want last 200 bars from symbol X i set only the limit at anytime of the day/week/year i know i’m getting that last 200 going backwards (request at sunday morning get you last 200 from friday in V2 you need to know in advance your start and end dates interval and make sure is valid trading hours( market close, holidays,) or else you get (bars:null) that’s alot of IFs in your code

  • since page_token works in farward mode, you can’t set an end date and go back to get your required N bars (“Give me last N bars” does not work)

  • dates in string format, why? dates in string format requires parsing in any usecase, which is slow in most programming languages, let alone increase in object size 4 bytes* 20 charaters=80byte (1000 bar is over 100kb) (in v1 1000 bar is arround 60kb), timetamp is more standard

  • API params not mentioned in documentation like symbol can be formated in URL or you can use the symbols (plural) param to fetch multiple symbols this isn’t mentioned anywhere

  • breaking changes should be announced, V1 for paid plans doesn’t work (only iex data is returned)

Solution for start and end dates is simple, 2 modes are required

  • (end and limit)
  • (start and end)

(end and limit) if i provide end + limit it should behave as “give me the last LIMIT bars before END”
interval mode is when i provide (start and end) (no limit should be present) both work with hard coded page size or additional param to step forward (page_token)

4 Likes
2

I’m fairly new to Alpaca so never used V1, but this has been bothering me in V2.

in V2 you need to know in advance your start and end dates interval

Yeah, I just want the last n bars. I don’t want to have to specify a start time.

1 Like
3

agree on this point , since many "indicators required more than 200 historical quotes i did this to make sure i have sufficent quotes to get the indicator i want

probably not the best solution but so far this work well.

The start date is the problem because as you know one ticker for 1000 quotes can have the start date as 2 days ago and one 15 days ago !

image
image1487×938 74.4 KB

4

In fact, it would be enough to just add a new optional parameter - sorting order into all historical requests. This change will not break the backward compatibility (if the parameter is missing it will be treated as the current default ascending order) and will allow you to fetch the last N bars using descending order and page size limit.

2 Likes
5

Has there been any ideal solutions to this challenge? I’m still using V1 bars for daily bots, because the new V2 takes a lot more processing on my end, and changes to my code. It almost forces you to pull more bars than you want, then filter, sort, etc yourself. Maybe the goal is to push towards people towards streaming and not use historical bars for live analysis? I’m using the python API and was looking for a good method to recreate the last 202 bars… as stated above.

3 Likes
6

Giving the last ‘N’ bars from a certain date is pretty easy from Alpaca’s side. Why don’t you put this back in your API? This information is truly important for fulfilling some indicators of technical analysis, and I believe that you know that. If you don’t want to change the query parameters, consider that if the request only uses the ‘end’ attribute, make the API return the ‘limit’ LAST bars from the ‘end’. Simple as that!

1 Like
7

@Prometeus It sounds good but in fact, it’s a breaking change in behavior (not in API itself), and potentially (just potentially) it can break existing client’s code. AFAIU the main problem that blocks Alpaca teams from implementing different sorting is pagination support and performance of underlying storage. From my own experience, some time-series-based DBs show very bad performance in case you read data in the order opposite to the natural one.

8

@oleg.rakhmatulin, thanks for your input. That behavior was already present in v1, according to other users. I didn’t use the previous version, but I guess that it was working well, based on previous users’ declarations here. About the eventual ‘breaking existing code’ situation I don’t see that, because the current API version demands the use of both ‘start’ and ‘end’ query parameters, and in the lack of one of them the request is not even processed, so currently if one of the parameters isn’t present it indeed configures a ‘bug’ on the client’s application. However, if this new feature (at least for the v2 API) is implemented, the previous client application won’t be affected as they currently need both parameters to be used, and only those who demand the last records will use only the ‘end’ parameter.

About the situation of time-series-based DB have bad performance when reading data in a backward fashion, it doesn’t need to be implemented that way. I guess the data is written to DB in a ‘basic’ timeframe, like 1s, but is regrouped internally according to the client’s requested timeframe. So, data will continue to be read in the natural order, without hassling the read performance. I also guess that data is written without ‘null’ values caused by a weekend or a holiday, of course. So all data will be there, in a known ‘basic’ timeframe, and reading the last ‘nth’ records (and processing according to the requested timeframe) won’t reduce DB performance.

I only don’t understand why a feature that was working well previously and is pretty useful for technical analysis is simply disabled in the new API version. It doesn’t make sense.

9

@Prometeus You are right - your proposal will not break the backward compatibility. It’s my fault - I forget that both start and end query parameters are required. But in any case, it’s not a very clear behavior and the explicit 'sorting_order` query parameter looks better for me.

The v1 version of API was just a wrapper/redirector around the Polygon.io REST API and therefore supported all features provided by the underlying API. I personally also think that alternative sorting order is not a big problem and I don’t know the exact reason why this feature is still not implemented. Maybe, the Alpaca team just doesn’t have enough resources and other tasks have higher priorities.

10

@oleg.rakhmatulin, your suggestion of a ‘sorting-order’ query parameter is good, and I believe that it fits your needs, but it still requires that you know previously where your ‘start’ date should be. Depending on the number of latest bars you are requesting you may need to do complicated calculations and adjustments in order to skip weekends and holidays, and eventually, you will need to request more data than you will use, which will waste resources of the API.

I didn’t know the previous history of ‘v1’, and you are probably right about their priorities at this time. I hope they find a solution for the ‘latest bars’ requests.

11

Hi!

Thank you for your feedback. I’m a software engineer at Alpaca, focused primarily on market data. I’m not an algotrader or a financial / stock market expert. Here’s my best shot to answer your questions:

  • in V1 when i want last 200 bars from symbol X i set only the limit at anytime of the day/week/year i know i’m getting that last 200 going backwards (request at sunday morning get you last 200 from friday in V2 you need to know in advance your start and end dates interval and make sure is valid trading hours( market close, holidays,) or else you get (bars:null) that’s alot of IFs in your code

You’re right, you can’t get the last N bars in v2. As @oleg.rakhmatulin already mentioned, this is because of technical limitations. To further clarify: it’s not that it would be slower, it’s that it’s not possible.

Of course one can always do workarounds, but it’s not easy and I personally feel that it wouldn’t even be beneficial. I know this must sound surprising, but here’s what I think: in most scenarios getting the latest N bars doesn’t make sense anyway. If you ask for the last 300 AAPL bars, you’ll most likely get the bars for the same day, but if you do the same for a rarely traded symbol, you can end up with bars days, weeks or even months old. I’m confident that in most scenarios what you really need is the bars for an interval, not “a number of” bars, which is exactly what the API encourages. But let me repeat myself: the reason it’s not supported is a technical one.

  • since page_token works in farward mode, you can’t set an end date and go back to get your required N bars (“Give me last N bars” does not work)

This is exactly the same issue: ordering by decreasing timestamps is not supported.

  • dates in string format, why? dates in string format requires parsing in any usecase, which is slow in most programming languages, let alone increase in object size 4 bytes* 20 charaters=80byte (1000 bar is over 100kb) (in v1 1000 bar is arround 60kb), timetamp is more standard

Both of your arguments are valid: the string representation is harder to parse and slightly larger than the unix timestamp. The reason we went for RFC 3339 is human readability. But I honestly agree with you here. What would you think about adding an option to choose unix timestamp over RFC 3339 on the REST API?

  • API params not mentioned in documentation like symbol can be formated in URL or you can use the symbols (plural) param to fetch multiple symbols this isn’t mentioned anywhere

Strongly agreed here. We have a lot of work to do on our documentation, and many we’re doing it, but we’re not quite there yet.

  • breaking changes should be announced, V1 for paid plans doesn’t work (only iex data is returned)

Breaking changes are always announced. This was especially true for the v1 deprecation. Besides, to give you more context, we had no choice but to use IEX only data for v1. Polygon simply stopped providing data for us, which was used for v1.

Finally, let me say that the title of this post is somewhat degrading to the work we did on v2. Here’s a very incomplete list of some features that were added to v2:

  • Historical & real-time trades
  • Historical & real-time quotes
  • Bar aggregation for custom timeframes
  • Bar adjustment for splits & dividends
  • Trade count & VWAP for bars
  • Crypto data
  • OTC data
  • Proper pagination

Don’t get me wrong, it’s not perfect by any means. The latest N bars is certainly a highly requested feature. We’re doing our utmost to deliver the best possible product for you, so I’m thrilled to have a conversation with you to move to that direction.

2 Likes