Abnormal latencies observed when multiple trading applications are running in parallel

Alpaca Trading
1

Logged Req#233285 for the same and pasting the discussion here so that it will be beneficial for everyone:

===== Question ======

Hi,

When I simply have one trading application connecting to an Alpaca paper account and making trades, latencies are coming within expectations, as was commented by Dan in Req#219388.

However, things get vastly different when I have multiple trading applications running in parallel, connecting to the same paper trading account and making trades (on different instruments). A significant number of trades(say around 25%) end up with huge latencies, beyond our expectations.

The reason I could think of is the following: When multiple trading applications are running in parallel, they may send orders at almost the same time to Alpaca’s order engine, which would result in some sort of a queue at the order engine, resulting in higher latencies. It looks to me that Alpaca’s order engine is more tailored toward handling sequential orders from one account.

Does this sound reasonable? It would be great if I could get someone’s opinion on this.

=== Answer =====

Hi ,

Thanks for reaching out and hope you’re well.

Your observation about increased latency when running multiple trading applications in parallel on the same paper account is valid and aligns with how Alpaca’s system processes orders. The paper trading environment does have specific characteristics that can affect performance when handling concurrent order submissions.

The paper trading system processes orders differently than live trading. When orders are submitted, they go through validation processes including buying power checks. When multiple applications submit orders simultaneously to the same account, these validation processes can create bottlenecks, particularly because the system needs to maintain account-level consistency and prevent race conditions.

The system uses account-level locking mechanisms to ensure data integrity when processing orders. This means that when multiple orders arrive at nearly the same time for the same account, they must be processed in a serialized manner to some extent, which can contribute to the latency you’re observing.

While the paper trading environment is designed to provide a realistic testing experience, it’s important to understand its limitations. As such, we suggest the below:

  1. Consider the simulation nature: Paper trading is a simulation and doesn’t account for all real-world factors like order queue position, market impact, or information leakage that would occur in live trading.

  2. Test with realistic scenarios: If your production strategy will involve multiple applications, testing this way in paper is valuable for understanding system behavior, but be aware that the latency characteristics may differ from live trading.

  3. Rate limiting: Consider implementing some rate limiting or coordination between your applications to avoid sending large bursts of orders simultaneously.

  4. Monitor order submission patterns: The system is designed to handle normal trading patterns, but extremely high concurrent order rates from a single account can strain the processing pipeline.

I hope this helps.