Duplicates in historic trades

It seems that the history often has repeat trades. Everything but the ID is the same. Which can happen but not that often. For example on IVOG 2021-May -28 10% of the transactions happened at the exact same time, price, quantity, condition and exchange.


repeat transaction 16 entries with:
t x p s c i z
67 2021-05-28T16:03:47.5472Z A 199.03 100 [ , F] 52983525229917 B

Wondering about the best way to deal wit this, e.g.:
What is the reason and remedy? I currently drop every thing at the same time, price, quantity and exchange. Does that remove too much or too little? This is relevant to understand the volume. Do the aggregate bars from Alpaca also count these transactions to determine volume?

@nthorwir Those are real trades. Those are not 'duplicates. If a trade has a unique ID then it is a unique trade. Not sure what your use case is but they probably should not be ‘dropped’.

Why do we see what seems to be duplicates? It would seem improbable that 16 trades (in this example) would all be for the same symbol at the same time? There are a couple of clues to what is going on and why this isn’t that ‘improbable’.

The first clue is the trade condition. Notice its an “F”. That is an “intermarket sweep” (ISO) order. That is a special kind of order that can fill across several exchanges and isn’t required to follow the order-protection rule under regulation NMS. In other words, it doesn’t need to fill at the NBBO quotes. The trader really just wants to fill the order as fast as possible at any cost.

The second clue is that all the orders are for a single 100 share block. I certainly do not know for sure, but this looks like some entity is exercising their stocks options. Options are purchased in blocks of 100 shares. The trades look as if they were placed to match up with each option (ie 1 order per option). They all got placed at the same time and executed at the same time.

Now, this actually highlights a larger issue. Not all trades are created equal. The trade API returns every trade which occurred on or off an exchange. It returns special kinds of trades which really don’t represent the price one would expect if placing a simple market order. For this reason, the Securities Information Providers (SIPs) publish guidance on which trade conditions to exclude from price and volume calculations. Basically only ‘regular’ ‘round lot’ orders (typical market orders over 100 shares) are included in prices but every trade is included in volume. Below is a summary of which conditions are typically excluded from pricing. If one is looking at raw trades one needs to be aware of the trade conditions and filter out any that you would deem not relavent.

To answer the specific question “Do the aggregate bars from Alpaca also count these transactions to determine volume?” Yes, all trades (except M and Q) are included in volume calculations. This isn’t on the list of ‘excluded’ conditions so these would also be included in prices.

EXCLUDED Trades from MarketCache

Feed Tape Condition Description
CTS AB B Average Price Trade
UTDF C W Average Price Trade
CTS/UTDF ABC 4 Derivatively Priced
CTS/UTDF ABC 7 Qualified Contingent Trade (“QCT”)
CTS/UTDF ABC 9 Corrected Consolidated Close (per listing market)
CTS/UTDF ABC G Bunched Sold Trade
CTS/UTDF ABC H Price Variation Trade
CTS/UTDF ABC I Odd Lot Trade
CTS/UTDF ABC M Market Center Official Close
CTS/UTDF ABC P Prior Reference Price
CTS/UTDF ABC Q Market Center Official Open
CTS/UTDF ABC U Extended Trading Hours (Sold Out of Sequence)
CTS/UTDF ABC V Contingent Trade
CTS/UTDF ABC Z Sold (out of Sequence)