NATS/Streaming - client disconnects from trade_updates

Alpaca Account Troubleshooting
1

I can connect to the trade_updates channel and receive streaming updates. However, after some time I get disconnected and stop receiving updates of my trades. I can see the Exception in the logs, but up to now I haven’t been able to catch it in try..except block.

How could I catch this disconnect?

I start listening to channel by doing:

    conn.run(
        ['trade_updates'] + tcs + qcs
    )

But a try…except block doesn’t catch the disconnection as suggested in the Momentum example script, with relevant code:

# Handle failed websocket connections by reconnecting
def run_ws(conn, channels):
    try:
        conn.run(channels)
    except Exception as e:
        print(e)
        conn.close
        run_ws(conn, channels)

The first exception I see on logs is:

Task exception was never retrieved
future: <Task finished coro=<StreamConn._consume_msg() done, defined at /mnt/home/an/trader/venv/lib/python3.6/site-packages/alpaca_trade_api/stream2.py:46> exception=ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='paper-api.alpaca.markets', port=443): Max retries exceeded with url: /v2/orders (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7fe18a88a898>, 'Connection to paper-api.alpaca.markets timed out. (connect timeout=2)'))",),)>
Traceback (most recent call last):
  File "/mnt/home/an/trader/venv/lib/python3.6/site-packages/urllib3/connection.py", line 159, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/mnt/home/an/trader/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection
    raise err
  File "/mnt/home/an/trader/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection
    sock.connect(sa)
socket.timeout: timed out
2

I wanted to update this thread by posting what the logging.DEBUG logs look like. I haven’t figured this out yet, and it hinders my algo… obviously…

2019-07-30 11:24:18,729 - client < Frame(fin=True, opcode=1, data=b'[{"ev":"Q","sym":"KO","c":0,"bx":10,"ax":10,"bp":54.08,"ap":54.09,"bs":8,"as":15,"t":1564500229400,"z":1}]', rsv1=False, rsv2=False, rsv3=False)
2019-07-30 11:24:18,730 - client ! timed out waiting for pong
2019-07-30 11:24:18,730 - client ! failing OPEN WebSocket connection with code 1011
2019-07-30 11:24:18,730 - client - state = CLOSING
2019-07-30 11:24:18,730 - client > Frame(fin=True, opcode=8, data=b'\x03\xf3', rsv1=False, rsv2=False, rsv3=False)
2019-07-30 11:24:18,764 - client > Frame(fin=True, opcode=9, data=b'\x9c\xd1\xcb\xc3', rsv1=False, rsv2=False, rsv3=False)
2019-07-30 11:24:18,768 - client < Frame(fin=True, opcode=10, data=b'\x9c\xd1\xcb\xc3', rsv1=False, rsv2=False, rsv3=False)
2019-07-30 11:24:18,768 - client - received solicited pong: 9cd1cbc3
2019-07-30 11:24:28,773 - client ! timed out waiting for TCP close
2019-07-30 11:24:28,774 - client x closing TCP connection
2019-07-30 11:24:29,067 - client < Frame(fin=True, opcode=9, data=b'', rsv1=False, rsv2=False, rsv3=False)
2019-07-30 11:24:29,067 - client - received ping, sending pong: [empty]
2019-07-30 11:24:29,068 - client > Frame(fin=True, opcode=10, data=b'', rsv1=False, rsv2=False, rsv3=False)
2019-07-30 11:24:38,779 - client > Frame(fin=True, opcode=9, data=b'\x855\xbe\xf8', rsv1=False, rsv2=False, rsv3=False)
2019-07-30 11:24:38,780 - client ! timed out waiting for TCP close
2019-07-30 11:24:38,780 - client x aborting TCP connection
2019-07-30 11:24:38,781 - client - event = connection_lost(None)
2019-07-30 11:24:38,781 - client - state = CLOSED
2019-07-30 11:24:38,781 - client x code = 1006, reason = [no reason]
2019-07-30 11:24:38,781 - client - aborted pending ping: d1950635
2019-07-30 11:24:38,788 - client < Frame(fin=True, opcode=10, data=b'\x855\xbe\xf8', rsv1=False, rsv2=False, rsv3=False)
2019-07-30 11:24:38,788 - client - received solicited pong: 8535bef8
1 Like