How to write to MarketStore?

Alpaca Account Troubleshooting
1

Hi all,

I’m new to this data analysis world.

I’m looking forward to include MarketStore to my trading project, right now working on Python, dumping dataframes to json files in parallel with sqlalchemy for other data.

I’d like to know what others approach is for things like “live” processing and writing to marketstore of indicators to the tickers received.

I’d like a solution over pymarketstore instead of getting down to code a plugin in Golang, if that’s possible.

I also have a question that is probably something I’m not yet understanding. What is the way to upload (ex. a RSI indicator serie) to an existing marketstore bucket? should this be an apply() over the serie or is there another method to do this in a straight way?

Thanks in advance, cheers!

2 Likes
2

Yeah that’s totally possible. Some of the serious production system using MarketStore indeed feeds data using pymarketstore. Did you check out the example in README?

In [1]: import pymarketstore as pymkts

## query data

In [2]: param = pymkts.Params('BTC', '1Min', 'OHLCV', limit=10)

In [3]: cli = pymkts.Client()

In [4]: reply = cli.query(param)

In [5]: reply.first().df()
Out[5]:
                               Open      High       Low     Close     Volume
Epoch
2018-01-17 17:19:00+00:00  10400.00  10400.25  10315.00  10337.25   7.772154
2018-01-17 17:20:00+00:00  10328.22  10359.00  10328.22  10337.00  14.206040
2018-01-17 17:21:00+00:00  10337.01  10337.01  10180.01  10192.15   7.906481
2018-01-17 17:22:00+00:00  10199.99  10200.00  10129.88  10160.08  28.119562
2018-01-17 17:23:00+00:00  10140.01  10161.00  10115.00  10115.01  11.283704
2018-01-17 17:24:00+00:00  10115.00  10194.99  10102.35  10194.99  10.617131
2018-01-17 17:25:00+00:00  10194.99  10240.00  10194.98  10220.00   8.586766
2018-01-17 17:26:00+00:00  10210.02  10210.02  10101.00  10138.00   6.616969
2018-01-17 17:27:00+00:00  10137.99  10138.00  10108.76  10124.94   9.962978
2018-01-17 17:28:00+00:00  10124.95  10142.39  10124.94  10142.39   2.262249

## write data

In [7]: import numpy as np

In [8]: import pandas as pd

In [9]: data = np.array([(pd.Timestamp('2017-01-01 00:00').value / 10**9, 10.0)], dtype=[('Epoch', 'i8'), ('Ask', 'f4')])

In [10]: cli.write(data, 'TEST/1Min/Tick')
Out[10]: {'responses': None}

In [11]: cli.query(pymkts.Params('TEST', '1Min', 'Tick')).first().df()
Out[11]:
                            Ask
Epoch
2017-01-01 00:00:00+00:00  10.0
3

Hi @hitoshi, thanks for your answer.

I realized my question was not well formulated.

I’ve gone through the readme, the example writes just one row, I’ve written in different ways and column types, but I just couldn’t get my head arround writting entire series right to the datastore.

Here is one try, at least no errors outputs, but couldn’t get to load the data.

import time
import numpy as np

last_time = time.time()
import pymarketstore as pymkts

from technical import indicators
param = pymkts.Params('binance_ETH-USDT', '1Day', 'OHLCV')
cli = pymkts.Client(endpoint='http://localhost:5993/rpc')

reply = cli.query(param)
df = reply.first().df()

last_time = time.time()

df = indicators.bollinger_bands(df.loc[:], field='Close', period=20, stdv=1.1)

df = df.fillna(0)
data = np.array(
        [
        df.index.values,
        df['bb_upper'].values,
        df['bb_lower'].values
        
        ], 
        dtype=[
                ('Epoch', 'i8'),
                ('bb_upper', 'f4'),
                ('bb_lower', 'f4')
                ]
        )

cli.write(data, 'binance_ETH-USDT/1Day/OHLCV')

print (cli.query(pymkts.Params('binance_ETH-USDT', '1Day', 'OHLCV')).first().df())

I couldn’t find any reference on how to do this. I couldn’t find a way yet and have done some hours of research on this. I know it looks realy dumb but I’d bet there are many dumb out there like myself :slight_smile:

Thans again, cheers.

4

Ok, now I got it. Thanks for letting me find it by myself, learned a lot :joy:

I had a problem with the interval, I was calling 1Day which seemed to work well to read but not for writting? Changed it to 1D and worked.

Here is my example for future generations, and please share better approaches.

import time
import numpy as np
import pandas as pd

last_time = time.time()
import pymarketstore as pymkts

from technical import indicators
param = pymkts.Params('binance_ETH-USDT', '1D', 'OHLCV', limit=100)
cli = pymkts.Client(endpoint='http://localhost:5993/rpc')

reply = cli.query(param)
df = reply.first().df()
df = indicators.bollinger_bands(df.loc[:], field='Close', period=20, stdv=1.1)
df=df.fillna(0)
df_save = df[['bb_lower', 'bb_upper']].copy()
df_save.index = df_save.index.astype('i8') // 10**9
data = df_save.to_records()
cli.write(data, 'binance_ETH-USDT/1D/Tick')
print (cli.query(pymkts.Params('binance_ETH-USDT', '1D', 'Tick')).first().df())

Now, let’s play with data. Cheers.

1 Like
5

Sorry I missed your last one. Glad to hear you could solve it :slight_smile:

6

Hey,
So I’m a little confused on this one… I can pull data from Marketstore no problem, but I’m not entirely sure I understand the writing process (I’ve looked at the README, it is pretty brief on the subject. I’m trying to write from a pandas dataframe.

So at this point I’ve pulled data out, and I’m now trying to write the exact same data back in (which you would think would be easy), but I’m getting this error: “cannot convert input with unit ‘s’” and it traces back to:

idxname = self.array.dtype.names[0]
73 df = pd.DataFrame(self.array).set_index(idxname)
—> 74 index = pd.to_datetime(df.index, unit=‘s’, utc=True)
75 tz = self.timezone
76 if tz.lower() != ‘utc’:"

Any tips for writing from a dataframe? This is basically what I’m doing:

df = cli.query(pymkts.Params('gdax_BTC-USD', '1D', 'OHLCV')).first().df()
mid = df.to_records(index_dtypes='i8',column_dtypes={"Open": "f4","High":"f4","Low":"f4","Close":"f4","Volume":"f4"})
cli.write(mid, 'BTC/1D/OHLCV')

The data is written, but format-wise it looks completely different than the existing data, but instead of being organized by year, it appears to be grouping between -32700 and +32700.

Any tips or tricks when writing from dataframes? Thanks.