This repository publishes dxFeed demo data to a Kafka Redpanda to stream. The Kafka stream is then consumed into Deephaven. Running redpanda_start.sh will create stock market tables on http://localhost:10000/ide. Note that the Deephaven's default authentication (pre-shared key). For more information on using pre-shared key authentication and setting your own key, see How to configure and use pre-shared key authentication.
This app runs using Deephaven's application mode.
Dockerfile- The dockerfile for the application. This extends the default Deephaven images to add dependencies. See our guide, How to install Python packages, for more information.docker-compose.yml- The Docker Compose file for the application. This is mostly the same as the Deephaven docker-compose file with modifications to run Redpanda, application mode, dxFeed Kafka producer and the custom dependencies.redpanda_start.sh- A simple helper script to launch the application.data/app.d/start.app- The Deephaven application mode app file.data/app.d/tables.py- The Deephaven queries to initialize tables.data/layouts/layout.json- The Deephaven layout to show all initialized tables.data/notebooks/query.py- A Deephaven sample query to run on tables.dxfeed/requirements.txt- Python dependencies for the application.dxfeed/fin_pub.py- The Python script that pulls the data from dxFeed and streams to Redpanda.
The demo feed contains a handful of symbols with 15 minute delayed publication during trading hours. In order to provide events during non-trading hours, the demo feed will replay random old events every few seconds.
The Repanda Kafka producer used in this guide creates the following Deephaven tables:
trade: Last Sale price for a given instrument + daily volumequote: Bid/Ask prices for a given instrumentcandle: Charting OHLCV candleprofile: Instrument profilesummary: Open-High-Low-Close values for current day and Close for previous trading dayorder: Market depth: Level 2 quote by market maker /regional exchange quote /element of order bookunderlying: Snapshot of computed values that are available for an option underlying symbol based on the option prices on the markettimeAndSale: Trade in a tape of trades for a given instrumentseries: Snapshot of computed values that are available for option series for a given underlying symbol based on the option prices on the market
With the instrument:
symbols = ['SPY', 'AAPL', 'IBM', 'MSFT', 'DIA', 'XLF', 'GOOG', 'AMZN', 'TSLA', 'SPX', 'HPQ', 'CSCO', 'INTC', 'AXP']- The Deephaven-core dependencies are required to build and run this project.
To launch the latest release, you can clone the repository via:
git clone https://github.com/deephaven-examples/redpanda-dxfeed-financial-data.git
cd redpanda-dxfeed-financial-dataA start script will install the needed python modules. It will also start the Deephaven IDE, Redpanda images and execute the python application to produce the stream from dxFeed.
To run it, execute:
./redpanda_start.shRunning this script will start several Docker containers that work together to pull data from dxFeed, publish that to various Kafka topics and then are consumed by Deephaven. To view the data navigate to http://localhost:10000/ide. To view the tables you might need to refresh the page by going in the top right Panels tab and clicking the circular refresh button.
Tables and sample query will appear in the IDE.
We can then write other queries which combine and/or aggregate data from these streams. This query uses an as-of join to correlate trade events with the most recent bid for the same symbol. A sample query is provided, edit this query write your own! Visit https://deephaven.io/ to learn more!
:::note
Some tables only work during trading hours, because the before/after hours events lack real timestamps. This data is demo data, it is provided here for demonstrative use without any warranty as to the accuracy, reliability, or completeness of the data.
:::