Attention: The code in this repository is intended for experimental use only and is not fully tested, documented, or supported by SingleStore. Visit the SingleStore Forums to ask questions about this repository.
- Code tour
- Data simulator
- Stored Procedures
- Web application
- Running the web interface locally
- Manually running the demo in pure SQL
This application is a demo of how to use SingleStore to serve ads to users based on their behavior and realtime location. The demo is based on location, purchase, and request history from millions of simulated subscribers for a hypothetical service company.
This application runs entirely within the web-browser and can be accessed publicly at https://digital-marketing.labs.singlestore.com
If you have any questions or issues, please file an issue on the GitHub repo or our forums.
This repository contains a data simulation tool written in go as well as a web application written in typescript. Refer to the following file tree for an overview of the code.
.
├── cmd
│ └── simulator (data simulation tool entrypoint)
├── data
│ ├── cities.ndjson (coordinates of many cities worldwide)
│ └── vendors.json (randomly generated vendors)
├── gen (go package for data generation)
├── output (go package for writing Parquet and JSON files out to the cloud)
├── README.md (this file!)
├── sql (all of the SQL data definition language (DDL) used in the demo)
├── util (go utility package)
└── web (the web application)
└── src
├── components (React components)
├── data (React hooks and all networking code)
│ ├── client.ts (HTTP client for querying SingleStore)
│ ├── queries.ts (SQL queries)
│ ├── recoil.ts (Recoil atoms and selectors)
└── main.tsx (web application entrypoint)
The simulator is written in Golang. Internally it is modular, allowing it to output data in multiple formats to different storage providers. The recommended production outputter is S3/Parquet.
The simulator can be configured using command line flags. By default it will write to /tmp/subscriber-sim (will error if this directory is missing). You can run it using default settings like so:
mkdir -p /tmp/subscriber-sim
go run cmd/simulator/main.goAfter running this command using default settings a bunch of JSON files will be written to /tmp/subscriber-sim. Inspect some of those files to get an idea of what the generated data looks like.
The gen package is responsible for generating the data. It has a driver which leverages many concurrent goroutines to iterate through all of the subscribers as fast as possible.
Data generated by the simulator is designed to be loaded repeatedly to simulate device streams coming from many different locations over time. To help achieve this, timestamps are not included in the generated data and location is output as coordinates on the unit circle (centered at [0,0])
The output package is responsible for encoding and uploading the data to a target. The primary interface it exposes is output.Writer. The output package supports the following encodings/output targets:
- Encodings
- Parquet
- JSON
- Targets
- Local filesystem
- Anything supported by https://gocloud.dev/blob (e.g. S3)
This demo makes extensive use of stored procedures which contain all of the logic required to load the data, segment users, and match subscribers to offers. You can find all of procedure code in ./sql/procedures.sql. Many procedures also depend on views which are all defined in ./sql/schema.sql.
The user interface is implemented as a single-page web application which runs entirely in the browser. The interface connects to any SingleStore cluster which has the Data API enabled. To get started quickly, we recommend either using our Docker image or the SingleStore Managed Service.
This will not work on a Mac M1 or ARM hardware
-
Sign up for a free SingleStore license. This allows you to run up to 4 nodes up to 32 gigs each for free. Grab your license key from SingleStore portal and set it as an environment variable.
export SINGLESTORE_LICENSE="singlestore license"
-
Start a SingleStore cluster-in-a-box using Docker:
docker run -it \ --name ciab \ -e LICENSE_KEY=${SINGLESTORE_LICENSE} \ -e ROOT_PASSWORD=test \ -e HTTP_API=on \ -p 3306:3306 -p 9000:9000 -p 8080:8080 \ singlestore/cluster-in-a-box docker start ciab -
Open the Digital Marketing demo in Chrome or Firefox
-
Plug in the connection details:
| Key | Value |
|---|---|
| Host & Port | http://localhost:9000 |
| Username | root |
| Password | test |
- Sign up for $500 in free managed service credits.
- Create a S-00 sized cluster in the portal
- Open the Digital Marketing demo in Chrome or Firefox
- Plug in the connection details (replacing placeholders as needed):
| Key | Value |
|---|---|
| Host & Port | https://CLUSTER_CONNECTION_URL |
| Username | admin |
| Password | CLUSTER_ADMIN_PASSWORD |
Rather than using https://digital-marketing.labs.singlestore.com it's possible to run the web interface locally via Docker. You can do this like so:
docker run -d --name demo -p 3000:3000 ghcr.io/singlestore-labs/demo-realtime-digital-marketingAfter running the above command the web interface will be running at http://localhost:3000.
This entire demo can be run standalone on any SingleStore cluster without needing the Web UI. To do this, run all of the SQL scripts in the ./sql folder in the following order:
- functions.sql
- schema.sql
- procedures.sql
- seed.sql
- pipelines.sql (requires variable replacement, see below)
We can do this on the command line like so:
mysql -u root -h 172.17.0.3 -ptest -e "create database martech"
mysql -u root -h 172.17.0.3 -ptest martech < sql/functions.sql
mysql -u root -h 172.17.0.3 -ptest martech < sql/schema.sql
mysql -u root -h 172.17.0.3 -ptest martech < sql/procedures.sql
mysql -u root -h 172.17.0.3 -ptest martech < sql/seed.sql
sed 's/${PARTITIONS}/2/;s/${SCALE_FACTOR}/v2\/100k-2p/' pipelines.sql | mysql -u root -h 172.17.0.3 -ptest martechNote that we are replacing the PARTITIONS and SCALE_FACTOR variables in pipelines.sql with acceptable values. See scalefactors.ts for additional options.