NS-Disruptions

• Scrapes the disruptions from api provided by the Dutch train service (NS)

Getting Started

This project is built using

• Scala
• FS2
• Flink
• HTTP4S
• Kafka
• Postgres

It contains a couple of microservices

• Scraper service that scrapes disruptions from the NS API and dumps in a Kafka topic
• Decoder service that decodes this data and dumps successfully decoded data into Kafka
• DeDuplicator service that filters out duplicate entries and store single entries in a kafka topic
• DatabaseSink

Running the project

Docker Compose

To run the project, a docker-compose file is provided.

From the base directory, you can run

sbt docker:publishLocal && docker-compose -f docker/docker-compose.yml up

Helm

Assuming you already have a Kubernetes cluster setup, you can start one locally with minikube. You can run the project on it.

You can start with exporting your NS API Keys

export SCRAPER_AUTH_KEY=

Run the helm installation script for the provided chart

Running the tests

sbt test