Skip to content

Latest commit

 

History

History
38 lines (26 loc) · 1.1 KB

README.md

File metadata and controls

38 lines (26 loc) · 1.1 KB
Raw

CarMeta

Car Information telegram bot that scrapes Russian car services

This is my final project for "Scala Developer" course at Otus.

Architecture

architecture

Done:

[ x ] Scraping Service (AutoRussia only, no Advertisement suport)
[ x ] Aggregation Service
[ - ] Warehouse
[ - ] Telegram Frontend

Project is built using SBT

Service decomposition is done poorly, development is done like it's Mutli-Module Monolith, but .jars should be deployed separetely

Technologies

  • Scala 2.13
  • Cats Effect 3
  • fs2
  • Ciris
  • Circe
  • ScalaPB
  • Kafka
  • Redis
  • Elasticsearch

Run

Docker compose includes Kafka, Confluent Control Panel, Redis, Kibana and Elasticsearch. You can run these dependencies by running docker compose up -d

Start scraper and aggregation services, you should see that Scraper is scraping data from AutoRussia and Aggregation service is getting records from definitions topic. Data Flow could be seen in Control Panel, after that, Definitions of cars could be seen in Kibana.