Meetup Data Scraper

Dowload group & events from Meetup-API into Elasticsearch to make a fulltext search on every event.

Data flow & Container diagram

Basic Commands

Flask Commands

Show help:

$ docker-compose -f local.yml run flask flask

Migrate all models into Elasticsearch:

$ docker-compose -f local.yml run flask flask migrate_models

Load a single Meetup Group:

$ docker-compose -f local.yml run flask flask get_group MeetupGoupUrlName
$ docker-compose -f local.yml run flask flask get_group --sandbox True # load sandbox group

Load all groups from JSON files from a path, default path is /app/meetup_groups:

$ docker-compose -f local.yml run flask flask get_groups # use default path
$ docker-compose -f local.yml run flask flask get_groups /app/meetup_groups # use custom path
$ docker-compose -f local.yml run flask flask get_groups --load_events False # don't load events from groups

Load new meetup zip codes from meetup.com within a boundingbox:

$ docker-compose -f local.yml run flask flask load_zip_codes 47.2701114 55.099161 5.8663153 15.0418087 # germany
$ docker-compose -f local.yml run flask flask load_zip_codes 45.817995 47.8084648 5.9559113 10.4922941 # switzerland
$ docker-compose -f local.yml run flask flask load_zip_codes 46.3722761 49.0205305 9.5307487 17.160776 # austria

Load groups for a country:

$ docker-compose -f local.yml run flask flask load_groups --country DE # germany
$ docker-compose -f local.yml run flask flask load_groups --country CH # switzerland
$ docker-compose -f local.yml run flask flask load_groups --country AT # austria

Start api-server:

$ docker-compose -f local.yml up flask

Type checks

Running type checks with mypy:

$ docker-compose -f local.yml run flask coverage run -m mypy /app

Test coverage

To run the tests, check your test coverage, and generate an HTML coverage report:

$ docker-compose -f local.yml run flask coverage run -m pytest
$ docker-compose -f local.yml run flask coverage run -m coverage html
$ open htmlcov/index.html

Running tests with py.test

$ docker-compose -f local.yml run flask coverage run -m pytest

Create Docs

For creating the docs use from sphinx the makefile:

$ docker-compose -f local.yml run flask make --directory docs html
$ open docs/_build/index.html

Than the docs will be generated into docs/_build/html.

For more options check out the help:

$ docker-compose -f local.yml run flask make --directory docs help