Skip to content

onepercentclub/bluebottle

Repository files navigation

Project Bluebottle

https://travis-ci.org/onepercentclub/bluebottle.png?branch=master https://coveralls.io/repos/github/onepercentclub/bluebottle/badge.svg?branch=master https://requires.io/github/onepercentclub/bluebottle/requirements.svg?branch=master

The repository for Project Bluebottle, the crowdsourcing framework initiated by GoodUp.

Contributors

For those who want to try out to the BlueBottle project, here's to get started:

  1. Fork and/or clone the repository.
  2. Navigate to your local repository directory.
  3. Create a bluebottle/settings/local.py base on bluebottle/settings/secrets.py.example

Now you have the option to install the application with, or without Docker. If you wish to not use Docker, continue below. Otherwise, see :ref:`docker`.

  1. Create a virtual environment within the repository directory, for example:

    $ virtualenv env
    $ source env/bin/activate
    
  2. Make sure you have a recent Python distro (3.7+ recommended).

  3. Make sure (a recent) virtualenv is installed.

  4. Install the project:

    $ pip install -e .[test]
    
  5. Migrate the database schemas:

    $ python manage.py migrate_schemas --shared --settings=bluebottle.settings.testing
    
  6. (optional) Create a new tenant (if you haven't imported a database dump):

    $ python manage.py create_tenant
    
  7. (optional) Re-index any imported data using Elasticsearch
    • For all tenants:

      $ python manage.py tenant_command search_index --rebuild -f
      
    • If for whatever reason this doesn't work, try running the migration step for a single tenant only, and indexing that tenant's data. E.g.:

      $ python manage.py migrate_schemas -s onepercent --settings=bluebottle.settings.testing
      $ python manage.py tenant_command --schema onepercent search_index --rebuild -f
      
  8. Start the server:

    $ python manage.py runserver
    
  9. You might still need to
    • Install libraries to get pip install working, like libxmlsec1, postgresql, postgis, elasticsearch 6.x
      • Installing these is not necessary when using Docker
    • Alter your hosts file (e.g. in /etc/hosts on Linux/OSX) to contain tenants you've created like:

      127.0.0.1 tenant35.localhost
      

Docker

It is possible to run thy Python server, PostgreSQL and Elasticsearch in a Docker environment using docker-compose. To get started, make sure to download a Docker client, like Docker Desktop.

Installation:

Make sure to download Docker (e.g. Docker Desktop) first.

In your local.py file, set the DATABASES variable to the following:

DATABASES = {
    'default': {
        'ENGINE': 'bluebottle.clients.postgresql_backend',
        'HOST': 'postgres',
        'PORT': '5432',
        'NAME': 'reef',
        'USER': 'postgres',
        'PASSWORD': 'postgres',
        'DISABLE_SERVER_SIDE_CURSORS': True # this prevents issues with connection pooling
    },
}

ELASTICSEARCH_DSL = {
    'default': {
        'hosts': 'elasticsearch:9200'
    },
}
  • To start the containers (if this is your first time, all dependencies will be installed automatically):

    $ docker-compose -u -d
    
  • To import a database dump file (please note that the last two commands run without the -t flag):

    $ docker exec -it -u postgres postgres dropdb reef
    $ docker exec -it -u postgres postgres createdb reef
    $ bzcat reef-prod-current.sql.bz2 | docker exec -i -u postgres postgres psql reef
    $ echo "UPDATE clients_client SET domain_url=CONCAT(client_name, '.localhost');" | docker exec -i -u postgres postgres psql reef
    
  • If you are running into errors related to max_map_count, then run the following command in your terminal: sysctl -w vm.max_map_count=262144

Running the containers:

To run the containers:

$ docker-compose -u -d

To run one specific container:

$ docker-compose -u -d [CONTAINER_NAME]

Or on other systems (some OSX):

$ docker compose up -d elasticsearch

To shut them down:

$ docker-compose down

The environment also comes with pgAdmin included so you can inspect the local database. Navigate to http://localhost:5050 and login with these credentials:

After that, you can add a new server using the details below to inspect the PostgreSQL database:

  • Host: host.docker.internal
  • Username: postgres
  • Password: postgres

To run commands in the Python container:

$ docker exec -it bluebottle python manage.py [YOUR_COMMAND]

- For example::

    $ docker exec -it bluebottle python manage.py migrate_schemas -s onepercent --settings=bluebottle.settings.local

If you need to rebuild the container, for example when you want to apply changes after pulling the latest version of a branch, run:

$ docker compose up --build

Testing

The BlueBottle test suite can be run completely using:

  1. Install the dependencies

    $ pip install -e .[test,dev]

  2. Create test db and restore testdata

    $ createdb test_reef $ psql test_reef < testdata.sql

  3. Run the tests

    $ python manage.py test -k