Feature - Separate containers for harvest (gather/consumer) and xloader

[mjanez]

Info

Background

First a deployment was developed with all workers in the background in the main CKAN container, but in order to handle high availability and scalability, it is proposed to separate the workers into isolated containers, connected to each other.

More info: #102

To do

To allow horizontal scalability, it would be better to use workers in separate containers rather than as background cron jobs within the `ckan' container.

They communicate with the ckan container via redis, and need access to DB, solr and filesystem (config, storage, logs, etc.)

References:

• ckanext-harvest production info
• Consider having worker containers for background processes ckan/ckan-docker#118
• ckan-docker/Datapusher replacement

Example

  gather_consumer:
    << : *default-common-ckan
    image: ${COMPOSE_PROJECT_NAME}-dcatapit_ckan:latest
    container_name: ${COMPOSE_PROJECT_NAME}-gather
    depends_on:
      ckan:
        condition: service_healthy
    entrypoint: /consumer-entrypoint.sh
    volumes:
      - ckan-data:/var/lib/ckan
      - ckan-config:/etc/ckan
      - ckan-logs:/var/log/ckan
      - ./docker/ckan/ckan_harvesting_gather.conf:/etc/supervisor/conf.d/ckan_harvesting.conf

  fetch_consumer:
    << : *default-common-ckan
    image: ${COMPOSE_PROJECT_NAME}-dcatapit_ckan:latest
    container_name: ${COMPOSE_PROJECT_NAME}-fetch
    depends_on:
      ckan:
        condition: service_healthy
    entrypoint: /consumer-entrypoint.sh
    volumes:
      - ckan-data:/var/lib/ckan
      - ckan-config:/etc/ckan
      - ckan-logs:/var/log/ckan
      - ./docker/ckan/ckan_harvesting_fetch.conf:/etc/supervisor/conf.d/ckan_harvesting.conf