Update IP addresses to and vagrant configs for example deployment

[FrankApiyo]

Upgrade Plan

• Ensure the ansible roles and inventories for the following services are updated before date of production deployment
  • Onadata
  • Pgbouncer
  • Nginx
  • PostgreSQL
  • RabbitMQ
  • Memcached
• Notify the QA team that an upgrade will be happening and a QA engineer will be needed after the upgrade is done
• [Optional] Take a snapshot of the servers if possible before performing any upgrades
• Stop all traffic to the OnaData servers; Ensure the API is not receiving any new submissions during the upgrade.
  • Stop the service by running systemctl stop onadata.service celery*
  • If the monit service is running, stop the service by running systemctl stop monit to avoid restarting the onadata service during the upgrade
• Backup existing configurations i.e onadata application settings i.e local_settings.py, uwsgi configs i.e uwsgi.ini, nginx configs and pgbouncer
• Take a full backup of the PostgreSQL Database using pg_dump
pg_dump --dbname onadata -h <db host> -U onadata --schema-only > schema-dump.sql and pg_dump --dbname onadata -h <db host> -U onadata > full-dump.sql
  • Ensure Backup/Database Dump file is present in external storage location
  • [Optional] Ensure to have a dump of the database schema
• [Optional] Run the sudo apt-get dist-upgrade command to update currently installed softwares
• Upgrade the servers from 18.04 to Ubuntu 22.04 via do-release-upgrade; Note: Do not restart the service when prompted to restart.
  • Restart the ssh and sshd services; Ensure that they are running as expected
  • Try to connect from a separate terminal
  • Restart the virtual machine
  • Verify that we still have connection to the server
  • Verify that the release version is 22.04; lsb_release -a
• Upgrade PostgreSQL version from 10.23 to 13.12
  • Confirm that it's possible to restore the PostgreSQL backups
  • Ensure PostgreSQL service is running after the OS update
  • Ensure that the DB schema has the correct DDL after the upgrade i.e indexes and constraints
• Deploy OnaData v3.14.2
  • Ansible deploy v3.14.2
    • ansible-playbook -i inventory deploy-everything.yml --limit onadata
  • Verify the migrations that need to be run: python manage.py showmigrations
  • Run migrations: python manage.py migrate
  • Check state of the service: systemctl status onadata.service celeryd-onadata celerybeat-onadata
  • Verify that you can still access some of the database entries:
    python from onadata.apps.logger.models import Instance Instance.objects.first()
  • Check Nginx logs and try to access the site to verify that the service is up and running
• Upgrade RabbitMQ to 3.10.7
  • Ansible deploy RabbitMQ v3.10.7 ansible-playbook -i inventory deploy-everything.yml --limit rabbitmq
  • Ensure that the rabbitmq service is running. Run sudo systemctl status rabbitmq-server & sudo rabbitmqctl status
  • Verify that the OnaData admin user has been created and has the correct tags with correct credentials. Run rabbitmqctl list_users. Expected entry: admin [administrator]
  • Ensure that onadata is able to connect to RabbitMQ and exports are working fine
• Deploy Nginx v1.25.2
  • Check nginx is running systemctl status nginx.service

[FrankApiyo]

Guid for upgrading postgres: https://severalnines.com/blog/upgrading-postgresql13/

[FrankApiyo]

Upgrading postgress:

Upgrading PostgreSQL using `pg_upgrade` on Ubuntu 20.04 involves several steps. The basic idea is to install a new version of PostgreSQL, initialize a new cluster, and then use `pg_upgrade` to migrate data from the old cluster to the new one. Here are the steps to upgrade PostgreSQL using `pg_upgrade`:

1. **Backup your database**: Before making any changes, it's crucial to backup your existing PostgreSQL database in case anything goes wrong. You can use the `pg_dump` utility for this purpose:

   ```bash
   pg_dumpall -U postgres > backup.sql
   ```

2. **Install the new version of PostgreSQL**: Use the `apt` package manager to install the desired version of PostgreSQL. For example, if you want to upgrade from PostgreSQL 12 to PostgreSQL 13:

   ```bash
   sudo apt update
   sudo apt install postgresql-13
   ```

3. **Stop the old PostgreSQL server**: Before you proceed, stop the old PostgreSQL server:

   ```bash
   sudo systemctl stop postgresql
   ```

4. **Initialize a new PostgreSQL cluster with the new version**: You can use the `pg_createcluster` command to create a new cluster with the desired version. For example, for PostgreSQL 13:

   ```bash
   sudo pg_createcluster 13 main --start
   ```

5. **Stop the newly created cluster**: You'll need to stop the new cluster to avoid conflicts:

   ```bash
   sudo systemctl stop postgresql@13-main
   ```

6. **Use `pg_upgrade` to migrate data**: Run `pg_upgrade` to migrate data from the old cluster to the new one. Ensure that you specify the correct old and new cluster data directories:

   ```bash
   sudo pg_upgrade -b /usr/lib/postgresql/12/bin -B /usr/lib/postgresql/13/bin -d /var/lib/postgresql/12/main -D /var/lib/postgresql/13/main
   ```

   Replace `/usr/lib/postgresql/12/bin` and `/usr/lib/postgresql/13/bin` with the paths to the `bin` directories for the old and new PostgreSQL versions. Replace `/var/lib/postgresql/12/main` and `/var/lib/postgresql/13/main` with the data directories for the old and new clusters.

7. **Start the new cluster**: After the data migration is complete, you can start the new PostgreSQL cluster:

   ```bash
   sudo systemctl start postgresql@13-main
   ```

8. **Check for errors**: After the migration, it's a good idea to check the PostgreSQL logs and test your applications to ensure everything is working correctly.

9. **Clean up**: Once you're confident that the upgrade was successful, you can remove the old PostgreSQL version if desired:

   ```bash
   sudo apt remove postgresql-12
   ```

Remember to adapt the specific version numbers and paths according to your PostgreSQL setup and requirements. Always make backups and perform upgrades in a test or non-production environment before attempting this in a production system.

[FrankApiyo]

pg_lsclusters