Skip to content

v0.9
Compare
Choose a tag to compare
@ghukill ghukill released this 21 May 12:48
· 310 commits to master since this release
v0.9
a6de141

Release Notes - v0.9

For changes see CHANGELOG

Upgrading to v0.9 (Ansible/Vagrant Server)

This version v0.9 introduces some changes at the server level that the normal update utility cannot address. Steps to manually make these changes are outlined below for Ansible/Vagrant Server build or Docker deployment.

Switch from standalone Spark cluster to running in local mode

To more closely align with the Docker deployment, and reduce some complexity without much/any noticeable change to performance, this switches the Spark application that is created by Livy from running in a standalone Spark cluster to running in what is called "local" mode, using N-threads.

This is optional, but recommended, as future updates and releases will likely assume running in local mode.

  1. First step is to stop Spark cluster, if running. The following can be run from anywhere:
# note the trailing colon, which is required
sudo supervisorctl stop spark:
  1. Second, is to prevent the Spark cluster from autostarting on reboot. Modify the file /etc/supervisor/supervisord.conf, and then under the sections [program:spark_driver] and [program:spark_worker], change autostart and autorestart to false. They should then look something like the following:
[program:spark_driver]
environment =
    SPARK_MASTER_HOST=0.0.0.0
command=/opt/spark/bin/spark-class org.apache.spark.deploy.master.Master
directory=/opt/spark/
autostart = false
autorestart = false
stdout_logfile = /var/log/spark/spark_driver.stdout
stderr_logfile = /var/log/spark/spark_driver.stderr
user = combine

[program:spark_worker]
command=/opt/spark/bin/spark-class org.apache.spark.deploy.worker.Worker spark://localhost:7077
directory=/opt/spark
autostart = false
autorestart = false
stdout_logfile = /var/log/spark/spark_worker.stdout
stderr_logfile = /var/log/spark/spark_worker.stderr
user = combine

To apply these changes, run the following:

sudo supervisorctl reread
sudo supervisorctl update

For reference sake, the configurations and binaries to run this standalone Spark cluster remain in the build, in the event it is deemed helpful, or might assist in configuring with another cluster.

  1. Finally, update the parameter livy.spark.master in /opt/livy/conf/livy.conf to the following:
livy.spark.master = local[*]

To apply, restart Livy:

sudo supervisorctl restart livy

Update DPLA's Ingestion3 build

As outlined in this issue, moving from forked version of Ingestion3 to pinned commits in DPLA repository to build from.

The Docker deployment ships with a .jar file already compiled, which can be used for our purposes here. Ansible/Vagrant builds as of v0.9 will build this newer, updated version of Ingestion3.

To upgrade in place:

# jump to directory where Ingestion3 jar is located
cd /opt/ingestion3/target/scala-2.11

# backup previous Ingestion3 jar file
mv ingestion3_2.11-0.0.1.jar ingestion3_2.11-0.0.1.jar.BAK

# download pre-built .jar file
wget https://github.com/WSULib/combine-docker/raw/15938f053ccdfad08e41d60e6385588a064dc062/combinelib/ingestion3_2.11-0.0.1.jar

Then, restart Livy:

sudo supervisorctl restart livy

Finally, run update script as per normal

cd /opt/combine
source activate combine
git checkout master
git pull
pip install -r requirements.txt
./manage.py update --release v0.9

Upgrading to v0.9 (Docker)

From the Combine-Docker git repository directory on your host machine, pull changes:

git pull

Checkout tagged release:

git checkout v0.9

Run update script:

./update_build.sh
Assets 2
Loading