Skip to content

4.0.0 Vireo Deployment AWS

Frank Smutniak edited this page Sep 20, 2019 · 17 revisions

Vireo 4 AWS Deployment

This deployment was successfully tested on a AWS t2.small. A t2.micro did not have enough memory for java to create a war file. Installations may vary on other systems.

A docker deployment is available at https://github.com/TexasDigitalLibrary/Vireo4Docker

Requirements

The vireo install will need PostgreSQL. The H2 database can be used for testing but PostgreSQL will be needed for persistence across reboots. MySQL is not yet supported.

The base system will need npm, nodejs, grunt and maven. Nodejs is needed for processing assets.

The base system will also need git but this typically comes with the base linux system.

Vireo will also need java 1.8 developer kit.

Edits to src/main/resources/application.yaml will also be needed.

Finally the install should use a proxy web server such as Apache2 or Nginx.

Sign in as root

sudo bash --login

Call yum update to be ready for other installs.

yum update

Verify that git is installed.

git --version

Typically it is already part of the base linux system but it may need to be explicitly installed. If so:

yum install git

Install Maven

wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo
sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo
yum install -y apache-maven

Install Java

yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel

There may have been an older java installed with the base system so find the newly installed java

alternatives --config java
    #select the correct number for java 1.8
java -version

export JAVA_HOME=/usr/lib/jvm/java-1.8.0

Create an account:

Set up an account under which the deploy will be performed. For these instructions we'll use the account named 'vireo4'

Add user 'vireo4' and then enter a requested initial password.

adduser vireo4

Set password for vireo4:

passwd vireo4

Give user 'vireo4' root level privileges:

visudo

Add the following line beneath the similar line for root:

    vireo4    ALL=(ALL) ALL 

Exit editor using ':wq'

Create a directory for assets:

Uploaded documents are stored in the operating systems directory structure. Create a [assets_store] directory such as /opt/vireo for these files.

mkdir /opt/vireo/
chown vireo4 /opt/vireo/
chgrp vireo4 /opt/vireo/

Sign out from the root account and sign in with the 'vireo4' account.

Alternatively you can use the vireo4 account with:

sudo -u vireo4 bash

Install npm, node, and grunt

This is the most system specific problem for deployments. This section may be different for later versions of ubuntu or AWS variants.

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash

Follow the instructions that the above command printed out:

export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"  # This loads nvm bash_completion

Then verify the version and install node and grunt-cli:

nvm --version

nvm install node

npm --version
node --version

npm install -g grunt-cli

Put

export JAVA_HOME=/usr/lib/jvm/java-1.8.0

and

export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"  # This loads nvm bash_completion

in ~/.bashrc

Install Vireo 4

Create a directory in vireo4 such as /home/vireo4/etd/ in which to clone Vireo:

mkdir /home/vireo4/etd/
cd /home/vireo4/etd/
git clone https://github.com/TexasDigitalLibrary/Vireo.git
cd Vireo

Use the default master branch or change to the latest point release branch of Vireo4

git checkout 4.0.x

Test!

Try to start vireo4 using spring boot to verify that it has everything it needs to build and run.

mvn clean spring-boot:run -Drun.arguments=console

After several minutes of downloads and processing you will see a generate> prompt. If you get this far then your system is set up properly. Use Control-C or other means to kill this process. If it failed review previous steps and verify the version of OS you are using. Most problems are related to getting the right npm and node versions for the OS.

If you point your browser to your URL or IP address followed by the port number, e.g. http://example.edu:9000, you will see a poorly formatted vireo page. This is OK. The presentation of this page will be remedied by changes to application.yaml further down.

Install PostgreSQL, init, and start the server:

sudo yum install -y postgresql postgresql-server

sudo postgresql-setup initdb
sudo service postgresql start
psql --version

This will give you a 'postgres=#' prompt.

Create role 'vireodb', give privileges, and create database vireo.

postgres=# CREATE ROLE vireodb WITH LOGIN PASSWORD '[your_vireo_postgres_password]';
postgres=# ALTER ROLE vireodb CREATEDB;
postgres=# CREATE DATABASE vireo;
postgres=# GRANT ALL PRIVILEGES ON DATABASE vireo TO vireodb;
postgres=# \q

Set postgres role password \password

Exit from the postgres account so you will be back in the vireo4 account.

exit

Edit /var/lib/pgsql/data/pg_hba.conf. (may require sudo) Change lines: # "local" is for Unix domain socket connections only local all all peer # IPv4 local connections: host all all 127.0.0.1/32 ident to # "local" is for Unix domain socket connections only local all all md5 # IPv4 local connections: local all all md5

Restart postgres

service postgresql restart

Sign in under the postgres account and then use psql to enter commands:

psql [dbname] [username]
psql vireo vireodb

OR

psql -U vireodb vireo

Determine the server's URL

Your URL can be found on the AWS console under EC2 listings. It is of the form 'ec2-[IP ADDRESS].compute-1.amazonaws.com

e.g. ec2-111-222-333-444.compute-1.amazonaws.com

You can also find the ip address with either:

curl http://checkip.amazonaws.com

or

curl http://icanhazip.com

Edit src/main/resources/application.yaml

A few values will be set at the command line but some should be set in the application.yaml file.

Set a port. For this install we will change port 9000 to port 8080 as we'll be running with embedded tomcat:

server:
  #port: 9000
  port: 8080

Set the database access by commenting out the H2 section and uncommenting the platform, url, and driverClassName in the PostgreSQL entry. Modify as needed for your particular setup (i.e. you elected a different port number):

spring:
  datasource:
    platform: postgresql
    url: jdbc:postgresql://localhost:5432/vireo
    driverClassName: org.postgresql.Driver

Also under datasource, set the database role name and password (as set in the Install PostreSQL section):

    username: vireodb
    password: [your_vireo_postgres_password]

Comment/uncomment the jpa entry to select the correct database-platform for PostgreSQL:

  jpa:
    database-platform: org.hibernate.dialect.PostgreSQLDialect

In the same section change:

    hibernate.ddl-auto: create-drop

To:

    hibernate.ddl-auto: update

Set the application URL with your url such as http://etd.example.edu or ec2-111-222-333-444.compute-1.amazonaws.com.

app:
  url: [your url]

Note that the values which appear under info: build: will be filled either via the pom.xml or the command line.

Finally, set the email relay and addresses:

  email:
    host: [your_smtp_relay]				#e.g. smtp-relay.example.edu
    from: [your_outbound_address]		#e.g. [email protected]
    replyTo: [your_replyto_address]		#e.g. [email protected]

Create user accounts (optional):

If you want to give users some sample accounts to experiment with, you will need to start vireo enabled with a command line interface. Use the -Drun.argumets=console flag, which only works when started with spring-boot.

mvn clean spring-boot:run -Drun.arguments=console

Once you get the prompt:

generate> accounts [number of accounts]
  #e.g. accounts 5
generate> admin_accounts [number of admin accounts]

If you also want to generate some sample data:

generate> generate [number of samples]

If you sign back in to the database you can see the accounts:

psql -U vireodb vireo
postgres=# SELECT * FROM weaver_users;
postgres=# \q

Build production mode war file with assets.uri specified:

mvn clean package -Dproduction -Dassets.uri=file:[assets_path]
	#e.g.  mvn clean package -Dproduction -Dassets.uri=file:/opt/vireo

Run it with the following command using the appropriate war file name based on [version-tag]:

nohup java -jar target/vireo-4.0.0-[version-tag].war &
	e.g.  nohup java -jar target/vireo-4.0.0-RC.war &

Run as a Service

Create the file /home/vireo4/etd/vireo4.sh and add the following 2 lines using the appropriate war file name based on [version-tag].

#!/bin/sh
sudo -u vireo4 /usr/bin/java -jar /home/vireo4/etd/Vireo/target/vireo-4.0.0-[version-tag].war

Change the file permissions with:

chmod 0764 vireo4.sh

Create /etc/systemd/system/vireo4.service - this may require sudo.

[Unit]
Description= Vireo 4 ETD service
[Service]
User=vireo4
#This must point to the Vireo base path
WorkingDirectory=/home/vireo4/etd/Vireo/
#This is a path to a bash script which launches war file with embedded tomcat8
ExecStart=/home/vireo4/etd/vireo4.sh
SuccessExitStatus=143
TimeoutStopSec=10
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target

Reload the systemd configuration and enable the vireo4 service to start automatically upon reboot:

sudo systemctl daemon-reload
sudo systemctl enable vireo4.service

Start, check status, stop the service, or check if it is enabled with:

sudo systemctl start vireo4
sudo systemctl status vireo4
sudo systemctl stop vireo4
sudo systemctl is-enabled vireo4

To test:

sudo reboot

Run with SSL Proxy

Since you are running on port 8080 you will likely want to have vireo4 served via a proxy server with SSL.

This example uses nginx.

sudo yum install -y nginx

This may tell you that it is not found but then direct you a particular version of 'amazon-linux-extras' to install it.

sudo amazon-linux-extras install nginx1.12

FSS

Follow? https://aws.amazon.com/blogs/compute/extending-amazon-linux-2-with-epel-and-lets-encrypt/

We'll need a certificate. You should use your institution's key, but for now we'll use a test key:

cd /etc/pki/tls/private
sudo openssl genrsa -out custom.key
sudo chown root.root custom.key
sudo chmod 600 custom.key
sudo openssl req -new -key custom.key -out csr.pem

Follow the prompts: Common Name must be the url such as

ec2-111-222-333-444.compute-1.amazonaws.com

This will have created the custom.key and csr.pem files.

Edit /etc/nginx/conf.d/virtual.conf such that it looks like:

server {
    listen       443 ssl;
    listen       [::]:443 ssl;
    server_name  ec2-3-222-186-66.compute-1.amazonaws.com;

    location / {
        proxy_pass http://127.0.0.1:8080;
    }

    ssl on;

    ssl_certificate /etc/pki/tls/private/csr.pem;
    ssl_certificate_key /etc/pki/tls/private/custom.key;
}

Stop the vireo4 service, rebuild the war file, and restart the vireo4 service.

You may also need to clean your browser's cache, especially if you see the vireo page with no formatting or other assets.

Nginx should have picked up the location changes. If not then restart the service.

	sudo service vireo4 restart
	sudo service nginx restart

Done!

You should be able to access vireo4 at your such as https://etd.example.edu

Post Install

Register

Register your account on the vireo4 website.

Grant privileges

There are 4 types of accounts: Admin, Manager, Reviewer, and Student. The Admin account will manage the site for the whole institution. You may need to manually set an initial Administrative permission on an account. This account can then be used to set permissions on other accounts through the user interface.

sudo su - postgres
psql
postgres=# UPDATE weaver_users SET role='ROLE_ADMIN' WHERE id=[user_id_in_weaver_users_table_you_want_to_make_admin];
postgres=#\q
exit

Database backups

While signed in under the unix vireo account create a directory for the backups and create a crontab entry

> mkdir /home/vireo/backups/
> crontab -e

Create an entry such as:

0 1 * * * pg_dump vireo > /home/vireo/backups/vireo_backup_`date +\%Y_\%m_\%dT\%H:\%M:\%S`.sql

This creates a file in your backups directory with a date stamp such as:

vireo_backup_2019_05_13T09:53:47.sql

To restore use:

> sudo -u postgres psql
postgres=# DROP DATABASE vireo;
postgres=# CREATE DATABASE vireo;
postgres=# \q
> sudo -u postgres psql -U postgres vireo < /tmp/vireo_backup_2019_05_13T09\:53\:47.sql 

Then restart:

> nohup java -jar target/vireo-4.0.0-SNAPSHOT.war &

Miscellaneous

If you are having trouble accessing your database, verify the port on which which PostgreSQL is running. The default is 5432.

sudo -u postgres psql
postgres=# SELECT * FROM pg_settings WHERE name = 'port';
postgres=# \q
exit

If you prefer running vireo4 as a jar file this can be generated by changing one line in the pom.xml file:

Edit pom.xml to use jar instead of war. Change:

  <packaging>war</packaging>

to

  <packaging>jar</packaging>