-
Notifications
You must be signed in to change notification settings - Fork 0
EuropeanaRS installation in a production environment
- A computer with:
- Ubuntu 14.04
- Internet connection
We will begin by ensuring our default user, from now on {username}, has root permission by adding it to the sudoers list:
sudo adduser {username} sudo
You will be prompted for the root password.
Next, we will ensure Ubuntu packages are updated to the latest version. For that purpose we will update the package database and the global system:
sudo apt-get update; sudo apt-get dist-upgrade
To ensure a sane fallback locale, execute the following commands:
export LANGUAGE=en_US.UTF-8
export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8
sudo locale-gen en_US.UTF-8
sudo dpkg-reconfigure locales
sudo update-locale LC_ALL="en_US.utf-8"
##Installing the production environment
###Installing dependencies The following packages are needed dependencies for our environment, and can be installed by executing:
sudo apt-get install git-core libxml2-dev nodejs libxslt1-dev libmysqlclient-dev libsqlite3-dev libpq-dev make apache2-threaded-dev apache2-mpm-worker libcurl4-openssl-dev
###Installing ruby First, we'll import the ruby repo gpg key:
gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
Next, we'll install a local ruby version 2.2.1 by rvm method
curl -L https://get.rvm.io | bash -s stable --ruby --autolibs=enable --auto-dotfiles
source /home/{username}/.rvm/scripts/rvm
echo 'gem: --no-rdoc --no-ri' >> ~/.gemrc
gem install bundler
echo "export rvmsudo_secure_path=1">> /home/{username}/.bashrc
###Installing Sphinx EuropeanaRS requires sphinx 2.2.10 or higher. The easiest way to install a valid and compliant version of sphinx is to download and install the corresponding deb package:
wget http://sphinxsearch.com/files/sphinxsearch_2.2.10-release-0ubuntu12~precise_amd64.deb
sudo dpkg -i sphinxsearch_2.2.10-release-0ubuntu12~precise_amd64.deb
###Installing postgres EuropeanaRS requires postgres 9.3.9+ as database management system. To install it:
sudo apt-get install postgresql postgresql-contrib
To assign a password to the postgres user:
sudo -u postgres psql postgres
posgres#> ALTER USER postgres with password '{your_db_password}';
posgres#> \q
To create the europeanars_production database:
sudo -u postgres createdb europeanars_production
To use the passenger app server for ruby, we'll install and configure it with the following commands:
sudo apt-get install libapache2-mod-passenger
sudo a2enmod passenger
sudo service apache2 restart
We will create the folder structure by executing the following commands:
sudo mkdir /u
sudo chown {username}:root /u
mkdir /u/apps
mkdir /u/apps/europeanars
mkdir /u/apps/europeanars/shared/
mkdir /u/apps/europeanars/shared/log
mkdir /u/apps/europeanars/shared/pids
Next, we'll download the source code from git into /u/apps/europeanars by executing the following commands:
cd /u/apps/europeanars
git clone https://github.com/agordillo/EuropeanaRS current
Copy the scripts into the shared folder:
cp /u/apps/europeanars/current/extras/check_daemon.sh /u/apps/europeanars/shared/check_daemon.sh
Copy the cron task:
sudo cp /u/apps/europeanars/current/extras/cron/europeanars_cron /etc/cron.d/europeanars_cron
Copy the thinking sphinx configuration file into the shared folder:
cp /u/apps/europeanars/current/config/thinking_sphinx.yml.example /u/apps/europeanars/shared/thinking_sphinx.yml
We will create the database file in /u/apps/europeanars/shared/database.yml , which will include the settings for accessing the postgresql database previously configured.
cp /u/apps/europeanars/current/config/database.yml.example /u/apps/europeanars/shared/database.yml
The /u/apps/europeanars/shared/database.yml file should have the following content:
production:
adapter: postgresql
database: europeanars_production
encoding: utf8
host: localhost
username: postgres
password: {your_db_password}
The configuration of a EuropeanaRS instance can be specified through the application_config.yml file.
cp /u/apps/europeanars/current/config/application_config.yml.example /u/apps/europeanars/shared/application_config.yml
Edit the /u/apps/europeanars/shared/application_config.yml file to specify the settings of your EuropeanaRS instance.
For more information about how to edit this file, please check setting up a EuropeanaRS instance.
production:
domain: "localhost:3000"
secret_token: "" # Execute 'bundle exec rake secret' to generate a new secret_token and copy it here
europeana:
api_key: "" #Request an API key here: http://labs.europeana.eu/api/introduction
private_key: ""
oauth2_client_id: ""
oauth2_client_key: ""
oai_pmh_key: "" #Request an access key for the OAI-PMH service here: http://labs.europeana.eu/api/oai-pmh-access-key
# my_europeana: "mimic" #To use a mimic of the MyEuropeana API instead of the real one. This should be used only for development.
facebook: #Get Facebook credentials for your app from https://developers.facebook.com/
app_id: ""
app_secret: ""
mail:
no_reply_mail: "[email protected]"
main_mail: "[email protected]"
type: "SMTP" #Type can be "SENDMAIL" or "SMTP" (default)
# gmail_credentials:
# username: ""
# password: ""
settings: #settings for the Recommender System
resource_types: ["TEXT","VIDEO","SOUND","IMAGE","3D"]
max_text_length: 50 #Maximun length in characters of the texts that will be taking into account when measuring semantic distances. High values could lead to performance issues.
max_user_los: 2 #Maximum number of LOs of the users to be processed on the RS. High values could lead to performance issues.
europeanars_database:
max_preselection_size: 500 #Maximun number of Learning Objects retrieved by the search engine in the preselection phase of the recommendations. High values could lead to performance issues.
europeana_database:
max_preselection_size: 500 #Maximun number of Learning Objects retrieved from Europeana in the preselection phase. High values could lead to performance issues.
default_settings: #Default settings for the Recommender System
database: "EuropeanaRS" #database used for generate the recommendations. Accepted values: "EuropeanaRS", "Europeana", "All"
query: ""
preselection_filter_resource_type: "true"
preselection_filter_languages: "true"
europeanars_database:
preselection_size: 500
europeana_database:
preselection_size: 500
europeanars_api:
require_key: "true" #if false, no app key would be required when using the EuropeanaRS API.
weights:
default_rs:
los_score: 0.4
us_score: 0.4
quality_score: 0.1
popularity_score: 0.1
default_los:
title: 0.2
description: 0.15
language: 0.5
year: 0.15
default_us:
language: 0.5
los: 0.5
popularity:
visit_count: 0.5
like_count: 0.5
filters:
default_rs:
los_score: 0
us_score: 0
quality_score: 0
popularity_score: 0
default_los:
title: 0
description: 0
language: 0
year: 0
default_us:
language: 0
los: 0
To allow remote access to our server, we'll remove the deny all directives in /etc/apache2/apache2.conf by removing the following lines:
<Directory />
Options FollowSymLinks
AllowOverride None
Require all denied
</Directory>
<Directory /usr/share>
AllowOverride None
Require all granted
</Directory>
<Directory /var/www/>
Options Indexes FollowSymLinks
AllowOverride None
Require all granted
</Directory>
To configure our web server, we need to create the file /etc/apache2/sites-available/europeanars.conf, with the following content:
<VirtualHost *:80>
ServerAdmin webmaster@localhost
ServerName example.com
ServerAlias www.example.com
DocumentRoot /u/apps/europeanars/current/public
<Directory /u/apps/europeanars/current/public/>
Options Indexes FollowSymLinks MultiViews
AllowOverride All
Order allow,deny
allow from all
Require all granted
</Directory>
RailsEnv production
ErrorLog ${APACHE_LOG_DIR}/error.log
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn
CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
Next, we will create symbolic links to the apache configuration files:
sudo rm /etc/apache2/sites-enabled/000-default.conf
sudo ln -s /etc/apache2/sites-available/europeanars.conf /etc/apache2/sites-enabled/europeanars.conf
To finish the installation:
cp /u/apps/europeanars/shared/thinking_sphinx.yml /u/apps/europeanars/current/config/thinking_sphinx.yml
cp /u/apps/europeanars/shared/database.yml /u/apps/europeanars/current/config/database.yml
cp /u/apps/europeanars/shared/application_config.yml /u/apps/europeanars/current/config/application_config.yml
cd /u/apps/europeanars/current
bundle install --without development test
bundle exec rake db:populate:install_production RAILS_ENV=production
cd ..
sudo rm -rf /u/apps/europeanars/current/
sudo chown {username}:www-data /u/apps/europeanars/shared/log/searchd.production.pid
sudo chmod 777 /u/apps/europeanars/shared/log/searchd.production.pid
The next steps are taken in a development machine.
To install a development environment you can read the following instructions:
EuropeanaRS Installation.
In the development machine, we'll customize the production environment based on our deployment file in europeanars/config/deploy/{environment}.yml. The europeanars/config/deploy/example.yml contains all the information that should be filled in:
message: "Deploying to EuropeanaRS"
repository: [email protected]:agordillo/EuropeanaRS.git
branch: master
server_url: 127.0.0.1 #put here the server URL (e.g. example.com)
username: ubuntu
keys: /home/username/.ssh/ssh-keypair #remove this line if you don't want to use ssh keys
Finally, we can deploy our EuropeanaRS application by executing:
bundle exec cap deploy DEPLOY={environment}
This way, if we define a production environment in europeanars/config/deploy/production.yml, we can deploy it with the following command:
bundle exec cap deploy DEPLOY=production
This way, we can specify several production environments for the same application in our development machine.
To enable automatic startup for EuropeanaRS and sphinx services we'll follow the next steps.
First, we should disable the autostart for sphinx if it's enabled (which is the default option in the sphinx version considered in this guide). To do that, we have to edit the /etc/default/sphinxsearch file:
#
# Settings for the sphinxsearch searchd daemon
# Please read /usr/share/doc/sphinxsearch/README.Debian for details.
#
# Should sphinxsearch run automatically on startup? (default: no)
# Before doing this you might want to modify /etc/sphinxsearch/sphinx.conf
# so that it works for you.
START=no
After that, we should modify the /etc/rc.local file to load the check_daemon.sh script on startup. At this point the file /u/apps/europeanars/shared/check_daemon.sh should exists. If not, copy it from the sources, it's located at extras/check_daemon.sh. Furthermore, we have to disable again the autostart of sphinx on the rc.local file. The final rc.local file should look like this:
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
#/usr/bin/searchd --pidfile --config /u/apps/europeanars/current/config/production.sphinx.conf
bash '/u/apps/europeanars/shared/check_daemon.sh'
exit 0
Cron tasks are needed for:
a) allowing the web server to periodically check if all services (like sphinx) are running and to restart the ones that are stopped.
b) executing scheduled tasks in the background.
At this point the file /etc/cron.d/europeanars_cron should exists. If not, copy it from the git repository, the file its located at extras/cron/europeanars_cron.
We need to edit /etc/cron.d/europeanars_cron to replace 'username' with our {username}.
Then, we should edit the crontab file with
sudo crontab -e
And add the following line:
*/10 * * * * export CAP_USER={username} && /u/apps/europeanars/shared/check_daemon.sh
Finally, for ease of use, we create the following symbolic link in our home directory:
cd /home/{username}
ln -s /u/apps/europeanars/current europeanars