- Operating system install
- User setup and management
- Set up networking
- Install PostgreSQL and related software
- Virtual network computing (remote desktop)
- Install R and supporting programs
- Install git and related software
- Other stuff
Steps followed to install Debian 8 Jessie (stable) and base software on a server. Workstation is a Puget Systems Obsidian, with 32 GB RAM, 500 GB SSD, and Intel Xeon 3.6 GHz Quad-core.
A net install .iso of Debain was downloaded for amd64, and put on a USB key using Linux Live USB creator. A graphical install was used. A root password, and a new user name and password are entered. Default settings were used in the install, with the following exceptions:
- Guided partitioning, with Logical volume management enabled
- Software selection: GNOME desktop env., web server, print server, SSH server, standard system utilities
- Installed GRUB boot loader to Master boot record (MBR)
After logging into Debian for the first time, start up a terminal. First we need to give sudo (root privileges) to our user we created in the install process (e.g., user1):
Switch to root user:
su root
Install sudo, and enable sudo access for user1:
apt-get update
apt-get install sudo
usermod -a -G sudo user1
Now switch back to user1 and create any new users using the adduser command:
su user1
sudo adduser user2
To delete a user, check if they are logged in first (using who), then enter the following:
sudo deluser --remove-home user2
We now can access the server remotely using ssh user1@computer_name. For security, we can disallow remote logins as the root user, by modifying the /etc/ssh/sshd_config file:
sudo nano /etc/ssh/sshd_config
Change the line #PermitRootLogin yes to PermitRootLogin no, and then restart ssh:
systemctl restart ssh
Install necessary packages for mounting (Windows) network folders:
sudo apt-get install samba
sudo apt-get install smbclient
sudo apt-get install cifs-utils
Make new local directories to link to network folders:
sudo mkdir /mnt/basille_lab
sudo mkdir /mnt/dbucklin
Since network folders require authentication, create a credentials text file with the following lines:
username=*******
password=*******
domain=ad.ufl.edu
Network folders can be mounted using the following commands:
sudo mount.cifs //ifs-flrec-1mps/data/Users/dbucklin /mnt/dbucklin -o credentials=/path/to/file,uid=user1,gid=user1
sudo mount.cifs //ifs-flrec-1mps/data/Groups/basille_lab /mnt/basille_lab -o credentials=/path/to/file,uid=user1,gid=user1
To load network folder for the lab on computer startup, add the following line to /etc/fstab (Note that permissions are restricted to only the user marked in uid= through the use of dir_mode=0700:
//ifs-flrec-1mps/data/Groups/basille_lab /mnt/basille_lab cifs credentials=/path/to/file,uid=user1,rw,dir_mode=0700 0 0
Install the base server, client, and development files - more information can be found here:
sudo apt-get update
sudo apt-get install postgresql-9.4 postgresql-client-9.4 postgresql-server-dev-9.4
The install creates a new system user postgres. This user can create additional database users (roles), using createuser and answering the questions that follow:
su root
su postgres
createuser --interactive
Enter name of role to add: user1
Shall the new role be a superuser? (y/n) y
Shall the new role be allowed to create databases? (y/n) y
Shall the new role be allowed to create more new roles? (y/n) y
Now we can log into psql as user1 using the following command:
psql -d database_name
sudo apt-get install pgadmin3
sudo apt-get install postgis
Databases can then be imported using the pgAdmin3 restore tool (Tools->Restore). Make sure prior to restore that all roles who have privileges on the restored databases already exist on the server. You could also restore the database using psql.
First create a new folder in the lab network folder:
sudo mkdir /mnt/basille_lab/db_backups
Now open cron, a task scheduling file, using the following command:
crontab -e
Add the following lines to the file, which create a backup for two databases as well as the entire server, and delete old files in the backup folder (all performed daily):
#backup databases, with dates in filenames
10 23 * * * pg_dump wood_stork_tracking | gzip > /mnt/basille_lab/db_backups/wood_stork_tracking_`date +'%Y_%m_%d'`.gz
20 23 * * * pg_dump keys_gps_tracking | gzip > /mnt/basille_lab/db_backups/keys_gps_tracking_`date +'%Y_%m_%d'`.gz
#backup full database server
30 23 * * * pg_dumpall | gzip > /mnt/basille_lab/db_backups/fullDB_`date +'%Y_%m'`.gz
#delete files older than 60 days
0 23 * * * find /mnt/basille_lab/db_backups -type f -mtime +60 -delete
The main settings for the Postgresql server can be altered by editing postgresql.conf, and connection settings in pg_hba.conf:
sudo nano /etc/postgresql/9.4/main/postgresql.conf
sudo nano /etc/postgresql/9.4/main/pg_hba.conf
Following changes, restart the server using:
sudo pg_ctlcluster 9.4 main [status][reload][restart][start][stop]
To view log files, log into root and go to directory:
cd /var/lib/postgresql/9.5/main/pg_log
To upgrade a major version (e.g., 9.4 -> 9.5), first install the new binaries:
sudo apt_get update && sudo apt_get upgrade postgresql
sudo apt_get update && sudo apt_get upgrade postgis
Drop the newly installed cluster:
sudo pg_dropcluster 9.5 main
Run the upgrade utility - make sure to note any restore errors during the process:
sudo pg_upgradecluster -v 9.5 9.4 main
The new cluster (9.5) will now run on port 5432, with the old cluster on port 5433. Once you are sure that the new cluster is functioning normally, drop the old cluster:
sudo pg_dropcluster --stop 9.4 main
Install vnc4server and the xfce4 desktop environment (there are issues with GNOME and Debian 8 on VNC):
sudo apt-get install vnc4server
sudo apt-get install xfce4 xfce4-goodies
Modify the file:
sudo nano /home/user1/.vnc/xstartup
Full file below:
#!/bin/sh
unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS
startxfce4 &
[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
Now launch a vnc server, and note the name and number (e.g., computer-name:1) given to it:
vnc4server -geometry 1920x1080 -depth 24
To stop the server computer-name:1, use:
vnc4server -kill :1
Install main packages:
sudo apt-get update
sudo apt-get install r-base r-base-dev
sudo apt-get install libatlas3-base
By default (on Debian Jessie), R 3.1.1 is installed. To set up backports for Jessie to allow for updating R, add an appropriate mirror source to /etc/apt/sources.list:
deb http://archive.linux.duke.edu/cran/bin/linux/debian jessie-cran3/
We also need to add a PUBKEY for the R mirror we chose:
gpg --keyserver pgpkeys.mit.edu --recv-key 06F90DE5381BA480
gpg -a --export 06F90DE5381BA480 | sudo apt-key add -
Now we can upgrade R (and any other system packages) to the newest version (as of Nov 2015, 3.2.2), using:
sudo apt-get update
sudo apt-get upgrade
To install R packages globally, we need to open R with root privileges (sudo R). Instead of doing this every time, we can add users to the group staff, which then allows those users to install to the global R library folder (/usr/local/lib/R/site-library) by default, e.g.:
su root
adduser user1 staff
su user1
R
> install.packages('shiny')
Download and install gdebi and RStudio Server 64-bit, and start it:
sudo apt-get install gdebi-core
wget https://download2.rstudio.org/rstudio-server-0.99.489-amd64.deb
sudo gdebi rstudio-server-0.99.489-amd64.deb
sudo rstudio-server start
To serve RStudio server from a custom path (instead of the port), using nginx - edit the nginx configuration /etc/nginx/nginx.conf, adding the following lines inside the http{}
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
Then edit the nginx default site configuration, adding the following location:
location /rstudio/ {
rewrite ^/rstudio/(.*)$ /$1 break;
proxy_pass http://localhost:8787;
proxy_redirect http://localhost:8787/ $scheme://$host/rstudio/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_read_timeout 20d;
}
Server can now be accessed here.
Before istalling the Shiny server in Debian 8, a prerequiste package (libssl0.9.8) needs to be installed:
wget http://ftp.us.debian.org/debian/pool/main/o/openssl/libssl0.9.8_0.9.8o-4squeeze14_amd64.deb
sudo dpkg -i libssl0.9.8_0.9.8o-4squeeze14_amd64.deb
Now install the Shiny server:
wget https://download3.rstudio.org/ubuntu-12.04/x86_64/shiny-server-1.4.0.756-amd64.deb
sudo gdebi shiny-server-1.4.0.756-amd64.deb
To share an app on the server, just copy it's project folder (containing server.r and ui.r) to server folder, e.g.:
sudo cp -R /usr/local/lib/R/site-library/shiny/examples/04_mpg /srv/shiny-server/
Apps are shared at http://basille-flrec.ad.ufl.edu:3838/app_name by default.
To edit configuration of shiny server, edit file:
sudo nano /etc/shiny-server/shiny-server.conf
To allow shiny apps to upload/download files to the app folder, create a shiny-apps group and add the shiny user (and any other users) to it:
sudo groupadd shiny-apps
sudo usermod -aG shiny-apps user1
sudo usermod -aG shiny-apps shiny
Then modify the permissions in the main shiny folder:
cd /srv/shiny-server
sudo chown -R user1:shiny-apps .
sudo chmod g+w .
sudo chmod g+s .
To give the shiny user full ownership of a certain app folder:
cd /srv/shiny-server/shinyapp/
sudo chown shiny:shiny-apps .
Install gdal, and necessary packages for using the R package rgdal:
sudo apt-get install gdal-bin
sudo apt-get install libproj-dev
sudo apt-get install libgdal-dev
Install necessary packages for the R package devtools:
sudo apt-get install libssl-dev
sudo apt-get install libxml2-dev
sudo apt-get install libcurl4-openssl-dev
sudo apt-get install git
Set user name and email:
git config --global user.name "first last"
git config --global user.email [email protected]
Set default text editor (nano for now):
git config --global core.editor nano
You can check existing settings with:
git config --list
Several packages are pre-requisites for GitLab, install them:
sudo apt-get install curl openssh-server ca-certificates postfix
Postfix is an email program which requires configuration (see here and here). To reconfigure postfix, use the command:
dpkg-reconfigure postfix
And then reload it:
sudo /etc/init.d/postfix reload
Now install and reconfigure gitlab:
curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash
sudo apt-get install gitlab-ce
sudo gitlab-ctl reconfigure
To run on an Apache2 webserver, some configuration is needed. The instructions from here were followed. This invloved setting up a new vhost file /etc/apache2/sites-available/gitlab-8.0-apache2.4.conf, taken from here. After this, the file /etc/apache2/ports.conf was modified, adding the line Listen 4554 (the gitlab port).
After this, run the following to enable apache2 modules necessary:
sudo a2enmod proxy proxy_http rewrite
Now enable the site and restart apache2:
sudo a2ensite gitlab-8.0-apache2.4.conf
sudo service apache2 restart
The gitlab site can be accessed here.
Allow gitlab to be upgraded using apt-get by adding the package repository to your system (more info here):
curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash
Upgrades can now be done as any other package:
sudo apt-get install gitlab-ce
Add the Debian Jessie backports repository to /etc/apt/sources.list
deb http://http.debian.net/debian jessie-backports main
Change document root in for main website:
sudo nano /etc/apache2/sites-enabled/000-default.conf
Change to DocumentRoot /srv/www/html. Then allow use of authentication (.htaccess) files in this new location in /srv:
sudo nano /etc/apache2/apache2.conf
Add the following lines:
<Directory /srv/>
Options Indexes FollowSymLinks
AllowOverride All
Require all granted
</Directory>
Give ownership of the password folder to user www-data:
sudo chown -R www-data:www-data /srv/.htpasswd
Finally, restart the server:
sudo service apache2 restart
Install nginx:
sudo apt-get install nginx
To configure nginx to allow authentication on shiny server pages - more information here:
Stop the shiny and nginx services:
sudo service nginx stop
sudo stop shiny-server
Edit the default configuration of nginx, adding a new location for shiny server:
sudo nano /etc/nginx/sites-available/default
File should have at least the following:
server {
listen 80;
location / {
proxy_pass http://127.0.0.1:3838/;
proxy_redirect http://127.0.0.1:3838/ $scheme://$host/;
auth_basic "Username and Password are required";
auth_basic_user_file /etc/nginx/.htpasswd;
}
}
Edit the shiny server configuration:
sudo nano /etc/shiny-server/shiny-server.conf
File should have at least the following:
server{
listen 3838 127.0.0.1;
location / {
site_dir /srv/shiny-server;
log_dir /var/log/shiny-server;
directory_index on;
}
}
Add users/passwords with the htpasswd utility from apache2-utils:
cd /etc/nginx
sudo htpasswd -c /etc/nginx/.htpasswd exampleuser
Postfix was installed with GitLab. As the server is behind a firewall, changes were made to the default configuration, following the instructions here.
Add the line transport_maps = hash:/etc/postfix/transport to /etc/postfix/main.cf.
Create the file transport with the following lines, specifying SMTP relay in external delivery:
# Internal delivery.
ufl.edu :
.ufl.edu :
# External delivery.
* smtp:[smtp.ufl.edu]
After editing the transport file, always check it with the command:
postmap /etc/postfix/transport