Skip to content

Commit

Permalink
Merge pull request #92 from TU-Delft-DCC/master
Browse files Browse the repository at this point in the history
Update develop from master
  • Loading branch information
mwakok authored Mar 4, 2024
2 parents 5b1bdd7 + d1cc921 commit e5cc608
Show file tree
Hide file tree
Showing 11 changed files with 202 additions and 90 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/links.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
- uses: actions/checkout@v3

- name: Link Checker
uses: lycheeverse/lychee-action@v1.5.1
uses: lycheeverse/lychee-action@v1.9.3
with:
args: --exclude-path 'docs/_under_development' --verbose --no-progress './**/*.md' './**/*.html'
fail: true
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/links_fail_fast.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ jobs:
- uses: actions/checkout@v3

- name: Link Checker
uses: lycheeverse/lychee-action@v1.6.1
uses: lycheeverse/lychee-action@v1.9.3
with:
args: --exclude-path 'docs/_under_development' --verbose --no-progress './**/*.md' './**/*.html'
fail: true
Expand Down
3 changes: 2 additions & 1 deletion .lycheeignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,5 @@ help.dreamhost.com
nl.linkedin.com
https://data.4tu.nl/
twitter.com
https://webdata.tudelft.nl
https://webdata.tudelft.nl
https://www.putty.org
4 changes: 2 additions & 2 deletions docs/community/maintainers.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ _This content is automatically generated, all changes made will be lost._
| Section | Title | Lead maintainer | Backup maintainer |
|:---------------|:------------------------------------------|:----------------------|:--------------------|
| containers | Docker users | Maurits Kok | |
| data | RequestProject Drive | Ashley Cryan | |
| data | Request Project Drive | Ashley Cryan | |
| data | Sync with Unison | Ashley Cryan | |
| data | Mount Project Drive | Ashley Cryan | |
| data | Mount Project Drive | Raúl Ortiz Merino | Maurits Kok |
| data | Data publishing | Aleksandra Wilczynska | |
| gitlab | Transfer ownership of a GitLab repository | Lora Armstrong | Maurits Kok |
| gitlab | CI with Gitlab | Ashley Cryan | Maurits Kok |
Expand Down
140 changes: 102 additions & 38 deletions docs/data/project_drive_mounting.md
Original file line number Diff line number Diff line change
@@ -1,78 +1,142 @@
---
section: data
title: Mount Project Drive
author_1: Ashley Cryan
author_2:
author_1: Raúl Ortiz Merino
author_2: Maurits Kok
---

# Mount Project Drive on Server

## Background

## What this documentation will help achieve
Project drive storage from TU Delft ICT can be mounted and made accessible in your (TU Delft) Virtual Private Server.

## Prerequisites
A (TU Delft) Virtual Private Server and space on the (TU Delft) Project Drive
- A (TU Delft) [Virtual Private Server](../infrastructure/VPS_request.md)
- (TU Delft) [Project Drive](./project_drive_request.md)

## Steps
1. Locate the URL of your project storage
1. Connect to your TU Delft VPS via SSH
2. Think of a mount point and create a new directory for it (It is a convention to mount external drives into /media)
3. Locate the URL of your project storage
4. Open the fstab file
5. Edit the fstab file to include project storage technical details
6. Save the fstab file
7. Mounting the project drive
1. Create a new directory as a mounting point
1. Retrieve your Linux user and group details
1. Edit the fstab file to include project storage technical details
1. Mount the project drive

### Step 1. Locate the URL of your project storage
The URL for your project drive can be obtained from either
- the email from TU Delft ICT with the confirmation of your project drive request, or
- by navigating to https://webdata.tudelft.nl/, and then through **WebDav Web Links > Staff-Umbrella > "Enter your netID and password"** to retrieve a list of your project drives.

Copy everything after "https://webdata.tudelft.nl/" (this will be staff-umbrella/`<your_project_name>`)

### Step 2. Connect to your TU Delft VPS via SSH
Follow instructions in TU Delft ICT email from initial server setup or [configure a 1-step connection via SSH](../infrastructure/VPS_SSH.md).

### Step 3. Create a new directory as the mounting point
The convention is to create mounting points in the folder `/media`. Navigate to the folder and create a new folder with

```bash
cd /media
mkdir <server_mount_point>
```
Replace `<server_mount_point>` with the name of your choice. This will be the name of the folder where your project drive will be mounted.

### Step 4. Find and save your user and group details

### Step 1. Connect to your TU Delft VPS via SSH
Follow instructions in TU Delft ICT email from initial server setup or configure 1-step connection via SSH.
In the terminal, you can retrieve your local user and group details with:

### Step 2. Create a new directory as the mounting point preferably in /media
This is where you will mount the project drive. Use `cd /media` to enter the media directory, and `mkdir <yourprojectname>` to create a new directory where the TU Delft project drive will be mounted.
```bash
id -u <your_netID> # User ID
id -g <your_netID> # Group ID
```

### Step 3. Locate the URL of your project storage
This can be found either in the email from TU Delft ICT confirming project drive storage setup, or by going to https://webdata.tudelft.nl/, and then WebDav Web Links > Staff-Umbrella > Enter your netID and password > "Your Project Name". Copy everything after "https://webdata.tudelft.nl/" (this will be staff-umbrella/yourprojectname)
You may need the values for `uid` and `gid` for step 5.

### Step 4. Open the fstab file
The fstab file is where you can list the addresses of external file systems you want to mount. Find it in the /etc/ directory by typing `sudo vi fstab`. The fstab file must be in the /etc/ directory.
```{note}
These commands are server-specific, so make sure to execute them on the server where the project drives will be mounted.
```

### Step 5. Edit the fstab file to include project storage technical details
The fstab file containes a list of the addresses of external file systems. In this file, the details of your project drive will need to be added in a single line. This line consists of four parts:
1. *filesystem* - the address of the project drive
2. *mount point* - the location in the VPS where you want to mount the project drive
3. *type* - the type of the filesystem
4. *options* - additional option such as user privileges


Basically you need to add one line to this file. This line consists of four parts: (i) filesystem, (ii) mount point, (iii) type, and (iv) options. The filesystem refers to the project drive address which is specified in the ICT email. The mount point is where you want to mount the project drive in the VPS. The third part determines the type of the filesystem and in the last part you specify some options such as privileges.
The fstab file must be in the `/etc/` directory and can be opened with the `vi` or `nano` editor:

To add the filesystem, open the /etc/fstab file with the sudo privilage and switch to the insert mode (hit "i" to switch to insert mode and be able to type), and write `//tudelft.net/` and paste the latter half of the URL you copied from the WebDav links. The full URL should be in this format: `//tudelft.net/staff-umbrella/yourprojectname`.
````{tab-set}
```{tab-item} vi
In the terminal, enter the following command to open the fstab file in the vi editor:
`sudo vi /etc/fstab`
Then, switch to the insert mode (hit "i" to switch to insert mode and be able to type)
```
For the mount point, add a space followed by the location in your VPS where you want to mount the Project Drive storage (where you created the folder previously, e.g., `/media/<server_mount_point>` ).
```{tab-item} nano
In the terminal, enter the following command to open the fstab file in the nano editor:
`sudo nano /etc/fstab`
```
````

The full new record in the fstab file should be formatted like this:
Add the following line to the file:

```
<your_netID>@sftp.tudelft.nl:/staff-umbrella/<Project_Drive_space> /media/<server_mount_point> fuse.sshfs rw,noauto,users,_netdev 0 0
<your_netID>@sftp.tudelft.nl:/staff-umbrella/<project_drive_name> /media/<server_mount_point> fuse.sshfs rw,noauto,users,_netdev 0 0
```

replacing the values between `<` and `>` with your NetID, the name of your project drive, and the name of the folder you created in step 3.

````{note}
If this configuration throws a permission error during mounting, try:
```
If that throws a permissions error, try:
//tudelft.net/staff-umbrella/<project_drive_name>/ /media/<server_mount_point> cifs username=<your_netID>,noauto,uid=<your_uid>,gid=<your_gid>,forcegid,rw,_netdev
```
//tudelft.net/staff-umbrella/ODCOR/ /media/odcor cifs username=acryan,noauto,uid=549192,gid=5004,forcegid,rw,_netdev
Use the values for `uid` and `gid` from step 4.
````

Close the file editor and save the changes:
````{tab-set}
```{tab-item} vi
Use `Control`+`C` followed by `:wq` to save the file and close it to get back to your terminal.
```
### Step 6. Save the fstab file
Use Control+C followed by `:wq` to save the file and close it to get back to your terminal.
```{tab-item} nano
As indicated by the nano interface, use `Control`+`O` to write the file. Then, confirm your choice of filename by hitting `enter`. Finally, exit the file with `Control`+`X`
```
````

### Step 7. Mounting the project drive
To mount the project drive use the `sudo mount /media/<server_mount_point>` command. You can also unmount it using `fusermount -u /media/<server_mount_point>`.
### Step 6. Mount the project drive
To mount the project drive execute the command

## Notes and next steps
The project drive will not mount automatically, so you will need to mount it manually each time you start the server. To do so, from the home (`/`) directory in your server run `mount /media/<server_mount_point>`. You should now be able to `cd /media/<server_mount_point>` and view your files stored on the project drive using `ls`.
```bash
sudo mount /media/<server_mount_point>
```

Note, if the step above does not work, it probably means that the packages for mounting cifs-type filesystems haven't been installed. Depending on your linux flavour you will need to install them first using:
You can also unmount the drive with
```bash
fusermount -u /media/<server_mount_point>
```

`sudo apt install cifs-utils`
The project drive will not mount automatically, so you will need to remount it manually each time you restart the server.

on ubuntu/debian like linuxes, or
`````{note}
If the step above does not work, it probably means that the packages for mounting cifs-type filesystems haven't been installed. Depending on your linux flavour you will need to install them using:
`sudo yum install cifs-utils`
````{tab-set}
```{tab-item} Ubuntu/Debian
`sudo apt install cifs-utils`
```
on redhat/centos/fedora flavours.
```{tab-item} Redhat/Centos/Fedora
`sudo yum install cifs-utils`
```
````
The steps above can also be used to mount any storage offered by TU Delft with a WebDav link (staff-homes, staff-groups, staff-bulk, student-homes, student-groups and apps). Simply use the latter half of the URL from the WebDav web link of your storage drive (step 5), which will change from staff-umbrella (project drive) to something else depending on the storage drive you would like to mount.
`````

## Notes and next steps
The steps above can also be used to mount any storage offered by TU Delft with a WebDav link (staff-homes, staff-groups, staff-bulk, student-homes, student-groups and apps). Simply use the latter half of the URL from the WebDav web link of your storage drive, which will change from staff-umbrella (project drive) to something else depending on the storage drive you would like to mount.
4 changes: 1 addition & 3 deletions docs/infrastructure/getting_started.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,9 +74,7 @@ If you need more computational power, then you might consider the national compu
- HPC cloud – for individual or group - https://servicedesk.surfsara.nl/wiki/display/WIKI/HPC+Cloud
- Grid – extremely large datasets - https://userinfo.surfsara.nl/systems/grid

For ‘large’ requests, applications go through NWO: https://www.nwo.nl/en/researchprogrammes/computing-time-national-computer-facilities

For ‘small’ requests, applications go through SURF: https://www.surf.nl/en/applying-for-access-to-compute-services
For small and large requests, applications go through NWO: https://www.surf.nl/en/access-to-compute-services

Alternatively, your FIM can arrange access to Cartesius and Lisa on a pay per use basis: https://userinfo.surfsara.nl/systems/shared/rccs

Expand Down
Loading

0 comments on commit e5cc608

Please sign in to comment.