From d88605c6c57a5e87a0017a126362c50bd2b50261 Mon Sep 17 00:00:00 2001 From: Niket <15953349+niketagrawal@users.noreply.github.com> Date: Wed, 29 Nov 2023 16:13:53 +0100 Subject: [PATCH 01/15] registering runner using registration token is deprecated --- docs/infrastructure/gitlab/gitlab_docker.md | 84 ++++++++++++--------- 1 file changed, 49 insertions(+), 35 deletions(-) diff --git a/docs/infrastructure/gitlab/gitlab_docker.md b/docs/infrastructure/gitlab/gitlab_docker.md index 48baece..65cc183 100644 --- a/docs/infrastructure/gitlab/gitlab_docker.md +++ b/docs/infrastructure/gitlab/gitlab_docker.md @@ -1,6 +1,6 @@ --- section: gitlab -title: CI with Gitlab +title: CI with Gitlab author_1: Ashley Cryan author_2: Maurits Kok --- @@ -8,7 +8,7 @@ author_2: Maurits Kok # Continuous Integration with Gitlab ## Background -If you happen to be working with TU Delft Gitlab instance and you want to implement DevOps or CI/CD pipelines, then you need to install a gitlab runner on your own. This should runner should be in a server, responding to changes such as commits or pull requests in your Gitlab repository. +If you happen to be working with TU Delft Gitlab instance and you want to implement DevOps or CI/CD pipelines, then you need to install a gitlab runner on your own. This should runner should be in a server, responding to changes such as commits or pull requests in your Gitlab repository. ## Quick overview of how it works In order to be ready to run CI/CD pipeline, a gitlab-runner Docker container is running on the server all the time. When a new commit is made in the GitLab repository, this triggers the CI/CD process to run a job (e.g., unit test) based on the pipeline defined in the .gitlab-ci.yml file in the repository. The container used to carry out the CI/CD tests is defined in the .gitlab-ci.yml file in the first line, and spawned from within the continuously running gitlab-runner container. In our example, we define `image:python:3.7.3` so every time a commit is made in the repository, a new container based on the python:3.7.3 Docker image is started and used to run tests on the python scripts and generate artifacts as defined in the .gitlab-ci.yml file. @@ -21,7 +21,7 @@ The documentation below will help you deploy GitLab runner in a Docker container ## Prerequisites **Server:** This example uses a server to run the whole process. You can request a server from TU Delft ICT service following these directions [here](../VPS_request.md). It is useful to set this up on a server so that Docker can be running continuously, and be ready to run CI/CD tests whenever a new commit occurs in the repository. -**Docker:** We use a Docker container to run the Gitlab runner and initialise the CI/CD pipeline. +**Docker:** We use a Docker container to run the Gitlab runner and initialise the CI/CD pipeline. **Gitlab runner:** (from GitLab documentation) "Runners are the agents that run the CI/CD jobs that come from GitLab. When you register a runner, you are setting up communication between your GitLab instance and the machine where GitLab Runner is installed. Runners usually process jobs on the same machine where you installed GitLab Runner." [Link](https://docs.gitlab.com) @@ -44,7 +44,7 @@ The documentation below will help you deploy GitLab runner in a Docker container 7. Set up the GitLab runner 8. Deploy the GitLab runner in a Docker container 9. Register the runner -10. Test the CI/CD pipeline +10. Test the CI/CD pipeline ### Step 1. Request server running Ubuntu If you don't have a VPS already, you can request one from TU Delft ICT. Instructions for requesting a server and storage from ICT can be found under [Remote servers/Request a Virtual Private Server](../VPS_request.md). @@ -55,17 +55,17 @@ We recommend the following configuration for configuring a Gitlab runner: - Additional space if your Docker images are larger than ~10Gb. ### Step 2. Connect to the server via ssh -The email response from Sysadmin@TUDelft.nl confirming the successful deployment of your server should contain instructions to connect via ssh. +The email response from Sysadmin@TUDelft.nl confirming the successful deployment of your server should contain instructions to connect via ssh. The default login procedure is to connect to the Bastion host (an intermediary server) and then to your server, so it is a two-step process. Please check your email for Steps A and B as described by ICT admin. -You can also connect to your server using Putty (Windows) or, on Mac/Linux, configure one-step access by storing ssh keys between your local machine and your server and designating an alias. +You can also connect to your server using Putty (Windows) or, on Mac/Linux, configure one-step access by storing ssh keys between your local machine and your server and designating an alias. -When you are successfully connected, you should see in your terminal/command prompt something like this: +When you are successfully connected, you should see in your terminal/command prompt something like this: ![Screen_Shot_2020-11-17_at_10.30.31](https://gitlab.tudelft.nl/acryan/data-management-for-researchers/-/wikis/uploads/048a06a817852a399c697582ae1e3a08/Screen_Shot_2020-11-17_at_10.30.31.png) -### Step 3. Install Docker on the server +### Step 3. Install Docker on the server Servers often do not come with Docker installed, so you may need to do it by yourself. To check whether Docker is installed, run `docker --version`. If you get an error message, you can install it using the following commands: @@ -89,7 +89,7 @@ This file in the root of your repository defines what CI/CD tests to run upon ea In the first line of the file, you can write `image: :` to indicate you want to run the runner in a Docker container container. `` should be replaced by the name of the image you want to use to start a container and `` should be replaced by the tag of the image you want to use(image names can be found on DockerHub). Tag will be set to `latest` by default. -**Important:** The file must be named exactly `.gitlab-ci.yml` so it will be recognizable by GitLab runner. Below is an example .gitlab-ci.yml file. +**Important:** The file must be named exactly `.gitlab-ci.yml` so it will be recognizable by GitLab runner. Below is an example .gitlab-ci.yml file. ```yaml # This is a sample build configuration for Python. @@ -116,11 +116,11 @@ test: Tags can be added as a final section in your .gitlab-ci.yml file (they are optional, we didn’t enter any in this example). You then need to enter these tags exactly when registering the runner. -### Step 7. Set up GitLab runner +### Step 7. Setup the GitLab runner -Now that you have the .gitlab-ci.yml file configured, you can link your repository to a runner with a token and tags, if you have included those in your .gitlab-ci.yml file. The token can be generated in GitLab by going to Your Project -> Settings -> CI/CD -> Expand Runners -> Setup a specific runner manually. Copy the token you see there. +- Follow the instructions [here](https://docs.gitlab.com/ee/tutorials/create_register_first_runner/#create-and-register-a-project-runner) **till step 7** to create a GitLab runner for your repository -![token](https://gitlab.tudelft.nl/acryan/data-management-for-researchers/-/wikis/uploads/89288dcfd011627a4397cf4a4ad00cd7/token.PNG) +- Copy the authentication token generated and keep it handy. It will be required in Step 9. ### Step 8. Deploy GitLab runner in a Docker container @@ -135,7 +135,7 @@ gitlab/gitlab-runner:latest Check that gitlab-runner container is running using `docker ps -a` -### Step 9. Register the runner +### Step 9. Register the runner using authentication token Run the following command to register your runner and configure it to deploy in a Docker container on your server. @@ -143,41 +143,55 @@ Run the following command to register your runner and configure it to deploy in **on Mac OS, use /Users/Shared/ instead of /srv/** -In response to this command you will be prompted to answer a series of questions, about the gitlab-ci coordinator URL (for TU Delft this is https://gitlab.tudelft.nl), gitlab-ci token (copied from your repository CI/CD settings), an optional description (can hit enter to skip), tgas (must exactly match the tags in your .gitlab-ci.yml file, or hit enter to skip), executor (docker), and default Docker image (same as the one specified in your .gitlab-ci.yml file). +In response to this command you will be prompted to answer the following questions +- GitLab URL URL: Enter https://gitlab.tudelft.nl +- gitlab-ci token: Paste the authentication token generated in Step 7. +- Enter name of the runner +- Type of executor: Choose `docker` +- Default Docker image: Specify the same image as the one specified in the 'image' field of the .gitlab-ci.yml file. + +See an example below. ``` -Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/): +~$ docker run --rm -it -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register + + +Enter the GitLab instance URL (for example, https://gitlab.com/): https://gitlab.tudelft.nl -Please enter the gitlab-ci token for this runner: - -Please enter the gitlab-ci description for this runner: (or hit enter to skip) -Please enter the gitlab-ci tags for this runner (comma separated): (or hit enter to skip) -Please enter the executor: custom, docker-ssh, shell, virtualbox, docker+machine, docker, parallels, ssh, docker-ssh+machine, kubernetes: +Enter the registration token: +xxxxxxxxxxxxxxx +Verifying runner... is valid runner=xxxxxxx +Enter a name for the runner. This is stored only in the local config.toml file: +[xxxxxxx]: example-runner +Enter an executor: instance, kubernetes, docker-windows, docker-autoscaler, parallels, shell, ssh, virtualbox, docker+machine, custom, docker: docker -Please enter the default Docker image (e.g. ruby:2.6): -python:3.7.3 +Enter the default Docker image (for example, ruby:2.7): +julia:1.6 +Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded! -Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded! +Configuration (with the authentication token) was saved in "/etc/gitlab-runner/config.toml" ``` ### Step 10. Check that the CI/CD process is activated and your pipeline runs successfully -In order to activate the CI/CD process, you should make a new commit to the GitLab repository. This will trigger the CI/CD pipeline to run. +- In your repository, navigate to Settings > CI/CD > Click on Expand under Runners: You should see that the runner is active as indicated by a green dot. This means that the CI pipeline is ready to run, but it needs to be triggered. -Navigate to Your Project -> CI/CD -> Pipelines to check the status of CI/CD pipelines connected to your repository. If you find a green message that says “passed” with a check mark then congratulations, your pipeline works! If you see a red message that says “failed”, check to see the error message associated with it - sometimes you need to reconfigure your .gitlab-ci.yml file to make sure it uses the correct formatting and defines the tests appropriately. +- In order to trigger the CI pipeline, you should make a new commit to the GitLab repository. + +Navigate to Your Project -> CI/CD -> Pipelines to check the status of CI/CD pipelines connected to your repository. If you find a green message that says “passed” with a check mark then congratulations, your pipeline works! If you see a red message that says “failed”, check to see the error message associated with it - sometimes you need to reconfigure your .gitlab-ci.yml file to make sure it uses the correct formatting and defines the tests appropriately. ![Screen_Shot_2020-11-17_at_11.56.41](https://gitlab.tudelft.nl/acryan/data-management-for-researchers/-/wikis/uploads/f7aadbc87f49d31de4da37a2f8a76ef3/Screen_Shot_2020-11-17_at_11.56.41.png) - + ## Notes and References -This guide was created using multiple references on GitLab runner and Docker. If you would like more information or details, please refer to the links below. -* https://www.hostinger.com/tutorials/how-to-install-docker-on-ubuntu -* https://cloudcone.com/docs/article/how-to-install-and-run-docker-on-ubuntu-20-04-lts/ -* https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-20-04 -* https://docs.gitlab.com/ee/ci/yaml/README.html -* https://docs.gitlab.com/runner/executors/docker.html#define-image-and-services-from-gitlab-ciyml +This guide was created using multiple references on GitLab runner and Docker. If you would like more information or details, please refer to the links below. +* https://www.hostinger.com/tutorials/how-to-install-docker-on-ubuntu +* https://cloudcone.com/docs/article/how-to-install-and-run-docker-on-ubuntu-20-04-lts/ +* https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-20-04 +* https://docs.gitlab.com/ee/ci/yaml/README.html +* https://docs.gitlab.com/runner/executors/docker.html#define-image-and-services-from-gitlab-ciyml * https://gitlab.tudelft.nl/help/ci/yaml/index.md -* https://docs.gitlab.com/ee/ci/runners/ -* https://docs.gitlab.com/ee/ci/docker/using_docker_images.html -* https://docs.gitlab.com/ee/ci/examples/README.html +* https://docs.gitlab.com/ee/ci/runners/ +* https://docs.gitlab.com/ee/ci/docker/using_docker_images.html +* https://docs.gitlab.com/ee/ci/examples/README.html * https://semaphoreci.com/blog/cicd-pipeline From 7d7493bff2741e10d7e99a97d6a76ada09bdf5af Mon Sep 17 00:00:00 2001 From: Niket <15953349+niketagrawal@users.noreply.github.com> Date: Wed, 29 Nov 2023 16:49:01 +0100 Subject: [PATCH 02/15] fix formatting typos --- docs/infrastructure/gitlab/gitlab_docker.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/docs/infrastructure/gitlab/gitlab_docker.md b/docs/infrastructure/gitlab/gitlab_docker.md index 65cc183..b749348 100644 --- a/docs/infrastructure/gitlab/gitlab_docker.md +++ b/docs/infrastructure/gitlab/gitlab_docker.md @@ -120,6 +120,8 @@ Tags can be added as a final section in your .gitlab-ci.yml file (they are optio - Follow the instructions [here](https://docs.gitlab.com/ee/tutorials/create_register_first_runner/#create-and-register-a-project-runner) **till step 7** to create a GitLab runner for your repository +- Choose 'Linux' under operating systems + - Copy the authentication token generated and keep it handy. It will be required in Step 9. ### Step 8. Deploy GitLab runner in a Docker container @@ -144,15 +146,18 @@ Run the following command to register your runner and configure it to deploy in **on Mac OS, use /Users/Shared/ instead of /srv/** In response to this command you will be prompted to answer the following questions -- GitLab URL URL: Enter https://gitlab.tudelft.nl + +```shell +- GitLab URL: https://gitlab.tudelft.nl - gitlab-ci token: Paste the authentication token generated in Step 7. -- Enter name of the runner -- Type of executor: Choose `docker` +- Enter name of the runner: example-runner +- Type of executor: docker - Default Docker image: Specify the same image as the one specified in the 'image' field of the .gitlab-ci.yml file. +``` See an example below. -``` +```shell ~$ docker run --rm -it -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register From 39a2573e76d65eae6dd9e5b1be4983d8ec41f51a Mon Sep 17 00:00:00 2001 From: Niket <15953349+niketagrawal@users.noreply.github.com> Date: Wed, 29 Nov 2023 16:56:55 +0100 Subject: [PATCH 03/15] check CI status only new after commit is made --- docs/infrastructure/gitlab/gitlab_docker.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/infrastructure/gitlab/gitlab_docker.md b/docs/infrastructure/gitlab/gitlab_docker.md index b749348..8d83a88 100644 --- a/docs/infrastructure/gitlab/gitlab_docker.md +++ b/docs/infrastructure/gitlab/gitlab_docker.md @@ -183,7 +183,7 @@ Configuration (with the authentication token) was saved in "/etc/gitlab-runner/c - In order to trigger the CI pipeline, you should make a new commit to the GitLab repository. -Navigate to Your Project -> CI/CD -> Pipelines to check the status of CI/CD pipelines connected to your repository. If you find a green message that says “passed” with a check mark then congratulations, your pipeline works! If you see a red message that says “failed”, check to see the error message associated with it - sometimes you need to reconfigure your .gitlab-ci.yml file to make sure it uses the correct formatting and defines the tests appropriately. +- After you have made a new commit to the repository, navigate to Your Project -> CI/CD -> Pipelines to check the status of CI/CD pipelines connected to your repository. If you find a green message that says “passed” with a check mark then congratulations, your pipeline works! If you see a red message that says “failed”, check to see the error message associated with it - sometimes you need to reconfigure your .gitlab-ci.yml file to make sure it uses the correct formatting and defines the tests appropriately. ![Screen_Shot_2020-11-17_at_11.56.41](https://gitlab.tudelft.nl/acryan/data-management-for-researchers/-/wikis/uploads/f7aadbc87f49d31de4da37a2f8a76ef3/Screen_Shot_2020-11-17_at_11.56.41.png) From e725630b47b6fbc5287f07ed6638f192898a06ab Mon Sep 17 00:00:00 2001 From: Niket <15953349+niketagrawal@users.noreply.github.com> Date: Wed, 29 Nov 2023 16:58:40 +0100 Subject: [PATCH 04/15] specify correct location to see pipeline status --- docs/infrastructure/gitlab/gitlab_docker.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/infrastructure/gitlab/gitlab_docker.md b/docs/infrastructure/gitlab/gitlab_docker.md index 8d83a88..3beb971 100644 --- a/docs/infrastructure/gitlab/gitlab_docker.md +++ b/docs/infrastructure/gitlab/gitlab_docker.md @@ -183,7 +183,7 @@ Configuration (with the authentication token) was saved in "/etc/gitlab-runner/c - In order to trigger the CI pipeline, you should make a new commit to the GitLab repository. -- After you have made a new commit to the repository, navigate to Your Project -> CI/CD -> Pipelines to check the status of CI/CD pipelines connected to your repository. If you find a green message that says “passed” with a check mark then congratulations, your pipeline works! If you see a red message that says “failed”, check to see the error message associated with it - sometimes you need to reconfigure your .gitlab-ci.yml file to make sure it uses the correct formatting and defines the tests appropriately. +- After you have made a new commit to the repository, navigate to Your Project -> Build -> Pipelines to check the status of CI/CD pipelines connected to your repository. If you find a green message that says “passed” with a check mark then congratulations, your pipeline works! If you see a red message that says “failed”, check to see the error message associated with it - sometimes you need to reconfigure your .gitlab-ci.yml file to make sure it uses the correct formatting and defines the tests appropriately. ![Screen_Shot_2020-11-17_at_11.56.41](https://gitlab.tudelft.nl/acryan/data-management-for-researchers/-/wikis/uploads/f7aadbc87f49d31de4da37a2f8a76ef3/Screen_Shot_2020-11-17_at_11.56.41.png) From 7ffd45ef5f5bf3b3d750826770dfb7e59510e1eb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Ortiz?= <41450568+rortizmerino@users.noreply.github.com> Date: Mon, 29 Jan 2024 14:36:17 +0100 Subject: [PATCH 05/15] Update project_drive_mounting.md Edited instructions according to issue #84 --- docs/data/project_drive_mounting.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/data/project_drive_mounting.md b/docs/data/project_drive_mounting.md index da4b8fe..d5abc63 100644 --- a/docs/data/project_drive_mounting.md +++ b/docs/data/project_drive_mounting.md @@ -52,9 +52,18 @@ The full new record in the fstab file should be formatted like this: If that throws a permissions error, try: ``` -//tudelft.net/staff-umbrella/ODCOR/ /media/odcor cifs username=acryan,noauto,uid=549192,gid=5004,forcegid,rw,_netdev +//tudelft.net/staff-umbrella// /media/ cifs username=,noauto,uid=,gid=,forcegid,rw,_netdev ``` +Te values for `uid` and `gid` can be found, correspondingly, with the following commands: + +``` +id -u +id -g +``` + +These commands are VPS-specific so make sure to execute them within the same VPS where the project drives are expected to be mounted. + ### Step 6. Save the fstab file Use Control+C followed by `:wq` to save the file and close it to get back to your terminal. From 22a2eea957053aa616b2e8778be2767e0789eb1e Mon Sep 17 00:00:00 2001 From: Maurits Kok Date: Mon, 29 Jan 2024 15:47:57 +0100 Subject: [PATCH 06/15] Update broken links --- .lycheeignore | 3 ++- docs/infrastructure/getting_started.md | 4 +--- docs/infrastructure/gitlab/gitlab_docker.md | 4 +--- docs/resources/curriculum.md | 2 +- 4 files changed, 5 insertions(+), 8 deletions(-) diff --git a/.lycheeignore b/.lycheeignore index f34e478..93eab99 100644 --- a/.lycheeignore +++ b/.lycheeignore @@ -4,4 +4,5 @@ help.dreamhost.com nl.linkedin.com https://data.4tu.nl/ twitter.com -https://webdata.tudelft.nl \ No newline at end of file +https://webdata.tudelft.nl +https://www.putty.org \ No newline at end of file diff --git a/docs/infrastructure/getting_started.md b/docs/infrastructure/getting_started.md index 64933a1..19da1d9 100644 --- a/docs/infrastructure/getting_started.md +++ b/docs/infrastructure/getting_started.md @@ -74,9 +74,7 @@ If you need more computational power, then you might consider the national compu - HPC cloud – for individual or group - https://servicedesk.surfsara.nl/wiki/display/WIKI/HPC+Cloud - Grid – extremely large datasets - https://userinfo.surfsara.nl/systems/grid -For ‘large’ requests, applications go through NWO: https://www.nwo.nl/en/researchprogrammes/computing-time-national-computer-facilities - -For ‘small’ requests, applications go through SURF: https://www.surf.nl/en/applying-for-access-to-compute-services +For small and large requests, applications go through NWO: https://www.surf.nl/en/access-to-compute-services Alternatively, your FIM can arrange access to Cartesius and Lisa on a pay per use basis: https://userinfo.surfsara.nl/systems/shared/rccs diff --git a/docs/infrastructure/gitlab/gitlab_docker.md b/docs/infrastructure/gitlab/gitlab_docker.md index 3beb971..04a2051 100644 --- a/docs/infrastructure/gitlab/gitlab_docker.md +++ b/docs/infrastructure/gitlab/gitlab_docker.md @@ -193,10 +193,8 @@ This guide was created using multiple references on GitLab runner and Docker. If * https://www.hostinger.com/tutorials/how-to-install-docker-on-ubuntu * https://cloudcone.com/docs/article/how-to-install-and-run-docker-on-ubuntu-20-04-lts/ * https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-20-04 -* https://docs.gitlab.com/ee/ci/yaml/README.html +* https://docs.gitlab.com/ee/topics/build_your_application.html * https://docs.gitlab.com/runner/executors/docker.html#define-image-and-services-from-gitlab-ciyml * https://gitlab.tudelft.nl/help/ci/yaml/index.md -* https://docs.gitlab.com/ee/ci/runners/ * https://docs.gitlab.com/ee/ci/docker/using_docker_images.html -* https://docs.gitlab.com/ee/ci/examples/README.html * https://semaphoreci.com/blog/cicd-pipeline diff --git a/docs/resources/curriculum.md b/docs/resources/curriculum.md index af7fc12..35ed72b 100644 --- a/docs/resources/curriculum.md +++ b/docs/resources/curriculum.md @@ -30,7 +30,7 @@ These materials represent a curated curriculum designed to help you develop and * [Understanding the GitHub flow](https://guides.github.com/introduction/flow/) – guide from GitHub on how and why to work with branches * [Collaborative distributed version control](https://coderefinery.github.io/git-collaborative/) - We have learned how to make a git repository for a single person. What about sharing? * [SSH connection to GitHub](https://coderefinery.github.io/installation/ssh/) – instructions to set up SSH connection to GitHub so that you do not need to input your login credentials with every push/pull -* [Gitlab and SSH keys](https://docs.gitlab.com/ee/ssh/) - instructions to add an SSH key to your (TU Delft) GitLab account for the same reason as above +* [Gitlab and SSH keys](https://docs.gitlab.com/ee/user/ssh.html) - instructions to add an SSH key to your (TU Delft) GitLab account for the same reason as above * [GitHub without the Command Line from Code Refinery](https://coderefinery.github.io/github-without-command-line/) - practice collaborating and sharing using either the GitHub website or GitHub desktop application * [GitHub Guides: Mastering Markdown](https://guides.github.com/features/mastering-markdown/) - Markdown is a lightweight and easy-to-use syntax for styling all forms of writing on the GitHub platform. From 89790323ed3a219e17e85dee0710973a92ca0cc5 Mon Sep 17 00:00:00 2001 From: Maurits Kok Date: Mon, 29 Jan 2024 16:02:10 +0100 Subject: [PATCH 07/15] Update version lyche-action --- .github/workflows/links_fail_fast.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/links_fail_fast.yml b/.github/workflows/links_fail_fast.yml index e88e1a1..11b973b 100644 --- a/.github/workflows/links_fail_fast.yml +++ b/.github/workflows/links_fail_fast.yml @@ -18,7 +18,7 @@ jobs: - uses: actions/checkout@v3 - name: Link Checker - uses: lycheeverse/lychee-action@v1.6.1 + uses: lycheeverse/lychee-action@v1.9.3 with: args: --exclude-path 'docs/_under_development' --verbose --no-progress './**/*.md' './**/*.html' fail: true From 5f5d2a53bf7fa0625250926535e2d60bc8defef3 Mon Sep 17 00:00:00 2001 From: Maurits Kok Date: Mon, 29 Jan 2024 16:03:45 +0100 Subject: [PATCH 08/15] Update version lychee-action in links.yml --- .github/workflows/links.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/links.yml b/.github/workflows/links.yml index 8d3c411..22f8f14 100644 --- a/.github/workflows/links.yml +++ b/.github/workflows/links.yml @@ -13,7 +13,7 @@ jobs: - uses: actions/checkout@v3 - name: Link Checker - uses: lycheeverse/lychee-action@v1.5.1 + uses: lycheeverse/lychee-action@v1.9.3 with: args: --exclude-path 'docs/_under_development' --verbose --no-progress './**/*.md' './**/*.html' fail: true From 9636a30b825c410aa77b4d2c0b6f8194bcb02354 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Ortiz?= <41450568+rortizmerino@users.noreply.github.com> Date: Thu, 1 Feb 2024 09:49:13 +0100 Subject: [PATCH 09/15] Update project_drive_mounting.md Added a new step as preparation to find uid and gid values, related to issues #84 and #85. Also fixed a typo --- docs/data/project_drive_mounting.md | 30 ++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/docs/data/project_drive_mounting.md b/docs/data/project_drive_mounting.md index d5abc63..f060c09 100644 --- a/docs/data/project_drive_mounting.md +++ b/docs/data/project_drive_mounting.md @@ -33,10 +33,23 @@ This is where you will mount the project drive. Use `cd /media` to enter the med ### Step 3. Locate the URL of your project storage This can be found either in the email from TU Delft ICT confirming project drive storage setup, or by going to https://webdata.tudelft.nl/, and then WebDav Web Links > Staff-Umbrella > Enter your netID and password > "Your Project Name". Copy everything after "https://webdata.tudelft.nl/" (this will be staff-umbrella/yourprojectname) -### Step 4. Open the fstab file +### Step 4. Find and save your user and group details as preparation + +Type the following commands: + +``` +id -u +id -g +``` + +You may need their values for `uid` and `gid`, correspondingly, for step 6. + +Note: These commands are VPS-specific so make sure to execute them within the same VPS where the project drives are expected to be mounted. + +### Step 5. Open the fstab file The fstab file is where you can list the addresses of external file systems you want to mount. Find it in the /etc/ directory by typing `sudo vi fstab`. The fstab file must be in the /etc/ directory. -### Step 5. Edit the fstab file to include project storage technical details +### Step 6. Edit the fstab file to include project storage technical details Basically you need to add one line to this file. This line consists of four parts: (i) filesystem, (ii) mount point, (iii) type, and (iv) options. The filesystem refers to the project drive address which is specified in the ICT email. The mount point is where you want to mount the project drive in the VPS. The third part determines the type of the filesystem and in the last part you specify some options such as privileges. @@ -55,19 +68,10 @@ If that throws a permissions error, try: //tudelft.net/staff-umbrella// /media/ cifs username=,noauto,uid=,gid=,forcegid,rw,_netdev ``` -Te values for `uid` and `gid` can be found, correspondingly, with the following commands: - -``` -id -u -id -g -``` - -These commands are VPS-specific so make sure to execute them within the same VPS where the project drives are expected to be mounted. - -### Step 6. Save the fstab file +### Step 7. Save the fstab file Use Control+C followed by `:wq` to save the file and close it to get back to your terminal. -### Step 7. Mounting the project drive +### Step 8. Mounting the project drive To mount the project drive use the `sudo mount /media/` command. You can also unmount it using `fusermount -u /media/`. ## Notes and next steps From b41d448ac5a47844957aebcf7b3f02d7be55d7dc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Ortiz?= <41450568+rortizmerino@users.noreply.github.com> Date: Thu, 1 Feb 2024 10:15:36 +0100 Subject: [PATCH 10/15] Update project_drive_mounting.md, vi and nano Added instructions for nano, kept vi as well --- docs/data/project_drive_mounting.md | 43 +++++++++++++++++++++++++++-- 1 file changed, 41 insertions(+), 2 deletions(-) diff --git a/docs/data/project_drive_mounting.md b/docs/data/project_drive_mounting.md index f060c09..a891f20 100644 --- a/docs/data/project_drive_mounting.md +++ b/docs/data/project_drive_mounting.md @@ -47,7 +47,27 @@ You may need their values for `uid` and `gid`, correspondingly, for step 6. Note: These commands are VPS-specific so make sure to execute them within the same VPS where the project drives are expected to be mounted. ### Step 5. Open the fstab file -The fstab file is where you can list the addresses of external file systems you want to mount. Find it in the /etc/ directory by typing `sudo vi fstab`. The fstab file must be in the /etc/ directory. +The fstab file is where you can list the addresses of external file systems you want to mount. The fstab file must be in the /etc/ directory. + +Click on the arrow to display instructions for your favorite editor + +
+ +vi + + + Within the /etc/ directory type `sudo vi fstab`. + +
+ +
+ +nano + + + Within the /etc/ directory type `sudo nano fstab`. + +
### Step 6. Edit the fstab file to include project storage technical details @@ -69,7 +89,26 @@ If that throws a permissions error, try: ``` ### Step 7. Save the fstab file -Use Control+C followed by `:wq` to save the file and close it to get back to your terminal. + +Click on the arrow to display instructions for the editor chosen in step 5. + +
+ +vi + + + Use `Control`+`C` followed by `:wq` to save the file and close it to get back to your terminal. + +
+ +
+ +nano + + + As indicated by the nano interface, use `Control`+`O` to write the file. Then, confirm your choice of filename by hitting `enter`. Finally, exit the file with `Control`+`X` + +
### Step 8. Mounting the project drive To mount the project drive use the `sudo mount /media/` command. You can also unmount it using `fusermount -u /media/`. From e90c08378d1b07048c854aca9cd55c2799628e1e Mon Sep 17 00:00:00 2001 From: Maurits Kok Date: Thu, 1 Feb 2024 12:08:32 +0100 Subject: [PATCH 11/15] Update project drive mounting guide --- docs/data/project_drive_mounting.md | 170 +++++++++++++++------------- 1 file changed, 91 insertions(+), 79 deletions(-) diff --git a/docs/data/project_drive_mounting.md b/docs/data/project_drive_mounting.md index a891f20..7fb73dd 100644 --- a/docs/data/project_drive_mounting.md +++ b/docs/data/project_drive_mounting.md @@ -1,130 +1,142 @@ --- section: data title: Mount Project Drive -author_1: Ashley Cryan -author_2: +author_1: Raúl Ortiz Merino +author_2: Maurits Kok --- # Mount Project Drive on Server -## Background - ## What this documentation will help achieve Project drive storage from TU Delft ICT can be mounted and made accessible in your (TU Delft) Virtual Private Server. ## Prerequisites -A (TU Delft) Virtual Private Server and space on the (TU Delft) Project Drive +- A (TU Delft) [Virtual Private Server](../infrastructure/VPS_request.md) +- (TU Delft) [Project Drive](./project_drive_request.md) ## Steps +1. Locate the URL of your project storage 1. Connect to your TU Delft VPS via SSH -2. Think of a mount point and create a new directory for it (It is a convention to mount external drives into /media) -3. Locate the URL of your project storage -4. Open the fstab file -5. Edit the fstab file to include project storage technical details -6. Save the fstab file -7. Mounting the project drive - -### Step 1. Connect to your TU Delft VPS via SSH -Follow instructions in TU Delft ICT email from initial server setup or configure 1-step connection via SSH. +1. Create a new directory as a mounting point +1. Retrieve your Linux user and group details +1. Edit the fstab file to include project storage technical details +1. Mount the project drive -### Step 2. Create a new directory as the mounting point preferably in /media -This is where you will mount the project drive. Use `cd /media` to enter the media directory, and `mkdir ` to create a new directory where the TU Delft project drive will be mounted. +### Step 1. Locate the URL of your project storage +The URL for your project drive can be obtained from either +- the email from TU Delft ICT with the confirmation of your project drive request, or +- by navigating to https://webdata.tudelft.nl/, and then through **WebDav Web Links > Staff-Umbrella > "Enter your netID and password"** to retrieve a list of your project drives. -### Step 3. Locate the URL of your project storage -This can be found either in the email from TU Delft ICT confirming project drive storage setup, or by going to https://webdata.tudelft.nl/, and then WebDav Web Links > Staff-Umbrella > Enter your netID and password > "Your Project Name". Copy everything after "https://webdata.tudelft.nl/" (this will be staff-umbrella/yourprojectname) +Copy everything after "https://webdata.tudelft.nl/" (this will be staff-umbrella/``) -### Step 4. Find and save your user and group details as preparation +### Step 2. Connect to your TU Delft VPS via SSH +Follow instructions in TU Delft ICT email from initial server setup or [configure a 1-step connection via SSH](../infrastructure/VPS_SSH.md). -Type the following commands: +### Step 3. Create a new directory as the mounting point +The convention is to create mounting points in the folder `/media`. Navigate to the folder and create a new folder with +```bash +cd /media +mkdir ``` -id -u -id -g -``` +Replace `` with the name of your choice. This will be the name of the folder where your project drive will be mounted. -You may need their values for `uid` and `gid`, correspondingly, for step 6. +### Step 4. Find and save your user and group details -Note: These commands are VPS-specific so make sure to execute them within the same VPS where the project drives are expected to be mounted. +In the terminal, you can retrieve your local user and group details with: -### Step 5. Open the fstab file -The fstab file is where you can list the addresses of external file systems you want to mount. The fstab file must be in the /etc/ directory. +```bash +id -u # User ID +id -g # Group ID +``` -Click on the arrow to display instructions for your favorite editor +You may need the values for `uid` and `gid` for step 5. -
- -vi - +```{note} +These commands are server-specific, so make sure to execute them on the server where the project drives will be mounted. +``` - Within the /etc/ directory type `sudo vi fstab`. +### Step 5. Edit the fstab file to include project storage technical details +The fstab file containes a list of the addresses of external file systems. In this file, the details of your project drive will need to be added in a single line. This line consists of four parts: +1. *filesystem* - the address of the project drive +2. *mount point* - the location in the VPS where you want to mount the project drive +3. *type* - the type of the filesystem +4. *options* - additional option such as user privileges -
-
- -nano - +The fstab file must be in the `/etc/` directory and can be opened with the `vi` or `nano` editor: - Within the /etc/ directory type `sudo nano fstab`. +````{tab-set} +```{tab-item} vi +In the terminal, enter the following command to open the fstab file in the vi editor: +`sudo vi /etc/fstab` +Then, switch to the insert mode (hit "i" to switch to insert mode and be able to type) +``` -
+```{tab-item} nano +In the terminal, enter the following command to open the fstab file in the nano editor: +`sudo nano /etc/fstab` +``` +```` -### Step 6. Edit the fstab file to include project storage technical details +Add the following line to the file: -Basically you need to add one line to this file. This line consists of four parts: (i) filesystem, (ii) mount point, (iii) type, and (iv) options. The filesystem refers to the project drive address which is specified in the ICT email. The mount point is where you want to mount the project drive in the VPS. The third part determines the type of the filesystem and in the last part you specify some options such as privileges. +``` +@sftp.tudelft.nl:/staff-umbrella/ /media/ fuse.sshfs rw,noauto,users,_netdev 0 0 +``` -To add the filesystem, open the /etc/fstab file with the sudo privilage and switch to the insert mode (hit "i" to switch to insert mode and be able to type), and write `//tudelft.net/` and paste the latter half of the URL you copied from the WebDav links. The full URL should be in this format: `//tudelft.net/staff-umbrella/yourprojectname`. +replacing the values between `<` and `>` with your NetID, the name of your project drive, and the name of the folder you created in step 3. -For the mount point, add a space followed by the location in your VPS where you want to mount the Project Drive storage (where you created the folder previously, e.g., `/media/` ). +````{note} -The full new record in the fstab file should be formatted like this: +If this configuration throws a permission error during mounting, try: ``` -@sftp.tudelft.nl:/staff-umbrella/ /media/ fuse.sshfs rw,noauto,users,_netdev 0 0 -``` -If that throws a permissions error, try: +//tudelft.net/staff-umbrella// /media/ cifs username=,noauto,uid=,gid=,forcegid,rw,_netdev ``` -//tudelft.net/staff-umbrella// /media/ cifs username=,noauto,uid=,gid=,forcegid,rw,_netdev +Use the values for `uid` and `gid` from step 4. +```` + +Close the file editor and save the changes: +````{tab-set} +```{tab-item} vi +Use `Control`+`C` followed by `:wq` to save the file and close it to get back to your terminal. ``` -### Step 7. Save the fstab file +```{tab-item} nano +As indicated by the nano interface, use `Control`+`O` to write the file. Then, confirm your choice of filename by hitting `enter`. Finally, exit the file with `Control`+`X` +``` +```` -Click on the arrow to display instructions for the editor chosen in step 5. +### Step 6. Mount the project drive +To mount the project drive execute the command -
- -vi - +```bash +sudo mount /media/ +``` - Use `Control`+`C` followed by `:wq` to save the file and close it to get back to your terminal. +You can also unmount the drive with +```bash +fusermount -u /media/ +``` -
+The project drive will not mount automatically, so you will need to remount it manually each time you restart the server. -
- -nano - +`````{note} +If the step above does not work, it probably means that the packages for mounting cifs-type filesystems haven't been installed. Depending on your linux flavour you will need to install them using: - As indicated by the nano interface, use `Control`+`O` to write the file. Then, confirm your choice of filename by hitting `enter`. Finally, exit the file with `Control`+`X` + ````{tab-set} + ```{tab-item} Ubuntu/Debian + `sudo apt install cifs-utils` + ``` -
+ ```{tab-item} Redhat/Centos/Fedora + `sudo yum install cifs-utils` + ``` + ```` -### Step 8. Mounting the project drive -To mount the project drive use the `sudo mount /media/` command. You can also unmount it using `fusermount -u /media/`. +````` ## Notes and next steps -The project drive will not mount automatically, so you will need to mount it manually each time you start the server. To do so, from the home (`/`) directory in your server run `mount /media/`. You should now be able to `cd /media/` and view your files stored on the project drive using `ls`. - -Note, if the step above does not work, it probably means that the packages for mounting cifs-type filesystems haven't been installed. Depending on your linux flavour you will need to install them first using: - -`sudo apt install cifs-utils` - -on ubuntu/debian like linuxes, or - -`sudo yum install cifs-utils` - -on redhat/centos/fedora flavours. - -The steps above can also be used to mount any storage offered by TU Delft with a WebDav link (staff-homes, staff-groups, staff-bulk, student-homes, student-groups and apps). Simply use the latter half of the URL from the WebDav web link of your storage drive (step 5), which will change from staff-umbrella (project drive) to something else depending on the storage drive you would like to mount. - +The steps above can also be used to mount any storage offered by TU Delft with a WebDav link (staff-homes, staff-groups, staff-bulk, student-homes, student-groups and apps). Simply use the latter half of the URL from the WebDav web link of your storage drive, which will change from staff-umbrella (project drive) to something else depending on the storage drive you would like to mount. From e1f41a08545e200b1a6514c22572794c2355a7ce Mon Sep 17 00:00:00 2001 From: Maurits Kok Date: Thu, 1 Feb 2024 14:57:00 +0100 Subject: [PATCH 12/15] Fix encoder bug --- docs/community/maintainers.md | 2 +- scripts/create_page_maintainers.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/community/maintainers.md b/docs/community/maintainers.md index 6448bc8..5a15a52 100644 --- a/docs/community/maintainers.md +++ b/docs/community/maintainers.md @@ -7,7 +7,7 @@ _This content is automatically generated, all changes made will be lost._ | containers | Docker users | Maurits Kok | | | data | RequestProject Drive | Ashley Cryan | | | data | Sync with Unison | Ashley Cryan | | -| data | Mount Project Drive | Ashley Cryan | | +| data | Mount Project Drive | Ral Ortiz Merino | Maurits Kok | | data | Data publishing | Aleksandra Wilczynska | | | gitlab | Transfer ownership of a GitLab repository | Lora Armstrong | Maurits Kok | | gitlab | CI with Gitlab | Ashley Cryan | Maurits Kok | diff --git a/scripts/create_page_maintainers.py b/scripts/create_page_maintainers.py index d511b5e..90cd70c 100644 --- a/scripts/create_page_maintainers.py +++ b/scripts/create_page_maintainers.py @@ -42,7 +42,7 @@ def get_filenames_in_toc(path: str) -> List: """ toc_file = os.path.join(path, "_toc.yml") files = [] - with open(toc_file, "r") as f: + with open(toc_file, "r", encoding="utf8") as f: for line in f: if "- file:" in line: files.append(line.strip().split()[-1]) From 65a66b484117e412880d6cd1630fcd67f63611b1 Mon Sep 17 00:00:00 2001 From: Maurits Kok Date: Thu, 1 Feb 2024 14:57:18 +0100 Subject: [PATCH 13/15] Rename conda environment --- environment.yml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/environment.yml b/environment.yml index e8ec641..2edc205 100644 --- a/environment.yml +++ b/environment.yml @@ -1,11 +1,10 @@ -name: dcc_guides_env +name: dcc_guides channels: - conda-forge - bioconda - defaults dependencies: - python=3.8 - - pip - numpy - pandas - tabulate From 87251c8277ee31d1eb79c07c140addf8320000df Mon Sep 17 00:00:00 2001 From: Maurits Kok Date: Fri, 2 Feb 2024 16:44:16 +0100 Subject: [PATCH 14/15] Update encoding --- docs/community/maintainers.md | 4 ++-- scripts/create_page_maintainers.py | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/community/maintainers.md b/docs/community/maintainers.md index 5a15a52..7e5299b 100644 --- a/docs/community/maintainers.md +++ b/docs/community/maintainers.md @@ -5,9 +5,9 @@ _This content is automatically generated, all changes made will be lost._ | Section | Title | Lead maintainer | Backup maintainer | |:---------------|:------------------------------------------|:----------------------|:--------------------| | containers | Docker users | Maurits Kok | | -| data | RequestProject Drive | Ashley Cryan | | +| data | Request Project Drive | Ashley Cryan | | | data | Sync with Unison | Ashley Cryan | | -| data | Mount Project Drive | Ral Ortiz Merino | Maurits Kok | +| data | Mount Project Drive | Raúl Ortiz Merino | Maurits Kok | | data | Data publishing | Aleksandra Wilczynska | | | gitlab | Transfer ownership of a GitLab repository | Lora Armstrong | Maurits Kok | | gitlab | CI with Gitlab | Ashley Cryan | Maurits Kok | diff --git a/scripts/create_page_maintainers.py b/scripts/create_page_maintainers.py index 90cd70c..7841d2b 100644 --- a/scripts/create_page_maintainers.py +++ b/scripts/create_page_maintainers.py @@ -96,7 +96,7 @@ def write_authors_page(df: pd.DataFrame, path): directory of the jupyter book """ file = os.path.join(path, "community", "maintainers.md") - with open(file, "w") as f: + with open(file, "w", encoding="utf8") as f: f.write("# Guide maintainers \n\n") f.write( "_This content is automatically generated, all changes made will be lost._ \n\n" From b7b081e437960550679a16245e9edb6ea2ecc2ef Mon Sep 17 00:00:00 2001 From: Maurits Kok Date: Fri, 16 Feb 2024 16:13:07 +0100 Subject: [PATCH 15/15] Update fair cards --- docs/software/project_cards.md | 41 ++++++++++++++++++++++++++++++---- 1 file changed, 37 insertions(+), 4 deletions(-) diff --git a/docs/software/project_cards.md b/docs/software/project_cards.md index ccb4dfd..0c20374 100644 --- a/docs/software/project_cards.md +++ b/docs/software/project_cards.md @@ -10,7 +10,7 @@ Add the cards below as issues to your repository to track the progress of adopti ^^^ ```md -Essential_ +_Essential_ - [ ] Use [git](https://www.atlassian.com/git) as a version control system - [ ] Upload your project on [GitHub](https://github.com/) or [TU Delft GitLab](https://gitlab.tudelft.nl/) @@ -94,8 +94,8 @@ _Recommended_ ```md _Essential_ -- [ ] Document how users can verify the proper -- [ ] Document verification for user installation and software execution +- [ ] Document verification of user installation +- [ ] Document how users can verify the proper functioning of the software _Recommended_ - [ ] [Defensive programming](https://swcarpentry.github.io/python-novice-inflammation/10-defensive/index.html) @@ -104,5 +104,38 @@ _Recommended_ - [ ] Code coverage check ([Codecov](https://about.codecov.io/), [Sonarcloud](https://sonarcloud.io/), [Travis](https://www.travis-ci.com/)) ``` -```` +--- +**Code quality** +^^^ + +```md +_Essential_ +- [ ] Project [organisation](https://coderefinery.github.io/reproducible-research/organizing-projects/) +- [ ] Record [software dependencies](https://coderefinery.github.io/reproducible-research/dependencies/) + +_Recommended_ +- [ ] Follow a consistent code style, such as [PEP8 guidelines](https://realpython.com/python-pep8/) +- [ ] Use a linter ([pylint](https://pypi.org/project/pylint/), [flake8](https://pypi.org/project/flake8/)) +- [ ] Use a formatter ([black](https://github.com/psf/black), [yapf](https://github.com/google/yapf)) +- [ ] Follow [best coding practices](https://alan-turing-institute.github.io/rse-course/html/module07_construction_and_design/index.html) +``` + +--- +**Releases** +^^^ + +```md +_Essential_ +- [ ] Obtain a DOI from ([Zenodo](https://zenodo.org/) or [4TU.ResearchData](https://data.4tu.nl/info/about-your-data/getting-started)) + +_Recommended_ +- [ ] [Semantic versioning](https://semver.org/) +- [ ] Tagged releases ([GitHub](https://docs.github.com/en/repositories/releasing-projects-on-github)) +- [ ] [CHANGELOG](https://keepachangelog.com/en/1.0.0/) +- [ ] Upload to registry ([PyPI](https://realpython.com/pypi-publish-python-package/), [conda](https://conda.io/projects/conda-build/en/latest/user-guide/tutorials/build-pkgs.html), [DockerHub](https://docs.docker.com/docker-hub/repos/)) +- [ ] [Release guide](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository) +- [ ] [Continuous Integration](https://the-turing-way.netlify.app/reproducible-research/ci/ci-options) for build and release +``` + +````