Built site for gh-pages

.nojekyll

@@ -1 +1 @@
-526e7cb3
+21f5a325

data-storage.html

@@ -148,7 +148,7 @@ <h2 id="toc-title">On this page</h2>
   <ul class="collapse">
   <li><a href="#the-_shared-directory" id="toc-the-_shared-directory" class="nav-link" data-scroll-target="#the-_shared-directory">The <code>_shared</code> directory</a></li>
   </ul></li>
-  <li><a href="#how-to-archive-old-home-directories-admin" id="toc-how-to-archive-old-home-directories-admin" class="nav-link" data-scroll-target="#how-to-archive-old-home-directories-admin">How to archive old home directories (admin)</a>
+  <li><a href="#how-to-archive-old-home-dir" id="toc-how-to-archive-old-home-dir" class="nav-link" data-scroll-target="#how-to-archive-old-home-dir">How to archive old home directories (admin)</a>
   <ul class="collapse">
   <li><a href="#finding-large-home-directories" id="toc-finding-large-home-directories" class="nav-link" data-scroll-target="#finding-large-home-directories">Finding large <code>$HOME</code> directories</a></li>
   <li><a href="#authenticate-with-s3-archive-bucket" id="toc-authenticate-with-s3-archive-bucket" class="nav-link" data-scroll-target="#authenticate-with-s3-archive-bucket">Authenticate with S3 archive bucket</a></li>
@@ -196,11 +196,11 @@ <h1 class="title">Data Storage Policies</h1>
 </ul>
 <section id="using-s3-bucket-storage" class="level3">
 <h3 class="anchored" data-anchor-id="using-s3-bucket-storage">Using S3 Bucket Storage</h3>
-<p>Please see the short tutorial in the Earthdata Cloud Cookbook on <a href="./how-tos/using-s3-storage.html">Using S3 Bucket Storage in NASA-Openscapes Hub</a>.</p>
+<p>Please see the short tutorial in the Earthdata Cloud Cookbook on <a href="https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/using-s3-storage.html">Using S3 Bucket Storage in NASA-Openscapes Hub</a>.</p>
 </section>
 <section id="data-retention-and-archiving-policy" class="level2">
 <h2 class="anchored" data-anchor-id="data-retention-and-archiving-policy">Data retention and archiving policy</h2>
-<p>User <code>$HOME</code> directories will be retained for six months after their last use. After a home directory has been idle for six months, it will be <a href="#how-to-archive-old-home-directories">archived to our “archive” S3 bucket, and removed</a>. If a user requests their archive back, an admin can restore it for them.</p>
+<p>User <code>$HOME</code> directories will be retained for six months after their last use. After a home directory has been idle for six months, it will be <a href="#how-to-archive-old-home-dir">archived to our “archive” S3 bucket, and removed</a>. If a user requests their archive back, an admin can restore it for them.</p>
 <p>Once a user’s home directory archive has been sitting in the archive for an additional six months, it will be permanently removed from the archive. After this it can no longer be retrieved. <!-- TODO make this automatic policy in S3 console --></p>
 <p>In addition to these policies, admins will keep an eye on the <a href="https://grafana.openscapes.2i2c.cloud/d/bd232539-52d0-4435-8a62-fe637dc822be/home-directory-usage-dashboard?orgId=1">Home Directory Usage Dashboard</a> in Grafana. When a user’s home directory increases in size to over 100GB, we will contact them and work with them to reduce the size of their home directory - by removing large unnecessary files, and moving the rest to the appropriate S3 bucket (e.g., <code>$PERSISTENT_BUCKET</code>).</p>
 <section id="the-_shared-directory" class="level3">
@@ -209,8 +209,8 @@ <h3 class="anchored" data-anchor-id="the-_shared-directory">The <code>_shared</c
 <p>This directory will follow the same policies as users’ home directories: after six months, contents will be archived to the “archive” S3 bucket (more below). After an additional six months, the archive will be deleted.</p>
 </section>
 </section>
-<section id="how-to-archive-old-home-directories-admin" class="level2">
-<h2 class="anchored" data-anchor-id="how-to-archive-old-home-directories-admin">How to archive old home directories (admin)</h2>
+<section id="how-to-archive-old-home-dir" class="level2">
+<h2 class="anchored" data-anchor-id="how-to-archive-old-home-dir">How to archive old home directories (admin)</h2>
 <p>To start, you will need to be an admin of the Openscapes Jupyterhub so that the <code>allusers</code> directory is mounted in your home directory. This will contain all users’ home directories, and you will have full read-write access.</p>
 <section id="finding-large-home-directories" class="level3">
 <h3 class="anchored" data-anchor-id="finding-large-home-directories">Finding large <code>$HOME</code> directories</h3>

index.html

@@ -154,7 +154,7 @@ <h1><img src="images/openscapes_hex.png" class="inline img-fluid"> Openscapes Ju
 <div>
 
 </div>
-<div class="quarto-layout-panel" data-layout-ncol="2" style="margin-top: 3em;">
+<div class="quarto-layout-panel" style="margin-top: 3em;" data-layout-ncol="2">
 <div class="quarto-layout-row">
 <div class="text-center quarto-layout-cell" style="flex-basis: 50.0%;justify-content: flex-start;">
 <div class="login-area">

password-access.html

@@ -197,7 +197,7 @@ <h2 class="anchored" data-anchor-id="using-the-openscapes-2i2c-hub-in-a-workshop
 What hours of the day will participants be active? 1:00pm - 5pm EEST (Athens, Greece).<br>
 Number of attendees: ~40<br>
 Resources per user: 14.8GB RAM / up to 3.7 CPU<br>
-The URL of the hub that will be used for the event: https://workshop.openscapes.2i2c.cloud/<br>
+The URL of the hub that will be used for the event: <a href="https://workshop.openscapes.2i2c.cloud/" class="uri">https://workshop.openscapes.2i2c.cloud/</a><br>
 Access method: shared password<br>
 Password choice: [YouChooseAPassword]</p>
 <p>Thank you!</p>

search.json

@@ -84,8 +84,8 @@
     "text": "Data retention and archiving policy\nUser $HOME directories will be retained for six months after their last use. After a home directory has been idle for six months, it will be archived to our “archive” S3 bucket, and removed. If a user requests their archive back, an admin can restore it for them.\nOnce a user’s home directory archive has been sitting in the archive for an additional six months, it will be permanently removed from the archive. After this it can no longer be retrieved. \nIn addition to these policies, admins will keep an eye on the Home Directory Usage Dashboard in Grafana. When a user’s home directory increases in size to over 100GB, we will contact them and work with them to reduce the size of their home directory - by removing large unnecessary files, and moving the rest to the appropriate S3 bucket (e.g., $PERSISTENT_BUCKET).\n\nThe _shared directory\nThe _shared directory is a place where instructors can put workshop materials for participants to access. It is mounted as /home/jovyan/shared, and is read only for all users. For those with admin access to the Hub, it is also mounted as a writeable directory as /home/jovyan/shared-readwrite.\nThis directory will follow the same policies as users’ home directories: after six months, contents will be archived to the “archive” S3 bucket (more below). After an additional six months, the archive will be deleted."
   },
   {
-    "objectID": "data-storage.html#how-to-archive-old-home-directories-admin",
-    "href": "data-storage.html#how-to-archive-old-home-directories-admin",
+    "objectID": "data-storage.html#how-to-archive-old-home-dir",
+    "href": "data-storage.html#how-to-archive-old-home-dir",
     "title": "Data Storage Policies",
     "section": "How to archive old home directories (admin)",
     "text": "How to archive old home directories (admin)\nTo start, you will need to be an admin of the Openscapes Jupyterhub so that the allusers directory is mounted in your home directory. This will contain all users’ home directories, and you will have full read-write access.\n\nFinding large $HOME directories\nLook at the Home Directory Usage Dashboard in Grafana to see the directories that haven’t been used in a long time and/or are very large.\nYou can also view and sort users’ directories by size in the Hub with the following command, though this takes a while because it has to summarize a lot of files and directories. This will show the 30 largest home directories:\ndu -h --max-depth=1 /home/jovyan/allusers/ | sort -hr | head -n 30\n\n\nAuthenticate with S3 archive bucket\nWe have created an AWS IAM user called archive-homedirs with appropriate permissions to write to the openscapeshub-prod-homedirs-archive bucket. Get access keys for this user from the AWS console, and use these keys to authenticate in the Hub:\nIn the terminal, type:\nawsv2 configure\nEnter the access key and secret key at the prompts, and set default region to us-west-2.\nYou will also need to temporarily unset some AWS environment variables that have been configured to authenticate with NASA S3 storage. (These will be reset the next time you log in):\nunset AWS_ROLE_ARN\nunset AWS_WEB_IDENTITY_TOKEN_FILE\nTest to make sure you can access the archive bucket:\n# test s3 access:\nawsv2 s3 ls s3://openscapeshub-prod-homedirs-archive/archives/\ntouch test123.txt\nawsv2 s3 mv test123.txt s3://openscapeshub-prod-homedirs-archive/archives/\nawsv2 s3 rm s3://openscapeshub-prod-homedirs-archive/archives/test123.txt\n\n\nSetting up and running the archive script\nWe use a python script, developed by @yuvipanda, that reproducibly archives a list of users’ directories into a specified S3 bucket.\nCopy the script into your home directory in the Hub.\nIn the Hub as of 2024-05-17, a couple of dependencies for the script are missing; you can install them before running the script:\npip install escapism\n\n# I had solver errors with pigz so needed to use the classic solver. \n# Also, the installation of pigz required a machine with >= 3.7GB memory\nconda install pigz --solver classic\nCreate a text file, with one username per line, of users’ home directories you would like to archive to s3. It will look like:\nusername1\nusername2\n# etc...\nFinally, run the script from the terminal, changing the parameter values as required:\npython3 archive-home-dirs.py \\\n    --archive-name=\"archive-$(date +'%Y-%m-%d')\" \\\n    --basedir=/home/jovyan/allusers/ \\\n    --bucket-name=openscapeshub-prod-homedirs-archive \\\n    --object-prefix=\"archives/\" \\\n    --usernames-file=users-to-archive.txt \\\n    --temp-path=/home/jovyan/archive-staging/\nOmitted in the above example, but available to use, is the --delete flag, which will delete the users’ home directory once the archive is completed.\nIf you don’t use the --delete flag, first verify that the archive was successfully completed and then remove the user’s home directory manually.\n\n\nArchiving the shared directory\nYou can use the same script to archive directories in the shared directory, by modifying the inputs slightly:\n\nSet --basedir=/home/jovyan/shared/, (or --basedir=/home/jovyan/shared-readwrite/ if you want to be able use the --delete flag).\nCreate a file with a list of directories in the shared directory you want to archive, and pass it to the --usernames-file argument.\nSet --object-prefix=\"archives/_shared/ to put the archives in the _shared subdirectory in the archive bucket.\n\nE.g.:\npython3 archive-home-dirs.py \\\n    --archive-name=\"archive-$(date +'%Y-%m-%d')\" \\\n    --basedir=/home/jovyan/shared/ \\\n    --bucket-name=openscapeshub-prod-homedirs-archive \\\n    --object-prefix=\"archives/_shared/\" \\\n    --usernames-file=/home/jovyan/shared-to-archive.txt \\\n    --temp-path=/home/jovyan/archive-staging/\nBy default, archives (.tar.gz) are created in your /tmp directory before upload to the S3 bucket. The /tmp directory is cleared out when you shut down the Hub. However, /tmp has limited space (80GB shared by up to four users on a single node), so if you are archiving many large directories, you will likely need to specify a location in your $HOME directory by passing a path to the --temp-path argument. The script will endeavour to clean up after itself and remove the tar.gz file after uploading, but double check that directory when you are finished or you may have copies of all of the other user directories in your own $HOME!"

sitemap.xml

@@ -2,30 +2,30 @@
 <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <url>
     <loc>https://openscapes.cloud/about.html</loc>
-    <lastmod>2024-09-27T22:41:50.688Z</lastmod>
+    <lastmod>2024-10-02T17:42:29.050Z</lastmod>
   </url>
   <url>
     <loc>https://openscapes.cloud/fledging.html</loc>
-    <lastmod>2024-09-27T22:41:50.688Z</lastmod>
+    <lastmod>2024-10-02T17:42:29.050Z</lastmod>
   </url>
   <url>
     <loc>https://openscapes.cloud/github-access.html</loc>
-    <lastmod>2024-09-27T22:41:50.688Z</lastmod>
+    <lastmod>2024-10-02T17:42:29.050Z</lastmod>
   </url>
   <url>
     <loc>https://openscapes.cloud/data-storage.html</loc>
-    <lastmod>2024-09-27T22:41:50.688Z</lastmod>
+    <lastmod>2024-10-02T17:42:29.050Z</lastmod>
   </url>
   <url>
     <loc>https://openscapes.cloud/index.html</loc>
-    <lastmod>2024-09-27T22:41:50.701Z</lastmod>
+    <lastmod>2024-10-02T17:42:29.063Z</lastmod>
   </url>
   <url>
     <loc>https://openscapes.cloud/password-access.html</loc>
-    <lastmod>2024-09-27T22:41:50.701Z</lastmod>
+    <lastmod>2024-10-02T17:42:29.064Z</lastmod>
   </url>
   <url>
     <loc>https://openscapes.cloud/access-policies.html</loc>
-    <lastmod>2024-09-27T22:41:50.688Z</lastmod>
+    <lastmod>2024-10-02T17:42:29.050Z</lastmod>
   </url>
 </urlset>