Merge branch 'main' of https://github.com/CCBR/spacesavers2

CCBR · Feb 28, 2024 · d87df1b · d87df1b
2 parents 5daaa31 + d7e4d5d
commit d87df1b
Show file tree

Hide file tree

Showing 15 changed files with 242 additions and 687 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,9 +1,17 @@
 ## spacesavers2 development version
 
+- Move the report to a separate internal repository (#79, @kelly-sovacool)
+
 ### New features
 
 ### Bug fixes
 
+## spacesavers2 0.11.5
+
+### New features
+
+- new command `spacesavers2_pdq` to get per-user number of files and number of bytes
+
 ## spacesavers2 0.11.4
 
 ### New features
@@ -54,11 +62,11 @@
 - `grubbers` `--limit` can be < 1 GiB (float) (#70, @kopardev)
 - `grubbers` output file format changed. New original file column added. Original file is required by `usurp`.
 - `mimeo` `--duplicateonly` now correctly handles duplicates owned by different UIDs. (#71, @kopardev)
-    - Update `blamematrix` and to account for corrected duplicate handling in `mimeo`.
+  - Update `blamematrix` and to account for corrected duplicate handling in `mimeo`.
 - `usurp` now uses the new "original file" column from `grubbers` while creating hard-links.
 - Total size now closely resembles `df` results (fix #75 @kopardev)
 - Files with future timestamps are handled correctly (fix #76, @kopardev)
-  
+
 ## spacesavers2 0.10.2
 
 - Now tracking user-facing changes with a changelog. (#61, @kelly-sovacool)

diff --git a/README.md b/README.md
@@ -29,6 +29,7 @@ Welcome! `spacesavers2`:
 - spacesavers2_grubbers
 - spacesavers2_e2e
 - spacesavers2_usurp
+- spacesavers2_pdq
 
 ## `spacesavers2` typical workflow looks like this:
 

diff --git a/bin/render.R b/bin/render.R
diff --git a/bin/render.sh b/bin/render.sh
diff --git a/bin/render_report_biowulf.sh b/bin/render_report_biowulf.sh
diff --git a/bin/spacesavers2_pdq b/bin/spacesavers2_pdq
@@ -0,0 +1 @@
+redirect
diff --git a/docker/spacesavers2/Dockerfile b/docker/spacesavers2/Dockerfile
diff --git a/docker/spacesavers2/environment.txt b/docker/spacesavers2/environment.txt
diff --git a/docker/spacesavers2/meta.yml b/docker/spacesavers2/meta.yml
diff --git a/docs/index.md b/docs/index.md
@@ -30,6 +30,7 @@
 - [spacesavers2_blamematrix](blamematrix.md)
 - [spacesavers2_usurp](usurp.md)
 - [spacesavers2_e2e](e2e.md)
+- [spacesavers2_pdq](pdq.md)
 
 ## Use case
 

diff --git a/docs/pdq.md b/docs/pdq.md
@@ -0,0 +1,60 @@
+## spacesavers2_pdq
+
+pdq = Pretty Darn Quick
+
+This uses `glob` library to list all files in a user-provided folder recursively. 
+
+For each user it gathers information like:
+ - total number of files
+ - total number of bytes
+
+It is quick tool to gather datapoints to monitor filesystem usage. Typically, can be run once daily and compared with previous days run to find large changes.
+
+### Inputs
+ - `--folder`: Path to the folder to run `spacesavers2_pdq` on.
+ - `--threads`: `spacesavers2_pdq` uses multiprocessing library to parallelize orchestration. This defines the number of threads to run in parallel.
+ - `--outfile`: If not supplied then the optput is written to the screen.
+
+> NOTE: `spacesavers2_pdq` reports errors (eg. cannot read file) to STDERR
+
+```bash
+usage: spacesavers2_pdq [-h] -f FOLDER [-p THREADS] [-o OUTFILE] [-v]
+
+spacesavers2_pdq: get quick per user info (number of files and bytes).
+
+options:
+  -h, --help            show this help message and exit
+  -f FOLDER, --folder FOLDER
+                        spacesavers2_pdq will be run on all files in this folder and its subfolders
+  -p THREADS, --threads THREADS
+                        number of threads to be used (default 4)
+  -o OUTFILE, --outfile OUTFILE
+                        outfile ... catalog file .. by default output is printed to screen
+  -v, --version         show program's version number and exit
+
+Version:
+    v0.11.5
+Example:
+    > spacesavers2_pdq -f /path/to/folder -p 4 -o /path/to/output_file
+```
+
+### Output
+
+## tab-delimited output (file)
+
+`spacesavers2_pdq` creates one tab seperated output line per user:
+
+```bash
+% head -n1 test.out
+user1       1386138 6089531321856
+user2  230616  2835680212992
+user3      1499    126442496
+```
+The 3 items in the line are as follows:
+
+
+| Column | Description              | Example                                                                                        |
+| ------ | ------------------------ | ---------------------------------------------------------------------------------------------- |
+| 1      | username                 | "user1" |
+| 2      | total no. of files owned     | 1386138                                                                                          |
+| 3      | total no. of bytes occupied      | 6089531321856                                                                                        |