Skip to content

Commit

Permalink
Rename add-many / add-one CLI commands
Browse files Browse the repository at this point in the history
fixes #24
  • Loading branch information
chrisjsewell committed Mar 12, 2020
1 parent 9a43e29 commit 6b2756d
Show file tree
Hide file tree
Showing 4 changed files with 54 additions and 49 deletions.
88 changes: 44 additions & 44 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ to come ...

## Example CLI usage

<!-- This section was auto-generated by: tests/make_cli_readme.py -->
<!-- This section was auto-generated on 2020-03-12 16:55 by: /Users/cjs14/GitHub/jupyter-cache/tests/make_cli_readme.py -->

From the checked-out repository folder:

Expand Down Expand Up @@ -83,13 +83,13 @@ Options:
--help Show this message and exit.

Commands:
add-many Cache notebook(s) that have already been executed.
add-one Cache a notebook, with possible artefact files.
cat-artifact Print the contents of a cached artefact.
diff-nb Print a diff of a notebook to one stored in the cache.
list List cached notebook records in the cache.
remove Remove notebooks stored in the cache.
show Show details of a cached notebook in the cache.
add Cache notebook(s) that have already been executed.
add-with-artefacts Cache a notebook, with possible artefact files.
cat-artifact Print the contents of a cached artefact.
diff-nb Print a diff of a notebook to one stored in the cache.
list List cached notebook records in the cache.
remove Remove notebooks stored in the cache.
show Show details of a cached notebook in the cache.
```

The first time the cache is required, it will be lazily created:
Expand All @@ -107,7 +107,7 @@ When caching, a check will be made that the notebooks look to have been executed
correctly, i.e. the cell execution counts go sequentially up from 1.

```console
$ jcache cache add-many tests/notebooks/basic.ipynb
$ jcache cache add tests/notebooks/basic.ipynb
Caching: ../tests/notebooks/basic.ipynb
Validity Error: Expected cell 1 to have execution_count 1 not 2
The notebook may not have been executed, continue caching? [y/N]: y
Expand All @@ -117,7 +117,7 @@ Success!
Or to skip validation:

```console
$ jcache cache add-many --no-validate tests/notebooks/basic.ipynb tests/notebooks/basic_failing.ipynb tests/notebooks/basic_unrun.ipynb tests/notebooks/complex_outputs.ipynb tests/notebooks/external_output.ipynb
$ jcache cache add --no-validate tests/notebooks/basic.ipynb tests/notebooks/basic_failing.ipynb tests/notebooks/basic_unrun.ipynb tests/notebooks/complex_outputs.ipynb tests/notebooks/external_output.ipynb
Caching: ../tests/notebooks/basic.ipynb
Caching: ../tests/notebooks/basic_failing.ipynb
Caching: ../tests/notebooks/basic_unrun.ipynb
Expand All @@ -141,17 +141,17 @@ You can remove cached records by their ID.
$ jcache cache list
ID URI Created Accessed
---- ------------------------------------- ---------------- ----------------
5 tests/notebooks/external_output.ipynb 2020-02-29 17:48 2020-02-29 17:48
4 tests/notebooks/complex_outputs.ipynb 2020-02-29 17:48 2020-02-29 17:48
3 tests/notebooks/basic_unrun.ipynb 2020-02-29 17:48 2020-02-29 17:48
2 tests/notebooks/basic_failing.ipynb 2020-02-29 17:48 2020-02-29 17:48
5 tests/notebooks/external_output.ipynb 2020-03-12 16:55 2020-03-12 16:55
4 tests/notebooks/complex_outputs.ipynb 2020-03-12 16:55 2020-03-12 16:55
3 tests/notebooks/basic_unrun.ipynb 2020-03-12 16:55 2020-03-12 16:55
2 tests/notebooks/basic_failing.ipynb 2020-03-12 16:55 2020-03-12 16:55
```

You can also cache notebooks with artefacts
(external outputs of the notebook execution).

```console
$ jcache cache add-one -nb tests/notebooks/basic.ipynb tests/notebooks/artifact_folder/artifact.txt
$ jcache cache add-with-artefacts -nb tests/notebooks/basic.ipynb tests/notebooks/artifact_folder/artifact.txt
Caching: ../tests/notebooks/basic.ipynb
Validity Error: Expected cell 1 to have execution_count 1 not 2
The notebook may not have been executed, continue caching? [y/N]: y
Expand All @@ -164,8 +164,8 @@ Show a full description of a cached notebook by referring to its ID
$ jcache cache show 6
ID: 6
URI: ../tests/notebooks/basic.ipynb
Created: 2020-02-29 17:48
Accessed: 2020-02-29 17:48
Created: 2020-03-12 16:55
Accessed: 2020-03-12 16:55
Hashkey: 818f3412b998fcf4fe9ca3cca11a3fc3
Artifacts:
- artifact_folder/artifact.txt
Expand All @@ -174,7 +174,7 @@ Artifacts:
Note artefact paths must be 'upstream' of the notebook folder:

```console
$ jcache cache add-one -nb tests/notebooks/basic.ipynb tests/test_db.py
$ jcache cache add-with-artefacts -nb tests/notebooks/basic.ipynb tests/test_db.py
Caching: ../tests/notebooks/basic.ipynb
Artifact Error: Path '../tests/test_db.py' is not in folder '../tests/notebooks''
```
Expand Down Expand Up @@ -236,12 +236,12 @@ Options:
--help Show this message and exit.

Commands:
add-many Stage notebook(s) for execution.
add-one Stage a notebook, with possible asset files.
list List notebooks staged for possible execution.
remove-ids Un-stage notebook(s), by ID.
remove-uris Un-stage notebook(s), by URI.
show Show details of a staged notebook.
add Stage notebook(s) for execution.
add-with-assets Stage a notebook, with possible asset files.
list List notebooks staged for possible execution.
remove-ids Un-stage notebook(s), by ID.
remove-uris Un-stage notebook(s), by URI.
show Show details of a staged notebook.
```

Staged notebooks are recorded as pointers to their URI,
Expand All @@ -252,7 +252,7 @@ you can list them to see which have existing records in the cache (by hash),
and which will require execution:

```console
$ jcache stage add-many tests/notebooks/basic.ipynb tests/notebooks/basic_failing.ipynb tests/notebooks/basic_unrun.ipynb tests/notebooks/complex_outputs.ipynb tests/notebooks/external_output.ipynb
$ jcache stage add tests/notebooks/basic.ipynb tests/notebooks/basic_failing.ipynb tests/notebooks/basic_unrun.ipynb tests/notebooks/complex_outputs.ipynb tests/notebooks/external_output.ipynb
Staging: ../tests/notebooks/basic.ipynb
Staging: ../tests/notebooks/basic_failing.ipynb
Staging: ../tests/notebooks/basic_unrun.ipynb
Expand All @@ -265,11 +265,11 @@ Success!
$ jcache stage list
ID URI Created Assets Cache ID
---- ------------------------------------- ---------------- -------- ----------
5 tests/notebooks/external_output.ipynb 2020-02-29 17:48 0 5
4 tests/notebooks/complex_outputs.ipynb 2020-02-29 17:48 0
3 tests/notebooks/basic_unrun.ipynb 2020-02-29 17:48 0 6
2 tests/notebooks/basic_failing.ipynb 2020-02-29 17:48 0 2
1 tests/notebooks/basic.ipynb 2020-02-29 17:48 0 6
5 tests/notebooks/external_output.ipynb 2020-03-12 16:55 0 5
4 tests/notebooks/complex_outputs.ipynb 2020-03-12 16:55 0
3 tests/notebooks/basic_unrun.ipynb 2020-03-12 16:55 0 6
2 tests/notebooks/basic_failing.ipynb 2020-03-12 16:55 0 2
1 tests/notebooks/basic.ipynb 2020-03-12 16:55 0 6
```

You can remove a staged notebook by its URI or ID:
Expand Down Expand Up @@ -315,10 +315,10 @@ that are inside the notebook folder, and data supplied by the executor.
$ jcache stage list
ID URI Created Assets Cache ID
---- ------------------------------------- ---------------- -------- ----------
5 tests/notebooks/external_output.ipynb 2020-02-29 17:48 0 5
3 tests/notebooks/basic_unrun.ipynb 2020-02-29 17:48 0 6
2 tests/notebooks/basic_failing.ipynb 2020-02-29 17:48 0
1 tests/notebooks/basic.ipynb 2020-02-29 17:48 0 6
5 tests/notebooks/external_output.ipynb 2020-03-12 16:55 0 5
3 tests/notebooks/basic_unrun.ipynb 2020-03-12 16:55 0 6
2 tests/notebooks/basic_failing.ipynb 2020-03-12 16:55 0
1 tests/notebooks/basic.ipynb 2020-03-12 16:55 0 6
```

Execution data (such as execution time) will be stored in the cache record:
Expand All @@ -327,11 +327,11 @@ Execution data (such as execution time) will be stored in the cache record:
$ jcache cache show 6
ID: 6
URI: ../tests/notebooks/basic_unrun.ipynb
Created: 2020-02-29 17:48
Accessed: 2020-02-29 17:48
Created: 2020-03-12 16:55
Accessed: 2020-03-12 16:55
Hashkey: 818f3412b998fcf4fe9ca3cca11a3fc3
Data:
execution_seconds: 1.2727476909999993
execution_seconds: 1.0545749530000004

```

Expand All @@ -341,18 +341,18 @@ Failed notebooks will not be cached, but the exception traceback will be added t
$ jcache stage show 2
ID: 2
URI: ../tests/notebooks/basic_failing.ipynb
Created: 2020-02-29 17:48
Created: 2020-03-12 16:55
Failed Last Execution!
Traceback (most recent call last):
File "../jupyter_cache/executors/basic.py", line 152, in execute
executenb(nb_bundle.nb, cwd=tmpdirname)
File "//anaconda/envs/mistune/lib/python3.7/site-packages/nbconvert/preprocessors/execute.py", line 737, in executenb
File "/anaconda/envs/mistune/lib/python3.7/site-packages/nbconvert/preprocessors/execute.py", line 737, in executenb
return ep.preprocess(nb, resources, km=km)[0]
File "//anaconda/envs/mistune/lib/python3.7/site-packages/nbconvert/preprocessors/execute.py", line 405, in preprocess
File "/anaconda/envs/mistune/lib/python3.7/site-packages/nbconvert/preprocessors/execute.py", line 405, in preprocess
nb, resources = super(ExecutePreprocessor, self).preprocess(nb, resources)
File "//anaconda/envs/mistune/lib/python3.7/site-packages/nbconvert/preprocessors/base.py", line 69, in preprocess
File "/anaconda/envs/mistune/lib/python3.7/site-packages/nbconvert/preprocessors/base.py", line 69, in preprocess
nb.cells[index], resources = self.preprocess_cell(cell, resources, index)
File "//anaconda/envs/mistune/lib/python3.7/site-packages/nbconvert/preprocessors/execute.py", line 448, in preprocess_cell
File "/anaconda/envs/mistune/lib/python3.7/site-packages/nbconvert/preprocessors/execute.py", line 448, in preprocess_cell
raise CellExecutionError.from_cell_and_msg(cell, out)
nbconvert.preprocessors.execute.CellExecutionError: An error occurred while executing the following cell:
------------------
Expand Down Expand Up @@ -388,15 +388,15 @@ As with artefacts, these files must be in the same folder as the notebook,
or a sub-folder.

```console
$ jcache stage add-one -nb tests/notebooks/basic.ipynb tests/notebooks/artifact_folder/artifact.txt
$ jcache stage add-with-assets -nb tests/notebooks/basic.ipynb tests/notebooks/artifact_folder/artifact.txt
Success!
```

```console
$ jcache stage show 1
ID: 1
URI: ../tests/notebooks/basic.ipynb
Created: 2020-02-29 17:48
Created: 2020-03-12 16:55
Cache ID: 6
Assets:
- ../tests/notebooks/artifact_folder/artifact.txt
Expand Down
4 changes: 2 additions & 2 deletions jupyter_cache/cli/commands/cmd_cache.py
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ def cache_file(db, nbpath, validate, overwrite, artifact_paths=()):
return True


@cmnd_cache.command("add-one")
@cmnd_cache.command("add-with-artefacts")
@arguments.ARTIFACT_PATHS
@options.NB_PATH
@options.CACHE_PATH
Expand All @@ -145,7 +145,7 @@ def cache_nb(cache_path, artifact_paths, nbpath, validate, overwrite):
click.secho("Success!", fg="green")


@cmnd_cache.command("add-many")
@cmnd_cache.command("add")
@arguments.NB_PATHS
@options.CACHE_PATH
@options.VALIDATE_NB
Expand Down
4 changes: 2 additions & 2 deletions jupyter_cache/cli/commands/cmd_stage.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ def cmnd_stage():
pass


@cmnd_stage.command("add-many")
@cmnd_stage.command("add")
@arguments.NB_PATHS
@options.CACHE_PATH
def stage_nbs(cache_path, nbpaths):
Expand All @@ -26,7 +26,7 @@ def stage_nbs(cache_path, nbpaths):
click.secho("Success!", fg="green")


@cmnd_stage.command("add-one")
@cmnd_stage.command("add-with-assets")
@arguments.ASSET_PATHS
@options.NB_PATH
@options.CACHE_PATH
Expand Down
7 changes: 6 additions & 1 deletion tests/make_cli_readme.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
from datetime import datetime
from glob import glob
import os
from textwrap import dedent
Expand Down Expand Up @@ -30,7 +31,11 @@ def main():
get_string(cmd_main.clear_cache, input="y")

strings = []
strings.append("<!-- This section was auto-generated by: {} -->".format(__file__))
strings.append(
"<!-- This section was auto-generated on {} by: {} -->".format(
datetime.now().isoformat(" ", "minutes"), __file__
)
)
strings.append("From the checked-out repository folder:")
strings.append(get_string(cmd_main.jcache, None, ["--help"]))
strings.append(
Expand Down

0 comments on commit 6b2756d

Please sign in to comment.