Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Modernize the RACK cli packaging #993

Merged
merged 8 commits into from
Aug 25, 2023
Merged

Modernize the RACK cli packaging #993

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
ba71902
Modernize the RACK cli packaging
glguy Aug 21, 2023
8413681
adjust ci
glguy Aug 21, 2023
f10f94b
Try to fix CI
glguy Aug 23, 2023
bad024f
fix python lints
glguy Aug 23, 2023
862a0ab
update semtk-python3 version in scrappingtoolkit
glguy Aug 23, 2023
5ac3679
Try to fix wheel installation
glguy Aug 23, 2023
35fb0f0
Fix docker image build
glguy Aug 23, 2023
8145ffe
CI cleanups
glguy Aug 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions .github/workflows/assemble-files.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ jobs:
# This path is specific to Ubuntu
path: ~/.cache/pip
# Look to see if there is a cache hit for the corresponding requirements file
key: ${{ runner.os }}-pip-${{ hashFiles('RACK/cli/requirements.txt') }}
key: ${{ runner.os }}-pip-${{ hashFiles('RACK/cli/pyproject.toml') }}
restore-keys: |
${{ runner.os }}-pip-
${{ runner.os }}-
Expand Down Expand Up @@ -88,7 +88,6 @@ jobs:
run: |
cd RACK/cli
python3 -m pip install --upgrade pip setuptools wheel
pip3 wheel --wheel-dir=wheels -r requirements.txt
pip3 wheel --wheel-dir=wheels .
cd ${{ github.workspace }}
tar cfz RACK/rack-box/files/rack-cli.tar.gz RACK/cli/{*.sh,wheels}
Expand All @@ -112,7 +111,7 @@ jobs:
- name: Generate CDR files
shell: bash
run: |
pip3 install RACK/cli/wheels/*.whl
pip3 install --no-dependencies RACK/cli/wheels/*.whl
tar xfz RACK/rack-box/files/semtk.tar.gz semtk-opensource/standaloneExecutables/target/standaloneExecutables-jar-with-dependencies.jar
RACK/nodegroups/generate-cdrs.sh semtk-opensource/standaloneExecutables/target/standaloneExecutables-jar-with-dependencies.jar

Expand Down
16 changes: 9 additions & 7 deletions .github/workflows/continuous.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,19 +29,21 @@ jobs:
# This path is specific to Ubuntu
path: ~/.cache/pip
# Look to see if there is a cache hit for the corresponding requirements file
key: ${{ runner.os }}-pip-${{ hashFiles('cli/dev/requirements.txt') }}
key: ${{ runner.os }}-pip-${{ hashFiles('cli/pyproject.toml') }}
restore-keys: |
${{ runner.os }}-pip-
${{ runner.os }}-

- name: Install RACK CLI development dependencies
run: pip3 install -q -r cli/dev/requirements.txt
run: |
cd cli
pip3 install ".[dev]"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not familiar with this ".[dev]". What does it mean?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is (kind of concerning) pip syntax that means to install the package in the current directory . using the optional dependency set dev. Another option was for me to write rack[dev]


- name: Lint RACK CLI
run: |
cd cli
pylint .
mypy .
pylint rack
mypy rack

- name: Lint RACK Ontology
run: |
Expand Down Expand Up @@ -108,9 +110,9 @@ jobs:

- name: Run rack-box tests
run: |
cd RACK
pip3 install -r cli/requirements.txt
pip3 install cli/.
cd RACK/cli
pip3 install .
cd ..
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Too bad that you need two cd commands. What makes pip3 install ./cli no longer work?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First I tried what you wrote, and that didn't work. I'm new to using pyproject.toml and when I found that the code committed above working, I went with it.

Your question prompted me to try harder and I learned that a trailing / indicates to pip that what I want is for it to install a directory, rather than a package name. I've updated the script to use a single command and saved the extra cd for another day!

pip3 install -r tests/requirements.txt
python3 -m pytest tests

Expand Down
2 changes: 1 addition & 1 deletion ScrapingToolKit/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@ colorama
lxml
graphviz
ftfy
semtk-python3 @ git+https://github.com/ge-semtk/semtk-python3@d44a45941d06ec6466bbe61bfd81cd667d6a5137
semtk-python3 @ git+https://github.com/ge-semtk/semtk-python3@7995623e12e93b9b252200b9bd507949b8c0a430
8 changes: 4 additions & 4 deletions cli/dev/README.md → cli/DEVELOPMENT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ are used to statically type-check the code. Usage is simple:

```shell
source venv/bin/activate
pip install -r dev/requirements.txt
mypy .
pip install ".[dev]"
mypy rack
```

## Tests
Expand All @@ -25,8 +25,8 @@ You can run the tests with `pytest`:

```shell
source venv/bin/activate
pip install -r dev/requirements.txt
python3 setup.py install && pytest
pip install ".[dev]"
pytest
```

Note that you should _always install the code before testing_.
Expand Down
6 changes: 2 additions & 4 deletions cli/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,17 +46,15 @@ the RACK-Box README.
```shell
python3 -m venv venv
source venv/bin/activate
pip install --force -r requirements.txt
python3 setup.py install
pip install --force-reinstall .
glguy marked this conversation as resolved.
Show resolved Hide resolved
```

If running on Windows, [GitBash](https://gitforwindows.org/) can be used with the following commands.

```shell
python -m venv venv
source venv/Scripts/activate
pip install --force -r requirements.txt
python setup.py install
pip install --force-reinstall .
glguy marked this conversation as resolved.
Show resolved Hide resolved
```

## Initialize RACK-in-a-Box
Expand Down
8 changes: 0 additions & 8 deletions cli/dev/requirements.txt
View file
Open in desktop

This file was deleted.

48 changes: 48 additions & 0 deletions cli/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "rack"
version = "0.1"
description = "Command-line interface to RACK"
requires-python = ">=3.7"

dependencies = [
"ase==3.22.1",
"attrs==23.1.0",
"certifi==2023.7.22",
"chardet==5.1.0",
"colorama==0.4.6",
"idna==3.4",
"importlib-metadata==6.7.0",
"jsonschema==4.17.3",
"pyrsistent==0.19.3",
"python-dateutil==2.8.2",
"PyYAML==6.0",
"requests==2.31.0",
"Pillow==9.4.0",
"plotly==5.15.0",
"semtk-python3 @ git+https://github.com/ge-semtk/semtk-python3@7995623e12e93b9b252200b9bd507949b8c0a430",
"six==1.16.0",
"tabulate==0.9.0",
"urllib3==2.0.3",
"zipp==3.15.0",
]

optional-dependencies.dev = [
"mypy==1.0.0",
"pylint==2.16.2",
"pytest-docker==1.0.1",
"pytest==7.2.1",
"typed-ast==1.5.4",
"types-PyYAML==6.0.12.6",
"types-requests==2.28.11.13",
"types-tabulate==0.9.0.0",
]
glguy marked this conversation as resolved.
Show resolved Hide resolved

[tool.setuptools]
packages = ["rack"]

[project.scripts]
rack = "rack.rack_cmd:main"
9 changes: 6 additions & 3 deletions cli/rack/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -793,10 +793,10 @@ def delete() -> None:
semtk3.delete_nodegroup_from_store(nodegroup)
delete()

def delete_store_item(id: str, item_type: str) -> None:
@with_status(f'Deleting {str_highlight(id)}')
def delete_store_item(ident: str, item_type: str) -> None:
@with_status(f'Deleting {str_highlight(ident)}')
def delete() -> None:
semtk3.delete_store_item(id, item_type)
semtk3.delete_store_item(ident, item_type)
delete()

def delete_nodegroups_driver(nodegroups: List[str], ignore_nonexistent: bool, yes: bool, use_regexp: bool, base_url: Url) -> None:
Expand Down Expand Up @@ -868,17 +868,20 @@ def dispatch_manifest_build(args: SimpleNamespace) -> None:

def dispatch_data_import(args: SimpleNamespace) -> None:
"""Implementation of the data import subcommand"""
global cliMethod
cliMethod = CLIMethod.DATA_IMPORT
ingest_data_driver(Path(args.config), args.base_url, args.model_graph, args.data_graph, args.triple_store, args.triple_store_type, args.clear)

def dispatch_data_cardinality(args: SimpleNamespace) -> None:
"""Implementation of the data cardinality subcommand"""
global cliMethod
cliMethod = CLIMethod.DATA_IMPORT
cardinality_driver(args.base_url, args.model_graph, args.data_graph, args.triple_store, args.triple_store_type,
export_format=args.format, headers=not args.no_headers, concise=args.concise, max_rows=args.max_rows)

def dispatch_model_import(args: SimpleNamespace) -> None:
"""Implementation of the plumbing model subcommand"""
global cliMethod
cliMethod = CLIMethod.MODEL_IMPORT
ingest_owl_driver(Path(args.config), args.base_url, args.model_graph, args.triple_store, args.triple_store_type, args.clear)

Expand Down
2 changes: 1 addition & 1 deletion cli/bin/rack → cli/rack/rack_cmd.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@

logger = logging.getLogger(__name__)

if __name__ == "__main__":
def main() -> None:
# Sets up colors for Windows users
colorama.init()

Expand Down
19 changes: 0 additions & 19 deletions cli/requirements.txt
View file
Open in desktop

This file was deleted.

7 changes: 0 additions & 7 deletions cli/setup.py
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion rack-box/scripts/install.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,5 +183,5 @@ done
# Setup the RACK dataset using the RACK CLI

cd "/home/${USER}/RACK/cli/"
python3 -m pip install ./wheels/*.whl
python3 -m pip install --no-dependencies ./wheels/*.whl
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not pip3 install?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only because I wasn't thinking about changing this line of code at all until it failed due to dependency checks. I was singularly focused on --no-dependencies when I visited this line. I'll make the change.

In case you wondered and didn't consult the failed build logs, the --no-dependencies became necessary because pip was able to tell that the version of python3-semtk that we'd installed was not the GitHub version, as specified in the pyproject.toml but it was the pre-compiled wheel file which self-descripted as version 0.1. This flag disables all the checks. I believe that this is OK because all the checks were done at the point at which we generated the wheels.

./setup-rack.sh