Start by forking main repository (/FAIRmat-NDFI/dev_distro) that will house all your plugins.
Benefits
- One-step installations: Install everything at once with editable mode.
- Centralized codebase: Easier navigation and searching across projects.
- Better editor support: Improved autocompletion and refactoring.
- Consistent tooling: Shared linting, testing, and formatting.
- Flexible plugin management: If you're developing plugins for different deployments with varying requirements, you can easily create different branches for each deployment and configure which plugins to use in the specific branch for that deployment.
Below are instructions for how to create a dev environment for developing nomad-lab and its plugins.
-
Make sure you have docker installed. Docker nowadays comes with
docker compose
built in. Prior, you needed to install the stand-alone docker-compose. -
Make sure you have uv installed. We will use it to setup, maintain, and use the development environment. The standalone installer or a global installation is the recommended way. (
brew install uv
on macOS ordnf install uv
on Fedora). -
For Windows users, we recommend using the Devcontainer plugin in VSCode to run the repository inside a container, or alternatively, using GitHub Codespaces to run the project.
-
Clone the forked repository.
git clone https://github.com/<your-username>/dev_distro.git cd dev_distro
-
Run the docker containers with docker compose in detached (--detach or -d) mode.
docker compose up -d
To shutdown the containers:
docker compose down
This guide explains how to set up a streamlined development environment for nomad-lab and its plugins using
uv
workspaces.
This approach eliminates the need for multiple pip install commands by leveraging a monorepo and a single installation step.
In this example, we'll set up the development environment for a developer working
on the following plugins: nomad-parser-plugins-electronic
and
nomad-measurements
. The first plugin already comes as a dependency
in this dev distribution. On the contrary, the second plugin is not listed as a
dependency. In the following, we take a look at how to setup the environment in
these two situations.
-
Update submodules
This loads the
nomad-lab
package which is already listed as a submodule.git submodule update --init --recursive
-
Add local plugins
Assuming that you already have a git repo for your plugins, add them to the
packages/
directory as submodules. In case, you are looking to create a plugin repo from scratch, consider using our plugin template: nomad-plugin-template.git submodule add https://github.com/<package_name>.git packages/<package_name>
Repeat for all the plugin packages you want to add and develop. For our example:
git submodule add https://github.com/nomad-coe/electronic-parsers.git packages/nomad-parser-plugins-electronic git submodule add https://github.com/FAIRmat-NFDI/nomad-measurements.git packages/nomad-measurements
-
Modify
pyproject.toml
To ensure
uv
recognizes the local plugins (a local copy of your plugin repository available inpackages/
directory), we need to make some modifications in thepyproject.toml
. These include adding the plugin package to[project.dependencies]
and[tool.uv.sources]
tables. For the packages listed under[tool.uv.sources]
,uv
uses the local code directory made available underpackages/
with the previous step.Some of the plugins are already listed under
[project.dependencies]
. If you want to develop one of them, you have to add them under[tool.uv.sources]
. We do this fornomad-parser-plugins-electronics
.[tool.uv.sources] ... nomad-parser-plugins-electronic = { workspace = true }
If you're developing a plugin not listed under
[project.dependencies]
, you must first add it as a dependency. After adding the dependencies, update the[tool.uv.sources]
section in yourpyproject.toml
file to reflect the new plugins. You can either add it manually abd runuv sync
:[project] dependencies = [ ... "nomad-measurements", ] [tool.uv.sources] ... nomad-measurements = { workspace = true }
Or, you can use
uv add
which adds the dependency and the source inpyproject.toml
and sets up the environment:uv add nomad-measurements
Note
You can also use uv
to install a specific branch of the plugin submodule.
uv add https://github.com/FAIRmat-NFDI/nomad-measurements --branch <specific-branch-name>
-
Install dependencies
Run the following command to install all dependencies, including the local packages in editable mode:
uv sync
Note
uv sync
and uv run
automatically manages the virtual environment for you.
There's no need to manually create or activate a venv.
Any uv run
commands will automatically use the correct environment by default.
Read more about uv
commands to manage the dependencies here.
-
Setup NOMAD GUI
This step only needs to be done once.
cd packages/nomad-FAIR/gui uv run python -m nomad.cli dev gui-env > .env.development
After the initial setup, here’s how to manage your daily development tasks.
-
Update submodules
git submodule update --init --recursive
-
Installing dependencies
If you've added new dependencies or made changes to your environment, install or update them by running:
uv sync
This will sync all packages and ensure everything is installed in editable mode.
-
Running
nomad
api app.uv run nomad admin run appworker
-
Start NOMAD GUI
cd packages/nomad-FAIR/gui yarn yarn start
-
Run the docs server (optional: only if you wish to run the documentation server):
uv run --directory packages/nomad-FAIR mkdocs serve
-
Running tests
To run tests across the project, use the uv run command to execute pytest in the relevant directory. For instance:
uv run --directory packages/package_name pytest
This allows you to run tests for a specific parser or package. For running tests across all packages, simply repeat the command for each directory.
-
Making code changes
Since all packages are installed in editable mode, changes you make to the code are immediately reflected. Edit your code and rerun tests or the application as needed, without needing to reinstall the packages.
-
Linting & code formatting
To check for code style issues using ruff, run the following command:
uv run ruff check .
This will lint all files.
For auto-formatting:
uv run ruff format .
-
Adding new plugins
To add a new package, follow setup guide and add it into the
packages/
directory and ensure it's listed inpyproject.toml
under[tool.uv.sources]
. Then, install it by running:uv sync
-
Removing an existing plugin
To remove an existing plugin from the workspace in
packages/
directory, do the following and commit:git rm <path-to-submodule>
Then you can remove the plugin from
[tool.uv.sources]
inpyproject.toml
to stop uv from using the local plugin repository.Additionally, if you want to remove the plugin from being a dependency of your NOMAD installation, you can use
uv
to entirely remove it:uv remove <plugin-name>
-
Modifying dependencies in packages.
uv add --package <PACKAGE_NAME> <DEPENDENCY_NAME>
For example:
uv add --package nomad-measurements "pandas>=2.0" uv remove --package nomad-lab numpy
-
Keeping Up-to-Date
To pull updates from the main repository and submodules, run:
git pull --recurse-submodules
Afterward, sync your environment:
uv sync
To keep your fork up to date with the latest changes from the original repository (upstream), follow these steps:
-
Add the upstream Remote
If you haven't already, add the original repository as upstream:
git remote add upstream https://github.com/FAIRmat-NFDI/dev_distro.git
-
Fetch the Latest Changes from upstream
Fetch the latest commits from the upstream repository:
git fetch upstream
-
Merge upstream/main into Your Local Branch
Switch to the local branch (e.g., main) you want to update, and merge the changes from upstream/main:
git checkout main git merge upstream/main
Resolve any merge conflicts if necessary, and commit the merge.
-
Push the Updates to Your Fork
After merging, push the updated branch to your fork on GitHub:
git push origin main