Updated radCAD to be compatible with latest cadCAD and Python 3.8-3.12 (

#68) * Updated cadCAD and radCAD to v0.14.0, and passed all tests with latest cadCAD v0.5.3 * Updated cadCAD and radCAD to v0.14.0, and passed all tests with latest cadCAD v0.5.3 * Updated cadCAD and radCAD to v0.14.0, and passed all tests with latest cadCAD v0.5.3 * Update pyproject.toml * Update documentation from Poetry to PDM * Migrate requirements from Poetry to PDM * Add Nox (Tox alternative) tests * Update tests to be compatible with Python 3.11+ * Update GitHub action * Update GitHub action * Update GitHub action * Update GitHub action * Update GitHub action * Update GitHub action * Update GitHub action * Update Nox file to fix recursion depth limit issue on Windows * Update Nox file to fix recursion depth limit issue on Windows * Update pytest version * Update pytest version * Remove pytest-monitor due to CI error * Increase recursion limit * Add RADCAD_BACKEND env variable to override during Windows tests * Add RADCAD_BACKEND env variable to override during Windows tests * Add RADCAD_BACKEND env variable to override during Windows tests * Resolve issue with tables and remove Windows from CI pipeline * Update CHANGELOG, README, CI to include Windows for one last attempt * Remove tables override for Python3.8 * Override deepcopy method when using Windows to avoid recurrsion errors * Override radCAD backend when using Windows to avoid recurrsion errors * Override radCAD backend when using Windows to avoid recurrsion errors * Remove deepcopy method switch for Windows * Pop Engine argument even if overridden by env variable --------- Co-authored-by: Benjamin Scholtz <[email protected]>
CADLabs · Sep 5, 2024 · 3e2d1b6 · 3e2d1b6
1 parent 1a0519e
commit 3e2d1b6
Show file tree

Hide file tree

Showing 18 changed files with 6,593 additions and 4,632 deletions.
diff --git a/.github/workflows/python.yml b/.github/workflows/python.yml
@@ -6,28 +6,47 @@ name: Build Status
 on:
   push:
     branches: [ master ]
+    paths:
+    - '**.py'
   pull_request:
     branches: [ master ]
+    paths:
+    - '**.py'
 
 jobs:
   build:
-
-    runs-on: ubuntu-latest
+    runs-on: ${{ matrix.os }}-latest
     strategy:
+      fail-fast: false
       matrix:
-        python-version: [3.8, 3.9, '3.10']
+        python-major-version: [3]
+        python-minor-version: [8, 9, 10, 11, 12]
+        os: [ubuntu, macOS, windows]
+    env:
+      python-version: ${{ format('{0}.{1}', matrix.python-major-version, matrix.python-minor-version) }}
 
     steps:
-    - uses: actions/checkout@v2
-    - name: Set up Python ${{ matrix.python-version }}
+    - uses: actions/checkout@v3
+    - name: Set up Python ${{ env.python-version }}
       uses: actions/setup-python@v2
       with:
-        python-version: ${{ matrix.python-version }}
-    - name: Install dependencies
+        python-version: ${{ env.python-version }}
+    - name: Install build dependencies
       run: |
         python -m pip install --upgrade pip setuptools wheel
-        python -m pip install poetry
-        poetry install -E compat
-    - name: Test with pytest
+        python -m pip install pdm nox
+    - name: Install build dependencies for MacOS
+      if: ${{ matrix.os == 'macOS' }}
+      run: |
+        brew install hdf5 c-blosc
+    - name: Install radCAD dependencies for Python 3.8
+      if: ${{ matrix.python-minor-version == 8 }}
+      run: |
+        pdm sync -d -G compat --lockfile pdm-py38.lock
+    - name: Install radCAD dependencies for Python 3.9+
+      if: ${{ matrix.python-minor-version >= 9 }}
+      run: |
+        pdm sync -d -G compat --lockfile pdm.lock
+    - name: Run tests using Nox
       run: |
-        PYTHONPATH=. poetry run python3 -m pytest tests
+        python -m nox -s tests-${{ env.python-version }}
diff --git a/.gitignore b/.gitignore
@@ -161,3 +161,5 @@ scratchpad**
 .pymon
 .benchmarks/
 .idea/
+*.prof
+.pdm-python
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,182 +1,273 @@
 # Change Log
+
 All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.14.0] - 2024-09-04
+
+### Added
+
+- Support for Python versions 3.11, 3.12
+- GitHub Action CI pipeline tests for Windows, MacOS
+
+### Changed
+
+- Migrated from using Poetry to PDM to support overriding/ignoring subdependencies and switching based on Python version (e.g. cadCAD, which has removed support for Python 3.8, but which we'd like to continue supporting)
+- Implemented tests and benchmarking across Python versions using Nox (equivalent of Tox that uses Python instead of yaml for configuration)
+- Updated cadCAD dependency to v0.5.3 to ensure compatibility with Python 3.10+ when using "compatibility mode"
+- Added RADCAD_BACKEND environment variable to override radCAD's backend, specifically used for Windows tests to avoid recursion depth errors
+
 ## [0.13.0] - 2023-10-26
+
 ### Added
+
 - Ability to override the method used for performing deep copies, e.g. using Pickle instead of standard library for better performance
 - Ability to use dataclasses instead of dictionaries to configure System Parameters, enabling much better developer experience including typing of System Parameters
 - Ability to use nested dataclasses to configure System Parameters to enable creating categories or namespaces of System Parameters for better management of large models, see tests for examples of how this works
 - Better typing of all core radCAD code, see `radcad.types`
 
 ### Changed
+
 - End of support for Python 3.7, due to dependecies we now only support >= 3.8
 - Major refactor of core code to use specification for simulation execution in the form of an abstract base class named `SimulationExecutionSpecification`, enabling much better understanding, customisation, and extension of radCAD core code
 
 ## [0.12.0] - 2022-09-05
+
 ### Added
+
 - Add typing
 
 ### Changed
+
 - Enable mutation of simulation hook `Context`
 - Possibly breaking change for anyone using hooks: `before_simulation(simulation: Simulation)` changed to `before_simulation(context: Context)`
 
 ## [0.11.1] - 2022-09-03
+
 ### Added
+
 - Add support for parameters dataclass
 - Add support for single value parameters (i.e. not a list)
 
 ## [0.11.0] - 2022-09-03
+
 ### Changed
+
 - Adapted core deepcopy processes to be more efficient and avoid unintended mutation of state between policy and state update functions
 
 ## [0.10.1] - 2022-09-02
+
 ### Changed
+
 - Updated to use "radCAD" logging instance
 
 ## [0.10.0] - 2022-09-01
+
 ### Added
+
 - Add a `deepcopy_method` Engine argument to allow setting a custom deepcopy method e.g. `copy.deepcopy` instead of default Pickle methods
 
 ### Changed
+
 - Fix edge case of unintended mutation of state passed to state update function from policy function (see issue #53)
 
 ## [0.9.1] - 2022-07-26
+
 ### Added
+
 - Thanks to @vmeylan for a developer experience contribution in `radcad/core.py`: make `_update_state()` error messages more verbose for easier debugging
 
 ## [0.9.0] - 2022-06-14
+
 ### Changed
+
 - `Context` passed to `before_subset(context)` hook will now correctly receive a single parameter subset and not all subsets
 - This could be a breaking change for anyone that relied on the `before_subset()` hook - the `before_run()` hook will still receive all subsets, as at that point of simulation the specific subset is unknown
 - See "NOTE" in `engine.py`: Each parameter is a list of all subsets in before_run() method and a single subset in before_subset()
 
 ## [0.8.4] - 2021-09-04
+
 ### Added
+
 - Overide for `__deepcopy__` method of Executable class to enable deepcopy after a simulation/experiment has been run
 - Add regression test for above `__deepcopy__` method
 
 ## [0.8.3] - 2021-08-30
+
 ### Changed
-- Update radCAD package version in __init__.py
+
+- Update radCAD package version in **init**.py
 
 ## [0.8.2] - 2021-08-27
+
 ### Changed
+
 - Update cadCAD from v0.4.23 to v0.4.27, fixing breaking changes to compat module
 
 ## [0.8.1] - 2021-08-24
+
 ### Changed
+
 - Minor update: Module package version & package details
 
 ## [0.8.0] - 2021-06-28
+
 ### Changed
+
 - `engine.deepcopy` setting now disables deepcopy of State as well as Policy Signals
 
 ## [0.7.1] - 2021-06-11
+
 ### Added
-- Fix for Pathos multiprocessing issue - "NameError: Name '_' is not defined". See https://github.com/uqfoundation/multiprocess/issues/6
+
+- Fix for Pathos multiprocessing issue - "NameError: Name '\_' is not defined". See https://github.com/uqfoundation/multiprocess/issues/6
 
 ## [0.7.0] - 2021-06-09
+
 ### Changed
+
 - Experiment and Simulation extend the Executable class (not a breaking change, but significant improvement to API)
 
 ## [0.6.7] - 2021-06-08
+
 ### Added
+
 - `generate_cartesian_product_parameter_sweep(...)` method to `radcad.utils`
 
 ## [0.6.6] - 2021-06-08
+
 ### Added
+
 - Base wrapper class
 
 ## [0.6.5] - 2021-06-03
+
 ### Added
+
 - Iterable Models
 
 ## [0.6.4] - 2021-05-31
+
 ### Added
+
 - Support for Python 3.9 (<4.0)
 
 ## [0.6.3] - 2021-04-30
+
 ### Added
+
 - Update dependencies
 
 ## [0.6.2] - 2021-04-30
+
 ### Added
+
 - Removed Streamlit which requires Python <3.9
 
 ## [0.6.1] - 2021-04-30
+
 ### Added
+
 - Support for Python 3.9
 
 ### Changed
+
 - Added tests for hook functionality, fixed minor error, removed WIP tag
 - Updated log statement for "Starting simulation ..."
 
 ## [0.6.0] - 2021-02-20
+
 ### Changed
+
 - Moved Ray and dependencies to optional extension `extension-backend-ray`
 - Refactored `Backend` module
 - Introduced idea of extensions
 
 ## [0.5.6] - 2021-02-10
+
 ### Added
+
 - `drop_substeps` (default False) option to Engine
 
 ## [0.5.5] - 2021-02-08
+
 ### Changed
+
 - Incorrect argument given to Pathos pool for number of processes
 - Update memory benchmarks to not test A/B testing
 - Change Python max version from <3.9 to <=3.9
 - Update hook API and add subset hook
 
 ## [0.5.4] - 2021-02-08
+
 ### Added
+
 - Error tracebacks to exceptions data
 
 ## [0.5.3] - 2021-02-07
+
 ### Changed
+
 - Close Pathos, Multiprocessing pools
 
 ## [0.5.2] - 2021-02-06
+
 ### Changed
+
 - Pandas version ^1.0.0
 
 ## [0.5.1] - 2021-02-06
+
 ### Added
+
 - Add hook example for saving experiment results to HDF5 file format
 
 ## [0.5.0] - 2021-02-05
+
 ### Added
+
 - Predator-prey benchmark test
 
 ### Changed
+
 - Refactor core from Rust to Python
 
 ## [0.4.1] - 2021-02-02
+
 ### Added
+
 - Add engine `deepcopy` option, to disable deepcopy of state
 
 ### Changed
+
 - Significant performance tuning of Rust core for 2x increase in speed and reduction in memory use
 
 ## [0.4.0] - 2021-02-01
+
 ### Changed
+
 - Fix of critical state update bug, added regression test `test_paralell_state_update()` to `test_regression.py`
 
 ## [0.3.1] - 2021-01-31
+
 ### Changed
+
 - Python version downgrade to `^3.7` for Streamlit
 
 ## [0.3.0] - 2021-01-30
+
 ### Added
+
 - Error handling (default `Engine(raise_exceptions=True)`)
 - Partial simulation results (on run failure / exception)
 - Memory profiling using mprof
 
 ### Changed
+
 - Default simulation engine (changed from MULTIPROCESSING to PATHOS)
 
 ### Removed
+
 -
diff --git a/Makefile b/Makefile
@@ -1,21 +1,41 @@
-test:
-	poetry run python3 -m pytest tests
-
-profile-memory-radcad:
-	poetry run python3 -m mprof run --include-children benchmarks/benchmark_memory_radcad.py && poetry run python3 -m mprof plot
-
-profile-memory-cadcad:
-	poetry run python3 -m mprof run --include-children benchmarks/benchmark_memory_cadcad.py && poetry run python3 -m mprof plot
-
-coverage:
-	poetry run python3 -m pytest tests --cov=radcad
-	poetry run coveralls
-
 start-jupyter-lab:
-	poetry run python3 -m jupyter lab 
+	pdm run jupyter lab
 
 plotly-jupyter-lab-support:
 	# JupyterLab renderer support
 	jupyter labextension install [email protected]
 	# OPTIONAL: Jupyter widgets extension
-	jupyter labextension install @jupyter-widgets/jupyterlab-manager [email protected]
+	jupyter labextension install @jupyter-widgets/jupyterlab-manager [email protected]
+
+lock-py38:
+	pdm lock \
+		-G compat -G extension-backend-ray \
+		--python="==3.8.*" \
+		--lockfile pdm-py38.lock
+
+lock-py39+:
+	pdm lock \
+		-G compat -G extension-backend-ray \
+		--python=">=3.9" \
+		--lockfile pdm.lock
+
+lock: lock-py38 lock-py39+
+
+install-py38:
+	pdm install --lockfile pdm-py38.lock
+
+install-py39+:
+	pdm install --lockfile pdm.lock
+
+test:
+	pdm run pytest tests
+
+coverage:
+	pdm run pytest tests --cov=radcad
+	pdm run coveralls
+
+profile-memory-radcad:
+	pdm run mprof run --include-children benchmarks/benchmark_memory_radcad.py && pdm run mprof plot
+
+profile-memory-cadcad:
+	pdm run mprof run --include-children benchmarks/benchmark_memory_cadcad.py && pdm run mprof plot