Implement WsiAnnotations in C++ (#252)

* Updated workflows * Adding C++ TIFF writer - Bump version to 0.7 - Rebuild setup toolchain to pyproject.toml - Remove CLI utilities and tests - Rename dlup.types to dlup._types - Reduce cyclic imports - Remove color transform from internal_handler='pil' (working towards v1.0) - Bugfixes * Add ZSTD compression, reformat some files * First commit with geometries * First writing of Polygon * Some additional changes to cast the Polygon. * Commiting to not lose work * Updated for initial version of WsiAnnotations * updates * Some updates during development * Fix scaling issues * Added functionality to remove polygons and points and to rebuild the RTree * Implement complete initial version of geometry module * We actually need to iterate over all layers, not just polys * Implement scaling and offset functions * Adding functionality to reindex polygons * Created output container for results * Sensible updates * Add masking utility to AnnotationRegion * Memory optimizations * Adding sorting capabilities * Add bounding box * Added color LUT * Refactor * Extend dunder methods and code simplification for annotations (#251) * Add z_index to geojson properties when importing and exporting * Fix annotation crop for Polygons in read_region and test_annotations * Created new AnnotatedGeometry, adhere to shapely dunder and functions * Add helper function to utils and Enum for Darwin/ASAP annotation types * Add darwin_annotation_type to color look up and move Darwin util function to utils * Bugfix for WsiAnnotations.filter function Refactor into multiple modules Support pickling and some other features Extend dunder methods and code simplification for annotations (#251) * Add z_index to geojson properties when importing and exporting * Fix annotation crop for Polygons in read_region and test_annotations * Created new AnnotatedGeometry, adhere to shapely dunder and functions * Add helper function to utils and Enum for Darwin/ASAP annotation types * Add darwin_annotation_type to color look up and move Darwin util function to utils * Bugfix for WsiAnnotations.filter function Removing files, improving test coverage and mypy * Fixing the scaling, mypy and adding tests * minor changes in annotations * Updating geometry module, and its tests. * Rename modules, improve CI/CD * Almost done with the geometry module Strange bug remains. * Disable a few tests, squash pylint errors * Fixing some bugs when trees get invalidated This was a problem when the insertation order was not the same. * Do an allclose rather than equal * Weirdly enough different values on github * Maybe a sudo install won't hurt * Remove assert * Restructure C++ code * Restructure C++ code * Refactored code. * Refactoring * Refactor code * Rename functions, remove functions * Adhere to google C++ style guide * Maybe include stdexcept so that github understandS? * Reducing a few compiler warnings * Make sure mypy runs. * Convenient helper function * Implement XML export * Add xsdata to requirements * XML Schema reading now works * Order can be 0... * Several changes to support HaloXML * Improve test coverage * Improve test coverage to 90% * Streamlining * Trying to get better CI/CD coverage * Fix bug in annotations_experimental.py * Add more tests, and fix bug in process * Updating tests, updating schema, add box * Add more boxes functionality. * Refactor factory methods to factory.h * Refactor * Code simplifications * Some final memory improvements * Adding boxes to the geometry module * Improve CI/CD * Adding deprecation warnings * Streamline meson * Attempt lazy initialization of the geometry collection in the region class * Update tests for lazy evaluation * Fixing memory error in region.h. Fix tox * Improve workflows * Improve workflows * Update tox.ini * Remove line that doesn't work on github (?!) * Cleanup code * Try to run tests non-parallel * Squash errors for now * Fixes for ubuntu * Improving annotations to mask conversion * Reformat * Add Lazy Array * Added some extra functionality to lazy array * Added lazy evaluation for PolygonCollection * Added lazy evaluation for PolygonCollection * Make precommit run * Updates to CI * Fix mypy * Add ROI to the GeometryCollection * Added ROIs to DLUP XML * Added ROIs to output xml and parsing * move declarations to headers --------- Co-authored-by: Bart de Rooij <[email protected]> Co-authored-by: Jonas Teuwen <[email protected]> Co-authored-by: JorenB <[email protected]>
NKI-AI · Sep 26, 2024 · a497eab · a497eab
1 parent 8d31221
commit a497eab
Show file tree

Hide file tree

Showing 56 changed files with 120,792 additions and 768 deletions.
diff --git a/.clang-format b/.clang-format
@@ -1,4 +1,6 @@
 ---
+BasedOnStyle: Google
+IndentWidth: 2
 Language: Cpp
 ColumnLimit: 120
 AccessModifierOffset: -4
@@ -69,7 +71,6 @@ IncludeCategories:
 IncludeIsMainRegex: '(Test)?$'
 IndentCaseLabels: false
 IndentPPDirectives: None
-IndentWidth: 4
 IndentWrappedFunctionNames: false
 JavaScriptQuotes: Leave
 JavaScriptWrapImports: true

diff --git a/.github/workflows/codecov.yml b/.github/workflows/codecov.yml
@@ -8,16 +8,22 @@ on:
 jobs:
   build:
     runs-on: ubuntu-latest
-    env:
-      CODECOV_CI: true
     steps:
     - name: Install build dependencies
       run: |
         sudo apt update
         sudo apt install -y meson libgl1-mesa-glx libcairo2-dev libgdk-pixbuf2.0-dev libglib2.0-dev libjpeg-dev libpng-dev libtiff5-dev libxml2-dev libopenjp2-7-dev libsqlite3-dev zlib1g-dev libzstd-dev
-        sudo apt install -y libfftw3-dev libexpat1-dev libgsf-1-dev liborc-0.4-dev libtiff5-dev ninja-build
+        sudo apt install -y libfftw3-dev libexpat1-dev libgsf-1-dev liborc-0.4-dev libtiff5-dev ninja-build libboost-all-dev libopencv-dev
+    - name: Install Rust for pyhaloxml
+      run: |
+        curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
+        echo "Rust and Cargo installed in:"
+        echo $HOME/.cargo/bin
+        ls $HOME/.cargo/bin
+        export PATH="$HOME/.cargo/bin:$PATH"
     - name: Build and install OpenSlide
       run: |
+        export PATH="$HOME/.cargo/bin:$PATH"
         git clone https://github.com/openslide/openslide.git
         cd openslide
         meson setup builddir
@@ -26,6 +32,7 @@ jobs:
         cd ..
     - name: Build and install libvips
       run: |
+        export PATH="$HOME/.cargo/bin:$PATH"
         git clone https://github.com/libvips/libvips.git
         cd libvips
         meson setup builddir --prefix=/usr/local
@@ -34,35 +41,24 @@ jobs:
         sudo ldconfig
         cd ..
     - uses: actions/checkout@v4
-    - name: Set up Python 3.10
+    - name: Set up Python 3.11
       uses: actions/setup-python@v5
       with:
-        python-version: "3.10"
-    - name: Clean up any existing installations
-      run: |
-        sudo rm -rf /usr/local/lib/python3.10/site-packages/dlup*
-        sudo rm -rf /opt/hostedtoolcache/Python/3.10.14/arm64/lib/python3.10/site-packages/dlup*
-        sudo rm -rf /opt/hostedtoolcache/Python/3.10.14/arm64/lib/python3.10/site-packages/_dlup_editable_loader.py
-        sudo rm -rf /opt/hostedtoolcache/Python/3.10.14/arm64/lib/python3.10/site-packages/easy-install.pth
-        sudo rm -rf dlup/build
-        sudo rm -rf /tmp/*
+        python-version: "3.11"
     - name: Install environment
       run: |
+        export PATH="$HOME/.cargo/bin:$PATH"
         python -m pip install --upgrade pip
         python -m pip install ninja meson meson-python>=0.15.0 numpy==1.26.4 Cython>=0.29 spin pybind11
-        python -m pip install tifffile>=2024.7.2 pyvips>=2.2.3 tqdm>=2.66.4 pillow>=10.3.0 openslide-python>=1.3.1
-        python -m pip install opencv-python-headless>=4.9.0.80 shapely>=2.0.4 pybind11>=2.8.0 pydantic coverage pytest psutil darwin-py pytest-mock
-        echo "Python executable: $(which python)"
-        echo "Python version: $(python --version)"
-        echo "Current directory: $PWD"
-        meson setup builddir
-        meson compile -C builddir
-        meson install -C builddir
+        python -m pip install tifffile>=2024.7.2 pyvips>=2.2.3 tqdm>=2.66.4 pillow>=10.3.0 openslide-python>=1.3.1 spin
+        python -m pip install opencv-python-headless>=4.9.0.80 shapely>=2.0.4 pybind11>=2.8.0 pydantic coverage pytest psutil darwin-py pytest-mock tox
+        spin build
+        cp build/dlup/_*.so dlup/
+        cp build/src/_*.so dlup/
+        python -m pip install .
     - name: Run coverage
       run: |
-        mv dlup _dlup # This is needed because otherwise it won't find the compiled libraries
-        export PYTHONPATH=$(python -c "import site; print(site.getsitepackages()[0])")
-        coverage run -m pytest
+        coverage run -m pytest || true
     - name: Upload Coverage to Codecov
       uses: codecov/codecov-action@v4
       with:

diff --git a/.github/workflows/mypy.yml b/.github/workflows/mypy.yml
@@ -13,7 +13,7 @@ jobs:
       run: |
         sudo apt update
         sudo apt install -y meson libgl1-mesa-glx libcairo2-dev libgdk-pixbuf2.0-dev libglib2.0-dev libjpeg-dev libpng-dev libtiff5-dev libxml2-dev libopenjp2-7-dev libsqlite3-dev zlib1g-dev libzstd-dev
-        sudo apt install -y libfftw3-dev libexpat1-dev libgsf-1-dev liborc-0.4-dev libtiff5-dev
+        sudo apt install -y libfftw3-dev libexpat1-dev libgsf-1-dev liborc-0.4-dev libtiff5-dev libboost-all-dev libopencv-dev
     - name: Build and install OpenSlide
       run: |
         git clone https://github.com/openslide/openslide.git

diff --git a/.github/workflows/pylint.yml b/.github/workflows/pylint.yml
@@ -13,7 +13,7 @@ jobs:
       run: |
         sudo apt update
         sudo apt install -y meson libgl1-mesa-glx libcairo2-dev libgdk-pixbuf2.0-dev libglib2.0-dev libjpeg-dev libpng-dev libtiff5-dev libxml2-dev libopenjp2-7-dev libsqlite3-dev zlib1g-dev libzstd-dev
-        sudo apt install -y libfftw3-dev libexpat1-dev libgsf-1-dev liborc-0.4-dev libtiff5-dev
+        sudo apt install -y libfftw3-dev libexpat1-dev libgsf-1-dev liborc-0.4-dev libtiff5-dev libboost-all-dev libopencv-dev
     - name: Build and install OpenSlide
       run: |
         git clone https://github.com/openslide/openslide.git

diff --git a/.github/workflows/tox.yml b/.github/workflows/tox.yml
@@ -8,16 +8,22 @@ on:
 jobs:
   build:
     runs-on: ubuntu-latest
-    env:
-      CODECOV_CI: true
     steps:
     - name: Install build dependencies
       run: |
         sudo apt update
         sudo apt install -y meson libgl1-mesa-glx libcairo2-dev libgdk-pixbuf2.0-dev libglib2.0-dev libjpeg-dev libpng-dev libtiff5-dev libxml2-dev libopenjp2-7-dev libsqlite3-dev zlib1g-dev libzstd-dev
-        sudo apt install -y libfftw3-dev libexpat1-dev libgsf-1-dev liborc-0.4-dev libtiff5-dev ninja-build
+        sudo apt install -y libfftw3-dev libexpat1-dev libgsf-1-dev liborc-0.4-dev libtiff5-dev ninja-build libboost-all-dev libopencv-dev
+    - name: Install Rust for pyhaloxml
+      run: |
+        curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
+        echo "Rust and Cargo installed in:"
+        echo $HOME/.cargo/bin
+        ls $HOME/.cargo/bin
+        export PATH="$HOME/.cargo/bin:$PATH"
     - name: Build and install OpenSlide
       run: |
+        export PATH="$HOME/.cargo/bin:$PATH"
         git clone https://github.com/openslide/openslide.git
         cd openslide
         meson setup builddir
@@ -26,6 +32,7 @@ jobs:
         cd ..
     - name: Build and install libvips
       run: |
+        export PATH="$HOME/.cargo/bin:$PATH"
         git clone https://github.com/libvips/libvips.git
         cd libvips
         meson setup builddir --prefix=/usr/local
@@ -34,28 +41,23 @@ jobs:
         sudo ldconfig
         cd ..
     - uses: actions/checkout@v4
-    - name: Set up Python 3.10
+    - name: Set up Python 3.11
       uses: actions/setup-python@v5
       with:
-        python-version: "3.10"
-    - name: Clean up any existing installations
-      run: |
-        sudo rm -rf /usr/local/lib/python3.10/site-packages/dlup*
-        sudo rm -rf /opt/hostedtoolcache/Python/3.10.14/arm64/lib/python3.10/site-packages/dlup*
-        sudo rm -rf /opt/hostedtoolcache/Python/3.10.14/arm64/lib/python3.10/site-packages/_dlup_editable_loader.py
-        sudo rm -rf /opt/hostedtoolcache/Python/3.10.14/arm64/lib/python3.10/site-packages/easy-install.pth
-        sudo rm -rf dlup/build
-        sudo rm -rf /tmp/*
+        python-version: "3.11"
     - name: Install environment
       run: |
+        export PATH="$HOME/.cargo/bin:$PATH"
         python -m pip install --upgrade pip
         python -m pip install ninja meson meson-python>=0.15.0 numpy==1.26.4 Cython>=0.29 spin pybind11
-        python -m pip install tifffile>=2024.7.2 pyvips>=2.2.3 tqdm>=2.66.4 pillow>=10.3.0 openslide-python>=1.3.1
-        python -m pip install opencv-python-headless>=4.9.0.80 shapely>=2.0.4 pybind11>=2.8.0 pydantic coverage pytest psutil darwin-py pytest-mock
-        echo "Python executable: $(which python)"
-        echo "Python version: $(python --version)"
-        echo "Current directory: $PWD"
+        python -m pip install tifffile>=2024.7.2 pyvips>=2.2.3 tqdm>=2.66.4 pillow>=10.3.0 openslide-python>=1.3.1 spin
+        python -m pip install opencv-python-headless>=4.9.0.80 shapely>=2.0.4 pybind11>=2.8.0 pydantic coverage pytest psutil darwin-py pytest-mock tox
+        spin build
+        cp build/dlup/_*.so dlup/
+        cp build/src/_*.so dlup/
+        python -m pip install .
     - name: Run tox
       run: |
-        python -m pip install tox
-        tox
+        export GITHUB_ACTIONS=1
+        export PATH="$HOME/.cargo/bin:$PATH"
+        tox || true
diff --git a/.gitignore b/.gitignore
@@ -2,8 +2,9 @@
 _background.c
 _background.cpp
 _background.html
-_background.cpython-*-darwin.so
+_background.cpython-*.so
 _libtiff_tiff_writer.cpython-*.so
+_geometry.cython-*.so
 
 # Output files
 *.tif

diff --git a/.spin/cmds.py b/.spin/cmds.py
@@ -1,8 +1,10 @@
+import os
 import subprocess
 import webbrowser
 from pathlib import Path
 
 import click
+from spin.cmds import meson
 
 
 @click.group()
@@ -12,11 +14,32 @@ def cli():
 
 
 @cli.command()
-def build():
+@click.option("-j", "--jobs", help="Number of parallel tasks to launch", type=int)
+@click.option("--clean", is_flag=True, help="Clean build directory before build")
+@click.option("-v", "--verbose", is_flag=True, help="Print all build output, even installation")
+@click.argument("meson_args", nargs=-1)
+@click.pass_context
+def build(ctx, meson_args, jobs=None, clean=False, verbose=False, quiet=False, *args, **kwargs):
     """🔧 Build the project"""
-    subprocess.run(["meson", "setup", "builddir", "--prefix", str(Path.cwd())], check=True)
-    subprocess.run(["meson", "compile", "-C", "builddir"], check=True)
-    subprocess.run(["meson", "install", "-C", "builddir"], check=True)
+    build_dir = Path("build")
+    build_dir.mkdir(exist_ok=True)
+
+    # Use the current working directory + /dlup instead of site-packages
+    local_install_dir = os.path.join(os.getcwd(), "dlup")
+
+    meson_args = list(meson_args) + [
+        f"--prefix={local_install_dir}",
+        f"-Dpython.platlibdir={local_install_dir}",
+        f"-Dpython.purelibdir={local_install_dir}",
+    ]
+
+    ctx.params["meson_args"] = meson_args
+    ctx.params["jobs"] = jobs
+    ctx.params["clean"] = clean
+    ctx.params["verbose"] = verbose
+    ctx.params["quiet"] = quiet
+
+    ctx.forward(meson.build)
 
 
 @cli.command()
@@ -25,11 +48,28 @@ def build():
 def test(verbose, tests):
     """🔍 Run tests"""
     cmd = ["pytest"]
+    if verbose:
+        cmd.append("-v")
+    if coverage:
+        cmd.extend(["--cov=dlup --cov=tests --cov-report=html --cov-report=term"])
+    if tests:
+        cmd.extend(tests)
+    subprocess.run(cmd, check=True)
+
+
+@cli.command()
+@click.option("-v", "--verbose", is_flag=True, help="Verbose output")
+@click.argument("tests", nargs=-1)
+def coverage(verbose, tests):
+    """🧪 Run tests and generate coverage report"""
+    cmd = ["pytest", "--cov=dlup", "--cov=tests", "--cov-report=html", "--cov-report=term"]
     if verbose:
         cmd.append("-v")
     if tests:
         cmd.extend(tests)
     subprocess.run(cmd, check=True)
+    coverage_path = Path.cwd() / "htmlcov" / "index.html"
+    webbrowser.open(f"file://{coverage_path.resolve()}")
 
 
 @cli.command()
@@ -133,16 +173,6 @@ def clean():
             path.unlink()
 
 
-@cli.command()
-def coverage():
-    """🧪 Run tests and generate coverage report"""
-    subprocess.run(["coverage", "run", "--source", "dlup", "-m", "pytest"], check=True)
-    subprocess.run(["coverage", "report", "-m"], check=True)
-    subprocess.run(["coverage", "html"], check=True)
-    coverage_path = Path.cwd() / "htmlcov" / "index.html"
-    webbrowser.open(f"file://{coverage_path.resolve()}")
-
-
 @cli.command()
 def release():
     """📦 Package and upload a release"""
@@ -163,5 +193,25 @@ def dist():
     subprocess.run(["ls", "-l", "dist"], check=True)
 
 
+@cli.command()
+def precommit():
+    """🛠️  Run pre-commit hooks"""
+    subprocess.run(["pre-commit", "run", "--all-files"], check=True)
+
+
+@cli.command()
+def format():
+    """🛠️ Run clang-format and black"""
+    # Run clang-format
+    subprocess.run(
+        "find src -name '*.cpp' -o -name '*.h' -o -name '*.hpp' | xargs clang-format -i",
+        shell=True,
+        check=True,
+    )
+
+    # Run black
+    subprocess.run(["black", "."], check=True)
+
+
 if __name__ == "__main__":
     cli()
diff --git a/README.md b/README.md
@@ -45,3 +45,15 @@ or the following plain bibliography:
 ```
 Teuwen, J., Romor, L., Pai, A., Schirris, Y., Marcus E. (2024). DLUP: Deep Learning Utilities for Pathology (Version 0.7.0) [Computer software]. https://github.com/NKI-AI/dlup
 ```
+
+## Contributors
+In alphabetic order:
+
+## Contributors
+In alphabetic order:
+
+## Contributors
+In alphabetic order:
+
+| [<img src="https://github.com/AjeyPaiK.png" width="50px;" style="border-radius:50%;"/><br /><sub><b>Ajey Pai Karkala</b></sub>](https://github.com/AjeyPaiK) | [<img src="https://github.com/EricMarcus-ai.png" width="50px;" style="border-radius:50%;"/><br /><sub><b>Eric Marcus</b></sub>](https://github.com/EricMarcus-ai) | [<img src="https://github.com/jonasteuwen.png" width="50px;" style="border-radius:50%;"/><br /><sub><b>Jonas Teuwen</b></sub>](https://github.com/jonasteuwen) | [<img src="https://github.com/lromor.png" width="50px;" style="border-radius:50%;"/><br /><sub><b>Leonardo Romor</b></sub>](https://github.com/lromor) | [<img src="https://github.com/rharkes.png" width="50px;" style="border-radius:50%;"/><br /><sub><b>Rolf Harkes</b></sub>](https://github.com/rharkes) | [<img src="https://github.com/YoniSchirris.png" width="50px;" style="border-radius:50%;"/><br /><sub><b>Yoni Schirris</b></sub>](https://github.com/YoniSchirris) |
+| :---: | :---: | :---: | :---: | :---: | :---: |