Pixi build directories for workflows (#225)

* add vendor pkgs

* remove ci

* pixi migration WIP

* bump ecoscope version in vendor

* rebuild ecoscope vendor

* test all in ci

* split tests across a few different jobs

* add doctest and rename workflow

* parametrize python version

* move strategy

* test python versions

* rename back to CI

* parallelize e2e test runs

* try to fix rtd

* lock docs env

* fix command

* fix docs env

* simplify pyproject python definitions

* alphabetize features

* mypy wip

* mypy WIP continued

* mypy WIP continued

* more typing

* try to run mypy with pixi in pre-commit

* bump ruff, remove mypy from pre-commit

* remove lint

* run mypy in ci

* moar mypy

* some import untyped ignores

* use ruamel.yaml in CLI

* mypy cont

* more mypy

* typing typing

* pa_typing

* give up with er io return values

* only 9 errors left

* cast according to pandera docs

* wow one error

* mypy 312 passes

* fix circular imports

* some tweaks for pydantic.mypy

* cast to save a few more type ignores

* vendor version check WIP

* vendor version check appears to work

* vendor version check

* vendor version check task

* vendor version check

* add manual build/push actions

* update lockfile

* README update for pixi install

* mypy fixes for #223

* recompile params since legend is optional

* remove etd cached parquet

* relock and add io marker

* rebuild + repush for lonboard 0.0.3

* relock

* oh maybe we dont need 0.0.3

* mypy check tests

* use ruamel.yaml in tests

* type ignore mypy failures in tests

* cast colors to strs before passing them to lonboard

* revert examples params changes

* mode map build dir first commit

* mode map pyproject wip

* simplify e2e test

* maybe pixi.toml makes more sense

* move pixi.toml into bld (bc it will be generated)

* add pixi.lock for mode-map

* move mode-map spec to top level of examples dir

* test cases WIP

* sequential test passes

* will this run

* fallback version for scm

* move test_case to testing

* some light renaming

* update gh workflow

* fix test

* a little build dir data modeling

* some more artifact modeling

* compiler rewrite first pass

* compile build dir command wip continued

* dags compile correctly

* add params-jsonschema

* pixi toml WIP

* PixiToml.from_file works

* toml roundtrip complete

* ecoscope_core feature WIP

* add tasks(requires=[..]) features

* fix features to get decorator tests to pass

* skip features reserved module name

* fix compiler test

* fix examples tests (not e2e)

* get e2e tests passing

* so do all tests pass now?

* get_pixi_toml WIP

* mypy fix task overload

* complete mypy fixes

* fix examples test

* try to fix sequential test

* move README into workflow dir (it will be generated)

* hatchling

* rm symlinked pyproject.toml

* relock mode-map with minimal pyproject.toml

* relock main envs, retry on mode-map env

* fix entrypoints table

* custom build hook / feature registry WIP

* add pixi.sh decision matrix

* update decision matrix

* namespaces take 2

* biggest commit i've ever made?

* move groupby to core

* remove duplicate aggretation from extension

* add test envs to core

* move classification to extension

* move filtering to extension

* core tasks tests pass'

* run core task tests in ci

* test all python versions

* extension tasks pass locally

* test extension tasks in ci

* fix workflows typo

* rename extension -> ext.ecoscope

* ext.ecoscope continued

* fix ext.ecoscope tasks

* enable io tests

* add connections tests to ext.ecoscope

* add test annotations to core tests

* add compiler tests to ext.ecoscope test

* add test config to ext.ecoscope

* add decorator tests to core

* add executors tests to core

* add graph tests to core

* add test registry to ext.ecoscope

* fix broken import

* fix more broken imports

* add test serde to core

* add testing utils test to ext.ecoscope

* add test type adapter to core

* remove all original tests except examples

* oops forgot to commit thos

* fix tests?

* the last problem with tests?

* apparently not

* mypy WIP

* mypy WIP continued

* get mypy to pass for core

* add py.typed marker to core

* add py.typed marker to ext.ecoscope as well

* mypy passing for ext.ecoscope

* doctests pass for core

* fix doctest type

* ext.ecoscope doesn't have doctests

* recompile mode-map build dir

* build conda dist from examples WIP

* move conda-dist to top level

* ecoscope-workflows-core builds locally

* build release in ci

* build ext-ecoscope conda dist

* lock mode-map builddir

* add tests to artifacts

* work towards testable build

* remove .conda

* get namespaces packaged to build correctly on conda-dist

* relock

* i think this hatch config is finally correct

* still broken

* namespaces are hard

* relock and update ci

* fix stray type

* oops unsaved files

* fix doctest

* fix registry test

* recompile mode-map

* fix testing util test

* wow sequential test passes locally in mode-map

* up and download release artifacts

* maybe fix docs

* test more things

* try to fix docs again

* fix redirect

* artifacts WIP

* mypy caught an issues

* factor requirements into separate module

* use py-rattler types migration first commit

* get mypy passing for core

* i guess we need to do it like this

* try to get pixitoml working

* ok this works

* fix mypy and note issue

* manually sync core deps to ext-ecoscope and publish

* relock ext-ecoscope

* get default pixi toml test artifact tests passing

* drop unused (old) dependencies defintions

* add deps WIP and workaround rattler bug

* fix artifacts test

* fix mypy

* add requirements section to workflow spec

* add label switch to flaky io tests

* implement add_dependency function

* drop pixitoml default from artifacts, create onthefly in compiler

* work on recompiling workflow with generated pixitoml

* serialize our channels by name

* recompile pixi.toml dynamically

* add --lock option to compile

* move mapvalues to build dir

* remove old artifacts for mapvalues

* add testcases doc for mapvalues

* fix mypy

* enable mapvalues test in ci

* reformat testcases for mapvalues

* compile build dir for patrol events

* alternate chdir impl for py310

* patrol events test cases

* enable patrol events e2e test

* migrate subject tracking

* patrol workflow build dir

* enable subject tracking in ci

* migrate patrol workflow and enable in ci

* remove unnecessary reference tests

* rm last of the old examples artifacts

* fix workflow

* fix missing quotation mark in testcases

* Update .gitignore

* Update src/ecoscope-workflows-core/ecoscope_workflows_core/registry.py

* remove version checking, will address in #249

* revert readme changes

* reset ecoscope-workflows-core README to be equal to main

* Remove unused context manager

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update ext-ecoscope README header to reflect current name

* remove last of old tests

* add pull_request types to examples workflow

* comment out custom types

* just run CI on all pushes

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

.github/workflows/ci.yml

@@ -0,0 +1,110 @@
+name: CI
+
+on:
+  push:
+    branches: ["*"]
+  # pull_request:
+  #   branches: ["*"]
+  #   # types: [ opened, reopened, synchronize, labeled ]
+  schedule:
+    - cron: '0 4 * * *' # run once a day at 4 AM
+
+jobs:
+  mypy:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        namespace: ["core", "ext-ecoscope"]
+        python-version: ["310", "311", "312"]
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install pixi
+        run: curl -fsSL https://pixi.sh/install.sh | bash && echo "$HOME/.pixi/bin" >> $GITHUB_PATH
+      - name: Mypy
+        run: |
+          pixi run \
+          --manifest-path src/ecoscope-workflows-${{ matrix.namespace }}/pyproject.toml \
+          --locked --environment mypy-py${{ matrix.python-version }} \
+          mypy
+
+  doctest:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        namespace: ["core"]  #, "ext/ecoscope"]  # todo: enable once ext/ecoscope has doctests
+        python-version: ["310", "311", "312"]
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install pixi
+        run: curl -fsSL https://pixi.sh/install.sh | bash && echo "$HOME/.pixi/bin" >> $GITHUB_PATH
+      - name: Test
+        run: |
+          pixi run \
+          --manifest-path src/ecoscope-workflows-${{ matrix.namespace }}/pyproject.toml \
+          --locked --environment test-py${{ matrix.python-version }} \
+          pytest -v src/ecoscope-workflows-${{ matrix.namespace }} \
+          --doctest-modules
+
+  test-core:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["310", "311", "312"]
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install pixi
+        run: curl -fsSL https://pixi.sh/install.sh | bash && echo "$HOME/.pixi/bin" >> $GITHUB_PATH
+      - name: Test
+        run: |
+          pixi run \
+          --manifest-path src/ecoscope-workflows-core/pyproject.toml \
+          --locked --environment test-py${{ matrix.python-version }} \
+          pytest src/ecoscope-workflows-core/tests -vx
+
+  test-ext-ecoscope-not-io:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["310", "311", "312"]
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install pixi
+        run: curl -fsSL https://pixi.sh/install.sh | bash && echo "$HOME/.pixi/bin" >> $GITHUB_PATH
+      - name: Test
+        run: |
+          pixi run \
+          --manifest-path src/ecoscope-workflows-ext-ecoscope/pyproject.toml \
+          --locked --environment test-py${{ matrix.python-version }} \
+          pytest src/ecoscope-workflows-ext-ecoscope/tests -vx \
+          -m "not io"
+
+  test-ext-ecoscope-io:
+    # run if:
+    #  - all pushes to main
+    #  - schedule defined above
+    #  - a PR was just labeled 'test-io'
+    #  - a PR with labeled 'test-io' label was opened, reopened, or synchronized
+    if: |
+      github.event_name == 'push' ||
+      github.event_name == 'schedule' ||
+      github.event.label.name == 'test-io' ||
+      contains( github.event.pull_request.labels.*.name, 'test-io')
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["310", "311", "312"]
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install pixi
+        run: curl -fsSL https://pixi.sh/install.sh | bash && echo "$HOME/.pixi/bin" >> $GITHUB_PATH
+      - name: Test
+        env:
+          EARTHRANGER_SERVER: ${{ secrets.EARTHRANGER_SERVER }}
+          EARTHRANGER_USERNAME: ${{ secrets.EARTHRANGER_USERNAME }}
+          EARTHRANGER_PASSWORD: ${{ secrets.EARTHRANGER_PASSWORD }}
+        run: |
+          pixi run \
+          --manifest-path src/ecoscope-workflows-ext-ecoscope/pyproject.toml \
+          --locked --environment test-py${{ matrix.python-version }} \
+          pytest src/ecoscope-workflows-ext-ecoscope/tests -vx \
+          -m "io"

.github/workflows/examples.yml

@@ -0,0 +1,66 @@
+name: Examples
+
+on:
+  push:
+    branches: ["*"]
+    # types: [ opened, reopened, synchronize, labeled ]
+  # pull_request:
+  #   branches: ["*"]
+
+jobs:
+  build-release:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install pixi
+        run: curl -fsSL https://pixi.sh/install.sh | bash && echo "$HOME/.pixi/bin" >> $GITHUB_PATH
+      - name: Build release
+        run: |
+          pixi run \
+          --manifest-path ./pixi.toml \
+          --locked --environment conda-dist \
+          build-release
+      - name: Upload conda channel
+        uses: actions/upload-artifact@v4
+        with:
+          name: release-artifacts
+          path: |
+            /tmp/ecoscope-workflows/release/artifacts/
+            !/tmp/ecoscope-workflows/release/artifacts/bld
+            !/tmp/ecoscope-workflows/release/artifacts/src_cache
+          if-no-files-found: error
+          compression-level: 0
+
+  test-examples:
+    needs: build-release
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        script: [
+          "async-local",
+          "sequential-local",
+        ]
+        compilation-spec: [
+          "mode-map",
+          "mode-mapvalues",
+          "patrol-events",
+          "patrol-workflow",
+          "subject-tracking",
+        ]
+    steps:
+      - uses: actions/checkout@v4
+      - name: Download conda channel
+        uses: actions/download-artifact@v4
+        with:
+          name: release-artifacts
+          path: /tmp/ecoscope-workflows/release/artifacts
+      - name: Log conda channel contents
+        run: ls -lR /tmp/ecoscope-workflows/release/artifacts
+      - name: Install pixi
+        run: curl -fsSL https://pixi.sh/install.sh | bash && echo "$HOME/.pixi/bin" >> $GITHUB_PATH
+      - name: Test
+        working-directory: ./examples/${{ matrix.compilation-spec }}/workflow
+        run: |
+          pixi run --locked --manifest-path pixi.toml -e test \
+          test-${{ matrix.script }}-mock-io --case test1

.gitignore

@@ -1,5 +1,7 @@
 .DS_Store
 
+# pixi
+.pixi
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]

.pre-commit-config.yaml

@@ -1,5 +1,5 @@
 # https://pre-commit.com/
-# Adapted from https://github.com/TomNicholas/VirtualiZarr/blob/main/.pre-commit-config.yaml
+# xref https://github.com/zarr-developers/VirtualiZarr/blob/main/.pre-commit-config.yaml
 
 ci:
   autoupdate_schedule: monthly
@@ -13,32 +13,10 @@ repos:
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
     # Ruff version.
-    rev: "v0.4.3"
+    rev: "v0.6.3"
     hooks:
       # Run the linter.
       - id: ruff
         args: [--fix]
       # Run the formatter.
       - id: ruff-format
-
-  - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: v1.10.0
-    hooks:
-      - id: mypy
-        exclude: "examples/dags/"
-        additional_dependencies: [
-            # Type stubs
-            types-python-dateutil,
-            types-PyYAML,
-            types-pytz,
-            # Dependencies that are typed
-            numpy,
-            typing-extensions>=4.1.0,
-          ]
-  # run this occasionally, ref discussion https://github.com/pydata/xarray/pull/3194
-  # - repo: https://github.com/asottile/pyupgrade
-  #   rev: v3.15.2
-  #   hooks:
-  #     - id: pyupgrade
-  #       args:
-  #         - "--py310-plus"

.readthedocs.yaml

@@ -4,7 +4,6 @@ build:
   tools:
     python: "3.12"
   commands:
-    - pip install ".[lithops,visualize]"
-    - pip install pdoc3
-    - pdoc --html --output-dir $READTHEDOCS_OUTPUT/html/ ecoscope_workflows
-    - echo "<!DOCTYPE html><html><head><meta http-equiv=\"refresh\" content=\"0; url=ecoscope_workflows/index.html\"></head><body><p>If you are not redirected, <a href=\"ecoscope_workflows/index.html\">click here</a>.</p></body></html>" > $READTHEDOCS_OUTPUT/html/index.html
+    - curl -fsSL https://pixi.sh/install.sh | bash
+    - $HOME/.pixi/bin/pixi run --manifest-path src/ecoscope-workflows-core/pyproject.toml --locked -e docs pdoc --html --output-dir $READTHEDOCS_OUTPUT/html/ ecoscope_workflows_core
+    - echo "<!DOCTYPE html><html><head><meta http-equiv=\"refresh\" content=\"0; url=ecoscope_workflows_core/index.html\"></head><body><p>If you are not redirected, <a href=\"ecoscope_workflows_core/index.html\">click here</a>.</p></body></html>" > $READTHEDOCS_OUTPUT/html/index.html

development.md

@@ -0,0 +1,6 @@
+| feature            | pip | poetry | conda | mamba | micromamba | pixi.sh |
+| ---------------    | --- | ------ | ----- | ----- | ---------- | ------- |
+| python deps        |  ✅ |   ✅   | ✅    | ✅    | ✅         |  ✅     |
+| conda-compatible   |  ❌ |   ❌   | ✅    | ✅    | ✅         |  ✅     |
+| native lockfiles   |  ❌ |   ✅   | ❌    | ❌    | ❌         |  ✅     |
+| performance        |  ✅ |   ✅   | ❌    | 🤷    | 🤷         |  🔥     |