Tasks: set groupers + split groups #160
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* pull in typevar upgrades * carve out arg prevalidators & return postvalidator * oh thank goodness * restore dropped validate call args * fix preprocessing test * move prepocessing task tests into tasks/ * fix task test factoring
…`map` and `mapvalues` methods (#144) * executors first commit * executor wip * reset on test decorators module * simple decorator tests * implement .map argname mapping in Task * try to implement mapvalues * mapvalues working * upacking works but mypy is not happy * try to satisfy mypy * try to satisfy mypy * ensure argnames and argvalues len match * mapvalues arg unpacking not implemented * mapvalues arg unpacking not implemented * lithops executor placeholder * DRYify type vars * remove stray operator kws refs * remove a few other stray registry objs
* drop replace method, add partial and validate methods * use functools partial, a few partial tests * test parens and dot call * oops actually use functools now * let's just allow kws for now * test partial raises * drop frozen instance stuff * some validate tests * doctest for validate * doctest for partial * doctest for call * docstring for map * mapvalues doctest * top level doctest * run doctest in CI * update templates * fix e2e test mocks * minor docstring fmt tweak
* rewrite specs * compiler test find and replace * start on the compiler * fix a test * fix another test * method computed field * check only oneof map or mapvalues * fix stray 'with' * examples tests WIP * omit args WIP * fix omit args * generate dag smoke test * sequential matches old API but should use partial * use partial in sequential macros * fix #121 * compiler typing work * mode map templating including arg name serialization * okay mode map works * pass params as macro * consolidate macros in one file * rename macros file * make validate settable on macro * e2e tests passing 🎉 * fix a test * fix another test * add test for map deps topology raises * get map deps raises to pass * raise on mapvalues deps topology errors as well * check self deps in map or mapvalues * fix oneof test * trying to get last test to pass * get tests to pass + satisfy mypy * simplify Spec.task_instance_dependencies * some docs * a little type renaming * check and test parallel op field names are both required * last test to fix? * don't need iterargs with coercion * docstring typo fix * second times the charm
|
Momentarily blocked by: I will fix this now |
cisaacstern
added a commit
that referenced
this pull request
Aug 8, 2024
…#162) * Decorator retool (#157) * Decorator maintenance (#126) * pull in typevar upgrades * carve out arg prevalidators & return postvalidator * oh thank goodness * restore dropped validate call args * fix preprocessing test * move prepocessing task tests into tasks/ * fix task test factoring * Rename decorator to `Task` / `@task` (#139) * decorator module renaming * widget tasks rename * more results renaming * more results renaming * more decorator rename * fix testing issue * more renaming * last of the renaming * ExecutorABC, PythonExecutor, and matching Task interface with `call` `map` and `mapvalues` methods (#144) * executors first commit * executor wip * reset on test decorators module * simple decorator tests * implement .map argname mapping in Task * try to implement mapvalues * mapvalues working * upacking works but mypy is not happy * try to satisfy mypy * try to satisfy mypy * ensure argnames and argvalues len match * mapvalues arg unpacking not implemented * mapvalues arg unpacking not implemented * lithops executor placeholder * DRYify type vars * remove stray operator kws refs * remove a few other stray registry objs * Task.partial + Task.validate methods, plus decorator doctests (#147) * drop replace method, add partial and validate methods * use functools partial, a few partial tests * test parens and dot call * oops actually use functools now * let's just allow kws for now * test partial raises * drop frozen instance stuff * some validate tests * doctest for validate * doctest for partial * doctest for call * docstring for map * mapvalues doctest * top level doctest * run doctest in CI * update templates * fix e2e test mocks * minor docstring fmt tweak * Rewrite spec and templates for new decorator API (#153) * rewrite specs * compiler test find and replace * start on the compiler * fix a test * fix another test * method computed field * check only oneof map or mapvalues * fix stray 'with' * examples tests WIP * omit args WIP * fix omit args * generate dag smoke test * sequential matches old API but should use partial * use partial in sequential macros * fix #121 * compiler typing work * mode map templating including arg name serialization * okay mode map works * pass params as macro * consolidate macros in one file * rename macros file * make validate settable on macro * e2e tests passing 🎉 * fix a test * fix another test * add test for map deps topology raises * get map deps raises to pass * raise on mapvalues deps topology errors as well * check self deps in map or mapvalues * fix oneof test * trying to get last test to pass * get tests to pass + satisfy mypy * simplify Spec.task_instance_dependencies * some docs * a little type renaming * check and test parallel op field names are both required * last test to fix? * don't need iterargs with coercion * docstring typo fix * second times the charm * recompile json * get mode map e2e to pass * get e2e tests to pass * don't write test files to local disk * recompile json * oops don't remove jinja2 * Tasks: set groupers + split groups (#160) * Decorator maintenance (#126) * pull in typevar upgrades * carve out arg prevalidators & return postvalidator * oh thank goodness * restore dropped validate call args * fix preprocessing test * move prepocessing task tests into tasks/ * fix task test factoring * Rename decorator to `Task` / `@task` (#139) * decorator module renaming * widget tasks rename * more results renaming * more results renaming * more decorator rename * fix testing issue * more renaming * last of the renaming * ExecutorABC, PythonExecutor, and matching Task interface with `call` `map` and `mapvalues` methods (#144) * executors first commit * executor wip * reset on test decorators module * simple decorator tests * implement .map argname mapping in Task * try to implement mapvalues * mapvalues working * upacking works but mypy is not happy * try to satisfy mypy * try to satisfy mypy * ensure argnames and argvalues len match * mapvalues arg unpacking not implemented * mapvalues arg unpacking not implemented * lithops executor placeholder * DRYify type vars * remove stray operator kws refs * remove a few other stray registry objs * Task.partial + Task.validate methods, plus decorator doctests (#147) * drop replace method, add partial and validate methods * use functools partial, a few partial tests * test parens and dot call * oops actually use functools now * let's just allow kws for now * test partial raises * drop frozen instance stuff * some validate tests * doctest for validate * doctest for partial * doctest for call * docstring for map * mapvalues doctest * top level doctest * run doctest in CI * update templates * fix e2e test mocks * minor docstring fmt tweak * Rewrite spec and templates for new decorator API (#153) * rewrite specs * compiler test find and replace * start on the compiler * fix a test * fix another test * method computed field * check only oneof map or mapvalues * fix stray 'with' * examples tests WIP * omit args WIP * fix omit args * generate dag smoke test * sequential matches old API but should use partial * use partial in sequential macros * fix #121 * compiler typing work * mode map templating including arg name serialization * okay mode map works * pass params as macro * consolidate macros in one file * rename macros file * make validate settable on macro * e2e tests passing 🎉 * fix a test * fix another test * add test for map deps topology raises * get map deps raises to pass * raise on mapvalues deps topology errors as well * check self deps in map or mapvalues * fix oneof test * trying to get last test to pass * get tests to pass + satisfy mypy * simplify Spec.task_instance_dependencies * some docs * a little type renaming * check and test parallel op field names are both required * last test to fix? * don't need iterargs with coercion * docstring typo fix * second times the charm * forklift in tasks from prior attempt * forklift groupby tests * forklift mapvalues example from #120 * use new mapvalues syntax in mapvalues * handle mapvalues in jinja macro * generate jupytext for mapvalues * make dashboard from widgets * generate yaml and json first pass * pass groupers through to dashboard * failing test for yaml form bug * rename reframe and test per_task_omit_args * finish fix for #161 * recompile mapvalues params following fix for #161 * #161 continued; notebook params need full spec context * recompile mapvalues dag for patrol events * recompile mapvalues params * check and test for invalid argnames * parametrized invalid arg name test * get mapvalues e2e test to pass WIP * get mapvalues e2e test to pass 🎉 * add two more asserts for mapvalues * recompile json * fix split groups test * fix dashboard tests * set groupers as an object not a string * set help text * fix dashboard tests WIP * a little more headway on dashboard tests * more dashboard test fixes * all dashboard tests passing * fix mapvalues e2d * fix merge mistake * fix e2e tests * recompile json * doctest for _serialize_variables * doctest for _groupkey_to_composite_filter
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Built on #157, which should go in first
closes #72
closes #109
closes #156
closes #161
Supercedes #120