Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-39944: Update Execution butler builder to use Butler._registry #355

Merged
merged 4 commits into from
Jul 14, 2023
Merged

DM-39944: Update Execution butler builder to use Butler._registry #355

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a70de2f
Update Execution butler builder to use `Butler._registry`
andy-slac Jul 11, 2023
c40ae88
Do pre-commit autoupdate
andy-slac Jul 12, 2023
98dcaee
Fix noqa code
andy-slac Jul 12, 2023
3517803
Use Butler._datastore in test code to suppress warnings
andy-slac Jul 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ repos:
- id: trailing-whitespace
- id: check-toml
- repo: https://github.com/psf/black
rev: 23.1.0
rev: 23.7.0
hooks:
- id: black
# It is recommended to specify the latest version of Python
Expand All @@ -22,6 +22,6 @@ repos:
name: isort (python)
- repo: https://github.com/astral-sh/ruff-pre-commit
# Ruff version.
rev: v0.0.275
rev: v0.0.278
hooks:
- id: ruff
2 changes: 1 addition & 1 deletion python/lsst/pipe/base/connections.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ class PipelineTaskConnectionsMetaclass(type):
See `inputs` for additional information.
"""

def __prepare__(name, bases, **kwargs): # noqa: 805
def __prepare__(name, bases, **kwargs): # noqa: N804
# Create an instance of our special dict to catch and track all
# variables that are instances of connectionTypes.BaseConnection
# Copy any existing connections from a parent class
Expand Down
2 changes: 1 addition & 1 deletion python/lsst/pipe/base/executionButlerBuilder.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -240,7 +240,7 @@ def _export(butler: Butler, collections: Iterable[str] | None, inserts: DataSetT
# export/import
BackendClass = get_class_of(butler._config["repo_transfer_formats", "yaml", "export"])
backend = BackendClass(yamlBuffer, universe=butler.dimensions)
exporter = RepoExportContext(butler.registry, butler._datastore, backend, directory=None, transfer=None)
exporter = RepoExportContext(butler._registry, butler._datastore, backend, directory=None, transfer=None)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is looking suspiciously like RepoExportContext is not a public API...

Copy link
Contributor Author

@andy-slac andy-slac Jul 12, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think its instantiation is intended to happen inside Butler.export(), but execution butler does not like to use export() for some reason. I prefer to wait until we remove execution butler rather than fixing it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I consider everything but construction of RepoExportContext to be public API, but construction definitely is not.


# Need to ensure that the dimension records for outputs are
# transferred.
Expand Down
4 changes: 2 additions & 2 deletions python/lsst/pipe/base/tests/simpleQGraph.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -414,7 +414,7 @@ def makeSimpleQGraph(
query based on dataset existence, defaults to
`DatasetQueryConstraintVariant.ALL`.
makeDatastoreRecords : `bool`, optional
If `True` then add datstore records to generated quanta.
If `True` then add datastore records to generated quanta.
bind : `~collections.abc.Mapping`, optional
Mapping containing literal values that should be injected into the
``userQuery`` expression, keyed by the identifiers they replace.
Expand Down Expand Up @@ -446,7 +446,7 @@ def makeSimpleQGraph(
builder = GraphBuilder(
registry=butler.registry,
skipExistingIn=skipExistingIn,
datastore=butler.datastore if makeDatastoreRecords else None,
datastore=butler._datastore if makeDatastoreRecords else None,
)
if not run:
assert butler.run is not None, "Butler must have run defined"
Expand Down