Skip to content

Commit

Permalink
Prototype partial(?) replacements relation-tree objects.
Browse files Browse the repository at this point in the history
The current daf_relation tree in our Query objects is too contaminated
with implementation details to serve us well in the RemoteButler
client, and we could only partially mitigate that by defining a new
daf_relation Engine.  The fundamental issue is that daf_relation
objects expect to know exactly which columns they have at any given
time, and that's at odds with user expectations that columns
"magically" appear whenever they're requested (e.g. referenced by a
`where` string), and that this joins in the tables that provide them
as-needed.

The two new files here *heavily* duplicate stuff in daf_relation, and
in addition to being more vague about what the columns are, they're
simpler for two key reasons:

- They're just data, with no logic for maintaining invariants,
  contructing trees, serialization, or anything else.  This will have
  to change as we actually start to use them.

- They fully enumerate the kinds of expressions and operations we care
  about in the butler query system rather than trying to define
  abstract versions of those upstream of daf_butler that could be
  specialized in daf_butler.  I had not appreciated how much of a
  simplification this could be when writing daf_relation as a separate
  package, and if it holds up it may suggest that the right way to
  resolve the duplication is to rip a lot of stuff out of daf_relation.
  serialization.
  • Loading branch information
TallJimbo committed Nov 30, 2023
1 parent 044b3a4 commit 54d6588
Show file tree
Hide file tree
Showing 3 changed files with 520 additions and 0 deletions.
26 changes: 26 additions & 0 deletions python/lsst/daf/butler/queries/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# This file is part of daf_butler.
#
# Developed for the LSST Data Management System.
# This product includes software developed by the LSST Project
# (http://www.lsst.org).
# See the COPYRIGHT file at the top-level directory of this distribution
# for details of code ownership.
#
# This software is dual licensed under the GNU General Public License and also
# under a 3-clause BSD license. Recipients may choose which of these licenses
# to use; please see the files gpl-3.0.txt and/or bsd_license.txt,
# respectively. If you choose the GPL option then the following text applies
# (but note that there is still no warranty even if you opt for BSD instead):
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
212 changes: 212 additions & 0 deletions python/lsst/daf/butler/queries/abstract_expressions.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,212 @@
# This file is part of daf_butler.
#
# Developed for the LSST Data Management System.
# This product includes software developed by the LSST Project
# (http://www.lsst.org).
# See the COPYRIGHT file at the top-level directory of this distribution
# for details of code ownership.
#
# This software is dual licensed under the GNU General Public License and also
# under a 3-clause BSD license. Recipients may choose which of these licenses
# to use; please see the files gpl-3.0.txt and/or bsd_license.txt,
# respectively. If you choose the GPL option then the following text applies
# (but note that there is still no warranty even if you opt for BSD instead):
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.

from __future__ import annotations

Check warning on line 28 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L28

Added line #L28 was not covered by tests

__all__ = (

Check warning on line 30 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L30

Added line #L30 was not covered by tests
"AbstractExpression",
"AbstractOrderExpression",
"AbstractPredicate",
)


import dataclasses
from typing import Literal, TypeAlias, Union, TYPE_CHECKING

Check warning on line 38 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L37-L38

Added lines #L37 - L38 were not covered by tests

import astropy.time

Check warning on line 40 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L40

Added line #L40 was not covered by tests

from lsst.sphgeom import Region

Check warning on line 42 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L42

Added line #L42 was not covered by tests

from ..dimensions import DataCoordinate
from .._column_tags import DimensionKeyColumnTag, DimensionRecordColumnTag, DatasetColumnTag
from .._timespan import Timespan

Check warning on line 46 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L44-L46

Added lines #L44 - L46 were not covered by tests

if TYPE_CHECKING:
from .abstract_relations import AbstractRelation


LiteralValue: TypeAlias = Union[int, bytes, str, float, astropy.time.Time, Timespan, Region]

Check warning on line 52 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L52

Added line #L52 was not covered by tests


@dataclasses.dataclass(frozen=True)
class ColumnLiteral:

Check warning on line 56 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L56

Added line #L56 was not covered by tests
"""A column expression that is a literal Python value."""

value: LiteralValue

Check warning on line 59 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L59

Added line #L59 was not covered by tests


@dataclasses.dataclass(frozen=True)
class ColumnReference:

Check warning on line 63 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L63

Added line #L63 was not covered by tests
"""A column expression that refers to a column obtainable from an abstract
relation.
"""

column: DimensionKeyColumnTag | DimensionRecordColumnTag | DatasetColumnTag

Check warning on line 68 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L68

Added line #L68 was not covered by tests


@dataclasses.dataclass(frozen=True)
class UnaryExpression:

Check warning on line 72 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L72

Added line #L72 was not covered by tests
"""A unary operation on a column expression that returns a non-bool."""

operand: AbstractExpression
operator: Literal["-", "begin_of", "end_of"]

Check warning on line 76 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L75-L76

Added lines #L75 - L76 were not covered by tests


@dataclasses.dataclass(frozen=True)
class BinaryExpression:

Check warning on line 80 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L80

Added line #L80 was not covered by tests
"""A binary operation on column expressions that returns a non-bool."""

a: AbstractExpression
b: AbstractExpression
operator: Literal["+", "-", "*", "/", "%"]

Check warning on line 85 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L83-L85

Added lines #L83 - L85 were not covered by tests


AbstractExpression: TypeAlias = Union[ColumnLiteral, ColumnReference, UnaryExpression, BinaryExpression]

Check warning on line 88 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L88

Added line #L88 was not covered by tests


@dataclasses.dataclass(frozen=True)
class Reversed:

Check warning on line 92 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L92

Added line #L92 was not covered by tests
"""A tag wrapper for `AbstractExpression` that indicate sorting in
reverse order.
"""

operand: AbstractExpression

Check warning on line 97 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L97

Added line #L97 was not covered by tests


AbstractOrderExpression: TypeAlias = Union[AbstractExpression, Reversed]

Check warning on line 100 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L100

Added line #L100 was not covered by tests


@dataclasses.dataclass(frozen=True)
class LogicalAnd:

Check warning on line 104 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L104

Added line #L104 was not covered by tests
"""A boolean column expression that is `True` only if all of its operands
are `True`.
"""

operands: tuple[AbstractPredicate]

Check warning on line 109 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L109

Added line #L109 was not covered by tests


@dataclasses.dataclass(frozen=True)
class LogicalOr:

Check warning on line 113 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L113

Added line #L113 was not covered by tests
"""A boolean column expression that is `True` if any of its operands are
`True`.
"""

operands: tuple[AbstractPredicate]

Check warning on line 118 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L118

Added line #L118 was not covered by tests


@dataclasses.dataclass(frozen=True)
class LogicalNot:

Check warning on line 122 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L122

Added line #L122 was not covered by tests
"""A boolean column expression that inverts its operand."""

operand: AbstractPredicate

Check warning on line 125 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L125

Added line #L125 was not covered by tests


@dataclasses.dataclass(frozen=True)
class IsNull:

Check warning on line 129 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L129

Added line #L129 was not covered by tests
"""A boolean column expression that tests whether its operand is NULL."""

operand: AbstractExpression

Check warning on line 132 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L132

Added line #L132 was not covered by tests


@dataclasses.dataclass(frozen=True)
class Comparison:

Check warning on line 136 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L136

Added line #L136 was not covered by tests
"""A boolean columns expression formed by comparing two non-boolean
expressions.
"""

a: AbstractExpression
b: AbstractExpression
operator: Literal["=", "!=", "<", ">", ">=", "<=", "overlaps"]

Check warning on line 143 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L141-L143

Added lines #L141 - L143 were not covered by tests


@dataclasses.dataclass(frozen=True)
class InContainer:

Check warning on line 147 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L147

Added line #L147 was not covered by tests
"""A boolean column expression that tests whether one expression is a
member of an explicit sequence of other expressions.
"""

member: AbstractExpression
container: tuple[AbstractExpression, ...]

Check warning on line 153 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L152-L153

Added lines #L152 - L153 were not covered by tests


@dataclasses.dataclass(frozen=True)
class InRange:

Check warning on line 157 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L157

Added line #L157 was not covered by tests
"""A boolean column expression that tests whether its expression is
included in an integer range.
"""

member: AbstractExpression
range: range

Check warning on line 163 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L162-L163

Added lines #L162 - L163 were not covered by tests


@dataclasses.dataclass(frozen=True)
class InRelation:

Check warning on line 167 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L167

Added line #L167 was not covered by tests
"""A boolean column expression that tests whether its expression is
included single-column projection of a relation.
This is primarily intended to be used on dataset ID columns, but it may
be useful for other columns as well.
"""

member: AbstractExpression
column: DimensionKeyColumnTag | DimensionRecordColumnTag | DatasetColumnTag
relation: AbstractRelation

Check warning on line 177 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L175-L177

Added lines #L175 - L177 were not covered by tests


@dataclasses.dataclass(frozen=True)
class StringPredicate:

Check warning on line 181 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L181

Added line #L181 was not covered by tests
"""A tag wrapper for boolean column expressions created by parsing a string
expression.
Remembering the original string is useful for error reporting.
"""

where: str
tree: AbstractPredicate

Check warning on line 189 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L188-L189

Added lines #L188 - L189 were not covered by tests


@dataclasses.dataclass(frozen=True)
class DataCoordinateConstraint:

Check warning on line 193 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L193

Added line #L193 was not covered by tests
"""A boolean column expression defined by interpreting data ID's key-value
pairs as a logical AND of equality constraints.
"""

data_coordinate: DataCoordinate

Check warning on line 198 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L198

Added line #L198 was not covered by tests


AbstractPredicate: TypeAlias = Union[

Check warning on line 201 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L201

Added line #L201 was not covered by tests
LogicalAnd,
LogicalOr,
LogicalNot,
IsNull,
Comparison,
InContainer,
InRange,
InRelation,
StringPredicate,
DataCoordinateConstraint,
]
Loading

0 comments on commit 54d6588

Please sign in to comment.