Skip to content

Commit

Permalink
Coalesce data ID query explanation into a single log.
Browse files Browse the repository at this point in the history
Logging a single statement as multiple logs causes problems with Loki,
because each line may be rearranged or independently filtered. Logging
a single multiline message fixes this problem, without changing
behavior for console users.

Constructing the multiline message makes it impossible to defer string
evaluation, but for an error-level log this is unlikely to come up
in practice.
  • Loading branch information
kfindeisen committed Aug 28, 2024
1 parent e2fa2c8 commit f6f0254
Show file tree
Hide file tree
Showing 2 changed files with 31 additions and 21 deletions.
1 change: 1 addition & 0 deletions doc/changes/DM-45722.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Explanatory logs for "initial data ID query returned no rows" now appear as a single log message instead of one entry per line. This improves display in log aggregators, but there is no change to console behavior.
51 changes: 30 additions & 21 deletions python/lsst/pipe/base/all_dimensions_quantum_graph_builder.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
import dataclasses
from collections.abc import Iterator, Mapping
from contextlib import contextmanager
from io import StringIO
from typing import TYPE_CHECKING, Any, final

from lsst.daf.butler.registry import MissingDatasetTypeError
Expand Down Expand Up @@ -507,31 +508,39 @@ def from_builder(
yield result

def log_failure(self, log: LsstLogAdapter) -> None:
"""Emit a series of ERROR-level log message that attempts to explain
"""Emit an ERROR-level log message that attempts to explain
why the initial data ID query returned no rows.
Parameters
----------
log : `logging.Logger`
The logger to use to emit log messages.
"""
log.error("Initial data ID query returned no rows, so QuantumGraph will be empty.")
for message in self.common_data_ids.explain_no_results():
log.error(message)
log.error(
"To reproduce this query for debugging purposes, run "
"Registry.queryDataIds with these arguments:"
)
# We could just repr() the queryArgs dict to get something
# the user could make sense of, but it's friendlier to
# put these args in an easier-to-reconstruct equivalent form
# so they can read it more easily and copy and paste into
# a Python terminal.
log.error(" dimensions=%s,", list(self.query_args["dimensions"].names))
log.error(" dataId=%s,", dict(self.query_args["dataId"].required))
if self.query_args["where"]:
log.error(" where=%s,", repr(self.query_args["where"]))
if "datasets" in self.query_args:
log.error(" datasets=%s,", list(self.query_args["datasets"]))
if "collections" in self.query_args:
log.error(" collections=%s,", list(self.query_args["collections"]))
# A single multiline log plays better with log aggregators like Loki.
buffer = StringIO()
try:
buffer.write("Initial data ID query returned no rows, so QuantumGraph will be empty.\n")

Check warning on line 522 in python/lsst/pipe/base/all_dimensions_quantum_graph_builder.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/pipe/base/all_dimensions_quantum_graph_builder.py#L520-L522

Added lines #L520 - L522 were not covered by tests
for message in self.common_data_ids.explain_no_results():
buffer.write(message)
buffer.write("\n")
buffer.write(

Check warning on line 526 in python/lsst/pipe/base/all_dimensions_quantum_graph_builder.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/pipe/base/all_dimensions_quantum_graph_builder.py#L524-L526

Added lines #L524 - L526 were not covered by tests
"To reproduce this query for debugging purposes, run "
"Registry.queryDataIds with these arguments:\n"
)
# We could just repr() the queryArgs dict to get something
# the user could make sense of, but it's friendlier to
# put these args in an easier-to-reconstruct equivalent form
# so they can read it more easily and copy and paste into
# a Python terminal.
buffer.write(f" dimensions={list(self.query_args['dimensions'].names)},")
buffer.write(f" dataId={dict(self.query_args['dataId'].required)},")

Check warning on line 536 in python/lsst/pipe/base/all_dimensions_quantum_graph_builder.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/pipe/base/all_dimensions_quantum_graph_builder.py#L535-L536

Added lines #L535 - L536 were not covered by tests
if self.query_args["where"]:
buffer.write(f" where={repr(self.query_args['where'])},")

Check warning on line 538 in python/lsst/pipe/base/all_dimensions_quantum_graph_builder.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/pipe/base/all_dimensions_quantum_graph_builder.py#L538

Added line #L538 was not covered by tests
if "datasets" in self.query_args:
buffer.write(f" datasets={list(self.query_args['datasets'])},")

Check warning on line 540 in python/lsst/pipe/base/all_dimensions_quantum_graph_builder.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/pipe/base/all_dimensions_quantum_graph_builder.py#L540

Added line #L540 was not covered by tests
if "collections" in self.query_args:
buffer.write(f" collections={list(self.query_args['collections'])},")

Check warning on line 542 in python/lsst/pipe/base/all_dimensions_quantum_graph_builder.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/pipe/base/all_dimensions_quantum_graph_builder.py#L542

Added line #L542 was not covered by tests
finally:
# If an exception was raised, write a partial.
log.error(buffer.getvalue())
buffer.close()

Check warning on line 546 in python/lsst/pipe/base/all_dimensions_quantum_graph_builder.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/pipe/base/all_dimensions_quantum_graph_builder.py#L545-L546

Added lines #L545 - L546 were not covered by tests

0 comments on commit f6f0254

Please sign in to comment.