DM-41117: Improvements to caching and Butler initialization

[andy-slac]

A bunch of improvements for collection caching to reduce the need to cache all collections in memory. In some cases when we need to filter collections based on regexes we still need to fetch everything into memory, but in other cases we only read collections that are explicitly specified.

Similarly, collection summary cache is eliminated completely to avoid upfront loading which takes significant time. Now we are only fetching summaries for collections that are needed.

Table reflection is eliminated for static tables, and is postponed for dynamic (tags/calib) tables until they are used.

Checklist

• ran Jenkins
• added a release note for user-visible changes to doc/changes

[codecov]

Codecov Report

Attention: 121 lines in your changes are missing coverage. Please review.

Comparison is base (faf02a0) 87.86% compared to head (922359c) 87.72%.

Files	Patch %	Lines
...st/daf/butler/registry/_collection_record_cache.py	33.33%	30 Missing ⚠️
...thon/lsst/daf/butler/registry/collections/_base.py	71.87%	16 Missing and 11 partials ⚠️
.../butler/registry/datasets/byDimensions/_manager.py	80.35%	11 Missing and 11 partials ⚠️
...t/daf/butler/registry/_collection_summary_cache.py	50.00%	9 Missing ⚠️
...on/lsst/daf/butler/registry/_dataset_type_cache.py	78.04%	7 Missing and 2 partials ⚠️
...butler/registry/datasets/byDimensions/summaries.py	83.67%	5 Missing and 3 partials ⚠️
...ython/lsst/daf/butler/registry/_caching_context.py	80.95%	4 Missing ⚠️
python/lsst/daf/butler/registry/sql_registry.py	62.50%	3 Missing ⚠️
python/lsst/daf/butler/_named.py	0.00%	0 Missing and 2 partials ⚠️
.../butler/registry/datasets/byDimensions/_storage.py	88.23%	1 Missing and 1 partial ⚠️
... and 5 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #904      +/-   ##
==========================================
- Coverage   87.86%   87.72%   -0.14%     
==========================================
  Files         281      285       +4     
  Lines       36440    36744     +304     
  Branches     7604     7690      +86     
==========================================
+ Hits        32017    32233     +216     
- Misses       3273     3340      +67     
- Partials     1150     1171      +21     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[TallJimbo]

Everything here looks great; definitely a step in the right direction.

[andy-slac]

@TallJimbo, sorry for another review request, I have added three more commits to try to improve things:

• 5833c91 - removes caching from dataset type manager, looking at the debug logs I realized that we now have too many SQL queries.
• 14721a2 - adds context manager to Registry that enables caching and adds caches for collections and collection summaries
• 1e59418 - adds cache for dataset types.

I'm not very happy with how it is structured now, cache for dataset types looks particularly ugly, but this is the best that I could do. As the result, number of queries is approximately the same in both QG building and pipetask run in ci_hsc_gen3. (In QG generation the number of queries (but not time, in my setup) is actually dominated by _skyFrameLookup).

[TallJimbo]

Looks good - I think it's time to wrap this one up and start one or more new tickets for further caching work, as I think this is a clear step in the right direction with some immediate benefits and no obvious downsides.

I do still think there's a ton of work ahead here, because the caching context interface implies a per-user cache and that'd be a massive pain to put on the server, and yet some of what we cache now can only go on the server (SQLAlchemy table objects) or needs to be on the server as a check even if it's also on the client (collection summaries). I think I'd advise some "step back and design it conceptually" work before further coding on this problem, though.

Typo: additiona in commit message for 14721a2.