Skip to content

Commit

Permalink
Decrease page size for queries
Browse files Browse the repository at this point in the history
Decrease query driver's default page size.  Queries were consuming excessive memory on Butler server, and lowering this value  does not seem to make queries any slower.  This reduces memory consumption for a dataset query with dimension records from ~120MB to ~30MB per in-flight query.
  • Loading branch information
dhirving committed Aug 28, 2024
1 parent 11dbc91 commit d249b51
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion python/lsst/daf/butler/direct_query_driver/_driver.py
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,10 @@ def __init__(
dimension_record_cache: DimensionRecordCache,
default_collections: Iterable[str],
default_data_id: DataCoordinate,
raw_page_size: int = 10000,
# Increasing raw_page_size increases memory usage for queries on
# Butler server, so if you increase this you may need to increase the
# memory allocation for the server in Phalanx as well.
raw_page_size: int = 2000,
constant_rows_limit: int = 1000,
postprocessing_filter_factor: int = 10,
):
Expand Down

0 comments on commit d249b51

Please sign in to comment.