Skip to content

Commit

Permalink
Decrease page size for queries
Browse files Browse the repository at this point in the history
Decrease query driver's default page size.  Queries were consuming excessive memory on Butler server (>100MB per in-flight query), and lowering this value  does not seem to make queries any slower.
  • Loading branch information
dhirving committed Aug 28, 2024
1 parent 11dbc91 commit 5a87659
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion python/lsst/daf/butler/direct_query_driver/_driver.py
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,10 @@ def __init__(
dimension_record_cache: DimensionRecordCache,
default_collections: Iterable[str],
default_data_id: DataCoordinate,
raw_page_size: int = 10000,
# Increasing raw_page_size increases memory usage for queries on
# Butler server, so if you increase this you may need to increase the
# memory allocation for the server in Phalanx as well.
raw_page_size: int = 2000,
constant_rows_limit: int = 1000,
postprocessing_filter_factor: int = 10,
):
Expand Down

0 comments on commit 5a87659

Please sign in to comment.