mirror of
https://github.com/apache/impala.git
synced 2026-01-22 00:01:21 -05:00
ca53f68525
This adds an advanced PREAGG_BYTES_LIMIT query option that allows limiting the memory consumption of streaming preaggregation operators in a query. It works by setting a maximum reservation on each grouping aggregator in a preaggregation node. The aggregators switch to passthrough mode automatically when hitting this limit, the same as if they were hitting the query memory limit. This does not override the minimum reservation computed for the aggregation - if the limit is less than the minimum reservation, the minimum reservation is used as the limit instead. The default behaviour is unchanged. Testing: Add a planner test with estimates higher and lower than limit to ensure that resource estimates correctly reflect the option. Add an end-to-end test that verifies that the option forces passthrough when the memory limit is hit. Change-Id: I87f7a5c68da93d068e304ef01afbcbb0d56807d9 Reviewed-on: http://gerrit.cloudera.org:8080/15463 Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>