mirror of
https://github.com/apache/impala.git
synced 2025-12-25 02:03:09 -05:00
101e10ba31
Recent changes such as IMPALA-11924 and IMPALA-8042 managed make NDV estimate more accurate in some cases. However, the more accurate (smaller) NDV estimates after these changes have exacerbated the problem with the 75% default FPP, which causes more cases of badly undersized filters. This patch lower default value of max_filter_error_rate flag from 75% to 10%. Lower target FPP will result in doubling runtime filter size most of the time when previous FPP is greater than 10%. Testing: - Pass exhaustive tests. - Manually ran a TPC-DS test at 3 TB comparing 10% to 75%. A value of 10% improves q94 by 2x and q95 by 5x, improves total query time and geomean time by a few percent, and doesn't cause a significant (> 10%) regression in any individual query. Change-Id: I4104e65cc3ce0ef4b36f6420f5044f2cdba9de04 Reviewed-on: http://gerrit.cloudera.org:8080/21552 Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
55 lines
2.4 KiB
Plaintext
55 lines
2.4 KiB
Plaintext
====
|
|
---- QUERY
|
|
set mem_limit=10mb;
|
|
select distinct * from functional_parquet.alltypesagg
|
|
---- CATCH
|
|
minimum memory reservation is greater than memory available to the
|
|
query for buffer reservations. Memory reservation needed given the
|
|
current plan: 68.09 MB. Adjust the MEM_LIMIT option for the query to allow the query
|
|
memory limit to be at least 100.09 MB. Note that changing the memory limit
|
|
may also change the plan. See 'Per Host Min Memory Reservation' in the query
|
|
profile for more information about the per-node memory requirements.
|
|
====
|
|
---- QUERY
|
|
set mem_limit=150mb;
|
|
select distinct * from functional_parquet.alltypesagg
|
|
---- CATCH
|
|
minimum memory reservation needed is greater than pool max mem resources.
|
|
Pool max mem resources: 40.00 MB.
|
|
Cluster-wide memory reservation needed: 208.26 MB.
|
|
Increase the pool max mem resources. See 'Per Host Min Memory Reservation' in the query
|
|
profile for more information about the per-node memory requirements.
|
|
====
|
|
---- QUERY
|
|
set buffer_pool_limit=10mb;
|
|
select distinct * from functional_parquet.alltypesagg
|
|
---- CATCH
|
|
row_regex:.*minimum memory reservation on backend '.*'
|
|
is greater than memory available to the query
|
|
for buffer reservations\. Increase the buffer_pool_limit to 72.09 MB\.
|
|
====
|
|
---- QUERY
|
|
set mem_limit=1024;
|
|
select count(*)
|
|
from tpch_parquet.lineitem join tpch_parquet.orders on l_orderkey = o_orderkey
|
|
---- CATCH
|
|
minimum memory reservation is greater than memory available to the
|
|
query for buffer reservations. Memory reservation needed given the
|
|
current plan: 46.00 MB. Adjust the MEM_LIMIT option for the query to allow the query
|
|
memory limit to be at least 78.00 MB. Note that changing the memory limit
|
|
may also change the plan. See 'Per Host Min Memory Reservation' in the query
|
|
profile for more information about the per-node memory requirements.
|
|
====
|
|
---- QUERY
|
|
set mem_limit=50mb;
|
|
select count(*)
|
|
from tpch_parquet.lineitem join tpch_parquet.orders on l_orderkey = o_orderkey
|
|
---- CATCH
|
|
minimum memory reservation is greater than memory available to the
|
|
query for buffer reservations. Memory reservation needed given the
|
|
current plan: 44.00 MB. Adjust the MEM_LIMIT option for the query to allow the query
|
|
memory limit to be at least 76.00 MB. Note that changing the memory limit
|
|
may also change the plan. See 'Per Host Min Memory Reservation' in the query
|
|
profile for more information about the per-node memory requirements.
|
|
====
|