mirror of
https://github.com/apache/impala.git
synced 2026-01-01 09:00:42 -05:00
c4d284f3cc
This is similar to the single-node execution optimisation, but applies to slightly larger queries that should run in a distributed manner but won't benefit from codegen. This adds a new query option disable_codegen_rows_threshold that defaults to 50,000. If fewer than this number of rows are processed by a plan node per impalad, the cost of codegen almost certainly outweighs the benefit. Using rows processed as a threshold is justified by a simple model that assumes the cost of codegen and execution per row for the same operation are proportional. E.g. if x is the complexity of the operation, n is the number of rows processed, C is a constant factor giving the cost of codegen and Ec/Ei are constant factor giving the cost of codegen'd and interpreted execution and d, then the cost of the codegen'd operator is C * x + Ec * x * n and the cost of the interpreted operator is Ei * x * n. Rearranging means that interpretation is cheaper if n < C / (Ei - Ec), i.e. that (at least with the simplified model) it makes sense to choose interpretation or codegen based on a constant threshold. The model also implies that it is somewhat safer to choose codegen because the additional cost of codegen is O(1) but the additional cost of interpretation is O(n). I ran some experiments with TPC-H Q1, varying the input table size, to determine what the cut-over point where codegen was beneficial was. The cutover was around 150k rows per node for both text and parquet. At 50k rows per node disabling codegen was very beneficial - around 0.12s versus 0.24s. To be somewhat conservative I set the default threshold to 50k rows. On more complex queries, e.g. TPC-H Q10, the cutover tends to be higher because there are plan nodes that process many fewer than the max rows. Fix a couple of minor issues in the frontend - the numNodes_ calculation could return 0 for Kudu, and the single node optimization didn't handle the case where for a scan node with conjuncts, a limit and missing stats correctly (it considered the estimate still valid.) Testing: Updated e2e tests that set disable_codegen to set disable_codegen_rows_threshold to 0, so that those tests run both with and without codegen still. Added an e2e test to make sure that the optimisation is applied in the backend. Added planner tests for various cases where codegen should and shouldn't be disabled. Perf: Added a targeted perf test for a join+agg over a small input, which benefits from this change. Change-Id: I273bcee58641f5b97de52c0b2caab043c914b32e Reviewed-on: http://gerrit.cloudera.org:8080/7153 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Impala Public Jenkins
31 lines
794 B
Plaintext
31 lines
794 B
Plaintext
====
|
|
---- QUERY
|
|
# alltypes has 7300 rows - codegen should be enabled if there
|
|
# are < 1000 backend daemons.
|
|
set disable_codegen_rows_threshold=8;
|
|
select count(*) from alltypes t1
|
|
join alltypestiny t2 on t1.id = t2.id
|
|
---- RESULTS
|
|
8
|
|
---- TYPES
|
|
bigint
|
|
---- RUNTIME_PROFILE
|
|
# Verify that codegen was enabled for join and scan
|
|
row_regex: .*Build Side Codegen Enabled.*
|
|
row_regex: .*TEXT Codegen Enabled.*
|
|
====
|
|
---- QUERY
|
|
# alltypes has 7300 rows - codegen should be disabled regardless
|
|
# of # of backend impala daemons.
|
|
set disable_codegen_rows_threshold=8000;
|
|
select count(*) from alltypes t1
|
|
join alltypestiny t2 on t1.id = t2.id
|
|
---- RESULTS
|
|
8
|
|
---- TYPES
|
|
bigint
|
|
---- RUNTIME_PROFILE
|
|
# Verify that codegen was disabled
|
|
row_regex: .*Codegen Disabled: disabled due to optimization hints.*
|
|
====
|