mirror of
https://github.com/apache/impala.git
synced 2026-01-08 03:02:48 -05:00
1ada739e81
This fixes the analytic push down optimization for the case where the ORDER BY expressions are compatible with the partitioning of the analytic *and* there is a rank() or row_number() predicate. In this case the rows returned are going to come from the first partitions, i.e. if the limit is 100, if we go through the partitions in order until the row count adds up to 100, then we know that the rows must come from those partitions. The problem is that predicates can discard rows from the partitions, meaning that a limit naively pushed down to the top-n will filter out rows that could be returned from the query. We can avoid the problem in the case where the partition limit >= order by limit, however. In this case the relevant set of partitions is the set of partitions that include the first <limit> rows, since the top-level limit generally kicks in before the per-partition limit. The only twist is that the orderings may be different within a partition, so we need to make sure to include all of the rows in the final partition. The solution implemented in this patch is to increase the pushed down limit so that it is always guaranteed to include all of the rows in the final partition to be returned. E.g. if you had a row_number() <= 100 predicate and limit 100, if you pushed down limit 200, then you'd be guaranteed to capture all of the rows in the final partition. One case we need to handle is that, in the case of a rank() predicate, we can have more than that number of rows in the partition because of ties. This patch implements tie handling in the backend (I took most of that implementation from my in-progress partitioned top-n patch, with the intention of rebasing that onto this patch). This also adds a check against TOPN_BYTES_LIMIT so that the limit can't be increased to an arbitarily large value. Testing: * Add new planner test with negative case where it's rejected because the transformation is incorrect. * Update other planner tests to reflect new limit calculation + tie handling required for correctness. * Add planner test for very high rank predicate that overflows int32 * Add planner test that checks TOPN_BYTES_LIMIT handling * Add planner test that checks that dense_rank() can't be pushed. * Existing planner tests already have adequate coverage for predicates : <=, <, = and row_number(). * Add some end-to-end tests that repro bugs that fall under the jira * Add an end-to-end test on TPC-H with more data to exercise the tie-handling logic in the execnode more. Perf: Ran TPC-DS q67 with mt_dop=1 on a single node, confirmed there was no measurable change in performance as a result of this patchset. Ran TPC-H scale 30 on a single node, no significant perf change. Ran a targeted query to check for regressions in the top-n node. The elapsed time for this targeted query did not change: use tpch30_parquet; set mt_dop=1; select l_extendedprice from lineitem order by 1 limit 100 Change-Id: I801d7799b0d649c73d2dd1703729a9b58a662509 Reviewed-on: http://gerrit.cloudera.org:8080/16942 Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
97 lines
2.3 KiB
Plaintext
97 lines
2.3 KiB
Plaintext
====
|
|
---- QUERY
|
|
# IMPALA-10296: this query returns correct results even before
|
|
# IMPALA-10296 because the limit is not hit. Reducing the limit
|
|
# should simply truncate the results because the ordering is total.
|
|
select tinyint_col, id from (
|
|
select *, rank() over (partition by tinyint_col order by id) rnk
|
|
from alltypestiny) v
|
|
where rnk < 4
|
|
order by tinyint_col, id desc
|
|
limit 10
|
|
---- TYPES
|
|
TINYINT,INT
|
|
---- RESULTS
|
|
0,4
|
|
0,2
|
|
0,0
|
|
1,5
|
|
1,3
|
|
1,1
|
|
====
|
|
---- QUERY
|
|
# IMPALA-10296: it is not safe to push the limit down past the analytic.
|
|
# This query reproduced incorrect results prior to the IMPALA-10296 fix.
|
|
select tinyint_col, id from (
|
|
select *, rank() over (partition by tinyint_col order by id) rnk
|
|
from alltypestiny) v
|
|
where rnk < 4
|
|
order by tinyint_col, id desc
|
|
limit 5
|
|
---- TYPES
|
|
TINYINT,INT
|
|
---- RESULTS
|
|
0,4
|
|
0,2
|
|
0,0
|
|
1,5
|
|
1,3
|
|
====
|
|
---- QUERY
|
|
# IMPALA-10296: this query returns correct results even before
|
|
# IMPALA-10296 because the limit is not hit. Reducing the limit
|
|
# should simply truncate the results because the ordering is total.
|
|
select tinyint_col, string_col, id, rnk from (
|
|
select *, rank() over (partition by tinyint_col order by string_col) rnk
|
|
from alltypestiny) v
|
|
where rnk <= 5
|
|
order by tinyint_col, string_col desc, id desc
|
|
limit 10
|
|
---- TYPES
|
|
TINYINT,STRING,INT,BIGINT
|
|
---- RESULTS
|
|
0,'0',6,1
|
|
0,'0',4,1
|
|
0,'0',2,1
|
|
0,'0',0,1
|
|
1,'1',7,1
|
|
1,'1',5,1
|
|
1,'1',3,1
|
|
1,'1',1,1
|
|
====
|
|
---- QUERY
|
|
# IMPALA-10296: the limit can be pushed past the analytic operator,
|
|
# but we need to increase the limit and include ties to guarantee
|
|
# correct results.
|
|
select tinyint_col, string_col, id, rnk from (
|
|
select *, rank() over (partition by tinyint_col order by string_col) rnk
|
|
from alltypestiny) v
|
|
where rnk <= 5
|
|
order by tinyint_col, string_col desc, id desc
|
|
limit 5
|
|
---- TYPES
|
|
TINYINT,STRING,INT,BIGINT
|
|
---- RESULTS
|
|
0,'0',6,1
|
|
0,'0',4,1
|
|
0,'0',2,1
|
|
0,'0',0,1
|
|
1,'1',7,1
|
|
====
|
|
---- QUERY
|
|
# IMPALA-10296: cannot push limit through an equality comparison to
|
|
# rank the predicate can filter out an arbitrary number of rows returned
|
|
# from the top-n sort.
|
|
select tinyint_col, string_col, id, rnk from (
|
|
select *, rank() over (partition by tinyint_col order by id) rnk
|
|
from alltypestiny) v
|
|
where rnk = 2
|
|
order by tinyint_col, string_col desc, id desc
|
|
limit 2
|
|
---- TYPES
|
|
TINYINT,STRING,INT,BIGINT
|
|
---- RESULTS
|
|
0,'0',2,2
|
|
1,'1',3,2
|
|
====
|