Gabor Kaszab 7e0feb4a8e
IMPALA-11701 Part1: Don't push down predicates to scanner if already applied by Iceberg ...
We push down predicates to Iceberg that uses them to filter out files
when getting the results of planFiles(). Using the
FileScanTask.residual() function we can find out if we have to use
the predicates to further filter the rows of the given files or if
Iceberg has already performed all the filtering.
Basically if we only filter on IDENTITY-partition columns then Iceberg
can filter the files and using these filters in Impala wouldn't filter
any more rows from the output (assuming that no partition evolution was
performed on the table).
An additional benefit of not pushing down no-op predicates to the
scanner is that we can potentially materialize less slots.
For example:
SELECT count(1) from iceberg_tbl where part_col = 10;
Another additional benefit comes with count(*) queries. If all the
predicates are skipped from being pushed to Impala's scanner for a
count(*) query then the Parquet scanner can go to an optimized path
where it uses stats instead of reading actual data to answer the query.
In the above query Iceberg filters the files using the predicate on
a partition column and then there won't be any need to materialize
'part_col' in Impala, nor to push down the 'part_col = 10' predicate.
Note, this is an all or nothing approach, meaning that assuming N
number of predicates we either push down all predicates to the scanner
or none of them. There is a room for improvement to identify a subset
of the predicates that we still have to push down to the scanner.
However, for this we'd need a mapping between Impala predicates and the
predicates returned by Iceberg's FileScanTask.residual() function that
would significantly increase the complexity of the relevant code.
Testing:
- Some existing tests needed some extra care as they were checking
for predicates being pushed down to the scanner, but with this
patch not all of them are pushed down. For these tests I added some
extra predicates to achieve that all of the predicates are pushed
down to the scanner.
- Added a new planner test suite for checking how predicate push down
works with Iceberg tables.
Change-Id: Icfa80ce469cecfcfbcd0dcb595a6b04b7027285b
Reviewed-on: http://gerrit.cloudera.org:8080/19534
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com >
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com >
2023-04-21 15:22:17 +00:00
2019-08-13 18:55:09 +00:00
2022-12-14 22:37:14 +00:00
2022-12-14 22:37:14 +00:00
2019-07-27 13:45:51 +00:00
2022-12-14 22:37:14 +00:00
2020-08-12 17:45:50 +00:00
2020-05-18 16:51:42 +00:00
2019-07-27 13:45:51 +00:00
2019-11-07 14:30:47 +00:00
2019-07-03 20:46:45 +00:00
2020-05-20 21:00:44 +00:00
2020-05-20 21:00:44 +00:00
2020-05-20 21:00:44 +00:00
2022-12-14 22:37:14 +00:00
2022-12-14 22:37:14 +00:00
2022-09-28 05:46:03 +00:00
2021-03-12 14:23:04 +00:00
2021-03-12 14:23:04 +00:00
2016-09-13 21:57:36 +00:00
2022-06-19 04:35:04 +00:00
2019-10-16 02:04:12 +00:00
2020-07-08 22:59:57 +00:00
2021-12-16 11:49:15 +00:00
2018-09-29 11:59:03 +00:00
2022-12-14 22:37:14 +00:00
2020-05-12 23:29:04 +00:00
2022-12-14 22:37:14 +00:00
2022-04-07 16:07:56 +00:00
2018-08-08 20:51:45 +00:00
2022-01-09 15:39:35 +00:00
2021-10-23 00:08:14 +00:00
2022-02-23 20:09:10 +00:00
2019-09-27 17:18:35 +00:00
2019-09-27 17:18:35 +00:00
2018-05-15 22:23:14 +00:00
2022-08-25 15:12:38 +00:00
2021-02-18 14:31:51 +00:00
2021-04-30 23:36:31 +00:00
2018-10-09 00:45:10 +00:00
2018-04-11 02:21:48 +00:00
2018-11-17 01:48:05 +00:00
2019-08-13 18:55:09 +00:00
2022-12-07 21:57:46 +00:00
2021-03-02 04:58:51 +00:00
2022-12-14 22:37:14 +00:00
2022-12-14 22:37:14 +00:00
2021-04-02 21:50:17 +00:00
2019-08-06 16:38:07 +00:00
2022-12-14 22:37:14 +00:00
2022-12-14 22:37:14 +00:00
2022-12-14 22:37:14 +00:00
2018-12-08 00:21:55 +00:00
2022-12-14 22:37:14 +00:00
2021-09-14 21:21:47 +00:00
2022-12-14 22:37:14 +00:00
2022-12-14 22:37:14 +00:00
2021-07-06 18:35:30 +00:00
2020-06-24 03:47:18 +00:00
2021-05-07 14:11:08 +00:00
2020-06-24 03:47:18 +00:00
2022-12-14 22:37:14 +00:00
2022-11-25 05:30:50 +00:00
2022-11-11 17:53:58 +00:00
2021-04-20 22:21:32 +00:00
2021-09-15 13:58:23 +00:00
2021-09-15 13:58:23 +00:00
2021-09-15 13:58:23 +00:00
2021-09-15 13:58:23 +00:00
2020-06-24 16:51:49 +00:00
2020-12-11 19:51:28 +00:00
2020-12-11 19:51:28 +00:00
2020-09-28 14:02:35 +00:00
2020-07-08 19:32:15 +00:00
2020-08-13 18:58:23 +00:00
2018-01-25 04:33:11 +00:00
2019-01-14 09:07:03 +00:00
2019-04-23 13:33:57 +00:00
2021-03-23 22:52:38 +00:00
2018-10-09 00:45:10 +00:00
2022-10-17 21:59:57 +00:00
2022-03-04 16:49:22 +00:00
2018-04-11 02:21:48 +00:00
2018-04-11 02:21:48 +00:00
2021-07-06 18:35:30 +00:00
2023-01-02 01:12:53 +00:00
2020-04-02 12:01:41 +00:00
2020-06-05 17:43:32 +00:00
2019-05-15 22:34:28 +00:00
2018-04-12 22:01:35 +00:00
2018-02-02 01:13:08 +00:00
2020-10-21 05:20:33 +00:00
2020-07-31 17:23:45 +00:00
2018-02-15 10:55:55 +00:00
2018-02-15 10:55:55 +00:00
2021-06-14 17:13:25 +00:00
2022-12-06 21:18:33 +00:00
2022-12-06 21:18:33 +00:00
2022-12-06 21:18:33 +00:00
2022-12-06 21:18:33 +00:00
2023-03-21 20:01:14 +00:00
2020-02-22 02:02:56 +00:00
2020-08-12 17:45:50 +00:00
2020-09-30 21:01:54 +00:00
2021-03-18 19:35:58 +00:00
2022-12-14 22:37:14 +00:00
2018-12-11 02:47:16 +00:00
2023-03-06 13:45:56 +00:00
2023-03-08 19:54:38 +00:00
2023-02-07 20:18:47 +00:00
2022-12-14 22:37:14 +00:00
2020-07-14 03:13:18 +00:00
2019-02-07 23:05:30 +00:00
2020-05-12 23:29:04 +00:00
2020-05-12 23:29:04 +00:00
2022-07-18 19:02:34 +00:00
2022-08-19 13:55:42 +00:00
2019-04-23 13:33:57 +00:00
2020-05-12 23:29:04 +00:00
2017-10-10 01:30:33 +00:00
2017-09-21 17:38:08 +00:00
2022-12-14 22:37:14 +00:00
2022-12-14 22:37:14 +00:00
2018-07-03 23:49:44 +00:00
2019-06-05 14:23:23 +00:00
2021-03-02 04:58:51 +00:00
2017-05-09 01:36:46 +00:00
2016-02-19 00:03:15 -08:00
2022-12-14 22:37:14 +00:00
2020-01-17 18:54:33 +00:00
2019-09-27 17:18:35 +00:00
2020-02-14 04:26:16 +00:00
2020-02-11 15:08:39 +00:00
2023-03-02 16:51:12 +00:00
2022-12-16 17:37:35 +00:00
2022-12-14 22:37:14 +00:00
2022-12-14 22:37:14 +00:00
2022-12-14 22:37:14 +00:00
2022-09-28 22:35:48 +00:00
2023-02-03 17:00:44 +00:00
2022-12-14 22:37:14 +00:00
2022-01-23 20:55:00 +00:00
2023-04-21 15:22:17 +00:00
2022-12-14 22:37:14 +00:00
2022-12-14 22:37:14 +00:00
2023-01-27 12:35:52 +00:00
2023-03-29 20:53:18 +00:00
2022-07-14 13:06:04 +00:00
2022-06-10 13:01:28 +00:00
2021-11-04 17:55:21 +00:00
2022-12-16 17:37:35 +00:00
2022-12-14 22:37:14 +00:00
2023-03-27 21:42:49 +00:00
2022-09-28 22:35:48 +00:00
2021-05-20 19:19:50 +00:00
2022-08-01 13:36:51 +00:00
2022-11-02 22:14:47 +00:00
2022-12-14 22:37:14 +00:00
2023-04-21 15:22:17 +00:00
2023-04-21 15:22:17 +00:00
2020-09-09 13:26:42 +00:00
2023-03-22 16:44:05 +00:00
2023-03-09 22:00:08 +00:00
2020-11-23 21:47:55 +00:00
2022-09-26 15:49:22 +00:00
2022-09-26 15:49:22 +00:00
2021-07-15 15:15:07 +00:00
2022-12-14 22:37:14 +00:00
2023-02-21 14:51:18 +00:00
2023-02-21 14:51:18 +00:00
2023-02-21 14:51:18 +00:00
2022-11-10 00:51:50 +00:00
2022-12-14 22:37:14 +00:00
2022-12-14 22:37:14 +00:00
2022-11-22 15:21:11 +00:00
2018-08-09 19:18:08 +00:00
2021-04-20 22:21:32 +00:00
2022-06-04 03:36:11 +00:00
2020-05-11 19:32:08 +00:00
2020-12-11 19:51:28 +00:00
2020-12-11 19:51:28 +00:00
2022-04-04 20:28:07 +00:00
2019-06-05 11:15:04 +00:00
2020-12-11 19:51:28 +00:00
2020-12-11 19:51:28 +00:00
2020-05-11 19:32:08 +00:00
2018-11-06 00:00:12 +00:00
2020-12-11 19:51:28 +00:00
2020-07-31 17:23:45 +00:00
2016-02-19 00:03:15 -08:00
2023-03-06 13:45:56 +00:00
2023-03-08 19:54:38 +00:00
2019-10-22 04:21:51 +00:00
2022-02-04 17:55:47 +00:00
2023-03-02 16:51:12 +00:00
2023-02-20 16:38:16 +00:00
2023-02-04 07:34:56 +00:00
2023-02-04 07:34:56 +00:00
2023-02-04 07:34:56 +00:00
2023-02-04 07:34:56 +00:00
2018-05-11 22:41:49 +00:00
2023-02-04 07:34:56 +00:00
2018-04-27 21:55:11 +00:00
2023-02-04 07:34:56 +00:00
2023-02-04 07:34:56 +00:00
2023-02-04 07:34:56 +00:00
2023-02-04 07:34:56 +00:00
2017-05-11 20:55:51 +00:00
2017-05-11 20:55:51 +00:00
2018-09-26 06:51:56 +00:00
2023-02-04 07:34:56 +00:00
2019-10-21 20:23:56 +00:00
2017-07-21 21:49:04 +00:00
2020-05-13 22:01:34 +00:00
2018-04-11 02:21:48 +00:00
2021-04-20 22:21:32 +00:00
2020-10-08 22:49:59 +00:00
2023-03-08 19:54:38 +00:00
2022-02-03 09:13:21 +00:00
2023-03-08 19:54:38 +00:00
2020-12-11 19:51:28 +00:00
2018-04-11 02:21:48 +00:00
2020-02-22 02:02:56 +00:00
2023-03-07 13:22:33 +00:00
2020-06-09 16:26:23 +00:00
2019-03-08 04:39:08 +00:00
2021-11-03 03:26:27 +00:00
2022-08-25 16:24:41 +00:00
2023-03-07 19:36:58 +00:00
2022-04-05 13:27:10 +00:00
2022-08-12 19:21:55 +00:00
2016-11-03 11:59:07 +00:00
2017-03-17 19:33:31 +00:00
2016-12-02 01:46:55 +00:00
2022-12-06 21:18:33 +00:00
2016-12-02 01:46:55 +00:00
2021-11-30 20:13:18 +00:00
2019-09-25 16:58:14 +00:00
2022-12-14 22:37:14 +00:00
2022-11-15 09:24:03 +00:00
2023-01-10 20:50:50 +00:00
2023-04-14 13:46:59 +00:00
2018-01-05 20:44:21 +00:00
2020-10-21 20:30:05 +00:00
2017-11-22 22:00:16 +00:00
2023-01-18 17:46:54 +00:00
2022-05-18 23:59:58 +00:00
2022-08-19 13:55:42 +00:00
2016-04-01 05:06:38 +00:00
2018-11-17 01:48:05 +00:00
2022-11-15 09:24:03 +00:00
2022-04-13 12:31:36 +00:00
2019-03-08 04:39:08 +00:00
2020-04-16 15:45:49 +00:00
2019-03-08 04:39:08 +00:00
2019-03-08 04:39:08 +00:00
2022-05-18 23:59:58 +00:00
2021-05-07 14:11:08 +00:00
2022-04-05 13:27:10 +00:00
2022-08-09 06:45:52 +00:00
2020-01-17 18:54:33 +00:00
2019-10-04 18:36:22 +00:00
2018-10-01 13:20:40 +00:00
2018-10-01 13:20:40 +00:00
2021-10-28 20:09:46 +00:00
2022-04-04 05:54:58 +00:00
2021-01-27 17:30:37 +00:00
2017-03-03 01:43:42 +00:00
2021-08-21 14:46:51 +00:00
2021-08-21 14:46:51 +00:00
2022-02-04 21:23:55 +00:00
2022-03-04 16:49:22 +00:00
2022-06-22 17:55:07 +00:00
2019-03-14 22:43:50 +00:00
2018-03-08 04:48:36 +00:00
2018-01-17 04:18:24 +00:00
2021-06-03 06:32:45 +00:00
2021-06-03 06:32:45 +00:00
2022-03-04 16:49:22 +00:00
2018-09-10 16:09:41 +00:00
2018-09-10 16:09:41 +00:00
2016-06-20 15:37:18 -07:00
2016-06-20 15:37:18 -07:00
2020-06-18 21:00:27 +00:00
2021-07-14 12:51:09 +00:00
2021-07-14 12:51:09 +00:00
2018-08-22 18:08:20 +00:00
2017-05-09 15:47:21 +00:00
2022-01-23 20:55:00 +00:00
2019-08-24 00:32:28 +00:00
2019-05-07 00:36:56 +00:00
2021-01-14 19:34:38 +00:00
2018-02-22 00:57:46 +00:00
2022-07-11 19:25:02 +00:00
2021-10-28 20:09:46 +00:00
2018-11-27 02:01:12 +00:00
2019-05-10 11:46:38 +00:00
2019-05-10 11:46:38 +00:00
2022-08-19 19:11:58 +00:00
2019-05-10 11:46:38 +00:00
2022-04-02 03:26:28 +00:00
2020-12-10 19:01:08 +00:00
2021-01-05 23:30:35 +00:00
2018-03-22 02:47:33 +00:00
2022-04-13 06:22:14 +00:00
2020-05-21 18:40:20 +00:00
2018-08-23 15:55:53 +00:00
2016-10-14 05:41:22 +00:00
2020-08-25 15:42:01 +00:00
2022-09-28 22:35:48 +00:00
2022-12-14 22:37:14 +00:00
2019-05-10 12:06:01 +00:00
2020-05-20 23:03:23 +00:00
2020-05-20 23:03:23 +00:00
2022-04-05 13:27:10 +00:00
2022-07-22 04:23:30 +00:00
2023-01-17 02:38:27 +00:00
2018-08-15 04:15:46 +00:00
2022-12-03 00:00:37 +00:00
2020-12-22 06:10:39 +00:00
2017-12-13 10:04:40 +00:00
2023-03-05 22:41:11 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2022-12-06 03:55:42 +00:00
2023-04-14 13:46:59 +00:00
2023-04-14 13:46:59 +00:00
2023-02-01 04:24:00 +00:00
2022-12-06 03:55:42 +00:00
2019-08-06 02:35:54 +00:00
2022-04-05 13:27:10 +00:00
2017-11-17 21:33:51 +00:00
2022-04-05 13:27:10 +00:00
2021-03-02 04:58:51 +00:00
2020-04-14 15:42:13 +00:00
2022-02-09 21:20:23 +00:00
2018-11-17 01:48:05 +00:00
2022-12-05 17:54:44 +00:00
2021-03-14 03:35:40 +00:00
2020-07-08 19:32:15 +00:00
2019-01-09 05:26:50 +00:00
2016-08-31 03:12:30 +00:00
2022-11-11 22:02:16 +00:00
2022-11-21 18:51:30 +00:00
2020-04-02 12:01:41 +00:00
2019-09-26 18:35:06 +00:00
2022-11-25 05:30:50 +00:00
2016-09-29 21:14:13 +00:00
2022-12-14 22:37:14 +00:00
2022-12-14 22:37:14 +00:00
2017-08-05 01:03:02 +00:00
2017-08-05 01:03:02 +00:00
2018-04-28 23:41:39 +00:00
2018-02-02 01:13:08 +00:00
2020-10-21 05:20:33 +00:00
2021-01-06 04:39:56 +00:00
2021-03-02 04:58:51 +00:00
2021-08-25 20:05:47 +00:00
2022-10-17 21:59:57 +00:00
2020-04-24 20:56:58 +00:00
2018-04-28 23:41:39 +00:00
2020-08-25 16:11:20 +00:00
2017-08-29 23:01:10 +00:00
2021-03-02 04:58:51 +00:00
2019-01-22 01:20:31 +00:00
2022-02-24 07:06:50 +00:00
2022-12-14 22:37:14 +00:00
2017-03-03 01:43:42 +00:00
2017-03-03 01:43:42 +00:00
2018-11-01 21:27:13 +00:00
2018-11-01 21:27:13 +00:00
2019-08-22 05:03:03 +00:00
2023-03-07 13:22:33 +00:00
2020-08-25 23:08:40 +00:00
2020-10-21 05:20:33 +00:00
2021-04-20 22:21:32 +00:00
2018-04-11 02:21:48 +00:00
2016-02-28 21:31:37 -08:00
2019-09-27 01:54:39 +00:00
2021-04-20 22:21:32 +00:00
2022-12-14 22:37:14 +00:00
2016-11-09 03:27:12 +00:00
2019-04-23 13:33:57 +00:00
2016-11-09 03:27:12 +00:00
2018-04-11 02:21:48 +00:00
2016-12-08 04:53:53 +00:00
2016-12-08 04:53:53 +00:00
2016-11-09 03:27:12 +00:00
2018-11-20 23:49:11 +00:00
2022-08-19 13:55:42 +00:00
2022-04-13 12:31:36 +00:00
2021-12-09 23:17:20 +00:00
2020-06-15 23:42:12 +00:00
2018-07-03 20:34:27 +00:00
2021-05-14 13:58:04 +00:00
2022-10-11 20:30:50 +00:00
2021-08-15 20:40:39 +00:00
2019-05-24 15:08:19 +00:00
2019-05-24 15:08:19 +00:00
2020-07-14 03:13:18 +00:00
2022-09-27 13:27:41 +00:00
2022-04-14 11:56:20 +00:00
2022-11-15 09:24:03 +00:00
2022-12-08 23:07:08 +00:00
2022-06-08 13:02:52 +00:00
2022-06-08 13:02:52 +00:00
2022-06-08 13:02:52 +00:00
2020-10-21 05:20:33 +00:00
2022-04-05 07:56:39 +00:00
2022-02-17 18:51:06 +00:00
2022-11-15 09:24:03 +00:00
7e0feb4a8e