Zoltan Borok-Nagy 301c3cebad
IMPALA-11591: Avoid calling planFiles() on Iceberg tables ...
Iceberg's planFiles() API is very expensive as it needs to read all
the relevant manifest files. It's especially expensive on object
stores like S3.
When there are no predicates on the table and we are not doing
time travel it's possible to avoid calling planFiles() and do the
scan planning from cached metadata. When none of the predicates are
on partition columns there's little benefit of pushing down predicates
to Iceberg. So with this patch we only push down predicates (and
hence invoke planFiles()) when at least one of the predicates are
on partition columns.
This patch introduces a new class to store content files:
IcebergContentFileStore. It separates data, delete, and "old" content
files. "Old" content files are the ones that are not part of the current
snapshot. We add such data files during time travel. Storing "old"
content files in a separate concurrent hash map also fixes a concurrency
bug in the current code.
Testing:
* executed current e2e tests
* updated predicate push down tests
Change-Id: Iadb883a28602bb68cf4f61e57cdd691605045ac5
Reviewed-on: http://gerrit.cloudera.org:8080/19043
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com >
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com >
2022-11-02 22:14:47 +00:00
2019-08-13 18:55:09 +00:00
2020-09-01 22:27:27 +00:00
2020-05-12 23:29:04 +00:00
2019-07-27 13:45:51 +00:00
2021-01-05 21:31:05 +00:00
2020-08-12 17:45:50 +00:00
2020-05-18 16:51:42 +00:00
2019-07-27 13:45:51 +00:00
2019-11-07 14:30:47 +00:00
2019-07-03 20:46:45 +00:00
2020-05-20 21:00:44 +00:00
2020-05-20 21:00:44 +00:00
2020-05-20 21:00:44 +00:00
2020-09-02 13:29:25 +00:00
2021-11-23 03:24:08 +00:00
2022-09-28 05:46:03 +00:00
2021-03-12 14:23:04 +00:00
2021-03-12 14:23:04 +00:00
2016-09-13 21:57:36 +00:00
2022-06-19 04:35:04 +00:00
2019-10-16 02:04:12 +00:00
2020-07-08 22:59:57 +00:00
2021-12-16 11:49:15 +00:00
2018-09-29 11:59:03 +00:00
2017-02-14 05:56:33 +00:00
2020-05-12 23:29:04 +00:00
2021-06-10 15:46:56 +00:00
2022-04-07 16:07:56 +00:00
2018-08-08 20:51:45 +00:00
2022-01-09 15:39:35 +00:00
2021-10-23 00:08:14 +00:00
2022-02-23 20:09:10 +00:00
2019-09-27 17:18:35 +00:00
2019-09-27 17:18:35 +00:00
2018-05-15 22:23:14 +00:00
2022-08-25 15:12:38 +00:00
2021-02-18 14:31:51 +00:00
2021-04-30 23:36:31 +00:00
2018-10-09 00:45:10 +00:00
2018-04-11 02:21:48 +00:00
2018-11-17 01:48:05 +00:00
2019-08-13 18:55:09 +00:00
2021-03-02 04:58:51 +00:00
2020-05-12 23:29:04 +00:00
2020-05-12 23:29:04 +00:00
2021-04-02 21:50:17 +00:00
2019-08-06 16:38:07 +00:00
2020-05-12 23:29:04 +00:00
2020-05-12 23:29:04 +00:00
2021-01-21 19:35:26 +00:00
2018-12-08 00:21:55 +00:00
2021-09-14 21:21:47 +00:00
2022-04-14 11:56:20 +00:00
2020-08-13 03:10:16 +00:00
2021-07-06 18:35:30 +00:00
2020-06-24 03:47:18 +00:00
2021-05-07 14:11:08 +00:00
2020-06-24 03:47:18 +00:00
2020-12-11 19:51:28 +00:00
2020-12-11 19:51:28 +00:00
2019-05-05 02:11:55 +00:00
2021-04-20 22:21:32 +00:00
2021-09-15 13:58:23 +00:00
2021-09-15 13:58:23 +00:00
2021-09-15 13:58:23 +00:00
2021-09-15 13:58:23 +00:00
2020-06-24 16:51:49 +00:00
2020-12-11 19:51:28 +00:00
2020-12-11 19:51:28 +00:00
2020-09-28 14:02:35 +00:00
2020-07-08 19:32:15 +00:00
2020-08-13 18:58:23 +00:00
2018-01-25 04:33:11 +00:00
2019-01-14 09:07:03 +00:00
2019-04-23 13:33:57 +00:00
2021-03-23 22:52:38 +00:00
2018-10-09 00:45:10 +00:00
2022-10-17 21:59:57 +00:00
2022-03-04 16:49:22 +00:00
2018-04-11 02:21:48 +00:00
2018-04-11 02:21:48 +00:00
2021-07-06 18:35:30 +00:00
2020-04-02 12:01:41 +00:00
2020-06-05 17:43:32 +00:00
2019-05-15 22:34:28 +00:00
2018-04-12 22:01:35 +00:00
2018-02-02 01:13:08 +00:00
2020-10-21 05:20:33 +00:00
2020-07-31 17:23:45 +00:00
2018-02-15 10:55:55 +00:00
2018-02-15 10:55:55 +00:00
2021-06-14 17:13:25 +00:00
2022-02-12 06:09:59 +00:00
2022-02-12 06:09:59 +00:00
2022-10-17 21:59:57 +00:00
2022-02-16 13:28:27 +00:00
2022-02-11 07:01:58 +00:00
2020-02-22 02:02:56 +00:00
2020-08-12 17:45:50 +00:00
2020-09-30 21:01:54 +00:00
2021-03-18 19:35:58 +00:00
2020-07-14 12:53:51 +00:00
2018-12-11 02:47:16 +00:00
2022-08-19 13:55:42 +00:00
2021-03-12 00:00:03 +00:00
2020-07-14 03:13:18 +00:00
2019-02-07 23:05:30 +00:00
2020-05-12 23:29:04 +00:00
2020-05-12 23:29:04 +00:00
2022-07-18 19:02:34 +00:00
2022-08-19 13:55:42 +00:00
2019-04-23 13:33:57 +00:00
2020-05-12 23:29:04 +00:00
2017-10-10 01:30:33 +00:00
2017-09-21 17:38:08 +00:00
2020-12-11 19:51:28 +00:00
2018-07-03 23:49:44 +00:00
2019-06-05 14:23:23 +00:00
2021-03-02 04:58:51 +00:00
2017-05-09 01:36:46 +00:00
2020-01-17 18:54:33 +00:00
2019-09-27 17:18:35 +00:00
2020-02-14 04:26:16 +00:00
2020-02-11 15:08:39 +00:00
2022-10-17 19:37:15 +00:00
2022-04-28 17:48:23 +00:00
2022-11-02 22:14:47 +00:00
2022-10-21 22:06:07 +00:00
2022-09-28 22:35:48 +00:00
2022-09-28 22:35:48 +00:00
2022-09-28 22:35:48 +00:00
2022-01-23 20:55:00 +00:00
2022-11-02 22:14:47 +00:00
2022-09-28 22:35:48 +00:00
2022-11-02 22:14:47 +00:00
2022-07-14 13:06:04 +00:00
2022-06-10 13:01:28 +00:00
2021-11-04 17:55:21 +00:00
2022-09-08 18:03:37 +00:00
2022-11-02 22:14:47 +00:00
2022-09-28 22:35:48 +00:00
2022-09-28 22:35:48 +00:00
2021-05-20 19:19:50 +00:00
2022-08-01 13:36:51 +00:00
2022-11-02 22:14:47 +00:00
2022-08-10 14:34:04 +00:00
2022-11-02 22:14:47 +00:00
2022-06-27 15:29:11 +00:00
2020-09-09 13:26:42 +00:00
2022-09-28 22:35:48 +00:00
2020-11-23 21:47:55 +00:00
2022-09-26 15:49:22 +00:00
2022-09-26 15:49:22 +00:00
2021-07-15 15:15:07 +00:00
2022-11-02 22:14:47 +00:00
2022-10-21 22:06:07 +00:00
2022-10-21 22:06:07 +00:00
2021-09-08 18:38:37 +00:00
2021-09-08 18:38:37 +00:00
2022-03-03 00:21:06 +00:00
2018-08-09 19:18:08 +00:00
2021-04-20 22:21:32 +00:00
2022-06-04 03:36:11 +00:00
2020-05-11 19:32:08 +00:00
2020-12-11 19:51:28 +00:00
2020-12-11 19:51:28 +00:00
2022-04-04 20:28:07 +00:00
2019-06-05 11:15:04 +00:00
2020-12-11 19:51:28 +00:00
2020-12-11 19:51:28 +00:00
2020-05-11 19:32:08 +00:00
2018-11-06 00:00:12 +00:00
2020-12-11 19:51:28 +00:00
2020-07-31 17:23:45 +00:00
2022-08-19 13:55:42 +00:00
2019-10-22 04:21:51 +00:00
2022-02-04 17:55:47 +00:00
2021-06-10 15:46:56 +00:00
2022-07-26 18:21:48 +00:00
2020-04-01 15:48:36 +00:00
2020-04-01 15:48:36 +00:00
2020-11-03 20:05:20 +00:00
2018-05-11 22:41:49 +00:00
2020-06-05 00:40:33 +00:00
2018-04-27 21:55:11 +00:00
2022-07-16 06:06:46 +00:00
2020-03-18 18:05:34 +00:00
2020-04-01 15:48:36 +00:00
2020-04-01 15:48:36 +00:00
2017-05-11 20:55:51 +00:00
2017-05-11 20:55:51 +00:00
2018-09-26 06:51:56 +00:00
2020-04-01 15:48:36 +00:00
2019-10-21 20:23:56 +00:00
2017-07-21 21:49:04 +00:00
2020-05-13 22:01:34 +00:00
2018-04-11 02:21:48 +00:00
2021-04-20 22:21:32 +00:00
2020-10-08 22:49:59 +00:00
2022-08-19 13:55:42 +00:00
2022-02-03 09:13:21 +00:00
2022-08-19 13:55:42 +00:00
2020-12-11 19:51:28 +00:00
2018-04-11 02:21:48 +00:00
2020-02-22 02:02:56 +00:00
2020-06-09 16:26:23 +00:00
2019-03-08 04:39:08 +00:00
2021-11-03 03:26:27 +00:00
2022-08-25 16:24:41 +00:00
2022-04-05 13:27:10 +00:00
2022-08-12 19:21:55 +00:00
2016-11-03 11:59:07 +00:00
2017-03-17 19:33:31 +00:00
2016-12-02 01:46:55 +00:00
2019-11-26 07:28:23 +00:00
2016-12-02 01:46:55 +00:00
2021-11-30 20:13:18 +00:00
2019-09-25 16:58:14 +00:00
2022-09-28 22:35:48 +00:00
2022-09-09 14:42:41 +00:00
2022-09-07 19:55:43 +00:00
2022-06-22 17:55:07 +00:00
2018-01-05 20:44:21 +00:00
2020-10-21 20:30:05 +00:00
2017-11-22 22:00:16 +00:00
2020-07-15 17:10:50 +00:00
2022-05-18 23:59:58 +00:00
2022-08-19 13:55:42 +00:00
2018-11-17 01:48:05 +00:00
2022-09-09 22:43:33 +00:00
2022-04-13 12:31:36 +00:00
2019-03-08 04:39:08 +00:00
2020-04-16 15:45:49 +00:00
2019-03-08 04:39:08 +00:00
2019-03-08 04:39:08 +00:00
2022-05-18 23:59:58 +00:00
2021-05-07 14:11:08 +00:00
2022-04-05 13:27:10 +00:00
2022-08-09 06:45:52 +00:00
2020-01-17 18:54:33 +00:00
2019-10-04 18:36:22 +00:00
2018-10-01 13:20:40 +00:00
2018-10-01 13:20:40 +00:00
2021-10-28 20:09:46 +00:00
2022-04-04 05:54:58 +00:00
2021-01-27 17:30:37 +00:00
2017-03-03 01:43:42 +00:00
2021-08-21 14:46:51 +00:00
2021-08-21 14:46:51 +00:00
2022-02-04 21:23:55 +00:00
2022-03-04 16:49:22 +00:00
2022-06-22 17:55:07 +00:00
2019-03-14 22:43:50 +00:00
2018-03-08 04:48:36 +00:00
2018-01-17 04:18:24 +00:00
2021-06-03 06:32:45 +00:00
2021-06-03 06:32:45 +00:00
2022-03-04 16:49:22 +00:00
2018-09-10 16:09:41 +00:00
2018-09-10 16:09:41 +00:00
2020-06-18 21:00:27 +00:00
2021-07-14 12:51:09 +00:00
2021-07-14 12:51:09 +00:00
2018-08-22 18:08:20 +00:00
2017-05-09 15:47:21 +00:00
2022-01-23 20:55:00 +00:00
2019-08-24 00:32:28 +00:00
2019-05-07 00:36:56 +00:00
2021-01-14 19:34:38 +00:00
2018-02-22 00:57:46 +00:00
2022-07-11 19:25:02 +00:00
2021-10-28 20:09:46 +00:00
2018-11-27 02:01:12 +00:00
2019-05-10 11:46:38 +00:00
2019-05-10 11:46:38 +00:00
2022-08-19 19:11:58 +00:00
2019-05-10 11:46:38 +00:00
2022-04-02 03:26:28 +00:00
2020-12-10 19:01:08 +00:00
2021-01-05 23:30:35 +00:00
2018-03-22 02:47:33 +00:00
2022-04-13 06:22:14 +00:00
2020-05-21 18:40:20 +00:00
2018-08-23 15:55:53 +00:00
2016-10-14 05:41:22 +00:00
2020-08-25 15:42:01 +00:00
2022-09-28 22:35:48 +00:00
2018-04-11 02:21:48 +00:00
2019-05-10 12:06:01 +00:00
2020-05-20 23:03:23 +00:00
2020-05-20 23:03:23 +00:00
2022-04-05 13:27:10 +00:00
2022-07-22 04:23:30 +00:00
2021-02-10 23:52:28 +00:00
2018-08-15 04:15:46 +00:00
2018-08-15 04:15:46 +00:00
2020-12-22 06:10:39 +00:00
2017-12-13 10:04:40 +00:00
2021-02-13 23:02:54 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2021-03-31 05:38:50 +00:00
2022-09-07 19:55:43 +00:00
2022-09-07 19:55:43 +00:00
2022-06-08 13:02:52 +00:00
2021-03-31 05:38:50 +00:00
2019-08-06 02:35:54 +00:00
2022-04-05 13:27:10 +00:00
2017-11-17 21:33:51 +00:00
2022-04-05 13:27:10 +00:00
2021-03-02 04:58:51 +00:00
2020-04-14 15:42:13 +00:00
2022-02-09 21:20:23 +00:00
2018-11-17 01:48:05 +00:00
2022-04-05 13:27:10 +00:00
2021-03-14 03:35:40 +00:00
2020-07-08 19:32:15 +00:00
2019-01-09 05:26:50 +00:00
2016-08-31 03:12:30 +00:00
2022-07-11 19:25:02 +00:00
2022-06-20 13:52:37 +00:00
2020-04-02 12:01:41 +00:00
2019-09-26 18:35:06 +00:00
2022-10-17 19:37:15 +00:00
2016-09-29 21:14:13 +00:00
2020-05-12 23:29:04 +00:00
2020-04-14 12:18:21 +00:00
2017-08-05 01:03:02 +00:00
2017-08-05 01:03:02 +00:00
2018-04-28 23:41:39 +00:00
2018-02-02 01:13:08 +00:00
2020-10-21 05:20:33 +00:00
2021-01-06 04:39:56 +00:00
2021-03-02 04:58:51 +00:00
2021-08-25 20:05:47 +00:00
2022-10-17 21:59:57 +00:00
2020-04-24 20:56:58 +00:00
2018-04-28 23:41:39 +00:00
2020-08-25 16:11:20 +00:00
2017-08-29 23:01:10 +00:00
2021-03-02 04:58:51 +00:00
2019-01-22 01:20:31 +00:00
2022-02-24 07:06:50 +00:00
2021-05-15 22:20:58 +00:00
2017-03-03 01:43:42 +00:00
2017-03-03 01:43:42 +00:00
2018-11-01 21:27:13 +00:00
2018-11-01 21:27:13 +00:00
2019-08-22 05:03:03 +00:00
2022-09-07 19:55:43 +00:00
2020-08-25 23:08:40 +00:00
2020-10-21 05:20:33 +00:00
2021-04-20 22:21:32 +00:00
2018-04-11 02:21:48 +00:00
2019-09-27 01:54:39 +00:00
2021-04-20 22:21:32 +00:00
2021-05-28 04:02:10 +00:00
2016-11-09 03:27:12 +00:00
2019-04-23 13:33:57 +00:00
2016-11-09 03:27:12 +00:00
2018-04-11 02:21:48 +00:00
2016-12-08 04:53:53 +00:00
2016-12-08 04:53:53 +00:00
2016-11-09 03:27:12 +00:00
2018-11-20 23:49:11 +00:00
2022-08-19 13:55:42 +00:00
2022-04-13 12:31:36 +00:00
2021-12-09 23:17:20 +00:00
2020-06-15 23:42:12 +00:00
2018-07-03 20:34:27 +00:00
2021-05-14 13:58:04 +00:00
2022-10-11 20:30:50 +00:00
2021-08-15 20:40:39 +00:00
2019-05-24 15:08:19 +00:00
2019-05-24 15:08:19 +00:00
2020-07-14 03:13:18 +00:00
2022-09-27 13:27:41 +00:00
2022-04-14 11:56:20 +00:00
2022-08-30 16:45:24 +00:00
2022-08-12 19:21:55 +00:00
2022-06-08 13:02:52 +00:00
2022-06-08 13:02:52 +00:00
2022-06-08 13:02:52 +00:00
2020-10-21 05:20:33 +00:00
2022-04-05 07:56:39 +00:00
2022-02-17 18:51:06 +00:00
2022-04-05 07:56:39 +00:00
301c3cebad