Gergely Fürnstáhl d0fe4c604f
IMPALA-11619: Improve Iceberg V2 reads with a custom Iceberg Position Delete operator ...
IcebergDeleteNode and IcebergDeleteBuild classes are based on
PartitionedHashJoin counterparts. The actual "join" part of the node is
optimized, while others are kept very similarly, to be able to integrate
features of PartitionedHashJoin if needed (partitioning, spilling).
ICEBERG_DELETE_JOIN is added as a join operator which is used only by
IcebergDeleteNode node.
IcebergDeleteBuild processes the data from the relevant delete files and
stores them in a {file_path: ordered row id vector} hash map.
IcebergDeleteNode tracks the processed file and progresses through the
row id vector parallel to the probe batch to check if a row is deleted
or hashes the probe row's file path and uses binary search to find the
closest row id if it is needed for the check.
Testing:
- Duplicated related planner tests to run both with new operator and
hash join
- Added a dimension for e2e tests to run both with new operator and
hash join
- Added new multiblock tests to verify assumptions used in new
operator to optimize probing
- Added new test with BATCH_SIZE=2 to verify in/out batch handling
with new operator
Change-Id: I024a61573c83bda5584f243c879d9ff39dd2dcfa
Reviewed-on: http://gerrit.cloudera.org:8080/19850
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com >
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com >
2023-07-05 20:32:23 +00:00
2019-08-13 18:55:09 +00:00
2022-12-14 22:37:14 +00:00
2022-12-14 22:37:14 +00:00
2019-07-27 13:45:51 +00:00
2022-12-14 22:37:14 +00:00
2020-08-12 17:45:50 +00:00
2020-05-18 16:51:42 +00:00
2019-07-27 13:45:51 +00:00
2019-11-07 14:30:47 +00:00
2019-07-03 20:46:45 +00:00
2020-05-20 21:00:44 +00:00
2020-05-20 21:00:44 +00:00
2020-05-20 21:00:44 +00:00
2022-12-14 22:37:14 +00:00
2022-12-14 22:37:14 +00:00
2022-09-28 05:46:03 +00:00
2021-03-12 14:23:04 +00:00
2021-03-12 14:23:04 +00:00
2016-09-13 21:57:36 +00:00
2022-06-19 04:35:04 +00:00
2019-10-16 02:04:12 +00:00
2020-07-08 22:59:57 +00:00
2021-12-16 11:49:15 +00:00
2018-09-29 11:59:03 +00:00
2022-12-14 22:37:14 +00:00
2020-05-12 23:29:04 +00:00
2022-12-14 22:37:14 +00:00
2022-04-07 16:07:56 +00:00
2018-08-08 20:51:45 +00:00
2023-06-06 19:06:24 +00:00
2021-10-23 00:08:14 +00:00
2022-02-23 20:09:10 +00:00
2019-09-27 17:18:35 +00:00
2019-09-27 17:18:35 +00:00
2018-05-15 22:23:14 +00:00
2022-08-25 15:12:38 +00:00
2021-02-18 14:31:51 +00:00
2021-04-30 23:36:31 +00:00
2018-10-09 00:45:10 +00:00
2018-04-11 02:21:48 +00:00
2018-11-17 01:48:05 +00:00
2019-08-13 18:55:09 +00:00
2022-12-07 21:57:46 +00:00
2021-03-02 04:58:51 +00:00
2022-12-14 22:37:14 +00:00
2022-12-14 22:37:14 +00:00
2021-04-02 21:50:17 +00:00
2019-08-06 16:38:07 +00:00
2022-12-14 22:37:14 +00:00
2022-12-14 22:37:14 +00:00
2022-12-14 22:37:14 +00:00
2018-12-08 00:21:55 +00:00
2022-12-14 22:37:14 +00:00
2021-09-14 21:21:47 +00:00
2022-12-14 22:37:14 +00:00
2022-12-14 22:37:14 +00:00
2021-07-06 18:35:30 +00:00
2020-06-24 03:47:18 +00:00
2021-05-07 14:11:08 +00:00
2020-06-24 03:47:18 +00:00
2022-12-14 22:37:14 +00:00
2022-11-25 05:30:50 +00:00
2022-11-11 17:53:58 +00:00
2021-04-20 22:21:32 +00:00
2021-09-15 13:58:23 +00:00
2021-09-15 13:58:23 +00:00
2021-09-15 13:58:23 +00:00
2021-09-15 13:58:23 +00:00
2020-06-24 16:51:49 +00:00
2020-12-11 19:51:28 +00:00
2020-12-11 19:51:28 +00:00
2020-09-28 14:02:35 +00:00
2020-07-08 19:32:15 +00:00
2020-08-13 18:58:23 +00:00
2018-01-25 04:33:11 +00:00
2019-01-14 09:07:03 +00:00
2019-04-23 13:33:57 +00:00
2021-03-23 22:52:38 +00:00
2018-10-09 00:45:10 +00:00
2022-10-17 21:59:57 +00:00
2022-03-04 16:49:22 +00:00
2018-04-11 02:21:48 +00:00
2018-04-11 02:21:48 +00:00
2021-07-06 18:35:30 +00:00
2023-01-02 01:12:53 +00:00
2020-04-02 12:01:41 +00:00
2020-06-05 17:43:32 +00:00
2019-05-15 22:34:28 +00:00
2018-04-12 22:01:35 +00:00
2018-02-02 01:13:08 +00:00
2020-10-21 05:20:33 +00:00
2020-07-31 17:23:45 +00:00
2018-02-15 10:55:55 +00:00
2018-02-15 10:55:55 +00:00
2021-06-14 17:13:25 +00:00
2022-12-06 21:18:33 +00:00
2022-12-06 21:18:33 +00:00
2022-12-06 21:18:33 +00:00
2022-12-06 21:18:33 +00:00
2023-03-21 20:01:14 +00:00
2020-02-22 02:02:56 +00:00
2020-08-12 17:45:50 +00:00
2020-09-30 21:01:54 +00:00
2021-03-18 19:35:58 +00:00
2022-12-14 22:37:14 +00:00
2018-12-11 02:47:16 +00:00
2023-03-06 13:45:56 +00:00
2023-03-08 19:54:38 +00:00
2023-02-07 20:18:47 +00:00
2022-12-14 22:37:14 +00:00
2020-07-14 03:13:18 +00:00
2019-02-07 23:05:30 +00:00
2020-05-12 23:29:04 +00:00
2020-05-12 23:29:04 +00:00
2022-07-18 19:02:34 +00:00
2022-08-19 13:55:42 +00:00
2019-04-23 13:33:57 +00:00
2020-05-12 23:29:04 +00:00
2017-10-10 01:30:33 +00:00
2017-09-21 17:38:08 +00:00
2022-12-14 22:37:14 +00:00
2022-12-14 22:37:14 +00:00
2018-07-03 23:49:44 +00:00
2019-06-05 14:23:23 +00:00
2021-03-02 04:58:51 +00:00
2017-05-09 01:36:46 +00:00
2016-02-19 00:03:15 -08:00
2022-12-14 22:37:14 +00:00
2020-01-17 18:54:33 +00:00
2019-09-27 17:18:35 +00:00
2020-02-14 04:26:16 +00:00
2020-02-11 15:08:39 +00:00
2023-03-02 16:51:12 +00:00
2022-12-16 17:37:35 +00:00
2022-12-14 22:37:14 +00:00
2023-05-09 00:05:36 +00:00
2022-12-14 22:37:14 +00:00
2022-09-28 22:35:48 +00:00
2023-02-03 17:00:44 +00:00
2022-12-14 22:37:14 +00:00
2023-06-13 11:33:32 +00:00
2023-07-05 16:52:28 +00:00
2023-06-13 11:33:32 +00:00
2022-01-23 20:55:00 +00:00
2023-05-09 00:05:36 +00:00
2022-12-14 22:37:14 +00:00
2023-05-09 00:05:36 +00:00
2023-01-27 12:35:52 +00:00
2023-03-29 20:53:18 +00:00
2023-07-03 08:17:41 +00:00
2022-07-14 13:06:04 +00:00
2022-06-10 13:01:28 +00:00
2021-11-04 17:55:21 +00:00
2022-12-16 17:37:35 +00:00
2022-12-14 22:37:14 +00:00
2023-06-29 18:11:50 +00:00
2022-09-28 22:35:48 +00:00
2021-05-20 19:19:50 +00:00
2022-08-01 13:36:51 +00:00
2022-11-02 22:14:47 +00:00
2022-12-14 22:37:14 +00:00
2023-05-09 00:05:36 +00:00
2023-05-09 00:05:36 +00:00
2020-09-09 13:26:42 +00:00
2023-03-22 16:44:05 +00:00
2023-03-09 22:00:08 +00:00
2020-11-23 21:47:55 +00:00
2022-09-26 15:49:22 +00:00
2022-09-26 15:49:22 +00:00
2023-06-16 22:32:11 +00:00
2021-07-15 15:15:07 +00:00
2023-05-09 00:05:36 +00:00
2023-05-09 00:05:36 +00:00
2023-05-09 00:05:36 +00:00
2023-07-05 20:32:23 +00:00
2022-11-10 00:51:50 +00:00
2023-06-29 18:11:50 +00:00
2022-12-14 22:37:14 +00:00
2022-12-14 22:37:14 +00:00
2022-11-22 15:21:11 +00:00
2018-08-09 19:18:08 +00:00
2021-04-20 22:21:32 +00:00
2022-06-04 03:36:11 +00:00
2020-05-11 19:32:08 +00:00
2020-12-11 19:51:28 +00:00
2020-12-11 19:51:28 +00:00
2022-04-04 20:28:07 +00:00
2019-06-05 11:15:04 +00:00
2020-12-11 19:51:28 +00:00
2020-12-11 19:51:28 +00:00
2020-05-11 19:32:08 +00:00
2018-11-06 00:00:12 +00:00
2023-06-29 18:45:38 +00:00
2020-12-11 19:51:28 +00:00
2020-07-31 17:23:45 +00:00
2016-02-19 00:03:15 -08:00
2023-03-06 13:45:56 +00:00
2023-06-20 17:00:35 +00:00
2019-10-22 04:21:51 +00:00
2022-02-04 17:55:47 +00:00
2023-03-02 16:51:12 +00:00
2023-02-20 16:38:16 +00:00
2023-05-02 02:47:46 +00:00
2023-02-04 07:34:56 +00:00
2023-02-04 07:34:56 +00:00
2023-02-04 07:34:56 +00:00
2018-05-11 22:41:49 +00:00
2023-02-04 07:34:56 +00:00
2018-04-27 21:55:11 +00:00
2023-02-04 07:34:56 +00:00
2023-02-04 07:34:56 +00:00
2023-02-04 07:34:56 +00:00
2023-02-04 07:34:56 +00:00
2017-05-11 20:55:51 +00:00
2017-05-11 20:55:51 +00:00
2018-09-26 06:51:56 +00:00
2023-02-04 07:34:56 +00:00
2019-10-21 20:23:56 +00:00
2017-07-21 21:49:04 +00:00
2020-05-13 22:01:34 +00:00
2018-04-11 02:21:48 +00:00
2021-04-20 22:21:32 +00:00
2023-06-30 23:27:31 +00:00
2023-03-08 19:54:38 +00:00
2022-02-03 09:13:21 +00:00
2023-06-20 17:00:35 +00:00
2020-12-11 19:51:28 +00:00
2018-04-11 02:21:48 +00:00
2020-02-22 02:02:56 +00:00
2023-03-07 13:22:33 +00:00
2020-06-09 16:26:23 +00:00
2019-03-08 04:39:08 +00:00
2021-11-03 03:26:27 +00:00
2022-08-25 16:24:41 +00:00
2023-05-18 09:56:55 +00:00
2023-05-06 22:55:05 +00:00
2022-08-12 19:21:55 +00:00
2016-11-03 11:59:07 +00:00
2017-03-17 19:33:31 +00:00
2016-12-02 01:46:55 +00:00
2023-05-03 03:16:29 +00:00
2016-12-02 01:46:55 +00:00
2021-11-30 20:13:18 +00:00
2019-09-25 16:58:14 +00:00
2022-12-14 22:37:14 +00:00
2023-05-19 22:11:03 +00:00
2023-05-19 22:11:03 +00:00
2023-04-14 13:46:59 +00:00
2018-01-05 20:44:21 +00:00
2020-10-21 20:30:05 +00:00
2017-11-22 22:00:16 +00:00
2023-01-18 17:46:54 +00:00
2022-05-18 23:59:58 +00:00
2022-08-19 13:55:42 +00:00
2016-04-01 05:06:38 +00:00
2018-11-17 01:48:05 +00:00
2022-11-15 09:24:03 +00:00
2023-06-13 07:05:41 +00:00
2019-03-08 04:39:08 +00:00
2020-04-16 15:45:49 +00:00
2019-03-08 04:39:08 +00:00
2019-03-08 04:39:08 +00:00
2022-05-18 23:59:58 +00:00
2021-05-07 14:11:08 +00:00
2023-06-20 10:24:33 +00:00
2020-01-17 18:54:33 +00:00
2019-10-04 18:36:22 +00:00
2018-10-01 13:20:40 +00:00
2018-10-01 13:20:40 +00:00
2021-10-28 20:09:46 +00:00
2022-04-04 05:54:58 +00:00
2023-04-22 23:24:10 +00:00
2017-03-03 01:43:42 +00:00
2021-08-21 14:46:51 +00:00
2021-08-21 14:46:51 +00:00
2022-02-04 21:23:55 +00:00
2022-03-04 16:49:22 +00:00
2022-06-22 17:55:07 +00:00
2019-03-14 22:43:50 +00:00
2018-03-08 04:48:36 +00:00
2018-01-17 04:18:24 +00:00
2021-06-03 06:32:45 +00:00
2021-06-03 06:32:45 +00:00
2022-03-04 16:49:22 +00:00
2018-09-10 16:09:41 +00:00
2018-09-10 16:09:41 +00:00
2016-06-20 15:37:18 -07:00
2016-06-20 15:37:18 -07:00
2020-06-18 21:00:27 +00:00
2021-07-14 12:51:09 +00:00
2021-07-14 12:51:09 +00:00
2018-08-22 18:08:20 +00:00
2017-05-09 15:47:21 +00:00
2022-01-23 20:55:00 +00:00
2019-08-24 00:32:28 +00:00
2019-05-07 00:36:56 +00:00
2021-01-14 19:34:38 +00:00
2018-02-22 00:57:46 +00:00
2022-07-11 19:25:02 +00:00
2021-10-28 20:09:46 +00:00
2018-11-27 02:01:12 +00:00
2019-05-10 11:46:38 +00:00
2019-05-10 11:46:38 +00:00
2022-08-19 19:11:58 +00:00
2019-05-10 11:46:38 +00:00
2022-04-02 03:26:28 +00:00
2020-12-10 19:01:08 +00:00
2021-01-05 23:30:35 +00:00
2018-03-22 02:47:33 +00:00
2023-05-06 22:55:05 +00:00
2020-05-21 18:40:20 +00:00
2018-08-23 15:55:53 +00:00
2023-05-12 18:31:03 +00:00
2016-10-14 05:41:22 +00:00
2020-08-25 15:42:01 +00:00
2022-09-28 22:35:48 +00:00
2022-12-14 22:37:14 +00:00
2019-05-10 12:06:01 +00:00
2020-05-20 23:03:23 +00:00
2020-05-20 23:03:23 +00:00
2023-05-06 22:55:05 +00:00
2022-07-22 04:23:30 +00:00
2023-05-18 09:56:55 +00:00
2023-06-20 09:09:17 +00:00
2018-08-15 04:15:46 +00:00
2022-12-03 00:00:37 +00:00
2020-12-22 06:10:39 +00:00
2017-12-13 10:04:40 +00:00
2023-03-05 22:41:11 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2022-12-06 03:55:42 +00:00
2023-04-14 13:46:59 +00:00
2023-04-14 13:46:59 +00:00
2023-06-28 08:42:06 +00:00
2023-06-28 08:42:06 +00:00
2019-08-06 02:35:54 +00:00
2023-05-06 22:55:05 +00:00
2017-11-17 21:33:51 +00:00
2023-05-06 22:55:05 +00:00
2021-03-02 04:58:51 +00:00
2020-04-14 15:42:13 +00:00
2022-02-09 21:20:23 +00:00
2018-11-17 01:48:05 +00:00
2023-05-06 22:55:05 +00:00
2021-03-14 03:35:40 +00:00
2020-07-08 19:32:15 +00:00
2019-01-09 05:26:50 +00:00
2016-08-31 03:12:30 +00:00
2022-11-11 22:02:16 +00:00
2022-11-21 18:51:30 +00:00
2020-04-02 12:01:41 +00:00
2019-09-26 18:35:06 +00:00
2022-11-25 05:30:50 +00:00
2016-09-29 21:14:13 +00:00
2022-12-14 22:37:14 +00:00
2022-12-14 22:37:14 +00:00
2017-08-05 01:03:02 +00:00
2017-08-05 01:03:02 +00:00
2018-04-28 23:41:39 +00:00
2018-02-02 01:13:08 +00:00
2023-05-18 09:56:55 +00:00
2020-10-21 05:20:33 +00:00
2021-01-06 04:39:56 +00:00
2021-03-02 04:58:51 +00:00
2021-08-25 20:05:47 +00:00
2022-10-17 21:59:57 +00:00
2020-04-24 20:56:58 +00:00
2018-04-28 23:41:39 +00:00
2020-08-25 16:11:20 +00:00
2017-08-29 23:01:10 +00:00
2021-03-02 04:58:51 +00:00
2019-01-22 01:20:31 +00:00
2022-02-24 07:06:50 +00:00
2022-12-14 22:37:14 +00:00
2017-03-03 01:43:42 +00:00
2017-03-03 01:43:42 +00:00
2018-11-01 21:27:13 +00:00
2018-11-01 21:27:13 +00:00
2019-08-22 05:03:03 +00:00
2023-03-07 13:22:33 +00:00
2023-05-24 01:18:56 +00:00
2020-08-25 23:08:40 +00:00
2020-10-21 05:20:33 +00:00
2021-04-20 22:21:32 +00:00
2018-04-11 02:21:48 +00:00
2016-02-28 21:31:37 -08:00
2019-09-27 01:54:39 +00:00
2023-05-18 09:56:55 +00:00
2021-04-20 22:21:32 +00:00
2022-12-14 22:37:14 +00:00
2016-11-09 03:27:12 +00:00
2019-04-23 13:33:57 +00:00
2016-11-09 03:27:12 +00:00
2018-04-11 02:21:48 +00:00
2016-12-08 04:53:53 +00:00
2016-12-08 04:53:53 +00:00
2016-11-09 03:27:12 +00:00
2018-11-20 23:49:11 +00:00
2022-08-19 13:55:42 +00:00
2023-06-13 07:05:41 +00:00
2021-12-09 23:17:20 +00:00
2020-06-15 23:42:12 +00:00
2018-07-03 20:34:27 +00:00
2021-05-14 13:58:04 +00:00
2022-10-11 20:30:50 +00:00
2021-08-15 20:40:39 +00:00
2019-05-24 15:08:19 +00:00
2019-05-24 15:08:19 +00:00
2020-07-14 03:13:18 +00:00
2022-09-27 13:27:41 +00:00
2022-04-14 11:56:20 +00:00
2022-11-15 09:24:03 +00:00
2022-12-08 23:07:08 +00:00
2022-06-08 13:02:52 +00:00
2022-06-08 13:02:52 +00:00
2022-06-08 13:02:52 +00:00
2020-10-21 05:20:33 +00:00
2022-04-05 07:56:39 +00:00
2022-02-17 18:51:06 +00:00
2022-11-15 09:24:03 +00:00
d0fe4c604f