Daniel Becker c5b474d3f5
IMPALA-13594: Read Puffin stats also from older snapshots ...
Before this change, Puffin stats were only read from the current
snapshot. Now we also consider older snapshots, and for each column we
choose the most recent available stats. Note that this means that the
stats for different columns may come from different snapshots.
In case there are both HMS and Puffin stats for a column, the more
recent one will be used - for HMS stats we use the
'impala.lastComputeStatsTime' table property, and for Puffin stats we
use the snapshot timestamp to determine which is more recent.
This commit also renames the startup flag 'disable_reading_puffin_stats'
to 'enable_reading_puffin_stats' and the table property
'impala.iceberg_disable_reading_puffin_stats' to
'impala.iceberg_read_puffin_stats' to make them more intuitive. The
default values are flipped to keep the same behaviour as before.
The documentation of Puffin reading is updated in
docs/topics/impala_iceberg.xml
Testing:
- updated existing test cases and added new ones in
test_iceberg_with_puffin.py
- reorganised the tests in TestIcebergTableWithPuffinStats in
test_iceberg_with_puffin.py: tests that modify table properties and
other state that other tests rely on are now run separately to
provide a clean environment for all tests.
Change-Id: Ia37abe8c9eab6d91946c8f6d3df5fb0889704a39
Reviewed-on: http://gerrit.cloudera.org:8080/22177
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com >
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com >
2025-01-23 15:25:59 +00:00
2023-10-18 10:22:05 +00:00
2021-06-02 19:34:15 +00:00
2017-03-01 22:49:32 +00:00
2024-12-11 02:18:18 +00:00
2024-05-23 03:54:00 +00:00
2024-09-24 17:10:34 +00:00
2018-10-10 18:18:09 +00:00
2024-10-23 05:27:45 +00:00
2018-03-21 17:50:35 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2021-06-02 19:34:15 +00:00
2024-03-11 10:10:43 +00:00
2022-10-18 00:13:15 +00:00
2018-10-10 18:18:09 +00:00
2023-10-18 10:22:05 +00:00
2017-07-11 00:24:36 +00:00
2017-02-17 01:10:12 +00:00
2019-10-31 23:02:49 +00:00
2019-08-12 17:42:42 +00:00
2024-08-05 16:19:45 +00:00
2017-01-18 19:31:57 +00:00
2019-10-03 18:43:38 +00:00
2023-10-18 10:22:05 +00:00
2019-02-28 20:58:26 +00:00
2018-10-10 18:18:09 +00:00
2017-02-17 01:10:12 +00:00
2018-05-03 20:45:28 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2018-07-24 00:35:48 +00:00
2024-12-11 02:18:18 +00:00
2018-09-04 17:44:21 +00:00
2017-01-18 19:31:57 +00:00
2024-08-13 21:38:30 +00:00
2024-08-07 02:31:36 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-11-14 01:15:34 +00:00
2019-02-11 20:19:58 +00:00
2018-10-26 02:17:07 +00:00
2023-10-18 10:22:05 +00:00
2017-03-14 16:48:20 +00:00
2019-08-05 21:31:33 +00:00
2019-12-10 22:54:26 +00:00
2025-01-10 07:13:34 +00:00
2018-10-30 23:55:25 +00:00
2017-01-18 19:31:57 +00:00
2017-03-10 23:56:51 +00:00
2019-06-25 19:44:43 +00:00
2021-06-02 19:34:15 +00:00
2024-03-11 10:10:43 +00:00
2022-10-18 00:13:15 +00:00
2019-04-02 01:26:24 +00:00
2023-11-06 18:57:18 +00:00
2018-10-15 19:35:51 +00:00
2019-11-13 19:00:47 +00:00
2020-05-05 17:44:40 +00:00
2020-11-25 13:52:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2018-05-01 22:10:42 +00:00
2019-04-02 01:26:24 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2021-11-24 12:16:27 +00:00
2023-10-18 10:22:05 +00:00
2023-08-17 15:44:06 +00:00
2018-04-12 20:19:38 +00:00
2017-01-18 19:31:57 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-03-02 17:25:55 +00:00
2018-10-30 23:55:25 +00:00
2017-07-11 00:26:13 +00:00
2017-01-18 19:31:57 +00:00
2017-03-10 23:56:51 +00:00
2017-03-08 22:40:19 +00:00
2021-06-02 19:34:15 +00:00
2017-03-21 22:01:36 +00:00
2017-03-10 23:56:51 +00:00
2017-01-18 19:31:57 +00:00
2023-10-18 10:22:05 +00:00
2024-08-08 22:07:48 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2019-12-09 19:19:22 +00:00
2019-03-13 21:26:10 +00:00
2017-11-16 22:31:38 +00:00
2020-12-01 22:46:31 +00:00
2023-10-18 10:22:05 +00:00
2022-02-03 03:13:41 +00:00
2023-03-10 22:23:18 +00:00
2017-07-11 00:26:13 +00:00
2024-11-20 15:03:06 +00:00
2024-11-20 15:03:06 +00:00
2021-06-02 19:34:15 +00:00
2017-05-30 19:01:20 +00:00
2024-08-01 02:36:25 +00:00
2024-08-01 02:36:25 +00:00
2024-08-01 02:36:25 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2019-12-11 20:41:29 +00:00
2018-10-05 19:36:21 +00:00
2020-03-12 18:49:31 +00:00
2025-01-23 15:25:59 +00:00
2024-01-30 14:00:31 +00:00
2023-10-18 10:22:05 +00:00
2019-02-11 20:19:58 +00:00
2021-07-12 13:36:54 +00:00
2021-06-02 19:34:15 +00:00
2019-01-18 00:46:24 +00:00
2017-11-27 19:20:43 +00:00
2021-06-02 19:34:15 +00:00
2019-04-02 01:26:24 +00:00
2024-06-20 18:05:29 +00:00
2019-12-10 22:17:19 +00:00
2023-10-18 10:22:05 +00:00
2019-02-14 22:28:53 +00:00
2019-04-02 01:26:24 +00:00
2020-03-17 20:16:58 +00:00
2023-10-18 10:22:05 +00:00
2023-03-31 23:29:23 +00:00
2021-06-02 19:34:15 +00:00
2024-09-24 17:10:34 +00:00
2022-08-19 14:40:56 +00:00
2021-06-02 19:34:15 +00:00
2019-08-12 17:40:48 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-01-09 12:04:46 +00:00
2021-06-02 19:34:15 +00:00
2017-02-17 01:10:12 +00:00
2024-03-08 16:34:50 +00:00
2017-01-18 19:31:57 +00:00
2024-05-10 11:20:41 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2017-01-18 19:31:57 +00:00
2024-05-23 03:54:00 +00:00
2024-10-09 11:32:18 +00:00
2022-01-26 23:50:14 +00:00
2023-10-18 10:22:05 +00:00
2017-01-18 19:31:57 +00:00
2019-07-23 21:40:59 +00:00
2024-09-25 12:14:52 +00:00
2021-04-01 23:35:48 +00:00
2021-07-12 13:36:54 +00:00
2020-12-01 22:46:31 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2017-03-17 00:00:21 +00:00
2019-09-16 20:57:08 +00:00
2023-10-18 10:22:05 +00:00
2022-02-03 03:13:41 +00:00
2019-02-11 20:19:58 +00:00
2023-01-30 21:50:32 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2019-10-03 22:39:24 +00:00
2018-06-14 20:35:24 +00:00
2018-06-18 17:39:15 +00:00
2018-05-21 23:41:49 +00:00
2018-05-30 17:50:04 +00:00
2017-03-14 17:23:08 +00:00
2017-03-09 23:27:53 +00:00
2020-03-24 21:14:31 +00:00
2017-03-09 23:27:53 +00:00
2017-01-18 19:31:57 +00:00
2017-01-18 19:31:57 +00:00
2018-08-08 20:33:08 +00:00
2019-10-18 18:09:22 +00:00
2023-10-18 10:22:05 +00:00
2018-12-14 19:27:49 +00:00
2019-02-12 22:07:12 +00:00
2019-12-14 02:40:27 +00:00
2019-10-04 18:03:21 +00:00
2023-10-18 10:22:05 +00:00
2024-08-13 21:38:30 +00:00
2017-01-18 19:31:57 +00:00
2021-06-02 19:34:15 +00:00
2018-06-08 18:18:05 +00:00
2023-10-18 10:22:05 +00:00
2021-06-02 19:34:15 +00:00
2017-03-14 17:29:57 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-11-24 01:14:11 +00:00
2019-01-09 20:58:08 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2021-06-02 19:34:15 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2024-08-09 17:49:03 +00:00
2024-08-09 17:49:03 +00:00
2023-10-18 10:22:05 +00:00
2020-01-13 20:02:16 +00:00
2021-07-17 01:35:28 +00:00
2021-06-02 19:34:15 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2017-03-09 23:27:53 +00:00
2017-01-18 19:31:57 +00:00
2023-10-18 10:22:05 +00:00
2021-06-02 19:34:15 +00:00
2017-03-06 19:08:31 +00:00
2017-02-09 22:19:44 +00:00
2017-03-13 23:49:56 +00:00
2021-06-02 19:34:15 +00:00
2019-09-12 18:58:26 +00:00
2023-10-18 10:22:05 +00:00
2018-11-02 00:02:58 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2022-12-14 15:49:30 +00:00
2018-07-09 20:01:08 +00:00
2024-12-20 06:20:37 +00:00
2023-10-18 10:22:05 +00:00
2019-02-21 23:40:39 +00:00
2023-10-18 10:22:05 +00:00
2023-01-31 00:32:54 +00:00
2017-01-18 19:31:57 +00:00
2024-12-02 15:15:15 +00:00
2022-04-21 22:24:49 +00:00
2017-02-17 01:10:12 +00:00
2017-10-21 00:06:38 +00:00
2017-01-18 19:31:57 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2023-01-30 21:50:32 +00:00
2019-02-07 20:03:07 +00:00
2023-10-18 10:22:05 +00:00
2023-10-18 10:22:05 +00:00
2024-04-03 03:25:10 +00:00
2020-09-14 16:40:25 +00:00
2018-09-25 19:42:59 +00:00
2023-10-18 10:22:05 +00:00
2019-09-17 17:52:47 +00:00
2020-12-01 22:46:31 +00:00
2017-01-18 19:31:57 +00:00
2017-01-18 19:31:57 +00:00
2017-01-18 19:31:57 +00:00
2017-01-18 19:31:57 +00:00
2017-01-18 19:31:57 +00:00
2019-02-12 22:07:12 +00:00
2018-10-09 19:18:10 +00:00
2024-12-20 06:20:37 +00:00
2020-03-17 20:16:58 +00:00
2020-05-08 04:18:26 +00:00
2018-01-18 21:53:40 +00:00
2024-08-13 21:38:30 +00:00
2024-01-02 10:49:13 +00:00
2019-07-19 22:10:58 +00:00
2018-04-04 00:55:57 +00:00
2024-06-12 12:27:05 +00:00
2017-01-18 19:31:57 +00:00
2023-10-18 10:22:05 +00:00
2022-08-23 22:34:42 +00:00
2019-11-13 19:00:47 +00:00
2017-03-08 22:40:19 +00:00
2017-01-18 19:31:57 +00:00
2017-10-21 00:06:38 +00:00
2024-02-23 21:01:08 +00:00
2017-11-16 22:31:38 +00:00
c5b474d3f5