Zoltan Borok-Nagy 981ef10465
IMPALA-10215: Implement INSERT INTO for non-partitioned Iceberg tables (Parquet) ...
This commit adds support for INSERT INTO statements against Iceberg
tables when the table is non-partitioned and the underlying file format
is Parquet.
We still use Impala's HdfsParquetTableWriter to write the data files,
though they needed some modifications to conform to the Iceberg spec,
namely:
* write Iceberg/Parquet 'field_id' for the columns
* TIMESTAMPs are encoded as INT64 micros (without time zone)
We use DmlExecState to transfer information from the table sink
operators to the coordinator, then updateCatalog() invokes the
AppendFiles API to add files atomically. DmlExecState is encoded in
protobuf, communication with the Frontend uses Thrift. Therefore to
avoid defining Iceberg DataFile multiple times they are stored in
FlatBuffers.
The commit also does some corrections on Impala type <-> Iceberg type
mapping:
* Impala TIMESTAMP is Iceberg TIMESTAMP (without time zone)
* Impala CHAR is Iceberg FIXED
Testing:
* Added INSERT tests to iceberg-insert.test
* Added negative tests to iceberg-negative.test
* I also did some manual testing with Spark. Spark is able to read
Iceberg tables written by Impala until we use TIMESTAMPs. In that
case Spark rejects the data files because it only accepts TIMESTAMPS
with time zone.
* Added concurrent INSERT tests to test_insert_stress.py
Change-Id: I5690fb6c2cc51f0033fa26caf8597c80a11bcd8e
Reviewed-on: http://gerrit.cloudera.org:8080/16545
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com >
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com >
2020-10-26 20:01:09 +00:00
2019-08-13 18:55:09 +00:00
2020-09-01 22:27:27 +00:00
2020-05-12 23:29:04 +00:00
2019-07-27 13:45:51 +00:00
2020-09-19 20:10:48 +00:00
2020-08-12 17:45:50 +00:00
2020-05-18 16:51:42 +00:00
2019-07-27 13:45:51 +00:00
2019-11-07 14:30:47 +00:00
2019-07-03 20:46:45 +00:00
2020-05-20 21:00:44 +00:00
2020-05-20 21:00:44 +00:00
2020-05-20 21:00:44 +00:00
2020-09-02 13:29:25 +00:00
2020-05-20 21:00:44 +00:00
2020-06-10 04:02:16 +00:00
2020-06-10 04:02:16 +00:00
2020-06-10 04:02:16 +00:00
2016-09-13 21:57:36 +00:00
2019-11-26 19:14:24 +00:00
2019-10-16 02:04:12 +00:00
2020-07-08 22:59:57 +00:00
2020-08-13 02:22:31 +00:00
2018-09-29 11:59:03 +00:00
2017-02-14 05:56:33 +00:00
2020-05-12 23:29:04 +00:00
2020-06-24 03:47:18 +00:00
2018-08-08 20:51:45 +00:00
2020-10-21 05:20:33 +00:00
2020-03-31 19:11:20 +00:00
2019-09-27 17:18:35 +00:00
2019-09-27 17:18:35 +00:00
2018-05-15 22:23:14 +00:00
2020-09-26 11:11:47 +00:00
2019-09-19 18:46:19 +00:00
2018-10-09 00:45:10 +00:00
2018-04-11 02:21:48 +00:00
2018-11-17 01:48:05 +00:00
2019-08-13 18:55:09 +00:00
2019-01-22 01:20:31 +00:00
2020-05-12 23:29:04 +00:00
2020-05-12 23:29:04 +00:00
2019-08-06 16:38:07 +00:00
2020-05-12 23:29:04 +00:00
2020-05-12 23:29:04 +00:00
2020-05-12 23:29:04 +00:00
2018-12-08 00:21:55 +00:00
2015-04-21 19:27:50 +00:00
2020-05-12 23:29:04 +00:00
2020-08-13 03:10:16 +00:00
2020-10-07 18:32:51 +00:00
2020-06-24 03:47:18 +00:00
2020-09-12 05:30:45 +00:00
2020-06-24 03:47:18 +00:00
2020-06-24 03:47:18 +00:00
2020-06-24 03:47:18 +00:00
2019-05-05 02:11:55 +00:00
2019-04-23 13:33:57 +00:00
2020-09-03 12:11:48 +00:00
2020-09-02 10:49:10 +00:00
2020-06-24 16:51:49 +00:00
2020-01-17 18:54:33 +00:00
2020-09-28 14:02:35 +00:00
2020-09-28 14:02:35 +00:00
2020-07-08 19:32:15 +00:00
2020-08-13 18:58:23 +00:00
2018-01-25 04:33:11 +00:00
2019-01-14 09:07:03 +00:00
2019-04-23 13:33:57 +00:00
2018-10-09 00:45:10 +00:00
2019-08-14 06:11:45 +00:00
2018-04-11 02:21:48 +00:00
2018-04-11 02:21:48 +00:00
2020-10-07 18:32:51 +00:00
2020-10-07 18:32:51 +00:00
2020-04-02 12:01:41 +00:00
2020-06-05 17:43:32 +00:00
2019-05-15 22:34:28 +00:00
2018-11-01 07:30:58 +00:00
2015-07-16 19:38:17 +00:00
2018-04-12 22:01:35 +00:00
2018-02-02 01:13:08 +00:00
2020-10-21 05:20:33 +00:00
2020-07-31 17:23:45 +00:00
2018-02-15 10:55:55 +00:00
2018-02-15 10:55:55 +00:00
2018-11-21 00:08:25 +00:00
2018-05-12 01:43:37 +00:00
2019-01-12 04:03:26 +00:00
2019-09-25 01:06:19 +00:00
2019-09-25 01:06:19 +00:00
2020-08-13 02:22:31 +00:00
2020-02-22 02:02:56 +00:00
2020-08-12 17:45:50 +00:00
2020-09-30 21:01:54 +00:00
2020-08-12 17:45:50 +00:00
2020-07-14 12:53:51 +00:00
2018-12-11 02:47:16 +00:00
2020-07-14 03:13:18 +00:00
2019-02-07 23:05:30 +00:00
2020-05-12 23:29:04 +00:00
2020-05-12 23:29:04 +00:00
2014-05-08 13:59:00 -07:00
2015-02-23 23:32:41 +00:00
2019-04-23 13:33:57 +00:00
2015-02-23 23:32:41 +00:00
2015-02-23 23:32:41 +00:00
2019-04-23 13:33:57 +00:00
2020-05-12 23:29:04 +00:00
2015-02-23 23:32:41 +00:00
2017-10-10 01:30:33 +00:00
2017-09-21 17:38:08 +00:00
2015-04-21 19:27:50 +00:00
2019-10-18 08:15:00 +00:00
2018-07-03 23:49:44 +00:00
2019-06-05 14:23:23 +00:00
2014-06-24 02:14:27 -07:00
2018-08-16 21:25:00 +00:00
2017-05-09 01:36:46 +00:00
2016-02-19 00:03:15 -08:00
2016-05-12 14:17:59 -07:00
2020-01-17 18:54:33 +00:00
2019-09-27 17:18:35 +00:00
2020-02-14 04:26:16 +00:00
2020-02-11 15:08:39 +00:00
2020-10-14 19:19:19 +00:00
2020-10-26 20:01:09 +00:00
2020-10-26 20:01:09 +00:00
2020-09-09 13:26:42 +00:00
2020-10-14 19:19:19 +00:00
2018-08-09 19:18:08 +00:00
2020-10-21 05:20:33 +00:00
2020-10-21 05:20:33 +00:00
2020-05-11 19:32:08 +00:00
2020-05-11 19:32:08 +00:00
2020-05-11 19:32:08 +00:00
2015-05-27 22:25:12 +00:00
2019-06-05 11:15:04 +00:00
2014-12-02 18:08:09 -08:00
2020-04-14 12:18:21 +00:00
2020-05-11 19:32:08 +00:00
2018-11-06 00:00:12 +00:00
2020-10-20 06:47:25 +00:00
2020-07-31 17:23:45 +00:00
2016-02-19 00:03:15 -08:00
2020-09-04 00:55:02 +00:00
2019-10-22 04:21:51 +00:00
2015-02-23 23:32:41 +00:00
2020-02-05 22:00:48 +00:00
2020-10-23 04:01:26 +00:00
2020-04-01 15:48:36 +00:00
2020-04-01 15:48:36 +00:00
2020-04-01 15:48:36 +00:00
2020-10-23 04:01:26 +00:00
2018-05-11 22:41:49 +00:00
2020-06-05 00:40:33 +00:00
2018-04-27 21:55:11 +00:00
2020-03-18 18:05:34 +00:00
2020-03-18 18:05:34 +00:00
2020-04-01 15:48:36 +00:00
2020-04-01 15:48:36 +00:00
2017-05-11 20:55:51 +00:00
2017-05-11 20:55:51 +00:00
2018-09-26 06:51:56 +00:00
2020-04-01 15:48:36 +00:00
2019-10-21 20:23:56 +00:00
2017-07-21 21:49:04 +00:00
2020-05-13 22:01:34 +00:00
2018-04-11 02:21:48 +00:00
2020-10-21 05:20:33 +00:00
2020-10-08 22:49:59 +00:00
2018-12-27 21:02:57 +00:00
2020-04-14 12:18:21 +00:00
2018-04-11 02:21:48 +00:00
2020-02-22 02:02:56 +00:00
2020-06-09 16:26:23 +00:00
2019-03-08 04:39:08 +00:00
2020-07-08 22:59:57 +00:00
2020-10-21 05:20:33 +00:00
2019-02-12 20:37:45 +00:00
2020-03-31 20:45:59 +00:00
2016-11-03 11:59:07 +00:00
2017-03-17 19:33:31 +00:00
2016-12-02 01:46:55 +00:00
2019-11-26 07:28:23 +00:00
2016-12-02 01:46:55 +00:00
2016-12-02 01:46:55 +00:00
2019-09-25 16:58:14 +00:00
2016-05-31 23:32:11 -07:00
2018-01-05 20:44:21 +00:00
2020-10-21 20:30:05 +00:00
2017-11-22 22:00:16 +00:00
2020-07-15 17:10:50 +00:00
2015-09-02 19:23:54 +00:00
2015-09-02 19:23:54 +00:00
2016-04-01 05:06:38 +00:00
2015-09-02 19:23:54 +00:00
2018-11-17 01:48:05 +00:00
2019-01-22 01:20:31 +00:00
2019-03-08 04:39:08 +00:00
2020-04-16 15:45:49 +00:00
2019-03-08 04:39:08 +00:00
2019-03-08 04:39:08 +00:00
2019-03-08 04:39:08 +00:00
2015-10-07 14:47:40 -07:00
2015-08-22 01:46:26 +00:00
2020-01-17 18:54:33 +00:00
2019-10-04 18:36:22 +00:00
2018-10-01 13:20:40 +00:00
2018-10-01 13:20:40 +00:00
2020-02-22 02:02:56 +00:00
2017-08-19 20:56:10 +00:00
2020-09-22 22:19:41 +00:00
2017-03-03 01:43:42 +00:00
2016-05-12 14:17:48 -07:00
2017-03-09 05:07:44 +00:00
2019-03-14 22:43:50 +00:00
2018-03-08 04:48:36 +00:00
2018-01-17 04:18:24 +00:00
2016-08-25 10:20:36 +00:00
2018-09-10 16:09:41 +00:00
2018-09-10 16:09:41 +00:00
2016-06-20 15:37:18 -07:00
2016-06-20 15:37:18 -07:00
2020-06-18 21:00:27 +00:00
2018-08-22 18:08:20 +00:00
2017-05-09 15:47:21 +00:00
2019-08-24 00:32:28 +00:00
2019-05-07 00:36:56 +00:00
2019-04-09 08:54:23 +00:00
2018-02-22 00:57:46 +00:00
2018-11-27 02:01:12 +00:00
2019-05-10 11:46:38 +00:00
2019-05-10 11:46:38 +00:00
2019-05-10 11:46:38 +00:00
2020-10-07 13:41:34 +00:00
2019-10-05 00:04:08 +00:00
2018-03-22 02:47:33 +00:00
2017-07-22 05:53:06 +00:00
2020-05-21 18:40:20 +00:00
2018-08-23 15:55:53 +00:00
2016-10-14 05:41:22 +00:00
2020-08-25 15:42:01 +00:00
2019-04-12 07:28:26 +00:00
2018-04-11 02:21:48 +00:00
2019-05-10 12:06:01 +00:00
2020-05-20 23:03:23 +00:00
2020-05-20 23:03:23 +00:00
2020-05-20 23:03:23 +00:00
2018-08-15 04:15:46 +00:00
2018-08-15 04:15:46 +00:00
2019-02-07 21:41:52 +00:00
2017-12-13 10:04:40 +00:00
2020-09-02 22:57:55 +00:00
2020-04-07 15:10:27 +00:00
2020-04-02 12:01:41 +00:00
2019-08-06 02:35:54 +00:00
2020-05-14 02:52:16 +00:00
2017-11-17 21:33:51 +00:00
2020-06-05 17:43:32 +00:00
2020-04-14 15:42:13 +00:00
2020-04-14 15:42:13 +00:00
2020-05-15 21:46:38 +00:00
2018-11-17 01:48:05 +00:00
2019-02-12 20:37:45 +00:00
2020-07-08 19:32:15 +00:00
2019-01-09 05:26:50 +00:00
2016-08-31 03:12:30 +00:00
2020-06-18 21:56:32 +00:00
2020-04-02 12:01:41 +00:00
2019-09-26 18:35:06 +00:00
2020-10-14 19:19:19 +00:00
2016-09-29 21:14:13 +00:00
2020-05-12 23:29:04 +00:00
2020-04-14 12:18:21 +00:00
2017-08-05 01:03:02 +00:00
2017-08-05 01:03:02 +00:00
2018-04-28 23:41:39 +00:00
2018-02-02 01:13:08 +00:00
2020-10-21 05:20:33 +00:00
2018-10-30 22:17:49 +00:00
2020-03-03 06:12:24 +00:00
2020-03-17 23:29:45 +00:00
2020-08-25 16:11:20 +00:00
2020-04-24 20:56:58 +00:00
2018-04-28 23:41:39 +00:00
2020-08-25 16:11:20 +00:00
2017-08-29 23:01:10 +00:00
2019-01-22 01:20:31 +00:00
2019-01-22 01:20:31 +00:00
2020-05-15 16:49:32 +00:00
2020-08-13 03:10:16 +00:00
2017-03-03 01:43:42 +00:00
2017-03-03 01:43:42 +00:00
2018-11-01 21:27:13 +00:00
2018-11-01 21:27:13 +00:00
2019-08-22 05:03:03 +00:00
2020-08-25 23:08:40 +00:00
2020-10-21 05:20:33 +00:00
2020-10-21 05:20:33 +00:00
2018-04-11 02:21:48 +00:00
2016-02-28 21:31:37 -08:00
2019-09-27 01:54:39 +00:00
2020-10-21 05:20:33 +00:00
2020-05-12 23:29:04 +00:00
2016-11-09 03:27:12 +00:00
2019-04-23 13:33:57 +00:00
2016-11-09 03:27:12 +00:00
2018-04-11 02:21:48 +00:00
2016-12-08 04:53:53 +00:00
2016-12-08 04:53:53 +00:00
2016-11-09 03:27:12 +00:00
2018-11-20 23:49:11 +00:00
2019-05-15 22:34:28 +00:00
2020-10-21 05:20:33 +00:00
2020-06-15 23:42:12 +00:00
2018-07-03 20:34:27 +00:00
2020-09-18 22:12:47 +00:00
2018-02-02 01:13:08 +00:00
2019-05-24 15:08:19 +00:00
2019-05-24 15:08:19 +00:00
2020-07-14 03:13:18 +00:00
2019-04-23 13:33:57 +00:00
2014-06-20 13:35:10 -07:00
2014-01-08 10:54:01 -08:00
2020-10-21 05:20:33 +00:00
981ef10465