Csaba Ringhofer 8c5ea90aa5
IMPALA-8836: Support COMPUTE STATS on insert only ACID tables ...
For ACID tables COMPUTE STATS needs to use a new HMS API, as the
old one is rejected by metastore. This API currently has some
counter intuitive parts:
- setPartitionColumnStatistics is used to set table stats, as there
is no similar function exposed by HMS client for tables at the
moment.
- A new writeId is allocated for the stat change, and this needs
a transaction, so a transaction is opened/committed/aborted even
though this doesn't seem necessary. The Hive code seems to use
internal API for this.
- Even though the HMS thrift Table object has a colStats field,
it is only applied during alter_table if there are other changes
like new columns in the tables, so alter_table couldn't be used
to change column stats.
Additional changes:
- DROP STATS is no longer allowed for transactional tables, as it
turned out that there is no transactional version of the old API.
- Remove COLUMN_STATS_ACCURATE table property during COMPUTE STATS
to ensure that Hive does use stats computed by Impala to return
answer queries like SELECT count(*)
- Changed CatalogOpExecutor.updateCatalog() to get the writeIds
earlier. This can mean unnecassary HMS RPC calls if no property
change is needed in the end, but I felt it hard to reason about
what happens if these RPC calls fail at their original location.
TODOs (My plan is to do these in IMPALA-8865):
- Tried to make the MetastoreShim API easier to use by adding a class
to encapsulate thing like txnId and writeId, but it feels rather
half baked and under documented.
A similar class is added in https://gerrit.cloudera.org/#/c/14071/ ,
it would be good to merge them.
- The validWriteIdList of the original SELECT(s) behind COMPUTE
STATS could be used in the HMS API calls, but this would need
more plumbing.
Change-Id: I5c06b4678c1ff75c5aa1586a78afea563e64057f
Reviewed-on: http://gerrit.cloudera.org:8080/14066
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com >
Tested-by: Tim Armstrong <tarmstrong@cloudera.com >
2019-08-16 23:22:53 +00:00
2019-08-13 18:55:09 +00:00
2019-05-27 13:08:45 +00:00
2019-08-16 23:22:53 +00:00
2019-07-27 13:45:51 +00:00
2019-08-15 13:02:55 +00:00
2019-08-16 23:22:53 +00:00
2019-07-27 13:45:51 +00:00
2019-05-29 14:37:18 +00:00
2019-07-03 20:46:45 +00:00
2019-08-13 18:07:21 +00:00
2019-06-21 03:28:43 +00:00
2019-06-19 05:58:10 +00:00
2019-06-19 05:58:10 +00:00
2016-09-13 21:57:36 +00:00
2019-01-31 02:08:31 +00:00
2018-01-24 22:47:18 +00:00
2018-09-29 11:59:03 +00:00
2018-09-29 11:59:03 +00:00
2017-02-14 05:56:33 +00:00
2018-12-08 00:21:55 +00:00
2019-06-03 09:24:38 +00:00
2018-08-08 20:51:45 +00:00
2019-04-23 13:33:57 +00:00
2016-12-07 09:45:11 +00:00
2016-07-14 19:04:43 +00:00
2018-05-15 22:23:14 +00:00
2018-02-13 08:29:03 +00:00
2019-06-26 01:58:10 +00:00
2018-10-09 00:45:10 +00:00
2018-04-11 02:21:48 +00:00
2018-11-17 01:48:05 +00:00
2019-08-13 18:55:09 +00:00
2019-01-22 01:20:31 +00:00
2019-05-02 23:33:32 +00:00
2019-05-02 23:33:32 +00:00
2019-08-06 16:38:07 +00:00
2019-04-23 13:33:57 +00:00
2018-12-08 00:21:55 +00:00
2019-08-15 14:40:04 +00:00
2018-12-08 00:21:55 +00:00
2015-04-21 19:27:50 +00:00
2019-08-15 14:40:04 +00:00
2019-04-16 22:40:13 +00:00
2018-08-17 00:54:28 +00:00
2017-05-12 15:43:30 +00:00
2018-04-11 02:21:48 +00:00
2018-04-11 02:21:48 +00:00
2018-04-11 02:21:48 +00:00
2019-05-05 02:11:55 +00:00
2019-04-23 13:33:57 +00:00
2019-05-15 22:34:28 +00:00
2019-05-07 00:36:56 +00:00
2019-06-27 14:39:46 +00:00
2019-06-27 14:39:46 +00:00
2018-01-25 04:33:11 +00:00
2019-01-14 09:07:03 +00:00
2019-04-23 13:33:57 +00:00
2018-10-09 00:45:10 +00:00
2019-08-14 06:11:45 +00:00
2018-04-11 02:21:48 +00:00
2018-04-11 02:21:48 +00:00
2019-08-07 01:41:15 +00:00
2019-08-07 01:41:15 +00:00
2018-04-11 02:21:48 +00:00
2019-05-15 22:34:28 +00:00
2018-05-18 03:44:46 +00:00
2018-11-01 07:30:58 +00:00
2015-07-16 19:38:17 +00:00
2018-04-12 22:01:35 +00:00
2018-02-02 01:13:08 +00:00
2016-01-27 20:41:45 +00:00
2018-02-15 10:55:55 +00:00
2018-02-15 10:55:55 +00:00
2018-11-21 00:08:25 +00:00
2018-05-12 01:43:37 +00:00
2019-01-12 04:03:26 +00:00
2019-06-21 03:28:43 +00:00
2019-02-26 23:29:56 +00:00
2018-12-12 04:47:41 +00:00
2018-12-11 02:47:16 +00:00
2018-04-11 02:21:48 +00:00
2018-12-11 02:47:16 +00:00
2019-02-07 23:05:30 +00:00
2019-04-23 13:33:57 +00:00
2019-04-23 13:33:57 +00:00
2014-05-08 13:59:00 -07:00
2015-02-23 23:32:41 +00:00
2019-04-23 13:33:57 +00:00
2015-02-23 23:32:41 +00:00
2015-02-23 23:32:41 +00:00
2019-04-23 13:33:57 +00:00
2017-02-22 23:10:34 +00:00
2014-01-08 10:52:36 -08:00
2015-02-23 23:32:41 +00:00
2017-10-10 01:30:33 +00:00
2017-09-21 17:38:08 +00:00
2015-04-21 19:27:50 +00:00
2016-11-15 03:27:36 +00:00
2018-07-03 23:49:44 +00:00
2019-06-05 14:23:23 +00:00
2014-06-24 02:14:27 -07:00
2018-08-16 21:25:00 +00:00
2017-05-09 01:36:46 +00:00
2016-02-19 00:03:15 -08:00
2014-01-08 10:52:14 -08:00
2016-05-12 14:17:59 -07:00
2019-05-07 00:36:56 +00:00
2018-08-09 19:18:08 +00:00
2018-10-09 00:45:10 +00:00
2019-06-21 03:28:43 +00:00
2018-04-11 02:21:48 +00:00
2015-08-22 01:46:26 +00:00
2014-01-08 10:53:51 -08:00
2015-05-27 22:25:12 +00:00
2019-06-05 11:15:04 +00:00
2014-12-02 18:08:09 -08:00
2018-04-11 02:21:48 +00:00
2018-05-11 22:41:49 +00:00
2018-11-06 00:00:12 +00:00
2018-05-11 22:41:49 +00:00
2016-02-19 00:03:15 -08:00
2018-12-27 21:02:57 +00:00
2019-03-19 01:20:13 +00:00
2015-02-23 23:32:41 +00:00
2019-04-23 13:33:57 +00:00
2019-06-06 22:16:27 +00:00
2019-04-13 02:42:32 +00:00
2018-04-11 02:21:48 +00:00
2019-05-03 20:28:48 +00:00
2019-06-06 22:16:27 +00:00
2018-05-11 22:41:49 +00:00
2019-01-24 03:14:35 +00:00
2018-04-27 21:55:11 +00:00
2018-04-11 02:21:48 +00:00
2018-04-11 02:21:48 +00:00
2018-04-11 02:21:48 +00:00
2018-04-11 02:21:48 +00:00
2017-05-11 20:55:51 +00:00
2017-05-11 20:55:51 +00:00
2018-09-26 06:51:56 +00:00
2017-11-29 05:53:15 +00:00
2017-07-21 21:49:04 +00:00
2019-05-05 00:00:31 +00:00
2018-04-11 02:21:48 +00:00
2018-10-09 00:45:10 +00:00
2018-12-27 21:02:57 +00:00
2018-04-11 02:21:48 +00:00
2018-04-11 02:21:48 +00:00
2018-08-03 17:45:25 +00:00
2019-03-08 04:39:08 +00:00
2019-01-14 09:07:03 +00:00
2019-04-23 13:33:57 +00:00
2019-02-12 20:37:45 +00:00
2016-11-03 11:59:07 +00:00
2017-03-17 19:33:31 +00:00
2016-12-02 01:46:55 +00:00
2016-12-02 01:46:55 +00:00
2016-12-02 01:46:55 +00:00
2018-10-24 01:03:38 +00:00
2016-05-31 23:32:11 -07:00
2018-01-05 20:44:21 +00:00
2019-05-10 11:46:38 +00:00
2017-11-22 22:00:16 +00:00
2017-11-08 07:39:02 +00:00
2015-09-02 19:23:54 +00:00
2015-09-02 19:23:54 +00:00
2016-04-01 05:06:38 +00:00
2015-09-02 19:23:54 +00:00
2018-11-17 01:48:05 +00:00
2019-01-22 01:20:31 +00:00
2019-03-08 04:39:08 +00:00
2019-03-08 04:39:08 +00:00
2019-04-09 10:23:13 +00:00
2019-03-08 04:39:08 +00:00
2019-03-08 04:39:08 +00:00
2015-10-07 14:47:40 -07:00
2015-08-22 01:46:26 +00:00
2019-05-07 00:36:56 +00:00
2018-10-01 13:20:40 +00:00
2018-10-01 13:20:40 +00:00
2018-11-17 01:48:05 +00:00
2017-08-19 20:56:10 +00:00
2017-03-03 01:43:42 +00:00
2016-05-12 14:17:48 -07:00
2017-03-09 05:07:44 +00:00
2019-03-14 22:43:50 +00:00
2018-03-08 04:48:36 +00:00
2018-01-17 04:18:24 +00:00
2016-08-25 10:20:36 +00:00
2018-09-10 16:09:41 +00:00
2018-09-10 16:09:41 +00:00
2016-06-20 15:37:18 -07:00
2016-06-20 15:37:18 -07:00
2018-08-02 20:21:12 +00:00
2018-08-22 18:08:20 +00:00
2017-05-09 15:47:21 +00:00
2018-10-27 04:07:51 +00:00
2019-05-07 00:36:56 +00:00
2019-04-09 08:54:23 +00:00
2018-02-22 00:57:46 +00:00
2018-11-27 02:01:12 +00:00
2019-05-10 11:46:38 +00:00
2019-05-10 11:46:38 +00:00
2019-05-10 11:46:38 +00:00
2019-05-10 11:46:38 +00:00
2019-03-14 22:43:50 +00:00
2018-03-22 02:47:33 +00:00
2017-07-22 05:53:06 +00:00
2019-05-07 00:36:56 +00:00
2018-08-23 15:55:53 +00:00
2016-10-14 05:41:22 +00:00
2017-05-25 02:55:36 +00:00
2019-04-12 07:28:26 +00:00
2018-04-11 02:21:48 +00:00
2019-05-10 12:06:01 +00:00
2018-08-15 04:15:46 +00:00
2018-08-15 04:15:46 +00:00
2019-02-07 21:41:52 +00:00
2017-12-13 10:04:40 +00:00
2019-08-06 02:35:54 +00:00
2017-11-17 21:33:51 +00:00
2017-11-28 03:01:39 +00:00
2019-06-21 03:28:43 +00:00
2018-04-28 23:41:39 +00:00
2018-11-17 01:48:05 +00:00
2019-02-12 20:37:45 +00:00
2018-03-13 22:10:18 +00:00
2019-01-09 05:26:50 +00:00
2016-08-31 03:12:30 +00:00
2019-07-31 12:57:09 +00:00
2018-10-06 07:25:49 +00:00
2018-08-13 21:36:16 +00:00
2016-09-29 21:14:13 +00:00
2019-08-06 16:38:07 +00:00
2016-07-07 10:41:29 -07:00
2017-08-05 01:03:02 +00:00
2017-08-05 01:03:02 +00:00
2018-04-28 23:41:39 +00:00
2018-02-02 01:13:08 +00:00
2016-06-02 21:33:08 -07:00
2018-10-30 22:17:49 +00:00
2018-09-26 03:54:49 +00:00
2018-02-02 01:13:08 +00:00
2018-04-28 23:41:39 +00:00
2018-04-28 23:41:39 +00:00
2018-11-19 17:27:13 +00:00
2017-08-29 23:01:10 +00:00
2019-01-22 01:20:31 +00:00
2019-01-22 01:20:31 +00:00
2018-06-05 01:14:35 +00:00
2019-06-21 03:28:43 +00:00
2017-03-03 01:43:42 +00:00
2017-03-03 01:43:42 +00:00
2018-11-01 21:27:13 +00:00
2018-11-01 21:27:13 +00:00
2017-12-06 00:41:11 +00:00
2019-01-22 01:20:31 +00:00
2019-04-23 13:33:57 +00:00
2018-04-11 02:21:48 +00:00
2016-02-28 21:31:37 -08:00
2018-06-19 03:02:49 +00:00
2018-11-05 21:58:45 +00:00
2019-05-21 17:01:43 +00:00
2016-11-09 03:27:12 +00:00
2019-04-23 13:33:57 +00:00
2016-11-09 03:27:12 +00:00
2018-04-11 02:21:48 +00:00
2016-12-08 04:53:53 +00:00
2016-12-08 04:53:53 +00:00
2016-11-09 03:27:12 +00:00
2018-11-20 23:49:11 +00:00
2019-05-15 22:34:28 +00:00
2018-10-09 00:45:10 +00:00
2018-05-18 03:44:46 +00:00
2018-07-03 20:34:27 +00:00
2014-01-08 10:48:09 -08:00
2017-07-19 22:17:13 +00:00
2018-02-02 01:13:08 +00:00
2019-05-24 15:08:19 +00:00
2019-05-24 15:08:19 +00:00
2019-05-24 15:08:19 +00:00
2019-04-23 13:33:57 +00:00
2014-06-20 13:35:10 -07:00
2014-01-08 10:54:01 -08:00
2015-10-07 14:47:40 -07:00
8c5ea90aa5