Aman Sinha 5530b62539
IMPALA-10314: Optimize planning time for simple limits ...
This patch optimizes the planning time for simple limit
queries by only considering a minimal set of partitions
whose file descriptors add up to N (the specified limit).
Each file is conservatively estimated to contain 1 row.
This reduces the number of partitions processed by
HdfsScanNode.computeScanRangeLocations() which, according
to query profiling, has been the main contributor to the
planning time especially for large number of partitions.
Further, within each partition, we only consider the number
of non-empty files that brings the total to N.
This is an opt-in optimization. A new planner option
OPTIMIZE_SIMPLE_LIMIT enables this optimization. Further,
if there's a WHERE clause, it must have an 'always_true'
hint in order for the optimization to be considered. For
example:
set optimize_simple_limit = true;
SELECT * FROM T
WHERE /* +always_true */ <predicate>
LIMIT 10;
If there are too many empty files in the partitions, it is
possible that the query may produce fewer rows although
those are still valid rows.
Testing:
- Added planner tests for the optimization
- Ran query_test.py tests by enabling the optimize_simple_limit
- Added an e2e test. Since result rows are non-deterministic,
only simple count(*) query on top of subquery with limit
was added.
Change-Id: I9d6a79263bc092e0f3e9a1d72da5618f3cc35574
Reviewed-on: http://gerrit.cloudera.org:8080/16723
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com >
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com >
2020-11-28 07:30:06 +00:00
2019-08-13 18:55:09 +00:00
2020-09-01 22:27:27 +00:00
2020-05-12 23:29:04 +00:00
2019-07-27 13:45:51 +00:00
2020-09-19 20:10:48 +00:00
2020-08-12 17:45:50 +00:00
2020-05-18 16:51:42 +00:00
2019-07-27 13:45:51 +00:00
2019-11-07 14:30:47 +00:00
2019-07-03 20:46:45 +00:00
2020-05-20 21:00:44 +00:00
2020-05-20 21:00:44 +00:00
2020-05-20 21:00:44 +00:00
2020-09-02 13:29:25 +00:00
2020-05-20 21:00:44 +00:00
2020-06-10 04:02:16 +00:00
2020-06-10 04:02:16 +00:00
2020-06-10 04:02:16 +00:00
2016-09-13 21:57:36 +00:00
2019-11-26 19:14:24 +00:00
2019-10-16 02:04:12 +00:00
2020-07-08 22:59:57 +00:00
2020-08-13 02:22:31 +00:00
2018-09-29 11:59:03 +00:00
2017-02-14 05:56:33 +00:00
2020-05-12 23:29:04 +00:00
2020-06-24 03:47:18 +00:00
2018-08-08 20:51:45 +00:00
2020-10-21 05:20:33 +00:00
2020-03-31 19:11:20 +00:00
2019-09-27 17:18:35 +00:00
2019-09-27 17:18:35 +00:00
2018-05-15 22:23:14 +00:00
2020-09-26 11:11:47 +00:00
2019-09-19 18:46:19 +00:00
2018-10-09 00:45:10 +00:00
2018-04-11 02:21:48 +00:00
2018-11-17 01:48:05 +00:00
2019-08-13 18:55:09 +00:00
2019-01-22 01:20:31 +00:00
2020-05-12 23:29:04 +00:00
2020-05-12 23:29:04 +00:00
2019-08-06 16:38:07 +00:00
2020-05-12 23:29:04 +00:00
2020-05-12 23:29:04 +00:00
2020-05-12 23:29:04 +00:00
2018-12-08 00:21:55 +00:00
2015-04-21 19:27:50 +00:00
2020-05-12 23:29:04 +00:00
2020-08-13 03:10:16 +00:00
2020-11-11 01:39:01 +00:00
2020-06-24 03:47:18 +00:00
2020-09-12 05:30:45 +00:00
2020-06-24 03:47:18 +00:00
2020-06-24 03:47:18 +00:00
2020-06-24 03:47:18 +00:00
2019-05-05 02:11:55 +00:00
2019-04-23 13:33:57 +00:00
2020-11-25 14:33:10 +00:00
2020-09-02 10:49:10 +00:00
2020-06-24 16:51:49 +00:00
2020-01-17 18:54:33 +00:00
2020-09-28 14:02:35 +00:00
2020-09-28 14:02:35 +00:00
2020-07-08 19:32:15 +00:00
2020-08-13 18:58:23 +00:00
2018-01-25 04:33:11 +00:00
2019-01-14 09:07:03 +00:00
2019-04-23 13:33:57 +00:00
2018-10-09 00:45:10 +00:00
2019-08-14 06:11:45 +00:00
2018-04-11 02:21:48 +00:00
2018-04-11 02:21:48 +00:00
2020-11-11 01:39:01 +00:00
2020-11-11 01:39:01 +00:00
2020-04-02 12:01:41 +00:00
2020-06-05 17:43:32 +00:00
2019-05-15 22:34:28 +00:00
2018-11-01 07:30:58 +00:00
2015-07-16 19:38:17 +00:00
2018-04-12 22:01:35 +00:00
2018-02-02 01:13:08 +00:00
2020-10-21 05:20:33 +00:00
2020-07-31 17:23:45 +00:00
2018-02-15 10:55:55 +00:00
2018-02-15 10:55:55 +00:00
2018-11-21 00:08:25 +00:00
2018-05-12 01:43:37 +00:00
2019-01-12 04:03:26 +00:00
2019-09-25 01:06:19 +00:00
2020-11-20 17:54:08 +00:00
2020-08-13 02:22:31 +00:00
2020-02-22 02:02:56 +00:00
2020-08-12 17:45:50 +00:00
2020-09-30 21:01:54 +00:00
2020-08-12 17:45:50 +00:00
2020-07-14 12:53:51 +00:00
2018-12-11 02:47:16 +00:00
2020-07-14 03:13:18 +00:00
2019-02-07 23:05:30 +00:00
2020-05-12 23:29:04 +00:00
2020-05-12 23:29:04 +00:00
2014-05-08 13:59:00 -07:00
2015-02-23 23:32:41 +00:00
2019-04-23 13:33:57 +00:00
2015-02-23 23:32:41 +00:00
2015-02-23 23:32:41 +00:00
2019-04-23 13:33:57 +00:00
2020-05-12 23:29:04 +00:00
2015-02-23 23:32:41 +00:00
2017-10-10 01:30:33 +00:00
2017-09-21 17:38:08 +00:00
2015-04-21 19:27:50 +00:00
2019-10-18 08:15:00 +00:00
2018-07-03 23:49:44 +00:00
2019-06-05 14:23:23 +00:00
2014-06-24 02:14:27 -07:00
2018-08-16 21:25:00 +00:00
2017-05-09 01:36:46 +00:00
2016-02-19 00:03:15 -08:00
2016-05-12 14:17:59 -07:00
2020-01-17 18:54:33 +00:00
2019-09-27 17:18:35 +00:00
2020-02-14 04:26:16 +00:00
2020-02-11 15:08:39 +00:00
2020-11-20 21:40:28 +00:00
2020-11-20 21:40:28 +00:00
2020-11-20 21:40:28 +00:00
2020-11-23 21:47:55 +00:00
2020-09-09 13:26:42 +00:00
2020-10-14 19:19:19 +00:00
2020-11-23 21:47:55 +00:00
2018-08-09 19:18:08 +00:00
2020-10-21 05:20:33 +00:00
2020-10-21 05:20:33 +00:00
2020-05-11 19:32:08 +00:00
2020-05-11 19:32:08 +00:00
2020-05-11 19:32:08 +00:00
2015-05-27 22:25:12 +00:00
2019-06-05 11:15:04 +00:00
2014-12-02 18:08:09 -08:00
2020-04-14 12:18:21 +00:00
2020-05-11 19:32:08 +00:00
2018-11-06 00:00:12 +00:00
2020-10-20 06:47:25 +00:00
2020-07-31 17:23:45 +00:00
2016-02-19 00:03:15 -08:00
2020-09-04 00:55:02 +00:00
2019-10-22 04:21:51 +00:00
2015-02-23 23:32:41 +00:00
2020-02-05 22:00:48 +00:00
2020-11-03 20:05:20 +00:00
2020-04-01 15:48:36 +00:00
2020-04-01 15:48:36 +00:00
2020-04-01 15:48:36 +00:00
2020-11-03 20:05:20 +00:00
2018-05-11 22:41:49 +00:00
2020-06-05 00:40:33 +00:00
2018-04-27 21:55:11 +00:00
2020-03-18 18:05:34 +00:00
2020-03-18 18:05:34 +00:00
2020-04-01 15:48:36 +00:00
2020-04-01 15:48:36 +00:00
2017-05-11 20:55:51 +00:00
2017-05-11 20:55:51 +00:00
2018-09-26 06:51:56 +00:00
2020-04-01 15:48:36 +00:00
2019-10-21 20:23:56 +00:00
2017-07-21 21:49:04 +00:00
2020-05-13 22:01:34 +00:00
2018-04-11 02:21:48 +00:00
2020-10-21 05:20:33 +00:00
2020-10-08 22:49:59 +00:00
2018-12-27 21:02:57 +00:00
2020-04-14 12:18:21 +00:00
2018-04-11 02:21:48 +00:00
2020-02-22 02:02:56 +00:00
2020-06-09 16:26:23 +00:00
2019-03-08 04:39:08 +00:00
2020-07-08 22:59:57 +00:00
2020-10-21 05:20:33 +00:00
2019-02-12 20:37:45 +00:00
2016-11-03 11:59:07 +00:00
2017-03-17 19:33:31 +00:00
2016-12-02 01:46:55 +00:00
2019-11-26 07:28:23 +00:00
2016-12-02 01:46:55 +00:00
2016-12-02 01:46:55 +00:00
2019-09-25 16:58:14 +00:00
2016-05-31 23:32:11 -07:00
2018-01-05 20:44:21 +00:00
2020-10-21 20:30:05 +00:00
2017-11-22 22:00:16 +00:00
2020-07-15 17:10:50 +00:00
2015-09-02 19:23:54 +00:00
2015-09-02 19:23:54 +00:00
2016-04-01 05:06:38 +00:00
2015-09-02 19:23:54 +00:00
2018-11-17 01:48:05 +00:00
2019-01-22 01:20:31 +00:00
2019-03-08 04:39:08 +00:00
2020-04-16 15:45:49 +00:00
2019-03-08 04:39:08 +00:00
2019-03-08 04:39:08 +00:00
2019-03-08 04:39:08 +00:00
2015-10-07 14:47:40 -07:00
2015-08-22 01:46:26 +00:00
2020-01-17 18:54:33 +00:00
2019-10-04 18:36:22 +00:00
2018-10-01 13:20:40 +00:00
2018-10-01 13:20:40 +00:00
2020-02-22 02:02:56 +00:00
2017-08-19 20:56:10 +00:00
2020-09-22 22:19:41 +00:00
2017-03-03 01:43:42 +00:00
2016-05-12 14:17:48 -07:00
2017-03-09 05:07:44 +00:00
2019-03-14 22:43:50 +00:00
2018-03-08 04:48:36 +00:00
2018-01-17 04:18:24 +00:00
2016-08-25 10:20:36 +00:00
2018-09-10 16:09:41 +00:00
2018-09-10 16:09:41 +00:00
2016-06-20 15:37:18 -07:00
2016-06-20 15:37:18 -07:00
2020-06-18 21:00:27 +00:00
2018-08-22 18:08:20 +00:00
2017-05-09 15:47:21 +00:00
2019-08-24 00:32:28 +00:00
2019-05-07 00:36:56 +00:00
2019-04-09 08:54:23 +00:00
2018-02-22 00:57:46 +00:00
2018-11-27 02:01:12 +00:00
2019-05-10 11:46:38 +00:00
2019-05-10 11:46:38 +00:00
2019-05-10 11:46:38 +00:00
2020-11-24 18:35:07 +00:00
2019-10-05 00:04:08 +00:00
2018-03-22 02:47:33 +00:00
2017-07-22 05:53:06 +00:00
2020-05-21 18:40:20 +00:00
2018-08-23 15:55:53 +00:00
2016-10-14 05:41:22 +00:00
2020-08-25 15:42:01 +00:00
2019-04-12 07:28:26 +00:00
2018-04-11 02:21:48 +00:00
2019-05-10 12:06:01 +00:00
2020-05-20 23:03:23 +00:00
2020-05-20 23:03:23 +00:00
2020-05-20 23:03:23 +00:00
2018-08-15 04:15:46 +00:00
2018-08-15 04:15:46 +00:00
2019-02-07 21:41:52 +00:00
2017-12-13 10:04:40 +00:00
2020-11-28 07:30:06 +00:00
2020-04-07 15:10:27 +00:00
2020-04-02 12:01:41 +00:00
2019-08-06 02:35:54 +00:00
2020-05-14 02:52:16 +00:00
2017-11-17 21:33:51 +00:00
2020-06-05 17:43:32 +00:00
2020-04-14 15:42:13 +00:00
2020-04-14 15:42:13 +00:00
2020-05-15 21:46:38 +00:00
2018-11-17 01:48:05 +00:00
2019-02-12 20:37:45 +00:00
2020-07-08 19:32:15 +00:00
2019-01-09 05:26:50 +00:00
2016-08-31 03:12:30 +00:00
2020-06-18 21:56:32 +00:00
2020-04-02 12:01:41 +00:00
2019-09-26 18:35:06 +00:00
2020-11-20 21:40:28 +00:00
2016-09-29 21:14:13 +00:00
2020-05-12 23:29:04 +00:00
2020-04-14 12:18:21 +00:00
2017-08-05 01:03:02 +00:00
2017-08-05 01:03:02 +00:00
2018-04-28 23:41:39 +00:00
2018-02-02 01:13:08 +00:00
2020-10-21 05:20:33 +00:00
2018-10-30 22:17:49 +00:00
2020-03-03 06:12:24 +00:00
2020-03-17 23:29:45 +00:00
2020-08-25 16:11:20 +00:00
2020-04-24 20:56:58 +00:00
2018-04-28 23:41:39 +00:00
2020-08-25 16:11:20 +00:00
2017-08-29 23:01:10 +00:00
2019-01-22 01:20:31 +00:00
2019-01-22 01:20:31 +00:00
2020-05-15 16:49:32 +00:00
2020-08-13 03:10:16 +00:00
2017-03-03 01:43:42 +00:00
2017-03-03 01:43:42 +00:00
2018-11-01 21:27:13 +00:00
2018-11-01 21:27:13 +00:00
2019-08-22 05:03:03 +00:00
2020-08-25 23:08:40 +00:00
2020-10-21 05:20:33 +00:00
2020-10-21 05:20:33 +00:00
2018-04-11 02:21:48 +00:00
2016-02-28 21:31:37 -08:00
2019-09-27 01:54:39 +00:00
2020-10-21 05:20:33 +00:00
2020-05-12 23:29:04 +00:00
2016-11-09 03:27:12 +00:00
2019-04-23 13:33:57 +00:00
2016-11-09 03:27:12 +00:00
2018-04-11 02:21:48 +00:00
2016-12-08 04:53:53 +00:00
2016-12-08 04:53:53 +00:00
2016-11-09 03:27:12 +00:00
2018-11-20 23:49:11 +00:00
2019-05-15 22:34:28 +00:00
2020-10-21 05:20:33 +00:00
2020-06-15 23:42:12 +00:00
2018-07-03 20:34:27 +00:00
2020-09-18 22:12:47 +00:00
2018-02-02 01:13:08 +00:00
2019-05-24 15:08:19 +00:00
2019-05-24 15:08:19 +00:00
2020-07-14 03:13:18 +00:00
2019-04-23 13:33:57 +00:00
2014-06-20 13:35:10 -07:00
2014-01-08 10:54:01 -08:00
2020-10-21 05:20:33 +00:00
5530b62539