Tim Armstrong 2a7c8b9011
IMPALA-5713: always reserve memory for preaggs ...
Before this change the preaggregation was frequently disabled when
running under some memory pressure, e.g. if the aggregation is at the
end of a pipeline of joins and those joins eat up all the memory.
This can result in huge performance degradation since all rows must then
be sent over the network.
This change always reserves 16 * (buffer size + 64kb) bytes per
preaggregation so that it is always able to build some hash tables and
reduce the input somewhat.
This has two parts:
* Changing the frontend reservation calculation
* Removing dead code in the backend that handled the case when the
initial partitions and hash tables could not be allocated.
Testing:
Passes exhaustive tests.
Change-Id: I2b544f9ec1a906719e2bbb074743926b95a3a573
Reviewed-on: http://gerrit.cloudera.org:8080/7739
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com >
Tested-by: Impala Public Jenkins
2017-08-25 00:48:28 +00:00
2016-09-13 21:57:36 +00:00
2017-07-06 01:26:44 +00:00
2017-08-17 03:12:48 +00:00
2017-04-29 02:23:51 +00:00
2017-02-14 05:56:33 +00:00
2017-02-22 23:10:34 +00:00
2017-06-22 19:08:59 +00:00
2017-05-27 05:19:53 +00:00
2017-08-22 08:27:12 +00:00
2016-12-07 09:45:11 +00:00
2016-07-14 19:04:43 +00:00
2014-09-26 12:28:03 -07:00
2014-09-26 12:24:07 -07:00
2015-03-07 09:51:27 +00:00
2016-11-24 08:03:39 +00:00
2017-02-22 23:10:34 +00:00
2017-07-25 03:31:19 +00:00
2016-05-12 14:17:50 -07:00
2015-04-21 19:27:50 +00:00
2017-02-22 23:10:34 +00:00
2016-08-31 00:58:03 +00:00
2016-09-02 02:47:02 +00:00
2017-05-12 15:43:30 +00:00
2017-06-09 23:29:17 +00:00
2017-05-12 15:43:30 +00:00
2017-05-12 15:43:30 +00:00
2017-05-02 01:12:14 +00:00
2016-07-14 19:04:44 +00:00
2017-03-02 20:12:05 +00:00
2017-02-22 06:31:14 +00:00
2016-06-07 09:34:30 -07:00
2016-06-07 09:34:30 -07:00
2016-06-07 09:34:30 -07:00
2016-06-07 09:34:30 -07:00
2017-06-29 21:14:59 +00:00
2017-08-24 06:27:05 +00:00
2017-08-07 00:57:46 +00:00
2015-07-16 19:38:17 +00:00
2016-02-19 00:03:15 -08:00
2016-08-19 06:04:18 +00:00
2016-01-27 20:41:45 +00:00
2016-03-02 23:23:04 -08:00
2017-08-12 08:10:07 +00:00
2017-08-12 08:10:07 +00:00
2017-08-12 08:10:07 +00:00
2017-08-12 08:10:07 +00:00
2017-08-04 21:51:19 +00:00
2017-08-17 03:12:48 +00:00
2017-07-26 05:43:01 +00:00
2017-07-21 19:16:57 +00:00
2017-02-22 23:10:34 +00:00
2017-02-22 23:10:34 +00:00
2014-05-08 13:59:00 -07:00
2015-02-23 23:32:41 +00:00
2015-08-22 01:46:26 +00:00
2015-02-23 23:32:41 +00:00
2015-02-23 23:32:41 +00:00
2016-10-13 00:40:41 +00:00
2015-02-23 23:32:41 +00:00
2017-02-22 23:10:34 +00:00
2014-01-08 10:52:36 -08:00
2015-02-23 23:32:41 +00:00
2017-04-26 01:12:01 +00:00
2015-04-21 19:27:50 +00:00
2016-11-15 03:27:36 +00:00
2015-10-12 14:41:00 -07:00
2014-06-24 02:14:27 -07:00
2017-05-09 01:36:46 +00:00
2016-02-19 00:03:15 -08:00
2014-01-08 10:52:14 -08:00
2016-05-12 14:17:59 -07:00
2014-01-08 10:46:49 -08:00
2016-02-19 00:03:15 -08:00
2015-10-07 14:47:40 -07:00
2015-08-22 01:46:26 +00:00
2014-01-08 10:53:51 -08:00
2015-05-27 22:25:12 +00:00
2014-12-02 18:08:09 -08:00
2016-05-12 14:17:49 -07:00
2017-05-22 00:40:04 +00:00
2016-02-19 00:03:15 -08:00
2017-07-07 01:30:59 +00:00
2015-02-23 23:32:41 +00:00
2015-10-12 14:41:05 -07:00
2016-10-25 05:52:33 +00:00
2017-06-22 19:08:59 +00:00
2017-08-07 19:53:59 +00:00
2016-12-07 07:31:16 +00:00
2017-05-11 20:55:51 +00:00
2017-07-22 00:28:36 +00:00
2016-12-10 00:05:50 +00:00
2016-12-07 05:11:13 +00:00
2016-12-06 10:41:53 +00:00
2016-12-07 07:31:16 +00:00
2017-05-11 20:55:51 +00:00
2017-05-11 20:55:51 +00:00
2017-05-18 21:09:51 +00:00
2017-07-21 21:49:04 +00:00
2017-07-21 21:49:04 +00:00
2017-02-15 01:33:23 +00:00
2016-11-09 03:27:12 +00:00
2015-02-23 23:32:41 +00:00
2016-11-09 03:27:12 +00:00
2016-07-19 23:30:02 -07:00
2016-06-08 16:30:32 -07:00
2016-09-03 00:39:07 +00:00
2016-11-17 05:31:34 +00:00
2014-01-08 10:48:09 -08:00
2016-11-03 11:59:07 +00:00
2017-03-17 19:33:31 +00:00
2016-12-02 01:46:55 +00:00
2016-12-02 01:46:55 +00:00
2016-12-02 01:46:55 +00:00
2016-05-31 23:32:11 -07:00
2017-03-01 02:00:19 +00:00
2016-05-12 23:06:36 -07:00
2015-09-02 19:23:54 +00:00
2015-09-02 19:23:54 +00:00
2016-04-01 05:06:38 +00:00
2015-09-02 19:23:54 +00:00
2015-09-02 19:23:54 +00:00
2017-08-18 02:53:23 +00:00
2017-08-05 01:03:02 +00:00
2015-10-07 14:47:40 -07:00
2015-08-22 01:46:26 +00:00
2016-12-08 02:20:50 +00:00
2016-12-08 02:20:50 +00:00
2017-08-19 20:56:10 +00:00
2017-03-03 01:43:42 +00:00
2016-05-12 14:17:48 -07:00
2017-03-09 05:07:44 +00:00
2017-03-09 05:07:44 +00:00
2016-08-25 10:20:36 +00:00
2016-06-20 15:37:18 -07:00
2016-06-20 15:37:18 -07:00
2017-05-09 15:47:21 +00:00
2017-05-31 06:49:43 +00:00
2017-03-03 10:20:07 +00:00
2017-07-22 05:53:06 +00:00
2017-07-06 01:26:44 +00:00
2016-10-14 05:41:22 +00:00
2017-05-25 02:55:36 +00:00
2016-09-08 04:31:27 +00:00
2017-07-25 03:31:19 +00:00
2016-11-22 00:42:57 +00:00
2016-05-12 23:06:35 -07:00
2017-06-08 21:31:08 +00:00
2017-08-05 01:03:02 +00:00
2017-06-08 21:31:08 +00:00
2016-11-03 23:08:56 +00:00
2015-09-27 15:13:32 -07:00
2016-08-31 03:12:30 +00:00
2017-07-19 06:48:06 +00:00
2017-08-23 03:27:26 +00:00
2017-05-12 15:43:30 +00:00
2016-09-29 21:14:13 +00:00
2017-02-22 23:10:34 +00:00
2016-07-07 10:41:29 -07:00
2017-08-05 01:03:02 +00:00
2017-08-05 01:03:02 +00:00
2016-01-26 03:13:05 +00:00
2016-09-30 01:21:05 +00:00
2016-06-02 21:33:08 -07:00
2017-08-25 00:48:28 +00:00
2017-08-23 03:27:26 +00:00
2017-08-24 04:24:10 +00:00
2017-08-24 04:24:10 +00:00
2017-08-24 06:27:05 +00:00
2017-08-23 03:59:02 +00:00
2017-08-12 08:10:07 +00:00
2017-03-03 01:43:42 +00:00
2017-03-03 01:43:42 +00:00
2016-05-12 23:06:36 -07:00
2016-07-19 23:30:02 -07:00
2016-02-28 21:31:37 -08:00
2014-09-26 12:28:03 -07:00
2016-05-12 14:17:45 -07:00
2017-02-22 23:10:34 +00:00
2016-11-09 03:27:12 +00:00
2017-08-17 03:12:48 +00:00
2016-11-09 03:27:12 +00:00
2016-11-09 03:27:12 +00:00
2016-12-08 04:53:53 +00:00
2016-12-08 04:53:53 +00:00
2016-11-09 03:27:12 +00:00
2016-12-08 04:53:53 +00:00
2017-02-03 23:35:25 +00:00
2017-05-04 02:04:03 +00:00
2014-01-08 10:48:09 -08:00
2017-07-19 22:17:13 +00:00
2016-01-23 23:59:27 +00:00
2014-09-18 00:36:03 -07:00
2016-09-02 02:47:02 +00:00
2014-06-20 13:35:10 -07:00
2014-01-08 10:54:01 -08:00
2015-10-07 14:47:40 -07:00
2a7c8b9011