Lars Volker 8ea21d099f
IMPALA-2523: Make HdfsTableSink aware of clustered input ...
IMPALA-2521 introduced clustering for insert statements. This change
makes the HdfsTableSink aware of clustered inputs, so that partitions
are opened, written, and closed one by one.
This change also adds/modifies tests in several ways:
- clustered insert tests switch from selecting all rows from
alltypessmall to alltypes. Together with varying settings for
batch_size, this results in a larger number of row batches being
written.
- clustered insert tests select from alltypes instead of
functional.alltypes to make sure we also select from various input
formats.
- clustered insert tests have been added to select from alltypestiny to
create inserts with 1 and 2 rows per partition respectively.
- exhaustive insert tests now use different values for batch_size: 1,
16, 0 (meaning default, 1024). This is limited to uncompressed parquet
files, to maintain a reasonable runtime. On my machine execution of
test.insert took 1778 seconds, compared to 1002 seconds with the just
default row batch size.
- There is additional testing in test_insert_behaviour.py to make sure
that insertion over several row batches only creates one file per
partition.
- It renames the test_insert method to make it unique in the file and
allow for effective filtering with -k.
- It adds tests to the Analyzer test suite.
Change-Id: Ibeda0bdabbfe44c8ac95bf7c982a75649e1b82d0
Reviewed-on: http://gerrit.cloudera.org:8080/4863
Reviewed-by: Lars Volker <lv@cloudera.com >
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com >
Tested-by: Internal Jenkins
2016-11-22 02:51:20 +00:00
2016-09-13 21:57:36 +00:00
2016-09-14 22:11:04 +00:00
2016-05-12 14:17:39 -07:00
2016-02-10 00:54:11 +00:00
2016-05-31 23:32:12 -07:00
2016-11-15 03:27:36 +00:00
2014-09-26 16:56:40 -07:00
2016-10-22 07:39:37 +00:00
2016-11-15 05:38:12 +00:00
2016-07-14 19:04:43 +00:00
2014-09-26 12:28:03 -07:00
2014-09-26 12:24:07 -07:00
2015-03-07 09:51:27 +00:00
2016-05-12 14:17:50 -07:00
2016-09-21 01:24:53 +00:00
2016-05-12 14:17:50 -07:00
2015-04-21 19:27:50 +00:00
2016-11-15 03:27:36 +00:00
2016-08-31 00:58:03 +00:00
2016-09-02 02:47:02 +00:00
2016-09-02 02:47:02 +00:00
2016-09-02 02:47:02 +00:00
2016-06-10 10:31:15 -07:00
2016-09-02 02:47:02 +00:00
2016-10-07 03:36:43 +00:00
2016-07-14 19:04:44 +00:00
2016-07-18 08:28:09 -07:00
2016-06-07 09:34:30 -07:00
2016-06-07 09:34:30 -07:00
2016-06-07 09:34:30 -07:00
2016-06-07 09:34:30 -07:00
2015-07-16 19:38:17 +00:00
2016-02-19 00:03:15 -08:00
2016-08-19 06:04:18 +00:00
2016-01-27 20:41:45 +00:00
2016-03-02 23:23:04 -08:00
2014-06-11 03:10:11 -07:00
2014-06-11 03:10:11 -07:00
2014-06-11 03:10:11 -07:00
2014-06-11 03:10:11 -07:00
2016-11-08 02:44:51 +00:00
2016-09-02 02:47:02 +00:00
2016-05-12 14:17:57 -07:00
2016-05-23 08:40:19 -07:00
2016-05-23 08:40:19 -07:00
2014-05-08 13:59:00 -07:00
2015-02-23 23:32:41 +00:00
2015-08-22 01:46:26 +00:00
2015-02-23 23:32:41 +00:00
2015-02-23 23:32:41 +00:00
2016-10-13 00:40:41 +00:00
2015-02-23 23:32:41 +00:00
2015-02-23 23:32:41 +00:00
2014-01-08 10:52:36 -08:00
2015-02-23 23:32:41 +00:00
2015-04-21 19:27:50 +00:00
2016-11-15 03:27:36 +00:00
2015-10-12 14:41:00 -07:00
2014-06-24 02:14:27 -07:00
2016-05-12 14:17:46 -07:00
2016-02-19 00:03:15 -08:00
2014-01-08 10:52:14 -08:00
2016-05-12 14:17:59 -07:00
2014-01-08 10:46:49 -08:00
2016-02-19 00:03:15 -08:00
2015-10-07 14:47:40 -07:00
2015-08-22 01:46:26 +00:00
2014-01-08 10:53:51 -08:00
2015-05-27 22:25:12 +00:00
2014-12-02 18:08:09 -08:00
2016-05-12 14:17:49 -07:00
2016-11-22 02:51:20 +00:00
2016-02-19 00:03:15 -08:00
2016-11-09 03:27:12 +00:00
2015-02-23 23:32:41 +00:00
2015-10-12 14:41:05 -07:00
2016-10-25 05:52:33 +00:00
2016-11-15 03:27:36 +00:00
2016-11-17 02:54:30 +00:00
2016-11-18 11:41:01 +00:00
2016-11-18 11:41:01 +00:00
2016-11-04 22:02:22 +00:00
2016-11-04 22:02:22 +00:00
2016-11-18 11:41:01 +00:00
2016-11-18 11:41:01 +00:00
2016-11-01 01:47:30 +00:00
2016-11-05 06:43:45 +00:00
2016-11-05 06:43:45 +00:00
2016-05-12 14:17:39 -07:00
2015-09-10 04:50:31 +00:00
2016-11-09 03:27:12 +00:00
2015-02-23 23:32:41 +00:00
2016-11-09 03:27:12 +00:00
2016-07-19 23:30:02 -07:00
2016-06-08 16:30:32 -07:00
2016-09-03 00:39:07 +00:00
2016-11-17 05:31:34 +00:00
2014-01-08 10:48:09 -08:00
2016-11-03 11:59:07 +00:00
2016-10-26 22:21:19 +00:00
2016-10-22 10:24:24 +00:00
2016-05-31 23:32:11 -07:00
2016-05-12 23:06:36 -07:00
2015-09-02 19:23:54 +00:00
2015-09-02 19:23:54 +00:00
2016-04-01 05:06:38 +00:00
2015-09-02 19:23:54 +00:00
2015-09-02 19:23:54 +00:00
2016-01-27 20:41:45 +00:00
2016-08-31 21:20:29 +00:00
2015-10-07 14:47:40 -07:00
2015-08-22 01:46:26 +00:00
2016-02-10 07:16:58 +00:00
2014-05-08 12:28:53 -07:00
2016-05-12 14:17:48 -07:00
2016-08-25 10:20:36 +00:00
2016-06-20 15:37:18 -07:00
2016-06-20 15:37:18 -07:00
2016-05-12 14:17:38 -07:00
2016-10-14 05:41:22 +00:00
2016-08-11 08:42:41 +00:00
2016-09-08 04:31:27 +00:00
2016-11-22 00:42:57 +00:00
2016-11-22 00:42:57 +00:00
2016-05-12 23:06:35 -07:00
2016-09-02 02:19:52 +00:00
2016-09-01 02:35:41 +00:00
2016-09-02 02:19:52 +00:00
2016-11-03 23:08:56 +00:00
2015-09-27 15:13:32 -07:00
2016-08-31 03:12:30 +00:00
2014-11-19 17:21:36 -08:00
2016-05-17 10:09:05 -07:00
2016-01-20 04:32:21 +00:00
2016-09-29 21:14:13 +00:00
2016-06-07 09:34:30 -07:00
2016-07-07 10:41:29 -07:00
2016-06-02 21:33:08 -07:00
2016-01-26 03:13:05 +00:00
2016-09-30 01:21:05 +00:00
2016-06-02 21:33:08 -07:00
2016-08-29 16:44:22 +00:00
2016-08-25 10:20:36 +00:00
2016-08-25 10:20:36 +00:00
2015-09-27 15:13:28 -07:00
2016-05-12 23:06:36 -07:00
2016-07-19 23:30:02 -07:00
2016-02-28 21:31:37 -08:00
2014-09-26 12:28:03 -07:00
2016-05-12 14:17:45 -07:00
2016-09-02 02:47:02 +00:00
2016-11-09 03:27:12 +00:00
2016-11-09 03:27:12 +00:00
2016-11-09 03:27:12 +00:00
2016-11-09 03:27:12 +00:00
2014-06-12 02:23:20 -07:00
2016-11-09 03:27:12 +00:00
2016-11-09 03:27:12 +00:00
2016-11-03 08:59:45 +00:00
2014-01-08 10:48:09 -08:00
2016-01-23 23:59:27 +00:00
2014-09-18 00:36:03 -07:00
2016-09-02 02:47:02 +00:00
2014-06-20 13:35:10 -07:00
2014-01-08 10:54:01 -08:00
2015-10-07 14:47:40 -07:00
8ea21d099f