Philip Zeyliger 11dbb3952a IMPALA-6070: Parallelize another bit of data load. ...

The two Kudu loads and Hive UDFs can all run in parallel. This
should shave about 4 minutes off of the data load. (Current
timings are 3.5, 4, and 0.6 minutes, see below.)

I've run dataload with this change many times.

   Loading Kudu functional (logging to /home/ubuntu/Impala/logs/data_loading/load-kudu.log)...
     Loading workload 'functional-query' using exploration strategy 'core' in table formats 'kudu/none/none' OK (Took: 3 min 29 sec)
   Loading Kudu TPCH (logging to /home/ubuntu/Impala/logs/data_loading/load-kudu-tpch.log)...
     Loading workload 'tpch' using exploration strategy 'core' in table formats 'kudu/none/none' OK (Took: 4 min 0 sec)
   Loading Hive UDFs (logging to /home/ubuntu/Impala/logs/data_loading/build-and-copy-hive-udfs.log)...
     Loading Hive UDFs OK (Took: 0 min 41 sec)

Change-Id: I7e93ee5a77ec9271b980b88bef7ad512ecbe0407
Reviewed-on: http://gerrit.cloudera.org:8080/8822
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: Impala Public Jenkins

2017-12-14 02:28:40 +00:00

..

cache_tables.py

IMPALA-4110: Clean up issues found by Apache RAT.

2016-09-14 22:09:24 +00:00

check-hbase-nodes.py

IMPALA-4684: Handle Zookeeper ConnentionLoss exceptions

2016-12-22 01:18:56 +00:00

check-schema-diff.sh

IMPALA-4294: Make check-schema-diff.sh executable from anywhere.

2016-10-15 04:05:04 +00:00

compute-table-stats.sh

IMPALA-6070: Parallel compute_table_stats.py

2017-10-24 23:54:15 +00:00

copy-data-sources.sh

IMPALA-3918: Remove Cloudera copyrights and add ASF license header

2016-08-09 08:19:41 +00:00

copy-udfs-udas.sh

IMPALA-4649: add a mechanism to pass flags into make

2016-12-15 21:37:17 +00:00

create-data-source-table.sql

IMPALA-3786: Replace "cloudera" with "apache" (part 2)

2016-09-29 21:14:13 +00:00

create-hbase.sh

IMPALA-3918: Remove Cloudera copyrights and add ASF license header

2016-08-09 08:19:41 +00:00

create-load-data.sh

IMPALA-6070: Parallelize another bit of data load.

2017-12-14 02:28:40 +00:00

create-mini.sql

IMPALA-4110: Clean up issues found by Apache RAT.

2016-09-14 22:09:24 +00:00

create-table-many-blocks.sh

IMPALA-5975: Work around broken beeline clients

2017-09-27 03:27:45 +00:00

generate-block-ids.sh

Expose $IMPALA_MAVEN_OPTIONS for configuring Maven.

2017-11-14 01:29:56 +00:00

generate-load-nested.sh

Expose $IMPALA_MAVEN_OPTIONS for configuring Maven.

2017-11-14 01:29:56 +00:00

generate-schema-statements.py

IMPALA-6068: Scale back fixing functional-types

2017-12-04 23:46:44 +00:00

generate-test-vectors.py

IMPALA-3918: Remove Cloudera copyrights and add ASF license header

2016-08-09 08:19:41 +00:00

kill-all.sh

IMPALA-3918: Remove Cloudera copyrights and add ASF license header

2016-08-09 08:19:41 +00:00

kill-hbase.sh

IMPALA-3918: Remove Cloudera copyrights and add ASF license header

2016-08-09 08:19:41 +00:00

kill-hive-server.sh

IMPALA-3918: Remove Cloudera copyrights and add ASF license header

2016-08-09 08:19:41 +00:00

kill-java-service.sh

IMPALA-3918: Remove Cloudera copyrights and add ASF license header

2016-08-09 08:19:41 +00:00

kill-mini-dfs.sh

IMPALA-3918: Remove Cloudera copyrights and add ASF license header

2016-08-09 08:19:41 +00:00

kill-sentry-service.sh

IMPALA-3918: Remove Cloudera copyrights and add ASF license header

2016-08-09 08:19:41 +00:00

load_nested.py

IMPALA-5162,IMPALA-5163: stress test support on secure clusters

2017-05-02 04:56:01 +00:00

load-dependent-tables.sql

IMPALA-4110: Clean up issues found by Apache RAT.

2016-09-14 22:09:24 +00:00

load-hive-builtins.sh

IMPALA-4110: Clean up issues found by Apache RAT.

2016-09-14 22:09:24 +00:00

load-metastore-snapshot.sh

IMPALA-3918: Remove Cloudera copyrights and add ASF license header

2016-08-09 08:19:41 +00:00

load-test-warehouse-snapshot.sh

IMPALA-3224: De-Cloudera non-docs JIRA URLs

2017-05-07 04:44:57 +00:00

load-tpc-kudu.py

IMPALA-3739: Enable stress tests on Kudu

2016-10-21 11:01:37 +00:00

lzo_indexer.sh

IMPALA-3918: Remove Cloudera copyrights and add ASF license header

2016-08-09 08:19:41 +00:00

minikdc_env.sh

IMPALA-3918: Remove Cloudera copyrights and add ASF license header

2016-08-09 08:19:41 +00:00

minikdc.sh

IMPALA-3918: Remove Cloudera copyrights and add ASF license header

2016-08-09 08:19:41 +00:00

random_avro_schema.py

IMPALA-3786: Replace "cloudera" with "apache" (part 2)

2016-09-29 21:14:13 +00:00

README-BENCHMARK-TEST-GENERATION

Added scripts for generating and running benchmarks across different data sets and file formats

2012-05-08 16:06:45 -07:00

run-all.sh

IMPALA-3918: Remove Cloudera copyrights and add ASF license header

2016-08-09 08:19:41 +00:00

run-hbase.sh

IMPALA-5223: Add waiting for HBase Zookeeper nodes to retry loop

2017-06-13 05:57:49 +00:00

run-hive-server.sh

IMPALA-6108, IMPALA-6070: Parallel data load (re-instated).

2017-11-02 00:40:19 +00:00

run-mini-dfs.sh

IMPALA-3918: Remove Cloudera copyrights and add ASF license header

2016-08-09 08:19:41 +00:00

run-sentry-service.sh

IMPALA-3786: Replace "cloudera" with "apache" (part 2)

2016-09-29 21:14:13 +00:00

run-step.sh

IMPALA-6108, IMPALA-6070: Parallel data load (re-instated).

2017-11-02 00:40:19 +00:00

setup-hdfs-env.sh

IMPALA-3224: De-Cloudera non-docs JIRA URLs

2017-05-07 04:44:57 +00:00

split-hbase.sh

Expose $IMPALA_MAVEN_OPTIONS for configuring Maven.

2017-11-14 01:29:56 +00:00

wait-for-hiveserver2.py

Increase wait times for startup of Hive and its Metastore

2016-11-15 20:35:01 +00:00

wait-for-metastore.py

Increase wait times for startup of Hive and its Metastore

2016-11-15 20:35:01 +00:00