Various build and test machines have multiple versions of java
installed and relying on the default "java" command being compatible
isn't practical (a machine may also build an older version of Impala
that might require a different java version). Since JAVA_HOME is already
required that can/should be used to determine which java binary to use.
This also includes a minor change to replace a block of code that was
using 4-space indent. Instead of using 2-space indent, that block was
replaced with one line.
Change-Id: I4b8698b2aa5411b5fa6c5bc06291625999478955
Reviewed-on: http://gerrit.cloudera.org:8080/310
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
This is the first iteration of a kerberized development environment.
All the daemons start and use kerberos, with the sole exception of the
hive metastore. This is sufficient to test impala authentication.
When buildall.sh is run using '-kerberize', it will stop before
loading data or attempting to run tests.
Loading data into the cluster is known to not work at this time, the
root causes being that Beeline -> HiveServer2 -> MapReduce throws
errors, and Beeline -> HiveServer2 -> HBase has problems. These are
left for later work.
However, the impala daemons will happily authenticate using kerberos
both from clients (like the impala shell) and amongst each other.
This means that if you can get data into the mini-cluster, you could
query it.
Usage:
* Supply a '-kerberize' option to buildall.sh, or
* Supply a '-kerberize' option to create-test-configuration.sh, then
'run-all.sh -format', re-source impala-config.sh, and then start
impala daemons as usual. You must reformat the cluster because
kerberizing it will change all the ownership of all files in HDFS.
Notable changes:
* Added clean start/stop script for the llama-minikdc
* Creation of Kerberized HDFS - namenode and datanodes
* Kerberized HBase (and Zookeeper)
* Kerberized Hive (minus the MetaStore)
* Kerberized Impala
* Loading of data very nearly working
Still to go:
* Kerberize the MetaStore
* Get data loading working
* Run all tests
* The unknown unknowns
* Extensive testing
Change-Id: Iee3f56f6cc28303821fc6a3bf3ca7f5933632160
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4019
Reviewed-by: Michael Yoder <myoder@cloudera.com>
Tested-by: jenkins
There are at least two problems:
1) generate-schema-statements.py wasn't putting a newline on the very
last insert stmt, and beeline apparently was then ignoring it.
2) If HBaseTestDataRegionAssignment fails, then we reload a couple
tables, but were not recomputing stats for those tables. And some
query-tests expect those tables to have stats.
Tesing: Ran the following commands and see that the tables are now
not-empty and include stats:
$IMPALA_HOME/bin/load-data.py -w functional-query \
--table_names=alltypesagg,alltypessmall --table_formats=hbase/none --force
$IMPALA_HOME/tests/util/compute_table_stats.py --db_names=functional_hbase \
--table_names=alltypesagg,alltypessmall
Change-Id: I5183e037d0f5499c81b79f2cc1060b71be2d4873
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3794
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
(cherry picked from commit 306b87b37edbf10fa4b89ed2206484e158cc8e0d)
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3802
Reviewed-by: Daniel Hecht <dhecht@cloudera.com>
