This patch implements a new feature to read the auth_to_local
configs from hdfs configuration files, using the parameter
hadoop.security.auth_to_local. This is done by modifying the
User#getShortName() method to use its hdfs equivalent.
This patch includes an end to end authorization test using
sentry where we add specific auth_to_local setting for a certain
user and test if the sentry authorization passes for this user
after applying these rules. Given we don't have tests that run
on a kerberized min-cluster, this patch adds a hack to load this
configuration during even on non-kerberized 'test runs'.
However this feature is disabled by default to preserve the
existing behavior. To enable it,
1. Use kerberos as authentication mechanism (by setting --principal) and
2. Add "--load_auth_to_local_rules=true" to the cluster startup args
Change-Id: I76485b83c14ba26f6fce66e5f83e8014667829e0
Reviewed-on: http://gerrit.cloudera.org:8080/2800
Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com>
Tested-by: Internal Jenkins
/tmp isn't necessarily on the same filesystem as the Kudu data
directory. Fix the check so that it checks the actual Kudu directory.
Change-Id: Ic6aa27569a0650db7dcf5759952cd50c8e47f8c9
Reviewed-on: http://gerrit.cloudera.org:8080/2967
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
Changes:
1) Previously when a service would fail, the user would have to find the
the log file and open it. Now the end of the log is dumped to stdout.
2) Add start, stop, and restart commands to the "admin" script. For
example now you can run
testdata/cluster/admin restart kudu
3) Wait up to 120 seconds for services to shutdown. The timeout is the
same as for the Impala processes. If the services fail to stop an
error will be raised.
Change-Id: I537ea5656df2081d4f1f27a9f3fcef4547fdc2fe
Reviewed-on: http://gerrit.cloudera.org:8080/2751
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
By default Kudu requires the underlying file system to support hole
punching. If support isn't there Kudu will fail to start. People using
such a file system can instead start Kudu with -block_manager=file.
Before starting Kudu in the local mini-cluster, the "fallocate"
command will be used to automatically determine if the special flag is
needed.
Note, users who need this must run bin/create-test-configuration.sh
after pulling in this commit.
This also fixes a bug in the delete_kudu_data() in the cluster admin
script. A directory name was incorrect.
Change-Id: I1ca7fedb367444c41e462b72b0b76091ee94e27c
Reviewed-on: http://gerrit.cloudera.org:8080/2750
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
The directory structure of the newer Kudu toolchain artifacts has
changed. Now the root directory is split into /release and /debug. A few
little updates are needed to the build and service scripts.
Since the toolchain no longer provides stubs for platforms that Kudu
doesn't support the stubs need to be generated. This will be done as
part of the toolchain bootstrapping.
Also this upgrades Kudu to 0.8 RC1.
Developers will need to run bin/create-test-configuration.sh after
pulling in this change. Otherwise the Kudu service will fail to start.
Change-Id: I625903bd92afece0ad819a96fc275d5812b5eb2a
Reviewed-on: http://gerrit.cloudera.org:8080/2720
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
The Kudu team recommended disabling this for testing purposes. This
should help with timeouts in cloud machines (ec2/gce). Disabling
fsyncs could lead to data loss if the system crashed before the OS had a
chance to write the data to disk. Our test setups don't need that level
of reliability.
Change-Id: I72fd85ce5c4bc71f071b854ea6a9ebe60fc1305f
Reviewed-on: http://gerrit.cloudera.org:8080/2734
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
This is for review purposes only. This patch will be merged with David's
big merge patch.
Changes:
1) Make Kudu compilation dependent on the OS since not all OSs support
Kudu.
2) Only run Kudu related tests when Kudu is supported (see #1).
3) Look for Kudu locally, but in a different location. To use a local
build of Kudu, set KUDU_BUILD_DIR to the path Kudu was built in and
set KUDU_CLIENT_DIR to the path KUDU was installed in.
Example:
git clone https://github.com/cloudera/kudu.git
...build 3rd party etc...
mkdir -p $KUDU_BUILD_DIR
cd $KUDU_BUILD_DIR
cmake <path to Kudu source dir>
make
DESTDIR=$KUDU_CLIENT_DIR make install
4) Look for Kudu in the toolchain if not using a local Kudu build.
5) Add Kudu service startup scripts. The Kudu in the toolchain is
actually a parcel that has been renamed (the contents were not
modified in any way), that mean the Kudu service binaries are there.
Those binaries are now used to run the Kudu service.
Change-Id: I3db88cbd27f2ea2394f011bc8d1face37411ed58
The major changes are:
1) Collect backtrace and fatal log on crash.
2) Poll memory usage. The data is only displayed at this time.
3) Support kerberos.
4) Add random queries.
5) Generate random and TPC-H nested data on a remote cluster. The
random data generator was converted to use MR for scaling.
6) Add a cluster abstraction to run data loading for #5 on a
remote or local cluster. This also moves and consolidates some
Cloudera Manager utilities that were in the stress test.
7) Cleanup the wrappers around impyla. That stuff was getting
messy.
Change-Id: I4e4b72dbee1c867626a0b22291dd6462819e35d7
Reviewed-on: http://gerrit.cloudera.org:8080/1298
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
Some of the tests rely on hdfs trash mechanism to be enabled and poll
the paths in the trash directory during test runs. These tests are
failing intermittenly due to a race with the hdfs trash checkpointing
mechanism which moves all the trash contents to another directory.
This checkpointing runs every fs.trash.checkpoint.interval minutes
and defaults to fs.trash.interval (when set to 0). Currently there
seems to be no way to disable this checkpointing. This patch increases
the fs.trash.interval from the current value of 30 minutes to 24 hours
so that the test runs never hit this race condition.
Change-Id: I42fcaee70a461712f1df6bac23c71f915718b015
Reviewed-on: http://gerrit.cloudera.org:8080/1703
Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com>
Tested-by: Internal Jenkins
Until now, our YARN configuration was broken so that we weren't able to
run local Map Reduce jobs. The jobs would fail with a class not found
exception of the LZO codec. This patch fixes this issues and corrects
the classpath.
Change-Id: I689cca7a079dbd269d4bd96f1b4e3d91147d527c
Reviewed-on: http://gerrit.cloudera.org:8080/1667
Reviewed-by: Martin Grund <mgrund@cloudera.com>
Tested-by: Internal Jenkins
This commit adds PURGE option to DROP TABLE/ALTER TABLE DROP
PARTITION statements. Following is the usage:
1. DROP TABLE <tablename> takes an optional argument PURGE. Adding
purge purges the table data by skipping trash, if configured.
DROP TABLE [<database>.]<tablename> [IF EXISTS] [PURGE]
2. PURGE is also supported with alter table drop partition query
with the following syntax. If specified, impala purges the partition
data by skipping trash.
ALTER TABLE [<database>.]<tablename> DROP PARTITION [IF EXISTS] [PURGE]
This patch also helps the use case where Trash and the data directories
are in different encryption zones, in which case we cannot move the data
during ALTER/DROP. Then purge option can be used to skip the trash and
make sure data is actually deleted.
Change-Id: I64bf71d660b719896c32e0f3a7ab768f30ec7b3b
(cherry picked from commit 585d4f8d9e809f3bf194018dd161a22d3f144270)
Reviewed-on: http://gerrit.cloudera.org:8080/1244
Reviewed-by: Juan Yu <jyu@cloudera.com>
Tested-by: Internal Jenkins
This patch enables running Impala tests against Isilon as the default file system. The
intention is to run tests against a realistic deployment, i.e, Isilon replacing HDFS as
the underlying filesystem.
Specifically, it does the following:
- Adds a new environment variable DEFAULT_FS, which points to HDFS by default.
- Makes the fs.defaultFs property in core-site.xml use the DEFAULT_FS environment
variable, such that all clients talk to Isilon implicitly.
- Unset FILESYSTEM_PREFIX when the TARGET_FILESYSTEM is Isilon, since path prefixes
are no longer needed.
- Only starts the Hive Metastore and the Impala service stack when running
tests against Isilon.
We don't start KMS/HBase because they're not relevant to Isilon. We also don't
start YARN, Hive and LLama because hive queries are disabled with Isilon.
The scripts that start/stop Hive, YARN and Llama should be modified to point to a
filesystem other than HDFS in the future.
Change-Id: Id66bfb160fe57f66a64a089b465b536c6c514b63
Reviewed-on: http://gerrit.cloudera.org:8080/449
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: Internal Jenkins
Update the yarn-site.xml to reduce the latency of
resource acquisition.
Also changes the log4j properties to reduce the very
verbose logging for the hadoop daemons which was consuming
huge amounts of space very quickly.
Change-Id: I8532fb5125b604974e26ddad76aee93b9c4e64fb
Reviewed-on: http://gerrit.cloudera.org:8080/381
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Internal Jenkins
This patch enables the Impala test suite to run the end to end tests
against an isilon namenode. There are a few caveats:
- The fe test will currently not work.
- Only loading data from both the test-warehouse snapshot and the metadata snapshot is
supported.
- The test suite cannot be run by multiple people (unless we have access to multiple
isilon namenodes)
Change-Id: I786b4e4f51b99e79ad42abc676f537ebfc189237
Reviewed-on: http://gerrit.cloudera.org:8080/356
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: Internal Jenkins
The templates for starting the services of the cluster had a bad
declaration of the shebang that made it impossible to start kms when
using a non-bash default shell.
Change-Id: I6b105b328dc61e71095c2d5e5d6859f65ca56a18
Reviewed-on: http://gerrit.cloudera.org:8080/293
Reviewed-by: Martin Grund <mgrund@cloudera.com>
Tested-by: Internal Jenkins
First change for IMPALA-1209 to address Impala limitations when
using HDFS encryption. This adds a KMS process to the testdata
cluster. This was tested manually by creating a key and an
encryption zone.
Change-Id: I499154506386f04e71c5371b128c10868b1e1318
Reviewed-on: http://gerrit.cloudera.org:8080/41
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Internal Jenkins
This patch enables loading data to s3 instead of hdfs. It is preliminary in nature,
as such, there are a few caveats:
- The fe tests do not work.
- Only loading from a test-warehouse snapshot and metastore snapshot is enabled.
- Until hive works with s3, only a subset of all the tests will work.
Change-Id: Ia66a5f836b4245e3b022a49de805eec337a51324
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/5851
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: jenkins
This will allow you to create tables around data that already exists on
S3. (Though INSERT and LOAD DATA don't support S3 yet). Also this will
make it easier to create some test tables that are not on HDFS.
Also, workaround HDFS-7031 (which is a "won't fix") where non-defaultFS
paths can be qualified with the wrong authority. This is needed for
Impala now that it can take non-HDFS paths as input.
Change-Id: Ie513d50b26dfe5a71be284ad31a8c8151d0e30d3
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/5417
Reviewed-by: Daniel Hecht <dhecht@cloudera.com>
Tested-by: jenkins
Prior to this work, the impalad could either authenticate with
Kerberos, or authenticate with LDAP. This fixes that so that both can
co-exist in the same daemon. Prior code had both a
KerberosAuthProvider and an LdapAuthProvider; this is refactored into
a single SaslAuthProvider that potentially contains both LDAP and
Kerberos.
The terminology of "client facing" and "server facing" has been
replaced with "external" and "internal". External is for clients like
the impala shell, odbc, jdbc, etc. Internal is for daemon <-> daemon
communication.
The notion of the "auxprop" plugin is removed, as that was dead code.
The Thrift code is enhanced to pass the Realm information from the
SaslAuthProvider down to the underlying SASL library.
Change-Id: I0a0b968a107c0b25610ca37295c3fee345ecdd6d
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4051
Reviewed-by: Michael Yoder <myoder@cloudera.com>
Tested-by: jenkins
This is the first iteration of a kerberized development environment.
All the daemons start and use kerberos, with the sole exception of the
hive metastore. This is sufficient to test impala authentication.
When buildall.sh is run using '-kerberize', it will stop before
loading data or attempting to run tests.
Loading data into the cluster is known to not work at this time, the
root causes being that Beeline -> HiveServer2 -> MapReduce throws
errors, and Beeline -> HiveServer2 -> HBase has problems. These are
left for later work.
However, the impala daemons will happily authenticate using kerberos
both from clients (like the impala shell) and amongst each other.
This means that if you can get data into the mini-cluster, you could
query it.
Usage:
* Supply a '-kerberize' option to buildall.sh, or
* Supply a '-kerberize' option to create-test-configuration.sh, then
'run-all.sh -format', re-source impala-config.sh, and then start
impala daemons as usual. You must reformat the cluster because
kerberizing it will change all the ownership of all files in HDFS.
Notable changes:
* Added clean start/stop script for the llama-minikdc
* Creation of Kerberized HDFS - namenode and datanodes
* Kerberized HBase (and Zookeeper)
* Kerberized Hive (minus the MetaStore)
* Kerberized Impala
* Loading of data very nearly working
Still to go:
* Kerberize the MetaStore
* Get data loading working
* Run all tests
* The unknown unknowns
* Extensive testing
Change-Id: Iee3f56f6cc28303821fc6a3bf3ca7f5933632160
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4019
Reviewed-by: Michael Yoder <myoder@cloudera.com>
Tested-by: jenkins
Our testdata/run-all.sh can be brittle depending on the state of your Hdfs.
In particular, Yarn depends on the NN not being in safe mode, but it may take
some time for the NN to exit safe mode immediately after starting Hdfs.
This patch makes the NN startup script complete only after the NN has exited
safe mode.
Change-Id: I8b30cd07128dc48d79d91726eafed4174fb91a6d
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3005
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3021
Since we're no longer using the MiniLlama, we need to explicitly set
whether or not the cluster is pseudo-distributed. Impala needs this
information to correctly translate datanode addresses to a format that
Llama understands.
This change (adapted from one made by Casey) adds a method to the
frontend (callable via JNI) to get a configuration value from the Hadoop
configuration. We'll set that configuration value for local RM testing.
Change-Id: Ifd51db98a993ac0270dac2b832babbc394483c1a
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2549
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
The hdfs-site template in CDH5 is different from the one we fine in CDH5. Specifically:
- It has entries that enable hdfs caching.
- It uses the correct parameter name for hdfs block locations timeout.
Change-Id: I0ca6bd84b074ccbb8f42243d37c5082b305f9bcf
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2338
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: jenkins
This should allow individual service components, such as a single nodemanager,
to be shutdown for failure testing. The mini-cluster bundled with hadoop is a
single process that does not expose the ability to control individual roles.
Now each role can be controlled and configured independently of the others.
Change-Id: Ic1d42e024226c6867e79916464d184fce886d783
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1432
Tested-by: Casey Ching <casey@cloudera.com>
Reviewed-by: Casey Ching <casey@cloudera.com>
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2297
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: Ishaan Joshi <ishaan@cloudera.com>
