3a2f5f28c9
The goals and non-goals of this patch could be summarized as follows.
Goals:
- Add changes to the minicluster configuration that allow a non-default
version of Ranger (possibly built locally) to run in the context of
the minicluster, and to be used as the authorization server by
Impala.
- Switch to the new constructor when instantiating
RangerAccessRequestImpl. This resolves IMPALA-12985 and also makes
Impala compatible with Apache Ranger if RangerAccessRequestImpl from
Apache Ranger is consumed.
- Prepare Ranger and Impala patches as supplemental material to verify
what authorization-related tests could be passed if Apache Ranger is
the authorization provider. Merging IMPALA-12921_addendum.diff to
the Impala repository is not in the scope of this patch in that the
diff file changes the behavior of Impala and thus more discussion is
required if we'd like to merge it in the future.
Non-goals:
- Set up any automation for building Ranger from source.
- Pass all Impala authorization-related tests with a non-default
version of Ranger.
Instructions on running Impala with locally built Ranger:
Suppose the Ranger project is under the folder $RANGER_SRC_DIR. We could
execute the following to build Apache Ranger for easy reference. By
default, the compressed tarball is produced under
$RANGER_SRC_DIR/target.
mvn clean compile -B -nsu -DskipCheck=true -Dcheckstyle.skip=true \
package install -DskipITs -DskipTests -Dmaven.javadoc.skip=true
After building Ranger, we need to build Impala's Java code so that
Impala's Java code could consume the locally produced Ranger classes. We
will need to export the following environment variables before building
Impala. This prevents bootstrap_toolchain.py from trying to download the
compressed Ranger tarball.
1. export RANGER_VERSION_OVERRIDE=\
$(mvn -f $RANGER_SRC_DIR/pom.xml -q help:evaluate \
-Dexpression=project.version -DforceStdout)
2. export RANGER_HOME_OVERRIDE=$RANGER_SRC_DIR/target/\
ranger-${RANGER_VERSION_OVERRIDE}-admin
It then suffices to execute the following to point
Impala to the locally built Ranger server before starting Impala.
1. source $IMPALA_HOME/bin/impala-config.sh
2. tar zxv -f $RANGER_SRC_DIR/target/\
ranger-${IMPALA_RANGER_VERSION}-admin.tar.gz \
-C $RANGER_SRC_DIR/target/
3. $IMPALA_HOME/bin/create-test-configuration.sh
4. $IMPALA_HOME/bin/create-test-configuration.sh \
-create_ranger_policy_db
5. $IMPALA_HOME/testdata/bin/run-ranger.sh
(run-all.sh has to be executed instead if other underlying services
have not been started)
6. $IMPALA_HOME/testdata/bin/setup-ranger.sh
Testing:
- Manually verified that we could point Impala to a locally built
Apache Ranger on the master branch (with tip being
https://github.com/apache/ranger/commit/4abb993).
- Manually verified that with RANGER-4771.diff and
IMPALA-12921_addendum.diff, only 3 authorization-related tests
failed. They failed because the resource type of 'storage-type' is
not supported in Apache Ranger yet and thus the test cases added in
IMPALA-10436 could fail.
- Manually verified that the log files of Apache and CDP Ranger's Admin
server could be created under ${RANGER_LOG_DIR} after we start the
Ranger service.
- Verified that this patch passed the core tests when CDP Ranger is
used.
Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
Reviewed-on: http://gerrit.cloudera.org:8080/21160
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Welcome to Impala
Lightning-fast, distributed SQL queries for petabytes of data stored in open data and table formats.
Impala is a modern, massively-distributed, massively-parallel, C++ query engine that lets you analyze, transform and combine data from a variety of data sources:
- Best of breed performance and scalability.
- Support for data stored in Apache Iceberg, HDFS, Apache HBase, Apache Kudu, Amazon S3, Azure Data Lake Storage, Apache Hadoop Ozone and more!
- Wide analytic SQL support, including window functions and subqueries.
- On-the-fly code generation using LLVM to generate lightning-fast code tailored specifically to each individual query.
- Support for the most commonly-used Hadoop file formats, including Apache Parquet and Apache ORC.
- Support for industry-standard security protocols, including Kerberos, LDAP and TLS.
- Apache-licensed, 100% open source.
More about Impala
The fastest way to try out Impala is a quickstart Docker container. You can try out running queries and processing data sets in Impala on a single machine without installing dependencies. It can automatically load test data sets into Apache Kudu and Apache Parquet formats and you can start playing around with Apache Impala SQL within minutes.
To learn more about Impala as a user or administrator, or to try Impala, please visit the Impala homepage. Detailed documentation for administrators and users is available at Apache Impala documentation.
If you are interested in contributing to Impala as a developer, or learning more about Impala's internals and architecture, visit the Impala wiki.
Supported Platforms
Impala only supports Linux at the moment. Impala supports x86_64 and has experimental support for arm64 (as of Impala 4.0). Impala Requirements contains more detailed information on the minimum CPU requirements.
Supported OS Distributions
Impala runs on Linux systems only. The supported distros are
- Ubuntu 16.04/18.04
- CentOS/RHEL 7/8
Other systems, e.g. SLES12, may also be supported but are not tested by the community.
Export Control Notice
This distribution uses cryptographic software and may be subject to export controls. Please refer to EXPORT_CONTROL.md for more information.
Build Instructions
See Impala's developer documentation to get started.
Detailed build notes has some detailed information on the project layout and build.