mirror of
https://github.com/apache/impala.git
synced 2026-01-09 06:05:09 -05:00
8ad6d03310
Impala's admission controller relies on the YARN fair-scheduler.xml for configuration. That configuration is loaded using YARN directly (ie. as a library by the frontend). In Hadoop 3, a number of changes were made to the YARN resourcemanager which break Impala. While we eventually want to rethink the admission control configuration (IMPALA-4159), in the meantime we at least should avoid using unsupported YARN APIs. This patch removes the fe dependency on the YARN artifact 'hadoop-yarn-server-resourcemanager' which contains private APIs and isn't meant to be used as a library. A subset of the required code has been added in 'common/yarn-extras', taken from Hadoop 2.6.0 in CDH, e.g. see [1]. The code is added in packages 'org.apache.impala.*' instead of 'org.apache.yarn.*'. Some code could be copied as-is, those files are marked with the comment: //YARNUTIL: VANILLA Files that required some modifications are marked with: //YARNUTIL: MODIFIED Or, if all code except a dummy interface could be added: //YARNUTIL: DUMMY IMPL The goal the 'yarn-extras' is to make Impala's handling of the AC configuration self-sufficient, i.e. it shouldn't matter what version of Hadoop exists. As-is, this was tested and found to work when Hadoop 2.6 is installed. Because the yarn-extras/pom.xml still references hadoop-common, hadoop-yarn-common, and hadoop-yarn-api, additional testing will be required to ensure Impala using yarn-extras works when installed along side Hadoop 3. That testing for Hadoop 3 will be done later. Future changes will make any other changes required for existing code to work when Hadoop 3 is installed. Testing: * Ran existing tests on master. 1: https://www.cloudera.com/documentation/enterprise/release-notes/topics/cm_vd_cdh_package_tarball_512.html Change-Id: I7efdd8ebea298836ca2a82c0a4ae037ac9285bcf Reviewed-on: http://gerrit.cloudera.org:8080/8035 Reviewed-by: Matthew Jacobs <mjacobs@apache.org> Tested-by: Matthew Jacobs <mjacobs@apache.org>