mirror of
https://github.com/apache/impala.git
synced 2026-01-31 09:00:19 -05:00
4186727fe6
Like IMPALA-8369, this patch adds a compatibility shim in fe so that Impala can interoperate with Hive 3.1.2. we need adds a new Metastoreshim class under compat-apache-hive-3 directory. These shim classes implement method which are different in cdp-hive-3 vs apache-hive-3 and are used by front end code. At the build time, based on the environment variable IMPALA_HIVE_DIST_TYPE one of the two shims is added to as source using the fe/pom.xml build plugin. Some codes that directly use Hive 4 APIs need to be ignored in compilation, eg. fe/src/main/java/org/apache/impala/catalog/metastore/. Use Maven profile to ignore some codes, profile will automatically activated based on the IMPALA_HIVE_DIST_TYPE. Testing: 1. Code compiles and runs against both HMS-3 and ASF-HMS-3 2. Ran full-suite of tests against HMS-3 3. Running full-tests against ASF-HMS-3 will need more work supporting Tez in the mini-cluster (for dataloading) and HMS transaction support. This will be on-going effort and test failures on ASF-Hive-3 will be fixed in additional sub-tasks. Notes: 1. Patch uses a custom build of Apache Hive to be deployed in mini-cluster. This build has the fixes for HIVE-21569, HIVE-20038. This hack will be added to the build script in additional sub-tasks. Change-Id: I9f08db5f6da735ac431819063060941f0941f606 Reviewed-on: http://gerrit.cloudera.org:8080/17774 Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
75 lines
2.1 KiB
Bash
Executable File
75 lines
2.1 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
#
|
|
# Licensed to the Apache Software Foundation (ASF) under one
|
|
# or more contributor license agreements. See the NOTICE file
|
|
# distributed with this work for additional information
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
# to you under the Apache License, Version 2.0 (the
|
|
# "License"); you may not use this file except in compliance
|
|
# with the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing,
|
|
# software distributed under the License is distributed on an
|
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
# KIND, either express or implied. See the License for the
|
|
# specific language governing permissions and limitations
|
|
# under the License.
|
|
|
|
# This script is used to repair service startup and task running problems that occur when
|
|
# Apache Hive 3 is integrated.
|
|
# Repair method:
|
|
# - Replace jar
|
|
# - Backport the patch (Hive 3.x has not been released for a long time, apply the patch
|
|
# as a transitional solution until the new version is released).
|
|
|
|
set -euo pipefail
|
|
. $IMPALA_HOME/bin/report_build_error.sh
|
|
setup_report_build_error
|
|
|
|
if [[ "${USE_APACHE_HIVE}" != true ]]; then
|
|
exit 0
|
|
fi
|
|
|
|
# Cache applied patches
|
|
PATCHED_CACHE_FILE="$HIVE_SRC_DIR/.patched"
|
|
if [ ! -f "$PATCHED_CACHE_FILE" ]; then touch "$PATCHED_CACHE_FILE"; fi
|
|
# Apache Hive patch dir
|
|
HIVE_PARCH_DIR="${IMPALA_HOME}/testdata/cluster/hive"
|
|
|
|
# Apply the patch and save the patch name to .patched
|
|
function apply_patch {
|
|
p="$1"
|
|
status=1
|
|
while IFS= read -r line
|
|
do
|
|
if [ "$line" == "$p" ]; then
|
|
status=0
|
|
break
|
|
fi
|
|
done < $PATCHED_CACHE_FILE
|
|
if [ $status = "1" ] ;then
|
|
echo "Apply patch: $p"
|
|
patch -p1 < ${HIVE_PARCH_DIR}/$p
|
|
echo $p >> $PATCHED_CACHE_FILE
|
|
fi
|
|
}
|
|
|
|
# 1. Fix HIVE-22915
|
|
echo "Fix HIVE-22915"
|
|
rm $HIVE_HOME/lib/guava-*jar
|
|
cp $HADOOP_HOME/share/hadoop/hdfs/lib/guava-*.jar $HIVE_HOME/lib/
|
|
|
|
# 2. Apply patches
|
|
pushd "$HIVE_SRC_DIR"
|
|
for file in `ls ${HIVE_PARCH_DIR}/patch*.diff | sort`
|
|
do
|
|
p=$(basename $file)
|
|
apply_patch $p
|
|
done
|
|
|
|
# 3. Repackage the hive submodules affected by the patch
|
|
|
|
popd
|