mirror of
https://github.com/apache/impala.git
synced 2025-12-25 11:04:13 -05:00
ea0969a772
Python 3 changed the behavior of imports with PEP328. Existing imports become absolute unless they use the new relative import syntax. This adapts the impala-shell code to use absolute imports, fixing issues where it is imported from our test code. There are several parts to this: 1. It moves impala shell code into shell/impala_shell. This matches the directory structure of the PyPi package. 2. It changes the imports in the shell code to be absolute paths (i.e. impala_shell.foo rather than foo). This fixes issues with Python 3 absolute imports. It also eliminates the need for ugly hacks in the PyPi package's __init__.py. 3. This changes Thrift generation to put it directly in $IMPALA_HOME/shell rather than $IMPALA_HOME/shell/gen-py. This means that the generated Thrift code is rooted in the same directory as the shell code. 4. This changes the PYTHONPATH to include $IMPALA_HOME/shell and not $IMPALA_HOME/shell/gen-py. This means that the test code is using the same import paths as the pypi package. With all of these changes, the source code is very close to the directory structure of the PyPi package. As long as CMake has generated the thrift files and the Python version file, only a few differences remain. This removes those differences by moving the setup.py / MANIFEST.in and other files from the packaging directory to the top-level shell/ directory. This means that one can pip install directly from the source code. i.e. pip install $IMPALA_HOME/shell This also moves the shell tarball generation script to the packaging directory and changes bin/impala-shell.sh to use Python 3. This sorts the imports using isort for the affected Python files. Testing: - Ran a regular core job with Python 2 - Ran a core job with Python 3 and verified that the absolute import issues are gone. Change-Id: Ica75a24fa6bcb78999b9b6f4f4356951b81c3124 Reviewed-on: http://gerrit.cloudera.org:8080/22330 Reviewed-by: Riza Suminto <riza.suminto@cloudera.com> Reviewed-by: Michael Smith <michael.smith@cloudera.com> Tested-by: Riza Suminto <riza.suminto@cloudera.com>
80 lines
3.0 KiB
Bash
Executable File
80 lines
3.0 KiB
Bash
Executable File
#!/bin/bash
|
|
#
|
|
# Licensed to the Apache Software Foundation (ASF) under one
|
|
# or more contributor license agreements. See the NOTICE file
|
|
# distributed with this work for additional information
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
# to you under the Apache License, Version 2.0 (the
|
|
# "License"); you may not use this file except in compliance
|
|
# with the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing,
|
|
# software distributed under the License is distributed on an
|
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
# KIND, either express or implied. See the License for the
|
|
# specific language governing permissions and limitations
|
|
# under the License.
|
|
|
|
|
|
# This script makes a tarball of the Python-based shell that can be unzipped and
|
|
# run out-of-the-box with no configuration. The final tarball is left in
|
|
# ${IMPALA_HOME}/shell/build.
|
|
|
|
set -euo pipefail
|
|
. $IMPALA_HOME/bin/report_build_error.sh
|
|
setup_report_build_error
|
|
|
|
if [ "x${IMPALA_HOME}" == "x" ]; then
|
|
echo "\$IMPALA_HOME must be set"
|
|
exit 1
|
|
fi
|
|
|
|
if [ $# -lt 2 ]; then
|
|
echo "Must provide pypi package and at least one python interpreter"
|
|
exit 1
|
|
fi
|
|
|
|
PYPI_PACKAGE=$1
|
|
shift
|
|
SHELL_HOME=${IMPALA_HOME}/shell
|
|
BUILD_DIR=${SHELL_HOME}/build
|
|
TARBALL_ROOT=${BUILD_DIR}/impala-shell-${IMPALA_VERSION}
|
|
|
|
for PYTHON_EXE in $*; do
|
|
PYTHON_NAME=$(basename ${PYTHON_EXE})
|
|
PYTHON_VERSION=$(${PYTHON_EXE} -c 'import sys; \
|
|
print("{}.{}".format(sys.version_info.major, sys.version_info.minor))')
|
|
PYTHON_MAJOR_VERSION=$(${PYTHON_EXE} -c 'import sys; print(sys.version_info.major)')
|
|
# pip install the wheel into the external dependencies directory
|
|
PIP_CACHE="~/.cache/impala_pip/${PYTHON_NAME}"
|
|
BUILD_TMP_DIR="$(mktemp -d)"
|
|
|
|
echo "Deleting all files in ${TARBALL_ROOT}/install_py${PYTHON_VERSION}"
|
|
rm -rf ${TARBALL_ROOT}/install_py${PYTHON_VERSION} 2>&1 > /dev/null
|
|
echo "Installing for python ${PYTHON_VERSION}"
|
|
# Use pip that matches the major version
|
|
if [[ $PYTHON_MAJOR_VERSION == 2 ]]; then
|
|
source ${IMPALA_HOME}/shell/build/python2_venv/bin/activate
|
|
else
|
|
source ${IMPALA_HOME}/shell/build/python3_venv/bin/activate
|
|
fi
|
|
mkdir -p ${TARBALL_ROOT}/install_py${PYTHON_VERSION}
|
|
pip install --cache ${PIP_CACHE} \
|
|
--target ${TARBALL_ROOT}/install_py${PYTHON_VERSION} ${PYPI_PACKAGE}
|
|
# We don't need the impala-shell binary for the installation. It contains
|
|
# a weird shebang from the virtualenv, so it is worth removing it.
|
|
rm ${TARBALL_ROOT}/install_py${PYTHON_VERSION}/bin/impala-shell
|
|
# Cleanup temp build directory
|
|
rm -rf ${BUILD_TMP_DIR}
|
|
done
|
|
|
|
# Copy the impala-shell driver script into the tarball root
|
|
cp ${SHELL_HOME}/packaging/impala-shell ${TARBALL_ROOT}
|
|
|
|
pushd ${BUILD_DIR} > /dev/null
|
|
echo "Making tarball in ${BUILD_DIR}"
|
|
tar czf ${BUILD_DIR}/impala-shell-${IMPALA_VERSION}.tar.gz --exclude="*.pyc" \
|
|
./impala-shell-${IMPALA_VERSION}/ || popd 2>&1 > /dev/null
|