mirror of
https://github.com/apache/impala.git
synced 2025-12-19 18:12:08 -05:00
ea0969a772
Python 3 changed the behavior of imports with PEP328. Existing imports become absolute unless they use the new relative import syntax. This adapts the impala-shell code to use absolute imports, fixing issues where it is imported from our test code. There are several parts to this: 1. It moves impala shell code into shell/impala_shell. This matches the directory structure of the PyPi package. 2. It changes the imports in the shell code to be absolute paths (i.e. impala_shell.foo rather than foo). This fixes issues with Python 3 absolute imports. It also eliminates the need for ugly hacks in the PyPi package's __init__.py. 3. This changes Thrift generation to put it directly in $IMPALA_HOME/shell rather than $IMPALA_HOME/shell/gen-py. This means that the generated Thrift code is rooted in the same directory as the shell code. 4. This changes the PYTHONPATH to include $IMPALA_HOME/shell and not $IMPALA_HOME/shell/gen-py. This means that the test code is using the same import paths as the pypi package. With all of these changes, the source code is very close to the directory structure of the PyPi package. As long as CMake has generated the thrift files and the Python version file, only a few differences remain. This removes those differences by moving the setup.py / MANIFEST.in and other files from the packaging directory to the top-level shell/ directory. This means that one can pip install directly from the source code. i.e. pip install $IMPALA_HOME/shell This also moves the shell tarball generation script to the packaging directory and changes bin/impala-shell.sh to use Python 3. This sorts the imports using isort for the affected Python files. Testing: - Ran a regular core job with Python 2 - Ran a core job with Python 3 and verified that the absolute import issues are gone. Change-Id: Ica75a24fa6bcb78999b9b6f4f4356951b81c3124 Reviewed-on: http://gerrit.cloudera.org:8080/22330 Reviewed-by: Riza Suminto <riza.suminto@cloudera.com> Reviewed-by: Michael Smith <michael.smith@cloudera.com> Tested-by: Riza Suminto <riza.suminto@cloudera.com>
54 lines
2.2 KiB
Bash
Executable File
54 lines
2.2 KiB
Bash
Executable File
#!/bin/bash
|
|
#
|
|
# Licensed to the Apache Software Foundation (ASF) under one
|
|
# or more contributor license agreements. See the NOTICE file
|
|
# distributed with this work for additional information
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
# to you under the Apache License, Version 2.0 (the
|
|
# "License"); you may not use this file except in compliance
|
|
# with the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing,
|
|
# software distributed under the License is distributed on an
|
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
# KIND, either express or implied. See the License for the
|
|
# specific language governing permissions and limitations
|
|
# under the License.
|
|
|
|
set -euo pipefail
|
|
|
|
# This script runs the impala shell from a dev environment.
|
|
PYTHONPATH=${PYTHONPATH:-}
|
|
SHELL_HOME=${IMPALA_SHELL_HOME:-${IMPALA_HOME}/shell}
|
|
|
|
# ${IMPALA_HOME}/bin has bootstrap_toolchain.py, required by bootstrap_virtualenv.py
|
|
PYTHONPATH=${PYTHONPATH}:${IMPALA_HOME}/bin
|
|
PYTHONPATH=${PYTHONPATH}:${SHELL_HOME}
|
|
|
|
export LD_LIBRARY_PATH=":$(PYTHONPATH=${PYTHONPATH} \
|
|
python "$IMPALA_HOME/infra/python/bootstrap_virtualenv.py" \
|
|
--print-ld-library-path)"
|
|
|
|
IMPALA_PY_DIR="$(dirname "$0")/../infra/python"
|
|
IMPALA_PY3_ENV_DIR="${IMPALA_PY_DIR}/env-gcc${IMPALA_GCC_VERSION}-py3"
|
|
# Allow overriding the python executable
|
|
IMPALA_PYTHON_EXECUTABLE="${IMPALA_PYTHON_EXECUTABLE:-${IMPALA_PY3_ENV_DIR}/bin/python3}"
|
|
|
|
# Note that this uses the external system python executable
|
|
PYTHONPATH=${PYTHONPATH} python "${IMPALA_PY_DIR}/bootstrap_virtualenv.py"
|
|
|
|
# Enable remote debugging if port was specified via environment variable
|
|
if [[ ${IMPALA_SHELL_DEBUG_PORT:-0} -ne 0 ]]; then
|
|
echo "installing debugpy if needed"
|
|
${IMPALA_PY_ENV_DIR}/bin/pip install debugpy
|
|
echo "impala python shell waiting for remote debugging connection on port" \
|
|
"${IMPALA_SHELL_DEBUG_PORT}"
|
|
EXTRA_ARGS=" -m debugpy --listen ${IMPALA_SHELL_DEBUG_PORT} --wait-for-client"
|
|
fi
|
|
|
|
# This uses the python executable in the impala python env
|
|
PYTHONIOENCODING='utf-8' PYTHONPATH=${PYTHONPATH} \
|
|
exec "${IMPALA_PYTHON_EXECUTABLE}" ${EXTRA_ARGS:-} -m "impala_shell.impala_shell" "$@"
|