jprdonnelly/impala
1
0
Fork 0
mirror of https://github.com/apache/impala.git synced 2025-12-19 18:12:08 -05:00
Code Issues Projects Releases Wiki Activity

IMPALA-12825: Install thrift into the impala-python virtualenv

Browse Source 
impala-python currently gets its Thrift from the toolchain
by adding the appropriate Thrift toolchain directories to
the PYTHONPATH. This is a problem when switching to Python 3,
because the toolchain Thrift was built with Python 2 and
this can produce complicated bugs. In general, it is also
not a good idea to get Python dependencies from the toolchain.

This switches to installing Thrift into the impala-python
virtualenv, which lets the different Python versions have
their own copy of compiled files.

Testing:
 - Ran a core job

Change-Id: Ib36e8a1ce8d446b69b08e81ea458f95c158e28f5
Reviewed-on: http://gerrit.cloudera.org:8080/21046
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Reviewed-by: Wenzhe Zhou <wzhou@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This commit is contained in:
Joe McDonnell
2024-02-21 10:47:57 -08:00
parent ade98362c8
commit 5071f54a4c
4 changed files with 8 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
README-build.md
View File

@@ -35,7 +35,7 @@ can do so through the environment variables and scripts listed below.
| IMPALA_JDK_VERSION | "system" | Set to 8 or 11 to select a system Java version. Default will set JAVA_HOME based on the javac symlink in PATH. |
| JAVA | "${JAVA_HOME}/bin/java" | Java binary location. |
| CLASSPATH | | See bin/set-classpath.sh for details. |
| PYTHONPATH | Will be changed to include: "${IMPALA_HOME}/shell/gen-py" "${IMPALA_HOME}/testdata" "${THRIFT_PY_HOME}/python/lib/python2.7/site-packages" |
| PYTHONPATH | | See bin/set-pythonpath.sh for details. |
## Source Directories for Impala

9
bin/impala-config.sh
View File

@@ -214,9 +214,12 @@ export IMPALA_SIMBA_JDBC_DRIVER_VERSION=42-2.6.32.1041
# Thrift related environment variables.
# IMPALA_THRIFT_POM_VERSION is used to populate IMPALA_THRIFT_JAVA_VERSION and
# thrift.version in java/pom.xml.
# If upgrading IMPALA_THRIFT_PY_VERSION, remember to also upgrade thrift version in
# shell/ext-py and shell/packaging/requirements.txt. IMPALA_THRIFT_PY_VERSION is used
# with Impyla and for the thrift compiler.
# IMPALA_THRIFT_PY_VERSION is used to find the thrift compiler to produce
# the generated Python code. The code that uses the generated Python code gets
# the corresponding Thrift runtime library by pip installing thrift (and does not
# respect this version). If upgrading IMPALA_THRIFT_PY_VERSION, also upgrade the
# thrift version in shell/ext-py, shell/packaging/requirements.txt, and
# infra/python/deps/requirements.txt.
export IMPALA_THRIFT_CPP_VERSION=0.16.0-p6
unset IMPALA_THRIFT_CPP_URL
if $USE_APACHE_HIVE; then

14
bin/set-pythonpath.sh
View File

@@ -28,17 +28,3 @@ export PYTHONPATH=${IMPALA_HOME}:${IMPALA_HOME}/bin
PYTHONPATH=${PYTHONPATH}:${IMPALA_HOME}/shell/gen-py
PYTHONPATH=${PYTHONPATH}:${IMPALA_HOME}/infra/python/env-gcc${IMPALA_GCC_VERSION}/lib
# There should be just a single version of python that created the
# site-packages directory. We find it by performing shell independent expansion
# of the following pattern:
# ${THRIFT_PY_HOME}/python/lib{64,}/python*/site-packages
# Note: this could go wrong if we have used two different versions of
# Python to build Thrift on this machine, and the first version is not
# compatible with the second.
for PYTHON_DIR in ${THRIFT_PY_HOME}/python/lib{64,}; do
[[ -d ${PYTHON_DIR} ]] || continue
for PKG_DIR in ${PYTHON_DIR}/python*/site-packages; do
PYTHONPATH=${PYTHONPATH}:${PKG_DIR}/
done
done

1
infra/python/deps/requirements.txt
View File

@@ -64,6 +64,7 @@ sqlparse == 0.3.1
texttable == 0.8.3
virtualenv == 16.7.10
avro==1.10.2
thrift==0.16.0
# Required for Kudu:
Cython == 0.29.14