mirror of
https://github.com/apache/impala.git
synced 2025-12-19 18:12:08 -05:00
c97191b6a5
Historically Impala used the Python2 version that was available on the hosting platform, as long as that version was at least v2.6. This caused constant headache as all Python syntax had to be kept compatible with Python 2.6 (for Centos 6). It also caused a recent problem on Centos 8: here the system Python version was compiled with the system's GCC version (v8.3), which was much more recent than the Impala standard compiler version (GCC 4.9.2). When the Impala virtualenv was built, the system Python version supplied C compiler switches for models containing native code that were unknown for the Impala version of GCC, thus breaking virtualenv installation. This patch changes the Impala virtualenv to always use the Python2 version from the toolchain, which is built with the toolchain compiler. This ensures that - Impala always has a known Python 2.7 version for all its scripts, - virtualenv modules based on native code will always be installable, as the Python environment and the modules are built with the same compiler version. Additional changes: - Add an auto-use fixture to conftest.py to check that the tests are being run with Python 2.7.x - Make bootstrap_toolchain.py independent from the Impala virtualenv: remove the dependency on the "sh" library Tests: - Passed core-mode tests on CentOS 7.4 - Passed core-mode tests in Docker-based mode for centos:7 and ubuntu:16.04 Most content in this patch was developed but not published earlier by Tim Armstrong. Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b Reviewed-on: http://gerrit.cloudera.org:8080/15624 Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
54 lines
2.3 KiB
Bash
Executable File
54 lines
2.3 KiB
Bash
Executable File
# Licensed to the Apache Software Foundation (ASF) under one
|
|
# or more contributor license agreements. See the NOTICE file
|
|
# distributed with this work for additional information
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
# to you under the Apache License, Version 2.0 (the
|
|
# "License"); you may not use this file except in compliance
|
|
# with the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing,
|
|
# software distributed under the License is distributed on an
|
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
# KIND, either express or implied. See the License for the
|
|
# specific language governing permissions and limitations
|
|
# under the License.
|
|
|
|
# Sets up the python path for impala-python. This is needed because tests and other
|
|
# utility scripts depend on some modules external to infra/python/env.
|
|
# TODO: we should try to reduce our reliance on PYTHONPATH if possible.
|
|
#
|
|
# Setting USE_THRIFT11_GEN_PY will add Thrift 11 Python generated code rather than the
|
|
# default Thrift Python code.
|
|
# Used to allow importing testdata, test, etc modules from other scripts.
|
|
|
|
# ${IMPALA_HOME}/bin has bootstrap_toolchain.py, required by bootstrap_virtualenv.py
|
|
export PYTHONPATH=${IMPALA_HOME}:${IMPALA_HOME}/bin
|
|
|
|
# Generated Thrift files are used by tests and other scripts.
|
|
if [ -n "${USE_THRIFT11_GEN_PY:-}" ]; then
|
|
PYTHONPATH=${PYTHONPATH}:${IMPALA_HOME}/shell/build/thrift-11-gen/gen-py
|
|
else
|
|
PYTHONPATH=${PYTHONPATH}:${IMPALA_HOME}/shell/gen-py
|
|
fi
|
|
|
|
PYTHONPATH=${PYTHONPATH}:${IMPALA_HOME}/infra/python/env/lib
|
|
|
|
# There should be just a single version of python that created the
|
|
# site-packages directory. We find it by performing shell independent expansion
|
|
# of the following pattern:
|
|
# ${THRIFT_HOME}/python/lib{64,}/python*/site-packages
|
|
# Note: this could go wrong if we have used two different versions of
|
|
# Python to build Thrift on this machine, and the first version is not
|
|
# compatible with the second.
|
|
for PYTHON_DIR in ${THRIFT_HOME}/python/lib{64,}; do
|
|
[[ -d ${PYTHON_DIR} ]] || continue
|
|
for PKG_DIR in ${PYTHON_DIR}/python*/site-packages; do
|
|
PYTHONPATH=${PYTHONPATH}:${PKG_DIR}/
|
|
done
|
|
done
|
|
|
|
# Add Hive after Thrift because Hive supplies its own Thrift modules
|
|
PYTHONPATH=${PYTHONPATH}:${HIVE_HOME}/lib/py
|