mirror of
https://github.com/apache/impala.git
synced 2026-02-01 03:00:22 -05:00
13fbe510c0
This upgrades GCC and libstdc++ to version 7.5.0. There
have been ABI changes since 4.9.2, so this means that
the native-toolchain produced with the new compiler is
not interoperable with one produced by the old compiler.
To allow that transition, IMPALA_TOOLCHAIN_PACKAGES_HOME
is now a subdirectory of IMPALA_TOOLCHAIN
(toolchain-packages-gcc${IMPALA_GCC_VERSION}) to distinguish
it from the old packages.
Some Python packages in the impala-python virtualenv are
compiled using the toolchain GCC and now use the new ABI.
This leads to two changes:
1. When constructing the LD_LIBRARY_PATH for impala-python,
we include the GCC libstdc++ libraries. Otherwise, certain
Python packages that use C++ fail on older OSes like Centos 7.
This fixes IMPALA-9804.
2. Since developers work on various branches, this changes
the virtualenv's directory location to a directory with
the GCC version in the name. This allows the virtualenv
built with GCC 7 to coexist with the current virtualenv
built with GCC 4.9.2. The location for the old virtualenv is
${IMPALA_HOME}/infra/python/env. The new location is
${IMPALA_HOME}/infra/python/env-gcc${IMPALA_GCC_VERSION}. This
required updating several impala-python scripts.
There are various odds-and-ends related to the transition:
1. Due to the small string optimization, the size of std::string
changed, which means that various data structures also changed
in size. This required updating some static asserts.
2. There is a bug in clang-tidy that reports a use-after-free
for some code using std::shared_ptr. Clang is not modeling
the shared_ptr correctly, so it is a false-positive. As a workaround,
this disables the clang-analyzer-cplusplus.NewDelete diagnostic.
3. Various small compilation fixes (includes, etc).
Performance testing:
- Ran single-node performance tests on TPC-H for the following
configurations:
- TPC-H Parquet scale 30 with normal configurations
- TPC-H Parquet scale 30 with codegen disabled
- TPC-H Kudu scale 10
None found any significant regressions. Full results are
posted on the JIRA.
- Ran single-node performance tests on targeted-perf scale 10.
No significant regressions.
- The size of binaries (impalad, etc) is slightly smaller with the new GCC:
GCC 4.9.2 release impalad binary: 545664
GCC 7.5.0 release impalad binary: 539900
- Compilation in DEBUG mode is roughly 15-25% faster
Functional testing:
- Ran core jobs, exhaustive release jobs, UBSAN
Change-Id: Ia0beb2b618ba669c9699f8dbc0c52d1203d004e4
Reviewed-on: http://gerrit.cloudera.org:8080/16045
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
56 lines
2.2 KiB
Bash
Executable File
56 lines
2.2 KiB
Bash
Executable File
#!/bin/bash
|
|
#
|
|
# Licensed to the Apache Software Foundation (ASF) under one
|
|
# or more contributor license agreements. See the NOTICE file
|
|
# distributed with this work for additional information
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
# to you under the Apache License, Version 2.0 (the
|
|
# "License"); you may not use this file except in compliance
|
|
# with the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing,
|
|
# software distributed under the License is distributed on an
|
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
# KIND, either express or implied. See the License for the
|
|
# specific language governing permissions and limitations
|
|
# under the License.
|
|
|
|
set -euo pipefail
|
|
|
|
# This script runs the impala shell from a dev environment.
|
|
PYTHONPATH=${PYTHONPATH:-}
|
|
SHELL_HOME=${IMPALA_SHELL_HOME:-${IMPALA_HOME}/shell}
|
|
|
|
# ${IMPALA_HOME}/bin has bootstrap_toolchain.py, required by bootstrap_virtualenv.py
|
|
PYTHONPATH=${PYTHONPATH}:${IMPALA_HOME}/bin
|
|
|
|
# Default version of thrift for the impala-shell is thrift >= 0.11.0.
|
|
PYTHONPATH=${PYTHONPATH}:${SHELL_HOME}/build/thrift-11-gen/gen-py
|
|
IMPALA_THRIFT_PY_VERSION=${IMPALA_THRIFT11_VERSION}
|
|
|
|
THRIFT_PY_ROOT="${IMPALA_TOOLCHAIN_PACKAGES_HOME}/thrift-${IMPALA_THRIFT_PY_VERSION}"
|
|
|
|
export LD_LIBRARY_PATH=":$(PYTHONPATH=${PYTHONPATH} \
|
|
python "$IMPALA_HOME/infra/python/bootstrap_virtualenv.py" \
|
|
--print-ld-library-path)"
|
|
|
|
IMPALA_PY_DIR="$(dirname "$0")/../infra/python"
|
|
IMPALA_PY_ENV_DIR="${IMPALA_PY_DIR}/env-gcc${IMPALA_GCC_VERSION}"
|
|
IMPALA_PYTHON_EXECUTABLE="${IMPALA_PY_ENV_DIR}/bin/python"
|
|
|
|
for PYTHON_LIB_DIR in ${THRIFT_PY_ROOT}/python/lib{64,}; do
|
|
[[ -d ${PYTHON_LIB_DIR} ]] || continue
|
|
for PKG_DIR in ${PYTHON_LIB_DIR}/python*/site-packages; do
|
|
PYTHONPATH=${PYTHONPATH}:${PKG_DIR}/
|
|
done
|
|
done
|
|
|
|
# Note that this uses the external system python executable
|
|
PYTHONPATH=${PYTHONPATH} python "${IMPALA_PY_DIR}/bootstrap_virtualenv.py"
|
|
|
|
# This uses the python executable in the impala python env
|
|
PYTHONIOENCODING='utf-8' PYTHONPATH=${PYTHONPATH} \
|
|
exec "${IMPALA_PYTHON_EXECUTABLE}" ${SHELL_HOME}/impala_shell.py "$@"
|