jprdonnelly/impala
1
0
Fork 0
mirror of https://github.com/apache/impala.git synced 2025-12-19 18:12:08 -05:00
Code Issues Projects Releases Wiki Activity

IMPALA-9668: Obey SKIP_TOOLCHAIN_BOOTSTRAP during virtualenv bootstrap

Browse Source 
IMPALA-9626 broke the use case where the toolchain binaries are not
downloaded from the native-toolchain S3 bucket, because
SKIP_TOOLCHAIN_BOOTSTRAP is set to true.

Fix this use case by checking SKIP_TOOLCHAIN_BOOTSTRAP in
bin/bootstrap_environment.py:
- if true: just check if the specified version of the Python binary is
  present at the expected toolchain location. If it is there, use it,
  otherwise throw an exception and abort the bootstrap process.
- in any other case: proceed to download the Python binary as in
  bootstrap_toolchain.py.

Test:
- simulate the custom toolchain setup by downloading the toolchain
  binaries from the S3 bucket, copying them to a separate directory,
  symlinking them into Impala/toolchain, then executing buildall.sh
  with SKIP_BOOTSTRAP_TOOLCHAIN set to "true".

Change-Id: Ic51b3c327b3cebc08edff90de931d07e35e0c319
Reviewed-on: http://gerrit.cloudera.org:8080/15759
Reviewed-by: Laszlo Gaal <laszlo.gaal@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This commit is contained in:
Laszlo Gaal
2020-04-20 14:39:47 +02:00
committed by Impala Public Jenkins
parent 7e1d774a15
commit b921d982b5
2 changed files with 8 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
bin/bootstrap_toolchain.py
View File

@@ -611,7 +611,7 @@ def get_toolchain_downloads():
toolchain_packages += map(ToolchainPackage,
["avro", "binutils", "boost", "breakpad", "bzip2", "cctz", "cmake", "crcutil",
"flatbuffers", "gdb", "gflags", "glog", "gperftools", "gtest", "libev",
"libunwind", "lz4", "openldap", "openssl", "orc", "protobuf",
"libunwind", "lz4", "openldap", "openssl", "orc", "protobuf", "python",
"rapidjson", "re2", "snappy", "thrift", "tpc-h", "tpc-ds", "zlib", "zstd"])
toolchain_packages += [ToolchainPackage("thrift",
explicit_version=os.environ.get("IMPALA_THRIFT11_VERSION"))]

9
infra/python/bootstrap_virtualenv.py
View File

@@ -50,6 +50,8 @@ from bootstrap_toolchain import ToolchainPackage
LOG = logging.getLogger(os.path.splitext(os.path.basename(__file__))[0])
SKIP_TOOLCHAIN_BOOTSTRAP = "SKIP_TOOLCHAIN_BOOTSTRAP"
DEPS_DIR = os.path.join(os.path.dirname(__file__), "deps")
ENV_DIR = os.path.join(os.path.dirname(__file__), "env")
@@ -192,7 +194,9 @@ def find_file(*paths):
def download_toolchain_python():
'''Grabs the Python implementation from the Impala toolchain, using the machinery from
bin/bootstrap_toolchain.py
bin/bootstrap_toolchain.py.
Skip the download if SKIP_TOOLCHAIN_BOOTSTRAP=true in the environment. In that case
only the presence of the Python executable is checked in the toolchain location.
'''
toolchain_root = os.environ.get("IMPALA_TOOLCHAIN")
@@ -201,7 +205,8 @@ def download_toolchain_python():
"Impala environment not set up correctly, make sure $IMPALA_TOOLCHAIN is set.")
package = ToolchainPackage("python")
package.download()
if not (os.environ.get(SKIP_TOOLCHAIN_BOOTSTRAP) == 'true'):
package.download()
python_cmd = os.path.join(package.pkg_directory(), "bin/python")
if not os.path.exists(python_cmd):
raise Exception("Unexpected error bootstrapping python from toolchain: {0} does not "