mirror of
https://github.com/apache/impala.git
synced 2025-12-25 02:03:09 -05:00
c8e15e484c
Build C/C++ packages with toolchain GCC to avoid ABI compatibility issues. This requires a multi-step bootstrapping process: 1. install basic non-C/C++ packages into the virtualenv 2. use Python 2.7 from the virtualenv to bootstrap the toolchain 3. use toolchain gcc to build C/C++ packages 4. build the kudu-python package with toolchain gcc and Cython To avoid potentially pulling in cached versions of packages built with a different compiler, this patch also disables pip's caching. This should not have a significant effect on performance since we've enabled ccache and cache downloaded packages in infra/python/deps. Improve bootstrapping time significantly by using ccache and by parallelising the numpy build - the most expensive part of the install process. On a system with a warmed-up ccache, bootstrapping after deleting infra/python/env takes 1m16s. Previously it could take over 5m. Testing: Tested manually on Ubuntu 16.04 to confirm that it fixes the ABI problem mentioned in IMPALA-4593. Initially "import kudu" failed in my dev environment. After deleting infra/python/env and re-bootstrapping, "import kudu" succeeded. Also ran the standard test suite on CentOS 6 and built Impala on a range of platforms (CentOS 5,6,7; SLES 11,12; Debian 6,7; Ubuntu12.04,14.04,16.04) to make sure nothing broke. Change-Id: I9e807510eddeb354069e0478363f649a1c1b75cf Reviewed-on: http://gerrit.cloudera.org:8080/6218 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Impala Public Jenkins
15 lines
673 B
Plaintext
15 lines
673 B
Plaintext
To install new packages:
|
|
|
|
1) Add your package to deps/requirements.txt, or deps/compiled-requirements.txt if the
|
|
the package needs a C/C++ compiler to build . You should specify the version number
|
|
using the "foo == x.y.z" notation so future upgrades can be done automatically.
|
|
2) Run deps/download_requirements, it will download the package to the deps dir.
|
|
3) Run the "impala-python" command, this should detect that requirements.txt changed and
|
|
automatically rebuild the virtualenv.
|
|
4) Now in the python prompt, you should be able to import the new module.
|
|
|
|
To upgrade a package:
|
|
|
|
1) Edit deps/requirement.txt to use the version you need.
|
|
2) Go to step 2 above.
|