mirror of
https://github.com/apache/impala.git
synced 2025-12-19 18:12:08 -05:00
566df80891
This adds a Python 3 equivalent to the impala-python virtualenv base on the toolchain Python 3.7.16. This modifies bootstrap_virtualenv.py to support the two different modes. This adds py2-requirements.txt and py3-requirements.txt to allow some differences between the Python 2 and Python 3 virtualenvs. Here are some specific package changes: - allpairs is replaced with allpairspy, as allpairs did not support Python 3. - requests is upgraded slightly, because otherwise is has issues with idna==2.8. - pylint is limited to Python 3, because we are adding it and don't need it on both - flake8 is limited to Python 2, because it will take some work to switch to a version that works on Python 3 - cm_api is limited to Python 2, because it doesn't support Python 3 - pytest-random does not support Python 3 and it is unused, so it is removed - Bump the version of setuptool-scm to support Python 3 This adds impala-pylint, which can be used to do further Python 3 checks via --py3k. This also adds a bin/check-pylint-py3k.sh script to enforce specific py3k checks. The banned py3k warnings are specified in the bin/banned_py3k_warnings.txt. This is currently empty, but this can ratchet up the py3k strictness over time to avoid regressions. This pulls in a new toolchain with the fix for IMPALA-11956 to get Python 3.7.16. Testing: - Hand tested that the allpairs libraries produce the same results - The python3 virtualenv has no influence on regular tests yet Change-Id: Ica4853f440c9a46a79bd5fb8e0a66730b0b4efc0 Reviewed-on: http://gerrit.cloudera.org:8080/19567 Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com> Tested-by: Joe McDonnell <joemcdonnell@cloudera.com>
To install new packages: 1) Add your package to deps/requirements.txt, or deps/compiled-requirements.txt if the the package needs a C/C++ compiler to build . You should specify the version number using the "foo == x.y.z" notation so future upgrades can be done automatically. 2) Run deps/download_requirements, it will download the package to the deps dir. 3) Run the "impala-python" command, this should detect that requirements.txt changed and automatically rebuild the virtualenv. 4) Now in the python prompt, you should be able to import the new module. To upgrade a package: 1) Edit deps/requirement.txt to use the version you need. 2) Go to step 2 above.