When running the shell in a terminal with live_progress=true, live
progress overwrites its output by using the ANSI up character to
rewrite lines with updated on the query progress. On Python 3,
we found that the updates to clear the live progress were overwriting
the actual output in the terminal. e.g.
+----------+
| count(*) |
+----------+
Fetched 1 row(s) in 5.20s
To avoid this, the live progress lines need to be fully flushed to stderr
before starting to output the result to stdout. This adds a flush call
in OverwritingStdErrOutputStream::clear() to force this.
Testing:
- Hand tested queries with live progress
- Added test that redirects stdout and stderr to the same file and
verifies that no ANSI up character comes after the query output
Change-Id: Id2e21224253f76b2a04767a57b3ade49ce2c914f
Reviewed-on: http://gerrit.cloudera.org:8080/22941
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Riza Suminto <riza.suminto@cloudera.com>
Python 3 changed the behavior of imports with PEP328. Existing
imports become absolute unless they use the new relative import
syntax. This adapts the impala-shell code to use absolute
imports, fixing issues where it is imported from our test code.
There are several parts to this:
1. It moves impala shell code into shell/impala_shell.
This matches the directory structure of the PyPi package.
2. It changes the imports in the shell code to be
absolute paths (i.e. impala_shell.foo rather than foo).
This fixes issues with Python 3 absolute imports.
It also eliminates the need for ugly hacks in the PyPi
package's __init__.py.
3. This changes Thrift generation to put it directly in
$IMPALA_HOME/shell rather than $IMPALA_HOME/shell/gen-py.
This means that the generated Thrift code is rooted in
the same directory as the shell code.
4. This changes the PYTHONPATH to include $IMPALA_HOME/shell
and not $IMPALA_HOME/shell/gen-py. This means that the
test code is using the same import paths as the pypi
package.
With all of these changes, the source code is very close
to the directory structure of the PyPi package. As long as
CMake has generated the thrift files and the Python version
file, only a few differences remain. This removes those
differences by moving the setup.py / MANIFEST.in and other
files from the packaging directory to the top-level
shell/ directory. This means that one can pip install
directly from the source code. i.e. pip install $IMPALA_HOME/shell
This also moves the shell tarball generation script to the
packaging directory and changes bin/impala-shell.sh to use
Python 3.
This sorts the imports using isort for the affected Python files.
Testing:
- Ran a regular core job with Python 2
- Ran a core job with Python 3 and verified that the absolute
import issues are gone.
Change-Id: Ica75a24fa6bcb78999b9b6f4f4356951b81c3124
Reviewed-on: http://gerrit.cloudera.org:8080/22330
Reviewed-by: Riza Suminto <riza.suminto@cloudera.com>
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Tested-by: Riza Suminto <riza.suminto@cloudera.com>
