This takes steps to make Python 2 behave like Python 3 as
a way to flush out issues with running on Python 3. Specifically,
it handles two main differences:
1. Python 3 requires absolute imports within packages. This
can be emulated via "from __future__ import absolute_import"
2. Python 3 changed division to "true" division that doesn't
round to an integer. This can be emulated via
"from __future__ import division"
This changes all Python files to add imports for absolute_import
and division. For completeness, this also includes print_function in the
import.
I scrutinized each old-division location and converted some locations
to use the integer division '//' operator if it needed an integer
result (e.g. for indices, counts of records, etc). Some code was also using
relative imports and needed to be adjusted to handle absolute_import.
This fixes all Pylint warnings about no-absolute-import and old-division,
and these warnings are now banned.
Testing:
- Ran core tests
Change-Id: Idb0fcbd11f3e8791f5951c4944be44fb580e576b
Reviewed-on: http://gerrit.cloudera.org:8080/19588
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Tested-by: Joe McDonnell <joemcdonnell@cloudera.com>
Recent testing showed that the pytests are not
respecting the log level and format set in
conftest.py's configure_logging(). It is using
the default log level of WARNING and the
default formatter.
The issue is that logging.basicConfig() is only
effective the first time it is called. The code
in lib/python/impala_py_lib/helpers.py does a
call to logging.basicConfig() at the global
level, and conftest.py imports that file. This
renders the call in configure_logging()
ineffective.
To avoid this type of confusion, logging.basicConfig()
should only be called from the main() functions for
libraries. This removes the call in lib/python/impala_py_lib
(as it is not needed for a library without a main function).
It also fixes up various other locations to move the
logging.basicConfig() call to the main() function.
Testing:
- Ran the end to end tests and custom cluster tests
- Confirmed the logging format
- Added an assert in configure_logging() to test that
the INFO log level is applied to the root logger.
Change-Id: I5d91b7f910b3606c50bcba4579179a0bc8c20588
Reviewed-on: http://gerrit.cloudera.org:8080/16679
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Some negative tests produce core dumps intentionally. We should have a
way of removing these as part of test cleanup.
For custom cluster tests, it's likely the cores may actually be generated
during the base class setup phase, which means it's too early for the
test fixture to really be useful. Such was the case with the test case
TestAuthorizationProvider::test_invalid_provider_flag. In this instance,
we had to add the same steps directly to the tests.
Testing done:
For test_invalid_provider_flag, I made sure I had pre-existing core files
in the IMPALA_HOME directory, then ran the test to confirm new cores were
removed.
-- 2019-11-06 19:53:27,303 INFO MainThread: Removing core.impalad.61852 created by test_invalid_provider_flag
-- 2019-11-06 19:53:27,375 INFO MainThread: Removing core.impalad.61856 created by test_invalid_provider_flag
-- 2019-11-06 19:53:27,450 INFO MainThread: Removing core.impalad.61849 created by test_invalid_provider_flag
...and then made sure the pre-existing cores were still present.
Change-Id: I778f27e820a6983894c1294d35627ddb04f5a51a
Reviewed-on: http://gerrit.cloudera.org:8080/14640
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
