This takes steps to make Python 2 behave like Python 3 as
a way to flush out issues with running on Python 3. Specifically,
it handles two main differences:
1. Python 3 requires absolute imports within packages. This
can be emulated via "from __future__ import absolute_import"
2. Python 3 changed division to "true" division that doesn't
round to an integer. This can be emulated via
"from __future__ import division"
This changes all Python files to add imports for absolute_import
and division. For completeness, this also includes print_function in the
import.
I scrutinized each old-division location and converted some locations
to use the integer division '//' operator if it needed an integer
result (e.g. for indices, counts of records, etc). Some code was also using
relative imports and needed to be adjusted to handle absolute_import.
This fixes all Pylint warnings about no-absolute-import and old-division,
and these warnings are now banned.
Testing:
- Ran core tests
Change-Id: Idb0fcbd11f3e8791f5951c4944be44fb580e576b
Reviewed-on: http://gerrit.cloudera.org:8080/19588
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Tested-by: Joe McDonnell <joemcdonnell@cloudera.com>
custom_cluster/test_query_concurrency created two threads to poll the
query_plan debug webpage without joining them. Then they could outlive
the test itself and continued to poll the webpage while the minicluster
of the next test started up.
During startup of the coordinator, the ImpalaServer would register its
HTTP handlers with the webserver before registering itself with the
ExecEnv. When the incoming request from the polling threads called
GetClientRequestState(), that call would dereference a nullptr returned
by the ExecEnv, which would cause the process to crash.
To fix this we join the threads in test_query_concurrency before
returning from the test method.
To fix the underlying race that made the crash possible we change the
initialization order to register the HTTP handlers after the
ImpalaServer has been registered with the ExecEnv. We also add some
DCHECKs to make sure that we access the ImpalaServer only through a
singleton instance and that it is properly registered with the ExecEnv.
Change-Id: If22f71ab6edaf9a6b46afc0985c73dc4625b5103
Reviewed-on: http://gerrit.cloudera.org:8080/12019
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Fixes use of an undefined variable.
I saw the following message in a build failure, which
clearly wasn't intended:
MainThread: Debug webpage not yet available.
Exception in thread Thread-862:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File "/data/jenkins/workspace/impala-asf-2.x-exhaustive-rhel7/repos/Impala/tests/custom_cluster/test_query_concurrency.py", line 58, in poll_query_page
except e:
NameError: global name 'e' is not defined
Change-Id: If507409b8945b16a9510bb6195343eed7d8538fc
Reviewed-on: http://gerrit.cloudera.org:8080/10049
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Holding client_request_state_map_lock_ and CRS::lock_ together in certain
paths could potentially block the impalad from registering new queries.
The most common occurrence of this is while loading the webpage of a
query while the query planning is still in progress. Since we hold the
CRS::lock_ during planning, it blocks the web page from loading which
inturn blocks incoming queries by holding client_request_state_map_lock_.
This patch makes client_request_state_map_lock_ a terminal lock so that
we don't have interleaving locking with CRS::lock_.
Testing: Tested it locally by adding a long sleep in
JniFrontend.createExecRequest() and still was able to refresh the web UI
and run parallel queries. Also added a custom cluster test that does the
same sequence of actions by injecting a metadata loading pause.
Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Reviewed-on: http://gerrit.cloudera.org:8080/6707
Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com>
Tested-by: Impala Public Jenkins
