This patch migrate query_test/test_kudu.py to use hs2 client protocol.
Here are the steps taken:
- Override default_test_protocol() to return 'hs2'.
See documentation in ImpalaTestSuite about what this method does.
- Remove usage of deprecated cursor and unique_cursor fixture.
- Replace all direct ImpalaTestSuite.client usage with helper
function call such as execute_query() or execute_query_using_vector().
- Remove all "SET" query invocation and replace it with passing
exec_option dictionary to helper method.
- Replace veryfing kudu modified / inserted rows from reading query
output to reading runtime profile counters.
- Add HS2_TYPES section at test cases where only TYPES exist.
- Remove all drop_impala_table_after_context() calls and replace it with
proper use of unique_database fixture.
KuduTestSuite is fixed with hs2 protocol dimension. Meanwhile,
CustomKuduTest is fixed to use beeswax protocol dimension until proper
migration can be done.
Added following convenience methods:
- ImpalaTestSuite.default_test_protocol() to allow individual test
class to override its default test procol.
- ImpylaHS2ResultSet.tuples() to access the raw HS2 result set that is
a list of tuples.
This patch also added several literal constants around test vector
dimension to help with traceability.
Fixed a bug where "SHOW PARTITIONS" via hs2 over kudu table will shows
NULL number of #Replicas because TResultRowBuilder does not have
overload for int type value. Adjust numFiles variable inside
HdfsTable.getTableStats() from int to long to match Type.BIGINT of
column '#Files'.
Fixed py.test classes that does not inherit BaseTestSuite. Fixed flake8
issues in test_statestore.py.
Testing:
- Run and pass all tests extended from KuduTestSuite in exhaustive mode.
Change-Id: I5f38baf5a0bbde1a1ad0bb4666c300f4f3cabd33
Reviewed-on: http://gerrit.cloudera.org:8080/22358
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This takes steps to make Python 2 behave like Python 3 as
a way to flush out issues with running on Python 3. Specifically,
it handles two main differences:
1. Python 3 requires absolute imports within packages. This
can be emulated via "from __future__ import absolute_import"
2. Python 3 changed division to "true" division that doesn't
round to an integer. This can be emulated via
"from __future__ import division"
This changes all Python files to add imports for absolute_import
and division. For completeness, this also includes print_function in the
import.
I scrutinized each old-division location and converted some locations
to use the integer division '//' operator if it needed an integer
result (e.g. for indices, counts of records, etc). Some code was also using
relative imports and needed to be adjusted to handle absolute_import.
This fixes all Pylint warnings about no-absolute-import and old-division,
and these warnings are now banned.
Testing:
- Ran core tests
Change-Id: Idb0fcbd11f3e8791f5951c4944be44fb580e576b
Reviewed-on: http://gerrit.cloudera.org:8080/19588
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Tested-by: Joe McDonnell <joemcdonnell@cloudera.com>
This commit adds the necessary tooling to automate diagnostics
collection for Impala daemons. Following diagnostics are supported.
1. Native core dump (+ shared libs)
2. GDB/Java thread dump (pstack + jstack)
3. Java heap dump (jmap)
4. Minidumps (using breakpad) *
5. Profiles
Given the required inputs, the script outputs a zip compressed
impala diagnostic bundle with all the diagnostics collected.
The script can be run manually with the following command.
python collect_diagnostics.py --help
Tested with python 2.6 and later.
* minidumps collected here correspond to the state of the Impala
process at the time this script is triggered. This is different
from collect_minidumps.py which archives the entire minidump
directory.
Change-Id: I166e726f1dd1ce81187616e4f06d2404fa379bf8
Reviewed-on: http://gerrit.cloudera.org:8080/10056
Reviewed-by: Philip Zeyliger <philip@cloudera.com>
Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com>
Tested-by: Bharath Vissapragada <bharathv@cloudera.com>
A couple of things donot work in python2.6
-- Multiple with statements in the same context
-- shutil.make_archive()
I need a little more time to test the fix with python2.6.
Meanwhile, reverting this to unblock others. I'll resubmit
the fix when I'm confident that it works with python2.6
This reverts commit 2883c99500.
Change-Id: I221ede9d5eb4d89ea20992cc27a8284803af3223
Reviewed-on: http://gerrit.cloudera.org:8080/9872
Reviewed-by: Michael Ho <kwho@cloudera.com>
Tested-by: Michael Ho <kwho@cloudera.com>
This commit adds the necessary tooling to automate diagnostics
collection for Impala daemons. Following diagnostics are supported.
1. Native core dump (+ shared libs)
2. GDB/Java thread dump (pstack + jstack)
3. Java heap dump (jmap)
4. Minidumps (using breakpad) *
5. Profiles
Given the required inputs, the script outputs a zip compressed
impala diagnostic bundle with all the diagnostics collected.
The script can be run manually with the following command.
python collect_diagnostics.py --help
* minidumps collected here correspond to the state of the Impala
process at the time this script is triggered. This is different
from collect_minidumps.py which archives the entire minidump
directory.
Change-Id: Ib29caec7c3be5b6a31e60461294979c318300f64
Reviewed-on: http://gerrit.cloudera.org:8080/9815
Reviewed-by: Lars Volker <lv@cloudera.com>
Reviewed-by: Philip Zeyliger <philip@cloudera.com>
Tested-by: Impala Public Jenkins
