Before this patch, when writing pytest that exercise custom query option
values, we need to declare it by making new test dimension, followed by
deepcopying the original vector, and inserting the selected dimension
value into 'exec_option' dictionary in generated vector.
This patch simplify this steps by accounting dimensions that is intended
to be part of 'exec_option' and automatically combining them during
vector generation in test_vector.py. Such dimension should be registered
via the new ImpalaTestMatrix.add_exec_option_dimension() function.
function add_exec_option_dimension() in test_dimensions.py is renamed to
add_mandatory_exec_option() to make it consistent with the same
functionality in ImpalaTestMatrix and avoid confusion with the new
ImpalaTestMatrix.add_exec_option_dimension() function. Function name
add_exec_option_dimension() in test_dimensions.py is then repurposed as
a shorthand for ImpalaTestMatrix.add_exec_option_dimension().
The remaining changes for other pytest files will be done gradually.
Testing:
- Fix bug in TestIcebergV2Table and confirm that both True and False
value for 'disable_optimized_iceberg_v2_read' options are exercised.
- Run and pass all modified tests in this patch.
Change-Id: I3adba260990fccf4d2f2e7c8c4e4fadc6fd43fe1
Reviewed-on: http://gerrit.cloudera.org:8080/20625
Reviewed-by: Zoltan Borok-Nagy <boroknagyz@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
This takes steps to make Python 2 behave like Python 3 as
a way to flush out issues with running on Python 3. Specifically,
it handles two main differences:
1. Python 3 requires absolute imports within packages. This
can be emulated via "from __future__ import absolute_import"
2. Python 3 changed division to "true" division that doesn't
round to an integer. This can be emulated via
"from __future__ import division"
This changes all Python files to add imports for absolute_import
and division. For completeness, this also includes print_function in the
import.
I scrutinized each old-division location and converted some locations
to use the integer division '//' operator if it needed an integer
result (e.g. for indices, counts of records, etc). Some code was also using
relative imports and needed to be adjusted to handle absolute_import.
This fixes all Pylint warnings about no-absolute-import and old-division,
and these warnings are now banned.
Testing:
- Ran core tests
Change-Id: Idb0fcbd11f3e8791f5951c4944be44fb580e576b
Reviewed-on: http://gerrit.cloudera.org:8080/19588
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Tested-by: Joe McDonnell <joemcdonnell@cloudera.com>
This commit introduces optional asynchronous code generation.
Asynchronous code generation means that instead of waiting for codegen
to finish, the query starts in interpreted mode while codegen is done on
another thread.
All the function pointers that point to codegen'd functions are changed
to be atomic, wrapped in a CodegenFnPtr. These are initialised to
nullptr and as long as they are nullptr, the corresponding interpreted
functions are used (as before). When code generation is ready, the
funtion pointers are set by the codegen thread. No synchronisation is
needed as the function pointers are atomic and it is not a problem if,
at a given moment, only a subset of the codegen'd function pointers are
set and the rest are interpreted.
Asynchronous code generation can be turned on using the ASYNC_CODEGEN
boolean query option.
Testing:
- In exhaustive mode, a limited number of end-to-end tests are run in
async mode and with debug actions randomly delaying the codegen
thread and the main thread after starting codegen to test various
scenarios of relative timing. The number of such tests is kept
small to avoid increasing the running time of the tests by too much.
- Added a new end-to-end test, tests/query_test/test_async_codegen.py,
which tests three relative timings:
1. Async codegen finishes before query execution starts (only
codegen'd code runs).
2. Query execution finishes before async codegen finishes (only
interpreted code runs).
3. Async codegen finishes during query execution (both interpreted
and condegen'd code runs, switching to codegen from interpreted
mode.
Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b
Reviewed-on: http://gerrit.cloudera.org:8080/15105
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
