Adding a new column "Different error message" for the report that
shows the differences on e2e tests between the Calcite planner and
the current Impala planner. This new column will measure the test
cases that fail under both planners but have different error messages.
Change-Id: I897ed4c454d3c0f0873942ceec4edce719d63ac9
Reviewed-on: http://gerrit.cloudera.org:8080/22410
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Tested-by: Joe McDonnell <joemcdonnell@cloudera.com>
This introduces a Calcite report mode for pytests. The mode
changes the behavior for run_test_case() so that it continues
past failures and exports information about the test results
to JSON. The JSON files can then be processed into an HTML
summary by the bin/calcite_report_generator.py. The HTML has
multiple layers of reporting:
1. Reporting on individual tests shows the test section along
with the reported result
2. There are multiple aggregations of these individual results
to have clear summaries and organization for browing:
a. Level 1 is a report at the test function level
(e.g. query_test/test_foo.py::TestFoo::test_foo)
with links to the individaul results.
b. Level 2 is at the test file level (e.g. query_test/test_foo.py)
with links down to the test function level.
c. Level 3 is a top level view with a summary across all the tests
with links down to the test file level.
The errors are classified into different categories (e.g.
parse failures, analysis failures, result differences, etc).
In general, parse failures and unsupported features are lower
priority issues while result differences and runtime failures
are higher priority.
The report is designed to compare two different points in time
to see differences. For example, someone can run tests for a
baseline and then do a comparison run with a new commit.
Testing:
- Ran on the code change for IMPALA-13468 and browsed the results
- Ran tests normally and verified that they continue to work
Change-Id: I453c219c22b6cbc253574e0467d2c0d7b1fac092
Reviewed-on: http://gerrit.cloudera.org:8080/21866
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
