This wraps each command executed by CMake with a wrapper that
generates a JUnitXML file if the command fails. If the command
succeeds, the wrapper does nothing. The wrapper applies to C++
compilation, linking, and custom shell commands (such as
building the frontend via maven). It does not apply to failures
coming from CMake itself. It can be disabled by setting
DISABLE_CMAKE_JUNITXML.
The command output can include Unicode (e.g. smart quotes for
g++), so this also updates generate_junitxml.py to handle
Unicode.
The wrapper interacts poorly with add_custom_command/add_custom_target
CMake commands that use 'cd directory && do_something', so this
switches those locations (in /docker) to use CMake's WORKING_DIRECTORY.
Testing:
- Verified it does not impact a successful build (including with
ccache and/or distcc).
- Verified it generates JUnitXML for C++ and Java compilation
failures.
- Verified it doesn't use the wrapper when DISABLE_CMAKE_JUNITXML
is set.
Change-Id: If71f2faf3ab5052b56b38f1b291fee53c390ce23
Reviewed-on: http://gerrit.cloudera.org:8080/12668
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Python 2.6's ElementTree.write() does not have an xml_declaration
argument, so junitxml_prune_notrun.py fails on python 2.6.
This fixes junitxml_prune_notrun.py by using minidom to write
the output. This mirrors how bin/generate_junitxml.py outputs
XML.
Verified that tests now pass on python 2.6 and python 2.7 does
not change.
Change-Id: I9ef8fb77b1ac8c51e3dfb6b04690ae9ccc490d62
Reviewed-on: http://gerrit.cloudera.org:8080/12479
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
There are 100+ backend tests and each requires 400+ MB
of disk space when statically linked (the default).
This requires a large amount of disk space and adds
considerable link time.
This introduces a framework to link multiple backend
tests into a single executable. Currently it does
this for several tests in be/src/util. It saves
about 10GB of space.
It maintains several of the same properties that
the current tests have:
1. "make <testname>" rebuilds that test.
2. It generates an executable shell script with the
same name as the original backend test that runs
the same subset of tests.
3. It generates JUnitXML and log files with names that
match the test name.
4. One can run the shell script with "--gtest_filter"
and run a subset of the tests.
5. ctest commands such as ctest -R continue to function.
It validates at build time that every test linked into
the unified executable is covered by an equivalent test
filter pattern. This means that every test in the unified
executable will run as part of normal testing.
Introducing the framework along with a limited number
of trial backend tests gives us a chance to evaluate
this change before continuing to convert tests.
Change-Id: Ia03ef38719b1fbc0fe2025e16b7b3d3dd4488842
Reviewed-on: http://gerrit.cloudera.org:8080/12124
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
With python 2.6, the syntax "{}".format(1) doesn't work:
$docker run centos:6 python -c 'print "{}".format(1)'
Traceback (most recent call last):
File "<string>", line 1, in <module>
ValueError: zero length field name in format
generate_junitxml was using this incantation and failing.
I've updated the syntax to be py2.6-friendly, and tested
it like so:
$docker run -v $(pwd):/mnt centos:6 bash -c "yum install -y python-argparse; /mnt/lib/python/impala_py_lib/jenkins/generate_junitxml.py --phase phase --step step --stdout out --stderr err; cat /extra_junit_xml_logs/*.xml"
[output from yum...]
Installed:
python-argparse.noarch 0:1.2.1-2.1.el6
Complete!
Generated: ./extra_junit_xml_logs/generate_junitxml.phase.step.20180904_18_04_56.xml
<?xml version="1.0" ?>
<testsuites errors="0" failures="0" tests="1" time="0.0">
<testsuite disabled="0" errors="0" failures="0" file="None" log="None" name="generate_junitxml.phase.step" skipped="0" tests="1" time="0" timestamp="2018-09-04 18:04:56+00:00" url="None">
<testcase classname="generate_junitxml.phase" name="step">
<system-out>
out
</system-out>
<system-err>
err
</system-err>
</testcase>
</testsuite>
</testsuites>
Change-Id: Ic0c1e837a9ed6c2d59906aed1d1098bde6f5d815
Reviewed-on: http://gerrit.cloudera.org:8080/11384
Reviewed-by: David Knupp <dknupp@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
The original patch for this Jira relied on a third party python lib
for generating Junit XML output, That proved to be limiting because
setting up the necessary virtualenv across a variety of dev and test
scenarios (private dev environment, jenkins.impala.io, and others)
proved to be confusing and messy.
This update to the script maintains the same functionality and the
same interface, but uses only the python standard library. A symlink
has also been added to Impala/bin for convenience.
Change-Id: I958ee0d8420b6a4197aaf0a7e0538a566332ea97
Reviewed-on: http://gerrit.cloudera.org:8080/11235
Reviewed-by: David Knupp <dknupp@cloudera.com>
Tested-by: David Knupp <dknupp@cloudera.com>
This patch adds a script to generate junit XML reports for arbitrary
build steps. It's also being used to seed the creation of an internal
python library for Impala development that can be pip installed into
a development environment.
Change-Id: If6024d74075ea69b8ee20d1fc3cc9c1ff821ba5b
Reviewed-on: http://gerrit.cloudera.org:8080/11128
Reviewed-by: David Knupp <dknupp@cloudera.com>
Tested-by: David Knupp <dknupp@cloudera.com>
