When hitting a DCHECK/CHECK the daemons do not write minidumps. This is
caused by glog's own stack unwinding mechanism, which catches SIGABRT
and removes all other handlers before aborting.
This change bumps the glog version to include a patch, which backports a
change from glog, which only resets the SIGABRT handler, if it is the
one installed by glog itself.
cda16b3443
Change-Id: I08e6b83af1b4ff1b8c916fe6c9052b88b760e188
Reviewed-on: http://gerrit.cloudera.org:8080/3286
Reviewed-by: Lars Volker <lv@cloudera.com>
Tested-by: Lars Volker <lv@cloudera.com>
The breakpad custom cluster tests expect to run in exhaustive only, but
the option to do that isn't passed appropriately. That means they aren't
being run at all in Jenkins.
Fix the problem by setting a common set of arguments in run-all-tests.sh
and passing them along as needed.
Change-Id: I57f9ab6235f99e6a4d5fb50b87ceba3584b6b6d2
Reviewed-on: http://gerrit.cloudera.org:8080/3307
Reviewed-by: Silvius Rus <srus@cloudera.com>
Reviewed-by: Michael Brown <mikeb@cloudera.com>
Tested-by: Michael Brown <mikeb@cloudera.com>
Apparently, some pre-historic packaging script still relies
on build_thirdparty.sh to work around some problem related to
snappy. The build team cannot quite tell me why they cannot
use the toolchain instead so let's just revert the removal
of this precious script for now.
Change-Id: I5e6f9d3b86623ddeac6c4a1632c26ac50c555684
Reviewed-on: http://gerrit.cloudera.org:8080/3342
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Michael Ho <kwho@cloudera.com>
This patch adds a script bin/cppclean.sh that runs cppclean
over the BE code and writes the output to stdout.
cppclean is a tool for detecting various issues with C++ code:
https://github.com/myint/cppclean
To simplify the above step this change also modifies CMakeLists.txt
to allow dumping of all include paths to a file, like this:
cmake -DDUMP_INCLUDE_PATHS=output_file.txt
Change-Id: I5cd064344bdf47d77058fb570f034bb324308103
Reviewed-on: http://gerrit.cloudera.org:8080/3237
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Alex Behm <alex.behm@cloudera.com>
This change removes the option to build without specifying
the environment variable $IMPALA_TOOLCHAIN. By default, if
it's not set, sourcing impala-config.sh will set it to
$IMPALA_HOME/toolchain. A user can override it by setting
$IMPALA_TOOLCHAIN to his/her own toolchain directory. The
user can also set $SKIP_TOOLCHAIN_BOOTSTRAP to true to
avoid running the toolchain bootstrapping script (e.g. a
particular component in toolchain is at a version not
checked into S3).
$IMPALA_TOOLCHAIN holds some third party binaries which
Impala relies on. They can be compiled from source in the
native toolchain which is public. This commit also removes
build_thirdparty.sh as it's no longer used.
By default, Impala will be built with the compiler in
$IMPALA_TOOLCHAIN but this option can be overridden by
setting environment variable $USE_SYSTEM_GCC to 1.
Change-Id: I42b60e99fb9caf1294be7ab242856ca3b9a5ab73
Reviewed-on: http://gerrit.cloudera.org:8080/3259
Reviewed-by: Michael Ho <kwho@cloudera.com>
Tested-by: Michael Ho <kwho@cloudera.com>
These come with significant memory overhead, meaning that the memory
usage of the debug build diverges significantly from the release build.
We should disable them by default. They can be enable by setting
ENABLE_IMPALA_IR_DEBUG_INFO=true.
Change-Id: Ia5426fe3f8be0b7a100c0c3683c8ef1eaf507146
Reviewed-on: http://gerrit.cloudera.org:8080/3223
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
Currently the default minidump location is /tmp/impala-minidumps, which can be wiped on
reboot on various distributions. This change moves the default location to
FLAGS_log_dir/minidumps/$daemon. The additional trailing $daemon folder is kept to prevent
name collisions in case of local test clusters and strangely configured installations.
For local test clusters the minidumps will be written to
$IMPALA_HOME/logs/cluster/minidumps/{catalogd,impalad,statestored}.
Change-Id: Idecf5a314bfb8b0870e8aa4819c4fb39a107702f
Reviewed-on: http://gerrit.cloudera.org:8080/3171
Reviewed-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com>
Tested-by: Internal Jenkins
This will bring in two changes in Thrift patch 7:
1. THRIFT-1248: fix TMemoryBuffer pointer arithmetic (fixed in Thrift 0.9.3)
2. Fix TMemoryBuffer overflow (THRIFT-3821, not fixed in 0.9.3)
Change-Id: I1e97b27d163627aa0f82ceff9e1d82400a4dac07
Reviewed-on: http://gerrit.cloudera.org:8080/3166
Reviewed-by: Huaisi Xu <hxu@cloudera.com>
Tested-by: Internal Jenkins
Maven has been downloading the postgres JDBC driver
all along. So, let's use the one in fe/target/dependency
instead of the one in thirdparty.
Change-Id: I76bce18fd308890e66615c8d08d5e58f02a8a132
Reviewed-on: http://gerrit.cloudera.org:8080/3232
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
This reverts commit f8dd5413b65d30646c3745dfc738ed812d50a51f and
effectively re-adds commit 9248dcb70478b8f93f022893776a0960f45fdc28. The
difference between this patch and its original is that I fixed the
changes introduced in infra/python/bootstrap_virtualenv.py to be
python2.4-compatible:
- removed the use of str.format(), preferring a str.join() pattern
- removed the call of the exit() builtin to prefer sys.exit()
The only testing I did for this patch was to ensure
CDH Impala-packaging-on-demand works.
Change-Id: I02ed97473868eacf45b25abe89b41e6fa2fce325
Reviewed-on: http://gerrit.cloudera.org:8080/3160
Reviewed-by: Michael Brown <mikeb@cloudera.com>
Tested-by: Internal Jenkins
Before this patch, the Breakpad minidump collection script expected the
start_time and end_time to be specified in epoch seconds UTC. This
patch changes this format to milliseconds (as requested by CM).
Change-Id: I9b91bffbf0d4ab37753566ea9c31bbb01ac41623
Reviewed-on: http://gerrit.cloudera.org:8080/3163
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com>
This patch integrates Jacoco into Impala's FE test runs
for getting a code coverage report.
The instrumentation and reporting functionality is disabled
by default, and must be enabled explicitly, e.g., like this:
mvn test -DcodeCoverage
The code coverage report is stored in this location:
$IMPALA_HOME/logs/fe_tests/coverage
With additional changes, Jacoco can also be used to get code
coverage reports for our end-to-end tests, but that is left
for future work.
Change-Id: Id5e4f1b8afb91210d40622aadd3d21d7ed94c2a7
Reviewed-on: http://gerrit.cloudera.org:8080/3151
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
Boost library header is already included in the toolchain.
Also removes the environment variable IMPALA_MIN_BOOST_VERSION
and standardizes on the boost library version in toolchain.
Change-Id: I297edac7053964bfa113e0d5bf411fa3934b3796
Reviewed-on: http://gerrit.cloudera.org:8080/3159
Reviewed-by: Michael Ho <kwho@cloudera.com>
Tested-by: Internal Jenkins
Changes:
1) Add the python Kudu module to the virtualenv. Building the virtualenv
is much slower now because Cython and numpy are required. To help with
the rebuild time --no-cache was removed. That option was added to help
when using the dev version of impyla, the version number would be the
same but the module contents were different and the cache used the old
module contents.
2) Add some py.test fixtures to help create Kudu and Impala connections.
Change-Id: I8e5e22b38d5bd09a36238e66a69aa42d1a941de7
Reviewed-on: http://gerrit.cloudera.org:8080/2855
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
Previously boost related symbols (and others) would get defined in the
Kudu client stub with a non-functional implementation. If these
implementations were used at runtime they would crash Impala.
Change-Id: I54292095692ce38c255a8df48cf8f3f655d797b0
Reviewed-on: http://gerrit.cloudera.org:8080/2864
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
Add two scripts:
collect_minidumps.py - generates a compressed tarball that contains
minidumps generated by Breakpad.
generate_minidump_collection_testdata.py - generates testdata for the
above script.
Change-Id: I85b3643133e28eca07507ac2a79acbf73128456f
Reviewed-on: http://gerrit.cloudera.org:8080/2997
Reviewed-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com>
Reviewed-by: Michael Brown <mikeb@cloudera.com>
Tested-by: Internal Jenkins
Ninja resolves dependencies much faster, so if only a couple of files
are changed "ninja -j ${IMPALA_BUILD_THREADS} impalad" returns within a
second or two, while make can take tens of seconds to resolve all the
dependencies.
This requires ninja to be installed. It is widely available, e.g. in the
ninja-build package on Ubuntu.
Ninja can be enabled by passing "-ninja" to buildall.sh or
make_impala.sh. The same targets should work as with make.
The default Ninja status output is fairly terse. It can be customised
with an environment variable. E.g. I have
export NINJA_STATUS="[%u to run/%r running/%f finished] "
Also fixes a bug in make_impala.sh where invalid arguments were ignored.
Change-Id: I2cea479615fe850c98d30110de043ecb6358dcda
Reviewed-on: http://gerrit.cloudera.org:8080/2923
Tested-by: Internal Jenkins
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
This changes add breakpad crash handling support to catalogd, impalad,
and statestored. The destination folder for minidump files can be
configured via the 'minidump_path' command line flag. Leaving it empty
will disable minidump generation. The daemons will rotate minidump
files. The number of files to keep can be configured with the
'max_minidumps' command line flag.
Change-Id: I7a37a38488716ffe34296f3490ae291bbb7228d6
Reviewed-on: http://gerrit.cloudera.org:8080/2028
Reviewed-by: Lars Volker <lv@cloudera.com>
Tested-by: Internal Jenkins
Sometimes it takes a while to kill Impala. We should understand why this
is the case, but in the meantime let's increase the timeout to reduce
build failures.
Change-Id: Idc309ecf1a6936fab5a80464888a8dec465706ad
Reviewed-on: http://gerrit.cloudera.org:8080/2878
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
In custom cluster tests, the Impala mini-cluster is restarted as part of
the setup phase of every test, which means this is done more than 10
times. This means that on the 11th and subsequent start, some logs are
rotated out, since the -max_log_files default is 10. This is a problem
if one of the earlier custom cluster tests failed, and there is no
access to the impalad et al logs during the test.
Plumb setting of -max_log_files to catalogd, statestored, and impalad
through start-impala-cluster.py via environment variable
IMPALA_MAX_LOG_FILES. Keep its default to 10 so as not to blow up the
size of log directories, except when running the custom cluster tests.
When running those tests, set IMPALA_MAX_LOG_FILES to 0 to preserve all
logs. Only allow one test run's logs to exist in the directory at a
time.
Change-Id: Iefbb2a8616adcb0cd2fb838505117e0e9ba39083
Reviewed-on: http://gerrit.cloudera.org:8080/2759
Reviewed-by: Michael Brown <mikeb@cloudera.com>
Tested-by: Internal Jenkins
This is the same as the previous LLVM upgrade patch, except we've
removed the libtinfo dependency, so we assume we're building against an
LLVM that doesn't require that.
This requires various changes for Impala to be fully functional with the
new version of LLVM.
The original JIT was removed from LLVM, we need to switch to the new
MCJIT API and implementation.
MCJIT only supports module-at-a-time compilation, so the module must
be finalised before any compilation happens. We didn't depend on the
old behaviour deeply, but various small fixes were required.
MCJIT requires that every IR module has a name.
We relied on the old JIT's workaround for the __dso_handle symbol,
which we have to emulate for MCJIT with a customer memory manager
until we can get rid of global initialisers in cross-compiled code.
LLVM made a number of incompatible API changes and reorganised headers.
Clang took over responsibility for padding structs by marking structs
as packed and inserting bytes so that members are aligned correctly
(previously it relies LLVM aligning struct members based on the
target's alignment rules). This means Impala also needs to manually
pad its structs since clang-emitted structs look to LLVM like they have
do not need to be inlined.
Our inlining pass would require some modification to work and is
redundant with LLVM's inlining pass, so was removed along with the
unused subexpr elimination pass.
There were various issues with __builtin_add_overflow and
__builtin_mul_overflow that are newly available in LLVM 3.8.
First, LLVM emitted a call to a function in libclang_rt, which
we don't link in and has symbols that conflict with
the gcc runtime library. Second, the performance actually regressed
by using the builtins (I tested this manually by copying across the
definition of the required function).
Change-Id: I60b18a40a2df3f1adf326721f0df2a639d53a7c2
Reviewed-on: http://gerrit.cloudera.org:8080/2866
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
This will bring in two patches:
1. Fix some compiler warnings
2. Enable TLSv1.1 and TLSv1.2
Change-Id: I39764e7d8566c692b8cc657daf72c082d9199ce4
Reviewed-on: http://gerrit.cloudera.org:8080/2863
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
Reverting until we can sort out libtinfo build dependencies on various
OSes.
This reverts commit 1e77048be06aeb511e3483193db4257c8dbc7cf3.
Change-Id: I281b0b040941d9e4e6a5199c5d228471ad8c031c
Reviewed-on: http://gerrit.cloudera.org:8080/2857
Tested-by: Internal Jenkins
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
This requires various changes for Impala to be fully functional with the
new version of LLVM.
The original JIT was removed from LLVM, we need to switch to the new
MCJIT API and implementation.
MCJIT only supports module-at-a-time compilation, so the module must
be finalised before any compilation happens. We did't depend on the
old behaviour deeply, but various small fixes were required.
MCJIT requires that every IR module has a name.
We relied on the old JIT's workaround for the __dso_handle symbol,
which we have to emulate for MCJIT with a customer memory manager
until we can get rid of global initialisers in cross-compiled code.
LLVM made a number of incompatible API changes and reorganised headers.
Clang took over responsibility for padding structs by marking structs
as packed and inserting bytes so that members are aligned correctly
(previously it relies LLVM aligning struct members based on the
target's alignment rules). This means Impala also needs to manually
pad its structs since clang-emitted structs look to LLVM like they have
do not need to be inlined.
Our inlining pass would require some modification to work and is
redundant with LLVM's inlining pass, so was removed along with the
unused subexpr elimination pass.
LLVM now depends on another system library libtinfo, so we use
llvm-config to get the required system libs directly.
There were various issues with __builtin_add_overflow and
__builtin_mul_overflow that are newly available in LLVM 3.8.
First, LLVM emitted a call to a function in libclang_rt, which
we don't link in and has symbols that conflict with
the gcc runtime library. Second, the performance actually regressed
by using the builtins (I tested this manually by copying across the
definition of the required function).
Change-Id: I17d7afd05ad3b472a0bfe035bfc3daada5597b2d
Reviewed-on: http://gerrit.cloudera.org:8080/2486
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
This change whitelists the supported filesystems which can be set
as Default FS for Impala to run on.
This patch configures Impala to use S3 as the default filesystem, rather
than a secondary filesystem as before.
Change-Id: I2f45bef6c94ece634045acb906d12591587ccfed
Reviewed-on: http://gerrit.cloudera.org:8080/1121
Reviewed-by: anujphadke <aphadke@cloudera.com>
Tested-by: Internal Jenkins
This is a drop-in replacement. There have been several performance
improvements in Snappy since the 1.0.5 release.
Change-Id: I681bd18bc9add210c9b592ff81e25618e437ca7e
Reviewed-on: http://gerrit.cloudera.org:8080/2827
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
The directory structure of the newer Kudu toolchain artifacts has
changed. Now the root directory is split into /release and /debug. A few
little updates are needed to the build and service scripts.
Since the toolchain no longer provides stubs for platforms that Kudu
doesn't support the stubs need to be generated. This will be done as
part of the toolchain bootstrapping.
Also this upgrades Kudu to 0.8 RC1.
Developers will need to run bin/create-test-configuration.sh after
pulling in this change. Otherwise the Kudu service will fail to start.
Change-Id: I625903bd92afece0ad819a96fc275d5812b5eb2a
Reviewed-on: http://gerrit.cloudera.org:8080/2720
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
As a temporary workaround to some Kudu issues, Kudu can now be disabled
by setting KUDU_IS_SUPPORTED=false before sourcing impala-config.sh.
That should disable all use of Kudu.
Please do not use this without first making sure the issue you have run
into is a known issue. If people use this without ever raising awareness
of the issues, the problems will never go away.
Change-Id: Ie0b529c436418617b01c73bc917bfdf0a85c5440
Reviewed-on: http://gerrit.cloudera.org:8080/2736
Tested-by: Internal Jenkins
Reviewed-by: Casey Ching <casey@cloudera.com>
Previously Kudu would only be started when the test configuration was
the standard mini-cluster. That led to failures during data loading when
testing without the mini-cluster (ex: local file system). Kudu doesn't
require any other services so now it'll be started for all test
environments.
Change-Id: I92643ca6ef1acdbf4d4cd2fa5faf9ac97a3f0865
Reviewed-on: http://gerrit.cloudera.org:8080/2690
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
If SKIP_TOOLCHAIN_BOOTSTRAP is set, toolchain bootstrap is skipped. This
means that even if you are running on a supported OS, your custom-built
toolchain artifacts will always be used.
Also use Ubuntu 14.04 toolchain artifacts for Ubuntu 15.10.
I have been using the artifacts locally for a while and it has been
working fine.
Change-Id: If3bae187cc8a829c693711482c0ec656e41b7bf2
Reviewed-on: http://gerrit.cloudera.org:8080/2665
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
Bug: Division by zero.
Starting only the statestored and catalogd can be
useful for debugging purposes. For example, it is
often convenient to start a single customized impalad
with start-impalad.sh, but that requires having
the statestored and catalogd already up.
Change-Id: I9abe40de6c6caea26b6faa03b7495f25cb07e0ac
Reviewed-on: http://gerrit.cloudera.org:8080/2666
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
The stubs in Impala broke during the merge commit. This commit removes
the stubs in hopes of improving robustness of the build. The original
problem (Kudu clients are only available for some OSs) is now addressed
by moving the stubbing into a dummy Kudu client. The dummy client only
allows linking to succeed, if any client method is called, Impala will
crash. Before calling any such method, Kudu availability must be
checked.
Change-Id: I4bf1c964faf21722137adc4f7ba7f78654f0f712
Reviewed-on: http://gerrit.cloudera.org:8080/2585
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
All logs, test results and SQL files generated during data
loading and testing are now consolidated under a single new
directory $IMPALA_HOME/logs. The goal is to simplify archiving
in Jenkins runs and debugging.
The new structure is as follows:
$IMPALA_HOME/logs/cluster
- logs of Hadoop components and Impala
$IMPALA_HOME/logs/data_loading
- logs and SQL files produced in data loading
$IMPALA_HOME/logs/fe_tests
- logs and test output of Frontend unit tests
$IMPALA_HOME/logs/be_tests
- logs and test output of Backend unit tests
$IMPALA_HOME/logs/ee_tests
- logs and test output of end-to-end tests
$IMPALA_HOME/logs/custom_cluster_tests
- logs and test output of custom cluster tests
I tested this change with a full data load which
was successful.
Change-Id: Ief1f58f3320ec39d31b3c6bc6ef87f58ff7dfdfa
Reviewed-on: http://gerrit.cloudera.org:8080/2456
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
With this commit the bootstrat_toolchain.py script can work on Ubuntu
15.04 systems by using the 14.04 prebuilt artifacts.
Change-Id: Ie61576cb3dc350420cfd327d85cdcd028dd0032c
Reviewed-on: http://gerrit.cloudera.org:8080/2283
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: Internal Jenkins
Switches the gperftools version from 2.0 to 2.5 which is
also updated in the native-toolchain. The unmodified source
is also checked into thirdparty for those not using the
toolchain.
This commit reverts "CDH-38434: Fix Impala packaging build"
(commit 5666ef84977c4b92dec5b10ed71bbe36740a50c7) now that
the toolchain dependencies have been built for sles12.
Change-Id: I3fdc5091dfa4557968bf1a40f7e6d3eab91e7c15
Reviewed-on: http://gerrit.cloudera.org:8080/2581
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
Switches the gperftools version from 2.0 to 2.5 which is
also updated in the native-toolchain. The unmodified source
is also checked into thirdparty for those not using the
toolchain.
Change-Id: Ic06dd692c4c045db1275fca9c59e267c909599a3
Reviewed-on: http://gerrit.cloudera.org:8080/2509
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Internal Jenkins
There were a few review items pointed out on the review only
version of the final impala-kudu merge. Since that patch was
a pure mechanical patch those are addressed here.
Change-Id: Ibc4b30180a8f23394c7afc32b32668b05f142eff
Reviewed-on: http://gerrit.cloudera.org:8080/2545
Reviewed-by: David Ribeiro Alves <david.alves@cloudera.com>
Tested-by: Internal Jenkins
1) Add Ubuntu 12 to the unsupported OSs list.
2) Update Kudu sink stub.
3) Don't try to download Kudu if it isn't supported.
Change-Id: I6412ea0c79c9f2a2e3285b532372076ca437400d
Reviewed-on: http://gerrit.cloudera.org:8080/2547
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Casey Ching <casey@cloudera.com>
This is for review purposes only. This patch will be merged with David's
big merge patch.
Changes:
1) Make Kudu compilation dependent on the OS since not all OSs support
Kudu.
2) Only run Kudu related tests when Kudu is supported (see #1).
3) Look for Kudu locally, but in a different location. To use a local
build of Kudu, set KUDU_BUILD_DIR to the path Kudu was built in and
set KUDU_CLIENT_DIR to the path KUDU was installed in.
Example:
git clone https://github.com/cloudera/kudu.git
...build 3rd party etc...
mkdir -p $KUDU_BUILD_DIR
cd $KUDU_BUILD_DIR
cmake <path to Kudu source dir>
make
DESTDIR=$KUDU_CLIENT_DIR make install
4) Look for Kudu in the toolchain if not using a local Kudu build.
5) Add Kudu service startup scripts. The Kudu in the toolchain is
actually a parcel that has been renamed (the contents were not
modified in any way), that mean the Kudu service binaries are there.
Those binaries are now used to run the Kudu service.
Change-Id: I3db88cbd27f2ea2394f011bc8d1face37411ed58
