Commit Graph

84 Commits

Author SHA1 Message Date
Henry Robinson
23100102c0 IMPALA-4758: (2/2) Impala-side changes to build with latest gutil
Meant to be taken as a whole with the previous commit. This patch makes
the necessary code changes to Impala and the gutil/ library to fix all
compilation errors. Future upgrades to gutil/ should redo the work in
this commit.

* Remove kudu/ include prefix with command:

git grep -l "include \"kudu/" | xargs sed -i 's/include \"kudu\//include
\"/g'

* Change KUDU_GUTIL_* guards to be GUTIL_*

git grep -l KUDU_GUTIL | xargs sed -i 's/KUDU_GUTIL/GUTIL/g'

* Replace glog/logging.h with common/logging.h

git grep -l "glog/logging" | xargs sed -i 's/glog\/logging/common\/logging/g'

* Provide our own implementation of since-removed MonotonicNanos()
* Reinstate COMPILE_FLAGS argument to ADD_EXPORTABLE_LIBRARY,
  used by gutil.
* Replay overwritten parts of following commits:

a7c3f30 - Remove AMD Opteron Rev E workaround from atomicops
54194af - IMPALA-4631: don't use floating point operations for time unit
conversions
152c586 - Improve AtomicInt abstraction and implementation

* Comment out non-compiling deprecated function definitions in numbers.h
* Overwrite changes from 92fafa "Use more efficient gutil implementation
  of Log2Ceiling" in favour of implementing them in Impala code only.
* Couple of misc fixes.

Change-Id: I4ac21d7d6401f21fcdfdd1132b8f322bfba4bb80
Reviewed-on: http://gerrit.cloudera.org:8080/5688
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Impala Public Jenkins
2017-03-29 02:52:34 +00:00
Henry Robinson
5a333c47c5 Fix typo in Flatbuffers cmake module
Change-Id: I0786344b5485a92c02a246b543b6acda279e199c
Reviewed-on: http://gerrit.cloudera.org:8080/6398
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: Impala Public Jenkins
2017-03-15 03:57:40 +00:00
Dimitris Tsirogiannis
60c1c6e81b IMPALA-4966: Add flatbuffers to build
FlatBuffers version 1.6.0 is already included in the toolchain. This
commit adds it to the build system.

Change-Id: I2ca255ddf08ac846b454bfa1470ed67b1338d2b0
Reviewed-on: http://gerrit.cloudera.org:8080/6180
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: Impala Public Jenkins
2017-03-02 09:43:03 +00:00
Henry Robinson
60c41c4f0f IMPALA-4652: Add crcutil to build
Add crcutil, built from a git hash since there are no released versions,
to Impala's build.

crcutil is available at https://github.com/rurban/crcutil

FindCrcutil.cmake was taken from Apache Kudu.

Change-Id: I095d1c6b8e9e8f40cf62c1ecfdc880e708a72c28
Reviewed-on: http://gerrit.cloudera.org:8080/5660
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: Henry Robinson <henry@cloudera.com>
2017-01-12 23:50:14 +00:00
Henry Robinson
a81ad5eaab IMPALA-4651: Add LibEv to build
Add libev 4.20 to the Impala build. This is a dependency of KRPC.

FindLibEv.cmake was taken from Apache Kudu.

Change-Id: Iaf0646533592e6a8cd929a8cb015b83a7ea3008f
Reviewed-on: http://gerrit.cloudera.org:8080/5659
Tested-by: Impala Public Jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
2017-01-12 23:44:26 +00:00
Henry Robinson
ed0aa66ee1 IMPALA-4650: Allow protobuf to find non-system libraries and binaries
This change makes PROTOBUF_GENERATE_CPP able to pick up Protobuf
libraries and binaries that are found by CMake but not installed on the
system LD_LIBRARY_PATH.

Change-Id: I942b3f18e25e2abc5aac167412b65abb680d3c5a
Reviewed-on: http://gerrit.cloudera.org:8080/5658
Tested-by: Impala Public Jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
2017-01-12 05:18:33 +00:00
Henry Robinson
4b3fdc3301 IMPALA-4650: Add Protobuf to build
This patch adds Protobuf 2.6.1 to Impala's build, and bumps the
toolchain version so that the dependency is available. Protobuf is
unused in this commit, but is required for KRPC.

FindProtobuf.cmake includes some utility CMake methods to generate
source code from Protobuf definitions. It is taken from Kudu.

Change-Id: Ic9357fe0f201cbf7df1ba19fe4773dfb6c10b4ef
Reviewed-on: http://gerrit.cloudera.org:8080/5657
Tested-by: Impala Public Jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
2017-01-12 05:18:17 +00:00
Henry Robinson
44bb99a61d Add Kudu cmake utilities
This commit imports some CMake utility methods from Kudu, in preparation
for adding KRPC and its dependencies to Impala's build.

The methods are unused in this patch, but will be used both by
thirdparty dependencies (e.g. Protobuf) and by the Kudu libraries
themselves.

Some methods are stubbed out to make it easier to import Kudu's
CMakeLists.txt files without adding extra test targets etc. to Impala's
build.

Change-Id: Ibaae645d650ab1555452e4cc2574d6c84a90d941
Reviewed-on: http://gerrit.cloudera.org:8080/5656
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Impala Public Jenkins
2017-01-12 02:53:45 +00:00
Jim Apple
14891fe004 IMPALA-3676: Use clang as a static analysis tool
This patch adds a script to run clang-tidy over the whole code
base. It is a first step towards running clang-tidy over patches as a
tool to help users spot bugs before code review.

Because of the number of clang-tidy checks, this patch only addresses
some of them. In particular, only checks starting with 'clang' are
considered. Many of them which are flaky or not part of our style are
excluded from the analysis. This patch also exlcudes some checks which
are part of our current style but which would be too laborious to fix
over the entire codebase, like using nullptr rather than NULL.

This patch also fixes a number of small bugs found by clang-tidy.

Finally, this patch adds the class AlignedNew, the purpose of which is
to provide correct alignment on heap-allocated data. The global new
operator only guarantees 16-byte alignment. A class that includes a
member variable that must be aligned on a k-byte boundary for k>16 can
inherit from AlignedNew<k> to ensure correct alignment on the heap,
quieting clang's -Wover-aligned warning. (Static and stack allocation
are required by the standard to respect the alignment of the type and
its member variables, so no extra code is needed for allocation in
those places.)

Change-Id: I4ed168488cb30ddeccd0087f3840541d858f9c06
Reviewed-on: http://gerrit.cloudera.org:8080/4758
Reviewed-by: Jim Apple <jbapple@cloudera.com>
Tested-by: Internal Jenkins
2016-11-04 00:13:12 +00:00
Tim Armstrong
df680cfe3a IMPALA-4277: allow overriding of Hive/Hadoop versions/locations
This is to help with IMPALA-4277 to make it easier to build against
Hadoop/Hive distributions where the directory layout doesn't exactly
match our current CDH dependencies, or where we may want to
temporarily override a version without making a source change.

Change-Id: I7da10e38f9c4309f2d193dc25f14a6ea308c9639
Reviewed-on: http://gerrit.cloudera.org:8080/4720
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Internal Jenkins
2016-10-18 05:54:09 +00:00
Jim Apple
bd2947329e IMPALA-4110: Clean up issues found by Apache RAT.
Change-Id: I5bfe77f9a871018e7a67553ed270e2df53006962
Reviewed-on: http://gerrit.cloudera.org:8080/4361
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
2016-09-14 22:09:24 +00:00
Dan Hecht
ffa7829b70 IMPALA-3918: Remove Cloudera copyrights and add ASF license header
For files that have a Cloudera copyright (and no other copyright
notice), make changes to follow the ASF source file header policy here:

http://www.apache.org/legal/src-headers.html#headers

Specifically:
1) Remove the Cloudera copyright.
2) Modify NOTICE.txt according to
   http://www.apache.org/legal/src-headers.html#notice
   to follow that format and add a line for Cloudera.
3) Replace or add the existing ASF license text with the one given
   on the website.

Much of this change was automatically generated via:

git grep -li 'Copyright.*Cloudera' > modified_files.txt
cat modified_files.txt | xargs perl -n -i -e 'print unless m#Copyright.*Cloudera#i;'
cat modified_files_txt | xargs fix_apache_license.py [1]

Some manual fixups were performed following those steps, especially when
license text was completely missing from the file.

[1] https://gist.github.com/anonymous/ff71292094362fc5c594 with minor
    modification to ORIG_LICENSE to match Impala's license text.

Change-Id: I2e0bd8420945b953e1b806041bea4d72a3943d86
Reviewed-on: http://gerrit.cloudera.org:8080/3779
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
2016-08-09 08:19:41 +00:00
Michael Ho
86ff18eee9 IMPALA-3223: Removal of non-toolchain builds.
This change removes the option to build without specifying
the environment variable $IMPALA_TOOLCHAIN. By default, if
it's not set, sourcing impala-config.sh will set it to
$IMPALA_HOME/toolchain. A user can override it by setting
$IMPALA_TOOLCHAIN to his/her own toolchain directory. The
user can also set $SKIP_TOOLCHAIN_BOOTSTRAP to true to
avoid running the toolchain bootstrapping script (e.g. a
particular component in toolchain is at a version not
checked into S3).

$IMPALA_TOOLCHAIN holds some third party binaries which
Impala relies on. They can be compiled from source in the
native toolchain which is public. This commit also removes
build_thirdparty.sh as it's no longer used.

By default, Impala will be built with the compiler in
$IMPALA_TOOLCHAIN but this option can be overridden by
setting environment variable $USE_SYSTEM_GCC to 1.

Change-Id: I42b60e99fb9caf1294be7ab242856ca3b9a5ab73
Reviewed-on: http://gerrit.cloudera.org:8080/3259
Reviewed-by: Michael Ho <kwho@cloudera.com>
Tested-by: Michael Ho <kwho@cloudera.com>
2016-06-07 17:29:59 -07:00
Michael Ho
0b7ae6e4eb IMPALA-3223: Relocate squeasel and mustache directories
This change moves the source and header files of squeasel
and mustache to be/src/thirdparty. This is a step towards
removing thirdparty as a preparation to move to ASF.

There is also corresponding change to Impala-lzo to update
its include path.

Change-Id: I782e493bc28086a1587274b3c474ea6b6f201855
Reviewed-on: http://gerrit.cloudera.org:8080/3206
Reviewed-by: Michael Ho <kwho@cloudera.com>
Tested-by: Michael Ho <kwho@cloudera.com>
2016-05-31 23:31:41 -07:00
Michael Ho
9a5e701209 IMPALA-3223: Remove boost multiprecision in thirdparty.
Boost library header is already included in the toolchain.
Also removes the environment variable IMPALA_MIN_BOOST_VERSION
and standardizes on the boost library version in toolchain.

Change-Id: I297edac7053964bfa113e0d5bf411fa3934b3796
Reviewed-on: http://gerrit.cloudera.org:8080/3159
Reviewed-by: Michael Ho <kwho@cloudera.com>
Tested-by: Internal Jenkins
2016-05-23 08:40:19 -07:00
Tim Armstrong
2b61ae7f2a IMPALA-3534: allow overriding of CMAKE_CXX_COMPILER for ASAN
This makes it consistent with the regular toolchain and makes it easier
to use wrapper scripts like distcc.

Change-Id: I3ab488182c46f9ccb1850a0a2b064653e7e3da26
Reviewed-on: http://gerrit.cloudera.org:8080/3050
Reviewed-by: Jim Apple <jbapple@cloudera.com>
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
2016-05-12 23:06:36 -07:00
Tim Armstrong
1c704f3cfd IMPALA-3166: basic perf support and asm dumps for codegened code
Adds support for communicating function-level symbols to perf by writing
/tmp/perf-<pid>.data if the --perf_map=true argument is set. Perf must
be run under the same user as Impala. I.e. 'sudo perf top' does not
work. To get perf to work under a non-root user you will probably need
to disable some kernel security features that perf complains about:

sudo bash -c 'echo -1 > /proc/sys/kernel/perf_event_paranoid'
sudo bash -c 'echo 0 > /proc/sys/kernel/kptr_restrict'

Once you get it working you should see IR function names concatenated with
the fragment instance id in 'perf top'. 'perf annotate' does not work.

Implements --asm_module_dir, analogous to --opt_module_dir. We dump
disassembly to files there. Debug symbols are interleaved with the
assembly if they are available. I enabled them for the debug
build, now that we have some purpose for them.  In some cases
it would be useful to have them for the release build, but
they make the llvm module much larger so I haven't enabled them
there.

The asm dump for a random exception constructor looks like this:

Disassembly for __cxx_global_var_init.165:324bc8754182e7c6:22735c36d7a2bc0 (0x7f50f2140300):
        date_facet.hpp:date_facet.hpp:<invalid>:363:0
        date_facet.hpp:date_facet.hpp:<invalid>:363:58
0:              movabsq $0, %rax
10:             movb    (%rax), %cl
12:             cmpb    $0, %cl
15:             jne     17
        date_facet.hpp:date_facet.hpp:<invalid>:363:58
17:             movabsq $0, %rax
27:             movq    $1, (%rax)
        date_facet.hpp:date_facet.hpp:<invalid>:363:58
34:             retq

Change-Id: If25de61e46f4db005956686cddbd4d71a1424528
Reviewed-on: http://gerrit.cloudera.org:8080/2793
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
2016-05-12 14:18:03 -07:00
Tim Armstrong
5c56ec0997 Fix some ASAN compile warnings and remove redundant flags
Change-Id: I7b2772d917449ca747820641c56e65545f610b23
Reviewed-on: http://gerrit.cloudera.org:8080/3025
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
2016-05-12 14:18:02 -07:00
Lars Volker
c9df348c38 IMPALA-2686: Add breakpad crash handler to all daemons
This changes add breakpad crash handling support to catalogd, impalad,
and statestored. The destination folder for minidump files can be
configured via the 'minidump_path' command line flag. Leaving it empty
will disable minidump generation. The daemons will rotate minidump
files. The number of files to keep can be configured with the
'max_minidumps' command line flag.

Change-Id: I7a37a38488716ffe34296f3490ae291bbb7228d6
Reviewed-on: http://gerrit.cloudera.org:8080/2028
Reviewed-by: Lars Volker <lv@cloudera.com>
Tested-by: Internal Jenkins
2016-05-12 14:17:52 -07:00
Tim Armstrong
d6613e9531 IMPALA-775,IMPALA-3374: Upgrade LLVM to 3.8.0
This is the same as the previous LLVM upgrade patch, except we've
removed the libtinfo dependency, so we assume we're building against an
LLVM that doesn't require that.

This requires various changes for Impala to be fully functional with the
new version of LLVM.

The original JIT was removed from LLVM, we need to switch to the new
MCJIT API and implementation.

MCJIT only supports module-at-a-time compilation, so the module must
be finalised before any compilation happens. We didn't depend on the
old behaviour deeply, but various small fixes were required.

MCJIT requires that every IR module has a name.

We relied on the old JIT's workaround for the __dso_handle symbol,
which we have to emulate for MCJIT with a customer memory manager
until we can get rid of global initialisers in cross-compiled code.

LLVM made a number of incompatible API changes and reorganised headers.

Clang took over responsibility for padding structs by marking structs
as packed and inserting bytes so that members are aligned correctly
(previously it relies LLVM aligning struct members based on the
target's alignment rules). This means Impala also needs to manually
pad its structs since clang-emitted structs look to LLVM like they have
do not need to be inlined.

Our inlining pass would require some modification to work and is
redundant with LLVM's inlining pass, so was removed along with the
unused subexpr elimination pass.

There were various issues with __builtin_add_overflow and
__builtin_mul_overflow that are newly available in LLVM 3.8.
First, LLVM emitted a call to a function in libclang_rt, which
we don't link in and has symbols that conflict with
the gcc runtime library. Second, the performance actually regressed
by using the builtins (I tested this manually by copying across the
definition of the required function).

Change-Id: I60b18a40a2df3f1adf326721f0df2a639d53a7c2
Reviewed-on: http://gerrit.cloudera.org:8080/2866
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
2016-05-12 14:17:42 -07:00
Tim Armstrong
b4a9dfcc92 Revert "IMPALA-775,IMPALA-3374: Upgrade LLVM to 3.8.0"
Reverting until we can sort out libtinfo build dependencies on various
OSes.

This reverts commit 1e77048be06aeb511e3483193db4257c8dbc7cf3.

Change-Id: I281b0b040941d9e4e6a5199c5d228471ad8c031c
Reviewed-on: http://gerrit.cloudera.org:8080/2857
Tested-by: Internal Jenkins
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
2016-05-12 14:17:40 -07:00
Tim Armstrong
be415f380f IMPALA-775,IMPALA-3374: Upgrade LLVM to 3.8.0
This requires various changes for Impala to be fully functional with the
new version of LLVM.

The original JIT was removed from LLVM, we need to switch to the new
MCJIT API and implementation.

MCJIT only supports module-at-a-time compilation, so the module must
be finalised before any compilation happens. We did't depend on the
old behaviour deeply, but various small fixes were required.

MCJIT requires that every IR module has a name.

We relied on the old JIT's workaround for the __dso_handle symbol,
which we have to emulate for MCJIT with a customer memory manager
until we can get rid of global initialisers in cross-compiled code.

LLVM made a number of incompatible API changes and reorganised headers.

Clang took over responsibility for padding structs by marking structs
as packed and inserting bytes so that members are aligned correctly
(previously it relies LLVM aligning struct members based on the
target's alignment rules). This means Impala also needs to manually
pad its structs since clang-emitted structs look to LLVM like they have
do not need to be inlined.

Our inlining pass would require some modification to work and is
redundant with LLVM's inlining pass, so was removed along with the
unused subexpr elimination pass.

LLVM now depends on another system library libtinfo, so we use
llvm-config to get the required system libs directly.

There were various issues with __builtin_add_overflow and
__builtin_mul_overflow that are newly available in LLVM 3.8.
First, LLVM emitted a call to a function in libclang_rt, which
we don't link in and has symbols that conflict with
the gcc runtime library. Second, the performance actually regressed
by using the builtins (I tested this manually by copying across the
definition of the required function).

Change-Id: I17d7afd05ad3b472a0bfe035bfc3daada5597b2d
Reviewed-on: http://gerrit.cloudera.org:8080/2486
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
2016-05-12 14:17:40 -07:00
Matthew Jacobs
62dbdb06d0 IMPALA-3162: Upgrade to gperftools 2.5 (take 2)
Switches the gperftools version from 2.0 to 2.5 which is
also updated in the native-toolchain. The unmodified source
is also checked into thirdparty for those not using the
toolchain.

This commit reverts "CDH-38434: Fix Impala packaging build"
(commit 5666ef84977c4b92dec5b10ed71bbe36740a50c7) now that
the toolchain dependencies have been built for sles12.

Change-Id: I3fdc5091dfa4557968bf1a40f7e6d3eab91e7c15
Reviewed-on: http://gerrit.cloudera.org:8080/2581
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
2016-03-18 23:08:09 +00:00
Dan Hecht
4e7732bb0d CDH-38434: Fix Impala packaging build
Revert "IMPALA-3162: Upgrade to gperftools 2.5"

This reverts commit 8772ca0d1962daa54ac81559960e7bdd6115f6f1.

Change-Id: Ib12fb2dfa9aa1b024b01c56fc75ddb3a261f8811
Reviewed-on: http://gerrit.cloudera.org:8080/2564
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Internal Jenkins
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
2016-03-16 22:38:27 +00:00
Matthew Jacobs
26be352d4c IMPALA-3162: Upgrade to gperftools 2.5
Switches the gperftools version from 2.0 to 2.5 which is
also updated in the native-toolchain. The unmodified source
is also checked into thirdparty for those not using the
toolchain.

Change-Id: Ic06dd692c4c045db1275fca9c59e267c909599a3
Reviewed-on: http://gerrit.cloudera.org:8080/2509
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Internal Jenkins
2016-03-16 05:37:13 +00:00
Tim Armstrong
2334c7da6d IMPALA-2980: use LLVM without assertions for release build
This patch depends on the llvm-3.3-no-asserts-p1 build being added to
the native toolchain. I tested by building with my local modified
toolchain. A previous commit also disabled automatic bootstrapping of
the toolchain on build machines, so to download the new module
automatically, I changed the build scripts to always bootstrap, but to
skip downloading packages that were already present.

The logic is changed so that LLVM without assertions is always used,
except for debug builds which link against the libraries with
assertions built in.

We want to always use the same clang to generate the IR, so that the IR
we are testing in debug mode is the same as in release mode. This
requires separating the LLVM binaries search from the LLVM libraries
search. Also requires the root CMakeLists.txt to know about debug
versus release builds so it can decide which library to use, so I
refactored some of that logic too.

This change fixes the lock contention problem of IMPALA-2980 (since a
global lock is acquired only to check an assertion) and generally
improves codegen times. On a simple inner join query I saw
OptimizationTime reduced from ~240ms to ~150ms and PrepareTime reduced
from ~120ms to ~90ms.

Change-Id: I4977815a42c66a74e34ebb6e5cf3931f51ed461a
Reviewed-on: http://gerrit.cloudera.org:8080/2231
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
2016-02-19 00:03:23 -08:00
casey
7a0a5012be Add option to use non-default compiler
This allows distcc to be used.

Change-Id: Id0b45de13dca6c2e3c37df43a8d347337f756915
Reviewed-on: http://gerrit.cloudera.org:8080/2094
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
2016-02-10 02:17:44 +00:00
Martin Grund
dfc05657de Link Cyrus SASL dynamically in Impala
Until now, we would statically link SASL in Impala and simply hope
that it is ABI compatible to the version where we install
Impala. However, SASL is as well a security library and thus Impala
should not statically link it but rather implicitly depend on it.

The only complication of this patch is that there was an API breaking
change in SASL that this patch has to deal with as it compiles on
different platforms.

In addition, this patch changes the behavior and default of the
--sasl_path command line option. If the option is not set, we rely on
the automatic resolution of the SASL plugin path by the dynamic library
and only if the option is set, we will override it with the custom value.

I tested this patch on Ubuntu and CentOS 6 with the two different
versions and everything worked fine.

Change-Id: I0523b47f15a63ac385e9036c5b76d43a55bb6771
Reviewed-on: http://gerrit.cloudera.org:8080/1692
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
2016-01-13 05:37:32 +00:00
Martin Grund
83199ec837 IMPALA-2761: OS X: Misc Fixes
Miscellaneous fixes to ensure portability across OS X and Linux. Most
prominent is the change to use gutils wallclock for the time calculation
in blocking join node.

In addition, configures impala-config.sh to pick-up the right versions of
the thirdparty libraries for OS X and makes sure the build is invoked
correctly.

Change-Id: Iedc62e81957ec2d80548b84bb8ce1a88acf0564c
Reviewed-on: http://gerrit.cloudera.org:8080/1616
Reviewed-by: Martin Grund <mgrund@cloudera.com>
Readability: Martin Grund <mgrund@cloudera.com>
Tested-by: Internal Jenkins
2015-12-15 22:09:42 +00:00
Martin Grund
65d7c1c41a Making ASAN use LLVM 3.7 instead of LLVM trunk
In addition, this adds a helper annotation that disables ASAN annotation
for a known and accepted leak in GetDocumentRoot(). Since LLVM 3.7 is
stricter when it comes to use function parameters as immediate values
for inline assembly, the SSE4 helper methods have been changed to use
templates instead.

Change-Id: I364c1612bfe3bba60817c8f453ccb4c11cef5fe7
Reviewed-on: http://gerrit.cloudera.org:8080/1265
Reviewed-by: Martin Grund <mgrund@cloudera.com>
Tested-by: Internal Jenkins
2015-10-28 22:24:41 +00:00
Martin Grund
daaedd7cf6 Propagating Avro patch-level and LLVM patch-level for toolchain.
Avro patch-level from IMPALA-1136 and IMPALA-2161.
LLVM patch-level from Mac OS X related changes.

Change-Id: Ie7450a57b5bc1cdcd43a0d11466e7726ba033da3
Reviewed-on: http://gerrit.cloudera.org:8080/1324
Tested-by: Internal Jenkins
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
2015-10-28 18:53:05 +00:00
Martin Grund
5afd5bc8f6 Toolchain Cleanup and ASAN Improvements
This patch provides the last fixes to finally enable the toolchain:

     - Remove static OpenSSL dependency
     - Fixing inline assembly problems in ASAN
     - Issues with non-relocatable LLVM 3.3 - adds manual system
       includes to fix issues with hardcoded header paths in clang.

When the toolchain is enabled and we build for ASAN we use a specific
toolchain file to build with LLVM-trunk as the main compiler. Even
though this uses LLVM-trunk for compiling the Impala code, this will use
LLVM 3.3 for codegen.  In addition, this enables us to follow up with
TSAN and LEAKSAN.

Change-Id: I0abb914ca3f192cb7edd83ead134bc9e2d02071f
Reviewed-on: http://gerrit.cloudera.org:8080/556
Tested-by: Internal Jenkins
Reviewed-by: Martin Grund <mgrund@cloudera.com>
2015-08-21 20:14:31 +00:00
Martin Grund
a4f84b1b1a Making CMake modules more modular for non-toolchain build
This patch makes sure that the Impala-lzo build can pickup the
cmake modules from Impala to avoid code duplication on the lzo side.

Change-Id: I7917946724ce4bfaa281e708e9ea5799b4e2cd37
Reviewed-on: http://gerrit.cloudera.org:8080/552
Reviewed-by: Martin Grund <mgrund@cloudera.com>
Tested-by: Internal Jenkins
2015-07-22 02:01:34 +00:00
Martin Grund
384ae3ab08 Fixes for Toolchain Issues
If a static version of zlib and bzip2 is picked up we assumed that it
would be compiled with -fPIC. However, this is not always the case. Thus
in the non-toolchain case we specifically dynamic link with zlib and
bzip2 for the dynamic targets.

In addition, this patch removes static linking of libgcc in the
toolchain case as LLVM is not able to find the exception handling
symbols even if they are present in the binary. Static linking of libgcc
is postponed.

Next, if Impala is build with -notests the external data source thrift
files would not be generated. This patch make sure the dependencies are
expressed correctly.

Finally, if a user would have google perftools installed on the system
we would accidentally pick up the system libraries and the thirdparty
headers which will end in linker errors. This patch fixes the path
issues.

Change-Id: Ic000101c33da26d75a0cd733f7ef02f1bd694937
Reviewed-on: http://gerrit.cloudera.org:8080/460
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
2015-06-15 23:14:32 +00:00
Martin Grund
81f247b171 Optional Impala Toolchain
This patch allows to optionally enable the new Impala binary
toolchain. For now there are now major version differences in the
toolchain dependencies and what is currently kept in thirdparty.

To enable the toolchain, export the variable IMPALA_TOOLCHAIN to the
folder where the binaries are available.

In addition this patch moves gutil from the thirdparty directory into
the source tree of be/src to allow easy propagation of compiler and
linker flags. Furthermore, the thrift-cpp target was added as a
dependency to all targets that require the generated thrift sources to
be available before the build is started.

What is the new toolchain: The goal of the toolchain is to homogenize
the build environment and to make sure that Impala is build nearly
identical on every platform. To achieve this, we limit the flexibility
of using the systems host libraries and rather rely on a set of custom
produced binaries including the necessary compiler.

Change-Id: If2dac920520e4a18be2a9a75b3184a5bd97a065b
Reviewed-on: http://gerrit.cloudera.org:8080/427
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Internal Jenkins
Reviewed-by: Martin Grund <mgrund@cloudera.com>
2015-06-13 03:11:44 +00:00
Nong Li
7da3d1a5c0 Fix sasl thirdparty build to work on all distros.
Our packaging jobs use ~ in the path containing IMPALA_HOME. Getting
sasl's 'make install' to work with ~ doesn't seem possible. Let's just
install it somewhere else.

Change-Id: Ie59a5a35609e7bff56e5684d34411b46f8ddcc0c
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4304
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins
2014-09-12 15:13:14 -07:00
root
b8ad1d0296 Updates to build on ubuntu 14.04/boost 1.5*.
Change-Id: Idfbbe6f0dc4e0888f404b23bb4046b8132d27940
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4296
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
2014-09-12 01:22:32 -07:00
Alex Behm
a1f504d18c Fix Ldap include path in cmake find module.
Change-Id: I877475e7f2b4888a712c3aef5bd207e35046afb1
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3865
Reviewed-by: Nong Li <nong@cloudera.com>
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Alex Behm <alex.behm@cloudera.com>
2014-08-17 12:45:59 -07:00
Henry Robinson
446d1061c3 Small changes to thirdparty builds
* Move openldap to /impala_install to avoid conflict with INSTALL file
* Don't build Thrift support for languages we don't use

Change-Id: I260b3b4bdecc26b5525e5117a265add6638773c0
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3854
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
2014-08-17 12:44:58 -07:00
Henry Robinson
4b8ca060b8 Add Mustache template library
The Mustache template library renders
Mustache (httP;//mustache.github.io) templates with Json contexts
produced by RapidJson.

Change-Id: I38fa33aea88bea7d19a2ee8cb548a50e3ca2f4a8
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/2196
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
(cherry picked from commit f7a30054a180867bcb492444bbcb4fa2849c7b93)
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3648
Reviewed-by: Henry Robinson <henry@cloudera.com>
2014-07-29 17:24:45 -07:00
Henry Robinson
55a054e1b8 Small fixes for FindRapidJson.cmake
This tiny fix cleans up the FindRapidJson.cmake file, which wasn't
actually searching for the rapidjson.h header correctly but instead
assuming a particular location for it, making some of the other code in
the file redundant.

Change-Id: I5fe3c664b8a078b6610440a8ae4173f38d57a84b
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3631
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
(cherry picked from commit 438d1512cc78899473d6684b9ca07a6a9b7b100a)
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3649
Tested-by: Henry Robinson <henry@cloudera.com>
Reviewed-by: Henry Robinson <henry@cloudera.com>
2014-07-29 17:20:55 -07:00
Paden Tomasello
0326f17bb3 Adding Lz4 Codec.
Change-Id: I037d4e0de3b2cd2b8582caea058c8e1f2f880ff3
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3027
Reviewed-by: Paden Tomasello <paden.tomasello@cloudera.com>
Tested-by: jenkins
2014-06-16 14:20:34 -07:00
Nong Li
d816e0fc51 Add the multi precision int library from boost.
This library provides the exact API and functionality we need for decimals.
I've included a simple example of how the API is used. Still looking into
the perf but I don't think we can build something much faster as easily.

I've taken the latest version from boost 1.55 but I don't think we can
upgrade that in all of our supported platforms easily. Instead I've taken
this part of the library and put it in thirdparty.

Change-Id: Icf9879d546aa6a8bde046940fb91eb5060fb2f47
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1691
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1723
2014-03-02 20:57:49 -08:00
Henry Robinson
177b9ba3b1 Remove nonblocking server (and dependencies) from build
Goodnight, sweet non-blocking prince. We didn't support, or test, this
configuration, and it doesn't work with security or sessions and brings
in some annoying dependencies that are a pain to build.

We have other RPC-stack options to investigate; we may wind up re-adding
the non-blocking server but only in a way that supports all required
features more regularly.

Change-Id: Ifbcabc5014441f6d31c342c4e288dd7fc6201443
2014-01-08 10:54:35 -08:00
Henry Robinson
6d9a7e290d Build Openldap as a thirdparty package
Change-Id: Ifbb0f468a23186f4160fceb462953bc321469c27
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1049
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: Henry Robinson <henry@cloudera.com>
2014-01-08 10:54:20 -08:00
Skye Wanderman-Milne
b7f83bcd73 Add support for LLVM IR UDFs.
This patch also adds a number of improvements to NativeUdfExpr. Highlights include:

* Correctly handling the lowering of AnyVal struct types (required for ABI compatibility)
* A rudimentary library cache for reusing handles produced by dlopen
* More complicated test cases

Change-Id: Iab9acdd7d7c4308e5d7ee3210f21b033fda5a195
Reviewed-on: http://gerrit.ent.cloudera.com:8080/540
Tested-by: jenkins
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Skye Wanderman-Milne <skye@cloudera.com>
2014-01-08 10:53:03 -08:00
Henry Robinson
dbed012396 Move from 'Mongoose' to 'Squeasel' webserver
We now maintain our own internal version of the Mongoose webserver,
renamed to 'Squeasel' for differentiation. This patch imports the new
code, and swaps all mentions of mongoose or mg_ for squeasel / sq_.

In the future, we might consider making Squeasel a git subproject so
that we can pull in changes more easily.

Change-Id: I83b595dc336a32f2c8aba59eee420b71274b681b
Reviewed-on: http://gerrit.ent.cloudera.com:8080/485
Tested-by: jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
2014-01-08 10:52:55 -08:00
Lenni Kuff
e0876b951b Add support for audit event logging
This change adds support audit event logging in Impala. This feature is
disabled by default and is enabled by setting the -audit_event_log_dir
flag. When auditing is enabled, details on each query that Impala executes
will be saved to the audit log along with the current session state. This
includes information such as the statement type, catalog objects accessed
by the query, and whether there the operation passed authorization.

Change-Id: I39b78664c971124ec79c5fcee998065dd53fd32e
Reviewed-on: http://gerrit.ent.cloudera.com:8080/142
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: Lenni Kuff <lskuff@cloudera.com>
2014-01-08 10:52:08 -08:00
Nong Li
9af39ad4c4 Add check for llvm version in CMake. 2014-01-08 10:51:55 -08:00
Skye Wanderman-Milne
3fecdeb793 IMPALA-441: support default values for Avro tables 2014-01-08 10:51:39 -08:00