This commit contains the simpler parts from
https://gerrit.cloudera.org/#/c/20602
This mainly means accessors for the header of the binary
format and bounding box check (st_envIntersects).
New tests for not yet covered functions / overloads are also added.
For details of the binary format see be/src/exprs/geo/shape-format.h
Differences from the PR above:
Only a subset of functions are added. The criteria was:
1. the native function must be fully compatible with the Java version*
2. must not rely on (de)serializing the full geometry
3. the function must be tested
1 implies 2 because (de)serialization is not implemented yet in
the original patch for >2d geometries, which would break compatibility
for the Java version for ZYZ/XYM/XYZM geometries.
*: there are 2 known differences:
1. NULL handling: the Java functions return error instead of NULL
when getting a NULL parameter
2. st_envIntersects() doesn't check if the SRID matches - the Java
library looks inconsistant about this
Because the native functions are fairly safe replacements for the Java
ones, they are always used when geospatial_library=HIVE_ESRI.
Change-Id: I0ff950a25320549290a83a3b1c31ce828dd68e3c
Reviewed-on: http://gerrit.cloudera.org:8080/23700
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch mainly implement the querying of paimon data table
through JNI based scanner.
Features implemented:
- support column pruning.
The partition pruning and predicate push down will be submitted
as the third part of the patch.
We implemented this by treating the paimon table as normal
unpartitioned table. When querying paimon table:
- PaimonScanNode will decide paimon splits need to be scanned,
and then transfer splits to BE do the jni-based scan operation.
- We also collect the required columns that need to be scanned,
and pass the columns to Scanner for column pruning. This is
implemented by passing the field ids of the columns to BE,
instead of column position to support schema evolution.
- In the original implementation, PaimonJniScanner will directly
pass paimon row object to BE, and call corresponding paimon row
field accessor, which is a java method to convert row fields to
impala row batch tuples. We find it is slow due to overhead of
JVM method calling.
To minimize the overhead, we refashioned the implementation,
the PaimonJniScanner will convert the paimon row batches to
arrow recordbatch, which stores data in offheap region of
impala JVM. And PaimonJniScanner will pass the arrow offheap
record batch memory pointer to the BE backend.
BE PaimonJniScanNode will directly read data from JVM offheap
region, and convert the arrow record batch to impala row batch.
The benchmark shows the later implementation is 2.x better
than the original implementation.
The lifecycle of arrow row batch is mainly like this:
the arrow row batch is generated in FE,and passed to BE.
After the record batch is imported to BE successfully,
BE will be in charge of freeing the row batch.
There are two free paths: the normal path, and the
exception path. For the normal path, when the arrow batch
is totally consumed by BE, BE will call jni to fetch the next arrow
batch. For this case, the arrow batch is freed automatically.
For the exceptional path, it happends when query is cancelled, or memory
failed to allocate. For these corner cases, arrow batch is freed in the
method close if it is not totally consumed by BE.
Current supported impala data types for query includes:
- BOOLEAN
- TINYINT
- SMALLINT
- INTEGER
- BIGINT
- FLOAT
- DOUBLE
- STRING
- DECIMAL(P,S)
- TIMESTAMP
- CHAR(N)
- VARCHAR(N)
- BINARY
- DATE
TODO:
- Patches pending submission:
- Support tpcds/tpch data-loading
for paimon data table.
- Virtual Column query support for querying
paimon data table.
- Query support with time travel.
- Query support for paimon meta tables.
- WIP:
- Snapshot incremental read.
- Complex type query support.
- Native paimon table scanner, instead of
jni based.
Testing:
- Create tests table in functional_schema_template.sql
- Add TestPaimonScannerWithLimit in test_scanners.py
- Add test_paimon_query in test_paimon.py.
- Already passed the tpcds/tpch test for paimon table, due to the
testing table data is currently generated by spark, and it is
not supported by impala now, we have to do this since hive
doesn't support generating paimon table for dynamic-partitioned
tables. we plan to submit a separate patch for tpcds/tpch data
loading and associated tpcds/tpch query tests.
- JVM Offheap memory leak tests, have run looped tpch tests for
1 day, no obvious offheap memory increase is observed,
offheap memory usage is within 10M.
Change-Id: Ie679a89a8cc21d52b583422336b9f747bdf37384
Reviewed-on: http://gerrit.cloudera.org:8080/23613
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Zoltan Borok-Nagy <boroknagyz@cloudera.com>
Reviewed-by: Riza Suminto <riza.suminto@cloudera.com>
Trace DML/DDL Queries
* Adds tracing for alter, compute, create, delete, drop, insert,
invalidate metadata, and with queries.
* Stops tracing beeswax queries since that protocol is deprecated.
* Adds Coordinator attribute to Init and Root spans for identifying
where the query is running.
Comment Handling
* Corrects handling of leading comments, both inline and full line.
Previously, queries with comments before the first keyword were
always ignored.
* Adds be ctest tests for determining whether or not a query should
be traced.
General Improvements
* Handles the case where the first query keyword is followed by a
newline character or an inline comment (without or with spaces
between).
* Corrects traces for errored/cancelled queries. These cases
short-circuit the normal query processing code path and have to be
handled accordingly.
* Ends the root span when the query ends instead of waiting for the
ClientRequestState to go out of scope. This change removes
use-after-free issues caused by reading from ClientRequestState
when the SpanManager went out of scope during that object's dtor.
* Simplified minimum tls version handling because the validators
on the ssl_minimum_version eliminate invalid values that previously
had to be accounted for.
* Removes the unnecessary otel_trace_enabled() function.
* Fixes IMPALA-14314 by waiting for the full trace to be written to
the output file before asserting that trace.
Testing
* Full test suite passed.
* ASAN/TSAN builds passed.
* Adds new ctest test.
* Adds custom cluster tests to assert traces for the new supported
query types.
* Adds custom cluster tests to assert traces for errored and
cancelled queries.
Generated-by: Github Copilot (Claude Sonnet 3.7)
Change-Id: Ie9e83d7f761f3d629f067e0a0602224e42cd7184
Reviewed-on: http://gerrit.cloudera.org:8080/23279
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Riza Suminto <riza.suminto@cloudera.com>
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Reviewed-on: http://gerrit.cloudera.org:8080/22924
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Adds the OpenTelemetry C++ SDK version 1.20.0 from the toolchain into
the cmake files for consumption during builds.
Testing was accomplished by building locally and in Jenkins.
Generated-by: Github Copilot (GPT-4.1)
Change-Id: Ib30123f79270e3f11233e28a2a34725e7d455f5e
Reviewed-on: http://gerrit.cloudera.org:8080/23101
Reviewed-by: Jason Fehr <jfehr@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
The RapidJSON library recalculates length of a StringRef each time
when using char*, until null character.
Within the codebase, many times we convert from string to char* for
RapidJSON.
Instead with RAPIDJSON_HAS_STDSTRING, we can use strings directly,
which inherently contain the size, instead of recalculating each time.
This is a major convenience and a bit more efficient during both
JSON generation and parsing.
This simplifies many methods(i.e. AddMember, HasMember, etc) and
the creation of Document::Value.
This does not remove compatibility for existing char* and still remains
as a stable feature within RapidJSON.
Change-Id: I84c6f17690f7a2e958a6dfbb3637832f5ab6ee53
Reviewed-on: http://gerrit.cloudera.org:8080/22613
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Adds "Select Columns", "Where Columns", "Join Columns", "Aggregate
Columns", and "OrderBy Columns" to the query profile and the workload
management active/completed queries tables. These fields are
presented as comma separate strings containing the fully qualified
column name in the format database.table_name.column_name. Aggregate
columns include all columns in the order by and having clauses.
Since new columns are being added, the workload management init
process is also being modified to allow for one-way upgrades of the
table schemas if necessary. Additionally, workload management can be
set up to run under a schema version that is not the latest. This
ability will be useful during troubleshooting. To enable these
upgrades, the workload management initialization that manages the
structure of the tables has been moved to the catalogd.
The changes in this patch must be backwards compatible so that Impala
clusters running previous workload management code can co-exist with
Impala clusters running this workload management code. To enable that
backwards compatibility, a new table property named
'wm_schema_version' is now used to track the schema version of the
workload management tables. Thus, the old property 'schema_version'
will always be set to '1.0.0' since modifying that property value
causes Impala running previous workload management code to error at
startup.
Testing accomplished by
* Adding/updating workload and custom cluster tests to assert the new
columns and the workload management upgrade process.
* JUnit tests added to verify the new workload management columns are
being correctly parsed.
* GTests added to ensure the workload management columns are
correctly defined and in the correct order.
Change-Id: I78f3670b067c0c192ee8a212fba95466fbcb51d7
Reviewed-on: http://gerrit.cloudera.org:8080/21142
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Riza Suminto <riza.suminto@cloudera.com>
This adds the IMPALA_SPLIT_DEBUG_INFO environment variable,
which controls whether the build uses -gsplit-dwarf. This option
puts debug information in a separate .dwo file for each C++
file. Executables contain metadata pointing to those .dwo files
and don't need to include the debug information themselves. This
reduces link time and disk space usage. The default for
IMPALA_SPLIT_DEBUG_INFO is off as this is intended to be an
opt-in option for developers.
For a debug build with compressed debug information,
it cuts disk space usage roughly in half:
Without backend tests (measuring "du -sh be"):
Regular: 5.6GB
Split debuginfo: 2.7GB
With backend tests:
Regular: 22GB
Split debuginfo: 12GB
This only works for the debug information from Impala itself.
The debug information from dependencies from the toolchain
are included in each executable.
Split debug information has been around for a long time,
so tools like GDB work. Resolving minidumps works properly.
Testing:
- Ran builds locally (with GCC and Clang)
- Attached to Impala with GDB and verified that symbols worked
- Resolved a minidump and checked the output
Change-Id: I3bbe700279a5dc3fde338fdfa0ea355d3570c9d0
Reviewed-on: http://gerrit.cloudera.org:8080/21720
Reviewed-by: Jason Fehr <jfehr@cloudera.com>
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Tested-by: Michael Smith <michael.smith@cloudera.com>
This adds support for using the mold linker. It changes
the existing USE_GOLD_LINKER environment variable to
IMPALA_LINKER, which accepts ld, gold, or mold as
values. It defaults to 'gold' to match current behavior.
Developers can override it in bin/impala-config-local.sh.
Clang does not implement -gz properly until version 12.
It does not enable compressed debuginfo in the final
binary. IMPALA_LINKER=mold doesn't work with
IMPALA_COMPRESSED_DEBUG_INFO=true on Clang due to this.
This detects Clang <12 and skips -gz as it is ineffective.
Mold follows similar to behavior to LLD and requires
--exclude-libs to use the full library name (i.e.
liblz4.a rather than liblz4). Gold will happily
accept the full library name, so this changes to use
the full library name.
Mold is much faster for incremental builds on my system:
(e.g. touch be/src/scheduling/scheduler.cc && make -j8 impalad)
gold: 15.8s
mold: 2.6s
Testing:
- Ran builds with IMPALA_LINKER=mold on Centos 7, Redhat 8,
and Ubuntu 20.
Change-Id: Ia9e9accd06b6ecd182d200d81afaae09a885c241
Reviewed-on: http://gerrit.cloudera.org:8080/21121
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Reviewed-by: Andrew Sherman <asherman@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
For current clang-tidy builds, we enable -Weverything and then
explicitly disable a couple verbose warnings. Clang-tidy uses
these warnings to produce its clang-diagnostic-* issues.
This gives clang-tidy the maximum flexibility to enforce any
warning the Clang can produce, but in practice, we disable
them via the .clang-tidy file's Checks: section.
The output from a -Weverything build is enormous and hard to
navigate. When there is a compilation issue, digging through
the logs is arduous. Clang discourages the use of -Weverything,
as it contains many noisy warnings.
This changes the clang-tidy build to use -Wextra instead and
shrinks the list of disabled clang-diagnostic-* issues
accordingly.
Testing:
- Verified that bin/run_clang_tidy.sh passes
Change-Id: Ib4f4cc9a6d040b90495496b2396a26f3c5189231
Reviewed-on: http://gerrit.cloudera.org:8080/21112
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This commit adds a scan node for querying Iceberg metadata tables. The
scan node creates a Java scanner object that creates and scans the
metadata table. The scanner uses the Iceberg API to scan the table after
that the scan node fetches the rows one by one and materialises them
into RowBatches. The Iceberg row reader on the backend does the
translation between Iceberg and Impala types.
There is only one fragment created to query the Iceberg metadata table
which is supposed to be executed on the coordinator node that already
has the Iceberg table loaded. This way there is no need for further
table loading on the executor side.
This change will not cover nested column types, these slots are set to
NULL, it will be done in IMPALA-12205.
Testing:
- Added e2e tests for querying metadata tables
- Updated planner tests
Performance testing:
Created a table and inserted ~5500 rows one by one, this generated
~270000 ALL_MANIFESTS metadata table records. This table is quite wide
and has a String column as well.
I only mention count(*) test on ALL_MANIFESTS, because every row is
materialized in every scenario currently:
- Cold cache: 15.76s
- IcebergApiScanTime: 124.407ms
- MaterializeTupleTime: 8s368ms
- Warm cache: 7.56s
- IcebergApiScanTime: 3.646ms
- MaterializeTupleTime: 7s477ms
Change-Id: I0e943cecd77f5ef7af7cd07e2b596f2c5b4331e7
Reviewed-on: http://gerrit.cloudera.org:8080/20010
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Prototype of HdfsJsonScanner implemented based on rapidjson, which
supports scanning data from splitting json files.
The scanning of JSON data is mainly completed by two parts working
together. The first part is the JsonParser responsible for parsing the
JSON object, which is implemented based on the SAX-style API of
rapidjson. It reads data from the char stream, parses it, and calls the
corresponding callback function when encountering the corresponding JSON
element. See the comments of the JsonParser class for more details.
The other part is the HdfsJsonScanner, which inherits from HdfsScanner
and provides callback functions for the JsonParser. The callback
functions are responsible for providing data buffers to the Parser and
converting and materializing the Parser's parsing results into RowBatch.
It should be noted that the parser returns numeric values as strings to
the scanner. The scanner uses the TextConverter class to convert the
strings to the desired types, similar to how the HdfsTextScanner works.
This is an advantage compared to using number value provided by
rapidjson directly, as it eliminates concerns about inconsistencies in
converting decimals (e.g. losing precision).
Added a startup flag, enable_json_scanner, to be able to disable this
feature if we hit critical bugs in production.
Limitations
- Multiline json objects are not fully supported yet. It is ok when
each file has only one scan range. However, when a file has multiple
scan ranges, there is a small probability of incomplete scanning of
multiline JSON objects that span ScanRange boundaries (in such cases,
parsing errors may be reported). For more details, please refer to
the comments in the 'multiline_json.test'.
- Compressed JSON files are not supported yet.
- Complex types are not supported yet.
Tests
- Most of the existing end-to-end tests can run on JSON format.
- Add TestQueriesJsonTables in test_queries.py for testing multiline,
malformed, and overflow in JSON.
Change-Id: I31309cb8f2d04722a0508b3f9b8f1532ad49a569
Reviewed-on: http://gerrit.cloudera.org:8080/19699
Reviewed-by: Quanlong Huang <huangquanlong@gmail.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Redhat 9 and Ubuntu 22.04 both use OpenSSL3, which deprecated
several APIs that we use. To support those platforms, we added
the -Wno-deprecated-declaration to the build. Historically, the
Impala build has also specified -Wno-deprecated due to
use of deprecated headers in gutils. These flags limit our
ability to notice use of deprecated code in other parts of the
code.
The code in gutils no longer requires -Wno-deprecated, so
this removes it completely. Additionally, this limits the
-Wno-deprecated-declaration flag to machines using
OpenSSL 3.
Reenabling deprecation warnings also reenables Clang Tidy's
clang-diagnostic-deprecated enforcement. This is currently
broken, so this turns off clang-diagnostic-deprecated
until it can be addressed properly.
Testing:
- Ran build-all-options on Ubuntu 22 and Ubuntu 16
- Ran a Rocky 9.2 build
Change-Id: I1b36450d084f342eeab5dac2272580ab6b0c988b
Reviewed-on: http://gerrit.cloudera.org:8080/20369
Reviewed-by: Laszlo Gaal <laszlo.gaal@cloudera.com>
Reviewed-by: Zoltan Borok-Nagy <boroknagyz@cloudera.com>
Tested-by: Joe McDonnell <joemcdonnell@cloudera.com>
This patch adds a new option 'BUILD_WITH_NO_TESTS' to tell CMake not
to generate test targets. In order to be consistent with the previous
test workflow, this option is only set ON when building impala using
the 'buildall.sh' script with '-notest' and '-package' flags. This
is useful for a packaging build which do not need to build all test
binaries.
Testing:
- Ran 'buildall.sh -release -package' with and without '-notests'
flag and verified generated executables.
Change-Id: I575ce76176c9f6a05fd2db0f420ebe6926d0272a
Reviewed-on: http://gerrit.cloudera.org:8080/20294
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Reviewed-by: Quanlong Huang <huangquanlong@gmail.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This adds support for Redhat 9 / Ubuntu 22. It updates
to a newer toolchain that has those builds, and it adds
supporting code in bootstrap_system.sh.
Redhat 9 and Ubuntu 22 use python = python3, which requires
various changes to build scripts and tests. Ubuntu 22 uses
Python 3.10, which deprecates certain ssl.PROTOCOL_TLS, so
this adapts test_client_ssl.py to that change until it
can be fully addressed in IMPALA-12219.
Various OpenSSL methods have been deprecated. As a workaround
until these can be addressed properly, this specifies
-Wno-deprecated-declarations. This can be removed once the
code is adapted to the non-deprecated APIs in IMPALA-12226.
Impala crashes with tcmalloc errors unless we update to a newer
gperftools, so this moves to gperftools 2.10. gperftools changed
the default for tcmalloc.aggressive_memory_decommit to off, so
this adapts our code to set it for backend tests. The gperftools
upgrade does not show any performance regression:
+----------+-----------------------+---------+------------+------------+----------------+
| Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) |
+----------+-----------------------+---------+------------+------------+----------------+
| TPCH(42) | parquet / none / none | 3.08 | -0.64% | 2.20 | -0.37% |
+----------+-----------------------+---------+------------+------------+----------------+
With newer Python versions, the impala-virtualenv command
fails to create a Python 3 virtualenv. This switches to
using Python 3's builtin venv command for Python >=3.6.
Kudu needed a newer version and LLVM required a couple patches.
Testing:
- Ran a core job on Ubuntu 22 and Redhat 9. The tests run
to completion without crashing. There are test failures
that will be addressed in follow-up JIRAs.
- Ran dockerised tests on Ubuntu 22.
- Ran dockerised tests on Ubuntu 20 and Rocky 8.5.
Change-Id: If1fcdb2f8c635ecd6dc7a8a1db81f5f389c78b86
Reviewed-on: http://gerrit.cloudera.org:8080/20073
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Tested-by: Joe McDonnell <joemcdonnell@cloudera.com>
This switches from C++14 to C++17, which allows Impala
code to interact with libraries written in C++17.
This fixes a few minor compilation differences, such
as the need for comparators to be const. Since C++17
includes -faligned-new, this removes the CMake code
that specified it.
Testing:
- Ran core job
Change-Id: Iadac41817fe5eaaa469a5f0e9f94056a409c14b9
Reviewed-on: http://gerrit.cloudera.org:8080/19183
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
The Clang Tidy build enables all warnings via -Wall
and -Weverything. This produces enormous output.
Looking at a recent failed Clang Tidy build, there
are ~4.5 million warnings generated. Of these,
about 4 million are from C++98 compatibility warnings.
A further 250 thousand are from padding warnings.
Since these are not particularly interesting, this
disables both of those to reduce the output size.
Disabling these warnings allowed Clang Tidy to find
some issues in DataSketches that it was previously
missing. Perhaps there is some limit on the number
or size of warnings that it was processing. This
modifies the DataSketches code to fix those (which
are all minor issues with const correctness).
Testing:
- Built with clang tidy locally
Change-Id: I28c6ed1e7a4f525d81a9c48e90d051b374d44941
Reviewed-on: http://gerrit.cloudera.org:8080/19182
Reviewed-by: Wenzhe Zhou <wzhou@cloudera.com>
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
recompilation
Currently, modifying most files incurs a rebuild of the LLVM IR, which
is a slow serial step.
This change splits off the *-ir.cc files into their own libraries that
have no dependency relation with the original libraries from which they
are separated. The build of the LLVM IR depends only on the new IR
libraries, so if a non-IR C++ source file is modified, the LLVM IR does
not need to be rebuilt.
Note that the LLVM IR build does not actually need or use the new IR
libraries - it is a completely separate procedure in which the *-ir.cc
files are included in impala-ir.cc and compiled with Clang into LLVM
bitcode. We add the dependency on the IR libraries to trigger the
rebuilding of the LLVM IR whenever an *-ir.cc file is modified.
However, the new IR libraries still need to be compiled into the main
Impala executable.
Also, dependence on the target 'gen_ir_descriptions' was not consistent.
As it is a fast target and needed by most other targets, 'gen-deps' now
depends on it to keep things safer and simpler. Direct dependencies on
'gen_ir_descriptions' were removed from that targets that also depend on
'gen-deps'.
Testing:
- manually verified for each modified target library that the LLVM IR
is not recompiled when a non-IR .cc file is modified, but is
recompiled if the corresponding header or *-ir.cc file is modified.
Change-Id: I63b9285bac6494a19f614d0ebc694a91bdf7a8a0
Reviewed-on: http://gerrit.cloudera.org:8080/19083
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Building with -so on Ubuntu 18 or higher fails due to
an issue finding dlopen in unwind_safeness.cc:
unwind_safeness.cc:76] Check failed: !error failed to find symbol dlopen
unwind_safeness.cc is using dlsym to load the dlopen symbol
so that it can wrap it with its own dlopen code. The Impala
build has issues with the ordering of libraries, and this
code does not find dlopen. This has previously happened with
the dl_iterate_phdr symbol in Kudu.
This is a problem starting with Ubuntu 18.04, because Ubuntu 16.04
uses a version of glibc that has a bug in reporting this
error. Ubuntu 18.04 uses a newer glibc with a fix for the bug. See
https://sourceware.org/bugzilla/show_bug.cgi?id=19509 .
As a workaround for this issue, this tolerates not finding
dlopen/dlclose when building with shared libraries. Impala shared
libraries are not used in production, so this bypasses the issue.
This also adds extra validation to make sure the symbols are non-null.
Specifically, this adds another CHECK in dlsym_or_die to verify
that the symbol is non-null. This also adds a DCHECK to verify that
the symbol is non-null at dereference.
This also fixes an issue where Boost was always using static
libraries, even for shared library builds. This makes Boost use
shared libraries for shared library builds.
Testing:
- The shared library build passes on Ubuntu 18 and Ubuntu 20
- Impala can boot and run queries with shared libraries
Change-Id: Iaab196b3d669ccc12854a98d0dbfbae2b9b91244
Reviewed-on: http://gerrit.cloudera.org:8080/19104
Reviewed-by: Daniel Becker <daniel.becker@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Impala's build produces dozens of C++ binaries
that link in all Impala libraries. Each binary is
hundreds of megabytes, leading to 10s of gigabytes
of disk space. A large proportion of this (~80%) is debug
information. The debug information increases in newer
versions of GCC such as GCC 10.
This introduces two options for reducing the size
of debug information:
- IMPALA_MINIMAL_DEBUG_INFO=true builds Impala with
minimal debug information (-g1). This contains line tables
and can resolve backtraces, but it does not contain
variable information and restricts further debugging.
- IMPALA_COMPRESSED_DEBUG_INFO=true builds Impala with
compressed debug information (-gz). This does not change
the debug information included, but the compression saves
significant disk space. gdb is known to work with
compressed debug information, but other tools may not
support it. The dump_breakpad_symbols.py script has been
adjusted to handle these binaries.
These are disabled by default.
Release impalad binary sizes:
Configuration | Size (bytes) | % reduction over base
Base | 707834808 | N/A
Stripped | 83351664 | 88%
Minimal debuginfo | 215924096 | 69%
Compressed debuginfo | 301619286 | 57%
Minimal + compressed debuginfo | 120886705 | 83%
Testing:
- Generated minidumps and resolved them
- Verified this is disabled by default
Change-Id: I04a20258a86053d8f3972b9c7c81cd5bec1bbb66
Reviewed-on: http://gerrit.cloudera.org:8080/18962
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Reviewed-by: Wenzhe Zhou <wzhou@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This upgrades GCC and libstdc++ to version 10.4. This
required patching or upgrading several dependencies
so they could compile with GCC 10. The toolchain
companion change has details on what items needed
to be upgraded and why.
The toolchain companion change switches GCC to build
with toolchain binutils rather than host binutils. This
means that the python virtualenv initialization needs
to include binutils on the path.
This disables two warnings introduced in the new GCC
versions (Wclass-memaccess and Winit-list-lifetime).
These two warnings occur in our code and also in
dependencies like LLVM and rapidjson. These are not
critical warnings, so they can be addressed
independently and reenabled later.
Binary sizes increase, particulary when including
debug symbols:
| GCC 7.5 | GCC 10.4
impalad RELEASE stripped | 83204768 | 88702824
impalad RELEASE | 707278904 | 971711456
impalad DEBUG stripped | 106677672 | 97391944
impalad DEBUG | 725864760 | 867647512
Testing:
- Multiple test jobs (core, release exhaustive, ASAN)
- Performance testing for TPC-H and TPC-DS shows
a modest improvement (2-4%).
- Code compiles without warnings on debug and release
Change-Id: Ibe6857b822925226d39fd4d6413457ef6bbaabec
Reviewed-on: http://gerrit.cloudera.org:8080/18134
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Csaba Ringhofer <csringhofer@cloudera.com>
This patch renames the Exec libraries to distinguish them from
external libraries like avro, kudu_client and orc
Due to IMPALA-11257, the cmake_minimum_required calls are
also removed from the new libraries
Change-Id: I042478e371049a7aed74f43c2bbfcb7900a53dd0
Reviewed-on: http://gerrit.cloudera.org:8080/18910
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
To avoid crash issue at OS with kernel lower than Linux 3.17, this patch
pick kudu's solution 35b5664f908cd1250c9f01e5dff77b653cfd12b7 into impala.
Testing:
- run pre-built impalad at CentOS 7 kernel 3.10 with no crash.
Change-Id: Ic48bd59b0a846bcb91a6faf77156c0a49cd08ae8
Reviewed-on: http://gerrit.cloudera.org:8080/18805
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Alexey Serbin <alexey@apache.org>
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
This fixes a few different CMake warnings:
1. This removes cmake_minimum_required invocations except for the
top-most CMakeLists.txt. This eliminates the warnings like this:
Compatibility with CMake < 2.8.12 will be removed from a future version of
CMake.
Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.
Moving to a later version also required setting CMAKE_ENABLE_EXPORTS
to continue exporting symbols.
2. This modifies the module names so that they match the corresponding
module names from Find*.cmake. This is mostly dealing with case
differences. This address warnings like:
The package name passed to `find_package_handle_standard_args` (PROTOBUF)
does not match the name of the calling package (Protobuf). This can lead
to problems in calling code that expects `find_package` result variables
(e.g., `_FOUND`) to follow a certain pattern.
This fixed the detection logic for KerberosPrograms, and so it required
adding more Kerberos packages to bin/bootstrap_build.sh.
3. This adds a missing .cc suffix. This addresses the following warning:
CMake Warning (dev) at be/src/util/CMakeLists.txt:141 (add_library):
Policy CMP0115 is not set: Source file extensions must be explicit. Run
"cmake --help-policy CMP0115" for policy details. Use the cmake_policy
command to set the policy and suppress this warning.
These fixes mostly match how these warnings were handled in
Apache Kudu.
Testing:
- Ran GVO
Change-Id: I2a97dd07cdd0831e90882a2035415ac71d670147
Reviewed-on: http://gerrit.cloudera.org:8080/18444
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Impala used to have one thrift compiler version to compile C++, Java,
and Python code.
Most Thrift serialization/deserialization between minor versions are
compatible with each other. So it is possible to have different thrift
compiler versions for different target codes. It is beneficial to do so
because it will allow Impala to upgrade separate components
independently.
This patch implements the infrastructure change required to do so. It
replace most of the 'THRIFT_*' environment variable and CMake variable
with 'THRFIT_CPP_*', 'THRFIT_JAVA_*', and 'THRFIT_PY_*' to compile C++,
Java, and Python code accordingly. All three still refer to the same
thrift version (thrift-0.11.0-p5).
Testing:
- Build Impala and pass core tests.
Change-Id: I56479dc69b79024d1a4d09211bbe88a61fa0c6a4
Reviewed-on: http://gerrit.cloudera.org:8080/18636
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
GCC's -Og applies optimizations that are compatible with
being debuggable. It is similar to -O1 and results
in a modest speed-up. This modifies the default debug
build to use -Og, so it is now more akin to a fastdebug
mode.
Even though -Og is intended to preserve debuggability,
optimization always impacts debuggability and -Og is
no exception. To enable the old behavior, this adds
a DEBUG_NOOPT build mode that retains the old
non-optimized behavior. Using the -debug_noopt flag
with buildall.sh enables this behavior.
Change-Id: Ie06c149c8181c90572b8668bd01dfd26c0a5971e
Reviewed-on: http://gerrit.cloudera.org:8080/18200
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Laszlo Gaal (Cloudera) <laszlo.gaal@cloudera.com>
Reviewed-by: Csaba Ringhofer <csringhofer@cloudera.com>
Currently, Impala compiles with -gdwarf-2 to use DWARF 2 debug
information. DWARF 4 provides better C++ language support and
better compression (reducing binary sizes), but it requires
GDB 7.0 or above. All of the Linux distributions that Impala
supports now have GDB 7.0 or above. This switches to using DWARF 4.
Binary sizes are reduced somewhat:
| DWARF 2 size | DWARF 4 size
impalad DEBUG | 573321352 | 519329200
impalad RELEASE | 713582048 | 693286800
This is also true for all unit tests. This has no impact on the
stripped binary sizes.
Testing:
- Ran core job
- Created a minidump and resolved it via the usual method
- Attached to impalad with gdb, set breakpoints, took backtraces, etc.
Change-Id: I7b6e75845ab137d0a7674289e4b331f682eee5b2
Reviewed-on: http://gerrit.cloudera.org:8080/18194
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
As part of moving to a newer protobuf, this updates the Kudu version
to get the fix for KUDU-3334. With this newer Kudu version, Clang
builds hit an error while linking:
lib/libLLVMCodeGen.a(TargetPassConfig.cpp.o):TargetPassConfig.cpp:
function llvm::TargetPassConfig::createRegAllocPass(bool):
error: relocation refers to global symbol "std::call_once<void (&)()>(std::once_flag&, void (&)())::{lambda()#2}::_FUN()",
which is defined in a discarded section
section group signature: "_ZZSt9call_onceIRFvvEJEEvRSt9once_flagOT_DpOT0_ENKUlvE0_clEv"
prevailing definition is from ../../build/debug/security/libsecurity.a(openssl_util.cc.o)
(This is from a newer binutils that will be pursued separately.)
As a hack to get around this error, this adds the calloncehack
shared library. The shared library publicly defines the symbol that
was coming from kudu_client. By linking it ahead of kudu_client, the
linker uses that rather than the one from kudu_client. This fixes
the Clang builds.
The new Kudu also requires a minor change to the flags for tserver
startup.
Testing:
- Ran debug tests and verified calloncehack is not used
- Ran ASAN tests
Change-Id: Ieccbe284f11445e1de792352ebc7c9e1fa2ca0c3
Reviewed-on: http://gerrit.cloudera.org:8080/18129
Reviewed-by: Wenzhe Zhou <wzhou@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
There are some header files deprecated in the new version of Boost
library. Need to define BOOST_ALLOW_DEPRECATED_HEADERS in
CMakeLists.txt to avoid compiling errors. Also define
BOOST_BIND_GLOBAL_PLACEHOLDERS to keep current behaviour of boost::bind
and avoid big code changes.
Define exception handler for a new boost::throw_exception() API since
BOOST_NO_EXCEPTIONS is defined in be/CMakeLists.txt and we have to
provide handlers which will be called by codegen'd code.
Revert the code change made by IMPALA-2846 and IMPALA-9571 since the
bug was fixed in Boost 1.74.0.
Testing:
- Passed core DEBUG build and exhaustive release build.
- Passed core ASAN build.
Change-Id: I78f32ae3c274274325e7af9e9bc9643814ae346a
Reviewed-on: http://gerrit.cloudera.org:8080/17996
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Wenzhe Zhou <wzhou@cloudera.com>
This patch added functionality to download JWKS from a given URL and
support key rotation by periodically checking the JWKS URL for updates.
We use Kudu's EasyCurl wrapper to download file from the given URL.
curl was added to native-toolchain. This patch modified makefiles
and bootstrap_toolchain.py to integrate libcurl and libkudu_curl_util.
Added end-end JWT authentication test cases with JWKS specified as
HTTP/HTTPS URL.
Testing:
- Passed core run, including new test cases.
Change-Id: Ic6ac8cf0010c13db30219776d1d275709bf211df
Reviewed-on: http://gerrit.cloudera.org:8080/17802
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch added JWT support with following functionality:
* Load and parse JWKS from pre-installed JSON file.
* Read the JWT token from the HTTP Header.
* Verify the JWT's signature with public key in JWKS.
* Get the username out of the payload of JWT token.
* Support following JSON Web Algorithms (JWA):
HS256, HS384, HS512, RS256, RS384, RS512.
We use third party library jwt-cpp to verify JWT token. jwt-cpp is a
headers only C++ library. It was added to native-toolchain.
This patch modified bootstrap_toolchain.py to download jwt-cpp from
toolchain s3 bucket, and modified makefiles to add jwt-cpp/include
in the include path.
Added BE unit-tests for loading JWKS file and verifying JWT token.
Also added FE custom cluster test for JWT authentication.
Testing:
- Passed core run.
Change-Id: I6b71fa854c9ddc8ca882878853395e1eb866143c
Reviewed-on: http://gerrit.cloudera.org:8080/17435
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This wraps each command executed by CMake with a wrapper that
generates a JUnitXML file if the command fails. If the command
succeeds, the wrapper does nothing. The wrapper applies to C++
compilation, linking, and custom shell commands (such as
building the frontend via maven). It does not apply to failures
coming from CMake itself. It can be disabled by setting
DISABLE_CMAKE_JUNITXML.
The command output can include Unicode (e.g. smart quotes for
g++), so this also updates generate_junitxml.py to handle
Unicode.
The wrapper interacts poorly with add_custom_command/add_custom_target
CMake commands that use 'cd directory && do_something', so this
switches those locations (in /docker) to use CMake's WORKING_DIRECTORY.
Testing:
- Verified it does not impact a successful build (including with
ccache and/or distcc).
- Verified it generates JUnitXML for C++ and Java compilation
failures.
- Verified it doesn't use the wrapper when DISABLE_CMAKE_JUNITXML
is set.
Change-Id: If71f2faf3ab5052b56b38f1b291fee53c390ce23
Reviewed-on: http://gerrit.cloudera.org:8080/12668
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Defined the BloomFilter class as the wrapper of kudu::BlockBloomFilter.
impala::BloomFilter build runtime bloom filter in kudu::BlockBloomFilter
APIs with FastHash as default hash algorithm.
Removed the duplicated functions from impala::BloomFillter class.
Pushed down bloom filter to Kudu through Kudu clinet API.
Added a new query option ENABLED_RUNTIME_FILTER_TYPES to set enabled
runtime filter types, which only affect Kudu scan node now. By default,
bloom filter is not enabled, only min-max filter will be enabled for
Kudu. With this option, user could enable bloom filter, min-max filter,
or both bloom and min-max runtime filters.
Added new test cases in PlannerTest and end-end runtime_filters test
for pushing down bloom filter to Kudu.
Added test cases to compare the number of rows returned from Kudu
scan when appling different types of runtime filter on same queries.
Updated bloom-filter-benchmark due to the bloom-filter implementation
change.
Bump Kudu version to d652cab17.
Testing:
- Passed all exhaustive tests.
Performance benchmark:
- Ran single_node_perf_run.py on TPC-H with scale as 30 for parquet
and Kudu. Verified that new hash function and bloom-filter
implementation don't cause regressions for HDFS bloom filters.
For Kudu, there is one regression for query TPCH-Q9 and there
are improvement for about 8 queris when appling both bloom and
min-max filters. The bloom filter reduce the number of rows
returned from Kudu scan, hence reduce the cost for aggregation
and hash join. But bloom filter evaluation add extra cost for
Kudu scan, which offset the gain on aggregation and join.
Kudu scan need to be optimized for bloom filter in following
tasks.
- Ran bloom-filter microbenchmarks and verified that there is no
regression for Insert/Find/Union functions with or without AVX2
due to bloom-filter implementation changes. There is small
performance degradation for Init function, but this function is
not in hot path.
Change-Id: I9100076f68ea299ddb6ec8bc027cac7a47f5d754
Reviewed-on: http://gerrit.cloudera.org:8080/15683
Reviewed-by: Thomas Tauber-Marshall <tmarshall@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
The locations for native-toolchain packages in IMPALA_TOOLCHAIN
currently do not include the compiler version. This means that
the toolchain can't distinguish between native-toolchain packages
built with gcc 4.9.2 versus gcc 7.5.0. The collisions can cause
issues when switching back and forth between branches.
This introduces the IMPALA_TOOLCHAIN_PACKAGES_HOME environment
variable, which is a location inside IMPALA_TOOLCHAIN that would
hold native-toolchain packages. Currently, it is set to the same
as IMPALA_TOOLCHAIN, so there is no difference in behavior.
This lays the groundwork to add the compiler version to this
path when switching to GCC7.
Testing:
- The only impediment to building with
IMPALA_TOOLCHAIN_PACKAGES_HOME=$IMPALA_TOOLCHAIN/test is
Impala-lzo. With a custom Impala-lzo, compilation succeeds.
Either Impala-lzo will be fixed or it will be removed.
- Core tests
Change-Id: I1ff641e503b2161baf415355452f86b6c8bfb15b
Reviewed-on: http://gerrit.cloudera.org:8080/15991
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This removes LD_LIBRARY_PATH and LD_PRELOAD from the
developer's shell and cleans it up. With the preceding
change, toolchain utilities like clang can be run without
a special LD_LIBRARY_PATH.
This fixes a bug where libjvm.so was registered as a
static instead of a shared library, which adds it to the
RUNPATH variable in the binary, which provides a default
search location that can be overriden by LD_LIBRARY_PATH.
Impala binaries don't have the rpath baked in for some
libraries, including Impala-lzo, libgcc and libstdc++.
, so we still need to set LD_LIBRARY_PATH when running
those. That is solved with wrapper scripts that sets
the environment variables only when invoking those
binaries, e.g. starting a daemon or running a backend
test. I added three scripts because there were 3 sets
of environment variables. The scripts are:
* run-binary.sh: just sets LD_LIBRARY_PATH
* run-jvm-binary.sh: sets LD_LIBRARY_PATH and CLASSPATH
* start-daemon.sh: sets LD_LIBRARY_PATH and CLASSPATH and
kerberos-related environment variables.
The binaries, in almost all cases, work fine without
those tweaks, because libstdc++ and libgcc are picked
up along with libkuduclient.so from the toolchain (they
are in the same directory). I decided to leave good enough
alone here. run-binary.sh and friends can be used in
any remaining edge cases to run binaries.
An alternative to the 3 scripts would be to have an
uber-script that set all the variables, but I felt
that it was better to be specific about what
each binary needed. Cleaning the LD_LIBRARY_PATH
mess up has given me a distaste for scattershot
setting of environment variables. I am open to
revisiting this.
Testing:
* Ran tests on centos 7
* Manually tested that my dev env with
LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu continued
to work (for now). All ubuntu 16.04 and 18.04 dev
envs that were set up with bootstrap_development.sh
will be in this state.
Change-Id: I61c83e6cca6debb87a12135e58ee501244bc9603
Reviewed-on: http://gerrit.cloudera.org:8080/14494
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
ARM64's L3 cacheline size is different according
to CPU vendor's architecture. If user defined
CACHELINESIZE_AARCH64 in impala-config-local.sh,
then we will use that value, if user did not
define it, then we will get the value from OS,
if fail, then we will use the default value 64.
Change-Id: Id56bfa63e4b6cd957c4997f10de78a5f4111f61f
Reviewed-on: http://gerrit.cloudera.org:8080/15555
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
So far we allowed short versions of Kudu macros in Impala by
defining KUDU_HEADERS_USE_SHORT_STATUS_MACROS in all compilation units.
I.e. DCHECK_OK meant KUDU_DCHECK_OK. This is a problem when we want to
define our own DCHECK_OK for an Impala Status object.
This commit limits KUDU_HEADERS_USE_SHORT_STATUS_MACROS to the Kudu
source files.
Unfortunately KUDU_HEADERS_USE_SHORT_STATUS_MACROS also controls some
Kudu log macros, e.g. KUDU_LOG. With the macro KUDU_LOG is defined to
LOG (glog). Without the macro KUDU_LOG is not defined, the Kudu library
expects that the clients will include 'kudu/client/stubs.h' which will
define KUDU_LOG to kudu::internal_logging::CerrLog. So far Impala didn't
use stubs.h (we define KUDU_HEADERS_NO_STUBS), and I think we still want
to use glog.
So in 'common/logging.h' I defined the KUDU logging macros to glog
macros (like if KUDU_HEADERS_USE_SHORT_STATUS_MACROS was still defined).
Change-Id: I06a65353d2a9eecf956e4ceb8d21eda2eebc69d5
Reviewed-on: http://gerrit.cloudera.org:8080/15596
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
In a previous change, the Kudu caching code from be/src/kudu/util was
copied to be/src/util/cache. This gets that code workable for Impala
and uses it for the DataCache.
This involves the following changes:
1. Fix up includes to point to Impala's cache implementation
2. Remove NVM Cache support and code (not used by Impala)
3. Remove CacheMetrics code (not used by Impala)
4. Move the cache code to the impala namespace
5. Modify cache-bench and cache-test to use Impala's backend
test infrastructure
6. Switch DataCache to use be/src/util/cache
These are only boilerplate changes. The cache implementation has not
meaningfully changed.
Testing:
- Ran core tests
- The modified version of Kudu's cache-test passes
- The modified version of Kudu's cache-bench runs successfully
Change-Id: I917f8352c9276373dd2761af986bf3487855271c
Reviewed-on: http://gerrit.cloudera.org:8080/15170
Reviewed-by: Thomas Tauber-Marshall <tmarshall@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch adds an additional build flag -full_tsan in addition to the
existing -tsan build flag. -full_tsan is equivalent to the current -tsan
behavior, and -tsan is changed to set the ignore_noninstrumented_modules
flag to true. ignore_noninstrumented_modules causes TSAN to ignore any
modules that are not TSAN-instrumented. This is necessary to get TSAN to
play nicely with Java, since Java is not TSAN-instrumented (see
https://wiki.openjdk.java.net/display/tsan/Main and JDK-8208520). While
this might decrease the number of issues surfaced by TSAN, it drastically
decreases the amount of noise produced by TSAN because the JVM is not
running TSAN-instrumented code. Without this flag set to true, almost
every single backend test fails with the error:
WARNING: ThreadSanitizer: data race (pid=12939)
Write of size 1 at 0x7fcbe379c4c6 by thread T31:
#0 strncpy /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:650 (unifiedbetests+0x1b2a4ad)
#1 <null> <null> (libjvm.so+0x90e706)
This patch fixes various TSAN bugs (e.g. data races) reported while
running backend tests and E2E against a TSAN build (it does not make
Impala completely TSAN-clean). This patch makes the following changes:
* Fixes several bugs involving issues with updating shared variables
between threads
* Fixes a few race conditions in test classes
* Where possible, existing locks are used to fix any data races; in cases
where the locking logic is non-trivial, atomics are used
* There are a few places where variables are marked as 'volatile'
presumably for synchronization purposes; TSAN flags these 'volatile'
variables as unsafe, and according to
https://github.com/isocpp/CppCoreGuidelines/blob/master/CppCoreGuidelines.md#Rconc-volatile
using 'volatile' for synchronization is dangerous; in these cases, the
'volatile' variables are changed to 'atomic' variables
* This patch adds a suppression file (bin/tsan-suppresions.txt) similar to
the UBSAN suppresion file (bin/ubsan-suppresions.txt)
Testing:
* Ran exhaustive tests
* Ran core tests w/ ASAN build
* Manually re-ran backend tests against a TSAN build and made sure the
reported errors are gone
Change-Id: I3d7ef5c228afd5882e145e6f53885b355d6c25a0
Reviewed-on: http://gerrit.cloudera.org:8080/15116
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
According to the ccache manpage, the correct way to compose it with
distcc is to use CCACHE_PREFIX. I think this explains why local
ccache wasn't working for me.
I updated the distcc wrapper scripts to do this instead and
confirmed that it works - after doing a clean and rebuilding
the same branch, it is much faster and "ccache -s" shows a
bunch of cache hit.
ccache on the distcc server side still works.
Change-Id: Ie0b080709bd765056b9296d3deb805038fc01e5d
Reviewed-on: http://gerrit.cloudera.org:8080/14408
Reviewed-by: Andrew Sherman <asherman@cloudera.com>
Tested-by: Tim Armstrong <tarmstrong@cloudera.com>
This converts tests with trivial main() functions to the unified
executable. This means that the code change is strictly removing
main() functions and updating the CMakeLists.txt files. Any test
that requires a change larger than that will be addressed
separately. The only exceptions are:
- exec/incr-stats-util-test.cc requires naming changes to avoid
conflicts with util/rle-test.cc
- runtime/decimal-test.cc simplified the naming to make the
CMakeLists.txt arguments easier.
The new test libraries are marked STATIC, because they are linked
into a single binary (unifiedbetests) and googletest has problems
with tests in shared libraries.
Converting this set of tests saves about 18GB of disk
space for a debug build and saves a minute or two of link time.
For any CMakeLists.txt that has unified tests, this adds a comment
for each test that is not unified.
Testing:
- Ran backend tests in DEBUG and ASAN modes on Centos7
- Ran backend tests in DEBUG mode on Centos6
Change-Id: I840d0f9b70edb3a7195a2a33b21fd2874d4c52bd
Reviewed-on: http://gerrit.cloudera.org:8080/13515
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Makefile was updated to include zstd in the ${IMPALA_HOME}/toolchain
directory. Other changes were made to make zstd headers and libs
accessible.
Class ZstandardCompressor/ZstandardDecompressor was added to provide
interfaces for calling ZSTD_compress/ZSTD_decompress functions. Zstd
supports different compression levels (clevel) from 1 to
ZSTD_maxCLevel(). Zstd also supports -ive clevels, but since the -ive
values represents uncompressed data they won't be supported. The default
clevel is ZSTD_CLEVEL_DEFAULT.
HdfsParquetTableWriter was updated to support ZSTD codec. The
new codecs can be set using existing query option as follows:
set COMPRESSION_CODEC=ZSTD:<clevel>;
set COMPRESSION_CODEC=ZSTD; // uses ZSTD_CLEVEL_DEFAULT
Testing:
- Added unit test in DecompressorTest class with ZSTD_CLEVEL_DEFAULT
clevel and a random clevel. The test unit decompresses an input
compressed data and validates the result. It also tests for
expected behavior when passing an over/under sized buffer for
decompressing.
- Added unit tests for valid/invalid values for COMPRESSION_CODEC.
- Added e2e test in test_insert_parquet.py which tests writing/read-
ing (null/non-null) data into/from a table (w different data type
columns) using multiple codecs. Other existing e2e tests were
updated to also use parquet/zstd table format.
- Manual interoperability tests were run between Impala and Hive.
Change-Id: Id2c0e26e6f7fb2dc4024309d733983ba5197beb7
Reviewed-on: http://gerrit.cloudera.org:8080/13507
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Added two new metrics tmp-file-mgr.scratch-space-bytes-used-high-water-mark
& tmp-file-mgr.scratch-space-bytes-used for tracking HWM and current
value for spilled bytes, respectively.
A new class AtomicHighWaterMarkGauge was added to keep track of the HWM
value. The new class also encapsulates a metric object which keeps track
of the current value for the spilled bytes.
The current value is incremented every time a new range is allocated from
a temporary file. The current value for spilled bytes is decremented when
a temporary file is closed. The new metrics are not updated when ranges
are recycled from a file. We can add a new metric in future for keeping
track of actual spilled bytes. The HWM value is updated whenever the
current value is greater than the HWM value.
Testing:
- Added new unit tests to the metrics-test test case.
- E2E testing for both the metrics by running concurrent spilling queries
and ensuring that both the current value metric and the HWM metric were
behaving as expected. Ran concurrent queries and monitored the metrics
on the impala daemon's metric page.
Change-Id: Ia1b3dd604c7234a8d8af34d70ca731544a46d298
Reviewed-on: http://gerrit.cloudera.org:8080/12956
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
