TestAcidRowValidation.test_row_validation fails with tuple caching
correction verification. The test creates a Full Hive ACID table
with a file using valid write ids, mimicking a streaming ingest.
As the valid write ids change, the scan of that file produces
different rows without the file changing. Tuple caching currently
doesn't understand valid write ids, so this produces incorrect
results.
This marks Full Hive ACID tables as ineligible for caching until
valid write ids can be supported properly. Insert-only tables are
still eligible.
Testing:
- Added test cases to TupleCacheTest
- Ran TestAcidRowValidation.test_row_validation with correctness
verification
Change-Id: Icab9613b8e2973aed1d34427c51d2fd8b37a9aba
Reviewed-on: http://gerrit.cloudera.org:8080/23454
Reviewed-by: Yida Wu <wydbaggio000@gmail.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
IMPALA-6984 changed the behavior to cancel backends when the
query reaches the RETURNED_RESULTS state. This ran into a regression
on large clusters where a query would end up waiting 10 seconds.
IMPALA-10047 reverted the core piece of the change.
For tuple caching, we found that a scan node can get stuck waiting
for a global runtime filter. It turns out that the coordinator will
not send out global runtime filters if the query is in a terminal
state. Tuple caching was causing queries to reach the RETURNED_RESULTS
phase before the runtime filter could be sent out. Reenabling the core
part of IMPALA-6984 sends out a cancel as soon as the query transitions
to RETURNED_RESULTS and wakes up any fragment instances waiting on
runtime filters.
The underlying cause of IMPALA-10047 is a tangle of locks that causes
us to exhaust the RPC threads. The coordinator is holding a lock on the
backend state while it sends the cancel synchronously. Other backends
that complete during that time run Coordinator::BackendState::LogFirstInProgress(),
which iterates through backend states to find the first that is not done.
The check to see if a backend state is done takes a lock on the backend
state. The problem case is that the coordinator may be sending a cancel
to a backend on itself. In that case, it needs an RPC thread on the coordinator
to be available to process the cancel. If all of the RPC threads are
processing updates, they can all call LogFirstInProgress() and get stuck
on the backend state lock for the coordinator's fragment. In that case,
it becomes a temporary deadlock as the cancel can't be processed and the
coordinator won't release the lock. It only gets resolved by the RPC timing
out.
To resolve this, this changes the Cancel() method to drop the lock while
doing the CancelQueryFInstances RPC. It reacquires the lock when it finishes
the RPC.
Testing:
- Hand tested with 10 impalads and control_service_num_svc_threads=1
Without the fix, it reproduces easily after reverting IMPALA-10047.
With the fix, it doesn't reproduce.
Change-Id: Ia058b03c72cc4bb83b0bd0a19ff6c8c43a647974
Reviewed-on: http://gerrit.cloudera.org:8080/23264
Reviewed-by: Yida Wu <wydbaggio000@gmail.com>
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Cleans up repetitive patterns in pom.xml.
Centralize plugin configuration in pluginManagement. Replace inline
maven-compiler-plugin configuration with newer maven.compiler.release
and update to latest plugin version.
Centralize common dependencies in dependencyManagement, including
exclusions when appropriate. Remove exclusions that are no longer
relevant.
Compared before and after with dependency:tree; only difference is that
commons-cli now comes from hadoop and jersey-serv{let,er} are
effectively excluded; all versions matched. Also ensured
USE_APACHE_COMPONENTS=true compiles.
Adds com.amazonaws:aws-java-sdk-bundle to exclusion checking to ensure
it's not accidentally included alongside impala-minimal-s3a-aws-sdk.
Removes missed io.netty exclusion from IMPALA-12816.
Updates commons-dbcp2 to 2.12.0 to match Hive.
Change-Id: If96649840e23036b4a73ee23e8d12516497994f0
Reviewed-on: http://gerrit.cloudera.org:8080/23432
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Scheduling does not sort scan ranges by modification time. When a new
file is added to a table, its order in the list of scan ranges is
not based on modification time. Instead, it is based on which partition
it belongs to and what its filename is. A new file that is added early
in the list of scan ranges can cause cascading differences in scheduling.
For tuple caching, this means that multiple runtime cache keys could
change due to adding a single file.
To minimize that disruption, this adds the ability to sort the scan
ranges by modification time and schedule scan ranges oldest to newest.
This enables it for scan nodes that feed into tuple cache nodes
(similar to deterministic scan range assignment).
Testing:
- Modified TestTupleCacheFullCluster::test_scan_range_distributed
to have stricter checks about how many cache keys change after
an insert (only one should change)
- Modified TupleCacheTest#testDeterministicScheduling to verify that
oldest to newest scheduling is also enabled.
Change-Id: Ia4108c7a00c6acf8bbfc036b2b76e7c02ae44d47
Reviewed-on: http://gerrit.cloudera.org:8080/23228
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Downstream error reports pointed out that the toolchain version picked
up for IMPALA-14139 contains toolchain binaries for Red Hat 9 (and
compatibles) that require at least the 9.5 minor version because of
OpenSSL library requirements. This was caused by the toolchain binary
build process not using package repo pinning for the redhat9 build
container definition, which caused the container process to install
"latest" packages, in this case packages released in Rocky / Red Hat
9.5.
This patch bumps the toolchain ID to a version in which the redhat9
binaries were produced in a build container "moved back in time" to the
9.2 release by pinning the package repos to the Rocky Linux 9.2 state,
using the Rocky Vault.
The patch also picks up a buffer overflow mitigation for the ORC
library.
Change-Id: I5c6921afdc69a4a6644b619de6b8d4e4cc69e601
Reviewed-on: http://gerrit.cloudera.org:8080/23448
Reviewed-by: Riza Suminto <riza.suminto@cloudera.com>
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
When admission control is enabled, but max memory for pool is not
configured, Impala skips memory-based admission completely, i.e. it
doesn't even take available host memory into account.
This behavior can lead to admitting many queries with large memory
consumption, potentially causing query failures due to memory
exhaustion.
Fixing the above behavior might cause regressions in some workloads,
so this patch just adds a new log message which makes it clear why
a query gets admitted, and also mentions possible failures.
Change-Id: Ib98482abc0fbcb53552adfd89cf6d157b17527fd
Reviewed-on: http://gerrit.cloudera.org:8080/23438
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch adds a new MetaProvider called MultiMetaProvider, which is
capable of handling multiple MetaProviders at once, prioritizing one
primary provider over multiple secondary providers. The primary
provider handles some methods exclusively for deterministic behavior.
In database listings, if one database name occurs multiple times the
contained tables are merged under that database name; if the two
separate databases contain a table with the same name, the query
analyzation fails with an error.
This change also modifies the local catalog implementation's
initialization. If catalogd is deployed, then it instantiates the
CatalogdMetaProvider and checks if the catalog configuration directory
is set as a backend flag. If it's set, then it tries to load every
configuration from the folder, and tries to instantiate the
IcebergMetaProvider from those configs. If the instantiation fails, an
error is reported to the logs, but the startup is not interrupted.
Tests:
- E2E tests for multi-catalog behavior
- Unit test for ConfigLoader
Change-Id: Ifbdd0f7085345e7954d9f6f264202699182dd1e1
Reviewed-on: http://gerrit.cloudera.org:8080/22878
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Zoltan Borok-Nagy <boroknagyz@cloudera.com>
Removes IMPALA_JAVA_HOME_OVERRIDE and updates version selection. In
order of priority
1. If IMPALA_JDK_VERSION is set, use the OS JDK version from a known
location. This is primarily used when also installing the JDK as part
of automated builds.
2. If JAVA_HOME is set, use it.
3. Look for the system default JDK.
The IMPALA_JDK_VERSION variable is no longer modified to avoid issues
when sourcing impala-config.sh multiple times. JAVA_HOME will be
modified if IMPALA_JDK_VERSION is set; both must be unset to restore
using the system default Java.
If switching between JDKs, now prefer setting JAVA_HOME. If relying on
system Java, unset JAVA_HOME after e.g. update-java-alternatives.
The detected Java version is set in IMPALA_JAVA_TARGET, which is used to
add Java 9+ options and configure the Java compilation target.
Eliminates IMPALA_JDK_VERSION_NUM as it's value was always identical to
IMPALA_JAVA_TARGET.
Stops printing from impala-config-java.sh. It made the output from
impala-config.sh look strange, and the decisions can all be clearly
determined from impala-config.sh printed variables later or the packages
installed in bootstrap_system.sh.
Fixes JAVA_HOME in bootstrap_build.sh on ARM64 systems.
Change-Id: I68435ca69522f8310221a0f3050f13d86568b9da
Reviewed-on: http://gerrit.cloudera.org:8080/23434
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This changes the default behavior of the tuple cache to consider
cost when placing the TupleCacheNodes. It tries to pick the best
locations within a budget. First, it eliminates unprofitable locations
via a threshold. Next, it ranks the remaining locations by their
profitability. Finally, it picks the best locations in rank order
until it reaches the budget.
The threshold is based on the ratio of processing cost for regular
execution versus the processing cost for reading from the cache.
If the ratio is below the threshold, the location is eliminated.
The threshold is specified by the tuple_cache_required_cost_reduction_factor
query option. This defaults to 3.0, which means that the cost of
reading from the cache must be less than 1/3 the cost of computing
the value normally. A higher value makes this more restrictive
about caching locations, which pushes in the direction of lower
overhead.
The ranking is based on the cost reduction per byte. This is given
by the formula:
(regular processing cost - cost to read from cache) / estimated serialized size
This prefers locations with small results or high reduction in cost.
The budget is based on the estimated serialized size per node. This
limits the total caching that a query will do. A higher value allows more
caching, which can increase the overhead on the first run of a query. A lower
value is less aggressive and can limit the overhead at the expense of less
caching. This uses a per-node limit as the limit should scale based on the
size of the executor group as each executor brings extra capacity. The budget
is specified by the tuple_cache_budget_bytes_per_executor.
The old behavior to place the tuple cache at all eligible locations is
still available via the tuple_cache_placement_policy query option. The
default is the cost_based policy described above, but the old behavior
is available via the all_eligible policy. This is useful for correctness
testing (and the existing tuple cache test cases).
This changes the explain plan output:
- The hash trace is only enabled at VERBOSE level. This means that the regular
profile will not contain the hash trace, as the regular profile uses EXTENDED.
- This adds additional information at VERBOSE to display the cost information
for each plan node. This can help trace why a particular location was
not picked.
Testing:
- This adds a TPC-DS planner test with tuple caching enabled (based on the
existing TpcdsCpuCostPlannerTest)
- This modifies existing tests to adapt to changes in the explain plan output
Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Reviewed-on: http://gerrit.cloudera.org:8080/23219
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This is a preparatory change for cost-based placement for
TupleCacheNodes. It reorders planning so that the processing cost and
filtered cardinality are calculated before running the TupleCachePlanner.
This computes the processing cost when enable_tuple_cache=true.
It also displays the cost information in the explain plan output
when enable_tuple_cache=true. This does not impact the adjustment
of fragment parallelism, which continues to be controlled by the
compute_processing_cost option.
This uses the processing cost to calculate a cumulative processing
cost in the TupleCacheInfo. This is all of the processing cost below
this point including other fragments. This is an indicator of how
much processing a cache hit could avoid. This does not accumulate the
cost when merging the TupleCacheInfo due to a runtime filter, as that
cost is not actually being avoided. This also computes the estimated
serialized size for the TupleCacheNode based on the filtered
cardinality and the row size.
Testing:
- Ran a core job
Change-Id: If78f5d002b0e079eef1eece612f0d4fefde545c7
Reviewed-on: http://gerrit.cloudera.org:8080/23164
Reviewed-by: Yida Wu <wydbaggio000@gmail.com>
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Tested-by: Michael Smith <michael.smith@cloudera.com>
In Apache Hive 3, HMS doesn't provide the API to retrive WriteEvents
info of a given transaction. COMMIT_TXN event just contains a
transaction id so Impala can't process it.
This patch ignores COMMIT_TXN events when building on Apache Hive 3.
Some tests in MetastoreEventsProcessorTest and EventExecutorServiceTest
are skipped due to this.
Tests:
- Manually tested on Apache Hive 3. Verified that EventProcessor still
works after receiving COMMIT_TXN events.
- Passed some tests in MetastoreEventsProcessorTest and
EventExecutorServiceTest that previously failed due to EventProcessor
going into ERROR state.
Change-Id: I863e39b3702028a14e83fed1fe912b441f2c28db
Reviewed-on: http://gerrit.cloudera.org:8080/23117
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
The feature of hms_event_incremental_refresh_transactional_table is
already mature that it has been enabled for years. We'd like to
deprecate the feature of turning it off. However, for older Hive
versions like Apache Hive 3 that don't provide sufficient APIs for
Impala to process COMMIT_TXN events, users can still turn this off.
This patch skips
test_no_hms_event_incremental_refresh_transactional_table when running
on CDP Hive.
To run the test on Apache Hive 3, adjust the test to create ACID table
using tblproperties instead of "create transactional table" statement.
Tests:
- Ran the test on CDP Hive and Apache Hive 3.
Change-Id: I93379e5331072bec1d3a4769f7d7ab59431478ee
Reviewed-on: http://gerrit.cloudera.org:8080/23435
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Currently, RELOAD events of partitioned table are processed one after
the other. Processing them one by one acquires the table lock multiple
times to load individual partitions in sequence. This also keeps the
table version changing which impacts performance of coordinators in
local-catalog mode - query planning needs retry to handle
InconsistentMetadataFetchException due to table version changes.
This patch handles the batch processing logic RELOAD events on same
table by reusing the exisiting logic of BatchPartitionEvent. This
implementation adds four new methods canBeBatched(),addToBatchEvents(),
getPartitionForBatching(), getBatchEventType()(pre-requisites to reuse
batching logic) to the RELOAD event class.
Testing:
- Added an end-to-end to verify the batching.
Change-Id: Ie3e9a99b666a1c928ac2a136bded1e5420f77dab
Reviewed-on: http://gerrit.cloudera.org:8080/23159
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Clang static analyzer found a potential memory leak in
TmpFileMgr. In some cases we forget the deletion of a
newly created TmpFileRemote object. This patch replaces
the raw pointer with a unique_ptr.
Change-Id: I5a516eab1a946e7368c6059f8d1cc430d2ee19e9
Reviewed-on: http://gerrit.cloudera.org:8080/23431
Reviewed-by: Riza Suminto <riza.suminto@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Fixes the code that detects if the OpenTelemetry collector is using
TLS. Previously, the code only worked properly when the collector URL
was all lowercase. Also removes unnecessary checks that could cause
TLS to be enabled even when the collector URL scheme was not https.
Testing accomplished by adding new ctest tests.
Change-Id: I3bf74f1353545d280575cdb94cf135e55c580ec7
Reviewed-on: http://gerrit.cloudera.org:8080/23397
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
All functions in the SpanManager class operate under the assumption
that child_span_mu_ in the SpanManager class will be locked before
the ClientRequestState lock. However, the
ImpalaServer::ExecuteInternal function takes the ClientRequestState
lock before calling SpanManager::EndChildSpanPlanning. If another
function in the SpanManager class has already taken the
child_span_mu_ lock and is waiting for the ClientRequestState lock,
a deadlock occurs.
This issue was found by running end-to-end tests with OpenTelemetry
tracing enabled and a release buildof Impala.
Testing accomplished by re-running the end-to-end tests with
OpenTelemetry tracing enabled and verifying that the deadlock no
longer occurs.
Change-Id: I7b43dba794cfe61d283bdd476e4056b9304d8947
Reviewed-on: http://gerrit.cloudera.org:8080/23422
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Contains the following improvements to the Impala queries as
OpenTelemetry traces custom cluster tests:
1. Supporting code for asserting traces was moved to
'tests/util/otel_trace.py'. The moved code was modified to remove
all references to 'self'. Since this code used
'self.assert_impalad_log_contains', it had to be modified so the
caller provides the correct log file path to search. The
'__find_span_log' function was updated to call a new generic file
grep function to run the necessary log file search regex. All
other code was moved unmodified.
2. Classes 'TestOtelTraceSelectsDMLs' and 'TestOtelTraceDDLs'
contained a total of 11 individual tests that used the
'unique_database' fixture. When this fixture is used in a test, it
results in two DDLs being run before the test to drop/create the
database and one DDL being run after the test to drop the database.
These classes now create a test database once during 'setup_class'
and drop it once during 'teardown_class' because creating a new
database for each test was unnecessary. This change dropped test
execution time from about 97 seconds to about 77 seconds.
3. Each test now has comments describing what the test is asserting.
4. The unnecessary sleep in 'test_query_exec_fail' was removed saving
five seconds of test execution time.
5. New test 'test_dml_insert_fail' added. Previously, the situation
where an insert DML failed was not tested. The test passed without
any changes to backend code.
6. Test 'test_ddl_createtable_fail' is greatly simplified by using a
debug action to fail the query instead of multiple parallel
queries where one dropped the database the other was inserting
into. The simplified setup eliminated test flakiness caused by
timing differences and sped up test execution by about 5 seconds.
7. Fixed test flakiness was caused by timing issues. Depending on
when the close process was initiated, span events are sometimes in
the QueryExecution span and sometimes in the Close span. Test
assertions cannot handle these situations. All span event
assertions for the Close span were removed. IMPALA-14334 will fix
these assertions.
8. The function 'query_id_from_ui' which retrieves the query profile
using the Impala debug ui now makes multiple attempts to retrieve
the query. In slower test situations, such as ASAN, the query may
not yet be available when the function is called initially which
used to cause tests to fail. This test flakiness is now eliminated
through the addition of the retries.
Testing accomplished by running tests in test_otel_trace.py both
locally and in a full Jenkins build.
Generated-by: Github Copilot (Claude Sonnet 3.7)
Change-Id: I0c3e0075df688c7ae601c6f2e5743f56d6db100e
Reviewed-on: http://gerrit.cloudera.org:8080/23385
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
IMPALA-14349 caused a regression due to change in
FileMetadataLoader.createFd(). When default FS is S3, all files is S3
should not have any FileBlock. However, after IMPALA-14349, CTAS query
that scans functional.alltypes table in S3 hit following Preconditions
in HdfsScanNode.java:
if (!fsHasBlocks) {
Preconditions.checkState(fileDesc.getNumFileBlocks() == 0);
This is because FileMetadataLoader.createFd() skip checking if the
originating FileSystem support supportsStorageIds() or not. S3
dataloading from HDFS snapshot consistently failed due this regression.
This patch fix the issue by restoring FileMetadataLoader.createFd() to
its state before IMPALA-14349. It also make
FileMetadataLoader.createFd() calls more consistent by not allowing null
parameters except for 'absPath' that is only not null for Iceberg data
files. Generalize numUnknownDiskIds parameter from Reference<Long> to
AtomicLong for parallel usage.
Testing:
Pass dataloading, FE_TEST, EE_TEST, and CLUSTER_TEST in S3.
Change-Id: Ie16c5d7b020a59b5937b52dfbf66175ac94f60cd
Reviewed-on: http://gerrit.cloudera.org:8080/23423
Reviewed-by: Zoltan Borok-Nagy <boroknagyz@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Update the following elements of the Impala build environment to enable
builds on Ubuntu 24.04:
- Recognize and handle (where necessary) Ubuntu 24.04 in various
bootstrap scripts (bootstrap_system.sh, bootstrap_toolchain.py, etc.)
- Bump IMPALA_TOOLCHAIN_ID to an official toolchain build that contains
Ubuntu 24.04-specific binary packages
- Bump binutils to 2.42, and
- Bump the GDB version to 12.1-p1, as required by the new toolchain
version
- Update unique_ptr usage syntax in be/src/util/webserver-test.cc to
compensate for new GLIBC funtion prototypes:
System headers in Ubuntu 24.04 adopted attributes on several widely
used function prototypes. Such attributes are not considered to be part
of the function's signature during template evaluation, so GCC throws a
warning when such a function is passed as a template argument, which
breaks the build, as warnings are treated as errors.
webserver-test.cc uses pclose() as the deleter for a unique_ptr in a
utility function. This patch encapsulates pclose() and its attributes in
an explicit specialization for std::default_delete<>, "hiding" the
attributes inside a functor.
The particular solution was inspired by Anton-V-K's proposal in
https://gist.github.com/t-mat/5849549
This commit builds on an earlier patch for the same purpose by Michael
Smith: https://gerrit.cloudera.org/c/23058/
Change-Id: Ia4454b0c359dbf579e6ba2f9f9c44cfa3f1de0d2
Reviewed-on: http://gerrit.cloudera.org:8080/23384
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Incorrect marker was included at the beginning (`--` instead of `---`)
which caused Apache Infrastructure to reject the file.
Tested with a YAML validator.
Change-Id: I554ba9b12655b1dbc8d38f6d533d51be92578369
Reviewed-on: http://gerrit.cloudera.org:8080/23426
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Tested-by: Michael Smith <michael.smith@cloudera.com>
Consume various utility functions added as part of previous changes.
Testing accomplished by running exhaustive tests in
test_query_log.py, test_query_live.py, and test_otel_trace.py both
locally and in jenkins.
Change-Id: If42a8b5b6fdb43fb2bb37dd2a3be4668e8a5e283
Reviewed-on: http://gerrit.cloudera.org:8080/23234
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
TestImpalaShell.test_cancellation start failing when run with Python 3.9
with following error message
RuntimeError: reentrant call inside <_io.BufferedWriter name='<stderr>'>
This patch is a quick fix the by changing the stderr write from using
print() to os.write(). Note that the thread-safetyness isssue within
_signal_handler in impala_shell.py during query cancellation still
remains.
Testing:
Run and pass test_cancellation in RHEL9 with Python 3.9.
Change-Id: I5403c7b8126b1a35ea841496fdfb6eb93e83376e
Reviewed-on: http://gerrit.cloudera.org:8080/23416
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Before this change, query plans and profile reported only a single
partition even for partitioned Iceberg tables, which was misleading
for users.
Now we can display the number of scanned partitions correctly for
both partitioned and unpartitioned Iceberg tables. This is achieved by
extracting the partition values from the file descriptors and storing
them in the IcebergContentFileStore. Instead of storing this information
redundantly in all file descriptors, we store them in one place and
reference the partition metadata in the FDs with an id.
This also gives the opportunity to optimize memory consumption in the
Catalog and Coordinator as well as reduce network traffic between them
in the future.
Time travel is handled similarly to oldFileDescMap. In that case
we don't know the total number of partitions in the old snapshot,
so the output is [Num scanned partitions]/unknown.
Testing:
- Planner tests
- E2E tests
- partition transforms
- partition evolution
- DROP PARTITION
- time travel
Change-Id: Ifb2f654bc6c9bdf9cfafc27b38b5ca2f7b6b4872
Reviewed-on: http://gerrit.cloudera.org:8080/23113
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
test_load_table_with_primary_key_attr has been intermittently fail with
following error message since Python3 switch in IMPALA-14333.
dbm.error: db type could not be determined
Error happen inside load_unique_col_metadata. But both
load_unique_col_metadata and persist_unique_col_metadata are seemingly
unused anymore.
While the root cause is still unknown, commenting
load_unique_col_metadata in db_connection.py can make the tests pass.
Tests are not affected because the they do not assert against any unique
columns. This patch implement that workaround until we can revisit the
issue with query generator infrastructure.
Fix few flake8 warnings in db_connection.py.
Testing:
Loop and pass test_show_create_table 10 times using downstream jenkins.
Change-Id: I2caf6b9cc5a4f06237323cb32bbd47afef576fa1
Reviewed-on: http://gerrit.cloudera.org:8080/23412
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
New test introduced in IMPALA-14290 depends on the Iceberg versions,
because newer ones (e.g. 1.5.2) will show
{"start_time_month":"646","end_time_day":"19916"}
instead of
{"start_time_day":null,"end_time_month":null,"start_time_month":"646","end_time_day":"19916"}
The test now accepts both cases.
Testing:
* ran query_test/test_iceberg.py with both Iceberg 1.3.1 and 1.5.2
Change-Id: I17e368ac043d1fbf80a78dcac6ab1be5a297b6ea
Reviewed-on: http://gerrit.cloudera.org:8080/23389
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Impalad crash (hitting DCHECK) when both enable_workload_mgmt and
gen_experimental_profile enabled. This is because lambda function
process_exec_profile expect "Averaged Fragment" node exist in query
profile. But it is actually not exist in V2 query profile.
This patch fix the issue by gathering ScratchBytesWritten,
ScannerIoWaitTime, and DataCacheHitBytes counters differently in V2
profile.
Testing:
- Add TestQueryLogTableHS2::test_with_experimental_profile.
- Manually start minicluster with both enable_workload_mgmt and
gen_experimental_profile flag enabled. Run few queries and confirm no
crash happen. Also verify that the columns of sys.impala_query_log
that summarize the scan node counters are correct.
Change-Id: Iccb4ad9279b0d66479b1e7816ffc732028e71734
Reviewed-on: http://gerrit.cloudera.org:8080/23396
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch mainly implement the creation/drop of paimon table
through impala.
Supported impala data types:
- BOOLEAN
- TINYINT
- SMALLINT
- INTEGER
- BIGINT
- FLOAT
- DOUBLE
- STRING
- DECIMAL(P,S)
- TIMESTAMP
- CHAR(N)
- VARCHAR(N)
- BINARY
- DATE
Syntax for creating paimon table:
CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
(
[col_name data_type ,...]
[PRIMARY KEY (col1,col2)]
)
[PARTITIONED BY (col_name data_type [COMMENT 'col_comment'], ...)]
STORED AS PAIMON
[LOCATION 'hdfs_path']
[TBLPROPERTIES (
'primary-key'='col1,col2',
'file.format' = 'orc/parquet',
'bucket' = '2',
'bucket-key' = 'col3',
];
Two types of paimon catalogs are supported.
(1) Create table with hive catalog:
CREATE TABLE paimon_hive_cat(userid INT,movieId INT)
STORED AS PAIMON;
(2) Create table with hadoop catalog:
CREATE [EXTERNAL] TABLE paimon_hadoop_cat
STORED AS PAIMON
TBLPROPERTIES('paimon.catalog'='hadoop',
'paimon.catalog_location'='/path/to/paimon_hadoop_catalog',
'paimon.table_identifier'='paimondb.paimontable');
SHOW TABLE STAT/SHOW COLUMN STAT/SHOW PARTITIONS/SHOW FILES
statements are also supported.
TODO:
- Patches pending submission:
- Query support for paimon data files.
- Partition pruning and predicate push down.
- Query support with time travel.
- Query support for paimon meta tables.
- WIP:
- Complex type query support.
- Virtual Column query support for querying
paimon data table.
- Native paimon table scanner, instead of
jni based.
Testing:
- Add unit test for paimon impala type conversion.
- Add unit test for ToSqlTest.java.
- Add unit test for AnalyzeDDLTest.java.
- Update default_file_format TestEnumCase in
be/src/service/query-options-test.cc.
- Update test case in
testdata/workloads/functional-query/queries/QueryTest/set.test.
- Add test cases in metadata/test_show_create_table.py.
- Add custom test test_paimon.py.
Change-Id: I57e77f28151e4a91353ef77050f9f0cd7d9d05ef
Reviewed-on: http://gerrit.cloudera.org:8080/22914
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Riza Suminto <riza.suminto@cloudera.com>
JniFrontend.getDbs() returns the thrift representation of all the dbs.
This might trigger multiple getPartialCatalogObject requests to catalogd
and could fail in InconsistentMetadataFetchException, e.g. if a db is
removed after coordinator fetching the db name list and before
coordinator fetching the msDb of that db.
This patch fixes the issue by retrying the above steps when hitting
InconsistentMetadataFetchException, similar to what other methods in
Frontend do. Adds getThriftDbs() in Frontend to directly return the
thrift db list so JniFrontend can use it directly and the retry can be
added inside Frontend.java.
TestAuthorization.test_local_catalog_show_dbs_with_transient_db is an
existing test to verify a similar problem. Running this test with
authorization disabled can reproduce the current bug. So this patch
extracts the test code into
TestLocalCatalogRetries._run_show_dbs_with_transient_db() and share it
in both authz enabled and disabled tests.
Tests
- Ran TestLocalCatalogRetries.test_show_dbs_retry 60 times. Without the
fix, it fails in about a dozen times.
Change-Id: Ib337f88a2ac0f35142417f6cee51d30497f12845
Reviewed-on: http://gerrit.cloudera.org:8080/23402
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Added '-udf_devel_package' option to buildall.sh. This generates
impala-udf-devel rpm which includes udf headers and static libraries -
ImpalaUdf-retail.a and ImpalaUdf-debug.a.
Testing:
- Tested that rpm is generated using build script:
./buildall.sh -release_and_debug -notests -udf_devel_package
- Tested that the rpm is also generated using standalone script:
./bin/make-impala-udf-devel-rpm.sh
- Generated impala-udf-devel package and tested compiling
impala_udf_samples:
https://github.com/cloudera/impala-udf-samples
Change-Id: I5b85df9c3f680a7e5551f067a97a5650daba9b50
Reviewed-on: http://gerrit.cloudera.org:8080/23060
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
With this patch we create Iceberg file descriptors from
LocatedFileStatus objects during IcebergFileMetadataLoader's
parallelListing(). This has the following benefits:
* We parallelize the creation of Iceberg file descriptor objects
* We don't need to maintain a large hash map with all the
LocatedFileStatus objects at once. Now we only need to keep a few
LocatedFileStatus objects per partition in memory while we are
converting them to Iceberg file descriptors. I.e., the GC is free to
destroy the LocatedFileStatus objects we don't use anymore.
This patch retires startup flag 'iceberg_reload_new_files_threshold'.
Since IMPALA-13254 we only list partitions that have new data files,
and we load them in parallel, i.e. efficient incremental table loading
is already covered. From that point the startup flag only added
unnecessary code complexity.
Measurements
I created two tables (from tpcds.store_sales) to measure table loading
times for large tables:
Table #1:
PARTITIONED BY SPEC(ss_item_sk, BUCKET(5, ss_sold_time_sk))
partitions: 107818
files: 754726
Table #2:
PARTITIONED BY SPEC(ss_item_sk)
partitions: 18000
files: 504224
Time taken in IcebergFileMetadataLoader.load() during full table reload:
+----------+-------+------+---------+
| | Base | New | Speedup |
+----------+-------+------+---------+
| Table #1 | 17.3s | 8.1s | 2.14 |
| Table #2 | 7.8s | 4.3s | 1.8 |
+----------+-------+------+---------+
I measured incremental table loading only for Table #2 (since there are
more files per partition this is the worse scenario for the new code, as
it only uses file listings, and each new file were created in a separate
partition)
Time taken in IcebergFileMetadataLoader.load() during incremental table
reload:
+------------+------+------+---------+
| #new files | Base | New | Speedup |
+------------+------+------+---------+
| 1 | 1.4s | 1.6s | 0.9 |
| 100 | 1.5s | 1.9s | 0.8 |
| 200 | 1.5s | 1.5s | 1 |
+------------+------+------+---------+
We lose a few tenths of a second, but I think the simplified code
justifies it.
Testing:
* some tests were updated because we we don't have
startup flag 'iceberg_reload_new_files_threshold' anymore
Change-Id: Ia1c2a7119d76db7ce7c43caec2ccb122a014851b
Reviewed-on: http://gerrit.cloudera.org:8080/23363
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
IMPALA-13850 (part 4) modify CatalogServiceCatalog.getDb() to delay
looking up catalog cache until initial reset() is complete.
EventProcessor can start processing event before reset() happen and
obtain versionLock_.readLock() when calling
CatalogServiceCatalog.getDbProperty(). Later on, it will hit deadlock
when attempting to obtain versionLock_.writeLock() through getDb() /
waitInitialResetCompletion(). This lock upgrade from read to write is
unsafe.
This patch mitigate the issue by changing waitInitialResetCompletion()
to not acquire write lock. After this patch, it will sleep for 100ms
before loop and checking again if initial reset has complete. Modified
CatalogResetManager.fetchingDbs_ to ConcurrentLinkedQueue so that
isActive() can be called without holding write lock.
Add helper class ReadLockAndLookupDb and WriteLockAndLookupDb. Both will
call waitInitialResetCompletion() before obtaining the appropriate lock.
In case of WriteLockAndLookupDb, it additionally will call
resetManager_.waitOngoingMetadataFetch() to ensure dbCache_ lookup is
safe for write purpose.
Skip calling catalog_.startEventsProcessor() in JniCatalog constructor.
Instead, let CatalogServiceCatalog.reset() start it at the end of cache
population.
Added @Nullable annotations on CatalogServiceCatalog methods that can
return null. Fixed some null check warnings that shows up afterwards.
Remove dead code CatalogServiceCatalog.addUserIfNotExists() and
CatalogOpExecutor.getCurrentEventId().
Testing:
Increase TRIGGER_RESET_METADATA_DELAY from 1s to 3s in
test_metadata_after_failover_with_delayed_reset. It was easy to hit the
deadlock with 3s delay before the patch. No more deadlock happen after
the patch.
Run and pass test_catalogd_ha.py and test_restart_services.py
exhaustively.
Change-Id: I3162472ea9531add77886bf1d0d73460ff34d07a
Reviewed-on: http://gerrit.cloudera.org:8080/23382
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Riza Suminto <riza.suminto@cloudera.com>
This patch modifies the string overload of
IcebergFunctions::TruncatePartitionTransform so that it always handles
strings as UTF-8-encoded ones, because the Iceberg specification states
that that strings are UTF-8 encoded.
Also, for an Iceberg table UrlEncode is called in not the
Hive-compatible way, rather than the standard way, similar to Java's
URLEncoder.encode() (which the Iceberg API also uses) to conform with
existing practices by Hive, Spark and Trino. This included a change in
the set of characters which are not escaped to follow the URL Standard's
application/x-www-form-urlencoded format. [1] Also renamed it from
ShouldNotEscape to IsUrlSafe for better readability.
Testing:
* add and extend e2e tests to check partitions with Unicode characters
* add be tests to coding-util-test.cc
[1]: https://url.spec.whatwg.org/#application-x-www-form-urlencoded-percent-encode-set
Change-Id: Iabb39727f6dd49b76c918bcd6b3ec62532555755
Reviewed-on: http://gerrit.cloudera.org:8080/23190
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch adds Keycloak as Identity Provider for Lakekeeper, so
now we can test Impala's Iceberg REST Catalog with an OAuth2
authentication (Client-Credential) flow. The Keycloak
instance is pre-configured with a Lakekeeper realm that contain
the necessary clients, users, scopes and roles.
Manual testing also revealed that our Iceberg REST Catalog
configuration is incomplete. This patch refactors config
handling in a way that both Iceberg native configuration
options and Trino-specific configuration options can be
used with Impala. This will help users use their Trino
connectors with Impala.
By default Impala uses Iceberg 1.3 which assumes that the
Iceberg REST server is also the authentication server. It is
not always true, e.g. Lakekeeper cannot even function as the
authententication server, but it can work with external authentication
servers. Btw, this is why we needed Keycloak in the first place.
It means if someone wants to try out Lakekeeper+Impala with Oauth2,
they need to configure Impala with Iceberg 1.5.
Testing
* manual testing with Iceberg 1.5
Change-Id: Ie5785cb72773e188b1de7c7924cc6f0b1f96de33
(cherry picked from commit a9cb94986a5791be2adcb2f7c576272a9c22e79c)
Reviewed-on: http://gerrit.cloudera.org:8080/23156
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This fix is needed before the join optimization fix can be committed.
The JoinTranspose rule provided by Calcite was having 2 issues:
1) For tpcds-q10 and q35, an exception was being thrown. There is
a bug in the Calcite code when the Join-Project gets matched but
the Join is of reltype SemiJoin. In this case, the Projects do not
get created correctly and the exception gets thrown.
2) We only want to transpose a Project above a Join if there is an
underlying Join underneath the Project. The whole purpose is to
be able to create adjacent Join RelNodes. We do not have to transpose
the Project when it is not sandwiched between two Join nodes. It is
preferable to keep it underneath the Join since the row width
calculation would be affected (the Project may reduce the number of
columns, thus reducing the row width).
This commit extends the given JoinProjectTranspose rule by Calcite
and handles these added restrictions.
Change-Id: I7f62ec030fc8fbe36e6150bf96c9673c44b7da1b
Reviewed-on: http://gerrit.cloudera.org:8080/23313
Reviewed-by: Aman Sinha <amsinha@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch implements an additional check when collecting table names
that are used in the given query. Specifically, for a table name that
is not fully qualified, we make sure the derived fully qualified table
name is not a common table expression (CTE) of a SqlWithItem in a
WITH clause since such CTE's are not actual tables.
Testing:
- Added a test in test_ranger.py to verify the issue is fixed.
Change-Id: I3f51af42d64cdcff3c26ad5a96c7f53ebef431b3
Reviewed-on: http://gerrit.cloudera.org:8080/23209
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Fang-Yu Rao <fangyu.rao@cloudera.com>
This commit fixes a crash in the sha2() function that occurs when
Impala is run on a FIPS enabled OS, particularly CentOS 7. Running
sha2() with 384 or 512-bit lengths would cause the impalad
to crash with an OpenSSL assertion failure:
"Low level API call to digest SHA384 forbidden in FIPS mode!"
The root cause was the direct use of low-level OpenSSL API calls
like SHA384(), SHA512(). OpenSSL 1.0 (used in RHEL/CentOS 7) is
particularly strict and forbids these calls in FIPS mode, causing
the module to terminate the process.
This patch changes to use the high-level, FIPS compliant EVP_Digest
API to perform the hash in sha2() function implementation.
Tests:
Ran sha2() in FIPS enabled CentOs 7 after the change and succeeded.
Passed exhaustive tests.
Change-Id: I694532350285534fd935c92b7a78bed91ded3cb5
Reviewed-on: http://gerrit.cloudera.org:8080/23373
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
The GetQueryStatus RPC could crash due to a use-after-free error
when accessing the reqest. When a query was rejected, the
function would call RespondAndReleaseRpc(), which can free the "req"
object, and then attempt to access req->query_id() for logging and
to delete the entry from the admission_state_map_.
This commit fixes the crash by moving the call to
RespondAndReleaseRpc() to the end of the function. This change
aligns this function with others like
AdmissionControlService::ReleaseQuery(), which also deletes from
admission_state_map_ before RespondAndReleaseRpc(), it ensures
that all logic is completed before the RPC resources are released.
Tests:
Reproduced the issue by running 100 times
TestAdmissionControllerStressWithACService::test_mem_limit, and
after the change, it can successfully run 100 times.
Passed exhaustive tests.
Change-Id: I688954c5c671671cc2dc669ecfdf2405476302d7
Reviewed-on: http://gerrit.cloudera.org:8080/23379
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Tested-by: Michael Smith <michael.smith@cloudera.com>
When creating or altering partitions of Iceberg tables, Impala only
accepts column names if they are in lowercase and throws
ImpalaRuntimeException otherwise.
This patch allows the usage of other cases as well in PARTITION SPEC
clauses. IcebergPartitionField converts field names to lower case in its
constructor, similar to ColumnDef and PartitionKeyValue.
Testing:
* ran existing tests
* add new test with mixed letter case columns
Change-Id: I4080a6b7468fff940435e2e780322d4ba1f0de49
Reviewed-on: http://gerrit.cloudera.org:8080/23334
Reviewed-by: Daniel Becker <daniel.becker@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Runs mvn clean on all Java subprojects - instead of just ext-data-source
- to avoid build failures when files from other versions of the code and
dependencies are left behind.
Change-Id: I8cf540f90adbff327de98f900059bfa3bbc8ef22
Reviewed-on: http://gerrit.cloudera.org:8080/23374
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Timestamp string can have a timezone offset at its end, e.g.
"2025-08-31 06:23:24.9392129 +08:00" has "+08:00" as the timezone
offset. When casting strings to DATE type, we try to find the default
format by matching the separators, i.e. '-', ':', ' ', etc in
SimpleDateFormatTokenizer::GetDefaultFormatContext(). The one that
matches this example is DEFAULT_DATE_TIME_CTX[] which represents the
default date/time context for "yyyy-MM-dd HH:mm:ss.SSSSSSSSS". The
fractional part at the end can have length from 0 to 9, matching
DEFAULT_DATE_TIME_CTX[0] to DEFAULT_DATE_TIME_CTX[9] respectively.
When calculating which item in DEFAULT_DATE_TIME_CTX is the matched
format, we use the index as str_len - 20 where 20 is the length of
"yyyy-MM-dd HH:mm:ss.". This causes the index overflow if the string
length is larger than 29. A wild pointer is returned from
GetDefaultFormatContext(), leading crash in following codes.
This patch fixes the issue by adding a check to make sure the string
length is smaller than the max length of the default date time format,
i.e. DEFAULT_DATE_TIME_FMT_LEN (29). Longer strings will use
DateTimeFormatContext created lazily.
Note that this just fixes the crash. Converting timestamp strings with
timezone offset at the end to DATE type is not supported yet and will be
followed up in IMPALA-14391.
Tests
- Added e2e tests on constant expressions. Also added a test table with
such timestamp strings and added test on it.
Change-Id: I36d73f4a71432588732b2284ac66552f75628a62
Reviewed-on: http://gerrit.cloudera.org:8080/23371
Reviewed-by: Daniel Becker <daniel.becker@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
