Recently, we see many timeout failures of test_concurrent_ddls.py in S3
builds, e.g. IMPALA-10280, IMPALA-10301, IMPALA-10363. It'd be helpful
to dump the server stacktraces so we can understand why some RPCs are
slow/stuck.
This patch extracts the logic of dumping stacktraces in
script-timeout-check.sh to a separate script, dump-stacktraces.sh.
The script also dumps jstacks of HMS and NameNode. Dumping all these
stacktraces is time-consuming so we do them in parallel, which also
helps to get consistent snapshots of all servers.
When any tests in test_concurrent_ddls.py timeout, we use
dump-stacktraces.sh to dump the stacktraces before exit. Previously,
some tests depend on pytest.mark.timeout for detecting timeouts. It's
hard to add a customized callback for dumping server stacktraces. So
this patch refactors test_concurrent_ddls.py to only use timeout of
multiprocessing.
Tests:
- Tested the scripts locally.
- Verified the error handling of timeout logics in Jenkins jobs
Change-Id: I514cf2d0ff842805c0abf7211f2a395151174173
Reviewed-on: http://gerrit.cloudera.org:8080/16800
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
test_concurrent_invalidate_metadata runs 20 iterations for concurrent
invalidate metadata commands. Each iteration could take more than 6s.
So it's easy to hit the current timeout limit, 120s. The main purpose of
this test is to detect metadata bugs that could cause invalidate
metadata hanging. It's not for performance. So this patch increases the
timeout limit to 300s to fix the flakiness.
Change-Id: I47e9f4793117b9a726fde165adea68ce31f539a8
Reviewed-on: http://gerrit.cloudera.org:8080/16333
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
The race condition is exposed intermittently, on certain builds which
causes test_event_processing::test_self_events test to fail. This
happens because we are checking for self-event identifiers in the Db
object without taking a lock. When a DDL like 'comment on
database is 'test'' is executed, it is possible that the event
processor thread is triggered as soon as the ALTER_DATABASE event is
generated. This may cause event processor fail the self-event detection
since the self-event identifiers are not yet added to the Db object.
The fix adds a Db lock similar to Table lock. Alter db operations
in CatalogOpExecutor now take db locks instead of metastoreDdlLock_
which makes it consistent with table locking protocol.
Testing:
1. Ran existing tests for events processor.
2. This test was failing on centos6 frequently (failed in 1/3 times).
After the fix I ran the test in a loop for 24 hrs (197 iterations) and
the test didn't fail.
3. Ran core tests with CDP and CDH builds.
Change-Id: I472fd8a55740769ee5cdb84e48422a4ab39a8d1e
Reviewed-on: http://gerrit.cloudera.org:8080/15260
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
The test uses a thread pool of 3 slots to run 3 concurrent CreateTable
statements. Each thread creates a client and closes it at the end of the
task(run_create_table_if_not_exists). The first 2 thread tasks start at
the same time. Then after one second, the 3rd thread task starts.
Ideally the 3 tasks should be executed in 3 different threads of the
thread pool. However, if any of the first 2 tasks finish in 1s, the 3rd
task could be executed by reusing the thread. Since the client in the
thread is closed, the task fails.
This patch removes the close operation at the end of the task.
Connection of the thread can be reused and will be closed when the
thread terminates.
Also fix the same problem in test_concurrent_ddls.py.
Tests:
- Run test_concurrent_ddls.py and test_concurrent_kudu_create.py. In
the logs of impalad, observe that connections are being reused and
closed at the end of the tests.
Change-Id: Idefba98ecd65efbd47b1618291330795ef13b910
Reviewed-on: http://gerrit.cloudera.org:8080/15151
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
The minimal catalog object version of valid catalog objects is used to
implement global invalidate metadata in legacy catalog mode. Coordinator
sends DDL RPC to catalogd for global invalidate metadata and gets the
expected min catalog version in the response. It's the version when
catalogd starts to reset the entire catalog, which means when the reset
is done, all valid catalog objects should be associated with a catalog
version larger than it. Coordinator will wait until its min catalog
version exceeds this value, which means it has processed all the updates
of the reset propagated from the catalogd via statestored. If SYNC_DDL
is set, the coordinator will also wait until other coordinators reach
the same statestore topic version with it, so they have also processed
the same updates and had the latest catalog after reset.
In local catalog mode, the coordinator does not cache all the metadata.
Instead, it caches them on-demand (based on query requests), and removes
them based on the Guava cache configurations (size or TTL) or explicit
invalidation from the catalog topic updates. So it's hard to track the
minimal catalog object version correctly.
This patch adds a new field (lastResetCatalogVersion) in TCatalog to
propagate the catalog version when catalogd starts to reset the entire
metadata. Each time when catalogd generates a new topic update, it will
generate a TCatalogObject of CATALOG type containing the state of the
catalog which includes this new field.
When coordinator receives a new value of lastResetCatalogVersion in a
topic update, it means catalogd has reset the entire catalog.
Coordinator will then clear its cache to remove all stale catalog
objects. It's possible that some fresh items being removed too. They
will be refetched on demand.
After the invalidation, there are no catalog object cached with catalog
version <= lastResetCatalogVersion. Because stale cache has been cleared
and all metadata from catalogd is newer than lastResetCatalogVersion. So
lastResetCatalogVersion + 1 is the lower bound (included) of min catalog
object version of a coordinator.
This patch also exposes the lower bound of catalog object version of via
a new metric "catalog.catalog-object-version-lower-bound" to ease
debugging.
IMPALA-9136 is also fixed in this patch.
Tests:
- Recover all existing tests that have been disabled due to this
missing feature
- Add custom cluster test for concurrent DDLs with INVALIDATE METADATA
- Run CORE tests
Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549
Reviewed-on: http://gerrit.cloudera.org:8080/14307
Reviewed-by: Vihang Karajgaonkar <vihang@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
