jprdonnelly/impala
1
0
Fork 0
mirror of https://github.com/apache/impala.git synced 2025-12-19 18:12:08 -05:00
Code Issues Projects Releases Wiki Activity

IMPALA-14617: Deflake TestMetadataReplicas.test_catalog_restart

Browse Source 
TestMetadataReplicas.test_catalog_restart creates a db and an underlying
table in Hive, then expects an INVALIDATE command can bring up the table
in impalads. The test runs in the legacy catalogd mode that has the bug
of IMPALA-12103. So if the INVALIDATE command runs in a state that
catalgod has the db in cache but the db doesn't show up in impalad's
cache yet, catalogd will just return the table and impalad will skip it
due to db not exists. Then the above assertion fails.

The db is added in catalogd cache by processing the CREATE_DATABASE HMS
event, which is asynchronous with executing the INVALIDATE command. If
the command is triggered before that, the test passes. If the command is
triggered after that, the test fails.

When the test was written, we don't have HMS event processing yet. It's
expected that the db is also added in catalogd by the INVALIDATE
command. To deflake the issue, this patch disables HMS event processing
in this test, so catalogd always has a consistent cache with impalad
when executing the INVALIDATE command.

This patch also changes the log level of a log in ImpaladCatalog to warn
if a table is not added due to its db is missing.

Tests:
 - Ran the test locally 10 times.

Change-Id: I2d17404cc8093eacf9b51df3d22caf5cbb6a61a9
Reviewed-on: http://gerrit.cloudera.org:8080/23798
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This commit is contained in:
stiga-huang
2025-12-17 17:02:44 +08:00
committed by Impala Public Jenkins
parent e22c0d33d6
commit 3725b4ea63
2 changed files with 7 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
View File

@@ -473,10 +473,8 @@ public class ImpaladCatalog extends Catalog implements FeCatalog {
long catalogVersion, long lastLoadedTime) throws TableLoadingException { long catalogVersion, long lastLoadedTime) throws TableLoadingException {
Db db = getDb(thriftTable.db_name); Db db = getDb(thriftTable.db_name);
if (db == null) { if (db == null) {
if (LOG.isTraceEnabled()) { LOG.warn("Not adding table since the database does not exist: " +
LOG.trace("Parent database of table does not exist: " +
thriftTable.db_name + "." + thriftTable.tbl_name); thriftTable.db_name + "." + thriftTable.tbl_name);
}
return; return;
} }

7
tests/custom_cluster/test_metadata_replicas.py
View File

@@ -26,11 +26,14 @@ from tests.util.hive_utils import HiveDbWrapper
@SkipIfFS.hive @SkipIfFS.hive
@CustomClusterTestSuite.with_args( @CustomClusterTestSuite.with_args(
impalad_args="--use_local_catalog=false", impalad_args="--use_local_catalog=false",
catalogd_args="--catalog_topic_mode=full", catalogd_args="--catalog_topic_mode=full --hms_event_polling_interval_s=0",
statestored_args="--statestore_update_frequency_ms=1000") statestored_args="--statestore_update_frequency_ms=1000")
class TestMetadataReplicas(CustomClusterTestSuite): class TestMetadataReplicas(CustomClusterTestSuite):
""" Validates metadata content across catalogd and impalad coordinators. """ Validates metadata content across catalogd and impalad coordinators.
This test is only valid in legacy catalog mode. """ This test is only valid in legacy catalog mode. Disables HMS event processing to avoid
catalogd cache being modified unintentionally by HMS events. The test expects the
INVALIDATE command to bring up the latest metadata from HMS. This avoids hitting the bug
of IMPALA-12103. """
@classmethod @classmethod
def setup_class(cls): def setup_class(cls):