mirror of
https://github.com/apache/impala.git
synced 2025-12-19 18:12:08 -05:00
6f3deabb9d
In global INVALIDATE METADATA (catalog reset), catalogd creates IncompleteTable for all the known table names. However, the createEventId is uninitialized so remain as -1. Tables could be dropped unintentionally by stale DropTable or AlterTableRename events. Ideally when catalogd creates an IncompleteTable during reset(), it should fetch the latest event on that table and use its event id as the createEventId. However, fetching such event ids for all tables is impractical to finish in a reasonable time. It also adds a significant load on HMS. As a compromise, this patch uses the current event id when the reset() operation starts, and sets it to all IncompleteTable objects created in this reset operation. This is enough to handle self CreateTable / DropTable / AlterTableRename events since such self-events generated before that id will be skipped. Such self-events generated after that id are triggered by concurrent DDLs which will wait until the corresponding table list is updated in reset(). The DDL will also update createEventId to skip stale DropTable / AlterTableRename events. Concurrent CreateTable DDLs could set a stale createEventId if their HMS operation finish before reset() and their catalog operations finish after reset() creates the table. To address this, we add a check in setCreateEventId() to skip stale event ids. The current event id of reset() is also used in DeleteEventLog to track tables removed by this operation. Refactored IncompleteTable.createUninitializedTable() to force passing a createEventId as a parameter. To ease debugging, adds logs when a table is added/removed in HMS events processing. Also adds logs when the catalog version of a table changes and adds logs when start processing a rename event. This patch also refactors CatalogOpExecutor.alterTableOrViewRename() by extracting some codes into methods. A race issue is identified and fixed that DeleteEventLog should be updated before renameTable() updates the catalog cache so the removed old table won't be added back by concurrently processing of a stale CREATE_TABLE event. _run_ddls_with_invalidation in test_concurrent_ddls.py could still fail with timeout when running with sync_ddl=true. The reason is when the DDL hits IMPALA-9135 and hangs, it needs catalogd to send new catalog updates to reach the max waiting attempts (see waitForSyncDdlVersion()). However, if all other concurrent threads already finish, there won't be any new catalog updates so the DDL will wait forever and finally result in the test timed out. To workaround this, this patch adds another concurrent thread that keeps creating new tables until the test finish. Tests: - Ran the following tests in test_concurrent_ddls.py 10 rounds. Each round takes 11 mins. - test_ddls_with_invalidate_metadata - test_ddls_with_invalidate_metadata_sync_ddl - test_mixed_catalog_ddls_with_invalidate_metadata - test_mixed_catalog_ddls_with_invalidate_metadata_sync_ddl - test_local_catalog_ddls_with_invalidate_metadata - test_local_catalog_ddls_with_invalidate_metadata_sync_ddl - test_local_catalog_ddls_with_invalidate_metadata_unlock_gap Change-Id: I6506821dedf7701cdfa58d14cae5760ee178c4ec Reviewed-on: http://gerrit.cloudera.org:8080/23346 Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
11 KiB
11 KiB