jprdonnelly/impala
1
0
Fork 0
mirror of https://github.com/apache/impala.git synced 2025-12-19 18:12:08 -05:00
Code Issues Projects Releases Wiki Activity

IMPALA-14227: In HA failover, passive catalogd should apply pending HMS events before being active

Browse Source 
After IMPALA-14074, the passive catalogd can have a warmed up metadata
cache during failover (with catalogd_ha_reset_metadata_on_failover=false
and a non-empty warmup_tables_config_file). However, it could still use
a stale metadata cache when some pending HMS events generated by the
previous active catalogd are not applied yet.

This patch adds a wait during HA failover to ensure HMS events before
the failover happens are all applied on the new active catalogd. The
timeout is configured by a new flag which defaults to 300 (5 minutes):
catalogd_ha_failover_catchup_timeout_s. When timeout happens, by default
catalogd will fallback to resetting all metadata. Users can decide
whether to reset or continue using the current cache. This is configured
by another flag, catalogd_ha_reset_metadata_on_failover_catchup_timeout.

Since the passive catalogd depends on HMS event processing to keep its
metadata up-to-date with the active catalogd, this patch adds validation
to avoid starting catalogd with catalogd_ha_reset_metadata_on_failover
set to false and hms_event_polling_interval_s <= 0.

This patch also makes catalogd_ha_reset_metadata_on_failover a
non-hidden flag so it's shown in the /varz web page.

Tests:
 - Ran test_warmed_up_metadata_after_failover 200 times. Without the
   fix, it usually fails in several runs.
 - Added new tests for the new flags.

Change-Id: Icf4fcb0e27c14197f79625749949b47c033a5f31
Reviewed-on: http://gerrit.cloudera.org:8080/23174
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This commit is contained in:
stiga-huang
2025-07-15 16:21:09 +08:00
committed by Impala Public Jenkins
parent 438461db9e
commit 64abca481f
13 changed files with 214 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
common/thrift/JniCatalog.thrift
View File

@@ -1083,6 +1083,13 @@ struct TCopyTestCaseReq {
1: required string input_path
}
struct TEventProcessorMetricsSummaryRequest {
// Whether to fetch the latest HMS event id using a HMS RPC or using the cached value
// in the EventProcessor. When set to true, 'latest_event_time_s' in the progress info
// will be -1 to save a HMS RPC.
1: required bool get_latest_event_from_hms = false
}
struct TEventBatchProgressInfo {
// Number of original HMS events received in the current batch.
1: required i32 num_hms_events