jprdonnelly/impala
1
0
Fork 0
mirror of https://github.com/apache/impala.git synced 2025-12-19 18:12:08 -05:00
Code Issues Projects Releases Wiki Activity

IMPALA-13788: [DOCS] Docs for query options SYNC_HMS_EVENTS_WAIT_TIME_S

Browse Source 
and SYNC_HMS_EVENTS_STRICT_MODE

The commit documents query options SYNC_HMS_EVENTS_WAIT_TIME_S
and SYNC_HMS_EVENTS_STRICT_MODE

Url: https://impala.apache.org/docs/build/html/topics/impala_set.html

Change-Id: Ia11663c5e84794d4bca658124cde59bf97aa7158
Reviewed-on: http://gerrit.cloudera.org:8080/23592
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Quanlong Huang <huangquanlong@gmail.com>
Reviewed-by: Jason Fehr <jfehr@cloudera.com>
This commit is contained in:
m-sanjana19
2025-10-24 15:56:24 +05:30
committed by Quanlong Huang
parent 54c0074b33
commit 134c28d445
3 changed files with 154 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/impala.ditamap
View File

@@ -265,6 +265,8 @@ under the License.
<topicref href="topics/impala_spool_query_results.xml"/>
<topicref href="topics/impala_support_start_over.xml"/>
<topicref href="topics/impala_sync_ddl.xml"/>
<topicref href="topics/impala_sync_hms_events_strict_mode.xml"/>
<topicref href="topics/impala_sync_hms_events_wait_time_s.xml"/>
<topicref href="topics/impala_thread_reservation_aggregate_limit.xml"/>
<topicref href="topics/impala_thread_reservation_limit.xml"/>
<topicref href="topics/impala_timezone.xml"/>

73
docs/topics/impala_sync_hms_events_strict_mode.xml Normal file
View File

@@ -0,0 +1,73 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
<concept rev="5.0.0" id="impala_sync_hms_events_strict_mode">
<title>SYNC_HMS_EVENTS_STRICT_MODE Query Option</title>
<titlealts audience="PDF"><navtitle>SYNC HMS EVENTS STRICT MODE</navtitle></titlealts>
<prolog>
<metadata>
<data name="Category" value="Impala"/>
<data name="Category" value="Impala Query Options"/>
<data name="Category" value="DDL"/>
<data name="Category" value="SQL"/>
<data name="Category" value="Developers"/>
<data name="Category" value="Data Analysts"/>
</metadata>
</prolog>
<conbody>
<p id="IMPALA-12152"><indexterm audience="hidden">SYNC_HMS_EVENTS_STRICT_MODE query
option</indexterm>This query option controls the behavior of the query coordinator if it
cannot successfully sync with the latest HMS events (e.g., if the waiting time set by
<codeph>SYNC_HMS_EVENTS_WAIT_TIME_S</codeph> is reached, or the event processor is in an
error state).</p>
<p conref="../shared/impala_common.xml#common/type_boolean"/>
<p conref="../shared/impala_common.xml#common/default_false_0"/>
<p><b>Query Range:</b>
<codeph>True</codeph> or <codeph>False</codeph></p>
<p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
<p>This option determines whether Impala favors consistency (failing the query) or availability
(starting the query planning despite potential lag).<ul id="ul_jpg_3qs_bhc">
<li><codeph>True</codeph> (Strict Mode): If Catalog service cannot sync the metadata with
the latest HMS event ID before the timeout is reached, the coordinator fails the query
with an error. This prioritizes correctness and metadata consistency.<p>If the coordinator
fails the query when <codeph>SYNC_HMS_EVENTS_STRICT_MODE</codeph> is set to
<codeph>TRUE</codeph> (Strict Mode), you will see an error message starting with
<msgph>Failed to sync events from Metastore:</msgph> and then the specific reason,
such as <msgph>HMS event processing is disabled</msgph>.</p></li>
<li><codeph>False</codeph> (Non-Strict Mode - Default): If the waiting times out, the
coordinator starts planning the query immediately but issues a warning message in the
query profile. This prioritizes availability, allowing the query to run, but risks working
with slightly stale metadata.</li>
</ul></p>
<note id="note_fhz_rqs_bhc">This option is only relevant if
<codeph>SYNC_HMS_EVENTS_WAIT_TIME_S</codeph> is set to a value greater than 0.</note>
<p><b>Example</b></p>
<p>The following example shows how to set <codeph>SYNC_HMS_EVENTS_STRICT_MODE</codeph> to
<codeph>TRUE</codeph> for a specific query to enforce
consistency:<codeblock id="codeblock_fnm_hsp_fhc">SET SYNC_HMS_EVENTS_STRICT_MODE=TRUE;
SELECT COUNT(*) FROM functional.alltypes;</codeblock></p>
<p><b>Added in:</b>
<keyword keyref="impala50_full"/></p>
<p conref="../shared/impala_common.xml#common/related_info"/>
<p>
<xref href="impala_ddl.xml#ddl"/>
</p>
</conbody>
</concept>

79
docs/topics/impala_sync_hms_events_wait_time_s.xml Normal file
View File

@@ -0,0 +1,79 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
<concept rev="5.0.0" id="impala_sync_hms_events_wait_time_s">
<title>SYNC_HMS_EVENTS_WAIT_TIME_S Query Option</title>
<titlealts audience="PDF"><navtitle>SYNC HMS EVENTS WAIT TIME S</navtitle></titlealts>
<prolog>
<metadata>
<data name="Category" value="Impala"/>
<data name="Category" value="Impala Query Options"/>
<data name="Category" value="DDL"/>
<data name="Category" value="SQL"/>
<data name="Category" value="Developers"/>
<data name="Category" value="Data Analysts"/>
</metadata>
</prolog>
<conbody>
<p id="IMPALA-12152"><indexterm audience="hidden">SYNC_HMS_EVENTS_WAIT_TIME_S query option</indexterm>This query
option controls the maximum time Impala will wait for the Catalog Service to sync with the
latest events from the Hive Metastore (HMS) before starting query planning.</p>
<p conref="../shared/impala_common.xml#common/type_integer"/>
<p conref="../shared/impala_common.xml#common/default_false_0"/>
<p><b>Query Range:</b> > = 0</p>
<p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
<p>Setting this option to a positive value (in seconds) enables a new mechanism where only the
planning thread of that query will pause and wait for the Catalog service's event processor to
fully apply any outstanding metadata changes from the HMS before the query is analyzed and
planned.<note id="note_sxq_srs_bhc">This mechanism only guarantees that HMS modifications
that occurred before query planning started will be synced by the Catalog service. Any HMS
modifications that happen after the query begins its planning phase are not guaranteed to be
applied.</note><ul id="ul_bh2_vrs_bhc">
<li>This is typically used after an external process (like Hive or Spark) has modified a
dependent table, ensuring Impala's query sees the most current metadata, such as newly
added partitions.</li>
<li>The default value of 0 disables this waiting mechanism.</li>
<li>The wait time could be set based on the maximum observed event processing lag in your
cluster (visible via the Catalogd WebUI /events page).</li>
</ul></p>
<p><b>Example:</b></p>
<p>When a Hive table is updated by an <codeph>INSERT</codeph> operation on dynamic partitions,
you can use the <codeph>sync_hms_events_wait_time_s</codeph> query option in Impala to ensure
the metadata is synchronized before you query the table.</p>
<p>Impala waits up to the specified time (<codeph>sync_hms_events_wait_time_s</codeph>) for Hive
Metastore Service (HMS) events to synchronize automatically.</p>
<p>The following example demonstrates setting the option to safely query a Hive table immediately after it is updated. In this case, Impala waits up to 300 seconds for the synchronization to complete.</p>
<codeblock>hive> insert into tbl partition(p) select * from tbl2;
impala> set sync_hms_events_wait_time_s=300;
impala> select * from tbl;</codeblock>
<p>
<note id="note_q4b_dqz_ghc" type="note">You do not need to run a <codeph>REFRESH</codeph>
command or wait explicitly on the client side.</note>
</p>
<p>
<b>Added in:</b>
<keyword keyref="impala50_full"/>
</p>
<p conref="../shared/impala_common.xml#common/related_info"/>
<p>
<xref href="impala_ddl.xml#ddl"/>
</p>
</conbody>
</concept>