mirror of
https://github.com/apache/impala.git
synced 2025-12-20 02:20:11 -05:00
acc3de40fb
When incremental metadata updates are enabled (by default), catalogd
sends incremental partition updates based on the last sent table
snapshot. Coordinators will apply these partition updates on their
existing table snapshots.
Each partition update is via a partition instance. Partition instances
are identified by partition ids. Each partition instance is a snapshot
of the metadata of a partition. When applying incremental partition
updates, ImpaladCatalog#addTable() has a Precondition check assuming
that new partition updates should not be duplicated with existing
partition ids.
The motivation of this check is to detect whether catalogd is sending
duplicate partition updates. However, it could be hitted when the
coordinator has a newer version of the table than the last sent table
snapshot in catalogd. This happens when two coordinators both execute
DMLs on the same table (e.g. insert into different partitions), and the
DMLs finish within a catalog topic update time window. Note that
coordinator will receive a table snapshot from catalogd as a response of
the DML request. So one of the coordinator will have a table version
that is lower than the latest version in catalogd but larger than the
last sent table version in catalogd. For an example, let's see the
following sequence of events on a table:
t0: coord1 and coord2 both have the latest version as catalogd
t1: coord1 executes a DML to add a partition p2
t2: coord2 executes a DML to add another partition p3
t3: catalogd sends topic update with {p2, p3}
t1 and t2 happen inside a topic-update window. So catalogd will send the
update of {p2, p3}. The following table shows the table version and
corresponding partition instances in each server.
+----+---------------+--------------+---------------+
| | catalogd | coordinator1 | coordinator2 |
+----+---------------+--------------+---------------+
| t0 | v0:{p1} | v0:{p1} | v0:{p1} |
+----+---------------+--------------+---------------+
| t1 | v1:{p1,p2} | v1:{p1,p2} | v0:{p1} |
+----+---------------+--------------+---------------+
| t2 | v2:{p1,p2,p3} | v1:{p1,p2} | v2:{p1,p2,p3} |
+----+---------------+--------------+---------------+
At t3, coordinator2 will skip the table update since it already has a
version equal to the one in the topic update. However, on coordinator1,
the table version is smaller than v2, so it will apply the incremental
updates of {p2,p3} and then hit the Precondition check complaining that
p2 already exists.
It's legal that a coordinator has got some partition instances in the
DML responses. So we can't assume that all partition updates in a topic
update don't exist in the coordinator. This patch removes this
Precondition check to accept this case.
Tests:
- Add a test to reproduce the scenario mentioned above. It fails
without this patch.
Change-Id: I1657684f8853b76b1524475a3b3c35fa22a0e36e
Reviewed-on: http://gerrit.cloudera.org:8080/16649
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
64 lines
2.8 KiB
Python
Executable File
64 lines
2.8 KiB
Python
Executable File
# Licensed to the Apache Software Foundation (ASF) under one
|
|
# or more contributor license agreements. See the NOTICE file
|
|
# distributed with this work for additional information
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
# to you under the Apache License, Version 2.0 (the
|
|
# "License"); you may not use this file except in compliance
|
|
# with the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing,
|
|
# software distributed under the License is distributed on an
|
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
# KIND, either express or implied. See the License for the
|
|
# specific language governing permissions and limitations
|
|
# under the License.
|
|
|
|
import pytest
|
|
from tests.common.custom_cluster_test_suite import CustomClusterTestSuite
|
|
|
|
|
|
class TestIncrementalMetadataUpdate(CustomClusterTestSuite):
|
|
""" Validates incremental metadata updates across catalogd and coordinators."""
|
|
|
|
@classmethod
|
|
def get_workload(cls):
|
|
return 'functional-query'
|
|
|
|
@classmethod
|
|
def setup_class(cls):
|
|
super(TestIncrementalMetadataUpdate, cls).setup_class()
|
|
|
|
@pytest.mark.execute_serially
|
|
@CustomClusterTestSuite.with_args(
|
|
statestored_args="--statestore_update_frequency_ms=5000")
|
|
def test_long_statestore_update_frequency(self, unique_database):
|
|
"""Test multiple inserts on the same table executed by different impalads. Increase
|
|
statestore_update_frequency_ms so inserts can finished in the same time window of
|
|
a catalog topic update. Different impalads will have different versions of the table
|
|
returned by DML responses. They should be able to correctly apply the incremental
|
|
partition updates from catalogd."""
|
|
self.execute_query(
|
|
"create table %s.multi_inserts_tbl (id int) partitioned by (p int)"
|
|
% unique_database)
|
|
# Refresh the table with sync_ddl=true so table metadata is loaded in all impalads.
|
|
self.execute_query("refresh %s.multi_inserts_tbl" % unique_database,
|
|
query_options={"sync_ddl": True})
|
|
# Run 3 inserts using different impalads.
|
|
for i in range(3):
|
|
impalad_client = self.create_client_for_nth_impalad(i)
|
|
# Set SYNC_DDL for the last query, so metadata should in sync along all impalads
|
|
# at the end.
|
|
impalad_client.set_configuration_option("sync_ddl", i == 2)
|
|
impalad_client.execute(
|
|
"insert into {0}.multi_inserts_tbl partition(p) values ({1}, {1})"
|
|
.format(unique_database, i))
|
|
|
|
# Verify each impalad has the latest table metadata.
|
|
for i in range(3):
|
|
impalad_client = self.create_client_for_nth_impalad(i)
|
|
res = impalad_client.execute(
|
|
"show partitions %s.multi_inserts_tbl" % unique_database)
|
|
assert len(res.data) == 4, "missing partitions"
|