TestLocalCatalogRetries.test_replan_limit runs REFRESH and SELECT
queries concurrently on a table, and expects one of the query hits
inconsistent metadata.
This patch increases the chance of inconsistent metadata by injecting
a latency (500ms) before each catalog fetch. So it's more likely that a
request is fetching stale metadata. Also bump up the timeout of
thread.join() so we can try out all the attempts.
Test
- Run test_replan_limit 1000 times without any error.
- Run all tests of TestLocalCatalogRetries 100 times without any error.
Change-Id: Ia5bdca7402039f1f24b7bf19595c2541fa32d0ad
Reviewed-on: http://gerrit.cloudera.org:8080/18537
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch adds support for COS(Cloud Object Storage). Using the
hadoop-cos, the implementation is similar to other remote FileSystems.
New flags for COS:
- num_cos_io_threads: Number of COS I/O threads. Defaults to be 16.
Follow-up:
- Support for caching COS file handles will be addressed in
IMPALA-10772.
- test_concurrent_inserts and test_failing_inserts in
test_acid_stress.py are skipped due to slow file listing on
COS (IMPALA-10773).
Tests:
- Upload hdfs test data to a COS bucket. Modify all locations in HMS
DB to point to the COS bucket. Remove some hdfs caching params.
Run CORE tests.
Change-Id: Idce135a7591d1b4c74425e365525be3086a39821
Reviewed-on: http://gerrit.cloudera.org:8080/17503
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Coordinator calls updateCatalog RPC to catalogd to finalize the INSERT
statement. Catalogd will create any new partitions and reload metadata
of the updated partitions (including new partitions).
After IMPALA-10502, each new partition has a createEventId which is used
to detect how "fresh" it is. If the event processor receives a
DROP_PARTITION event, it will compare its event id with the
createEventId of the partition. Only DROP_PARTITION events happen after
the createEventId will be evaluated.
There is a bug in CatalogOpExecutor#updateCatalog() that we loss the
createEventId of the new partitions. It should be used in the final call
of loadTableMetadata(). This bug causes intermittent failures in
TestReusePartitionMetadata.test_reuse_partition_meta. The last two DMLs
of the test is dropping a partition and then creating it back by an
INSERT. If the DROP_PARTITION event is processed after the INSERT
finishes, the partition will be dropped incorrectly.
Tests
- Ran the test 100 times locally. Without the fix, it fails in 10 runs.
Change-Id: I2622c28a5ce6084fc77f6ea475d2633445c7f8dd
Reviewed-on: http://gerrit.cloudera.org:8080/18066
Reviewed-by: Vihang Karajgaonkar <vihang@cloudera.com>
Tested-by: Vihang Karajgaonkar <vihang@cloudera.com>
This commit turns on events processing by default. The default
polling interval is set as 1 second which can be overrriden by
setting hms_event_polling_interval_s to non-default value.
When the event polling turned on by default this patch also
moves the test_event_processing.py to tests/metadata instead
of custom cluster test. Some tests within test_event_processing.py
which needed non-default configurations were moved to
tests/custom_cluster/test_events_custom_configs.py.
Additionally, some other tests were modified to take into account
the automatic ability of Impala to detect newly added tables
from hive.
Testing done:
1. Ran exhaustive tests by turning on the events processing multiple
times.
2. Ran exhaustive tests by disabling events processing.
3. Ran dockerized tests.
Change-Id: I9a8b1871a98b913d0ad8bb26a104a296b6a06122
Reviewed-on: http://gerrit.cloudera.org:8080/17612
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Quanlong Huang <huangquanlong@gmail.com>
This patch adds support for GCS(Google Cloud Storage). Using the
gcs-connector, the implementation is similar to other remote
FileSystems.
New flags for GCS:
- num_gcs_io_threads: Number of GCS I/O threads. Defaults to be 16.
Follow-up:
- Support for spilling to GCS will be addressed in IMPALA-10561.
- Support for caching GCS file handles will be addressed in
IMPALA-10568.
- test_concurrent_inserts and test_failing_inserts in
test_acid_stress.py are skipped due to slow file listing on
GCS (IMPALA-10562).
- Some tests are skipped due to issues introduced by /etc/hosts setting
on GCE instances (IMPALA-10563).
Tests:
- Compile and create hdfs test data on a GCE instance. Upload test data
to a GCS bucket. Modify all locations in HMS DB to point to the GCS
bucket. Remove some hdfs caching params. Run CORE tests.
- Compile and load snapshot data to a GCS bucket. Run CORE tests.
Change-Id: Ia91ec956de3b620cccf6a1244b56b7da7a45b32b
Reviewed-on: http://gerrit.cloudera.org:8080/17121
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch adds a feature flag, enable_incremental_metadata_updates, to
turn off incremental metadata (i.e. partition level metadata)
propagation from catalogd to coordinators. It defaults to true. When
setting to false, catalogd will send metadata updates in table
granularity (the legacy behavior).
Also fixes a bug of logging an empty aggregated partition update log
when no partitions are changed in a DDL.
Tests:
- Run CORE tests with this flag set to true and false.
- Add tests with enable_incremental_metadata_updates=false.
Change-Id: I98676fc8ca886f3d9f550f9b96fa6d6bff178ebb
Reviewed-on: http://gerrit.cloudera.org:8080/16436
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Currently, partitions are tightly integrated into the HdfsTable objects.
Catalogd has to transmit the entire table metadata even when few
partitions change. This is a waste of resources and can lead to OOM in
transmitting large tables due to the 2GB JVM array limit.
This patch makes HdfsPartition extend CatalogObject so the catalogd can
send partitions as individual catalog objects. Consequently, table
objects in the catalog topic update can have minimal partition maps that
only contain the partition ids, which reduces the thrift object size for
large tables. The catalog object key of HdfsPartition consists of db
name, table name and partition name.
In "full" topic mode (catalog_topic_mode=full), catalogd only sends
changed partitions with their latest table states. The latest table
states are table objects with the minimal partition map. Legacy
coordinators use the partition list to pick up existing (unchanged)
partitions from the existing table object and new partitions in the
catalog update.
Currently, partition instances are immutable - all partition
modifications are implemented by deleting the old instance and adding a
new one with a new partition id. Since partition ids are generated by a
global counter. Newer partition instances will have larger partition
ids. So catalogd maintains a watermark for each table as the max sent
partition id. Partition instances with ids larger than this are new
partitions that should be sent in the next catalog update. For the
deleted partition instances, they are kept in a set for each table until
the next catalog update. If there are no updates on the same partition
name, catalogd will send deletion on the partition.
For dropped or invalidated tables, catalogd will still send deletions on
their partitions. Although they are not used in coordinators
(coordinators delete the partitions when they delete the table
instances), they help in avoiding topic entry leak in the statestore
catalog topic.
In "minimal" topic mode (catalog_topic_mode=minimal), catalogd only
sends invalidations on tables and stale partition instances. Each
partition instance is identified by its partition id. LocalCatalog
coordinators use the partition invalidations to evict stale partitions
in time. For instance, let's say partition(year=2010) is updated in
catalogd. This is done by deleting the old partition instance
partition(id=0, year=2010) and adding a new partition instance
partition(id=1, year=2010). Catalogd will send invalidations on the
table and partition instance with id=0, but not the one with id=1. A
LocalCatalog coordinator will invalidate the partition instance(id=0) if
it's in the cache. If the partition instance(id=1) is cached, it's
already the latest version since partition instances are immutable. So
we don't need to invalidate it.
Tests
- Run exhaustive tests.
- Run exhaustive test_ddl.py in LocalCatalog mode.
- Add test in test_local_catalog.py to verify stale partitions are
invalidated in LocalCatalog when partitions are updated.
Change-Id: Ia0abfb346903d6e7cdc603af91c2b8937d24d870
Reviewed-on: http://gerrit.cloudera.org:8080/16159
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Vihang Karajgaonkar <vihang@cloudera.com>
Hive ACID supports row-level DELETE and UPDATE operations on a table.
It achieves it via assigning a unique row-id for each row, and
maintaining two sets of files in a table. The first set is in the
base/delta directories, they contain the INSERTed rows. The second set
of files are in the delete-delta directories, they contain the DELETEd
rows.
(UPDATE operations are implemented via DELETE+INSERT.)
In the filesystem it looks like e.g.:
* full_acid/delta_0000001_0000001_0000/0000_0
* full_acid/delta_0000002_0000002_0000/0000_0
* full_acid/delete_delta_0000003_0000003_0000/0000_0
During scanning we need to return INSERTed rows minus DELETEd rows.
This patch implements it by creating an ANTI JOIN between the INSERT and
DELETE sets. It is a planner-only modification. Every HDFS SCAN
that scans full ACID tables (that also have deleted rows) are converted
to two HDFS SCANs, one for the INSERT deltas, and one for the DELETE
deltas. Then a LEFT ANTI HASH JOIN with BROADCAST distribution mode is
created above them.
Later we can add support for other distribution modes if the performance
requires it. E.g. if we have too many deleted rows then probably we are
better off with PARTITIONED distribution mode. We could estimate the
number of deleted rows by sampling the delete delta files.
The current patch only works for primitive types. I.e. we cannot select
nested data if the table has deleted rows.
Testing:
* added planner test
* added e2e tests
Change-Id: I15c8feabf40be1658f3dd46883f5a1b2aa5d0659
Reviewed-on: http://gerrit.cloudera.org:8080/16082
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
In LocalCatlaog cache, partition metadata is cached with a composed key
of table name, table version and partition id. Whenever the table
version bumps, e.g. due to comments being changed, all the cached
partition values will be unreachable. Following queries have to reload
the partitions and cache them with the new table version. Actually, the
partition id is an unique id across the whole catalog. It's sufficient
to identify the partition. However, there are no partition level
invalidations if the partition is modified in-place in catalogd. So we
have to include the table version and depend on it.
After IMPALA-9778, there are no in-place modifications on partition
metadata. We can safely reuse partition meta across table versions in
LocalCatalog cache. This patch removes the table name and version in the
partition cache key. So metadata of unchanged partitions can be reused
when table version bumps.
Tests:
- Add tests in test_local_catalog.py to verify the partition metadata
is reused based on profile metrics.
Change-Id: I512f735b596bc51d553e6d395d108f49727619ed
Reviewed-on: http://gerrit.cloudera.org:8080/16081
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Vihang Karajgaonkar <vihang@cloudera.com>
In catalog server, when extracting the schema of a full ACID table from
the HMS table meta, we insert the synthetic "row__id" column. We need to
do the same thing in LocalCatalog mode coordinators, because they don't
fetch the loaded schema from catalog server. Instead, they fetch the HMS
table meta and extract the schema again.
Tests:
- Run test_acid.py in local catalog mode.
- Add a canary test in test_local_catalog.py. It can be removed once we
enable local catalog mode by default.
Change-Id: Ieb11e520325908e2e33a257568e15637d66c9901
Reviewed-on: http://gerrit.cloudera.org:8080/15783
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
The minimal catalog object version of valid catalog objects is used to
implement global invalidate metadata in legacy catalog mode. Coordinator
sends DDL RPC to catalogd for global invalidate metadata and gets the
expected min catalog version in the response. It's the version when
catalogd starts to reset the entire catalog, which means when the reset
is done, all valid catalog objects should be associated with a catalog
version larger than it. Coordinator will wait until its min catalog
version exceeds this value, which means it has processed all the updates
of the reset propagated from the catalogd via statestored. If SYNC_DDL
is set, the coordinator will also wait until other coordinators reach
the same statestore topic version with it, so they have also processed
the same updates and had the latest catalog after reset.
In local catalog mode, the coordinator does not cache all the metadata.
Instead, it caches them on-demand (based on query requests), and removes
them based on the Guava cache configurations (size or TTL) or explicit
invalidation from the catalog topic updates. So it's hard to track the
minimal catalog object version correctly.
This patch adds a new field (lastResetCatalogVersion) in TCatalog to
propagate the catalog version when catalogd starts to reset the entire
metadata. Each time when catalogd generates a new topic update, it will
generate a TCatalogObject of CATALOG type containing the state of the
catalog which includes this new field.
When coordinator receives a new value of lastResetCatalogVersion in a
topic update, it means catalogd has reset the entire catalog.
Coordinator will then clear its cache to remove all stale catalog
objects. It's possible that some fresh items being removed too. They
will be refetched on demand.
After the invalidation, there are no catalog object cached with catalog
version <= lastResetCatalogVersion. Because stale cache has been cleared
and all metadata from catalogd is newer than lastResetCatalogVersion. So
lastResetCatalogVersion + 1 is the lower bound (included) of min catalog
object version of a coordinator.
This patch also exposes the lower bound of catalog object version of via
a new metric "catalog.catalog-object-version-lower-bound" to ease
debugging.
IMPALA-9136 is also fixed in this patch.
Tests:
- Recover all existing tests that have been disabled due to this
missing feature
- Add custom cluster test for concurrent DDLs with INVALIDATE METADATA
- Run CORE tests
Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549
Reviewed-on: http://gerrit.cloudera.org:8080/14307
Reviewed-by: Vihang Karajgaonkar <vihang@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
getTCatalogObject() is not supported in local catalog mode
since metadata is partially fetched on demand. Removed hyperlinks
to the /catalog_object endpoints when local_catalog_mode is enabled.
Testing:
Added a test to test_local_catalog::TestObservability to verify
/catalog_mode endpoint is disabled when in local catalog mode.
Change-Id: Ia04797b32964c2edaa2e860dcf510d6f9cccd81c
Reviewed-on: http://gerrit.cloudera.org:8080/12443
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This handles a race condition in which a cache invalidation concurrent
with a cache load would potentially be skipped, causing out-of-date data
to persist in the cache. This would present itself as spurious "table
not found" errors.
A new test case triggers the issue reliably by injecting latency into
the metadata fetch RPC and running DDLs concurrently on the same
database across 8 threads. With the fix, the test passes reliably.
Another option to fix this might have been to switch to Caffeine instead
of Guava's loading cache. However, Caffeine requires Java 8, and
LocalCatalog is being backported to Impala 2.x which still can run on
Java 7. So, working around the Guava issue will make backporting (and
future backports) easier.
Change-Id: I70f377db88e204825a909389f28dc3451815235c
Reviewed-on: http://gerrit.cloudera.org:8080/13664
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
when enabled.
Added a banner on coordinator web UI for visibility of local
catalog mode.
Testing:
1. Added a test in test_local_catalog to scrape webpages and
search for strings. Tested banner appears when local catalog
mode is enabled.
2. Refactored parts of test_local_catalog into
TestObservability class.
Change-Id: Iaeb2c7b20742a2630d9509da9b629d5f45c55207
Reviewed-on: http://gerrit.cloudera.org:8080/12413
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
- Doubles the number of attempts to check for inconsistent metadata.
- Increases the chance of inconsistent metadata by adding more
concurrent 'refresh'es (for bumping the table versions).
Change-Id: I0eb914f4ab87de690fce3f24f356692163f069a4
Reviewed-on: http://gerrit.cloudera.org:8080/12105
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Current code throws a RuntimeException (RTE) when partial fetch RPCs
looking up partition metadata and the corresponding partition ID is
missing on the Catalog server. There are a couple of cases here.
1. The partition could be genuinely missing as it was dropped by a
concurrent operation.
2. Partial fetch RPCs lookup partitions by IDs instead of names. This is
problematic since the IDs can change over the lifetime of a table.
In both the cases, throwing a RTE is not the right approach and for (2)
we need to transparently retry the fetch with the new partition ID.
We eventually need to fix (2) as looking up by partition ID is not the
right approach.
Testing: Updated an e-e test which fails without the patch.
Change-Id: I2aa103ee159ce9478af9b5b27b36bc0cc286f442
Reviewed-on: http://gerrit.cloudera.org:8080/11732
Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
When configured to use the local catalog, concurrent metadata
reads and writes can cause the CatalogMetaProvider to throw
an InconsistentMetadataFetchException. Queries have been wrapped
with a retry loop, but the other frontend methods, such listing
table or partition information, can also fail from the same error.
These errors were seen under a workload consisting of concurrent
adding and showing partitions.
This change wraps call-sites (primarily in Frontend.java) that acquire
a Catalog, so have a chance of throwing an InconsistentMetadataFetchExecption.
Testing:
- added a test that checks whether inconsistent metadata exceptions
are seen in a concurrent workload.
Change-Id: I43a21571d54a7966c5c68bea1fe69dbc62be2a0b
Reviewed-on: http://gerrit.cloudera.org:8080/11608
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
The bug here was that any partial RPC on an IncompleteTable was throwing
an NPE.
Ideally, we attempt to load the table (if we find that it is not loaded)
before making the partial info request, but a concurrent invalidate could
reset the table state and move it back to an uninitialized state.
This patch handles this case better by propagating a meaningful error to
the caller.
Testing:
-------
- Added a test that fails consistently with an NPE without this patch.
Change-Id: I8533f73f25ca42a20f146ddfd95d4213add9b705
Reviewed-on: http://gerrit.cloudera.org:8080/11638
Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reapplies reverted IMPALA-7527. Adding a top-level entry to the profile
broke downstream consumers. The change here is to add the additional stats
to the summary profile.
This patch adds a Java wrapper for a RuntimeProfile object. The wrapper
supports some basic operations like non-hierarchical counters and
informational strings.
During planning, a profile is created, and passed back to the backend as
part of the ExecRequest. The backend then updates the query profile
based on the info emitted from the frontend.
This patch also adds the first use case for this profile information:
the CatalogdMetaProvider emits counters for cache hits, misses, and
fetch times, broken down by metadata category.
The emitted profile is a bit of a superset of the existing 'timeline'
functionality. However, it seems that some tools may parse the timeline
in its current location in the profile, so moving it might be
incompatible. I elected to leave that alone for now and just emit
counters in the new profile.
Change-Id: I419be157168cddb7521ea61e8f86733306b9315e
Reviewed-on: http://gerrit.cloudera.org:8080/11569
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Under heavy read/write load, the number of retries needed for queries
in order to skip over inconsistent metadata exceptions needs to be set
higher. This change makes the number of retries configurable. It can be
set with the newly added flag --local_catalog_max_fetch_retries.
In addition, this change increases the default from 10 to 40, which was
sufficient when handling several workloads with high read/write load.
Follow-up change for IMPALA-7597 will make use of this configuration
when retrying for cases other than analyzing queries.
Made several fixes to exception messages.
Testing:
- manual tests
- added an e2e test that sets the flag and checks for inconsistent metadata
Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
Reviewed-on: http://gerrit.cloudera.org:8080/11539
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch adds the aggregated CatalogdMetaProvider cache stats to
the catalog metrics on the coordinators. They can be accessed under
<coordinator>:<web-port>/metrics#catalog.
These metrics are refreshed at the end of planning, for each query run.
Testing:
-------
Visual inspection by running a few queries locally and making sure
stats are updated. Also modified existing tests to account for this
behavior.
Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Reviewed-on: http://gerrit.cloudera.org:8080/11511
Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch adds a Java wrapper for a RuntimeProfile object. The wrapper
supports some basic operations like non-hierarchical counters and
informational strings.
During planning, a profile is created, and passed back to the backend as
part of the ExecRequest. The backend then updates the query profile
based on the info emitted from the frontend.
This patch also adds the first use case for this profile information:
the CatalogdMetaProvider emits counters for cache hits, misses, and
fetch times, broken down by metadata category.
The emitted profile is a bit of a superset of the existing 'timeline'
functionality. However, it seems that some tools may parse the timeline
in its current location in the profile, so moving it might be
incompatible. I elected to leave that alone for now and just emit
counters in the new profile.
Change-Id: I808d55a225338912ebaebd0cf71c36db944b7276
Reviewed-on: http://gerrit.cloudera.org:8080/11388
Reviewed-by: Vuk Ercegovac <vercegovac@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Given the granular caching in CatalogdMetaProvider, it's possible to
have a situation like the following:
- impalad has cached the table list including some table "t"
- some other daemon has issued deletion of "t"
- we try to access "t" before the invalidation has reached us via the
statestore
In this case, we'll get an error back when we try to fetch the table
"t". This can confuse the planning process since it previously
determined that the table exists. Note that this may occur either when
"t" is first referenced or later during planning (eg when fetching a
specific partition post-pruning), so it wouldn't be possible to simply
convert it into a 'table does not exist' result.
The solution here is to throw InconsistentMetadataFetchException after
invalidating the table list and associated object. This then triggers a
re-plan of the query.
This patch implements such re-planning up to 10 times before eventually
giving up. Given that each attempt invalidates the inconsistent cache we
would normally expect it to succeed on the first such retry. The limit
of 10 retries is just to avoid infinite loops in the case of a bug.
Change-Id: I17389954780fa22d7866224c17e128458fffa545
Reviewed-on: http://gerrit.cloudera.org:8080/11403
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This implements cache invalidation inside CatalogdMetaProvider. The
design is as follows:
- when the catalogd collects updates into the statestore topic, it now
adds an additional entry for each table and database. These additional
entries are minimal - they only include the object's name, but no
metadata. This new behavior is conditional on a new flag
--catalog_topic_mode. The default mode is to keep the old style, but
it can be configured to mixed (support both v1 and v2) or v2-only.
- the old-style topic entries are prefixed with a '1:' whereas the new
minimal entries are prefixed with a '2:'. The impalad will subscribe
to one or the other prefix depending on whether it is running with
--use_local_catalog. Thus, old impalads will not be confused by the
new entries and vice versa.
- when the impalad gets these topic updates, it forwards them through to
the catalog implementation. The LocalCatalog implementation forwards
them to the CatalogdMetaProvider, which uses them to invalidate
cached metadata as appropriate.
This patch includes some basic unit tests. I also did some manual
testing by connecting to different impalads and verifying that a session
connected to impalad #1 saw the effects of DDLs made by impalad #2
within a short period of time (the statestore topic update frequency).
Existing end-to-end tests cover these code paths pretty thoroughly:
- if we didn't automatically invalidate the cache on a coordinator
in response to DDL operations, then any test which expects to
"read its own writes" (eg access a table after creating one)
would fail
- if we didn't propagate invalidations via the statestore, then
all of the tests that use sync_ddl would fail.
I verified the test coverage above using some of the tests in
test_ddl.py -- I selectively commented out a few of the invalidation
code paths in the new code and verified that tests failed until I
re-introduced them. Along the way I also improved test_ddl so that, when
this code is broken, it properly fails with a timeout. It also has a bit
of expanded coverage for both the SYNC_DDL and non-SYNC cases.
I also wrote a new custom-cluster test for LocalCatalog that verifies
a few of the specific edge cases like detecting catalogd restart,
SYNC_DDL behavior in mixed mode, etc.
One notable exception here is the implementation of INVALIDATE METADATA
This turned out to be complex to implement, so I left a lengthy TODO
describing the issue and filed a JIRA.
Change-Id: I615f9e6bd167b36cd8d93da59426dd6813ae4984
Reviewed-on: http://gerrit.cloudera.org:8080/11280
Reviewed-by: Todd Lipcon <todd@apache.org>
Tested-by: Todd Lipcon <todd@apache.org>
