This patch adds support for COS(Cloud Object Storage). Using the
hadoop-cos, the implementation is similar to other remote FileSystems.
New flags for COS:
- num_cos_io_threads: Number of COS I/O threads. Defaults to be 16.
Follow-up:
- Support for caching COS file handles will be addressed in
IMPALA-10772.
- test_concurrent_inserts and test_failing_inserts in
test_acid_stress.py are skipped due to slow file listing on
COS (IMPALA-10773).
Tests:
- Upload hdfs test data to a COS bucket. Modify all locations in HMS
DB to point to the COS bucket. Remove some hdfs caching params.
Run CORE tests.
Change-Id: Idce135a7591d1b4c74425e365525be3086a39821
Reviewed-on: http://gerrit.cloudera.org:8080/17503
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
When skip locked tables from topic updates is enabled
(topic_update_tbl_max_wait_time_ms > 0), it is possible that a thread
waiting for a topic update during a sync ddl execution to terminate
earlier. This is because the waitForSyncDdlVersion currently statically
counts the number of instances of topic updates against a maximum
and bails out when the maximum is reached. With topic update thread
skipping locked tables, this number of instances of topic updates
is more likely to hit the maximum attempt limit.
This commit changes the logic so that when locked tables are skipped from
topic updates, the sync ddl operation waits until a configurable
timeout. This timeout value is specified in seconds using the configuration
max_wait_time_for_sync_ddl_s. The default value of this configuration is 0
which means it waits indefinitely. This makes sure that only the sync ddl
queries on a table which has been locked wait for extended durations while
the other unrelated sync ddl queries can finish appropriately.
Also this commit changes the default values of following flags
which were introduced in the earlier patch for IMPALA-6671.
The current default value of topic_update_tbl_max_wait_time_ms
of 500ms is too low and may skip the locked tables more
aggressively than needed. The new defaults were set based on
analysis of a real world deployment.
topic_update_tbl_max_wait_time_ms = 120000
catalog_max_lock_skipped_topic_updates = 3
Testing:
1. Ran existing test test_topic_update_frequency.
2. Added a new test for the newly added flag.
Change-Id: I79e64cdec0e6aa7b597a47851b4b5c5441ca5528
Reviewed-on: http://gerrit.cloudera.org:8080/17253
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch adds support for GCS(Google Cloud Storage). Using the
gcs-connector, the implementation is similar to other remote
FileSystems.
New flags for GCS:
- num_gcs_io_threads: Number of GCS I/O threads. Defaults to be 16.
Follow-up:
- Support for spilling to GCS will be addressed in IMPALA-10561.
- Support for caching GCS file handles will be addressed in
IMPALA-10568.
- test_concurrent_inserts and test_failing_inserts in
test_acid_stress.py are skipped due to slow file listing on
GCS (IMPALA-10562).
- Some tests are skipped due to issues introduced by /etc/hosts setting
on GCE instances (IMPALA-10563).
Tests:
- Compile and create hdfs test data on a GCE instance. Upload test data
to a GCS bucket. Modify all locations in HMS DB to point to the GCS
bucket. Remove some hdfs caching params. Run CORE tests.
- Compile and load snapshot data to a GCS bucket. Run CORE tests.
Change-Id: Ia91ec956de3b620cccf6a1244b56b7da7a45b32b
Reviewed-on: http://gerrit.cloudera.org:8080/17121
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This change adds a mechanism for topic-update thread
to skip a table which is locked for more than a
configurable interval from the topic updates. This is
especially useful in scenarios where long running operations on a
locked table (refresh, recover partitions, compute stats) block the
topic update thread. This causes unrelated queries which are
waiting on metadata via topic updates (catalog-v1 mode)
to unnecessarily block.
The ideal solution of this problem would be to make HdfsTable
immutable so that there is no need for table lock. But that
is large change and not easily portable to older releases
of Impala. It would be taken up as a separate patch.
This change introduces 2 new configurations for catalogd:
1. topic_update_tbl_max_wait_time_ms: This defines the
maximum time in msecs the topic update thread waits on a locked table
before skipping the table from that iteration of topic updates.
The default value is 500. If this configuration is set to 0
the lock with timeout for topic update thread is disabled.
2. catalog_max_lock_skipped_topic_updates: This defines
the maximum number of distinct lock operations which are skipped
by topic update thread due to lock contention. Once this limit
is reached, topic update thread will block until it acquires
the table lock and adds it to the updates.
Testing:
1. Added a test case which introduces a simulated delay
in a few potentially long running statements. This causes the table
to be locked for a long time. The topic update thread skips
that table from updates and unrelated queries are unblocked
since they receive the required metadata from updates.
2. Added a test where multiple threads run blocking statements
in a loop to stress the table lock. It makes sure that topic
update thread is not starved and eventually blocks
on table lock by hitting the limit defined by
catalog_max_lock_skipped_topic_updates.
3. Ran exhaustive tests with default configurations.
Change-Id: Ic657b96edbcdc94c6b906e7ca59291f4e4715655
Reviewed-on: http://gerrit.cloudera.org:8080/16549
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
