Moves several custom cluster tests out of core and into exhaustive
only. The tests were chosen based on their stability, lack of recent
modifications, and coverage of rare/corner cases.
Testing was accomplished by running both core and exhaustive tests
and manually verifying the tests were or were not skipped as
expected.
Change-Id: If99c015a0cb5d95b1607ca2be48d2dea04194f81
Reviewed-on: http://gerrit.cloudera.org:8080/22963
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Python 3 changes list operators such as range, map, and filter
to be lazy. Some code that expects the list operators to happen
immediately will fail. e.g.
Python 2:
range(0,5) == [0,1,2,3,4]
True
Python 3:
range(0,5) == [0,1,2,3,4]
False
The fix is to wrap locations with list(). i.e.
Python 3:
list(range(0,5)) == [0,1,2,3,4]
True
Since the base operators are now lazy, Python 3 also removes the
old lazy versions (e.g. xrange, ifilter, izip, etc). This uses
future's builtins package to convert the code to the Python 3
behavior (i.e. xrange -> future's builtins.range).
Most of the changes were done via these futurize fixes:
- libfuturize.fixes.fix_xrange_with_import
- lib2to3.fixes.fix_map
- lib2to3.fixes.fix_filter
This eliminates the pylint warnings:
- xrange-builtin
- range-builtin-not-iterating
- map-builtin-not-iterating
- zip-builtin-not-iterating
- filter-builtin-not-iterating
- reduce-builtin
- deprecated-itertools-function
Testing:
- Ran core job
Change-Id: Ic7c082711f8eff451a1b5c085e97461c327edb5f
Reviewed-on: http://gerrit.cloudera.org:8080/19589
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Tested-by: Joe McDonnell <joemcdonnell@cloudera.com>
This takes steps to make Python 2 behave like Python 3 as
a way to flush out issues with running on Python 3. Specifically,
it handles two main differences:
1. Python 3 requires absolute imports within packages. This
can be emulated via "from __future__ import absolute_import"
2. Python 3 changed division to "true" division that doesn't
round to an integer. This can be emulated via
"from __future__ import division"
This changes all Python files to add imports for absolute_import
and division. For completeness, this also includes print_function in the
import.
I scrutinized each old-division location and converted some locations
to use the integer division '//' operator if it needed an integer
result (e.g. for indices, counts of records, etc). Some code was also using
relative imports and needed to be adjusted to handle absolute_import.
This fixes all Pylint warnings about no-absolute-import and old-division,
and these warnings are now banned.
Testing:
- Ran core tests
Change-Id: Idb0fcbd11f3e8791f5951c4944be44fb580e576b
Reviewed-on: http://gerrit.cloudera.org:8080/19588
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Tested-by: Joe McDonnell <joemcdonnell@cloudera.com>
Python 3 now treats print as a function and requires
the parenthesis in invocation.
print "Hello World!"
is now:
print("Hello World!")
This fixes all locations to use the function
invocation. This is more complicated when the output
is being redirected to a file or when avoiding the
usual newline.
print >> sys.stderr , "Hello World!"
is now:
print("Hello World!", file=sys.stderr)
To support this properly and guarantee equivalent behavior
between python 2 and python 3, all files that use print
now add this import:
from __future__ import print_function
This also fixes random flake8 issues that intersect with
the changes.
Testing:
- check-python-syntax.sh shows no errors related to print
Change-Id: Ib634958369ad777a41e72d80c8053b74384ac351
Reviewed-on: http://gerrit.cloudera.org:8080/19552
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Tested-by: Michael Smith <michael.smith@cloudera.com>
Combines all SkipIf* classes for different filesystems into a single
SkipIfFS class. Many cases are simplified to 'not IS_HDFS', with the
rest as filesystem-specific special cases. The 'jira' option is removed
in favor of specific flags for each issue.
Change-Id: Ib928a6274baaaec45614887b9e762346a25812a1
Reviewed-on: http://gerrit.cloudera.org:8080/18781
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch adds support for COS(Cloud Object Storage). Using the
hadoop-cos, the implementation is similar to other remote FileSystems.
New flags for COS:
- num_cos_io_threads: Number of COS I/O threads. Defaults to be 16.
Follow-up:
- Support for caching COS file handles will be addressed in
IMPALA-10772.
- test_concurrent_inserts and test_failing_inserts in
test_acid_stress.py are skipped due to slow file listing on
COS (IMPALA-10773).
Tests:
- Upload hdfs test data to a COS bucket. Modify all locations in HMS
DB to point to the COS bucket. Remove some hdfs caching params.
Run CORE tests.
Change-Id: Idce135a7591d1b4c74425e365525be3086a39821
Reviewed-on: http://gerrit.cloudera.org:8080/17503
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
When skip locked tables from topic updates is enabled
(topic_update_tbl_max_wait_time_ms > 0), it is possible that a thread
waiting for a topic update during a sync ddl execution to terminate
earlier. This is because the waitForSyncDdlVersion currently statically
counts the number of instances of topic updates against a maximum
and bails out when the maximum is reached. With topic update thread
skipping locked tables, this number of instances of topic updates
is more likely to hit the maximum attempt limit.
This commit changes the logic so that when locked tables are skipped from
topic updates, the sync ddl operation waits until a configurable
timeout. This timeout value is specified in seconds using the configuration
max_wait_time_for_sync_ddl_s. The default value of this configuration is 0
which means it waits indefinitely. This makes sure that only the sync ddl
queries on a table which has been locked wait for extended durations while
the other unrelated sync ddl queries can finish appropriately.
Also this commit changes the default values of following flags
which were introduced in the earlier patch for IMPALA-6671.
The current default value of topic_update_tbl_max_wait_time_ms
of 500ms is too low and may skip the locked tables more
aggressively than needed. The new defaults were set based on
analysis of a real world deployment.
topic_update_tbl_max_wait_time_ms = 120000
catalog_max_lock_skipped_topic_updates = 3
Testing:
1. Ran existing test test_topic_update_frequency.
2. Added a new test for the newly added flag.
Change-Id: I79e64cdec0e6aa7b597a47851b4b5c5441ca5528
Reviewed-on: http://gerrit.cloudera.org:8080/17253
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch adds support for GCS(Google Cloud Storage). Using the
gcs-connector, the implementation is similar to other remote
FileSystems.
New flags for GCS:
- num_gcs_io_threads: Number of GCS I/O threads. Defaults to be 16.
Follow-up:
- Support for spilling to GCS will be addressed in IMPALA-10561.
- Support for caching GCS file handles will be addressed in
IMPALA-10568.
- test_concurrent_inserts and test_failing_inserts in
test_acid_stress.py are skipped due to slow file listing on
GCS (IMPALA-10562).
- Some tests are skipped due to issues introduced by /etc/hosts setting
on GCE instances (IMPALA-10563).
Tests:
- Compile and create hdfs test data on a GCE instance. Upload test data
to a GCS bucket. Modify all locations in HMS DB to point to the GCS
bucket. Remove some hdfs caching params. Run CORE tests.
- Compile and load snapshot data to a GCS bucket. Run CORE tests.
Change-Id: Ia91ec956de3b620cccf6a1244b56b7da7a45b32b
Reviewed-on: http://gerrit.cloudera.org:8080/17121
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This change adds a mechanism for topic-update thread
to skip a table which is locked for more than a
configurable interval from the topic updates. This is
especially useful in scenarios where long running operations on a
locked table (refresh, recover partitions, compute stats) block the
topic update thread. This causes unrelated queries which are
waiting on metadata via topic updates (catalog-v1 mode)
to unnecessarily block.
The ideal solution of this problem would be to make HdfsTable
immutable so that there is no need for table lock. But that
is large change and not easily portable to older releases
of Impala. It would be taken up as a separate patch.
This change introduces 2 new configurations for catalogd:
1. topic_update_tbl_max_wait_time_ms: This defines the
maximum time in msecs the topic update thread waits on a locked table
before skipping the table from that iteration of topic updates.
The default value is 500. If this configuration is set to 0
the lock with timeout for topic update thread is disabled.
2. catalog_max_lock_skipped_topic_updates: This defines
the maximum number of distinct lock operations which are skipped
by topic update thread due to lock contention. Once this limit
is reached, topic update thread will block until it acquires
the table lock and adds it to the updates.
Testing:
1. Added a test case which introduces a simulated delay
in a few potentially long running statements. This causes the table
to be locked for a long time. The topic update thread skips
that table from updates and unrelated queries are unblocked
since they receive the required metadata from updates.
2. Added a test where multiple threads run blocking statements
in a loop to stress the table lock. It makes sure that topic
update thread is not starved and eventually blocks
on table lock by hitting the limit defined by
catalog_max_lock_skipped_topic_updates.
3. Ran exhaustive tests with default configurations.
Change-Id: Ic657b96edbcdc94c6b906e7ca59291f4e4715655
Reviewed-on: http://gerrit.cloudera.org:8080/16549
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
