jprdonnelly/impala
1
0
Fork 0
mirror of https://github.com/apache/impala.git synced 2025-12-19 18:12:08 -05:00
Code Issues Projects Releases Wiki Activity

IMPALA-7712: Support Google Cloud Storage

Browse Source 
This patch adds support for GCS(Google Cloud Storage). Using the
gcs-connector, the implementation is similar to other remote
FileSystems.

New flags for GCS:
 - num_gcs_io_threads: Number of GCS I/O threads. Defaults to be 16.

Follow-up:
 - Support for spilling to GCS will be addressed in IMPALA-10561.
 - Support for caching GCS file handles will be addressed in
   IMPALA-10568.
 - test_concurrent_inserts and test_failing_inserts in
   test_acid_stress.py are skipped due to slow file listing on
   GCS (IMPALA-10562).
 - Some tests are skipped due to issues introduced by /etc/hosts setting
   on GCE instances (IMPALA-10563).

Tests:
 - Compile and create hdfs test data on a GCE instance. Upload test data
   to a GCS bucket. Modify all locations in HMS DB to point to the GCS
   bucket. Remove some hdfs caching params. Run CORE tests.
 - Compile and load snapshot data to a GCS bucket. Run CORE tests.

Change-Id: Ia91ec956de3b620cccf6a1244b56b7da7a45b32b
Reviewed-on: http://gerrit.cloudera.org:8080/17121
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This commit is contained in:
stiga-huang
2021-02-25 20:19:49 +08:00
committed by Impala Public Jenkins
parent 6c6b0ee869
commit 2dfc68d852
68 changed files with 303 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
tests/shell/test_shell_commandline.py
View File

@@ -653,7 +653,9 @@ class TestImpalaShell(ImpalaTestSuite):
assert "Error retrieving LDAP password" in result_1.stderr
assert "command was: 'cmddoesntexist'" in result_1.stderr
assert "No such file or directory" in result_1.stderr
# On GCE instances, the error thrown in subprocess is "[Errno 13] Permission denied".
assert "No such file or directory" in result_1.stderr \
or "Permission denied" in result_1.stderr
result_2 = run_impala_shell_cmd(vector,
args + ['--ldap_password_cmd=cat filedoesntexist'],