jprdonnelly/impala
1
0
Fork 0
mirror of https://github.com/apache/impala.git synced 2025-12-23 21:08:39 -05:00
Code Issues Projects Releases Wiki Activity

IMPALA-11683: Support Aliyun OSS File System

Browse Source 
This patch adds support for OSS (Aliyun Object Storage Service).
Using the hadoop-aliyun, the implementation is similar to other
remote FileSystems.

Tests:
- Prepare:
  Initialize OSS-related environment variables:
  OSS_ACCESS_KEY_ID, OSS_SECRET_ACCESS_KEY, OSS_ACCESS_ENDPOINT.
  Compile and create hdfs test data on a ECS instance. Upload test data
  to an OSS bucket.
- Modify all locations in HMS DB to point to the OSS bucket.
  Remove some hdfs caching params. Run CORE tests.

Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Reviewed-on: http://gerrit.cloudera.org:8080/19165
Reviewed-by: Quanlong Huang <huangquanlong@gmail.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This commit is contained in:
yacai
2022-10-24 16:31:45 +08:00
committed by Impala Public Jenkins
parent 5f7210a8df
commit c953426692
14 changed files with 87 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
tests/custom_cluster/test_hdfs_fd_caching.py
View File

@@ -24,7 +24,8 @@ from tests.util.filesystem_utils import (
IS_ISILON,
IS_ADLS,
IS_GCS,
IS_COS)
IS_COS,
IS_OSS)
from time import sleep
@@ -135,7 +136,7 @@ class TestHdfsFdCaching(CustomClusterTestSuite):
# Caching applies to HDFS, Ozone, S3, and ABFS files. If this is HDFS, Ozone, S3, or
# ABFS, then verify that caching works. Otherwise, verify that file handles are not
# cached.
if IS_ADLS or IS_ISILON or IS_GCS or IS_COS:
if IS_ADLS or IS_ISILON or IS_GCS or IS_COS or IS_OSS:
caching_expected = False
else:
caching_expected = True
@@ -152,7 +153,7 @@ class TestHdfsFdCaching(CustomClusterTestSuite):
handle_timeout = 5
# Only test eviction on platforms where caching is enabled.
if IS_ADLS or IS_ISILON or IS_GCS or IS_COS:
if IS_ADLS or IS_ISILON or IS_GCS or IS_COS or IS_OSS:
return
caching_expected = True
self.run_fd_caching_test(vector, caching_expected, cache_capacity, handle_timeout)
@@ -203,7 +204,7 @@ class TestHdfsFdCaching(CustomClusterTestSuite):
eviction_timeout_secs = 5
# Only test eviction on platforms where caching is enabled.
if IS_ADLS or IS_ISILON or IS_GCS or IS_COS:
if IS_ADLS or IS_ISILON or IS_GCS or IS_COS or IS_OSS:
return
# Maximum number of file handles cached.