jprdonnelly/impala
1
0
Fork 0
mirror of https://github.com/apache/impala.git synced 2025-12-25 02:03:09 -05:00
Code Issues Projects Releases Wiki Activity

IMPALA-11697: Enable SkipIf.not_hdfs tests for Ozone

Browse Source 
Convert SkipIf.not_hdfs to SkipIf.not_dfs for tests that require
filesystem semantics, adding more feature test coverage with Ozone.

Creates a separate not_scratch_fs flag for scratch dir tests as they're
not supported with Ozone yet. Filed IMPALA-11730 to address this.

Preserves not_hdfs for a specific test that uses the dfsadmin CLI to put
it in safemode.

Adds sfs_ofs_unsupported for SmallFileSystem tests. This should work for
many of our filesystems based on
ebb1e2fa99/ql/src/java/org/apache/hadoop/hive/ql/io/SingleFileSystem.java (L62-L87). Makes sfs tests work on S3.

Adds hardcoded_uris for IcebergV2 tests where deletes are implemented as
hardcoded URIs in parquet files. Adding a parquet read/write library for
Python is beyond the scope if this patch.

Change-Id: Iafc1dac52d013e74a459fdc4336c26891a256ef1
Reviewed-on: http://gerrit.cloudera.org:8080/19254
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
This commit is contained in:
Michael Smith
2022-11-16 14:13:54 -08:00
parent 87e0077255
commit f8443d9828
12 changed files with 68 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
tests/common/skip.py
View File

@@ -98,7 +98,15 @@ class SkipIf:
skip_hbase = pytest.mark.skipif(pytest.config.option.skip_hbase,
reason="--skip_hbase argument specified")
not_s3 = pytest.mark.skipif(not IS_S3, reason="S3 Filesystem needed")
not_hdfs = pytest.mark.skipif(not IS_HDFS, reason="HDFS Filesystem needed")
not_hdfs = pytest.mark.skipif(not IS_HDFS, reason="HDFS admin needed")
not_dfs = pytest.mark.skipif(not (IS_HDFS or IS_OZONE),
reason="HDFS/Ozone Filesystem needed")
not_scratch_fs = pytest.mark.skipif(not IS_HDFS,
reason="Scratch dirs for temporary file spilling not supported")
sfs_unsupported = pytest.mark.skipif(not (IS_HDFS or IS_S3 or IS_ABFS or IS_ADLS
or IS_GCS), reason="Hive support for sfs+ is limited, HIVE-26757")
hardcoded_uris = pytest.mark.skipif(not IS_HDFS,
reason="Iceberg delete files hardcode the full URI in parquet files")
not_ec = pytest.mark.skipif(not IS_EC, reason="Erasure Coding needed")
no_secondary_fs = pytest.mark.skipif(not SECONDARY_FILESYSTEM,
reason="Secondary filesystem needed")