jprdonnelly/impala
1
0
Fork 0
mirror of https://github.com/apache/impala.git synced 2026-01-05 12:01:11 -05:00
Code Issues Projects Releases Wiki Activity

IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table

Browse Source 
When Impala/Hive creates a table they lowercase the schema elements.
When Spark creates an Iceberg table it doesn't lowercase the names
of the columns in the Iceberg metadata. This triggers a precondition
check in Impala which makes such Iceberg tables unloadable.

This patch converts column names to lowercase when converting Iceberg
schemas to Hive/Impala schemas.

Testing:
 * added e2e test

Change-Id: Iffd910f76844fbf34db805dda6c3053c5ad1cf79
Reviewed-on: http://gerrit.cloudera.org:8080/18368
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This commit is contained in:
Zoltan Borok-Nagy
2022-03-30 16:18:46 +02:00
committed by Impala Public Jenkins
parent 38efbee29d
commit 952f2af0ca
8 changed files with 272 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
testdata/datasets/functional/functional_schema_template.sql vendored
View File

@@ -3297,6 +3297,20 @@ STORED AS ICEBERG;
---- DATASET
functional
---- BASE_TABLE_NAME
iceberg_uppercase_col
---- CREATE
CREATE EXTERNAL TABLE IF NOT EXISTS {db_name}{db_suffix}.{table_name}
STORED AS ICEBERG
TBLPROPERTIES('write.format.default'='parquet', 'iceberg.catalog'='hadoop.catalog',
'iceberg.catalog_location'='/test-warehouse/iceberg_test/hadoop_catalog',
'iceberg.table_identifier'='ice.iceberg_uppercase_col');
---- DEPENDENT_LOAD
`hadoop fs -mkdir -p /test-warehouse/iceberg_test/hadoop_catalog/ice && \
hadoop fs -put -f ${IMPALA_HOME}/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_uppercase_col /test-warehouse/iceberg_test/hadoop_catalog/ice
====
---- DATASET
functional
---- BASE_TABLE_NAME
alltypes_date_partition_2
---- PARTITION_COLUMNS
date_col date