jprdonnelly/impala
1
0
Fork 0
mirror of https://github.com/apache/impala.git synced 2025-12-25 02:03:09 -05:00
Code Issues Projects Releases Wiki Activity

IMPALA-12899: Temporary workaround for BINARY in complex types

Browse Source 
The BINARY type is currently not supported inside complex types and a
cross-component decision is probably needed to support it (see
IMPALA-11491). We would like to enable EXPAND_COMPLEX_TYPES for Iceberg
metadata tables (IMPALA-12612), which requires that queries with BINARY
inside complex types don't fail. Enabling EXPAND_COMPLEX_TYPES is a more
prioritised issue than IMPALA-11491, so we have come up with a
temporary solution.

This change NULLs out BINARY values in complex types coming from Iceberg
metadata tables and logs a warning.

BINARYs in complex types from regular tables are not affected by this
change.

Testing:
 - Added test queries in iceberg-metadata-tables.test.

Change-Id: I0d834126c7d702a25e957bb6071ecbf0fda2c203
Reviewed-on: http://gerrit.cloudera.org:8080/21219
Reviewed-by: Gabor Kaszab <gaborkaszab@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This commit is contained in:
Daniel Becker
2024-03-14 18:29:02 +01:00
parent f05eac6476
commit a623447206
5 changed files with 64 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
fe/src/main/java/org/apache/impala/analysis/Analyzer.java
View File

@@ -1801,7 +1801,10 @@ public class Analyzer {
boolean isResolved = resolvedPath.resolve();
Preconditions.checkState(isResolved);
if (resolvedPath.destType().isBinary()) {
if (resolvedPath.destType().isBinary() &&
!collTblRef.getResolvedPath().comesFromIcebergMetadataTable()) {
// We allow BINARY fields in collections from Iceberg metadata tables but NULL them
// out.
throw new AnalysisException(
"Binary type inside collection types is not supported (IMPALA-11491).");
}

13
fe/src/main/java/org/apache/impala/analysis/Path.java
View File

@@ -456,6 +456,19 @@ public class Path {
return result;
}
/**
* Returns whether the given path belongs to a (possibly nested) field from an Iceberg
* metadata table.
*/
public boolean comesFromIcebergMetadataTable() {
Preconditions.checkState(rootTable_ != null || rootDesc_ != null);
if (rootDesc_ != null) {
return rootDesc_.getTable() instanceof IcebergMetadataTable;
} else {
return rootTable_ instanceof IcebergMetadataTable;
}
}
/**
* Returns the absolute explicit path starting from the fully-qualified table name.
* The goal is produce a canonical non-ambiguous path that can be used as an

4
fe/src/main/java/org/apache/impala/analysis/SlotRef.java
View File

@@ -214,7 +214,9 @@ public class SlotRef extends Expr {
throw new AnalysisException("Unsupported type '"
+ fieldType.toSql() + "' in '" + toSql() + "'.");
}
if (fieldType.isBinary()) {
if (fieldType.isBinary() && !desc_.getPath().comesFromIcebergMetadataTable()) {
// We allow BINARY fields in collections from Iceberg metadata tables but NULL
// them out.
throw new AnalysisException("Struct containing a BINARY type is not " +
"allowed in the select list (IMPALA-11491).");
}