mirror of
https://github.com/apache/impala.git
synced 2026-02-02 06:00:36 -05:00
e91c7810f0
IcebergScanNode interprets the timestamp literals as UTC timestamps during predicate pushdown to Iceberg. It causes problems when the Iceberg table uses TIMESTAMPTZ (which corresponds to TIMESTAMP WITH LOCAL TIME ZONE in SQL) because in the scanners we assume that the timestamp literals in a query are in local timezone. Hence, if the Iceberg table is partitioned by HOUR(ts), and Impala is running in a different timezone than UTC, then the following query doesn't return any rows: SELECT * from t WHERE ts = <some ts>; Because during predicate pushdown the timestamp is interpreted as a UTC timestamp (no conversion from local to UTC), but during query execution the timestamp data in the files are converted to local timezone, then compared to <some ts>. I.e. in the scanner the assumption is that <some ts> is in local timezone. On the other hand, when Iceberg type TIMESTAMP (which correcponds to TIMESTAMP WITHOUT TIME ZONE in SQL) is used, then we should just push down the timestamp values without any conversion. In this case there is no conversion in the scanners either. Testing: * added e2e test with TIMESTAMPTZ * added e2e test with TIMESTAMP Change-Id: I181be5d2fa004f69b457f69ff82dc2f9877f46fa Reviewed-on: http://gerrit.cloudera.org:8080/18399 Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com> Reviewed-by: Csaba Ringhofer <csringhofer@cloudera.com>