mirror of
https://github.com/apache/impala.git
synced 2025-12-26 14:02:53 -05:00
2c54dbe225
ORC scanner uses TimestampValue::FromUnixTimeNanos() to convert sec + nano representation to Impala's TimestampValue (day + nano). FromUnixTimeNanos was affected by flag use_local_tz_for_unix_timestamp_conversions, while that global option should not affect ORC. By default there was no conversion, but if the flag is 1, then timestamps were interpreted as UTC and converted to local time. This could be solved by creating a UTC version of FromUnixTimeNanos, but I decided to change the interface in the hope of making To/From timestamp functions less confusing. Changes: - Fixed the bug by passing UTC as timezone in the ORC scanner. - Changed the interface of these TimestampValue functions to expect a timezone pointer, interpret null as UTC and skip conversion. It would be also possible to pass the actual UTC timezone and check for this in the functions, but I guess it is easier to optimize the inlined functions this way. - Moved the checking of use_local_tz_for_unix_timestamp_conversions to RuntimeState and added property time_zone_for_unix_time_conversions() to return the timezone to use in Unix time conversions. This made TimestampValue's interface clearer and makes it easy to replace the flag with a query option if we want to. - Changed RuntimeState and the Parquet scanner to skip timezone conversion if convert_legacy_hive_parquet_utc_timestamps=1 but the timezone is UTC. This allows users to avoid the performance penalty of this flag by setting query option timezone to UTC in their session (IMPALA-7557). CCTZ is not good at this, actually conversions are slower with fixed offset timezones (including UTC) than with timezones that have DST/historical rule changes. Postponed changes: - Didn't remove the UTC versions of the functions yet, as that would require changing (and possibly rethinking) several BE tests and benchmarks (IMPALA-9409). Tests: - Added regression test for Orc and other file formats to check that they are not affected by this flag. - Extended test_hive_parquet_timestamp_conversion.py to cover the case when convert_legacy_hive_parquet_utc_timestamps=1 and timezone=UTC. Also did some cleanup there to use query option timezone instead of env var TZ. Change-Id: I14e2a7e512ccd013d5d9fe480a5467ed4c46b76e Reviewed-on: http://gerrit.cloudera.org:8080/15222 Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>