mirror of
https://github.com/apache/impala.git
synced 2026-01-07 18:02:33 -05:00
87b9fac2ad
No changes to writing were made. No changes to reading Impala written
files were made.
Hive writes TIMESTAMP values to parquet files differently than Impala
does. Hive converts the value from local time to UTC before writing;
Impala does not. This change adds a startup flag that will convert UTC
to local when reading files written by Hive.
The Hive-file detection actually checks for "parquet-mr" (which is the
library Hive uses) in the file metadata. A slight possibility exists
that TIMESTAMP values written by something other than Hive but also
using parquet-mr may become incorrect. The possibility should be very
small because TIMESTAMP values are stored and encoded in a non-standard
way other applications are unlikely to be aware of.
Flags from be/src/exec/hdfs-parquet-scanner.cc:
-convert_legacy_hive_parquet_utc_timestamps (When true, TIMESTAMPs
read from files written by Parquet-MR (used by Hive) will be
converted from UTC to local time. Writes are unaffected.) type: bool
default: false
Change-Id: I79a499fe24049b7025ee2dd76c9c3e07010d346a
Reviewed-on: http://gerrit.cloudera.org:8080/35
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
This directory contains Impala test data sets. The directory layout is structured as follows: datasets/ <data set>/<data set>_schema_template.sql <data set>/<data files SF1>/data files <data set>/<data files SF2>/data files Where SF is the scale factor controlling data size. This allows for scaling the same schema to different sizes based on the target test environment.