mirror of
https://github.com/apache/impala.git
synced 2026-01-28 18:00:14 -05:00
27fa27e808
This change is a follow-up to IMPALA-7368 and adds support for DATE type to the avro scanner. Similarly to parquet, avro uses DATE logical type for dates. DATE logical type annotates an INT32 that stores the number of days since the unix epoch, 1 January 1970. This representation introduces an avro interoperability issue between Impala and older versions of Hive: - Before version 3.1, Hive used Julian calendar to represent dates up to 1582-10-05 and Gregorian calendar for dates starting with 1582-10-15. Dates between 1582-10-05 and 1582-10-15 were lost. - Impala uses proleptic Gregorian calendar, extending the Gregorian calendar backward to dates preceding its official introduction in 1582-10-15. This means that pre-1582-10-15 dates written to an avro table by Hive will be read back incorrectly by Impala. Note that Hive 3.1 switched to proleptic Gregorian calendar too, so for Hive 3.1+ this is no longer an issue. Dependency changes: - BE uses avro 1.7.4-p5 from native-toolchain. Change-Id: I7a9d5b93a22cf3a00244037e187f8c145cacc959 Reviewed-on: http://gerrit.cloudera.org:8080/13944 Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
String Data ----------- Created by modifying Impala's HdfsAvroTableWriter(removed). These files' schemas have a single nullable string column 's'. negative_string_len.avro: contains two values, but the second value has a negative length. invalid_union.avro: contains three values, all of which have an invalid union value (4, which decodes to 2). Also has a single extra byte at the end (0x0a), which will be interpreted as a malformed data block. truncated_string.avro: contains one value, which is missing the last byte. Float Data ---------- Created by modifying Impala's HdfsAvroTableWriter(removed). These files' schemas have a single nullable float column 'c1'. truncated_float.avro: contains two float values. The second is missing the last byte. Bad Schema ---------- Created by editing the schema of a valid Avro file with vim. invalid_decimal_schema.avro: two columns, name STRING and value DECIMAL(5,7). The DECIMAL value is invalid. Date Data --------- out_of_range_date.avro: Small avro table with a single DATE column. Contains 5 values, two of which are out of range. hive2_pre_gregorian_date.avro: Small avro table with one DATE column, created by Hive 2.1.1. Used to demonstrate avro interoperability issues between Hive and Impala for dates before the introduction of Gregorian calendar in 1582-10-15. hive3_pre_gregorian_date.avro: Small avro table with one DATE column, created by Hive 3.1.0. Used to demonstrate that pre-Gregorian avro dates written by Hive3 are read back correctly by Impala.