jprdonnelly/impala
1
0
Fork 0
mirror of https://github.com/apache/impala.git synced 2025-12-25 02:03:09 -05:00
Code Issues Projects Releases Wiki Activity

IMPALA-7370: DATE: Read/Write to parquet.

Browse Source 
This change is a follow-up to IMPALA-7368 and adds support for DATE
type to the parquet scanner/writer. CREATE TABLE LIKE PARQUET
statements associated with data files that contain dates are also
supported.

Parquet uses DATE logical type for dates. DATE logical type annotates
an INT32 that stores the number of days from the Unix epoch, 1 January
1970.

This representation introduces a parquet interoperability issue
between Impala and older versions of Hive:
- Before version 3.1, Hive used Julian calendar to represent dates
  up to 1582-10-05 and Gregorian calendar for dates starting with
  1582-10-15. Dates between 1582-10-05 and 1582-10-15 were lost.
- Impala uses proleptic Gregorian calendar, extending the Gregorian
  calendar backward to dates preceding its official introduction in
  1582-10-15.
This means that pre-1582-10-15 dates written to a parquet table by
Hive will be read back incorrectly by Impala and vice versa.

Note that Hive 3.1 switched to proleptic Gregorian calendar too, so
for Hive 3.1+ this is no longer an issue.

Change-Id: I67da03754531660bc8de3b6935580d46deae1814
Reviewed-on: http://gerrit.cloudera.org:8080/13189
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This commit is contained in:
Attila Jeges
2019-04-23 16:02:25 +02:00
committed by Impala Public Jenkins
parent ae7aefe34c
commit 2bce974990
29 changed files with 465 additions and 148 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
tests/query_test/test_date_queries.py
View File

@@ -39,9 +39,9 @@ class TestDateQueries(ImpalaTestSuite):
'batch_size': [0, 1],
'disable_codegen': ['false', 'true'],
'disable_codegen_rows_threshold': [0]}))
# DATE type is only supported for text fileformat on HDFS and HBASE.
# DATE type is only supported for text & parquet fileformat on HDFS and HBASE.
cls.ImpalaTestMatrix.add_constraint(lambda v:
v.get_value('table_format').file_format in ('text', 'hbase'))
v.get_value('table_format').file_format in ('text', 'hbase', 'parquet'))
# Run these queries through both beeswax and HS2 to get coverage of date returned
# via both protocols.
@@ -63,7 +63,7 @@ class TestDateQueries(ImpalaTestSuite):
@SkipIfABFS.qualified_path
@SkipIfADLS.qualified_path
@SkipIfLocal.qualified_path
def test_text_only_support(self, vector, unique_database):
def test_fileformat_support(self, vector, unique_database):
""" Test that scanning and writing DATE is supported for text tables only."""
# This test specifies databases and locations explicitly. No need to execute it for
# anything other than text fileformat on HDFS.
@@ -83,7 +83,7 @@ class TestDateQueries(ImpalaTestSuite):
["/testdata/data/date_tbl.avro"])
# Partitioned table with parquet and avro partitions
TABLE_NAME = "date_tbl"
CREATE_SQL = """CREATE TABLE {0}.{1} (id_col INT, date_col DATE)
CREATE_SQL = """CREATE TABLE {0}.{1} (date_col DATE)
PARTITIONED BY (date_part DATE)""".format(unique_database, TABLE_NAME)
self.client.execute(CREATE_SQL)
# Add partitions
@@ -102,4 +102,5 @@ class TestDateQueries(ImpalaTestSuite):
SET FILEFORMAT AVRO""".format(unique_database, TABLE_NAME)
self.client.execute(SET_PART_FF_SQL)
# Test scanning/writing tables with different fileformats.
self.run_test_case('QueryTest/date-text-only-support', vector, use_db=unique_database)
self.run_test_case('QueryTest/date-fileformat-support', vector,
use_db=unique_database)