mirror of
https://github.com/apache/impala.git
synced 2026-01-06 06:01:03 -05:00
bce6b2b422
This change is a first step towards a more efficient Parquet scanner. The focus is on presenting the new code flow that materializes the table-level slots in a column-wise fashion, without going deep into actually improving scan efficieny. After these changes there are several obvious places that should be optimized to realize efficiency gains. Summary of changes - the table-level tuples are materialized in a column-wise fashion with new ColumnReader::ReadValueBatch() functions - this is done by materializing a 'scratch' batch, and transferring scratch tuples that survive filters/conjuncts to the output batch - the tuples of nested collections are still materialized in a row-wise fashion using the ColumnReader::ReadValue() function, just as before Mini benchmark I ran the following queries on a single impalad before and after my change using a synthetic 'huge_lineitem' table. I modified hdfs-scan-node.cc to set the number of rows of any row batch to 0 to focus the measurement on the scan time. Query options: set num_scanner_threads=1; set disable_codegen=true; set num_nodes=1; select * from huge_lineitem; Before: 22.39s Afer: 18.50s select * from huge_lineitem where l_linenumber < 0; Before: 25.11s After: 20.56s select * from huge_lineitem where l_linenumber % 2 = 0; Before: 26.32s After: 21.82s Change-Id: I72a613fa805c542e39df20588fb25c57b5f139aa Reviewed-on: http://gerrit.cloudera.org:8080/2779 Reviewed-by: Alex Behm <alex.behm@cloudera.com> Tested-by: Internal Jenkins
20 lines
698 B
Plaintext
20 lines
698 B
Plaintext
====
|
|
---- QUERY
|
|
# IMPALA-2558: Trigger bad parse_status_ in HdfsParquetScanner::AssembleRows().
|
|
# Abort on error must be used to trigger a status.
|
|
# Set a single node and scanner thread to make this test deterministic.
|
|
set num_nodes=1;
|
|
set num_scanner_threads=1;
|
|
select id, cnt from bad_column_metadata t, (select count(*) cnt from t.int_array) v
|
|
---- CATCH
|
|
Column metadata states there are 50 values, but read 100 values from column element.
|
|
====
|
|
---- QUERY
|
|
# IMPALA-2558. Same as above but only selecting a scalar column.
|
|
set num_nodes=1;
|
|
set num_scanner_threads=1;
|
|
select id from bad_column_metadata
|
|
---- CATCH
|
|
Column metadata states there are 11 values, but read 10 values from column id.
|
|
====
|