mirror of
https://github.com/apache/impala.git
synced 2025-12-30 03:01:44 -05:00
547be27e77
Added checks/error handling: * Negative string lengths while decoding dictionary or data page. * Buffer overruns while decoding dictionary or data page. * Some metadata FILECHECKs were converted to statuses. Testing: Unit tests for: * decoding of strings with negative lengths * truncation of all parquet types * dictionary creation correctly handling error returns from Decode(). End-to-end tests for handling of negative string lengths in dictionary- and plain-encoded data in corrupt files, and for handling of buffer overruns for string data. The corrupted parquet files were generated by hacking Impala's parquet writer to write invalid lengths, and by hacking it to write plain-encoded data instead of dictionary-encoded data by default. Performance: set num_nodes=1; set num_scanner_threads=1; select * from biglineitem where l_orderkey = -1; I inspected MaterializeTupleTime. Before the average was 8.24s and after was 8.36s (a 1.4% slowdown, within the standard deviation of 1.8%). Change-Id: Id565a2ccb7b82f9f92cc3b07f05642a3a835bece Reviewed-on: http://gerrit.cloudera.org:8080/3387 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Internal Jenkins
These Parquet files were created by modifying Impala's HdfsParquetTableWriter. String Data ----------- These files have a single nullable string column 's'. dict-encoded-negative-len.parq: a single dictionary-encoded value with a negative length. dict-encoded-out-of-bounds.parq: a single dictionary-encoded value with a length past end of page. plain-encoded-negative-len.parq: a single plain-encoded value with a negative length. plain-encoded-out-of-bounds.parq: a single plain-encoded value with a length past end of page.