mirror of
https://github.com/apache/impala.git
synced 2025-12-23 11:55:25 -05:00
ed5ec6772f
This change extends MemPool, FreePool and StringBuffer to support 64-bit allocations, fixes a bug in decompressor and extends various places in the code to support 64-bit allocation sizes. With this change, the text scanner can now decompress compressed files larger than 1GB. Note that the UDF interfaces FunctionContext::Allocate() and FunctionContext::Reallocate() still use 32-bit for the input argument to avoid breaking compatibility. In addition, the byte size of a tuple is still assumed to be within 32-bit. If it needs to be upgraded to 64-bit, it will be done in a separate change. A new test has been added to test the decompression of a 2GB snappy block compressed text file. Change-Id: Ic1af1564953ac02aca2728646973199381c86e5f Reviewed-on: http://gerrit.cloudera.org:8080/3575 Reviewed-by: Michael Ho <kwho@cloudera.com> Tested-by: Internal Jenkins
This folder contains a file necessary to test Impala's support for snappy block compressed file larger than 1GB. In particular, compressed_payload.snap is a string of 50176 bytes compressed using snappy. It's the building block for constructing a large snappy block compressed file. Please see test_compressed_formats.py for details.