jprdonnelly/impala
1
0
Fork 0
mirror of https://github.com/apache/impala.git synced 2025-12-19 18:12:08 -05:00
Code Issues Projects Releases Wiki Activity

IMPALA-1619: Support 64-bit allocations.

Browse Source 
This change extends MemPool, FreePool and StringBuffer to support
64-bit allocations, fixes a bug in decompressor and extends various
places in the code to support 64-bit allocation sizes. With this
change, the text scanner can now decompress compressed files larger
than 1GB.

Note that the UDF interfaces FunctionContext::Allocate() and
FunctionContext::Reallocate() still use 32-bit for the input
argument to avoid breaking compatibility.

In addition, the byte size of a tuple is still assumed to be
within 32-bit. If it needs to be upgraded to 64-bit, it will be
done in a separate change.

Change-Id: I7ed28083d809a86d801a9c063a0aa32c50d32b20
Reviewed-on: http://gerrit.cloudera.org:8080/2781
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
This commit is contained in:
Michael Ho
2016-03-30 19:34:57 -07:00
committed by Tim Armstrong
parent 5d9e13a6af
commit 5f3dfdf6c7
29 changed files with 363 additions and 338 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
infra/python/bootstrap_virtualenv.py
View File

@@ -123,8 +123,16 @@ def detect_python_cmd():
def install_deps():
toolchain_dir = os.environ.get("IMPALA_TOOLCHAIN", "")
snappy_version = os.environ.get("IMPALA_SNAPPY_VERSION", "")
snappy_dir = toolchain_dir + "/snappy-" + snappy_version
lib_path = snappy_dir + "/lib:" + os.environ.get("LD_LIBRARY_PATH", "")
include_dir = snappy_dir + "/include"
args = ["--global-option", "build_ext", "--global-option", "-L"+ lib_path,
"--global-option", "-I" + include_dir, "-r", REQS_PATH]
LOG.info("Installing packages into the virtualenv")
exec_pip_install(["-r", REQS_PATH])
exec_pip_install(args)
shutil.copyfile(REQS_PATH, INSTALLED_REQS_PATH)