mirror of
https://github.com/apache/impala.git
synced 2026-01-02 03:00:32 -05:00
c14a090400
This is also a step towards IMPALA-2399 (remove QueryMaintenance()). "local" allocations containing expression results (either intermediate or final results) have the following properties: * They are usually small allocations * They can be made frequently (e.g. every function call) * They are owned and managed by the Impala runtime * They are freed in bulk at various points in query execution. A MemPool (i.e. bump allocator) is the right mechanism to manage allocations with the above properties. Before this patch FunctionContext's used a FreePool + vector of allocations to emulate the above behaviour. This patch switches to using a MemPool to bring these allocations in line with the rest of the codebase. The steps required to do this conversion. * Use a MemPool for FunctionContext local allocations. * Identify appropriate MemPools for all of the local allocations from function contexts so that the memory lifetime is correct. * Various cleanup and documentation of existing MemPools. * Replaces calls to FreeLocalAllocations() with calls to MemPool::Clear() More involved surgery was required in a few places: * Made the Sorter own its comparator, exprs and MemPool. * Remove FunctionContextImpl::ReallocateLocal() and just have StringFunctions::Replace() do the doubling itself to avoid the need for a special interface. Worst-case this doubles the memory requirements for Replace() since n / 2 + n / 4 + n / 8 + .... bytes of memory could be wasted instead of recycled for an n-byte output string. * Provide a way redirect agg fn Serialize()/Finalize() allocations to come directly from the output RowBatch's MemPool. This is also potentially applicable to other places where we currently copy out strings from local allocations, e.g. AnalyticEvalNode::AddResultTuple() and Tuple::MaterializeExprs(). * --stress_free_pool_alloc was changed to instead intercept at the FunctionContext layer so that it retains the old behaviour even though allocations do not all come from FreePools. The "local" allocation concept was not exposed directly in udf.h so this patch also renames them to better reflect that they're used for expr results. Testing: * ran exhaustive and ASAN Change-Id: I4ba5a7542ed90a49a4b5586c040b5985a7d45b61 Reviewed-on: http://gerrit.cloudera.org:8080/8025 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Impala Public Jenkins
63 lines
1.9 KiB
Plaintext
63 lines
1.9 KiB
Plaintext
====
|
|
---- QUERY
|
|
# TODO: IMPALA-3350: Add 'group by' to these tests to exercise different code paths.
|
|
select min(string_col) from functional.alltypes
|
|
---- CATCH
|
|
FunctionContext::Allocate() failed to allocate 1 bytes.
|
|
====
|
|
---- QUERY
|
|
select max(string_col) from functional.alltypes
|
|
---- CATCH
|
|
FunctionContext::Allocate() failed to allocate 1 bytes.
|
|
====
|
|
---- QUERY
|
|
select sample(timestamp_col) from functional.alltypes
|
|
---- CATCH
|
|
FunctionContext::Allocate() failed to allocate 248 bytes.
|
|
====
|
|
---- QUERY
|
|
select group_concat(string_col) from functional.alltypes
|
|
---- CATCH
|
|
FunctionContext::Allocate() failed to allocate 4 bytes.
|
|
====
|
|
---- QUERY
|
|
select extract(year from timestamp_col) from functional.alltypes limit 10
|
|
---- CATCH
|
|
FunctionContextImpl::AllocateForResults() failed to allocate 4 bytes.
|
|
====
|
|
---- QUERY
|
|
select trunc(timestamp_col, 'YEAR') from functional.alltypes limit 10
|
|
---- CATCH
|
|
FunctionContextImpl::AllocateForResults() failed to allocate 4 bytes.
|
|
====
|
|
---- QUERY
|
|
select first_value(string_col) over (partition by month order by year) from functional.alltypes
|
|
---- CATCH
|
|
FunctionContext::Allocate() failed to allocate 1 bytes.
|
|
====
|
|
---- QUERY
|
|
select last_value(string_col) over (partition by month order by year) from functional.alltypes
|
|
---- CATCH
|
|
FunctionContext::Allocate() failed to allocate 1 bytes.
|
|
====
|
|
---- QUERY
|
|
select rand() from functional.alltypes;
|
|
---- CATCH
|
|
FunctionContext::Allocate() failed to allocate 4 bytes.
|
|
====
|
|
---- QUERY
|
|
select case when min(int_col) = 0 then 0 end from functional.alltypes
|
|
---- CATCH
|
|
FunctionContext::Allocate() failed to allocate 16 bytes.
|
|
====
|
|
---- QUERY
|
|
select cast(string_col as char(120)) from functional.alltypes
|
|
---- CATCH
|
|
FunctionContextImpl::AllocateForResults() failed to allocate 120 bytes.
|
|
====
|
|
---- QUERY
|
|
select appx_median(int_col) from functional.alltypes
|
|
---- CATCH
|
|
FunctionContext::Allocate() failed to allocate 248 bytes.
|
|
====
|