mirror of
https://github.com/apache/impala.git
synced 2026-01-02 03:00:32 -05:00
579be1c542
This is the first step to fix issues with large memory allocations. In this patch, the built-in `group_concat` is no longer allowed to allocate arbitraryly large strings and crash impala, but is limited to the upper bound of possible allocations in Impala. This patch does not perform any functional change, but rather avoids unnecessary crashes. However, it changes the parameter type of FindChunk() in MemPool to be a signed 64bit integer. This change allows the mempool to allocate internally memory of more than one 1GB, but the public interface of Allocate() is not changed, so the general limitation remains. The reason for this change is as follows: 1) In a UDF FunctionContext::Reallocate() would allocate slightly more than 512MB from the FreePool. 2) The free pool tries to double this size to alloocate 1GB from the MemPool. 3) The MemPool doubles the size again and overflows the signed 32bit integer in the FindChunk() method. This will then only allocate 1GB instead of the expected 2GB. What happens is that one of the callers expected a larger allocation than actually happened, which will in turn lead to memory corruption as soon as the memory is accessed. Change-Id: I068835dfa0ac8f7538253d9fa5cfc3fb9d352f6a Reviewed-on: http://gerrit.cloudera.org:8080/858 Tested-by: Internal Jenkins Reviewed-by: Dan Hecht <dhecht@cloudera.com>
56 lines
2.8 KiB
Plaintext
56 lines
2.8 KiB
Plaintext
====
|
|
---- QUERY
|
|
-- IMPALA-1619 group_concat() error
|
|
select length(group_concat(l_comment, "!")) from (select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem union all
|
|
select l_comment from tpch_parquet.lineitem) a;
|
|
---- CATCH
|
|
Concatenated string length larger than allowed limit of 1 GB character data.
|
|
===== |