jprdonnelly/impala
1
0
Fork 0
mirror of https://github.com/apache/impala.git synced 2026-02-03 00:00:40 -05:00
Code Issues Projects Releases Wiki Activity

IMPALA-6684: Fix untracked memory in KRPC

Browse Source 
During serialization of a row batch header, a tuple_data_ is created
which will hold the compressed tuple data for an outbound row batch.
We would like this tuple data to be trackable as it is responsible for
a significant portion of untrackable memory from the krpc data stream
sender. By using MemTrackerAllocator, we can allocate tuple data and
compression scratch and account for it in the memory tracker of the
KrpcDataStreamSender. This solution replaces the type for tuple data
and compression scratch from std::string to TrackedString, an
std:basic_string with MemTrackerAllocator as the custom allocator.

This patch adds memory estimation in DataStreamSink.java to account
for OutboundRowBatch memory allocation. This patch also removes the
thrift-based serialization because the thrift RPC has been removed
in the prior commit.

Testing:
 - Passed core tests.
 - Ran a single node benchmark which shows no regression.
 - Updated row-batch-serialize-test and row-batch-serialize-benchmark
   to test the row-batch serialization used by KRPC.
 - Manually collected query-profile, heap growth, and memory usage log
   showing untracked memory decreased by 1/2.
 - Add test_datastream_sender.py to verify the peak memory of EXCHANGE
   SENDER node.
 - Raise mem_limit in two of test_spilling_large_rows test case.
 - Print test line number in PlannerTestBase.java

New row-batch serialization benchmark:

Machine Info: Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
serialize:            10%   50%   90%     10%     50%     90%
                                        (rel)   (rel)   (rel)
-------------------------------------------------------------
   ser_no_dups_base  18.6  18.8  18.9      1X      1X      1X
        ser_no_dups  18.5  18.5  18.8  0.998X  0.988X  0.991X
   ser_no_dups_full  14.7  14.8  14.8  0.793X   0.79X  0.783X

  ser_adj_dups_base  28.2  28.4  28.8      1X      1X      1X
       ser_adj_dups  68.9  69.1  69.8   2.44X   2.43X   2.43X
  ser_adj_dups_full  56.2  56.7  57.1   1.99X      2X   1.99X

      ser_dups_base  20.7  20.9  20.9      1X      1X      1X
           ser_dups  20.6  20.8  20.9  0.994X  0.995X      1X
      ser_dups_full  39.8    40  40.5   1.93X   1.92X   1.94X

deserialize:          10%   50%   90%     10%     50%     90%
                                        (rel)   (rel)   (rel)
-------------------------------------------------------------
 deser_no_dups_base  75.9  76.6    77      1X      1X      1X
      deser_no_dups  74.9  75.6    76  0.987X  0.987X  0.987X

deser_adj_dups_base   127   128   129      1X      1X      1X
     deser_adj_dups   179   193   195   1.41X   1.51X   1.51X

    deser_dups_base   128   128   129      1X      1X      1X
         deser_dups   165   190   193   1.29X   1.48X   1.49X

Change-Id: I2ba2b907ce4f275a7a1fb8cf75453c7003eb4b82
Reviewed-on: http://gerrit.cloudera.org:8080/18798
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This commit is contained in:
TheOmid
2022-07-26 12:03:35 -07:00
committed by Impala Public Jenkins
parent dc912f016e
commit e327a28757
124 changed files with 2870 additions and 2743 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
testdata/workloads/functional-query/queries/QueryTest/dedicated-coord-mem-estimates.test vendored
View File

@@ -39,9 +39,9 @@ row_regex: .*Cluster Memory Admitted: 169.47 MB.*
# SELECT with non-merging exchange.
select * from functional.alltypes;
---- RUNTIME_PROFILE
row_regex: .*Per-Host Resource Estimates: Memory=20MB.*
row_regex: .*Per-Host Resource Estimates: Memory=21MB.*
row_regex: .*Dedicated Coordinator Resource Estimate: Memory=104MB.*
row_regex: .*Cluster Memory Admitted: 145.47 MB.*
row_regex: .*Cluster Memory Admitted: 146.20 MB.*
====
---- QUERY
# SELECT with a non-grouping aggregate in the coordinator fragment.

6
testdata/workloads/functional-query/queries/QueryTest/explain-level2.test vendored
View File

@@ -67,7 +67,7 @@ from functional_avro.alltypes t1
left outer join functional_avro.alltypes t3 on (t2.id = t3.id)
where t1.month = 1 and t2.year = 2009 and t3.bool_col = false
---- RESULTS: VERIFY_IS_SUBSET
'Per-Host Resource Estimates: Memory=54MB'
'Per-Host Resource Estimates: Memory=55MB'
'WARNING: The following tables are missing relevant table and/or column statistics.'
'functional_avro.alltypes, functional_parquet.alltypessmall'
====
@@ -91,6 +91,6 @@ explain select year, month from functional.alltypes
intersect
select year, month from functional.alltypes where year=2009;
---- RESULTS: VERIFY_IS_SUBSET
'Per-Host Resources: mem-estimate=13.97MB mem-reservation=5.88MB thread-reservation=1 runtime-filters-memory=2.00MB'
'Per-Host Resources: mem-estimate=13.98MB mem-reservation=5.88MB thread-reservation=1 runtime-filters-memory=2.00MB'
'04:HASH JOIN [LEFT SEMI JOIN, PARTITIONED]'
====
====

3
testdata/workloads/functional-query/queries/QueryTest/spilling-large-rows.test vendored
View File

@@ -21,6 +21,7 @@ Row of size 9.54 MB could not be materialized by AGGREGATION_NODE
====
---- QUERY
# Agg should be able to process the large strings if we increase the row size.
set mem_limit="1gb";
set max_row_size=10m;
select id, count(distinct bigstr)
from bigstrs
@@ -152,7 +153,7 @@ Row of size 9.54 MB could not be materialized by SORT_NODE (id=1). Increase the
---- QUERY
# Sort and analytic should be able to process the large strings if we increase the row
# size.
set mem_limit="1gb";
set mem_limit="1.4gb";
set max_row_size=10m;
SELECT id, int_col, substring(bigstr, 1, 10), substring(bigstr, 9999999, 1), rank
FROM (