jprdonnelly/impala
1
0
Fork 0
mirror of https://github.com/apache/impala.git synced 2025-12-25 02:03:09 -05:00
Code Issues Projects Releases Wiki Activity

IMPALA-8825: Add additional counters to PlanRootSink

Browse Source 
Adds the counters RowsSent and RowsSentRate to the PLAN_ROOT_SINK
section of the profile:

  PLAN_ROOT_SINK:
     - PeakMemoryUsage: 4.01 MB (4202496)
     - RowBatchGetWaitTime: 0.000ns
     - RowBatchSendWaitTime: 0.000ns
     - RowsSent: 10 (10)
     - RowsSentRate: 416.00 /sec

RowsSent tracks the number of rows sent to the PlanRootSink via
PlanRootSink::Send. RowsSentRate tracks the rate that rows are sent to
the PlanRootSink.

Adds the counters NumRowsFetched, NumRowsFetchedFromCache, and
RowMaterializationRate to the ImpalaServer section of the profile.

  ImpalaServer:
     - ClientFetchWaitTimer: 11.999ms
     - NumRowsFetched: 10 (10)
     - NumRowsFetchedFromCache: 10 (10)
     - RowMaterializationRate: 9.00 /sec
     - RowMaterializationTimer: 1s007ms

NumRowsFetched tracks the total number of rows fetched by the query,
but does not include rows fetched from the cache. NumRowsFetchedFromCache
tracks the total number of rows fetched from the query results cache.
RowMaterializationRate tracks the rate at which rows are materialized.
RowMaterializationTimer already existed and tracks how much time is
spent materializing rows.

Testing:
* Added tests to test_fetch_first.py and query_test/test_fetch.py
* Enabled some tests in test_fetch_first.py that were pending
the completion of IMPALA-8819
* Ran core tests

Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036
Reviewed-on: http://gerrit.cloudera.org:8080/14180
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Sahil Takiar <stakiar@cloudera.com>
This commit is contained in:
Sahil Takiar
2019-09-04 10:11:15 -07:00
committed by Sahil Takiar
parent 77aad0405f
commit 34d132c513
10 changed files with 199 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
tests/custom_cluster/test_result_spooling.py
View File

@@ -16,6 +16,7 @@
# under the License.
import pytest
import re
from tests.common.custom_cluster_test_suite import CustomClusterTestSuite
from tests.common.test_dimensions import (
@@ -24,7 +25,6 @@ from tests.common.test_dimensions import (
from tests.util.web_pages_util import (
get_num_completed_backends,
get_mem_admitted_backends_debug_page)
from time import sleep
class TestDedicatedCoordinator(CustomClusterTestSuite):
@@ -49,7 +49,9 @@ class TestDedicatedCoordinator(CustomClusterTestSuite):
"""Test the following when result spooling is enabled on a cluster with a dedicated
coordinator when all results are spooled: (1) all backends are shutdown besides the
coordinator and (2) all non-coordinator memory is released."""
query = "select id from functional_parquet.alltypes order by id limit 2000"
num_rows = 2000
query = "select id from functional_parquet.alltypes order by id limit {0}".format(
num_rows)
vector.get_value('exec_option')['spool_query_results'] = 'true'
# Amount of time to wait for the query to reach the finished state before throwing a
@@ -61,10 +63,9 @@ class TestDedicatedCoordinator(CustomClusterTestSuite):
# Wait for the query to finish (all rows are spooled). Assert that the executor
# has been shutdown and its memory has been released.
self.wait_for_state(handle, self.client.QUERY_STATES['FINISHED'], timeout)
# Since FINISHED does not necessarily mean all results are spooled, sleep for
# enough time to allow for all results to be spooled. This can be better enforced
# once the metrics in IMPALA-8825 have been added.
sleep(1)
self.assert_eventually(timeout, 0.5,
lambda: re.search("RowsSent:.*({0})".format(num_rows),
self.client.get_runtime_profile(handle)))
assert "NumCompletedBackends: 1 (1)" in self.client.get_runtime_profile(handle)
mem_admitted = get_mem_admitted_backends_debug_page(self.cluster)
assert mem_admitted['executor'][0] == 0