Tim Armstrong cf224f8461 IMPALA-9128: part 2: dump traces for slow RPCs ...

This adds trace events for data stream RPCs and
dumps them when they take longer than
--impala_slow_rpc_threshold_ms.

I needed to modify the KRPC code to do this because it
currently only dumps traces for RPCs with deadlines.
I plan to add some version of this upstream in Kudu
so that we don't diverge our KRPC implementation.

Example output from test_exchange_small_buffer:

I1111 08:38:53.732910 26509 rpcz_store.cc:265] Call impala.DataStreamService.TransmitData from 127.0.0.1:42434 (request call id 43) took 7799ms. Request Metrics: {}
I1111 08:38:53.732928 26509 rpcz_store.cc:269] Trace:
1111 08:38:45.933412 (+     0us) impala-service-pool.cc:167] Inserting onto call queue
1111 08:38:45.933449 (+    37us) impala-service-pool.cc:254] Handling call
1111 08:38:45.933470 (+    21us) krpc-data-stream-mgr.cc:227] Added early sender
1111 08:38:47.906542 (+1973072us) krpc-data-stream-recvr.cc:327] Enqueuing deferred RPC
1111 08:38:53.732858 (+5826316us) krpc-data-stream-recvr.cc:506] Processing deferred RPC
1111 08:38:53.732860 (+     2us) krpc-data-stream-recvr.cc:399] Deserializing batch
1111 08:38:53.732888 (+    28us) krpc-data-stream-recvr.cc:426] Enqueuing deserialized batch
1111 08:38:53.732895 (+     7us) inbound_call.cc:162] Queueing success response

Disabled +-clang-diagnostic-gnu-zero-variadic-macro-arguments because it
had false positives on the TRACE_TO invocations.

Testing:
* Ran exhaustive and ASAN tests
* Ran stress test

Change-Id: Ic7af4b45c43ec731d742d3696112c5f800849947
Reviewed-on: http://gerrit.cloudera.org:8080/14668
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>

2019-11-14 20:24:58 +00:00

3.6 KiB

Raw Blame History

View Raw