mirror of
https://github.com/apache/impala.git
synced 2025-12-19 18:12:08 -05:00
5391100c7e
This change converts ReportExecStatus() RPC from thrift based RPC to KRPC. This is done in part of the preparation for fixing IMPALA-2990 as we can take advantage of TCP connection multiplexing in KRPC to avoid overwhelming the coordinator with too many connections by reducing the number of TCP connection to one for each executor. This patch also introduces a new service pool for all query execution control related RPCs in the future so that control commands from coordinators aren't blocked by long-running DataStream services' RPCs. To avoid unnecessary delays due to sharing the network connections between DataStream service and Control service, this change added the service name as part of the user credentials for the ConnectionId so each service will use a separate connection. The majority of this patch is mechanical conversion of some Thrift structures used in ReportExecStatus() RPC to Protobuf. Note that the runtime profile is still retained as a Thrift structure as Impala clients will still fetch query profiles using Thrift RPCs. This also avoids duplicating the serialization implementation in both Thrift and Protobuf for the runtime profile. The Thrift runtime profiles are serialized and sent as a sidecar in ReportExecStatus() RPC. This patch also fixes IMPALA-7241 which may lead to duplicated dml stats being applied. The fix is by adding a monotonically increasing version number for fragment instances' reports. The coordinator will ignore any report smaller than or equal to the version in the last report. Testing done: 1. Exhaustive build. 2. Added some targeted test cases for profile serialization failure and RPC retries/timeout. Change-Id: I7638583b433dcac066b87198e448743d90415ebe Reviewed-on: http://gerrit.cloudera.org:8080/10855 Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
52 lines
1.4 KiB
Protocol Buffer
52 lines
1.4 KiB
Protocol Buffer
// Licensed to the Apache Software Foundation (ASF) under one
|
|
// or more contributor license agreements. See the NOTICE file
|
|
// distributed with this work for additional information
|
|
// regarding copyright ownership. The ASF licenses this file
|
|
// to you under the Apache License, Version 2.0 (the
|
|
// "License"); you may not use this file except in compliance
|
|
// with the License. You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing,
|
|
// software distributed under the License is distributed on an
|
|
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
// KIND, either express or implied. See the License for the
|
|
// specific language governing permissions and limitations
|
|
// under the License.
|
|
//
|
|
|
|
syntax="proto2";
|
|
|
|
package impala;
|
|
|
|
import "kudu/rpc/rpc_header.proto";
|
|
|
|
// Definitions for service used for rpc-mgr-test.
|
|
message PingRequestPB {
|
|
}
|
|
|
|
message PingResponsePB {
|
|
required int32 int_response = 1;
|
|
}
|
|
|
|
service PingService {
|
|
option (kudu.rpc.default_authz_method) = "Authorize";
|
|
|
|
rpc Ping(PingRequestPB) returns (PingResponsePB);
|
|
}
|
|
|
|
message ScanMemRequestPB {
|
|
required int32 pattern = 1;
|
|
required int32 sidecar_idx = 2;
|
|
}
|
|
|
|
message ScanMemResponsePB {
|
|
}
|
|
|
|
service ScanMemService {
|
|
option (kudu.rpc.default_authz_method) = "Authorize";
|
|
|
|
rpc ScanMem(ScanMemRequestPB) returns (ScanMemResponsePB);
|
|
}
|