mirror of
https://github.com/apache/impala.git
synced 2025-12-19 18:12:08 -05:00
ea6173440c
THRIFT-5237 Implement MAX_MESSAGE_SIZE and consolidate limits into a TConfiguration class. The MAX_MESSAGE_SIZE default to 100MB. This patch adds backend flag 'thrift_rpc_max_message_size' to override the default MAX_MESSAGE_SIZE by calling function AssignDefaultTConfiguration. We set higher default, 1GB, to minimize interruption on existing Impala workload. We should consider lowering this default once we ensure that all thrift rpc response can be made in batches, such as explained in IMPALA-11402. Thrift tuning for communication with HMS is fixed through HIVE-26633. Appropriate Hive version bump is required. Testing: - Add EXPECT_NO_THROW to verify that 'checkReadBytesAvailable' to does not throw exception given default FLAGS_thrift_rpc_max_message_size. - Add MaxMessageSizeFit and MaxMessageSizeExceeded tests in thrift-server-test. - Run and pass thrift-server-test Change-Id: I137683d43c72a34105fd7b32fea3a93532601ae3 Reviewed-on: http://gerrit.cloudera.org:8080/19162 Reviewed-by: Wenzhe Zhou <wzhou@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
45 lines
1.6 KiB
Bash
Executable File
45 lines
1.6 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
#
|
|
# Licensed to the Apache Software Foundation (ASF) under one
|
|
# or more contributor license agreements. See the NOTICE file
|
|
# distributed with this work for additional information
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
# to you under the Apache License, Version 2.0 (the
|
|
# "License"); you may not use this file except in compliance
|
|
# with the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing,
|
|
# software distributed under the License is distributed on an
|
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
# KIND, either express or implied. See the License for the
|
|
# specific language governing permissions and limitations
|
|
# under the License.
|
|
|
|
# This script load 150k partitions into /test-warehouse/1k_col_tbl
|
|
# with 3 long partition keys, each 250 chars.
|
|
|
|
mkdir batch-1
|
|
cd batch-1
|
|
|
|
# Create data for "p1=0000....0001" in local file system
|
|
p1=$(printf '%*s\n' 250 "1" | tr ' ' '0')
|
|
for j in {1..1000}; do
|
|
z=$(printf '%*s\n' 250 "$j" | tr ' ' '0')
|
|
ppath="p1=$p1/p2=$z/p3=$z"
|
|
delims=$(printf '%*s\n' 1000 "" | tr ' ' '|')
|
|
mkdir -p "$ppath"; echo "${j}${delims}" > "$ppath/data.txt";
|
|
done
|
|
|
|
# Put data for "p1=0000....0001" to HDFS.
|
|
hdfs dfs -put -f "p1=$p1" /test-warehouse/1k_col_tbl
|
|
|
|
# Create the other partitions by copying from "p1=0000....0001"
|
|
for i in {2..150}; do
|
|
pn=$(printf '%*s\n' 250 "$i" | tr ' ' '0')
|
|
echo "loading p1=$pn"
|
|
hdfs dfs -cp "/test-warehouse/1k_col_tbl/p1=$p1" "/test-warehouse/1k_col_tbl/p1=$pn"
|
|
done
|
|
|