mirror of
https://github.com/apache/impala.git
synced 2025-12-25 02:03:09 -05:00
c233634d74
Python 3 made the main dictionary methods lazy (items(), keys(), values()). This means that code that uses those methods may need to wrap the call in list() to get a list immediately. Python 3 also removed the old iter* lazy variants. This changes all locations to use Python 3 dictionary methods and wraps calls with list() appropriately. This also changes all itemitems(), itervalues(), iterkeys() locations to items(), values(), keys(), etc. Python 2 will not use the lazy implementation of these, so there is a theoretical performance impact. Our python code is mostly for tests and the performance impact is minimal. Python 2 will be deprecated when Python 3 is functional. This addresses these pylint warnings: dict-iter-method dict-keys-not-iterating dict-values-not-iterating Testing: - Ran core tests Change-Id: Ie873ece54a633a8a95ed4600b1df4be7542348da Reviewed-on: http://gerrit.cloudera.org:8080/19590 Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com> Tested-by: Joe McDonnell <joemcdonnell@cloudera.com>
51 lines
1.9 KiB
Python
Executable File
51 lines
1.9 KiB
Python
Executable File
#!/usr/bin/env impala-python
|
|
#
|
|
# Licensed to the Apache Software Foundation (ASF) under one
|
|
# or more contributor license agreements. See the NOTICE file
|
|
# distributed with this work for additional information
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
# to you under the Apache License, Version 2.0 (the
|
|
# "License"); you may not use this file except in compliance
|
|
# with the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing,
|
|
# software distributed under the License is distributed on an
|
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
# KIND, either express or implied. See the License for the
|
|
# specific language governing permissions and limitations
|
|
# under the License.
|
|
#
|
|
# Used to extract minimum memory data for test_mem_usage_scaling.py from stress test
|
|
# runtime info json.
|
|
#
|
|
# Usage
|
|
# =====
|
|
# Run the stress test binary search on the 3 node minicluster:
|
|
#
|
|
# ./tests/stress/concurrent_select.py --tpch-db=tpch_parquet \
|
|
# --runtime-info-path=mem_usage_scaling_runtime_info.json --samples 3 \
|
|
# --mem_limit_eq_threshold_percent=0.01 --mem_limit_eq_threshold_mb=5 \
|
|
# --common-query-options="default_spillable_buffer_size=256k"
|
|
#
|
|
# Then run this script to extract minimum memory:
|
|
#
|
|
# ./tests/stress/extract_min_mem.py mem_usage_scaling_runtime_info.json
|
|
#
|
|
from __future__ import absolute_import, division, print_function
|
|
import json
|
|
import sys
|
|
|
|
results = []
|
|
with open(sys.argv[1]) as f:
|
|
data = json.load(f)
|
|
for query_data in data['db_names']['tpch_parquet'].values():
|
|
runtime_info = query_data['[]']
|
|
# Build up list of query numbers and minimum memory.
|
|
results.append((int(runtime_info['name'][1:]),
|
|
runtime_info['required_mem_mb_with_spilling']))
|
|
|
|
results.sort()
|
|
print(', '.join(["'Q{0}': {1}".format(num, mem) for num, mem in results]))
|