mirror of
https://github.com/apache/impala.git
synced 2026-02-03 00:00:40 -05:00
9b4a3ef795
Enables support for caching remote file handles for Ozone. Local file handles were already cached unintentionally, similar to HDFS. Updates file handle cache enablement to be more stringent about enabling caching. File handle caching is enabled if a max_cached_file_handles is non-zero and any of the following are true - HDFS file is local - HDFS file is remote and cache_remote_file_handles is enabled - Ozone file is local or remote and cache_ozone_file_handles is enabled - S3 file is remote and cache_s3_file_handles is enabled - ABFS file is remote and cache_abfs_file_handles is enabled Enables testing Ozone in test_hdfs_fd_caching, and adds tests that remote caching can be disabled using individual flags. Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9 Reviewed-on: http://gerrit.cloudera.org:8080/18853 Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com> Reviewed-by: Csaba Ringhofer <csringhofer@cloudera.com>
32 lines
1.3 KiB
Python
32 lines
1.3 KiB
Python
# Licensed to the Apache Software Foundation (ASF) under one
|
|
# or more contributor license agreements. See the NOTICE file
|
|
# distributed with this work for additional information
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
# to you under the Apache License, Version 2.0 (the
|
|
# "License"); you may not use this file except in compliance
|
|
# with the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing,
|
|
# software distributed under the License is distributed on an
|
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
# KIND, either express or implied. See the License for the
|
|
# specific language governing permissions and limitations
|
|
# under the License.
|
|
|
|
# Tools for identifying network characteristics.
|
|
|
|
import socket
|
|
|
|
|
|
# Retrieves the host external IP rather than localhost/127.0.0.1 so we have an IP that
|
|
# Impala will consider distinct from storage backends to force remote scheduling.
|
|
def get_external_ip():
|
|
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
|
|
s.settimeout(0)
|
|
# This address is used to get the networking stack to identify a return IP address.
|
|
# Timeout=0 means it doesn't need to resolve.
|
|
s.connect(('10.254.254.254', 1))
|
|
return s.getsockname()[0]
|