fifteencai a0a25a61c3 IMPALA-10193: Limit the memory usage for the whole test cluster ...

This patch introduces a new approach of limiting the memory usage
for both mini-cluster and CDH cluster.

Without this limit, clusters are prone to getting killed when running
in docker containers with a lower mem limit than host's memory size.
i.e. The mini-cluster may running in a
container with 32GB limitted by CGROUPS, while the host machine has
128GB. Under this circumstance, if the container is started with
'-privileged' command argument, both mini and CDH clusters compute
their mem_limit according to 128GB rather than 32GB. They will be
killed when attempting to apply for extra resource.

Currently, the mem-limit estimating algorithms for Impalad and Node
Manager are different:

for Impalad:  mem_limit = 0.7 * sys_mem / cluster_size (default is 3)

for Node Manager:
        1. Leave aside 24GB, then fit the left into threasholds below.
        2. The bare limit is 4GB and maximum limit 48GB

In headge of over-consumption, we

- Added a new environment variable IMPALA_CLUSTER_MAX_MEM_GB
- Modified the algorithm in 'bin/start-impala-cluster.py', making it
  taking IMPALA_CLUSTER_MAX_MEM_GB rather than sys_mem into account.
- Modified the logic in
 'testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py'
  Similarly, making IMPALA_CLUSTER_MAX_MEM_GB substitutes for sys_mem .

Testing: this patch worked in a 32GB docker container running on a 128GB
         host machine. All 1188 unit tests get passed.

Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93
Reviewed-on: http://gerrit.cloudera.org:8080/16522
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>

2020-10-01 08:38:18 +00:00

38 KiB

Executable File

Raw Blame History

View Raw