Files
impala/docker/quickstart-kudu-minimal.yml
Tim Armstrong eb85c6eeca IMPALA-9793: Impala quickstart cluster with docker-compose
What works:
* A single node cluster can be started up with docker-compose
* HMS data is stored in Derby database in a docker volume
* Filesystem data is stored in a shared docker volume, using the
  localfs support in the Hadoop client.
* A Kudu cluster with a single master can be optionally added on
  to the Impala cluster.
* TPC-DS data can be loaded automatically by a data loading container.

We need to set up a docker network called quickstart-network,
purely because docker-compose insists on generating network names
with underscores, which are part of the FQDN and end up causing
problems with Java's URL parsing, which rejects these technically
invalid domain names.

How to run:

Instructions for running the quickstart cluster are in
docker/README.md.

How to build containers:

  ./buildall.sh -release -noclean -notests -ninja
  ninja quickstart_hms_image quickstart_client_image docker_images

How to upload containers to dockerhub:

  IMPALA_QUICKSTART_IMAGE_PREFIX=timgarmstrong/
  for i in impalad_coord_exec impalad_coordinator statestored \
           impalad_executor catalogd impala_quickstart_client \
           impala_quickstart_hms
  do
    docker tag $i ${IMPALA_QUICKSTART_IMAGE_PREFIX}$i
    docker push ${IMPALA_QUICKSTART_IMAGE_PREFIX}$i
  done

I pushed containers build from commit f260cce22, which
was branched from 6cb7cecacf on master.

Misc other stuff:
* Added more metadata to all images.

TODO:
* Test and instructions to run against Kudu quickstart
* Upload latest version of containers before merging.

Change-Id: Ifc0b862af40a368381ada7ec2a355fe4b0aa778c
Reviewed-on: http://gerrit.cloudera.org:8080/15966
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
2021-01-26 11:22:08 +00:00

129 lines
4.9 KiB
YAML

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# Minimal Apache Kudu cluster for quickstart with a single master and three tservers.
# The Kudu repository contains a docker compose file (docker/quickstart.yml) for a more
# production-like Kudu cluster with multiple masters, etc. This file is derived from that
# file.
version: "3"
services:
kudu-master-1:
image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest}
ports:
- "${QUICKSTART_LISTEN_ADDR:?Please set QUICKSTART_LISTEN_ADDR environment variable}:7051:7051"
- "${QUICKSTART_LISTEN_ADDR:?Please set QUICKSTART_LISTEN_ADDR environment variable}:8051:8051"
command: ["master"]
volumes:
- kudu-minimal-master-1:/var/lib/kudu
environment:
# TODO: Use `host.docker.internal` instead of QUICKSTART_IP when it
# works on Linux (https://github.com/docker/for-linux/issues/264)
- >
MASTER_ARGS=--fs_wal_dir=/var/lib/kudu/master
--rpc_bind_addresses=0.0.0.0:7051
--rpc_advertised_addresses=${QUICKSTART_IP:?Please set QUICKSTART_IP environment variable}:7051
--webserver_port=8051
--webserver_advertised_addresses=${QUICKSTART_IP}:8051
--webserver_doc_root=/opt/kudu/www
--stderrthreshold=0
--use_hybrid_clock=false
--memory_limit_hard_bytes=1073741824
networks:
- quickstart-network
kudu-tserver-1:
image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest}
depends_on:
- kudu-master-1
ports:
- "${QUICKSTART_LISTEN_ADDR:?Please set QUICKSTART_LISTEN_ADDR environment variable}:7050:7050"
- "${QUICKSTART_LISTEN_ADDR:?Please set QUICKSTART_LISTEN_ADDR environment variable}:8050:8050"
command: ["tserver"]
volumes:
- kudu-minimal-tserver-1:/var/lib/kudu
environment:
- KUDU_MASTERS=kudu-master-1:7051
- >
TSERVER_ARGS=--fs_wal_dir=/var/lib/kudu/tserver
--rpc_bind_addresses=0.0.0.0:7050
--rpc_advertised_addresses=${QUICKSTART_IP}:7050
--webserver_port=8050
--webserver_advertised_addresses=${QUICKSTART_IP}:8050
--webserver_doc_root=/opt/kudu/www
--stderrthreshold=0
--use_hybrid_clock=false
--memory_limit_hard_bytes=1073741824
networks:
- quickstart-network
kudu-tserver-2:
image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest}
depends_on:
- kudu-master-1
ports:
- "${QUICKSTART_LISTEN_ADDR:?Please set QUICKSTART_LISTEN_ADDR environment variable}:7150:7150"
- "${QUICKSTART_LISTEN_ADDR:?Please set QUICKSTART_LISTEN_ADDR environment variable}:8150:8150"
command: ["tserver"]
volumes:
- kudu-minimal-tserver-2:/var/lib/kudu
environment:
- KUDU_MASTERS=kudu-master-1:7051
- >
TSERVER_ARGS=--fs_wal_dir=/var/lib/kudu/tserver
--rpc_bind_addresses=0.0.0.0:7150
--rpc_advertised_addresses=${QUICKSTART_IP}:7150
--webserver_port=8150
--webserver_advertised_addresses=${QUICKSTART_IP}:8150
--webserver_doc_root=/opt/kudu/www
--stderrthreshold=0
--use_hybrid_clock=false
--memory_limit_hard_bytes=1073741824
networks:
- quickstart-network
kudu-tserver-3:
image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest}
depends_on:
- kudu-master-1
ports:
- "${QUICKSTART_LISTEN_ADDR:?Please set QUICKSTART_LISTEN_ADDR environment variable}:7250:7250"
- "${QUICKSTART_LISTEN_ADDR:?Please set QUICKSTART_LISTEN_ADDR environment variable}:8250:8250"
command: ["tserver"]
volumes:
- kudu-minimal-tserver-3:/var/lib/kudu
environment:
- KUDU_MASTERS=kudu-master-1:7051
- >
TSERVER_ARGS=--fs_wal_dir=/var/lib/kudu/tserver
--rpc_bind_addresses=0.0.0.0:7250
--rpc_advertised_addresses=${QUICKSTART_IP}:7250
--webserver_port=8250
--webserver_advertised_addresses=${QUICKSTART_IP}:8250
--webserver_doc_root=/opt/kudu/www
--stderrthreshold=0
--use_hybrid_clock=false
--memory_limit_hard_bytes=1073741824
networks:
- quickstart-network
volumes:
kudu-minimal-master-1:
kudu-minimal-tserver-1:
kudu-minimal-tserver-2:
kudu-minimal-tserver-3:
networks:
quickstart-network:
external: true