Using Impala through a Proxy for High Availability Load-Balancing Proxy for HA

For most clusters that have multiple users and production availability requirements, you might set up a proxy server to relay requests to and from Impala.

Currently, the Impala statestore mechanism does not include such proxying and load-balancing features. Set up a software package of your choice to perform these functions.

Overview of Proxy Usage and Load Balancing for Impala

Using a load-balancing proxy server for Impala has the following advantages:

  • Applications connect to a single well-known host and port, rather than keeping track of the hosts where the impalad daemon is running.
  • If any host running the impalad daemon becomes unavailable, application connection requests still succeed because you always connect to the proxy server rather than a specific host running the impalad daemon.
  • The coordinator node for each Impala query potentially requires more memory and CPU cycles than the other nodes that process the query. The proxy server can issue queries using round-robin scheduling, so that each connection uses a different coordinator node. This load-balancing technique lets the Impala nodes share this additional work, rather than concentrating it on a single machine.

The following setup steps are a general outline that apply to any load-balancing proxy software:

  1. Download the load-balancing proxy software. It should only need to be installed and configured on a single host. Pick a host other than the DataNodes where impalad is running, because the intention is to protect against the possibility of one or more of these DataNodes becoming unavailable.
  2. Configure the load balancer (typically by editing a configuration file). In particular:

    • Set up a port that the load balancer will listen on to relay Impala requests back and forth.

    • Consider enabling sticky sessions. Where practical, enable this setting so that stateless client applications such as impalad and Hue are not disconnected from long-running queries. Evaluate whether this setting is appropriate for your combination of workload and client applications.

    • For Kerberized clusters, follow the instructions in .

  3. Specify the host and port settings for each Impala node. These are the hosts that the load balancer will choose from when relaying each Impala query. See for when to use port 21000, 21050, or another value depending on what type of connections you are load balancing.

    In particular, if you are using Hue or JDBC-based applications, you typically set up load balancing for both ports 21000 and 21050, because these client applications connect through port 21050 while the impala-shell command connects through port 21000.

  4. Run the load-balancing proxy server, pointing it at the configuration file that you set up.
  5. For any scripts, jobs, or configuration settings for applications that formerly connected to a specific datanode to run Impala SQL statements, change the connection information (such as the -i option in impala-shell) to point to the load balancer instead.
The following sections use the HAProxy software as a representative example of a load balancer that you can use with Impala. Choosing the Load-Balancing Algorithm

Load-balancing software offers a number of algorithms to distribute requests. Each algorithm has its own characteristics that make it suitable in some situations but not others.

leastconn
Connects sessions to the coordinator with the fewest connections, to balance the load evenly. Typically used for workloads consisting of many independent, short-running queries. In configurations with only a few client machines, this setting can avoid having all requests go to only a small set of coordinators.
source affinity
Sessions from the same IP address always go to the same coordinator. A good choice for Impala workloads containing a mix of queries and DDL statements, such as CREATE TABLE and ALTER TABLE. Because the metadata changes from a DDL statement take time to propagate across the cluster, prefer to use source affinity in this case. If necessary, run the DDL and subsequent queries that depend on the results of the DDL through the same session, for example by running impala-shell -f script_file to submit several statements through a single session. An alternative is to set the query option SYNC_DDL=1 to hold back subsequent queries until the results of a DDL operation have propagated throughout the cluster, but that is a relatively expensive setting. Recommended for use with Hue.
sticky
Similar to source affinity. Sessions from the same IP address always go to the same coordinator. The maintenance overhead for the stick tables can cause long-running Hue sessions to disconnect, therefore source affinity is often a better choice.
round-robin
Distributes connections to all coordinator nodes. Typically not recommended for Impala.

You might need to perform benchmarks and load testing to determine which setting is optimal for your use case. If some client applications have special characteristics, such as long-running Hue queries working best with source affinity, you might configure multiple virtual IP addresses with a different load-balancing algorithm for each.

Special Proxy Considerations for Clusters Using Kerberos

In a cluster using Kerberos, applications check host credentials to verify that the host they are connecting to is the same one that is actually processing the request, to prevent man-in-the-middle attacks. To clarify that the load-balancing proxy server is legitimate, perform these extra Kerberos setup steps:

  1. This section assumes you are starting with a Kerberos-enabled cluster. See for instructions for setting up Impala with Kerberos. See for general steps to set up Kerberos.
  2. Choose the host you will use for the proxy server. Based on the Kerberos setup procedure, it should already have an entry impala/proxy_host@realm in its keytab. If not, go back over the initial Kerberos configuration steps for the keytab on each host running the impalad daemon.
  3. Copy the keytab file from the proxy host to all other hosts in the cluster that run the impalad daemon. (For optimal performance, impalad should be running on all DataNodes in the cluster.) Put the keytab file in a secure location on each of these other hosts.
  4. Add an entry impala/actual_hostname@realm to the keytab on each host running the impalad daemon.
  5. For each impalad node, merge the existing keytab with the proxy’s keytab using ktutil, producing a new keytab file. For example: $ ktutil ktutil: read_kt proxy.keytab ktutil: read_kt impala.keytab ktutil: write_kt proxy_impala.keytab ktutil: quit
  6. To verify that the keytabs are merged, run the command: klist -k keytabfile which lists the credentials for both principal and be_principal on all nodes.
  7. Make sure that the impala user has permission to read this merged keytab file.
  8. Change the following configuration settings for each host in the cluster that participates in the load balancing:
    • In the impalad option definition, add: --principal=impala/proxy_host@realm --be_principal=impala/actual_host@realm --keytab_file=path_to_merged_keytab Every host has different --be_principal because the actual hostname is different on each host. Specify the fully qualified domain name (FQDN) for the proxy host, not the IP address. Use the exact FQDN as returned by a reverse DNS lookup for the associated IP address.
    • Modify the startup options. See for the procedure to modify the startup options.
  9. Restart Impala to make the changes take effect. Restart the impalad daemons on all hosts in the cluster, as well as the statestored and catalogd daemons.
Example of Configuring HAProxy Load Balancer for Impala

If you are not already using a load-balancing proxy, you can experiment with HAProxy a free, open source load balancer. This example shows how you might install and configure that load balancer on a Red Hat Enterprise Linux system.

This is the sample haproxy.cfg used in this example:

global # To have these messages end up in /var/log/haproxy.log you will # need to: # # 1) configure syslog to accept network log events. This is done # by adding the '-r' option to the SYSLOGD_OPTIONS in # /etc/sysconfig/syslog # # 2) configure local2 events to go to the /var/log/haproxy.log # file. A line like the following can be added to # /etc/sysconfig/syslog # # local2.* /var/log/haproxy.log # log 127.0.0.1 local0 log 127.0.0.1 local1 notice chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon # turn on stats unix socket #stats socket /var/lib/haproxy/stats #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block # # You might need to adjust timing values to prevent timeouts. #--------------------------------------------------------------------- defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 maxconn 3000 contimeout 5000 clitimeout 50000 srvtimeout 50000 # # This sets up the admin page for HA Proxy at port 25002. # listen stats :25002 balance mode http stats enable stats auth username:password # This is the setup for Impala. Impala client connect to load_balancer_host:25003. # HAProxy will balance connections among the list of servers listed below. # The list of Impalad is listening at port 21000 for beeswax (impala-shell) or original ODBC driver. # For JDBC or ODBC version 2.x driver, use port 21050 instead of 21000. listen impala :25003 mode tcp option tcplog balance leastconn server symbolic_name_1 impala-host-1.example.com:21000 server symbolic_name_2 impala-host-2.example.com:21000 server symbolic_name_3 impala-host-3.example.com:21000 server symbolic_name_4 impala-host-4.example.com:21000 # Setup for Hue or other JDBC-enabled applications. # In particular, Hue requires sticky sessions. # The application connects to load_balancer_host:21051, and HAProxy balances # connections to the associated hosts, where Impala listens for JDBC # requests on port 21050. listen impalajdbc :21051 mode tcp option tcplog balance source server symbolic_name_5 impala-host-1.example.com:21050 server symbolic_name_6 impala-host-2.example.com:21050 server symbolic_name_7 impala-host-3.example.com:21050 server symbolic_name_8 impala-host-4.example.com:21050