Using Impala Logging Logging

The Impala logs record information about:

Formerly, the logs contained the query profile for each query, showing low-level details of how the work is distributed among nodes and how intermediate and final results are transmitted across the network. To save space, those query profiles are now stored in zlib-compressed files in /var/log/impala/profiles. You can access them through the Impala web user interface. For example, at http://impalad-node-hostname:25000/queries, each query is followed by a Profile link leading to a page showing extensive analytical data for the query execution.

The auditing feature introduced in Impala 1.1.1 produces a separate set of audit log files when enabled. See for details.

In and higher, you can control how many audit event log files are kept on each host through the --max_audit_event_log_files startup option for the impalad daemon, similar to the --max_log_files option for regular log files.

The lineage feature introduced in Impala 2.2.0 produces a separate lineage log file when enabled. See for details.

Locations and Names of Impala Log Files

Impala stores information using the glog_v logging system. You will see some messages referring to C++ file names. Logging is affected by:

Managing Impala Logs

Review Impala log files on each host, when you have traced an issue back to a specific system.

Rotating Impala Logs

Impala periodically switches the physical files representing the current log files, after which it is safe to remove the old files if they are no longer needed.

Impala can automatically remove older unneeded log files, a feature known as log rotation.

In Impala 2.2 and higher, the --max_log_files configuration option specifies how many log files to keep at each severity level. You can specify an appropriate setting for each Impala-related daemon (impalad, statestored, and catalogd). The default value is 10, meaning that Impala preserves the latest 10 log files for each severity level (INFO, WARNING, ERROR, and FATAL). Impala checks to see if any old logs need to be removed based on the interval specified in the logbufsecs setting, every 5 seconds by default.

A value of 0 preserves all log files, in which case you would set up set up manual log rotation using your Linux tool or technique of choice. A value of 1 preserves only the very latest log file.

Reviewing Impala Logs

By default, the Impala log is stored at /var/logs/impalad/. The most comprehensive log, showing informational, warning, and error messages, is in the file name impalad.INFO. View log file contents by using the web interface or by examining the contents of the log file. (When you examine the logs through the file system, you can troubleshoot problems by reading the impalad.WARNING and/or impalad.ERROR files, which contain the subsets of messages indicating potential problems.)

On a machine named impala.example.com with default settings, you could view the Impala logs on that machine by using a browser to access http://impala.example.com:25000/logs.

The web interface limits the amount of logging information displayed. To view every log entry, access the log files directly through the file system.

You can view the contents of the impalad.INFO log file in the file system. With the default configuration settings, the start of the log file appears as follows:

[user@example impalad]$ pwd /var/log/impalad [user@example impalad]$ more impalad.INFO Log file created at: 2013/01/07 08:42:12 Running on machine: impala.example.com Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg I0107 08:42:12.292155 14876 daemon.cc:34] impalad version 0.4 RELEASE (build 9d7fadca0461ab40b9e9df8cdb47107ec6b27cff) Built on Fri, 21 Dec 2012 12:55:19 PST I0107 08:42:12.292484 14876 daemon.cc:35] Using hostname: impala.example.com I0107 08:42:12.292706 14876 logging.cc:76] Flags (see also /varz are on debug webserver): --dump_ir=false --module_output= --be_port=22000 --classpath= --hostname=impala.example.com The preceding example shows only a small part of the log file. Impala log files are often several megabytes in size. Understanding Impala Log Contents

The logs store information about Impala startup options. This information appears once for each time Impala is started and may include:

There is information about each job Impala has run. Because each Impala job creates an additional set of data about queries, the amount of job specific data may be very large. Logs may contained detailed information on jobs. These detailed log entries may include:

Setting Logging Levels

Impala uses the GLOG system, which supports three logging levels. You can adjust logging levels by exporting variable settings. To change logging settings manually, use a command similar to the following on each node before starting impalad:

export GLOG_v=1 For performance reasons, do not enable the most verbose logging level of 3 unless there is no other alternative for troubleshooting.

For more information on how to configure GLOG, including how to set variable logging levels for different system components, see How To Use Google Logging Library (glog).

Understanding What is Logged at Different Logging Levels

As logging levels increase, the categories of information logged are cumulative. For example, GLOG_v=2 records everything GLOG_v=1 records, as well as additional information.

Increasing logging levels imposes performance overhead and increases log size. Where practical, use GLOG_v=1 for most cases: this level has minimal performance impact but still captures useful troubleshooting information.

Additional information logged at each level is as follows:

  • GLOG_v=1 - The default level. Logs information about each connection and query that is initiated to an impalad instance, including runtime profiles.
  • GLOG_v=2 - Everything from the previous level plus information for each RPC initiated. This level also records query execution progress information, including details on each file that is read.
  • GLOG_v=3 - Everything from the previous level plus logging of every row that is read. This level is only applicable for the most serious troubleshooting and tuning scenarios, because it can produce exceptionally large and detailed log files, potentially leading to its own set of performance and capacity problems.
Redacting Sensitive Information from Impala Log Files

redaction Log redaction is a security feature that prevents sensitive information from being displayed in locations used by administrators for monitoring and troubleshooting, such as log files and the Impala debug web user interface. You configure regular expressions that match sensitive types of information processed by your system, such as credit card numbers or tax IDs, and literals matching these patterns are obfuscated wherever they would normally be recorded in log files or displayed in administration or debugging user interfaces.

In a security context, the log redaction feature is complementary to the Sentry authorization framework. Sentry prevents unauthorized users from being able to directly access table data. Redaction prevents administrators or support personnel from seeing the smaller amounts of sensitive or personally identifying information (PII) that might appear in queries issued by those authorized users.

See for details about how to enable this feature and set up the regular expressions to detect and redact sensitive information within SQL statement text.