REFRESH Statement REFRESH

REFRESH statement To accurately respond to queries, the Impala node that acts as the coordinator (the node to which you are connected through impala-shell, JDBC, or ODBC) must have current metadata about those databases and tables that are referenced in Impala queries. If you are not familiar with the way Impala uses metadata and how it shares the same metastore database as Hive, see for background information.

REFRESH [db_name.]table_name [PARTITION (key_col1=val1 [, key_col2=val2...])]

Use the REFRESH statement to load the latest metastore metadata and block location data for a particular table in these scenarios:

In and higher, the syntax ALTER TABLE table_name RECOVER PARTITIONS is a faster alternative to REFRESH when the only change to the table data is the addition of new partition directories through Hive or manual HDFS operations. See for details.

You only need to issue the REFRESH statement on the node to which you connect to issue queries. The coordinator node divides the work among all the Impala nodes in a cluster, and sends read requests for the correct HDFS blocks without relying on the metadata on the other nodes.

REFRESH reloads the metadata for the table from the metastore database, and does an incremental reload of the low-level block location data to account for any new data files added to the HDFS data directory for the table. It is a low-overhead, single-table operation, specifically tuned for the common scenario where new data files are added to HDFS.

Only the metadata for the specified table is flushed. The table must already exist and be known to Impala, either because the CREATE TABLE statement was run in Impala rather than Hive, or because a previous INVALIDATE METADATA statement caused Impala to reload its entire metadata catalog.

The catalog service broadcasts any changed metadata as a result of Impala ALTER TABLE, INSERT and LOAD DATA statements to all Impala nodes. Thus, the REFRESH statement is only required if you load data through Hive or by manipulating data files in HDFS directly. See for more information on the catalog service.

Another way to avoid inconsistency across nodes is to enable the SYNC_DDL query option before performing a DDL statement or an INSERT or LOAD DATA.

The table name is a required parameter. To flush the metadata for all tables, use the INVALIDATE METADATA command.

A metadata update for an impalad instance is required if:

  • A metadata change occurs.
  • and the change is made through Hive.
  • and the change is made to a metastore database to which clients such as the Impala shell or ODBC directly connect.

A metadata update for an Impala node is not required after you run ALTER TABLE, INSERT, or other table-modifying statement in Impala rather than Hive. Impala handles the metadata synchronization automatically through the catalog service.

Database and table metadata is typically modified by:

  • Hive - through ALTER, CREATE, DROP or INSERT operations.
  • Impalad - through CREATE TABLE, ALTER TABLE, and INSERT operations. Such changes are propagated to all Impala nodes by the Impala catalog service.

REFRESH causes the metadata for that table to be immediately reloaded. For a huge table, that process could take a noticeable amount of time; but doing the refresh up front avoids an unpredictable delay later, for example if the next reference to the table is during a benchmark test.

Refreshing a single partition:

In and higher, the REFRESH statement can apply to a single partition at a time, rather than the whole table. Include the optional PARTITION (partition_spec) clause and specify values for each of the partition key columns.

The following examples show how to make Impala aware of data added to a single partition, after data is loaded into a partition's data directory using some mechanism outside Impala, such as Hive or Spark. The partition can be one that Impala created and is already aware of, or a new partition created through Hive.

create table p (x int) partitioned by (y int); impala> insert into p (x,y) values (1,2), (2,2), (2,1); impala> show partitions p; +-------+-------+--------+------+... | y | #Rows | #Files | Size |... +-------+-------+--------+------+... | 1 | -1 | 1 | 2B |... | 2 | -1 | 1 | 4B |... | Total | -1 | 2 | 6B |... +-------+-------+--------+------+... -- ... Data is inserted into one of the partitions by some external mechanism ... beeline> insert into p partition (y = 1) values(1000); impala> refresh p partition (y=1); impala> select x from p where y=1; +------+ | x | +------+ | 2 | <- Original data created by Impala | 1000 | <- Additional data inserted through Beeline +------+ ]]>

The same applies for tables with more than one partition key column. The PARTITION clause of the REFRESH statement must include all the partition key columns.

create table p2 (x int) partitioned by (y int, z int); impala> insert into p2 (x,y,z) values (0,0,0), (1,2,3), (2,2,3); impala> show partitions p2; +-------+---+-------+--------+------+... | y | z | #Rows | #Files | Size |... +-------+---+-------+--------+------+... | 0 | 0 | -1 | 1 | 2B |... | 2 | 3 | -1 | 1 | 4B |... | Total | | -1 | 2 | 6B |... +-------+---+-------+--------+------+... -- ... Data is inserted into one of the partitions by some external mechanism ... beeline> insert into p2 partition (y = 2, z = 3) values(1000); impala> refresh p2 partition (y=2, z=3); impala> select x from p where y=2 and z = 3; +------+ | x | +------+ | 1 | <- Original data created by Impala | 2 | <- Original data created by Impala | 1000 | <- Additional data inserted through Beeline +------+ ]]>

The following examples show how specifying a nonexistent partition does not cause any error, and the order of the partition key columns does not have to match the column order in the table. The partition spec must include all the partition key columns; specifying an incomplete set of columns does cause an error.

The following example shows how you might use the REFRESH statement after manually adding new HDFS data files to the Impala data directory for a table:

[impalad-host:21000] > refresh t1; [impalad-host:21000] > refresh t2; [impalad-host:21000] > select * from t1; ... [impalad-host:21000] > select * from t2; ...

For more examples of using REFRESH and INVALIDATE METADATA with a combination of Impala and Hive operations, see .

Related impala-shell options:

The impala-shell option -r issues an INVALIDATE METADATA statement when starting up the shell, effectively performing a REFRESH of all tables. Due to the expense of reloading the metadata for all tables, the impala-shell -r option is not recommended for day-to-day use in a production environment. (This option was mainly intended as a workaround for synchronization issues in very old Impala versions.)

The user ID that the impalad daemon runs under, typically the impala user, must have execute permissions for all the relevant directories holding table data. (A table could have data spread across multiple directories, or in unexpected paths, if it uses partitioning or specifies a LOCATION attribute for individual partitions or the entire table.) Issues with permissions might not cause an immediate error for this statement, but subsequent statements such as SELECT or SHOW TABLE STATS could fail.

All HDFS and Sentry permissions and privileges are the same whether you refresh the entire table or a single partition.

The REFRESH command checks HDFS permissions of the underlying data files and directories, caching this information so that a statement can be cancelled immediately if for example the impala user does not have permission to write to the data directory for the table. Impala reports any lack of write permissions as an INFO message in the log file, in case that represents an oversight. If you change HDFS permissions to make data readable or writeable by the Impala user, issue another REFRESH to make Impala aware of the change.

,