mirror of
https://github.com/apache/impala.git
synced 2025-12-25 02:03:09 -05:00
4d01d8eb51
Add short notes about this release. Tests: - Built the html and pdf locally. Verified the new content. Change-Id: I4b9cc838de018c954f419ebb71d65c0d5725a4a9 Reviewed-on: http://gerrit.cloudera.org:8080/17676 Reviewed-by: Zoltan Borok-Nagy <boroknagyz@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
1202 lines
59 KiB
XML
1202 lines
59 KiB
XML
<?xml version="1.0" encoding="UTF-8"?>
|
|
<!--
|
|
Licensed to the Apache Software Foundation (ASF) under one
|
|
or more contributor license agreements. See the NOTICE file
|
|
distributed with this work for additional information
|
|
regarding copyright ownership. The ASF licenses this file
|
|
to you under the Apache License, Version 2.0 (the
|
|
"License"); you may not use this file except in compliance
|
|
with the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing,
|
|
software distributed under the License is distributed on an
|
|
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
KIND, either express or implied. See the License for the
|
|
specific language governing permissions and limitations
|
|
under the License.
|
|
-->
|
|
<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
|
|
<concept rev="ver" id="incompatible_changes">
|
|
<title><ph audience="standalone">Incompatible Changes and Limitations in
|
|
Apache Impala</ph><ph audience="integrated">Apache Impala Incompatible
|
|
Changes and Limitations</ph></title>
|
|
<prolog>
|
|
<metadata>
|
|
<data name="Category" value="Impala"/>
|
|
<data name="Category" value="Release Notes"/>
|
|
<data name="Category" value="Incompatible Changes"/>
|
|
<data name="Category" value="Limitations"/>
|
|
<data name="Category" value="Upgrading"/>
|
|
<data name="Category" value="Troubleshooting"/>
|
|
<data name="Category" value="Administrators"/>
|
|
<data name="Category" value="Developers"/>
|
|
<data name="Category" value="Data Analysts"/>
|
|
</metadata>
|
|
</prolog>
|
|
<conbody>
|
|
<p> The Impala version covered by this documentation library contains the
|
|
following incompatible changes. These are things such as file format
|
|
changes, removed features, or changes to implementation, default
|
|
configuration, dependencies, or prerequisites that could cause issues
|
|
during or after an Impala upgrade. </p>
|
|
<p> Even added SQL statements or clauses can produce incompatibilities, if
|
|
you have databases, tables, or columns whose names conflict with the new
|
|
keywords. <ph audience="PDF">See <xref
|
|
href="impala_reserved_words.xml#reserved_words"/> for the set of
|
|
reserved words for the current release, and the quoting techniques to
|
|
avoid name conflicts.</ph>
|
|
</p>
|
|
<p outputclass="toc inpage"/>
|
|
</conbody>
|
|
<concept id="incompatible_changes_400x">
|
|
<title>Incompatible Changes Introduced in Impala 4.0.x</title>
|
|
<conbody>
|
|
<p> For the full list of incompatible changes introduced in this release,
|
|
see the <xref keyref="release_notes_40">release notes for <keyword
|
|
keyref="impala40"/></xref>.
|
|
</p>
|
|
</conbody>
|
|
</concept>
|
|
<concept id="incompatible_changes_340x">
|
|
<title>Incompatible Changes Introduced in Impala 3.4.x</title>
|
|
<conbody>
|
|
<p> For the full list of issues closed in this release, including any that
|
|
introduce behavior changes or incompatibilities, see the <xref
|
|
keyref="changelog_34">changelog for <keyword keyref="impala34"
|
|
/></xref>. <ul>
|
|
<li>To optimize query performance, Impala planner uses the value of
|
|
the <codeph>fs.s3a.block.size</codeph> startup flag when calculating
|
|
the split size on non-block based stores, e.g. S3, ADLS, etc.
|
|
Starting in this release, Impala planner uses the
|
|
<codeph>PARQUET_OBJECT_STORE_SPLIT_SIZE</codeph> query option to
|
|
get the Parquet file format specific split size.<p>For Parquet
|
|
files, the <codeph>fs.s3a.block.size</codeph> startup flag is no
|
|
longer used.</p><p>The default value of the
|
|
<codeph>PARQUET_OBJECT_STORE_SPLIT_SIZE</codeph> query option is
|
|
256 MB.</p></li>
|
|
</ul></p>
|
|
</conbody>
|
|
</concept>
|
|
<concept id="incompatible_changes_330x">
|
|
<title>Incompatible Changes Introduced in Impala 3.3.x</title>
|
|
<conbody>
|
|
<p> For the full list of issues closed in this release, including any that
|
|
introduce behavior changes or incompatibilities, see the <xref
|
|
keyref="changelog_33">changelog for <keyword keyref="impala33"
|
|
/></xref>. <ul>
|
|
<li>Default file format changed to Parquet<p>When you create a table,
|
|
the default format for that table data is now Parquet.</p><p>For
|
|
backward compatibility, you can use the DEFAULT_FILE_FORMAT query
|
|
option to set the default file format to the previous default,
|
|
text, or other formats.</p></li>
|
|
</ul></p>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="3.1.0" id="incompatible_changes_320x">
|
|
<title>Incompatible Changes Introduced in Impala 3.2.x</title>
|
|
<conbody>
|
|
<p> For the full list of issues closed in this release, including any that
|
|
introduce behavior changes or incompatibilities, see the <xref
|
|
keyref="changelog_32">changelog for <keyword keyref="impala32"
|
|
/></xref>. </p>
|
|
<ul>
|
|
<li>The Port change for the <codeph>SHUTDOWN</codeph> command<p>The
|
|
<codeph>SHUTDOWN</codeph> command for shutting down a remote
|
|
server used the backend port in Impala 3.1. Starting in Impala 3.2,
|
|
the command uses the KRPC port, e.g.<codeph>
|
|
:shutdown('host100:27000')</codeph>.</p></li>
|
|
</ul>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="3.1.0" id="incompatible_changes_310x">
|
|
<title>Incompatible Changes Introduced in Impala 3.1.x</title>
|
|
<conbody>
|
|
<p> For the full list of issues closed in this release, including any that
|
|
introduce behavior changes or incompatibilities, see the <xref
|
|
keyref="changelog_31">changelog for <keyword keyref="impala31"
|
|
/></xref>. </p>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="3.0.0" id="incompatible_changes_300x">
|
|
<title>Incompatible Changes Introduced in Impala 3.0.x</title>
|
|
<conbody>
|
|
<p> For the full list of issues closed in this release, including any that
|
|
introduce behavior changes or incompatibilities, see the <xref
|
|
keyref="changelog_300">changelog for <keyword keyref="impala30"
|
|
/></xref>. </p>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="2.12.0" id="incompatible_changes_212x">
|
|
<title>Incompatible Changes Introduced in Impala 2.12.x</title>
|
|
<conbody>
|
|
<p> For the full list of issues closed in this release, including any that
|
|
introduce behavior changes or incompatibilities, see the <xref
|
|
keyref="changelog_212">changelog for <keyword keyref="impala212"
|
|
/></xref>. </p>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="2.11.0" id="incompatible_changes_211x">
|
|
<title>Incompatible Changes Introduced in Impala 2.11.x</title>
|
|
<conbody>
|
|
<p> For the full list of issues closed in this release, including any that
|
|
introduce behavior changes or incompatibilities, see the <xref
|
|
keyref="changelog_211">changelog for <keyword keyref="impala211"
|
|
/></xref>. </p>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="2.10.0" id="incompatible_changes_210x">
|
|
<title>Incompatible Changes Introduced in Impala 2.10.x</title>
|
|
<conbody>
|
|
<p> For the full list of issues closed in this release, including any that
|
|
introduce behavior changes or incompatibilities, see the <xref
|
|
keyref="changelog_210">changelog for <keyword keyref="impala210"
|
|
/></xref>. </p>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="2.9.0" id="incompatible_changes_29x">
|
|
<title>Incompatible Changes Introduced in Impala 2.9.x</title>
|
|
<conbody>
|
|
<p> For the full list of issues closed in this release, including any that
|
|
introduce behavior changes or incompatibilities, see the <xref
|
|
keyref="changelog_29">changelog for <keyword keyref="impala29"
|
|
/></xref>. </p>
|
|
<!-- Prose explanations of specific incompatible changes - TBD.
|
|
<ul>
|
|
<li>
|
|
</li>
|
|
</ul>
|
|
-->
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="2.8.0" id="incompatible_changes_28x">
|
|
<title>Incompatible Changes Introduced in Impala 2.8.x</title>
|
|
<conbody>
|
|
<ul>
|
|
<li>
|
|
<p rev="IMPALA-4160"> Llama support is removed completely from Impala.
|
|
Related flags (<codeph>--enable_rm</codeph>) and query options (such
|
|
as <codeph>V_CPU_CORES</codeph>) remain but do not have any effect. </p>
|
|
<p rev="IMPALA-4160"> If <codeph>--enable_rm</codeph> is passed to
|
|
Impala, a warning is printed to the log on startup. </p>
|
|
</li>
|
|
<li>
|
|
<p rev="kudu"> The syntax related to Kudu tables includes a number of
|
|
new reserved words, such as <codeph>COMPRESSION</codeph>,
|
|
<codeph>DEFAULT</codeph>, and <codeph>ENCODING</codeph>, that
|
|
might conflict with names of existing tables, columns, or other
|
|
identifiers from older Impala versions. See <xref
|
|
href="impala_reserved_words.xml#reserved_words"/> for the full
|
|
list of reserved words. </p>
|
|
</li>
|
|
<li>
|
|
<p rev="kudu"> The DDL syntax for Kudu tables, particularly in the
|
|
<codeph>CREATE TABLE</codeph> statement, is different from the
|
|
special <codeph>impala_next</codeph> fork that was previously used
|
|
for accessing Kudu tables from Impala: </p>
|
|
<ul rev="kudu">
|
|
<li>
|
|
<p> The <codeph>DISTRIBUTE BY</codeph> clause is now
|
|
<codeph>PARTITIONED BY</codeph>. </p>
|
|
</li>
|
|
<li>
|
|
<p> The <codeph>INTO <varname>N</varname> BUCKETS</codeph> clause
|
|
is now <codeph>PARTITIONS <varname>N</varname></codeph>. </p>
|
|
</li>
|
|
<li>
|
|
<p> The <codeph>SPLIT ROWS</codeph> clause is replaced by
|
|
different syntax for specifying the ranges covered by each
|
|
partition. </p>
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li>
|
|
<p> The <codeph>DESCRIBE</codeph> output for Kudu tables includes
|
|
several extra columns. </p>
|
|
</li>
|
|
<li>
|
|
<p rev="kudu IMPALA-4527"> Non-primary-key columns can contain
|
|
<codeph>NULL</codeph> values by default. The <codeph>SHOW CREATE
|
|
TABLE</codeph> output for these columns displays the
|
|
<codeph>NULL</codeph> attribute. There was a period during early
|
|
experimental versions of Impala + Kudu where non-primary-key columns
|
|
had the <codeph>NOT NULL</codeph> attribute by default. </p>
|
|
</li>
|
|
<li>
|
|
<p rev="kudu IMPALA-3710"> The <codeph>IGNORE</codeph> keyword that
|
|
was present in early experimental versions of Impala + Kudu is no
|
|
longer present. The behavior of the <codeph>IGNORE</codeph> keyword
|
|
is now the default: DML statements continue with warnings, instead
|
|
of failing with errors, if they encounter conditions such as
|
|
<q>primary key already exists</q> for an <codeph>INSERT</codeph>
|
|
statement or <q>primary key already deleted</q> for a
|
|
<codeph>DELETE</codeph> statement. </p>
|
|
</li>
|
|
<li>
|
|
<p rev="IMPALA-4589"> The replication factor for Kudu tables must be
|
|
an odd number. </p>
|
|
</li>
|
|
<li>
|
|
<p rev="IMPALA-4432"> A UDF compiled into an LLVM IR bitcode module
|
|
(<codeph>.bc</codeph>) might encounter a runtime error when native
|
|
code generation is turned off by setting the query option
|
|
<codeph>DISABLE_CODEGEN=1</codeph>. This issue also applies when
|
|
running a built-in or native UDF with more than 20 arguments. See
|
|
<xref keyref="IMPALA-4432">IMPALA-4432</xref> for details. As a
|
|
workaround, either turn native code generation back on with the
|
|
query option <codeph>DISABLE_CODEGEN=0</codeph>, or use the regular
|
|
UDF compilation path that does not produce an IR module. </p>
|
|
</li>
|
|
</ul>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="2.7.0" id="incompatible_changes_27x">
|
|
<title>Incompatible Changes Introduced in Impala 2.7.x</title>
|
|
<conbody>
|
|
<ul>
|
|
<li>
|
|
<p rev="IMPALA-1731 IMPALA-3868"> Bug fixes related to parsing of
|
|
floating-point values (IMPALA-1731 and IMPALA-3868) can change the
|
|
results of casting strings that represent invalid floating-point
|
|
values. For example, formerly a string value beginning or ending
|
|
with <codeph>inf</codeph>, such as <codeph>1.23inf</codeph> or
|
|
<codeph>infinite</codeph>, now are converted to
|
|
<codeph>NULL</codeph> when interpreted as a floating-point value.
|
|
Formerly, they were interpreted as the special <q>infinity</q> value
|
|
when converting from string to floating-point. Similarly, now only
|
|
the string <codeph>NaN</codeph> (case-sensitive) is interpreted as
|
|
the special <q>not a number</q> value. String values containing
|
|
multiple dots, such as <codeph>3..141</codeph> or
|
|
<codeph>3.1.4.1</codeph>, are now interpreted as
|
|
<codeph>NULL</codeph> rather than being converted to valid
|
|
floating-point values. </p>
|
|
</li>
|
|
</ul>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="2.6.0" id="incompatible_changes_26x">
|
|
<title>Incompatible Changes Introduced in Impala 2.6.x</title>
|
|
<conbody>
|
|
<ul>
|
|
<li>
|
|
<p rev=""> The default for the <codeph>RUNTIME_FILTER_MODE</codeph>
|
|
query option is changed to <codeph>GLOBAL</codeph> (the highest
|
|
setting). </p>
|
|
</li>
|
|
<li rev="IMPALA-3007">
|
|
<p> The <codeph>RUNTIME_BLOOM_FILTER_SIZE</codeph> setting is now only
|
|
used as a fallback if statistics are not available; otherwise,
|
|
Impala uses the statistics to estimate the appropriate size to use
|
|
for each filter. </p>
|
|
</li>
|
|
<li>
|
|
<p rev="IMPALA-3199"> Admission control and dynamic resource pools are
|
|
enabled by default. When upgrading from an earlier release, you must
|
|
turn on these settings yourself if they are not already enabled. See
|
|
<xref href="impala_admission.xml#admission_control"/> for details
|
|
about admission control. </p>
|
|
</li>
|
|
<li>
|
|
<p> Impala reserves some new keywords, in preparation for support for
|
|
Kudu syntax: <codeph>buckets</codeph>, <codeph>delete</codeph>,
|
|
<codeph>distribute</codeph>, <codeph>hash</codeph>,
|
|
<codeph>ignore</codeph>, <codeph>split</codeph>, and
|
|
<codeph>update</codeph>. </p>
|
|
</li>
|
|
<li>
|
|
<p rev="IMPALA-3554"> For Kerberized clusters, the Catalog service now
|
|
uses the Kerberos principal instead of the operating sytem user that
|
|
runs the <cmdname>catalogd</cmdname> daemon. This eliminates the
|
|
requirement to configure a
|
|
<codeph>hadoop.user.group.static.mapping.overrides</codeph>
|
|
setting to put the OS user into the Sentry administrative group, on
|
|
clusters where the principal and the OS user name for this user are
|
|
different. </p>
|
|
</li>
|
|
<li>
|
|
<p> The mechanism for interpreting <codeph>DECIMAL</codeph> literals
|
|
is improved, no longer going through an intermediate conversion step
|
|
to <codeph>DOUBLE</codeph>: </p>
|
|
<ul>
|
|
<li>
|
|
<p rev="IMPALA-3163"> Casting a <codeph>DECIMAL</codeph> value to
|
|
<codeph>TIMESTAMP</codeph>
|
|
<codeph>DOUBLE</codeph> produces a more precise value for the
|
|
<codeph>TIMESTAMP</codeph> than formerly. </p>
|
|
</li>
|
|
<li>
|
|
<p rev="IMPALA-3439"> Certain function calls involving
|
|
<codeph>DECIMAL</codeph> literals now succeed, when formerly
|
|
they failed due to lack of a function signature with a
|
|
<codeph>DOUBLE</codeph> argument. </p>
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li>
|
|
<p rev="IMPALA-3155"> Improved type accuracy for <codeph>CASE</codeph>
|
|
return values. If all <codeph>WHEN</codeph> clauses of the
|
|
<codeph>CASE</codeph> expression are of <codeph>CHAR</codeph>
|
|
type, the final result is also <codeph>CHAR</codeph> instead of
|
|
being converted to <codeph>STRING</codeph>. </p>
|
|
</li>
|
|
<li>
|
|
<p conref="../shared/impala_common.xml#common/IMPALA-3662"/>
|
|
</li>
|
|
<li rev="IMPALA-3452">
|
|
<p> The <codeph>S3_SKIP_INSERT_STAGING</codeph> query option, which is
|
|
enabled by default, increases the speed of <codeph>INSERT</codeph>
|
|
operations for S3 tables. The speedup applies to regular
|
|
<codeph>INSERT</codeph>, but not <codeph>INSERT
|
|
OVERWRITE</codeph>. The tradeoff is the possibility of inconsistent
|
|
output files left behind if a node fails during
|
|
<codeph>INSERT</codeph> execution. See <xref
|
|
href="impala_s3_skip_insert_staging.xml#s3_skip_insert_staging"/>
|
|
for details. </p>
|
|
</li>
|
|
</ul>
|
|
<p> Certain features are turned off by default, to avoid regressions or
|
|
unexpected behavior following an upgrade. Consider turning on these
|
|
features after suitable testing: </p>
|
|
<ul>
|
|
<li>
|
|
<p rev="IMPALA-2660"> Impala now recognizes the
|
|
<codeph>auth_to_local</codeph> setting, specified through the HDFS
|
|
configuration setting
|
|
<codeph>hadoop.security.auth_to_local</codeph>. This feature is
|
|
disabled by default; to enable it, specify
|
|
<codeph>--load_auth_to_local_rules=true</codeph> in the
|
|
<cmdname>impalad</cmdname> configuration settings. </p>
|
|
</li>
|
|
<li>
|
|
<p rev="IMPALA-2069"> A new query option,
|
|
<codeph>PARQUET_ANNOTATE_STRINGS_UTF8</codeph>, makes Impala
|
|
include the <codeph>UTF-8</codeph> annotation metadata for
|
|
<codeph>STRING</codeph>, <codeph>CHAR</codeph>, and
|
|
<codeph>VARCHAR</codeph> columns in Parquet files created by
|
|
<codeph>INSERT</codeph> or <codeph>CREATE TABLE AS SELECT</codeph>
|
|
statements. </p>
|
|
</li>
|
|
<li>
|
|
<p rev="IMPALA-2835"> A new query option,
|
|
<codeph>PARQUET_FALLBACK_SCHEMA_RESOLUTION</codeph>, lets Impala
|
|
locate columns within Parquet files based on column name rather than
|
|
ordinal position. This enhancement improves interoperability with
|
|
applications that write Parquet files with a different order or
|
|
subset of columns than are used in the Impala table. </p>
|
|
</li>
|
|
</ul>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="2.5.x" id="incompatible_changes_25x">
|
|
<title>Incompatible Changes Introduced in Impala 2.5.x</title>
|
|
<conbody>
|
|
<ul>
|
|
<li rev="IMPALA-3044">
|
|
<p> The admission control default limit for concurrent queries (the
|
|
<uicontrol>max requests</uicontrol> setting) is now unlimited
|
|
instead of 200. </p>
|
|
</li>
|
|
<li>
|
|
<p rev="IMPALA-2749"> Multiplying a mixture of
|
|
<codeph>DECIMAL</codeph> and <codeph>FLOAT</codeph> or
|
|
<codeph>DOUBLE</codeph> values now returns <codeph>DOUBLE</codeph>
|
|
rather than <codeph>DECIMAL</codeph>. This change avoids some cases
|
|
where an intermediate value would underflow or overflow and become
|
|
<codeph>NULL</codeph> unexpectedly. The results of multiplying
|
|
<codeph>DECIMAL</codeph> and <codeph>FLOAT</codeph> or
|
|
<codeph>DOUBLE</codeph> might now be slightly less precise than
|
|
before. Previously, the intermediate types and thus the final result
|
|
depended on the exact order of the values of different types being
|
|
multiplied, which made the final result values difficult to reason
|
|
about. </p>
|
|
</li>
|
|
<li rev="IMPALA-2204">
|
|
<p> Previously, the <codeph>_</codeph> and <codeph>%</codeph> wildcard
|
|
characters for the <codeph>LIKE</codeph> operator would not match
|
|
characters on the second or subsequent lines of multi-line string
|
|
values. The fix for issue <xref keyref="IMPALA-2204"
|
|
>IMPALA-2204</xref> causes the wildcard matching to apply to the
|
|
entire string for values containing embedded <codeph>\n</codeph>
|
|
characters. This could cause different results than in previous
|
|
Impala releases for identical queries on identical data. </p>
|
|
</li>
|
|
<li rev="IMPALA-1748">
|
|
<p> Formerly, all Impala UDFs and UDAs required running the
|
|
<codeph>CREATE FUNCTION</codeph> statements to re-create them
|
|
after each <cmdname>catalogd</cmdname> restart. In <keyword
|
|
keyref="impala25_full"/> and higher, functions written in C++ are
|
|
persisted across restarts, and the requirement to re-create
|
|
functions only applies to functions written in Java. Adapt any
|
|
function-reloading logic that you have added to your Impala
|
|
environment. </p>
|
|
</li>
|
|
<li>
|
|
<p rev="IMPALA-1651">
|
|
<codeph>CREATE TABLE LIKE</codeph> no longer inherits HDFS caching
|
|
settings from the source table. </p>
|
|
</li>
|
|
<li>
|
|
<p rev="IMPALA-2070"> The <codeph>SHOW DATABASES</codeph> statement
|
|
now returns two columns rather than one. The second column includes
|
|
the associated comment string, if any, for each database. Adjust any
|
|
application code that examines the list of databases and assumes the
|
|
result set contains only a single column. </p>
|
|
</li>
|
|
<li>
|
|
<p> The output of the <codeph>SHOW FUNCTIONS</codeph> statement
|
|
includes two new columns, showing the kind of the function (for
|
|
example, <codeph>BUILTIN</codeph>) and whether or not the function
|
|
persists across catalog server restarts. For example, the
|
|
<codeph>SHOW FUNCTIONS</codeph> output for the
|
|
<codeph>_impala_builtins</codeph> database starts with: </p>
|
|
<codeblock>
|
|
+--------------+-------------------------------------------------+-------------+---------------+
|
|
| return type | signature | binary type | is persistent |
|
|
+--------------+-------------------------------------------------+-------------+---------------+
|
|
| BIGINT | abs(BIGINT) | BUILTIN | true |
|
|
| DECIMAL(*,*) | abs(DECIMAL(*,*)) | BUILTIN | true |
|
|
| DOUBLE | abs(DOUBLE) | BUILTIN | true |
|
|
...
|
|
</codeblock>
|
|
</li>
|
|
</ul>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="2.4.x" id="incompatible_changes_24x">
|
|
<title>Incompatible Changes Introduced in Impala 2.4.x</title>
|
|
<conbody>
|
|
<p> Other than support for DSSD storage, the Impala feature set for
|
|
<keyword keyref="impala24"/> is the same as for <keyword
|
|
keyref="impala23"/>. Therefore, there are no incompatible changes for
|
|
Impala introduced in <keyword keyref="impala24"/>. </p>
|
|
</conbody>
|
|
</concept>
|
|
<!-- All 2.3.x subsections go under here -->
|
|
<!-- Actually for 2.3 and higher, let's get away from doing a separate subhead for each maintenance release,
|
|
because in the normal course of events there will be nothing to add here until the next full release. If something new
|
|
needs to get noted, just add a new bullet with wording to indicate which x.y.z release it applies to. -->
|
|
<concept rev="2.3.x" id="incompatible_changes_23x">
|
|
<title>Incompatible Changes Introduced in Impala 2.3.x</title>
|
|
<conbody>
|
|
<note conref="../shared/impala_common.xml#common/impala_llama_obsolete"/>
|
|
<ul>
|
|
<li rev="IMPALA-2005" audience="hidden">
|
|
<p> If a <codeph>CREATE TABLE AS SELECT</codeph> operation fails while
|
|
data is being inserted, the table is automatically removed.
|
|
Previously, the table was left behind with no data. </p>
|
|
</li>
|
|
<li rev="IMPALA-2130">
|
|
<p> If Impala encounters a Parquet file that is invalid because of an
|
|
incorrect magic number, the query skips the file. This change is
|
|
caused by the fix for issue <xref keyref="IMPALA-2130"
|
|
>IMPALA-2130</xref>. Previously, Impala would attempt to read the
|
|
file despite the possibility that the file was corrupted. </p>
|
|
</li>
|
|
<li rev="IMPALA-2233">
|
|
<p> Previously, calls to overloaded built-in functions could treat
|
|
parameters as <codeph>DOUBLE</codeph> or <codeph>FLOAT</codeph> when
|
|
no overload had a signature that matched the exact argument types.
|
|
Now Impala prefers the function signature with
|
|
<codeph>DECIMAL</codeph> parameters in this case. This change
|
|
avoids a possible loss of precision in function calls such as
|
|
<codeph>greatest(0, 99999.8888)</codeph>; now both parameters are
|
|
treated as <codeph>DECIMAL</codeph> rather than
|
|
<codeph>DOUBLE</codeph>, avoiding any loss of precision in the
|
|
fractional value. This could cause slightly different results than
|
|
in previous Impala releases for certain function calls. </p>
|
|
</li>
|
|
<li rev="IMPALA-1675">
|
|
<p> Formerly, adding or subtracting a large interval value to a
|
|
<codeph>TIMESTAMP</codeph> could produce a nonsensical result. Now
|
|
when the result goes outside the range of <codeph>TIMESTAMP</codeph>
|
|
values, Impala returns <codeph>NULL</codeph>. </p>
|
|
</li>
|
|
<li rev="IMPALA-2251 IMPALA-2257">
|
|
<p> Formerly, it was possible to accidentally create a table with
|
|
identical row and column delimiters. This could happen
|
|
unintentionally, when specifying one of the delimiters and using the
|
|
default value for the other. Now an attempt to use identical
|
|
delimiters still succeeds, but displays a warning message. </p>
|
|
</li>
|
|
<li rev="">
|
|
<p> Formerly, Impala could include snippets of table data in log files
|
|
by default, for example when reporting conversion errors for data
|
|
values. Now any such log messages are only produced at higher
|
|
logging levels that you would enable only during debugging. </p>
|
|
</li>
|
|
<!-- placeholder -->
|
|
</ul>
|
|
</conbody>
|
|
</concept>
|
|
<!-- All 2.2.x subsections go under here -->
|
|
<concept rev="2.2.x" id="incompatible_changes_22x">
|
|
<title>Incompatible Changes Introduced in Impala 2.2.x</title>
|
|
<conbody>
|
|
<section id="files_220">
|
|
<title> Changes to File Handling </title>
|
|
<p conref="../shared/impala_common.xml#common/ignore_file_extensions"/>
|
|
<p> The log rotation feature in Impala 2.2.0 and higher means that older
|
|
log files are now removed by default. The default is to preserve the
|
|
latest 10 log files for each severity level, for each Impala-related
|
|
daemon. If you have set up your own log rotation processes that expect
|
|
older files to be present, either adjust your procedures or change the
|
|
Impala <codeph>-max_log_files</codeph> setting. <ph audience="PDF">See
|
|
<xref href="impala_logging.xml#logs_rotate"/> for details.</ph>
|
|
</p>
|
|
</section>
|
|
<section id="prereqs_210">
|
|
<title> Changes to Prerequisites </title>
|
|
<p conref="../shared/impala_common.xml#common/cpu_prereq"/>
|
|
</section>
|
|
</conbody>
|
|
</concept>
|
|
<!-- All 2.1.x subsections go under here -->
|
|
<concept rev="2.1.x" id="incompatible_changes_21x">
|
|
<title>Incompatible Changes Introduced in Impala 2.1.x</title>
|
|
<conbody>
|
|
<section id="prereqs_210">
|
|
<title> Changes to Prerequisites </title>
|
|
<p rev=""> Currently, Impala 2.1.x does not function on CPUs without the
|
|
SSE4.1 instruction set. This minimum CPU requirement is higher than in
|
|
previous versions, which relied on the older SSSE3 instruction set.
|
|
Check the CPU level of the hosts in your cluster before upgrading to
|
|
<keyword keyref="impala21_full"/>. </p>
|
|
</section>
|
|
<section id="output_format_210">
|
|
<title> Changes to Output Format </title>
|
|
<p> The <q>small query</q> optimization feature introduces some new
|
|
information in the <codeph>EXPLAIN</codeph> plan, which you might need
|
|
to account for if you parse the text of the plan output. </p>
|
|
</section>
|
|
<section id="reserved_words_210">
|
|
<title> New Reserved Words </title>
|
|
<p> New SQL syntax introduces additional reserved words:
|
|
<codeph>FOR</codeph>, <codeph>GRANT</codeph>,
|
|
<codeph>REVOKE</codeph>, <codeph>ROLE</codeph>,
|
|
<codeph>ROLES</codeph>, <codeph>INCREMENTAL</codeph>. <ph
|
|
audience="PDF">As always, see <xref
|
|
href="impala_reserved_words.xml#reserved_words"/> for the set of
|
|
reserved words for the current release, and the quoting techniques
|
|
to avoid name conflicts.</ph>
|
|
</p>
|
|
</section>
|
|
</conbody>
|
|
</concept>
|
|
<!-- All 2.0.x subsections go under here -->
|
|
<concept rev="2.0.5" id="incompatible_changes_205">
|
|
<title>Incompatible Changes Introduced in Impala 2.0.5</title>
|
|
<conbody>
|
|
<p> No incompatible changes. </p>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="2.0.4" id="incompatible_changes_204">
|
|
<title>Incompatible Changes Introduced in Impala 2.0.4</title>
|
|
<conbody>
|
|
<p> No incompatible changes. </p>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="2.0.3" id="incompatible_changes_203">
|
|
<title>Incompatible Changes Introduced in Impala 2.0.3</title>
|
|
<conbody> </conbody>
|
|
</concept>
|
|
<concept rev="2.0.2" id="incompatible_changes_202">
|
|
<title>Incompatible Changes Introduced in Impala 2.0.2</title>
|
|
<conbody>
|
|
<p> No incompatible changes. </p>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="2.0.1" id="incompatible_changes_201">
|
|
<title>Incompatible Changes Introduced in Impala 2.0.1</title>
|
|
<conbody>
|
|
<ul>
|
|
<li>
|
|
<p
|
|
conref="../shared/impala_common.xml#common/insert_hidden_work_directory"
|
|
/>
|
|
</li>
|
|
<li>
|
|
<p> The <codeph>abs()</codeph> function now takes a broader range of
|
|
numeric types as arguments, and the return type is the same as the
|
|
argument type. </p>
|
|
</li>
|
|
<li>
|
|
<p> Shorthand notation for character classes in regular expressions,
|
|
such as <codeph>\d</codeph> for digit, are now available again in
|
|
regular expression operators and functions such as
|
|
<codeph>regexp_extract()</codeph> and
|
|
<codeph>regexp_replace()</codeph>. Some other differences in
|
|
regular expression behavior remain between Impala 1.x and Impala 2.x
|
|
releases. See <xref
|
|
href="impala_incompatible_changes.xml#incompatible_changes_200"/>
|
|
for details. </p>
|
|
</li>
|
|
</ul>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="2.0.0" id="incompatible_changes_200">
|
|
<title>Incompatible Changes Introduced in Impala 2.0.0</title>
|
|
<conbody>
|
|
<section id="prereqs_200">
|
|
<title> Changes to Prerequisites </title>
|
|
<p rev=""> Currently, Impala 2.0.x does not function on CPUs without the
|
|
SSE4.1 instruction set. This minimum CPU requirement is higher than in
|
|
previous versions, which relied on the older SSSE3 instruction set.
|
|
Check the CPU level of the hosts in your cluster before upgrading to
|
|
<keyword keyref="impala20_full"/>. </p>
|
|
</section>
|
|
<section id="queries_200">
|
|
<title> Changes to Query Syntax </title>
|
|
<p> The new syntax where query hints are allowed in comments causes some
|
|
changes in the way comments are parsed in the
|
|
<cmdname>impala-shell</cmdname> interpreter. Previously, you could
|
|
end a <codeph>--</codeph> comment line with a semicolon and
|
|
<cmdname>impala-shell</cmdname> would treat that as a no-op
|
|
statement. Now, a comment line ending with a semicolon is passed as an
|
|
empty statement to the Impala daemon, where it is flagged as an error. </p>
|
|
<p> Impala 2.0 and later uses a different support library for regular
|
|
expression parsing than in earlier Impala versions. Now, Impala uses
|
|
the <xref href="https://code.google.com/p/re2/" scope="external"
|
|
format="html">Google RE2 library</xref> rather than Boost for
|
|
evaluating regular expressions. This implementation change causes some
|
|
differences in the allowed regular expression syntax, and in the way
|
|
certain regex operators are interpreted. The following are some of the
|
|
major differences (not necessarily a complete list): </p>
|
|
<ul>
|
|
<li>
|
|
<p>
|
|
<codeph>.*?</codeph> notation for non-greedy matches is now
|
|
supported, where it was not in earlier Impala releases. </p>
|
|
</li>
|
|
<li>
|
|
<p> By default, <codeph>^</codeph> and <codeph>$</codeph> now match
|
|
only begin/end of buffer, not begin/end of each line. This
|
|
behavior can be overridden in the regex itself using the
|
|
<codeph>m</codeph> flag. </p>
|
|
</li>
|
|
<li>
|
|
<p> By default, <codeph>.</codeph> does not match newline. This
|
|
behavior can be overridden in the regex itself using the
|
|
<codeph>s</codeph> flag. </p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<codeph>\Z</codeph> is not supported. </p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<codeph><</codeph> and <codeph>></codeph> for start of word
|
|
and end of word are not supported. </p>
|
|
</li>
|
|
<li>
|
|
<p> Lookahead and lookbehind are not supported. </p>
|
|
</li>
|
|
<li>
|
|
<p> Shorthand notation for character classes, such as
|
|
<codeph>\d</codeph> for digit, is not recognized. (This
|
|
restriction is lifted in Impala 2.0.1, which restores the
|
|
shorthand notation.) </p>
|
|
</li>
|
|
</ul>
|
|
</section>
|
|
<section id="output_format_210">
|
|
<title> Changes to Output Format </title>
|
|
<p conref="../shared/impala_common.xml#common/user_kerberized"/>
|
|
<p> The changed format for the user name in secure environments is also
|
|
reflected where the user name is displayed in the output of the
|
|
<codeph>PROFILE</codeph> command. </p>
|
|
<p> In the output from <codeph>SHOW FUNCTIONS</codeph>, <codeph>SHOW
|
|
AGGREGATE FUNCTIONS</codeph>, and <codeph>SHOW ANALYTIC
|
|
FUNCTIONS</codeph>, arguments and return types of arbitrary
|
|
<codeph>DECIMAL</codeph> scale and precision are represented as
|
|
<codeph>DECIMAL(*,*)</codeph>. Formerly, these items were displayed
|
|
as <codeph>DECIMAL(-1,-1)</codeph>. </p>
|
|
</section>
|
|
<section id="query_options_200">
|
|
<title> Changes to Query Options </title>
|
|
<p> The <codeph>PARQUET_COMPRESSION_CODEC</codeph> query option has been
|
|
replaced by the <codeph>COMPRESSION_CODEC</codeph> query option. <ph
|
|
audience="PDF">See <xref
|
|
href="impala_compression_codec.xml#compression_codec"/> for
|
|
details.</ph>
|
|
</p>
|
|
</section>
|
|
<section id="config_options_200">
|
|
<title> Changes to Configuration Options </title>
|
|
<p> The meaning of the <codeph>--idle_query_timeout</codeph>
|
|
configuration option is changed, to accommodate the new
|
|
<codeph>QUERY_TIMEOUT_S</codeph> query option. Rather than setting
|
|
an absolute timeout period that applies to all queries, it now sets a
|
|
maximum timeout period, which can be adjusted downward for individual
|
|
queries by specifying a value for the <codeph>QUERY_TIMEOUT_S</codeph>
|
|
query option. In sessions where no <codeph>QUERY_TIMEOUT_S</codeph>
|
|
query option is specified, the <codeph>--idle_query_timeout</codeph>
|
|
timeout period applies the same as in earlier versions. </p>
|
|
<p> The <codeph>--strict_unicode</codeph> option of
|
|
<cmdname>impala-shell</cmdname> was removed. To avoid problems with
|
|
Unicode values in <cmdname>impala-shell</cmdname>, define the
|
|
following locale setting before running
|
|
<cmdname>impala-shell</cmdname>: </p>
|
|
<codeblock>export LC_CTYPE=en_US.UTF-8
|
|
</codeblock>
|
|
</section>
|
|
<section id="reserved_words_210">
|
|
<title> New Reserved Words </title>
|
|
<p> Some new SQL syntax requires the addition of new reserved words:
|
|
<codeph>ANTI</codeph>, <codeph>ANALYTIC</codeph>,
|
|
<codeph>OVER</codeph>, <codeph>PRECEDING</codeph>,
|
|
<codeph>UNBOUNDED</codeph>, <codeph>FOLLOWING</codeph>,
|
|
<codeph>CURRENT</codeph>, <codeph>ROWS</codeph>,
|
|
<codeph>RANGE</codeph>, <codeph>CHAR</codeph>,
|
|
<codeph>VARCHAR</codeph>. <ph audience="PDF">As always, see <xref
|
|
href="impala_reserved_words.xml#reserved_words"/> for the set of
|
|
reserved words for the current release, and the quoting techniques
|
|
to avoid name conflicts.</ph>
|
|
</p>
|
|
</section>
|
|
<section id="output_files_200">
|
|
<title> Changes to Data Files </title>
|
|
<p id="parquet_block_size"> The default Parquet block size for Impala is
|
|
changed from 1 GB to 256 MB. This change could have implications for
|
|
the sizes of Parquet files produced by <codeph>INSERT</codeph> and
|
|
<codeph>CREATE TABLE AS SELECT</codeph> statements. </p>
|
|
<p> Although older Impala releases typically produced files that were
|
|
smaller than the old default size of 1 GB, now the file size matches
|
|
more closely whatever value is specified for the
|
|
<codeph>PARQUET_FILE_SIZE</codeph> query option. Thus, if you use a
|
|
non-default value for this setting, the output files could be larger
|
|
than before. They still might be somewhat smaller than the specified
|
|
value, because Impala makes conservative estimates about the space
|
|
needed to represent each column as it encodes the data. </p>
|
|
<p> When you do not specify an explicit value for the
|
|
<codeph>PARQUET_FILE_SIZE</codeph> query option, Impala tries to
|
|
keep the file size within the 256 MB default size, but Impala might
|
|
adjust the file size to be somewhat larger if needed to accommodate
|
|
the layout for <term>wide</term> tables, that is, tables with hundreds
|
|
or thousands of columns. </p>
|
|
<p> This change is unlikely to affect memory usage while writing Parquet
|
|
files, because Impala does not pre-allocate the memory needed to hold
|
|
the entire Parquet block. </p>
|
|
</section>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="1.4.4" id="incompatible_changes_144">
|
|
<title>Incompatible Changes Introduced in Impala 1.4.4</title>
|
|
<conbody>
|
|
<p> No incompatible changes. </p>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="1.4.3" id="incompatible_changes_143">
|
|
<title>Incompatible Changes Introduced in Impala 1.4.3</title>
|
|
<conbody>
|
|
<p> No incompatible changes. The TLS/SSL security fix does not require any
|
|
change in the way you interact with Impala. </p>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="1.4.2" id="incompatible_changes_142">
|
|
<title>Incompatible Changes Introduced in Impala 1.4.2</title>
|
|
<conbody>
|
|
<p> None. Impala 1.4.2 is purely a bug-fix release. It does not include
|
|
any incompatible changes. </p>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="1.4.1" id="incompatible_changes_141">
|
|
<title>Incompatible Changes Introduced in Impala 1.4.1</title>
|
|
<conbody>
|
|
<p> None. Impala 1.4.1 is purely a bug-fix release. It does not include
|
|
any incompatible changes. </p>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="1.4.0" id="incompatible_changes_140">
|
|
<title>Incompatible Changes Introduced in Impala 1.4.0</title>
|
|
<prolog>
|
|
<metadata>
|
|
<data name="Category" value="Deprecated Features"/>
|
|
</metadata>
|
|
</prolog>
|
|
<conbody>
|
|
<ul>
|
|
<li>
|
|
<p> There is a slight change to required security privileges in the
|
|
Sentry framework. To create a new object, now you need the
|
|
<codeph>ALL</codeph> privilege on the parent object. For example,
|
|
to create a new table, view, or function requires having the
|
|
<codeph>ALL</codeph> privilege on the database containing the new
|
|
object. See <xref href="impala_authorization.xml"/> for a full list
|
|
of operations and associated privileges. </p>
|
|
</li>
|
|
<li>
|
|
<p> With the ability of <codeph>ORDER BY</codeph> queries to process
|
|
unlimited amounts of data with no <codeph>LIMIT</codeph> clause, the
|
|
query options <codeph>DEFAULT_ORDER_BY_LIMIT</codeph> and
|
|
<codeph>ABORT_ON_DEFAULT_LIMIT_EXCEEDED</codeph> are now
|
|
deprecated and have no effect. <ph audience="PDF">See <xref
|
|
href="impala_order_by.xml#order_by"/> for details about
|
|
improvements to the <codeph>ORDER BY</codeph> clause.</ph>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p> There are some changes to the list of reserved words. <ph
|
|
audience="PDF">See <xref
|
|
href="impala_reserved_words.xml#reserved_words"/> for the most
|
|
current list.</ph> The following keywords are new: </p>
|
|
<ul>
|
|
<li>
|
|
<codeph>API_VERSION</codeph>
|
|
</li>
|
|
<li>
|
|
<codeph>BINARY</codeph>
|
|
</li>
|
|
<li>
|
|
<codeph>CACHED</codeph>
|
|
</li>
|
|
<li>
|
|
<codeph>CLASS</codeph>
|
|
</li>
|
|
<li>
|
|
<codeph>PARTITIONS</codeph>
|
|
</li>
|
|
<li>
|
|
<codeph>PRODUCED</codeph>
|
|
</li>
|
|
<li>
|
|
<codeph>UNCACHED</codeph>
|
|
</li>
|
|
</ul>
|
|
<p> The following were formerly reserved keywords, but are no longer
|
|
reserved: </p>
|
|
<ul>
|
|
<li>
|
|
<codeph>COUNT</codeph>
|
|
</li>
|
|
<li>
|
|
<codeph>GROUP_CONCAT</codeph>
|
|
</li>
|
|
<li>
|
|
<codeph>NDV</codeph>
|
|
</li>
|
|
<li>
|
|
<codeph>SUM</codeph>
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li>
|
|
<p> The fix for issue <xref keyref="IMPALA-973">IMPALA-973</xref>
|
|
changes the behavior of the <codeph>INVALIDATE METADATA</codeph>
|
|
statement regarding nonexistent tables. In Impala 1.4.0 and higher,
|
|
the statement returns an error if the specified table is not in the
|
|
metastore database at all. It completes successfully if the
|
|
specified table is in the metastore database but not yet recognized
|
|
by Impala, for example if the table was created through Hive.
|
|
Formerly, you could issue this statement for a completely
|
|
nonexistent table, with no error. </p>
|
|
</li>
|
|
</ul>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="1.3.3" id="incompatible_changes_133">
|
|
<title>Incompatible Changes Introduced in Impala 1.3.3</title>
|
|
<conbody>
|
|
<p> No incompatible changes. The TLS/SSL security fix does not require any
|
|
change in the way you interact with Impala. </p>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="1.3.2" id="incompatible_changes_132">
|
|
<title>Incompatible Changes Introduced in Impala 1.3.2</title>
|
|
<conbody>
|
|
<p> With the fix for IMPALA-1019, you can use HDFS caching for files that
|
|
are accessed by Impala. </p>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="1.3.1" id="incompatible_changes_131">
|
|
<title>Incompatible Changes Introduced in Impala 1.3.1</title>
|
|
<conbody>
|
|
<ul>
|
|
<li>
|
|
<p conref="../shared/impala_common.xml#common/regexp_matching"/>
|
|
</li>
|
|
<li>
|
|
<p> The result set for the <codeph>SHOW FUNCTIONS</codeph> statement
|
|
includes a new first column, with the data type of the return value.
|
|
<ph audience="PDF">See <xref href="impala_show.xml#show"/> for
|
|
examples.</ph>
|
|
</p>
|
|
</li>
|
|
</ul>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="1.3.0" id="incompatible_changes_130">
|
|
<title>Incompatible Changes Introduced in Impala 1.3.0</title>
|
|
<conbody>
|
|
<ul>
|
|
<li>
|
|
<p> The <codeph>EXPLAIN_LEVEL</codeph> query option now accepts
|
|
numeric options from 0 (most concise) to 3 (most verbose), rather
|
|
than only 0 or 1. If you formerly used <codeph>SET
|
|
EXPLAIN_LEVEL=1</codeph> to get detailed explain plans, switch to
|
|
<codeph>SET EXPLAIN_LEVEL=3</codeph>. If you used the mnemonic
|
|
keyword (<codeph>SET EXPLAIN_LEVEL=verbose</codeph>), you do not
|
|
need to change your code because now level 3 corresponds to
|
|
<codeph>verbose</codeph>. <ph audience="PDF">See <xref
|
|
href="impala_explain_level.xml#explain_level"/> for details
|
|
about the allowed explain levels, and <xref
|
|
href="impala_explain_plan.xml#explain_plan"/> for usage
|
|
information.</ph>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p> The keyword <codeph>DECIMAL</codeph> is now a reserved word. If
|
|
you have any databases, tables, columns, or other objects already
|
|
named <codeph>DECIMAL</codeph>, quote any references to them using
|
|
backticks (<codeph>``</codeph>) to avoid name conflicts with the
|
|
keyword. <note> Although the <codeph>DECIMAL</codeph> keyword is a
|
|
reserved word, currently Impala does not support
|
|
<codeph>DECIMAL</codeph> as a data type for columns. </note>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p> The query option formerly named <codeph>YARN_POOL</codeph> is now
|
|
named <codeph>REQUEST_POOL</codeph> to reflect its broader use with
|
|
the Impala admission control feature. <ph audience="PDF">See <xref
|
|
href="impala_request_pool.xml#request_pool"/> for information
|
|
about the option, and <xref
|
|
href="impala_admission.xml#admission_control"/> for details
|
|
about its use with the admission control feature.</ph>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p> There are some changes to the list of reserved words. <ph
|
|
audience="PDF">See <xref
|
|
href="impala_reserved_words.xml#reserved_words"/> for the most
|
|
current list.</ph>
|
|
</p>
|
|
<ul>
|
|
<li>
|
|
<p> The names of aggregate functions are no longer reserved words,
|
|
so you can have databases, tables, columns, or other objects
|
|
named <codeph>AVG</codeph>, <codeph>MIN</codeph>, and so on
|
|
without any name conflicts. </p>
|
|
</li>
|
|
<li>
|
|
<p> The internal function names <codeph>DISTINCTPC</codeph> and
|
|
<codeph>DISTINCTPCSA</codeph> are no longer reserved words,
|
|
although <codeph>DISTINCT</codeph> is still a reserved word.
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p> The keywords <codeph>CLOSE_FN</codeph> and
|
|
<codeph>PREPARE_FN</codeph> are now reserved words. <ph
|
|
audience="PDF">See <xref
|
|
href="impala_create_function.xml#create_function"/> for
|
|
their role in the <codeph>CREATE FUNCTION</codeph> statement,
|
|
and <xref href="impala_udf.xml#udf_threads"/> for usage
|
|
information.</ph>
|
|
</p>
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li>
|
|
<p> The HDFS property
|
|
<codeph>dfs.client.file-block-storage-locations.timeout</codeph>
|
|
was renamed to
|
|
<codeph>dfs.client.file-block-storage-locations.timeout.millis</codeph>,
|
|
to emphasize that the unit of measure is milliseconds, not seconds.
|
|
Impala requires a timeout of at least 10 seconds, making the minimum
|
|
value for this setting 10000. If you are not using cluster
|
|
management software, you might need to edit the
|
|
<filepath>hdfs-site.xml</filepath> file in the Impala
|
|
configuration directory for the new name and minimum value. </p>
|
|
</li>
|
|
</ul>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="1.2.4" id="incompatible_changes_124">
|
|
<title>Incompatible Changes Introduced in Impala 1.2.4</title>
|
|
<conbody>
|
|
<p> There are no incompatible changes introduced in Impala 1.2.4. </p>
|
|
<p> Previously, after creating a table in Hive, you had to issue the
|
|
<codeph>INVALIDATE METADATA</codeph> statement with no table name, a
|
|
potentially expensive operation on clusters with many databases, tables,
|
|
and partitions. Starting in Impala 1.2.4, you can issue the statement
|
|
<codeph>INVALIDATE METADATA <varname>table_name</varname></codeph> for
|
|
a table newly created through Hive. Loading the metadata for only this
|
|
one table is faster and involves less network overhead. Therefore, you
|
|
might revisit your setup DDL scripts to add the table name to
|
|
<codeph>INVALIDATE METADATA</codeph> statements, in cases where you
|
|
create and populate the tables through Hive before querying them through
|
|
Impala. </p>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="1.2.3" id="incompatible_changes_123">
|
|
<title>Incompatible Changes Introduced in Impala 1.2.3</title>
|
|
<conbody>
|
|
<p> Because the feature set of Impala 1.2.3 is identical to Impala 1.2.2,
|
|
there are no new incompatible changes. See <xref
|
|
href="impala_incompatible_changes.xml#incompatible_changes_122"/> if
|
|
you are upgrading from Impala 1.2.1 or 1.1.x. </p>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="1.2.2" id="incompatible_changes_122">
|
|
<title>Incompatible Changes Introduced in Impala 1.2.2</title>
|
|
<conbody>
|
|
<p> The following changes to SQL syntax and semantics in Impala 1.2.2
|
|
could require updates to your SQL code, or schema objects such as tables
|
|
or views: </p>
|
|
<ul>
|
|
<li>
|
|
<p> With the addition of the <codeph>CROSS JOIN</codeph> keyword, you
|
|
might need to rewrite any queries that refer to a table named
|
|
<codeph>CROSS</codeph> or use the name <codeph>CROSS</codeph> as a
|
|
table alias: </p>
|
|
<codeblock>-- Formerly, 'cross' in this query was an alias for t1
|
|
-- and it was a normal join query.
|
|
-- In 1.2.2 and higher, CROSS JOIN is a keyword, so 'cross'
|
|
-- is not interpreted as a table alias, and the query
|
|
-- uses the special CROSS JOIN processing rather than a
|
|
-- regular join.
|
|
select * from t1 cross join t2...
|
|
|
|
-- Now if CROSS is used in other context such as a table or column name,
|
|
-- use backticks to escape it.
|
|
create table `cross` (x int);
|
|
select * from `cross`;</codeblock>
|
|
</li>
|
|
<li>
|
|
<p> Formerly, a <codeph>DROP DATABASE</codeph> statement in Impala
|
|
would not remove the top-level HDFS directory for that database. The
|
|
<codeph>DROP DATABASE</codeph> has been enhanced to remove that
|
|
directory. (You still need to drop all the tables inside the
|
|
database first; this change only applies to the top-level directory
|
|
for the entire database.) </p>
|
|
</li>
|
|
<li> The keyword <codeph>PARQUET</codeph> is introduced as a synonym for
|
|
<codeph>PARQUETFILE</codeph> in the <codeph>CREATE TABLE</codeph>
|
|
and <codeph>ALTER TABLE</codeph> statements, because that is the
|
|
common name for the file format. (As opposed to SequenceFile and
|
|
RCFile where the <q>File</q> suffix is part of the name.)
|
|
Documentation examples have been changed to prefer the new shorter
|
|
keyword. The <codeph>PARQUETFILE</codeph> keyword is still available
|
|
for backward compatibility with older Impala versions. </li>
|
|
<li> New overloads are available for several operators and built-in
|
|
functions, allowing you to insert their result values into smaller
|
|
numeric columns such as <codeph>INT</codeph>,
|
|
<codeph>SMALLINT</codeph>, <codeph>TINYINT</codeph>, and
|
|
<codeph>FLOAT</codeph> without using a <codeph>CAST()</codeph> call.
|
|
If you remove the <codeph>CAST()</codeph> calls from
|
|
<codeph>INSERT</codeph> statements, those statements might not work
|
|
with earlier versions of Impala. </li>
|
|
</ul>
|
|
<p> Because many users are likely to upgrade straight from Impala 1.x to
|
|
Impala 1.2.2, also read <xref
|
|
href="impala_incompatible_changes.xml#incompatible_changes_121"/> for
|
|
things to note about upgrading to Impala 1.2.x in general. </p>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="1.2.1" id="incompatible_changes_121">
|
|
<title>Incompatible Changes Introduced in Impala 1.2.1</title>
|
|
<conbody>
|
|
<p> The following changes to SQL syntax and semantics in Impala 1.2.1
|
|
could require updates to your SQL code, or schema objects such as tables
|
|
or views: </p>
|
|
<ul>
|
|
<li>
|
|
<p conref="../shared/impala_common.xml#common/null_sorting_change"/>
|
|
<p audience="PDF"> See <xref href="impala_literals.xml#null"/> for
|
|
more information. </p>
|
|
</li>
|
|
</ul>
|
|
<p> The new <cmdname>catalogd</cmdname> service might require changes to
|
|
any user-written scripts that stop, start, or restart Impala services,
|
|
install or upgrade Impala packages, or issue <codeph>REFRESH</codeph> or
|
|
<codeph>INVALIDATE METADATA</codeph> statements: </p>
|
|
<ul conref="../shared/impala_common.xml#common/catalogd_xrefs">
|
|
<li/>
|
|
</ul>
|
|
</conbody>
|
|
</concept>
|
|
<concept rev="1.2" id="incompatible_changes_120">
|
|
<title>Incompatible Changes Introduced in Impala 1.2.0 (Beta)</title>
|
|
<conbody>
|
|
<p> There are no incompatible changes to SQL syntax in Impala 1.2.0
|
|
(beta). </p>
|
|
<p> The new <cmdname>catalogd</cmdname> service might require changes to
|
|
any user-written scripts that stop, start, or restart Impala services,
|
|
install or upgrade Impala packages, or issue <codeph>REFRESH</codeph> or
|
|
<codeph>INVALIDATE METADATA</codeph> statements: </p>
|
|
<ul conref="../shared/impala_common.xml#common/catalogd_xrefs">
|
|
<li/>
|
|
</ul>
|
|
<p> The new resource management feature interacts with both YARN and Llama
|
|
services. <ph audience="PDF">See <xref
|
|
href="impala_resource_management.xml#resource_management"/> for
|
|
usage information for Impala resource management.</ph>
|
|
</p>
|
|
</conbody>
|
|
</concept>
|
|
<concept id="incompatible_changes_111">
|
|
<title>Incompatible Changes Introduced in Impala 1.1.1</title>
|
|
<conbody>
|
|
<p> There are no incompatible changes in Impala 1.1.1. </p>
|
|
<!-- These couple of paragraphs were originally intended to be conref'ed from the Parquet section of Installing/Using. -->
|
|
<!-- But conbodydiv tag too restrictive, can't have just paragraphs and codeblocks inside. -->
|
|
<!-- So I will physically copy the info for the time being. -->
|
|
<!-- Also copying it under the Upgrading topic. -->
|
|
<!-- <conbodydiv conref="impala_parquet.xml#upgrade_parquet_metadata"/> -->
|
|
<p> Previously, it was not possible to create Parquet data through Impala
|
|
and reuse that table within Hive. Now that Parquet support is available
|
|
for Hive 10, reusing existing Impala Parquet data files in Hive requires
|
|
updating the table metadata. Use the following command if you are
|
|
already running Impala 1.1.1: </p>
|
|
<codeblock>ALTER TABLE <varname>table_name</varname> SET FILEFORMAT PARQUETFILE;
|
|
</codeblock>
|
|
<p> If you are running a level of Impala that is older than 1.1.1, do the
|
|
metadata update through Hive: </p>
|
|
<codeblock>ALTER TABLE <varname>table_name</varname> SET SERDE 'parquet.hive.serde.ParquetHiveSerDe';
|
|
ALTER TABLE <varname>table_name</varname> SET FILEFORMAT
|
|
INPUTFORMAT "parquet.hive.DeprecatedParquetInputFormat"
|
|
OUTPUTFORMAT "parquet.hive.DeprecatedParquetOutputFormat";
|
|
</codeblock>
|
|
<p> Impala 1.1.1 and higher can reuse Parquet data files created by Hive,
|
|
without any action required. </p>
|
|
<p> As usual, make sure to upgrade the Impala LZO package to the latest
|
|
level at the same time as you upgrade the Impala server. </p>
|
|
</conbody>
|
|
</concept>
|
|
<concept id="incompatible_changes_11">
|
|
<title>Incompatible Change Introduced in Impala 1.1</title>
|
|
<conbody>
|
|
<ul>
|
|
<li>
|
|
<p> The <codeph>REFRESH</codeph> statement now requires a table name;
|
|
in Impala 1.0, the table name was optional. This syntax change is
|
|
part of the internal rework to make <codeph>REFRESH</codeph> a true
|
|
Impala SQL statement so that it can be called through the JDBC and
|
|
ODBC APIs. <codeph>REFRESH</codeph> now reloads the metadata
|
|
immediately, rather than marking it for update the next time any
|
|
affected table is accessed. The previous behavior, where omitting
|
|
the table name caused a refresh of the entire Impala metadata
|
|
catalog, is available through the new <codeph>INVALIDATE
|
|
METADATA</codeph> statement. <codeph>INVALIDATE METADATA</codeph>
|
|
can be specified with a table name to affect a single table, or
|
|
without a table name to affect the entire metadata catalog; the
|
|
relevant metadata is reloaded the next time it is requested during
|
|
the processing for a SQL statement. See <xref
|
|
href="impala_refresh.xml#refresh"/> and <xref
|
|
href="impala_invalidate_metadata.xml#invalidate_metadata"/> for
|
|
the latest details about these statements. </p>
|
|
</li>
|
|
</ul>
|
|
</conbody>
|
|
</concept>
|
|
<concept id="incompatible_changes_10">
|
|
<title>Incompatible Changes Introduced in Impala 1.0</title>
|
|
<conbody>
|
|
<ul>
|
|
<li> If you use LZO-compressed text files, when you upgrade Impala to
|
|
version 1.0, also update the Impala LZO package to the latest level.
|
|
See <xref href="impala_txtfile.xml#lzo"/> for details. </li>
|
|
</ul>
|
|
</conbody>
|
|
</concept>
|
|
</concept>
|