mirror of
https://github.com/apache/impala.git
synced 2025-12-23 21:08:39 -05:00
df528fe2b1
This patch provides an unnest implementation for arrays where unnesting
multiple arrays in one query results the items of the arrays being
zipped together instead of joining. There are two different syntaxes
introduced for this purpose:
1: ISO:SQL 2016 compliant syntax:
SELECT a1.item, a2.item
FROM complextypes_arrays t, UNNEST(t.arr1, t.arr2) AS (a1, a2);
2: Postgres compatible syntax:
SELECT UNNEST(arr1), UNNEST(arr2) FROM complextypes_arrays;
Let me show the expected behaviour through the following example:
Inputs: arr1: {1,2,3}, arr2: {11, 12}
After running any of the above queries we expect the following output:
===============
| arr1 | arr2 |
===============
| 1 | 11 |
| 2 | 12 |
| 3 | NULL |
===============
Expected behaviour:
- When unnesting multiple arrays with zipping unnest then the 'i'th
item of one array will be put next to the 'i'th item of the other
arrays in the results.
- In case the size of the arrays is not the same then the shorter
arrays will be filled with NULL values up to the size of the longest
array.
On a sidenote, UNNEST is added to Impala's SQL language as a new
keyword. This might interfere with use cases where a resource (db,
table, column, etc.) is named "UNNEST".
Restrictions:
- It is not allowed to have WHERE filters on an unnested item of an
array in the same SELECT query. E.g. this is not allowed:
SELECT arr1.item
FROM complextypes_arrays t, UNNEST(t.arr1) WHERE arr1.item < 5;
Note, that it is allowed to have an outer SELECT around the one
doing unnests and have a filter there on the unnested items.
- If there is an outer SELECT filtering on the unnested array's items
from the inner SELECT then these predicates won't be pushed down to
the SCAN node. They are rather evaluated in the UNNEST node to
guarantee result correctness after unnesting.
Note, this restriction is only active when there are multiple arrays
being unnested, or in other words when zipping unnest logic is
required to produce results.
- It's not allowed to do a zipping and a (traditional) joining unnest
together in one SELECT query.
- It's not allowed to perform zipping unnests on arrays from different
tables.
Testing:
- Added a bunch of E2E tests to the test suite to cover both syntaxes.
- Did a manual test run on a table with 1000 rows, 3 array columns
with size of around 5000 items in each array. I did an unnest on all
three arrays in one query to see if there are any crashes or
suspicious slowness when running on this scale.
Change-Id: Ic58ff6579ecff03962e7a8698edfbe0684ce6cf7
Reviewed-on: http://gerrit.cloudera.org:8080/17983
Reviewed-by: Csaba Ringhofer <csringhofer@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
58 lines
1.0 KiB
Plaintext
58 lines
1.0 KiB
Plaintext
====
|
|
---- QUERY
|
|
create view view_unnested_arrays as
|
|
select id, unnest(arr1) as arr1_unnested, unnest(arr2) as arr2_unnested
|
|
from functional_orc_def.complextypes_arrays;
|
|
---- RESULTS
|
|
'View has been created.'
|
|
====
|
|
---- QUERY
|
|
# Query unnested array items from a view that does the unnesting itself.
|
|
select id, arr1_unnested, arr2_unnested from view_unnested_arrays;
|
|
---- RESULTS
|
|
1,1,'one'
|
|
1,2,'two'
|
|
1,3,'three'
|
|
1,4,'four'
|
|
1,5,'five'
|
|
2,1,'one'
|
|
2,NULL,'two'
|
|
2,3,'three'
|
|
2,4,'NULL'
|
|
2,5,'five'
|
|
3,10,'ten'
|
|
3,9,'NULL'
|
|
3,8,'NULL'
|
|
4,10,'ten'
|
|
4,NULL,'nine'
|
|
4,NULL,'eight'
|
|
5,10,'ten'
|
|
5,NULL,'eleven'
|
|
5,12,'twelve'
|
|
5,NULL,'thirteen'
|
|
6,NULL,'str1'
|
|
6,NULL,'str2'
|
|
7,1,'NULL'
|
|
7,2,'NULL'
|
|
9,NULL,'str1'
|
|
9,NULL,'str2'
|
|
10,1,'NULL'
|
|
10,2,'NULL'
|
|
10,3,'NULL'
|
|
---- TYPES
|
|
INT,INT,STRING
|
|
====
|
|
---- QUERY
|
|
# Same as above but there is a filter in the outer select.
|
|
select id, arr1_unnested, arr2_unnested from view_unnested_arrays
|
|
where arr1_unnested > 5;
|
|
---- RESULTS
|
|
3,10,'ten'
|
|
3,9,'NULL'
|
|
3,8,'NULL'
|
|
4,10,'ten'
|
|
5,10,'ten'
|
|
5,12,'twelve'
|
|
---- TYPES
|
|
INT,INT,STRING
|