jprdonnelly/impala
1
0
Fork 0
mirror of https://github.com/apache/impala.git synced 2026-01-31 09:00:19 -05:00
Code Issues Projects Releases Wiki Activity
Files
c802be42b66cf69a362f7d2c89175932bbe226cf
impala/testdata/workloads/functional-query/queries/QueryTest/nested-struct-in-select-list.test
Daniel Becker c802be42b6 IMPALA-10839: NULL values are displayed on a wrong level for nested structs (ORC) 
When querying a non-toplevel nested struct from an ORC file, the NULL
values are displayed at an incorrect level. E.g.:

select id, outer_struct.inner_struct3 from
functional_orc_def.complextypes_nested_structs where id >= 4;
+----+----------------------------+
| id | outer_struct.inner_struct3 |
+----+----------------------------+
| 4  | {"s":{"i":null,"s":null}}  |
| 5  | {"s":null}                 |
+----+----------------------------+

However, in the first row it is expected that 's' should be null and not
its members; in the second row the result should be 'NULL', i.e.
'outer_struct.inner_struct3' is null.
For reference see what is returned when querying 'outer_struct' instead
of 'outer_struct.inner_struct3':

+----+-------------------------------------------------------------------------------------------------------------------------------+
| 4  | {"str":"","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":1,"str":"string"},"inner_struct3":{"s":null}} |
| 5  | {"str":null,"inner_struct1":null,"inner_struct2":null,"inner_struct3":null}                                                   |
+----+-------------------------------------------------------------------------------------------------------------------------------+

The problem comes from the incorrect handling of the different depths of
the following trees:
 - the ORC type hierarchy (schema)
 - the tuple descriptor / slot descriptor hierarchy
as the ORC type hierarchy contains a node for every level in the schema
but the tuple/slot descriptor hierarchy omits the levels of structs that
are not in the select list (but an ancestor of theirs is), as these
structs are not materialised.

In the case of the example query, the two hierarchies are the following:
ORC:
 root --> outer_struct -> inner_struct3 -> s --> i
      |                                      \-> s
      \-> id
Tuple/slot descriptors:
 main_tuple --> inner_struct3 -> s --> i
            |                      \-> s
            \-> id

We create 'OrcColumnReader's for each node in the ORC type tree. Each
OrcColumnReader is assigned an ORC type node and a slot descriptor. The
incorrect behaviour comes from the incorrect pairing of ORC type nodes
with slot descriptors.

The old behaviour is described below:
Starting from the root, going along a path in both trees (for example
the path leading to outer_struct.inner_struct3.s.i), for each step we
consume a level in both trees until no more nodes remain in the
tuple/slot desc tree, and then we pair the last element from that tree
with the remaining ORC type node(s).

In the example, we get the following pairs:
(root, main_tuple) -> (outer_struct, inner_struct3) ->
(inner_struct3, s) -> (s, i) -> (i, i)

When we run out of structs in the tuple/slot desc tree, we still create
OrcStructReaders (because the ORC type is still a struct, but the slot
descriptor now refers to an INT), but we mark them incorrectly as
non-materialised.

Also, the OrcStructReaders for non-materialised structs do not need to
check for null-ness as they are not present in the select list, only
their descendants, and the ORC batch object stores null information also
for the descendants of null values.

Let's look at the row with id 4 in the example:
Because of the bug, the non-materialising OrcStructReader appears at the
level of the (s, i) pair, so the 's' struct is not checked for
null-ness, although it is actually null. One level lower, for 'i' (and
the inner 's' string field), the ORC batch object tells us that the
values are null (because their parent is). Therefore the nulls appear
one level lower than they should.

The correct behaviour is that ORC type nodes are paired with slot
descriptors if either
 - the ORC type node matches the slot descriptor (they refer to the same
   node in the schema) or
 - the slot descriptor is a descendant of the schema node that the ORC
   type node refers to.

This patch fixes the incorrect pairing of ORC types and slot
descriptors, so we have the following pairs:
(root, main_tuple) -> (outer_struct, main_tuple) ->
(inner_struct3, inner_struct3) -> (s, s) -> (i, i)

In this case the OrcStructReader for the pair (outer_struct, main_tuple)
becomes non-materialising and the one for (s, s) will be materialising,
so the 's' struct will also be null-checked, recognising null-ness at
the correct level.

This commit also fixes some comments in be/src/exec/orc-column-readers.h
and be/src/exec/hdfs-orc-scanner.h mentioning the field
HdfsOrcScanner::col_id_path_map_, which has been removed by
"IMPALA-10485: part(1): make ORC column reader creation independent of
schema resolution".

Testing:
  - added tests to
    testdata/workloads/functional-query/queries/QueryTest/nested-struct-in-select-list.test
    that query various levels of the struct 'outer_struct' to check that
    NULLs are at the correct level.

Change-Id: Iff5034e7bdf39c036aecc491fbd324e29150f040
Reviewed-on: http://gerrit.cloudera.org:8080/18403
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
2022-04-21 13:59:17 +00:00

183 lines
9.1 KiB
Plaintext
Raw Blame History

====
---- QUERY
# Select a struct that contains multiple structs.
select id, outer_struct
from functional_orc_def.complextypes_nested_structs;
---- RESULTS
1,'{"str":"somestr1","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":333222111,"str":"somestr3"},"inner_struct3":{"s":{"i":112288,"s":null}}}'
2,'{"str":"str","inner_struct1":null,"inner_struct2":{"i":100,"str":"str3"},"inner_struct3":{"s":{"i":321,"s":"dfgs"}}}'
3,'NULL'
4,'{"str":"","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":1,"str":"string"},"inner_struct3":{"s":null}}'
5,'{"str":null,"inner_struct1":null,"inner_struct2":null,"inner_struct3":null}'
---- TYPES
INT,STRING
====
---- QUERY
# Select a struct that contains multiple structs using a filter on a non-struct field.
select id, outer_struct
from functional_orc_def.complextypes_nested_structs
where id > 2;
---- RESULTS
3,'NULL'
4,'{"str":"","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":1,"str":"string"},"inner_struct3":{"s":null}}'
5,'{"str":null,"inner_struct1":null,"inner_struct2":null,"inner_struct3":null}'
---- TYPES
INT,STRING
====
---- QUERY
# Select a struct that contains multiple structs using a filter on a struct field.
select id, outer_struct
from functional_orc_def.complextypes_nested_structs
where length(outer_struct.str) > 3;
---- RESULTS
1,'{"str":"somestr1","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":333222111,"str":"somestr3"},"inner_struct3":{"s":{"i":112288,"s":null}}}'
---- TYPES
INT,STRING
====
---- QUERY
# Select a nested struct with an order by.
select id, outer_struct
from functional_orc_def.complextypes_nested_structs
order by id;
---- RESULTS: VERIFY_IS_EQUAL_SORTED
1,'{"str":"somestr1","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":333222111,"str":"somestr3"},"inner_struct3":{"s":{"i":112288,"s":null}}}'
2,'{"str":"str","inner_struct1":null,"inner_struct2":{"i":100,"str":"str3"},"inner_struct3":{"s":{"i":321,"s":"dfgs"}}}'
3,'NULL'
4,'{"str":"","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":1,"str":"string"},"inner_struct3":{"s":null}}'
5,'{"str":null,"inner_struct1":null,"inner_struct2":null,"inner_struct3":null}'
---- TYPES
INT,STRING
====
---- QUERY
# Select a nested struct with an order by.
select id, outer_struct
from functional_orc_def.complextypes_nested_structs
order by id desc;
---- RESULTS: VERIFY_IS_EQUAL_SORTED
5,'{"str":null,"inner_struct1":null,"inner_struct2":null,"inner_struct3":null}'
4,'{"str":"","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":1,"str":"string"},"inner_struct3":{"s":null}}'
3,'NULL'
2,'{"str":"str","inner_struct1":null,"inner_struct2":{"i":100,"str":"str3"},"inner_struct3":{"s":{"i":321,"s":"dfgs"}}}'
1,'{"str":"somestr1","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":333222111,"str":"somestr3"},"inner_struct3":{"s":{"i":112288,"s":null}}}'
---- TYPES
INT,STRING
====
---- QUERY
# Select the same nested struct multiple times in one query.
select id, outer_struct, outer_struct
from functional_orc_def.complextypes_nested_structs;
---- RESULTS
1,'{"str":"somestr1","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":333222111,"str":"somestr3"},"inner_struct3":{"s":{"i":112288,"s":null}}}','{"str":"somestr1","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":333222111,"str":"somestr3"},"inner_struct3":{"s":{"i":112288,"s":null}}}'
2,'{"str":"str","inner_struct1":null,"inner_struct2":{"i":100,"str":"str3"},"inner_struct3":{"s":{"i":321,"s":"dfgs"}}}','{"str":"str","inner_struct1":null,"inner_struct2":{"i":100,"str":"str3"},"inner_struct3":{"s":{"i":321,"s":"dfgs"}}}'
3,'NULL','NULL'
4,'{"str":"","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":1,"str":"string"},"inner_struct3":{"s":null}}','{"str":"","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":1,"str":"string"},"inner_struct3":{"s":null}}'
5,'{"str":null,"inner_struct1":null,"inner_struct2":null,"inner_struct3":null}','{"str":null,"inner_struct1":null,"inner_struct2":null,"inner_struct3":null}'
---- TYPES
INT,STRING,STRING
====
---- QUERY
# Select the same nested struct multiple times in one query and order the results.
select id, outer_struct, outer_struct
from functional_orc_def.complextypes_nested_structs
order by id desc;
---- RESULTS: VERIFY_IS_EQUAL_SORTED
5,'{"str":null,"inner_struct1":null,"inner_struct2":null,"inner_struct3":null}','{"str":null,"inner_struct1":null,"inner_struct2":null,"inner_struct3":null}'
4,'{"str":"","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":1,"str":"string"},"inner_struct3":{"s":null}}','{"str":"","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":1,"str":"string"},"inner_struct3":{"s":null}}'
3,'NULL','NULL'
2,'{"str":"str","inner_struct1":null,"inner_struct2":{"i":100,"str":"str3"},"inner_struct3":{"s":{"i":321,"s":"dfgs"}}}','{"str":"str","inner_struct1":null,"inner_struct2":{"i":100,"str":"str3"},"inner_struct3":{"s":{"i":321,"s":"dfgs"}}}'
1,'{"str":"somestr1","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":333222111,"str":"somestr3"},"inner_struct3":{"s":{"i":112288,"s":null}}}','{"str":"somestr1","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":333222111,"str":"somestr3"},"inner_struct3":{"s":{"i":112288,"s":null}}}'
---- TYPES
INT,STRING,STRING
====
---- QUERY
# Similar to the above query but here the 'id' field is not in the select list but still
# used in the order by.
select outer_struct, outer_struct
from functional_orc_def.complextypes_nested_structs
order by id desc;
---- RESULTS: VERIFY_IS_EQUAL_SORTED
'{"str":null,"inner_struct1":null,"inner_struct2":null,"inner_struct3":null}','{"str":null,"inner_struct1":null,"inner_struct2":null,"inner_struct3":null}'
'{"str":"","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":1,"str":"string"},"inner_struct3":{"s":null}}','{"str":"","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":1,"str":"string"},"inner_struct3":{"s":null}}'
'NULL','NULL'
'{"str":"str","inner_struct1":null,"inner_struct2":{"i":100,"str":"str3"},"inner_struct3":{"s":{"i":321,"s":"dfgs"}}}','{"str":"str","inner_struct1":null,"inner_struct2":{"i":100,"str":"str3"},"inner_struct3":{"s":{"i":321,"s":"dfgs"}}}'
'{"str":"somestr1","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":333222111,"str":"somestr3"},"inner_struct3":{"s":{"i":112288,"s":null}}}','{"str":"somestr1","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":333222111,"str":"somestr3"},"inner_struct3":{"s":{"i":112288,"s":null}}}'
---- TYPES
STRING,STRING
====
---- QUERY
# WITH clause creates an inline view containing a nested struct.
with sub as (
select id, outer_struct from functional_orc_def.complextypes_nested_structs)
select sub.id, sub.outer_struct from sub;
---- RESULTS
1,'{"str":"somestr1","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":333222111,"str":"somestr3"},"inner_struct3":{"s":{"i":112288,"s":null}}}'
2,'{"str":"str","inner_struct1":null,"inner_struct2":{"i":100,"str":"str3"},"inner_struct3":{"s":{"i":321,"s":"dfgs"}}}'
3,'NULL'
4,'{"str":"","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":1,"str":"string"},"inner_struct3":{"s":null}}'
5,'{"str":null,"inner_struct1":null,"inner_struct2":null,"inner_struct3":null}'
---- TYPES
INT,STRING
====
---- QUERY
# WITH clause creates an inline view containing a nested struct. Also has a filter on
# the inline view and ordering by a non-complex item from the view.
with sub as (
select id, outer_struct
from functional_orc_def.complextypes_nested_structs
where length(outer_struct.str) > 3)
select sub.id, sub.outer_struct from sub order by sub.id desc;
---- RESULTS
1,'{"str":"somestr1","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":333222111,"str":"somestr3"},"inner_struct3":{"s":{"i":112288,"s":null}}}'
---- TYPES
INT,STRING
====
---- QUERY
# Checks that "SELECT nested_struct.* ..." omits the nested structs from the output.
select id, outer_struct.* from functional_orc_def.complextypes_nested_structs;
---- RESULTS
1,'somestr1'
2,'str'
3,'NULL'
4,''
5,'NULL'
---- TYPES
INT,STRING
====
---- QUERY
# IMPALA-10839: Display nulls at the correct level.
select id, outer_struct.inner_struct3 from
functional_orc_def.complextypes_nested_structs;
---- RESULTS
1,'{"s":{"i":112288,"s":null}}'
2,'{"s":{"i":321,"s":"dfgs"}}'
3,'NULL'
4,'{"s":null}'
5,'NULL'
---- TYPES
INT,STRING
====
---- QUERY
# IMPALA-10839: Display nulls at the correct level.
select id, outer_struct.inner_struct3.s from
functional_orc_def.complextypes_nested_structs;
---- RESULTS
1,'{"i":112288,"s":null}'
2,'{"i":321,"s":"dfgs"}'
3,'NULL'
4,'NULL'
5,'NULL'
---- TYPES
INT,STRING
====
---- QUERY
# Subquery that returns a complex type is not supported.
# IMPALA-9500
select outer_struct
from functional_orc_def.complextypes_nested_structs
where outer_struct in
(select outer_struct from functional_orc_def.complextypes_nested_structs);
---- CATCH
AnalysisException: A subquery can't return complex types. (SELECT outer_struct FROM functional_orc_def.complex
types_nested_structs)
====
Reference in New Issue View Git Blame Copy Permalink