Daniel Becker
77dc20264c
IMPALA-11687: Select * with EXPAND_COMPLEX_TYPES=1 and explicit complex types fails
...
If EXPAND_COMPLEX_TYPES is set to true, some queries that combine star
expressions and explicitly given complex columns fail:
select outer_struct, * from
functional_orc_def.complextypes_nested_structs;
ERROR: IllegalStateException: Illegal reference to non-materialized
slot: tid=1 sid=1
select *, outer_struct.str from
functional_orc_def.complextypes_nested_structs;
ERROR: IllegalStateException: null
Having two stars in a table with complex columns also fails.
select *, * from functional_orc_def.complextypes_nested_structs;
ERROR: IllegalStateException: Illegal reference to non-materialized
slot: tid=6 sid=13
The error is because of this line in 'SelectStmt.addStarResultExpr()':
8e350d0a8a/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java (L811)
What we want to do is create 'SlotRef's for the struct children
(recursively) but 'reExpandStruct()' also creates new 'SlotDescriptor's
for the children. The new 'SlotDescriptor's are redundant and are not
inserted into the tree which can leave them unmaterialised or without a
correct memory layout.
The solution is to only create the 'SlotRef's for the struct children
without creating new 'SlotDescriptor's. This leads us to another
problem:
- for structs, it is 'SlotRef.analyzeImpl()' that creates the child
'SlotRef's
- the constructor 'SlotRef(SlotDescriptor desc)' sets 'isAnalyzed_' to
true.
Before structs were allowed, this was correct but now struct-typed
'SlotRef's created with the above constructor are counted as analysed
but lack child expressions, which would have been added if 'analyze()'
had been called on them. This essentially violates the contract of this
constructor.
This commit modifies 'SlotRef(SlotDescriptor desc)' so that child
expressions are generated for structs, restoring the correct semantics
of this constructor. After this, it is no longer necessary to call
'reExpandStruct()' in 'SelectStmt.addStarResultExpr()'.
Testing:
- Added the failing test cases and a few variations of them to
nested-types-star-expansion.test
Change-Id: Ia8cf53b0a7409faca668713228bfef275f3833f9
Reviewed-on: http://gerrit.cloudera.org:8080/19171
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com >
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com >
2022-11-04 04:33:02 +00:00
..
2019-08-13 18:55:09 +00:00
2020-09-01 22:27:27 +00:00
2020-05-12 23:29:04 +00:00
2019-07-27 13:45:51 +00:00
2021-01-05 21:31:05 +00:00
2020-08-12 17:45:50 +00:00
2020-05-18 16:51:42 +00:00
2019-07-27 13:45:51 +00:00
2019-11-07 14:30:47 +00:00
2019-07-03 20:46:45 +00:00
2020-05-20 21:00:44 +00:00
2020-05-20 21:00:44 +00:00
2020-05-20 21:00:44 +00:00
2020-09-02 13:29:25 +00:00
2022-11-03 00:52:08 +00:00
2022-09-28 05:46:03 +00:00
2021-03-12 14:23:04 +00:00
2021-03-12 14:23:04 +00:00
2016-09-13 21:57:36 +00:00
2022-06-19 04:35:04 +00:00
2019-10-16 02:04:12 +00:00
2020-07-08 22:59:57 +00:00
2021-12-16 11:49:15 +00:00
2018-09-29 11:59:03 +00:00
2017-02-14 05:56:33 +00:00
2020-05-12 23:29:04 +00:00
2021-06-10 15:46:56 +00:00
2022-04-07 16:07:56 +00:00
2018-08-08 20:51:45 +00:00
2022-01-09 15:39:35 +00:00
2021-10-23 00:08:14 +00:00
2022-02-23 20:09:10 +00:00
2019-09-27 17:18:35 +00:00
2019-09-27 17:18:35 +00:00
2018-05-15 22:23:14 +00:00
2022-08-25 15:12:38 +00:00
2021-02-18 14:31:51 +00:00
2021-04-30 23:36:31 +00:00
2018-10-09 00:45:10 +00:00
2018-04-11 02:21:48 +00:00
2018-11-17 01:48:05 +00:00
2019-08-13 18:55:09 +00:00
2021-03-02 04:58:51 +00:00
2020-05-12 23:29:04 +00:00
2020-05-12 23:29:04 +00:00
2021-04-02 21:50:17 +00:00
2019-08-06 16:38:07 +00:00
2020-05-12 23:29:04 +00:00
2020-05-12 23:29:04 +00:00
2021-01-21 19:35:26 +00:00
2018-12-08 00:21:55 +00:00
2021-09-14 21:21:47 +00:00
2022-04-14 11:56:20 +00:00
2020-08-13 03:10:16 +00:00
2021-07-06 18:35:30 +00:00
2020-06-24 03:47:18 +00:00
2021-05-07 14:11:08 +00:00
2020-06-24 03:47:18 +00:00
2020-12-11 19:51:28 +00:00
2020-12-11 19:51:28 +00:00
2019-05-05 02:11:55 +00:00
2021-04-20 22:21:32 +00:00
2021-09-15 13:58:23 +00:00
2021-09-15 13:58:23 +00:00
2021-09-15 13:58:23 +00:00
2021-09-15 13:58:23 +00:00
2020-06-24 16:51:49 +00:00
2020-12-11 19:51:28 +00:00
2020-12-11 19:51:28 +00:00
2020-09-28 14:02:35 +00:00
2020-07-08 19:32:15 +00:00
2020-08-13 18:58:23 +00:00
2018-01-25 04:33:11 +00:00
2019-01-14 09:07:03 +00:00
2019-04-23 13:33:57 +00:00
2021-03-23 22:52:38 +00:00
2018-10-09 00:45:10 +00:00
2022-10-17 21:59:57 +00:00
2022-03-04 16:49:22 +00:00
2018-04-11 02:21:48 +00:00
2018-04-11 02:21:48 +00:00
2021-07-06 18:35:30 +00:00
2020-04-02 12:01:41 +00:00
2020-06-05 17:43:32 +00:00
2019-05-15 22:34:28 +00:00
2018-04-12 22:01:35 +00:00
2018-02-02 01:13:08 +00:00
2020-10-21 05:20:33 +00:00
2020-07-31 17:23:45 +00:00
2018-02-15 10:55:55 +00:00
2018-02-15 10:55:55 +00:00
2021-06-14 17:13:25 +00:00
2022-02-12 06:09:59 +00:00
2022-02-12 06:09:59 +00:00
2022-10-17 21:59:57 +00:00
2022-02-16 13:28:27 +00:00
2022-02-11 07:01:58 +00:00
2020-02-22 02:02:56 +00:00
2020-08-12 17:45:50 +00:00
2020-09-30 21:01:54 +00:00
2021-03-18 19:35:58 +00:00
2020-07-14 12:53:51 +00:00
2018-12-11 02:47:16 +00:00
2022-08-19 13:55:42 +00:00
2022-11-03 05:48:17 +00:00
2020-07-14 03:13:18 +00:00
2019-02-07 23:05:30 +00:00
2020-05-12 23:29:04 +00:00
2020-05-12 23:29:04 +00:00
2022-07-18 19:02:34 +00:00
2022-08-19 13:55:42 +00:00
2019-04-23 13:33:57 +00:00
2020-05-12 23:29:04 +00:00
2017-10-10 01:30:33 +00:00
2017-09-21 17:38:08 +00:00
2020-12-11 19:51:28 +00:00
2018-07-03 23:49:44 +00:00
2019-06-05 14:23:23 +00:00
2021-03-02 04:58:51 +00:00
2017-05-09 01:36:46 +00:00
2020-01-17 18:54:33 +00:00
2019-09-27 17:18:35 +00:00
2020-02-14 04:26:16 +00:00
2020-02-11 15:08:39 +00:00
2022-10-17 19:37:15 +00:00
2022-04-28 17:48:23 +00:00
2022-11-02 22:14:47 +00:00
2022-10-21 22:06:07 +00:00
2022-09-28 22:35:48 +00:00
2022-09-28 22:35:48 +00:00
2022-09-28 22:35:48 +00:00
2022-01-23 20:55:00 +00:00
2022-11-02 22:14:47 +00:00
2022-09-28 22:35:48 +00:00
2022-11-02 22:14:47 +00:00
2022-07-14 13:06:04 +00:00
2022-06-10 13:01:28 +00:00
2021-11-04 17:55:21 +00:00
2022-09-08 18:03:37 +00:00
2022-11-02 22:14:47 +00:00
2022-09-28 22:35:48 +00:00
2022-09-28 22:35:48 +00:00
2021-05-20 19:19:50 +00:00
2022-08-01 13:36:51 +00:00
2022-11-02 22:14:47 +00:00
2022-08-10 14:34:04 +00:00
2022-11-02 22:14:47 +00:00
2022-06-27 15:29:11 +00:00
2020-09-09 13:26:42 +00:00
2022-09-28 22:35:48 +00:00
2020-11-23 21:47:55 +00:00
2022-09-26 15:49:22 +00:00
2022-09-26 15:49:22 +00:00
2021-07-15 15:15:07 +00:00
2022-11-02 22:14:47 +00:00
2022-10-21 22:06:07 +00:00
2022-10-21 22:06:07 +00:00
2021-09-08 18:38:37 +00:00
2021-09-08 18:38:37 +00:00
2022-03-03 00:21:06 +00:00
2018-08-09 19:18:08 +00:00
2021-04-20 22:21:32 +00:00
2022-06-04 03:36:11 +00:00
2020-05-11 19:32:08 +00:00
2020-12-11 19:51:28 +00:00
2020-12-11 19:51:28 +00:00
2022-04-04 20:28:07 +00:00
2019-06-05 11:15:04 +00:00
2020-12-11 19:51:28 +00:00
2020-12-11 19:51:28 +00:00
2020-05-11 19:32:08 +00:00
2018-11-06 00:00:12 +00:00
2020-12-11 19:51:28 +00:00
2020-07-31 17:23:45 +00:00
2022-08-19 13:55:42 +00:00
2019-10-22 04:21:51 +00:00
2022-02-04 17:55:47 +00:00
2021-06-10 15:46:56 +00:00
2022-07-26 18:21:48 +00:00
2020-04-01 15:48:36 +00:00
2020-04-01 15:48:36 +00:00
2020-11-03 20:05:20 +00:00
2018-05-11 22:41:49 +00:00
2020-06-05 00:40:33 +00:00
2018-04-27 21:55:11 +00:00
2022-07-16 06:06:46 +00:00
2020-03-18 18:05:34 +00:00
2020-04-01 15:48:36 +00:00
2020-04-01 15:48:36 +00:00
2017-05-11 20:55:51 +00:00
2017-05-11 20:55:51 +00:00
2018-09-26 06:51:56 +00:00
2020-04-01 15:48:36 +00:00
2019-10-21 20:23:56 +00:00
2017-07-21 21:49:04 +00:00
2020-05-13 22:01:34 +00:00
2018-04-11 02:21:48 +00:00
2021-04-20 22:21:32 +00:00
2020-10-08 22:49:59 +00:00
2022-08-19 13:55:42 +00:00
2022-02-03 09:13:21 +00:00
2022-08-19 13:55:42 +00:00
2020-12-11 19:51:28 +00:00
2018-04-11 02:21:48 +00:00
2020-02-22 02:02:56 +00:00
2020-06-09 16:26:23 +00:00
2019-03-08 04:39:08 +00:00
2021-11-03 03:26:27 +00:00
2022-08-25 16:24:41 +00:00
2022-04-05 13:27:10 +00:00
2022-08-12 19:21:55 +00:00
2016-11-03 11:59:07 +00:00
2017-03-17 19:33:31 +00:00
2016-12-02 01:46:55 +00:00
2019-11-26 07:28:23 +00:00
2016-12-02 01:46:55 +00:00
2021-11-30 20:13:18 +00:00
2019-09-25 16:58:14 +00:00
2022-09-28 22:35:48 +00:00
2022-09-09 14:42:41 +00:00
2022-09-07 19:55:43 +00:00
2022-06-22 17:55:07 +00:00
2018-01-05 20:44:21 +00:00
2020-10-21 20:30:05 +00:00
2017-11-22 22:00:16 +00:00
2020-07-15 17:10:50 +00:00
2022-05-18 23:59:58 +00:00
2022-08-19 13:55:42 +00:00
2018-11-17 01:48:05 +00:00
2022-11-04 04:33:02 +00:00
2022-04-13 12:31:36 +00:00
2019-03-08 04:39:08 +00:00
2020-04-16 15:45:49 +00:00
2019-03-08 04:39:08 +00:00
2019-03-08 04:39:08 +00:00
2022-05-18 23:59:58 +00:00
2021-05-07 14:11:08 +00:00
2022-04-05 13:27:10 +00:00
2022-08-09 06:45:52 +00:00
2020-01-17 18:54:33 +00:00
2019-10-04 18:36:22 +00:00
2018-10-01 13:20:40 +00:00
2018-10-01 13:20:40 +00:00
2021-10-28 20:09:46 +00:00
2022-04-04 05:54:58 +00:00
2021-01-27 17:30:37 +00:00
2017-03-03 01:43:42 +00:00
2021-08-21 14:46:51 +00:00
2021-08-21 14:46:51 +00:00
2022-02-04 21:23:55 +00:00
2022-03-04 16:49:22 +00:00
2022-06-22 17:55:07 +00:00
2019-03-14 22:43:50 +00:00
2018-03-08 04:48:36 +00:00
2018-01-17 04:18:24 +00:00
2021-06-03 06:32:45 +00:00
2021-06-03 06:32:45 +00:00
2022-03-04 16:49:22 +00:00
2018-09-10 16:09:41 +00:00
2018-09-10 16:09:41 +00:00
2016-06-20 15:37:18 -07:00
2016-06-20 15:37:18 -07:00
2020-06-18 21:00:27 +00:00
2021-07-14 12:51:09 +00:00
2021-07-14 12:51:09 +00:00
2018-08-22 18:08:20 +00:00
2017-05-09 15:47:21 +00:00
2022-01-23 20:55:00 +00:00
2019-08-24 00:32:28 +00:00
2019-05-07 00:36:56 +00:00
2021-01-14 19:34:38 +00:00
2018-02-22 00:57:46 +00:00
2022-07-11 19:25:02 +00:00
2021-10-28 20:09:46 +00:00
2018-11-27 02:01:12 +00:00
2019-05-10 11:46:38 +00:00
2019-05-10 11:46:38 +00:00
2022-08-19 19:11:58 +00:00
2019-05-10 11:46:38 +00:00
2022-04-02 03:26:28 +00:00
2020-12-10 19:01:08 +00:00
2021-01-05 23:30:35 +00:00
2018-03-22 02:47:33 +00:00
2022-04-13 06:22:14 +00:00
2020-05-21 18:40:20 +00:00
2018-08-23 15:55:53 +00:00
2016-10-14 05:41:22 +00:00
2020-08-25 15:42:01 +00:00
2022-09-28 22:35:48 +00:00
2018-04-11 02:21:48 +00:00
2019-05-10 12:06:01 +00:00
2020-05-20 23:03:23 +00:00
2020-05-20 23:03:23 +00:00
2022-04-05 13:27:10 +00:00
2022-07-22 04:23:30 +00:00
2021-02-10 23:52:28 +00:00
2018-08-15 04:15:46 +00:00
2018-08-15 04:15:46 +00:00
2020-12-22 06:10:39 +00:00
2017-12-13 10:04:40 +00:00
2021-02-13 23:02:54 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2021-01-15 13:01:53 +00:00
2021-03-31 05:38:50 +00:00
2022-09-07 19:55:43 +00:00
2022-09-07 19:55:43 +00:00
2022-06-08 13:02:52 +00:00
2021-03-31 05:38:50 +00:00
2019-08-06 02:35:54 +00:00
2022-04-05 13:27:10 +00:00
2017-11-17 21:33:51 +00:00
2022-04-05 13:27:10 +00:00
2021-03-02 04:58:51 +00:00
2020-04-14 15:42:13 +00:00
2022-02-09 21:20:23 +00:00
2018-11-17 01:48:05 +00:00
2022-04-05 13:27:10 +00:00
2021-03-14 03:35:40 +00:00
2020-07-08 19:32:15 +00:00
2019-01-09 05:26:50 +00:00
2016-08-31 03:12:30 +00:00
2022-07-11 19:25:02 +00:00
2022-06-20 13:52:37 +00:00
2020-04-02 12:01:41 +00:00
2019-09-26 18:35:06 +00:00
2022-10-17 19:37:15 +00:00
2016-09-29 21:14:13 +00:00
2020-05-12 23:29:04 +00:00
2020-04-14 12:18:21 +00:00
2017-08-05 01:03:02 +00:00
2017-08-05 01:03:02 +00:00
2018-04-28 23:41:39 +00:00
2018-02-02 01:13:08 +00:00
2020-10-21 05:20:33 +00:00
2021-01-06 04:39:56 +00:00
2021-03-02 04:58:51 +00:00
2021-08-25 20:05:47 +00:00
2022-10-17 21:59:57 +00:00
2020-04-24 20:56:58 +00:00
2018-04-28 23:41:39 +00:00
2020-08-25 16:11:20 +00:00
2017-08-29 23:01:10 +00:00
2021-03-02 04:58:51 +00:00
2019-01-22 01:20:31 +00:00
2022-02-24 07:06:50 +00:00
2021-05-15 22:20:58 +00:00
2017-03-03 01:43:42 +00:00
2017-03-03 01:43:42 +00:00
2018-11-01 21:27:13 +00:00
2018-11-01 21:27:13 +00:00
2019-08-22 05:03:03 +00:00
2022-09-07 19:55:43 +00:00
2020-08-25 23:08:40 +00:00
2020-10-21 05:20:33 +00:00
2021-04-20 22:21:32 +00:00
2018-04-11 02:21:48 +00:00
2019-09-27 01:54:39 +00:00
2021-04-20 22:21:32 +00:00
2021-05-28 04:02:10 +00:00
2016-11-09 03:27:12 +00:00
2019-04-23 13:33:57 +00:00
2016-11-09 03:27:12 +00:00
2018-04-11 02:21:48 +00:00
2016-12-08 04:53:53 +00:00
2016-12-08 04:53:53 +00:00
2016-11-09 03:27:12 +00:00
2018-11-20 23:49:11 +00:00
2022-08-19 13:55:42 +00:00
2022-04-13 12:31:36 +00:00
2021-12-09 23:17:20 +00:00
2020-06-15 23:42:12 +00:00
2018-07-03 20:34:27 +00:00
2021-05-14 13:58:04 +00:00
2022-10-11 20:30:50 +00:00
2021-08-15 20:40:39 +00:00
2019-05-24 15:08:19 +00:00
2019-05-24 15:08:19 +00:00
2020-07-14 03:13:18 +00:00
2022-09-27 13:27:41 +00:00
2022-04-14 11:56:20 +00:00
2022-08-30 16:45:24 +00:00
2022-08-12 19:21:55 +00:00
2022-06-08 13:02:52 +00:00
2022-06-08 13:02:52 +00:00
2022-06-08 13:02:52 +00:00
2020-10-21 05:20:33 +00:00
2022-04-05 07:56:39 +00:00
2022-02-17 18:51:06 +00:00
2022-04-05 07:56:39 +00:00