Alex Behm b89d69da90 Nested Types: Enforce and test maximum nesting depth of 100. ...

The limit of 100 was determined empirically by generating deeply
nested Parquet and Avro files and then trying to run queries with
and without subplans over them (one absolute table ref vs. all relative
table refs for maximally nested subplans).

Based on those experiments we can handle up to 200 levels of nesting,
but the queries get very slow. At 300 levels, we exceed the stack space
due to the recursive implementation of the scan. Also, we decode the
rep/def levels of Parquet as uint8_t. I settled with 100 because it is
safe, future proof and reasonably high for most practical cases.

Change-Id: Iebdfa96a6dd6060387e38eaedb8ddf0f9901ac24
Reviewed-on: http://gerrit.cloudera.org:8080/905
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins

2015-10-05 11:30:54 -07:00

8.6 KiB

Raw History

View Raw