Nested Types: Enforce and test maximum nesting depth of 100.

Browse Source

The limit of 100 was determined empirically by generating deeply
nested Parquet and Avro files and then trying to run queries with
and without subplans over them (one absolute table ref vs. all relative
table refs for maximally nested subplans).

Based on those experiments we can handle up to 200 levels of nesting,
but the queries get very slow. At 300 levels, we exceed the stack space
due to the recursive implementation of the scan. Also, we decode the
rep/def levels of Parquet as uint8_t. I settled with 100 because it is
safe, future proof and reasonably high for most practical cases.

Change-Id: Iebdfa96a6dd6060387e38eaedb8ddf0f9901ac24
Reviewed-on: http://gerrit.cloudera.org:8080/905
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins

...

This commit is contained in:

Alex Behm

2015-09-22 18:08:03 -07:00

committed by ishaan

parent 1f93a816df

commit b89d69da90

17 changed files with 410 additions and 10 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

BIN

testdata/max_nesting_depth/struct_array/file.parq vendored Normal file

Binary file not shown.