Add tests for TPCH-nested queries with column masking policies on the
PII columns (phone, name, address). Some queries have the same results
as without the column masking policies so we reuse their test files.
Change-Id: I4a6c9fc480923369952e8e215f4a90b2f6448028
Reviewed-on: http://gerrit.cloudera.org:8080/15655
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
We've supported reading primitive types from ORC files (IMPALA-5717).
In this patch we add support for complex types (struct/array/map).
In IMPALA-5717, we leverage the ORC lib to parse ORC binaries (data in
io buffer read from DiskIoMgr). The ORC lib can materialize ORC column
binaries into its representation (orc::ColumnVectorBatch). Then we
transform values in orc::ColumnVectorBatch into impala::Tuples in
hdfs-orc-scanner. We don't need to do anything about decoding/decompression
since they are handled by the ORC lib. Fortunately, the ORC lib already
supports complex types, we can still leverage it to support complex types.
What we need to add in IMPALA-6503 are two things:
1. Specify which nested columns we need in the form required by the ORC
lib (Get list of ORC type ids from tuple descriptors)
2. Transform outputs of ORC lib (nested orc::ColumnVectorBatch) into
Impala's representation (Slots/Tuples/RowBatches)
To format the materialization, we implement several ORC column readers
in hdfs-orc-scanner. Each kind of reader treats a column type and
transforms outputs of the ORC lib into tuple/slot values.
Tests:
* Enable existing tests for complex types (test_nested_types.py,
test_tpch_nested_queries.py) for ORC.
* Run exhaustive tests in DEBUG and RELEASE builds.
Change-Id: I244dc9d2b3e425393f90e45632cb8cdbea6cf790
Reviewed-on: http://gerrit.cloudera.org:8080/12168
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
In this commit we enable Decimal_V2 by default. We also update the
expected results in many of our tests.
Testing:
Ran an exhaustive test which almost passed. Updated the few failed
tests in it.
Cherry-pick: not for 2.x
Change-Id: Ibbdd05bf986b7947f106b396017faa3a0bd87fd7
Reviewed-on: http://gerrit.cloudera.org:8080/9062
Reviewed-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com>
Tested-by: Impala Public Jenkins
The concurrent test driver did not pick them up because the name
prefix did not match the workload dirname. The query test driver
used a hardcoded prefix.
Testing done: Ran tests/stress/concurrent_select.py,
tests/query_test/test_tpch_nested_queries.py locally; latter
passed, former hit IMPALA-5855 after correctly locating all 22
new tpch_nested query files.
Change-Id: Ie067b201ae20b4f4c61a98be7ac1ec5a3f8febd8
Reviewed-on: http://gerrit.cloudera.org:8080/7891
Reviewed-by: Michael Brown <mikeb@cloudera.com>
Reviewed-by: David Knupp <dknupp@cloudera.com>
Tested-by: Impala Public Jenkins
Marcel spotted that nested TPCH-Q18 can be expressed with
more efficient SQL.
Results on nested TPCH-300:
Before 160s
After 100s
Change-Id: I8b351b7f467e8bef0c256dc43cea325d7f177edf
Reviewed-on: http://gerrit.cloudera.org:8080/2418
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
- Changed the nested TPCH queries to use unqualified table names
- Replaced dash with an underscore in the workload name
Change-Id: Id1cbe5318fc9940ca7dc9dd4ff09d61593600a24
Reviewed-on: http://gerrit.cloudera.org:8080/502
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
