Cross joins should be handled like outer joins in the join order
optimization in that the right table referenced by a cross join may not
be reordered anywhere before tables referenced to the left of the cross
join. If there are inner joins to the right of the cross join, those
tables may be reordered before the cross join.
E.g., if we have A JOIN B CROSS JOIN C JOIN D, then C must come after A
and B, but D may be reordered to come before C.
Also adds test cases for join order optimization and predicate propagation.
Change-Id: I6b1022dd3e862efbff81e283b43284d846c8eca4
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1096
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Impala now exploits existing partitioning of the lhs and/or rhs of joins. Existing
partitioning effects the cost of choosing between a partitioned and a broadcast join,
as well as the final plan because Impala can avoid repartitioning the data.
An existing data partition is exploitable iff the lhs/rhs input partition is equivalent
to the target partition of the join. This matching procedure considers equivalence classes.
Change-Id: Ica080f35cf5063bea828963bc234ba69797e2030
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1070
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Adds the TPCDS queries as planner tests and fixes a few small issues
with the Planner test file parser. This adds the TPC-DS queries using
SQL-92 style joins that have a hand optimized (although
not perfect) join order.
Change-Id: I2d81e66af740b2d826b8ebd0c5ba8553b5faf0a2
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1019
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
Adds a CROSS JOIN (cartesian product). Common join code is moved from to
a new abstract base class BlockingJoinNode. We must keep all build RowBatches in
memory in order to iterate over them for every row from the left child. The
TupleRowList provides a convenient way to iterate over all of the rows.
A future change will address codegen for the CrossJoinNode.
Change-Id: I5e0caa6fb4ec802a9c87e700f9dd6238cea8cdf2
Reviewed-on: http://gerrit.ent.cloudera.com:8080/970
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Introduces STRAIGHT_JOIN keyword to prevent join order optimization.
Structural changes to the planning framework:
- slot materialization: the decision whether to materialize a slot now happens *prior* to
plan generation. This is needed in order to be able to generate accurate cost estimates
at plan generation time. see QueryStmt.materializeRequiredSlots()
- added PlanNode.init(), which initializes the entire state of a PlanNode; this subsumes
finalize()
* computeMemLayout() now happens per-tuple in the corresponding ScanNode's init()
* init() calls computeStats() by default; also marks slots as materialized and calls
TupleDescriptor.computeMemLayout()
- added PlanNode.tblRefIds_
- restructured UnionStmt and union plan generation to fit pred propagation model:
all tuples are created (and equiv predicates registered) prior to plan generation
- added Expr.isAuxExpr
Change-Id: I475c1645bfca9e84ae6e5f529e7781d9532e5c9a
Reviewed-on: http://gerrit.ent.cloudera.com:8080/955
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
Updates our compute stats script to execute using Impala. This allows us
to easily compute stats on all tables in a database or all tables in the
metastore.
The updated stats caused one of the TPCH plans to change so this also
updates the TPCH planner test results.
Change-Id: I17e5dcd1036a35e40eb4eb2c8e4a20702db9049c
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1024
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
Adds support for skipping a number of rows with an ORDER BY clause and a LIMIT. Hive
does not support OFFSET so creating a view with an OFFSET will not work in Hive.
For example, "SELECT * FROM T1 ORDER BY ID LIMIT 20 OFFSET 5" will do the sorting, skip
5 rows, then return the next 20. OFFSET requires an ORDER BY clause.
Note this is not very efficient as we must actually keep (limit+offset) rows in memory
in the topn-node, and all child sort nodes must as well. Users should be careful when
using this feature.
Change-Id: I4d7021c278296e7bdbfa0e6f2699cd6f23eef59d
Reviewed-on: http://gerrit.ent.cloudera.com:8080/900
Tested-by: jenkins
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Matthew Jacobs <mj@cloudera.com>
This patch redoes how the aggregation node is implemented. The functionality is
now split between aggregation-node, agg-expr and aggregate-functions. This is a working
progress (there's still a lot of debug stuff I added that needs to be cleaned up) but
it does pass the tests.
Aggregation-node is now very simple and now only deals with the grouping part.
Aggregate-expr serves as the glue between the agg node and the aggregate functions.
The aggregation functions are implemented with the UDA interface. I've reimplemented
our existing aggregate functions with this setup. For true UDAs, the binaries would be
loaded in aggregate-expr.
This also includes some preliminary changes in the FE. We now need to annotate each
AggNode as executing the update vs. merge phase (root aggs execute update, others
execute merge) and if it needs a finalize step (only the root does). This is more
general than our builtins which are too simple to need this structure.
There is a big TODO here to allow the intermediate types between agg nodes to change.
For example, in distinct estimate, the input type is the column type and the output type
is a bigint. We'd like the intermediate type to be CHAR(256). This is different since
currently, the intermediate type and output type have always been the same. We've hacked
around this by having both the intermediate and output type be TYPE_STRING. I've left
this for another patch (changing the BE to support this is trivial).
For aggregates that result in strings, we used to store some additional stuff past the
end of the tuple. The layout was:
<tuple> <length of 1st string buffer>,<length of 2nd string buffer>, etc
The rationale for this is that we want to reuse the buffer for min/max and grow the buffer
more quickly for group_concat. This breaks down the abstraction between agg-expr and
agg-node and is not something UDAs can use in general. Rather than try to hack around
this, I think the proper solution is to the intermediate type not be StringValue and
to contain the buffer length itself.
This patch also resurrects the distinct estimate code. The distinct estimate functions
exercise all of the code paths.
Change-Id: Ic152a2cd03bc1713967673681e1e6204dcd80346
Reviewed-on: http://gerrit.ent.cloudera.com:8080/564
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: Nong Li <nong@cloudera.com>
Fixed cost estimation of union queries and exchange nodes.
Fixed propagation of stats through cloning of exprs and plan nodes.
Fixed propagation of expr stats to slots they are materialized into (e.g., grouping columns in multi-level aggs).
Improved explain output for constant selects.
Change-Id: I96d1652c00d48e4093b85ae7fc8bad28d74b8b81
Reviewed-on: http://gerrit.ent.cloudera.com:8080/547
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Alex Behm <alex.behm@cloudera.com>
This fix contains two parts:
- functionality inside the analyzer to compute a value transfer graph (from equality predicates
between slotrefs) and from that equivalence classes for all slots; this functionality is required for this fix
but will be generally useful when adding propagation of binding predicates in the future
- a "shortest path" implementation inside the planner of a fix for the problem at hand; this leaves a lot to be desired:
* correct handling of assigned predicates: the added test case shows that the planner will try to assign all predicates
to some node in the tree, even if that predicate is superfluous because it was subsumed by an equality derived from
equivalence class membership
* complete lack of propagation of binding predicates (e.g., propagate "col1 = 5" to all slotrefs that are in the same
equivalence class as col1)
This is beyond what can be accomplished for 1.1 and therefore will have to wait for 1.2.
This is solved by repartitioning the input to the hdfs table sinks on the partition key columns of the hdfs
table, so that each partition is only written by a single node.
- added PlanNode.numNodes, PlanNode.avgRowSize and PlanNode.computeStats()
- fixing up some cardinality estimates
- Planner now tries to do a cost-based decision between broadcast join and join with full repartitioning (both inputs)
- ExchangeNode now distinguishes between its input and output row descriptor: the output potentially contains more tuples
- fixed problem related to cancellation and concurrent hash table builds.
Not included:
- partitioned joins that take advantage of existing partitions of the inputs; those will have to wait for a follow-on change
