- created new class PlanFragment, which encapsulates everything having to do with a single
plan fragment, including its partition, output exprs, destination node, etc.
- created new class DataPartition
- explicit classes for fragment and plan node ids, to avoid getting them mixed up, which is easy to do with ints
- Adding IdGenerator class.
- moved PlanNode.ExplainPlanLevel to Types.thrift, so it can also be used for
PlanFragment.getExplainString()
- Changed planner interface to return scan ranges with a complete list of server locations,
instead of making a server assignment.
Also included: cleaned up AggregateInfo:
- the 2nd phase of a DISTINCT aggregation is now captured separately from a merge aggregation.
- moved analysis functionality into AggregateInfo
Removing broken test cases from workload functional-planner (they're being handled correctly in functional-newplanner).
- executor takes report callback; passed in by ImpalaServer::FragmentExecState
- the PlanFragmentExecutor invokes profile reporting cb in background thread.
- RuntimeProfile is now thread-safe and has an RuntimeProfile::Update()
Also included:
- a number of bug fixes related to async cancellation of query
and propagation of errors through PlanFragmentExecutor/Coordinator/ImpalaServer.
- changing COUNTER_SCOPED_TIMER to SCOPED_TIMER
- derived counters: RuntimeProfile now lets you add counters that return a
value via a function call, which is useful for reporting something like normalized
ScanNode throughput; retrofitted to ScanNode and all subclasses
- changed coordinator to make cancellation atomic wrt recognition of an error status
for the overall query.
- Removed InProcessQueryExecutor from data-stream-test.
Added aggregate throughput counters to coordinator:
- all throughput counters are grouped in a sub-profile "AggregateThroughput"
- each scan node gets its own counter
- the value is aggregated across all registered backends which contain that node in
their plan fragments
"distinctpc" and "distinctpcsa".
We've gathered statistics on an internal dataset (all columns) which is
part of our regression data. It's roughly 400mb, ~100 columns,
int/bigint/string type.
On Hive, it took roughly 64sec.
On this Impala implementation, it took 35sec. By adding inline to hash-util.h (which we don't),
we can achieve 24~26sec.
Change-Id: Ibcba3c9512b49e8b9eb0c2fec59dfd27f14f84c3
[Submitting on behalf of Marcel]
- fragment ids weren't assigned correctly (they need to be unique across all
nodes on which they're executing)
- some of the execution logic that I checked in yesterday was flawed
- added DataStreamMgr::Cancel(), which is used to propagate cancellation from the
coordinator to all (possibly blocked) ExchangeNodes
- all exec nodes now check for cancellation before they do anything that might block for a while
- fixed up logic related to async cancellation
Added support for async query execution via beeswax interface:
- implemented ImpalaServer::query()
- QueryExecState now tracks beeswax's idea of the query state
- ImpalaServer::get_state() now returns the actual state
Fixed handling of ExecNode::Close():
- needs to be called for entire plan tree, regardless of what fails (can't use
RETURN_IF_ERROR() inside of it)
- needs to be called for every Open() call by coordinator/ImpalaServer
- making the coordinator asynchronous
- renamed ImpalaBackendService to ImpalaInternalService;
- new class ImpalaServer implements ImpalaService and ImpalaInternalService
- renaming ImpalaInternalService fields to conform to c++ style
- merged impala-service.{cc,h} and backend-service.{cc,h} into impala-server.{cc,h}
- added TStatusCode field to Status.ErrorDetail
- removed ImpalaInternalService.CloseChannel
Also removed JdbcDriverTest.java
Currently, subscriptions are per-subscriber. However, we don't want
to be stuck with this decision later on, and if subscriptions are
uniquely numbered, a 32-bit integer may not allow as many subscriptions
as we'd like.
