- Added execution summary to the beeswax client and QueryResult
- Modified report-benchmark-results to handle JSON and perform
execution summary comparison between runs
- Added comments to the new workload runner
Change-Id: I9c3c5f2fdc5d8d1e70022c4077334bc44e3a2d1d
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3598
Reviewed-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com>
Tested-by: jenkins
(cherry picked from commit fd0b1406be2511c202e02fa63af94fbbe5e18eee)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3618
This patch introduces new abstractions and changes the way queries are run via the
workload runner. A new class 'Workload' is introduced, which represents the notion of a
workload in the performance framework (i.e, A set of query names mapped to query
strings).
The new workflow is:
- run-workload acts as a driver. It accepts user parmaters for which queries to
run and their execution strategy. It generates workload objects and passes them to the
workload-runner.
- The workload runner takes a workload, its execution parameters and generates a set of
test vectors over which the workload is run iteratively.
- A workload is executed by initialiazing a QueryExecutor for each query being run in a
test vector. The workload executor is then responsible for execution and gathering
results.
- The execution details of every query being executed are are stored and returned to the
driver (run-workload).
Change-Id: Ia16360140d65e6733e534e823bc5d5614622ab5f
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3616
Reviewed-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com>
Tested-by: jenkins
Also add support for "SET", which returns a table of query options and
their respective values.
The front-end parses the option into a (key, value) pair and then the
existing backend logic is used to set the option, or return the result
sets.
Change-Id: I40dbd98537e2a73bdd5b27d8b2575a2fe6f8295b
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3582
Reviewed-by: Daniel Hecht <dhecht@cloudera.com>
Tested-by: jenkins
(cherry picked from commit aa0f6a2fc1d3fe21f22cc7bc56887e1fdb02250b)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3614
Adds an aggregate function to compute equi-depth histograms. The UDA
creates a sample of the column values using weighted reservoir sampling
and computes the histogram from the sorted sample.
TODO:
* Extract highly frequent values into separate buckets (i.e. 'compressed
histogram').
* Expose separate finalize fn to produce samples and histogram data for stats
Change-Id: I314ce5fb8c73b935c4d61ea5bbd6816c59b3b41e
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3552
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
(cherry picked from commit c5c475712f88244e15160befaf4e99d6e165a148)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3608
We recently changed user-initiated cancellation to not set the query state
to EXCEPTION. In FetchInternal() we relied on the previous behavior for
detecting cancellations/errors after BlockOnWait().
This patch fixes the cancellation/error check to use the query status
instead of the query state.
Change-Id: I48b4834e77b6e692fb6722637fb9fd5d8c8d9d97
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3597
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3600
This also means clients of the block mgr need to delete all blocks in close.
This is less important for sorting since it's typically at the end but will
be useful very soon.
Change-Id: Ia4ee188ad845540039ede5fe410a6048abe2bf5a
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3540
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3588
With this change we introduce a proper type for subqueries using the
recently added complex type hierarchy. The type of a subquery can be a
ScalarType, a StructType or an ArrayType depending on how many columns
and rows are returned by the subquery's statement. The subquery type is
used to simplify the analysis of subquery predicates.
Change-Id: I82e76fcb511397ca58c611f26e77fb764cfa21ae
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3547
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3581
This patch does a few things:
1. Moves the buffer block mgr from the sorter to the runtime state. This is now
one that is shared across the query fragment. The partitioned hash join and agg
will use this as well.
2. Adds a Client interface to the block mgr. Each exec node is a different client
and can reserve a minimum number of buffers. This avoid starvation.
3. Updated the BufferedBlockMgr interface's for getting pinned blocks to collapse
two existing APIs.
Change-Id: Ibb31fbe480f3726048457f26e24a9e33f7201d86
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3504
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: Nong Li <nong@cloudera.com>
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3574
We were setting the state to exception on Cancel() all the time.
We use the cancellation path as the normal cleanup path so this
gets called even when the query went fine (e.g. UnregisterQuery
calls Cancel()). We had already plumbed through a 'cause' argument
to differentiate.
Change-Id: Icf1091c165dec36d3dad7ce308367bbbc9edee4f
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3524
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3575
Currently, the scanner is throwing an IOException when encountering an
empty literal, and so no parse error is formulated. Fix this by
adding a token type for empty literals. This new token doesn't appear
in the parser's grammar, and so a nice parse error will be generated
when the parser encounters an empty literal token.
Also add a regression fe test case.
Change-Id: Ib1ad0470ebc30b6fc827c9420745ecd83fc5e1ae
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3539
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: jenkins
(cherry picked from commit 2160b527703caee853ccca239797b67090bda149)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3568
Reviewed-by: Daniel Hecht <dhecht@cloudera.com>
FromUtc and ToUtc use thirdparty libraries which use inline asm which
isn't currently supported with JIT. The UDFs are included in this
commit, but the function symbols were not changed in
impala_functions.py
Change-Id: I0824a434d4a26a39abf29bc6e47d51b5ad7991d6
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3390
Reviewed-by: Paden Tomasello <paden.tomasello@cloudera.com>
Tested-by: jenkins
(cherry picked from commit 8e149ccd78010b7a22d6fff1b0de5614848b02ac)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3548
Adds support for dropping all table and column stats from a table. Once incremental
stats are supported, this will provide the user a way to force a recompute of all
stats.
Change-Id: I27e03d5986b64eb91852bfc3417ffa971d432d6b
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3533
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
(cherry picked from commit f1f074f24bfdc77c4cef147fe9d26f27df80ab81)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3551
The following changes are included in this commit:
1. Modified the parser to parse nested queries.
2. Added functional parser tests for nested queries.
3. Modified the analyzer to perform semantic analysis of nested queries.
4. Added functional analysis tests for nested queries.
Change-Id: I0988cb22c9b52c79d57a7c59daa85ec4821643f9
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3419
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3530
There was an issue with the previous fix to IMPALA-1059
if the user tried to reconnect within the shell after
having passed in a database via the -d option. The
passed database would be doubly backticked. This makes
the backticking of the argument idempotent.
Change-Id: I6eaed997c2be73d8659a2a12046ce393b97ec82c
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3467
Reviewed-by: Abdullah Yousufi <abdullah.yousufi@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3502
We used to maintain a separate hash table (in the form of a boost
unordered set) to keep track of the build rows that have been matched.
This patch changes it by just keeping a bit in the hash table. It is not
possible to use boost::unordered_set for tables that are large.
Change-Id: Ie36e609bf79e5e7e403417a3c02a0817d37acc60
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3478
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
This patch does two things in preparation for external joins. The
hash table used to contain a directory structure (buckets and nodes)
both of which were contiguous. The nodes contained the tuple ptrs
within it.
This patch changes it so the nodes are not stored contiguously but
allocated in pages. (this structure is dense and does not require
random lookups by index). The bucket structure is still contiguous
since we rely on the doubling property and random lookup by index.
The second change is that the node's no longer store the tuple ptrs
within them. This makes it easier to build the hash table ontop of
existing data.
Here's a quick benchmark doing a self join on tpch lineitem. Both
build and probe times decreased a bit.
Before:
HASH_JOIN_NODE (id=2):(Total: 1s139ms, non-child: 985.939ms, % non-child: 86.50%)
- BuildBuckets: 2.10M (2097152)
- BuildRows: 6.00M (6001215)
- BuildTime: 527.991ms
- LeftChildRows: 6.00M (6001215)
- LeftChildTime: 451.964ms
- LoadFactor: 0.50
- RowsReturned: 30.01M (30012985)
- RowsReturnedRate: 26.33 M/sec
After:
HASH_JOIN_NODE (id=2):(Total: 1s019ms, non-child: 835.350ms, % non-child: 81.97%)
- BuildBuckets: 2.10M (2097152)
- BuildRows: 6.00M (6001215)
- BuildTime: 423.175ms
- LeftChildRows: 6.00M (6001215)
- LeftChildTime: 406.67ms
- LoadFactor: 0.50
- RowsReturned: 30.01M (30012985)
- RowsReturnedRate: 29.45 M/sec
Change-Id: I79e209a24c24fb4f2f99574bcf187746fddadc06
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3245
Tested-by: jenkins
Reviewed-by: Nong Li <nong@cloudera.com>
I'm not sure when we added this but it does not have any benefit. The join nodes
combine the tuple*'s from the LHS and RHS anyway and the extra Tuple* reserved in
the LHS row batch is never written to or read.
Change-Id: I40f88f417161ef72185e995b6c5b8f56f31fbfc4
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3438
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
If not backticked, arguments such as parquet are interpreted as
keywords, when it is possible a database by that name exists.
This could have been avoided via single quotes around backticks: -d '`parquet`'
Otherwise, -d `parquet` throws a commandline error.
In interactive mode, backticks alone (ex. use `parquet`) will pass the
name as an identifier rather than a keyword.
Change-Id: I24b43eeeb6b4bfda5388165856788a20b64bc2ba
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3307
Reviewed-by: Abdullah Yousufi <abdullah.yousufi@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3500
The tests pass every time locally (in a 60 minute run), but fail
intermittently on our build machines.
Change-Id: I62d5ea0df8c42728a538b29bd16006be3179bfd3
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3489
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: jenkins
The root cause of the problem was that columns of a Table were not
added to the colsByName_ map with lower case keys on the Table.load() path
that is only exercised by the catalog server (the Impalads "load" tables
via Table.loadFromThrift() which did the right thing).
The above led to an empty column stats object being sent to the HMS
after an otherwise successful compute stats.
The problem was sporadic for the following reasons:
1. Only certain file formats like avro/snap/block have uppercase
column names in the HMS because the table was created by Hive
2. Some of our tests executed via run-tests.py, notably the
cancellation tests, aren't deterministic in which test vectors
are executed in a particular run. As a result, we only see the
cancellation test run compute stats on an avro/snap/block
once in a while (this behavior is unaffected by this patch).
This patch includes other minor bugfixes and simplifications
related to compute stats.
Change-Id: I7cb5fe69404e35133eda314d9f7d072c78416ff1
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3468
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3479
I hit this in the expr refactoring. This makes sure we never expose a
function that returns a DecimalVal directly (rather than through an
extra return parameter as specified by the ABI), which will crash if
called from precompiled native code.
Change-Id: Ifb249086c221b53553d3e7fb39af065f4cca2bac
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3425
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
(cherry picked from commit 429448935555b098e324bcb97ab43a7c90e0b918)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3473
