- Added static order by tests to test_queries.py and QueryTest/sort.test
- test_order_by.py also contains tests with static queries that are run with
multiple memory limits.
- Added stress, scratch disk and failpoints tests
- Incorporated Srinath's change that copied all order by with limit tests into
the top-n.test file
Extra time required:
Serial:
scratch disk: 42 seconds
test queries sort : 77 seconds
test sort: 56 seconds
sort stress: 142 seconds
TOTAL: 5 min 17 seconds
Parallel(8 threads):
scratch disk: 40 seconds
test queries sort: 42 seconds
test sort: 49 seconds
sort stress: 93 seconds
TOTAL: 3 min 44 sec
Change-Id: Ic5716bcfabb5bb3053c6b9cebc9bfbbb9dc64a7c
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2820
Reviewed-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3205
The admission control tests could occasionally fail when cancelled
queries return OK (IMPALA-1047). Until fixed, we can just treat
such queries as if there were cancelled.
Change-Id: Id9fc8e9f585e466059d4ffefb4d9ed407206ad1d
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3019
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
(cherry picked from commit 2901a8a960076f2aec74cb5a1f5000953359a68f)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3025
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Some tests have constraints that were there only to help reduce runtime which
reduces coverage when running in exhaustive mode. The majority of the constraints
are because it adds no value to run the test across additional dimensions (or
it is invalid to run with those dimensions). Updates the tests that have
legitimate constraints to use two new helper methods for constraining the table format
dimension:
create_uncompressed_text_dimension()
create_parquet_dimension()
These will create a dimension that will produce a single test vector, either
uncompressed text or parquet respectively.
Change-Id: Id85387c1efd5d192f8059ef89934933389bfe247
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2149
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
(cherry picked from commit e02acbd469bc48c684b2089405b4a20552802481)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2290
their parent's permissions
This patch adds --insert_inherit_permissions. If true, all
new partition directories created by INSERT will inherit their
permissions from their parent. When false, the directories are created
with the default permissions.
Change-Id: Ib2b4c251e51ea5048387169678e8dde34ecfe5f6
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1917
Tested-by: jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
A crucial comparison was between time values with different units.
Tests didn't catch this because they only confirmed that sessions were
timed out within the correct time, not that they were *not* timed out
early.
Change-Id: Ia8c57d3d70e4702996d0225b167142b7bf88d236
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1926
Tested-by: jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
* Remove requirement that fair scheduler and Llama conf files be on the classpath if
specified as relative paths. Now they can be specified as any relative or absolute
path.
* Add flags to disable all per-pool max requests limits or mem limits.
* Rename RequestPoolUtils to RequestPoolService
* Make it more clear RequestPoolService is a singleton by putting it in ExecEnv
* FileWatchService: use Executors.newScheduledThreadPool instead of a thread
* Moved MEGABYTE (and related constants) to new Constants class (frontend)
* Test RequestPoolService: Removed AllocationFileLoaderServiceHelper, replaced with
reflection
Change-Id: Iadf79cf77a7894a469c3587d0019a6d0bee7e58f
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1787
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
(cherry picked from commit b9a167f6fdb4ab2595aca6035e1f9d926b909d94)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1858
In some rare cases on overloaded machines, the thread join timeout of 10 seconds isn't
long enough. Also, taking the lock at that time isn't necessary because the main thread
will not attempt to cancel a thread unless it is already in the list of running threads.
Threads are added to that list only after they submit their query.
Change-Id: I23a67d726bc25221f0e9331ca1a3e9f5363f821d
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1744
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
(cherry picked from commit 27cf239592fafdb36a5680c480914f38a16037da)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1760
Admission control adds support for configuring pools via a fair scheduler
allocation configuration, so the pool configuration mechanism is no longer
needed. This also renames the "yarn_pool" query option to the more general
"request_pool" as it can also be used to configure the admission controller
when RM/Yarn is not used. Similarly, the query profile shows the pool as
"Request Pool" rather than "Yarn Pool".
Change-Id: Id2cefb77ccec000e8df954532399d27eb18a2309
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1668
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
(cherry picked from commit 8d59416fb519ec357f23b5267949fd9682c9d62f)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1759
Adds RequestPoolUtils which exposes user to pool resolution, authorization,
and relevant pool configurations by wrapping Yarn classes that provide that
functionality. (To support CDH4, those Yarn classes will come from
thirdparty/cdh4-extras.) RequestPoolUtils is created once by the backend and
the instance lives for the duration of the process.
Change-Id: I53db075555578614356d33f9d939c5378b9ec797
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1566
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
(cherry picked from commit 8e385bdb54ed97e567c672a76723936c24cfe45f)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1758
The test works by submitting a number of queries (parameterized) with
some delay between submissions (parameterized) and the ability to
submit to one impalad or many. The queries are set with the WAIT debug
action so that we have more control over the state that the admission
controller uses to make decisions. Each query is submitted on a
separate thread. Depending on the test parameters a varying number of
queries will be admitted, queued, and rejected. Once queries are
admitted, the query execution blocks and we can cancel the query in
order to allow another queued query to be admitted. The test tracks
the state of the admission controller using metric counters on each
impalad.
Change-Id: I455484a7f899032890b22c38592fcea1875f5399
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1413
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: jenkins
(cherry picked from commit bc2a74d6da622de877422f926ff1892bed867bb1)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1624
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Matthew Jacobs <mj@cloudera.com>
This is the first step in cleaning up the test logging. It provides a common connection
interface that provides tracing around all operations. When a test fails the output will
be executable SQL. It also logs actions such as when a connection is opened, close, or
when an operation is cancelled. Currently only beeswax connections are supported, but
I have a seperate patch that adds support for executing using HS2 as well as Beeswax.
Example of new logging:
-- connecting to: localhost:21000
-- executing against localhost:21000
use functional;
SET disable_codegen=False;
SET abort_on_error=1;
SET batch_size=0;
SET num_nodes=0;
-- executing against localhost:21000
select a.timestamp_col from alltypessmall a inner join alltypessmall b on
(a.timestamp_col = b.timestamp_col)
where a.year=2009 and a.month=1 and b.year=2009 and b.month=1;
-- closing connection to: localhost:21000
Change-Id: Iedc7d4d3a84bfeff6cc1daae6ed1ca97613d7700
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1133
Tested-by: jenkins
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Test suites that derive from common.CustomClusterTestSuite have a brand
new cluster for every tests case, which they can configure as they wish
with custom arguments using the @with_args() decorator.
A future improvement is to optionally only have one cluster per test
suite, to allow multiple tests to run more quickly if they share
configuration options.
Change-Id: I6abd5740e644996d7ca2800edf4ff11b839d1bc4
Reviewed-on: http://gerrit.ent.cloudera.com:8080/882
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins
