This change fixes two problems:
1. The query options OPTIMIZE_PARTITION_KEY_SCANS and
DISABLE_STREAMING_PREAGGREGATIONS are both boolean
so they should accept 'true' and '1' as input values.
Previously, these two options are treated as int and
value such as 'true' doesn't work with them.
2. The break statement in the case statement of the option
SCAN_NODE_CODEGEN_THRESHOLD was 'stolen' by the option
DISABLE_STREAMING_PREAGGREGATIONS when it was added.
This change adds the missing break statement back for
SCAN_NODE_CODEGEN_THRESHOLD.
Change-Id: I5c74a1e5c49e3bda15a91b40740fc7310303207b
Reviewed-on: http://gerrit.cloudera.org:8080/2776
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
For a table with location "ABC", most partitions will have locations
like "ABC/DEF=2". The "ABC" part of the location does not need to be
stored in Catalog for each partition; we can compress it down to one
int in the common case.
This is done by stripping from each partition location the last N
directories (where N is the number of clustering columns) and storing
the resulting string in a cache of partition location prefixes. In the
cache, this location prefix string is mapped to an int. Partition
locations are then stored as a tuple consisting of that int and a
suffix string; the partition location can be reconstructed as the
concatenation of the prefix string (from the cache) and the suffix.
Though this scheme was designed in the expectation that most
partitions will be stored in directories like
"/part_col_1=1.23/part_col_2=234/", it works even when that is not the
case.
TODO: Since each partition stores the literal values for the
partitioning columns, we could also elide the column names and values
when partitions are placed in directories like
"/part_col_1=1.23/part_col_2=234/"
Change-Id: I8c67b6ce0f83de2f5277a528a9ce67e47d638adb
Reviewed-on: http://gerrit.cloudera.org:8080/2355
Reviewed-by: Jim Apple <jbapple@cloudera.com>
Tested-by: Internal Jenkins
There was an observed race between TestUdfs.test_java_udfs and
TestUdfs.test_libs_with_same_filenames, because they both used the
same database name.
This patch just changes the name of the database used by
test_libs_with_same_filenames.
Change-Id: Icc38cbe720a3b9d864935416eb10612171132e17
Reviewed-on: http://gerrit.cloudera.org:8080/2767
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Internal Jenkins
This patch introduces a new query option,
PARQUET_FALLBACK_SCHEMA_RESOLUTION which allows Parquet files' schemas
to be resolved by either name or position. It's "fallback" because
eventually field IDs will be the primary schema resolution scheme, and
we don't want to create an option that we will have to change the name
of later. The default is still by position. I chose to do a query
option because it will make testing easier and also be easier to
diagnose resolution problems quickly in the field. If users want to
switch the default behavior to be by name (like Hive), they can use
the --default_query_options flag.
This patch also introduces a new test section, SHELL, which can be
used to execute shell commands in a .test file. This is useful for
copying files into test tables.
Change-Id: Id0c715ea23792b2a6872610839a40532aabbb5a6
Reviewed-on: http://gerrit.cloudera.org:8080/2384
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Internal Jenkins
This field was included in the schema and data files, but the
checked-in generated parquet files didn't include it. It's not
referenced in any tests so we didn't catch it.
Change-Id: I5d394f074e7082fa12fafb7e57a144a83b3099a6
Reviewed-on: http://gerrit.cloudera.org:8080/2562
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
The stubs in Impala broke during the merge commit. This commit removes
the stubs in hopes of improving robustness of the build. The original
problem (Kudu clients are only available for some OSs) is now addressed
by moving the stubbing into a dummy Kudu client. The dummy client only
allows linking to succeed, if any client method is called, Impala will
crash. Before calling any such method, Kudu availability must be
checked.
Change-Id: I4bf1c964faf21722137adc4f7ba7f78654f0f712
Reviewed-on: http://gerrit.cloudera.org:8080/2585
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
These tests functionally test whether the following type of files
are able to be scanned properly:
1) Add a parquet file with multiple blocks such that each node has to
scan multiple blocks.
2) Add a parquet file with multiple blocks but only one row group
that spans the entire file. Only one scan range should do any work
in this case.
Change-Id: I4faccd9ce3fad42402652c8f17d4e7aa3d593368
Reviewed-on: http://gerrit.cloudera.org:8080/1500
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Internal Jenkins
The PHJ may disable runtime filter production for one of several
reasons, including a predicted high false-positive rate. If the filters
are not produced, any scans will wait for their entire timeout before
continuing.
This patch changes the filter logic to always send a filter, even if one
wasn't actually produced by the PHJ. To preserve correctness, that
filter must contain every element of the set. Such a filter is
represented by (BloomFilter*)NULL. This allows us to make no changes to
RuntimeFilter::Eval(), which already returns true if the member Bloom
filter is NULL.
In RPCs, a new field is added to TBloomFilter to identify filters that
are always true.
The HdfsParquetScanner checks to see if filters would always return true
for any element, and disables them if so.
There is some miscellaneous cleanup in this patch, particularly the
removal of unused members in BloomFilter.
This patch has been manually tested on queries that would otherwise take
a long time to time-out. A unit test was added to ensure that queries do
not wait.
Change-Id: I04b3e6542651c1e7b77a9bab01d0e3d9506af42f
Reviewed-on: http://gerrit.cloudera.org:8080/2475
Tested-by: Internal Jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
When running with ASAN enabled, runtime filters may take a lot longer to
be produced, triggering timeouts in the filter tests. This patch triples
the timeout time.
We still want the timeout to be reasonable as protection against
excessive regressions in filter production time, which is why I've not
set the timeout to a very large value, plus if the test fails and
filters aren't produced we don't want to hang the build for a large
timeout delay.
Change-Id: Ife1d36a78d6ad587462fe112afda573f6e480441
Reviewed-on: http://gerrit.cloudera.org:8080/2609
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
This patch adds functional tests for runtime filters. It relies on
setting RUNTIME_FILTER_WAIT_TIME_MS high enough to ensure that filters
are received.
To make the test files more readable, this patch also adds a new COMMENT
section to the test syntax, and allows blank spaces between queries so
that the separation of different test cases can be made more obvious.
Currently missing is a test for disabling probe-side filters based on
selectivity, as we lack suitable tables to trigger the disable condition.
Change-Id: I94d617c6d23ffa394a6eb7ead56f1cfb701e0d90
Reviewed-on: http://gerrit.cloudera.org:8080/2603
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: Internal Jenkins
Added the ability for the "GRANT/REVOKE ALL ON SERVER TO ROLE <role>"
statement to optionally take a server name parameter as:
"GRANT/REVOKE ALL ON SERVER <server> TO ROLE <role>" since Hive
allows this.
The specified server name is checked against the expected server
name from the config during analysis, and an exception is thrown if
they do not match.
Change-Id: Id6c136d9a171ec062d4ff803682d026422497e8b
Reviewed-on: http://gerrit.cloudera.org:8080/2296
Tested-by: Internal Jenkins
Reviewed-by: Thomas Tauber-Marshall <tmarshall@cloudera.com>
This commit fixes the following two issues
- A drop table statement can drop a view with same name when "IF EXISTS" is specified.
- A drop view statement can drop a table with same name when "IF EXISTS" is specified.
This happens due to lack of checks in the Catalog before the drop executes.
Change-Id: I0d35cd1f50d9b8d50223660f753c56529cbbc311
Reviewed-on: http://gerrit.cloudera.org:8080/2458
Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com>
Tested-by: Internal Jenkins
This merges the 'feature/kudu' branch with cdh5-trunk as of commit:
055500cc753f87f6d1c70627321fcc825044e183
This patch is not a pure merge patch in the sense that goes beyond conflict
resolution to also address reviews to the 'feature/kudu' branch as a whole.
The review items and their resolution can be inspected at:
http://gerrit.cloudera.org:8080/#/c/1403/
Change-Id: I6dd4270cd17a4f5c02811c343726db3504275a92
Previously, thread resource manager only supports a single callback
for each resource pool. The callback is invoked when a thread token
is available. This mostly works as scan node is the only consumer
and there is usually one scan node in a plan fragment. As shown in
IMPALA-3064 and IMPALA-561, it's possible to generate a plan fragment
with more than one scan nodes. In which case, one of the scan nodes
may be running with single thread and in debug builds, a DCHECK will
be hit.
This change fixes the problem by allowing more than one callbacks in
a given resource pool. The thread resource manager will go through all
the registered callbacks in round robin fashion.
This change also adds a missing thread token release call in
HdfsScanNode::ThreadTokenAvailableCb().
Change-Id: Iddfff1feef0b59d407994ad3bc560166acbfa623
Reviewed-on: http://gerrit.cloudera.org:8080/2430
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
The bug:
Evaluating !empty() predicates at non-scan nodes interacts
poorly with our BE projection of collection slots. For example,
rows could incorrectly be filtered if a !empty() predicate is
assigned to a plan node that comes after the unnest of the
collection that also performs the projection.
The fix:
This patch reworks the generation of !empty() predicates
introduced in IMPALA-2663 for correctness purposes.
The predicates are generated in cases where we can ensure that
they will be assigned only by the parent scan, and no other
plan node.
The conditions are as follows:
- collection table ref is relative and non-correlated
- collection table ref represents the rhs of an inner/cross/semi join
- collection table ref's parent tuple is not outer joined
Change-Id: Ie975ce139a103285c4e9f93c59ce1f1d2aa71767
Reviewed-on: http://gerrit.cloudera.org:8080/2399
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Reviewed-by: Silvius Rus <srus@cloudera.com>
Tested-by: Internal Jenkins
Add a custom cluster test that tests for delays in registering data
stream receivers. We add a stress option to artificially delay this
registration to ensure that it can be handled correctly.
Change-Id: Id5f5746b6023c301bacfa305c525846cdde822c9
Reviewed-on: http://gerrit.cloudera.org:8080/2306
Tested-by: Internal Jenkins
Reviewed-by: Silvius Rus <srus@cloudera.com>
Fix a bug in which Impala only reads the first stream
of a multi-stream bz2/gzip file.
Changes the bz2 decoder to read the file in a streaming
fashion rather than reading the entire file into memory
before it can be decompressed.
Change-Id: Icbe617d03a69953f0bf3aa0f7c30d34bc612f9f8
(cherry picked from commit b6d0b4e059329633dc50f1f73ebe35b7ac317a8e)
Reviewed-on: http://gerrit.cloudera.org:8080/2219
Reviewed-by: Juan Yu <jyu@cloudera.com>
Tested-by: Internal Jenkins
We need to skip queries that select from tables wiht nested types is
running with the old aggs and joins. To achieve this, move the failing
test to a separate test and use the skip decorator.
Change-Id: Iaf1351c711b524be66a99084657926909425cbff
Reviewed-on: http://gerrit.cloudera.org:8080/2272
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
After this patch structs can be parsed/created with field names
that are regular identifiers or keywords, even if unquoted.
This fix is needed for parsing type strings stored in the
Hive Metastore which could contain unquoted identifiers that
correspond to Impala keywords.
The parser changes required an upgrade of Cup and its Maven plugin.
In the old version, the generated parser would not compile because
of a giant method that exceeded the JVM maximum allowed size for a
single method.
Change-Id: Ic989c7afd034216f6db4c8f9f3901c025cceb524
Reviewed-on: http://gerrit.cloudera.org:8080/2249
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
We were missing handling booleans in the Kudu scanner, though
we handled them in the sink.
This patch fixes this issue and adds some tests.
Change-Id: If8edbe85ae257c6374eddf757845c1ec917b1693
In the frontend we support creating Kudu tables with VARCHAR
but in the backend we don't handle it. Moreover we were swallowing
the error in release mode, causing inserts to just skip values when
this type is used.
This patch adds support for VARCHAR, along with the corresponding
tests.
Change-Id: Ic734890e1b3aae2eef1e0a3d45a7561d02eeb917
This commit adds a new feature to persist hive/java udfs across
catalog restarts. IMPALA-1748 already added this for non-java
udfs by storing them in parameters map of the Db object and
reading them back at catalog startup. However we follow a
different approach for hive udfs by converting them to Hive's
function format and adding them as hive functions to the metastore.
This makes it possible to share udfs between hive and Impala as the
udfs added from one service are accessible to other. This commit
takes care of format conversions between hive and impala and user
can just add function once in either of the services.
Background: Hive and impala treat udfs differently. Hive resolves the
evaluate function in the udf class at runtime depending on the data
types of the input arguments. So user can add one function by name and
can pass any arguments to it as long as there is a compatible evaluate
function in the udf class. However Impala takes the input types of the
udf as a part of function definition (that maps to only one evaluate
function) and loads the function only for those set of input argument
types. If we have multiple 'evaluate' methods, we need to add multiple
functions one for each of them.
This commit adds new variants of CREATE | DROP FUNCTIONS to Impala which
lets the user to create and drop hive/java udfs without input argument
types or return types. Catalog takes care of loading/dropping the udf
signatures corresponding to each "evaluate" method in the udf symbol
class. The syntax is as follows,
CREATE FUNCTION [IF NOT EXISTS] <function name> <function_opts>
DROP FUNCTION [IF EXISTS] <function name>
Examples:
CREATE FUNCTION IF NOT EXISTS foo location '/path/to/jar' SYMBOL='TestUdf';
CREATE FUNCTION bar location '/path/to/jar' SYMBOL='TestUdf2';
DROP FUNCTION foo;
DROP FUNCTION IF EXISTS bar;
The older way of creating hive/java udfs with specific signature is still supported,
however they are *not* persisted across restarts. So a restart of catalog can
wipe them out. Additionally this commit also loads all the compatible java udfs
added outside of Impala and they needn't be separately loaded. One thing
to note here is that the functions added using the new CREATE FUNCTION
can only be dropped using the new DROP FUNCTION syntax (without
signature). The same rule applies for the java udfs added using the old
CREATE FUNCTION syntax (with signature).
Change-Id: If31ed3d5ac4192e3bc2d57610a9a0bbe1f62b42d
Reviewed-on: http://gerrit.cloudera.org:8080/2250
Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com>
Tested-by: Internal Jenkins
We do not have exceptions enabled for codegen'd code, so exceptions
thrown by functions called by codegen'd functions cannot be caught by
the codegen'd functions. TimestampValue::UnixTimeToPtime() has a
try/catch around boost::posix_time::ptime_from_tm(), but since it was
inlined into the TimestampFunctions::FromUnix() IR the try/catch
didn't work. This patch moves the UnixTimeToPtime() implementation to
the .cc file so it doesn't get included in the IR. It does the same
for TimestampParser::Parse() in case it gets inlined into IR code as
well.
Change-Id: Ic0af73629e1e3b6bf18cbf5d832973712b068527
Reviewed-on: http://gerrit.cloudera.org:8080/2210
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Internal Jenkins
Hive allows udfs with primitive data types as return values (along
with Writables) and input arguments. This commmit adds this support
for Impala.
Change-Id: I2ec24eab5a824772a8618d7fb97ae5c7ea2a0e39
Reviewed-on: http://gerrit.cloudera.org:8080/2207
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: Internal Jenkins
Test files in testdata/workloads/functional-query/queries/QueryTest
are parsed by test_file_parser.py, which used to ignore everything
before the first ==== line as a file header. This change fixes all
affected files.
This change also modifies the test file parser to forbid headers
starting with what looks like a subsection title ('----'), which
should prevent the reintroduction of similar errors in the future.
Change-Id: Iaa1bc5ffd02782e24289c7843dcb35401c334519
Reviewed-on: http://gerrit.cloudera.org:8080/2220
Reviewed-by: Lars Volker <lv@cloudera.com>
Tested-by: Internal Jenkins
AllocBuffer() and CopyStringVal() are two helper functions used by
various UDAs to allocate buffers for StringVal during their Init()
and Update() functions. Previously, these functions assumed that
the buffer length is always greater than 0. That turned out to be
an invalid assumption. This change removes this assumption and
handles zero-length StringVal by initializing its 'ptr' to NULL and
'len' to 0. A new test is also added to exercise this case.
Change-Id: Ia1e4140376c65ca3c734c40ecc3cce15b8bf2d3f
Reviewed-on: http://gerrit.cloudera.org:8080/2211
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
This test query is supposed to check the error path for when a
collection buffer cannot be allocated. However, it's flaky because the
collection allocations are not very big (< 2KB), so it's possible for
a different operator to trigger OOM.
I think the correct solution is to create a test file that contains
very large collections, so a large collection allocation will trigger
OOM, rather than many small collection allocations. For now though,
let's disable the specific collection allocation check to unblock the
build, even though we risk losing coverage.
Change-Id: Iab4c9b605186926c522cf692246a37882fbdfcdb
Reviewed-on: http://gerrit.cloudera.org:8080/2208
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Internal Jenkins
A regression test for IMPALA-2265, IMPALA-2559 expected a
query to fail with an OOM but the mem limit is now too high.
This reduces the mem limit of the test case to be as low as
it can be without failing to set up the operators.
Change-Id: I056c3ad4067e5466e3690c3b4d597b9815a7a234
Reviewed-on: http://gerrit.cloudera.org:8080/2186
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Internal Jenkins
(cherry picked from commit 45ba3109e752dfdeefdf5627a5d57079f73b24c9)
The test was recently reenabled in commit
71a0a7d998702781ae44270f8c742b10c34c0efc.
Continue running the test but loosen the memory limit and don't check
the runtime profile. The memory limits for this set of tests needs
revisiting in any case.
Change-Id: I195e8ad3b67c8ff85d5d15c2646a13f5feb57553
Reviewed-on: http://gerrit.cloudera.org:8080/2183
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
(cherry picked from commit 51632f39a45ba9deac9b86bbdb14ff10cbee35ac)
A couple of tests were disabled because of IMPALA-1305. Now that the fix
is in, those tests can be reenabled. I ran them in a loop to make sure
that they weren't flaky.
Also fix the spelling mistake in the file name.
Change-Id: I1bfcc619911a92d93b871be3a14852aa11f78da9
Reviewed-on: http://gerrit.cloudera.org:8080/2150
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
This patch implements codegen support for aggregations with decimal
input and intermediate type. For the following benchmark query:
SELECT l_discount, count(*) AS cnt
FROM biglineitem
GROUP BY l_discount
HAVING cnt > 9999999999999
Query time went from 8.85s to 3.74s (2.4x faster).
Change-Id: I25934fcd6324e5bf1fa6859496107bf2ec68b8d3
Reviewed-on: http://gerrit.cloudera.org:8080/2050
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Internal Jenkins
The bug: In SingleNodePlanner.createInlineViewPlan() we need to wrap some
exprs with TupleIsNullPredicates to preserve correctness if the inline view
is outer joined. The bug was that we used to perform this wrapping on
the rhs of the inline view's smap, and not the final output smap after
those rhs exprs have been resolved against the physical output of the
inline view's plan root. As a result, the TupleIsNullWrapping did not
work correctly for deeply nested inline views with exprs that require
wrapping at various nesting levels.
The fix: Resolve the exprs against the physical output of the inline view's
plan root before performing the TupleIsNullPredicate wrapping.
Change-Id: I183bba6a36bf5e19a88687ed8c82977ae769ddf4
Reviewed-on: http://gerrit.cloudera.org:8080/2092
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
test_alloc_fail_update() aims to stress memory allocation
failure in the Update(), Serialize() and/or Finalize() functions
of UDAs. However, this test included some UDFs which allocated
memory in their Init() functions and not during their Update()
functions. This change removes those UDFs from the test.
Change-Id: I1ecc7e838e34ebc9ea3c878fee8ea2497b5fa23e
Reviewed-on: http://gerrit.cloudera.org:8080/2005
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
IMPALA-2862 fixed parsing for regular expressions in the result
verifier. This change fixes a test that had a broken regular expression,
which was not caught by the exhaustive test suite.
I search for tests with a similar issue but couldn't find any:
git grep "regex:[^,]\+'"
Change-Id: I3aaca6bdfdc1eaab715929aa5fc6b64e6c969656
Reviewed-on: http://gerrit.cloudera.org:8080/2089
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
When the query option OPTIMIZE_PARTITION_KEY_SCANS is true, we may
acquire the partition key values from the metadata and generate a
union node containing constant expressions only. There is a bug in
the planner when generating the union node as it skips evaluating
the constant expressions for unmaterialized slots but union node
expects an entry in the constant expression lists for each slot
in the tuple descriptor even if the slot is not materialized.
This change fixes the problem by inserting a dummy null values
in the constant expression list for unmaterialized slots and lets
the union node filter them out. A test is also added to verify
the fix.
Change-Id: I9ed49dca0101b96bd9b20e6d1e5b1d56f654e911
Reviewed-on: http://gerrit.cloudera.org:8080/2067
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
The bug was that we were not updating child_row_idx_
when the output batch was at capacity, leading us to
double count that last child_row_idx_, and incorrectly
returning extra rows.
Change-Id: I85b2f1c146861ec7756887b0d2c574365d90233e
Reviewed-on: http://gerrit.cloudera.org:8080/2044
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
The bug: In order to preserve tuple nullability information through analytic sorts
we materialize the relevant expressions that contain TupleIsNullPredicates, and
with appropriate changes to the analytic sort's output smap. However, in some
edge cases, we incorrectly materialized an expr with a TupleIsNullPredicate
that could not be evaluated at that sort node because the tuple ids referenced
by the expr were not produced by the sort's input. For example, this scenario
was possible when a constant expr was wrapped in a TupleIsNullPredicate, and our
isBoundByTupleIds() check failed to filter out the expr from materialization
at the analytic sort.
The fix: Our existing code in the AnalyticPlanner already does the right thing.
We were simply missing the implementation of TupleIsNullPredicate.isBoundByTupleIds().
Change-Id: I72774f698545220922dd8ffbfa514aa87d26f97d
Reviewed-on: http://gerrit.cloudera.org:8080/2008
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
This commit fixes an issue where wrong results are returned if an EXISTS subquery
contains a HAVING clause and non-equality correlated binary predicates. This case does
not have a valid rewrite as the HAVING clause needs to be applied after the correlated
predicates have been evaluated. With this fix, we detect cases like this and throw an
AnalysisException.
Change-Id: I159f956e2b01f408601829b5d2afcf11d76bedcd
Reviewed-on: http://gerrit.cloudera.org:8080/1927
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: Internal Jenkins
Test results can be verified using regular expressions. The extraction
of the regular expression substring from the expected test results had a
bug where only the first character of an expression was considered. This
lead to wrong but undetected test results.
Change-Id: Ia670da6e0758455a86dc44744b96b9465d890af3
Reviewed-on: http://gerrit.cloudera.org:8080/1818
Reviewed-by: Lars Volker <lv@cloudera.com>
Tested-by: Internal Jenkins
A UNION is special because it may cause a scan node to be started
without any scan ranges. The Kudu scanner didn't expect that scenario
and would hang waiting for data from scanner threads that would never be
started. The fix is to exit early when there are no scan ranges.
Change-Id: Id53fb880ba23ee9bbcf3169598f97fa1a3285dd9
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/10044
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: jenkins
This commit improves the performance of DDL statements on tables with
large number of partitions. Previously, the catalog would force-reload
the entire table metadata during the execution of DDL and insert
statements, causing significant delays for tables with large number of
partitions. With this commit the catalog is reusing any cached table
entries to partially reload table metadata for only those partitions
that have been modified. With this change we've improved the performance
of some DDL and insert statements by at least 4-5X.
This commit also adds basic table-level locking to protect table
metadata from concurrent DDL operations.
Preliminary performance measurements
-----------------------------------
Workload: insert into table partition () select ... limit 10
Iterations: 10
Num partitions OLD (avg time sec) NEW (avg time sec)
1K 1.15 0.45
5K 3.65 0.9
10K 5.75 1.38
15K 10.1 2.02
30K 25.4 4.46
Workload: alter table partition() set location...
Iterations: 10
Num partitions OLD (avg time sec) NEW (avg time sec)
1K 0.8 0.47
5K 4.3 0.71
10K 7.1 1.2
15K 13.2 1.8
30K 26.8 3.4
Change-Id: I4da7fb6df0a71162b0cb60e6025a4019cb9572bf
Reviewed-on: http://gerrit.cloudera.org:8080/1706
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: Internal Jenkins
We cannot run certain nested types queries with the legacy joins/aggs,
so to fix a build I just moved a recently added test into a different
.test file that already does not run with legacy joins/agggs.
Change-Id: I0ec0e61535ad01333129bd49beca4aa481f04d74
Reviewed-on: http://gerrit.cloudera.org:8080/1918
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: Internal Jenkins
This commit fixes a DDL test that was failing because some newly added test cases were
using a database that had been dropped by another test case. The temporary fix is to use
fully qualified names for the specified tables.
Change-Id: I3bb022e2497283faeb84c85f922cda95beca2a32
Reviewed-on: http://gerrit.cloudera.org:8080/1909
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
MathFunctions::RandPrepare() allocates a 4-bytes seed and
stores it in the FunctionContext's thread local state.
However, it was never freed. This change fixes the problem
by adding a close function for Rand() so it has a chance to
free the seed. A new test is also added to verify the fix.
Change-Id: Ibcc2e1ca0d052b86defe80aad471f9fdaac5a453
Reviewed-on: http://gerrit.cloudera.org:8080/1855
Reviewed-by: Michael Ho <kwho@cloudera.com>
Tested-by: Internal Jenkins
