Enforces that the planner treats IS NOT DISTINCT FROM as eligible for
hash joins, but does not find the minimum spanning tree of
equivalences for use in optimizing query plans; this is left as future
work.
Change-Id: I62c5300b1fbd764796116f95efe36573eed4c8d0
Reviewed-on: http://gerrit.cloudera.org:8080/710
Reviewed-by: Jim Apple <jbapple@cloudera.com>
Tested-by: Internal Jenkins
input_stream_ is now set to NULL on eos, but execution can
continue through the function to l753 which touches
input_stream_ without checking for NULL. This occurs
occasionally when the transfer of memory from the mem pool
to the output row batch occurs at eos.
Change-Id: I8ca88ef10d48e19cfde7f3c6de9512eefcae561e
Reviewed-on: http://gerrit.cloudera.org:8080/1757
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Internal Jenkins
Fixes tests in show.test that executes 'show files' on the
'insert_string_partitioned' table in our functional db. The expected
output relied on modifications to 'insert_string_partitioned' made
in test_insert.py. Tests should not rely on the overall ordering
of test execution.
This patch also fixes 'show files' to produce a consistently ordered
output.
Change-Id: Ic736b94b70677b0e3f4f8a9838ffdfdde2ba17ab
Reviewed-on: http://gerrit.cloudera.org:8080/1748
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
The original purpose of the escapechartesttable was to test
Impala's behavior on text tables that have the same character
as line terminator and escape character. Recent changes in
Hive have made creating such a table impossible because
1) Only newline is allowed as the line terminator
2) Newline is forbidden as the escape character
See HIVE-11785 for details on the Hive changes.
This commit removes escapechartesttable and all associated
tests, but does not add the same enforcement rules as Hive.
These enforcement rules should be added in a follow-on change.
Change-Id: I2bd9755f4c2cc3d7dfd8d67c3759885951550f08
Reviewed-on: http://gerrit.cloudera.org:8080/1690
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Alex Behm <alex.behm@cloudera.com>
This commit fixes a bug where catalog incorrectly converts all sentry
authorizables to lower case. Since hdfs URIs are case sensitive,
this bug can result in incorrect grants when the URIs have uppercase
letters.
Change-Id: I642c34d9046729dc904cc45871c7d7959ae828bc
Reviewed-on: http://gerrit.cloudera.org:8080/1675
Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com>
Tested-by: Internal Jenkins
FunctionContext::Allocate(), FunctionContextImpl::AllocateLocal()
and FunctionContext::Reallocate() allocate memory without taking
memory limits into account. The problem is that these functions
invoke FreePool::Allocate() which may call MemPool::Allocate()
that doesn't check against the memory limits. This patch fixes
the problem by making these FunctionContext functions check for
memory limits and set an error in the FunctionContext object if
memory limits are exceeded.
An alternative would be for these functions to call
MemPool::TryAllocate() instead and return NULL if memory limits
are exceeded. However, this may break some existing external
UDAs which don't check for allocation failures, leading to
unexpected crashes of Impala. Therefore, we stick with this
ad hoc approach until the UDF/UDA interfaces are updated in
the future releases.
Callers of these FunctionContext functions are also updated to
handle potential failed allocations instead of operating on
NULL pointers. The query status will be polled at various
locations and terminate the query.
This patch also fixes MemPool to handle the case in which malloc
may return NULL. It propagates the failure to the callers instead
of continuing to run with NULL pointers. In addition, errors during
aggregate functions' initialization are now properly propagated.
Change-Id: Icefda795cd685e5d0d8a518cbadd37f02ea5e733
Reviewed-on: http://gerrit.cloudera.org:8080/1445
Reviewed-by: Michael Ho <kwho@cloudera.com>
Tested-by: Internal Jenkins
SHOW CREATE TABLE now supports views. It returns a CREATE VIEW statement
with column names and the original sql statement.
Authorization allows SHOW CREATE TABLE to be run on view if the user has
VIEW_METADATA privilege on the view and SELECT privilege on all
underlying views and table.
E.g. "SHOW CREATE TABLE some_view" returns output of form:
CREATE VIEW a_database.some_view (id, bool_col, tinyint_col) AS
SELECT id, bool_col, tinyint_col FROM functional.alltypes
Change-Id: Id633af2f5c1f5b0e01c13ed85c4bf9c045dc0666
Reviewed-on: http://gerrit.cloudera.org:8080/713
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
Allow Impala to start only with a running HMS (and no additional services like HDFS,
HBase, Hive, YARN) and use the local file system.
Skip all tests that need these services, use HDFS caching or assume that multiple impalads
are running.
To run Impala with the local filesystem, set TARGET_FILESYSTEM to 'local' and
WAREHOUSE_LOCATION_PREFIX to a location on the local filesystem where the current user has
permissions since this is the location where the test data will be extracted.
Test coverage (with core strategy) in comparison with HDFS and S3:
HDFS 1348 tests passed
S3 1157 tests passed
Local Filesystem 1161 tests passed
Change-Id: Ic9718c7e0307273382b1cc6baf203ff2fb2acd03
Reviewed-on: http://gerrit.cloudera.org:8080/1352
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
Readability: Alex Behm <alex.behm@cloudera.com>
As part of change, refactor catalog and frontend functions to return
TDatabase/Db objects instead of just the string names of databases -
this required a lot of method/variable renamings.
Add test for creating database with comment. Modify existing tests
that assumed only a single column in SHOW DATABASES results.
Change-Id: I400e99b0aa60df24e7f051040074e2ab184163bf
Reviewed-on: http://gerrit.cloudera.org:8080/620
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
When building hash tables for the build side in partitioned
hash join or aggreagtion, we will evaluate the build or probe
side expressions to compute the hash values for each TupleRow.
Evaluation of certain expressions (e.g. CastToChar) requires
"local" memory allocation. "Local" memory allocation is supposed
to be freed after processing each row batch.
However, the calls to free local allocations are missing in
PartitionedHashJoinNode::BuildHashTableInternal() and
PartitionedAggregationNode::ProcessStream(). This causes all
"local" memory allocation to accumulate potentially for the
entire duration of the query or until GetNext() is called.
This may lead to unnecessary memory allocation failure as
memory limit is exceeded.
This patch calls ExecNode::FreeLocalAllocations() at least once
per row-batch when building hash tables. It also adds the missing
checks for the query status in the loop building hash tables.
Please note that QueryMaintenance() isn't called due to its
overhead in memory limit checks.
Change-Id: Idbeab043a45b0aaf6b6a8c560882bd1474a1216d
Reviewed-on: http://gerrit.cloudera.org:8080/1448
Reviewed-by: Michael Ho <kwho@cloudera.com>
Tested-by: Internal Jenkins
This patch reduces memory usage of scanners by adjusting how batch
capacity is checked and handled and by freeing unneeded memory.
Change RowBatch::AtCapacity(MemPool) so that batches with no rows
cannot hold onto an unbounded amount of memory - instead they
will pass these batches up operator tree so that the resources
can be freed.
The Parquet scanner also only checked capacity every 1024 rows.
With large rows (e.g. nested collections), it can overrun the
intended 8mb limit. It also didn't include the MemPool usage
in its checks. After the change the scanner will produce smaller
batches if rows contain large nested collections or strings.
I benchmarked this with a scan of the nested TPC-H customers
tables. The row batch sized decrease from ~16MB to ~8MB. If the
nested collections were larger this would be more drastic.
Also pass at capacity up the tree if no rows passed the conjuncts in
the DataSourceScanNode and Parquet scanner so that resources can be
freed.
HdfsTableSink is modified to avoid the incorrect assumption that a batch
only has 0 rows at eos. It is also refactored to pass a related flag as
an argument to make the semantics clearer.
Two simple benchmarks (one column and many columns) shows no change
in scanner performance:
> set num_scanner_threads=1;
> select count(l_orderkey) from biglineitem;
> select count(l_orderkey), count(l_partkey), count(l_suppkey),
count(l_returnflag), count(l_quantity), count(l_linenumber),
count(l_extendedprice), count(l_linestatus), count(l_shipdate),
count(l_commitdate) from biglineitem;
Change-Id: I3b79671ffd3af50a2dc20c643b06cc353ba13503
Reviewed-on: http://gerrit.cloudera.org:8080/1239
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
This commmit implements DESCRIBE DATABASE [FORMATTED|EXTENDED] <db_name>.
Without FORMATTED|EXTENDED this statement only prints database's location and
comment. With FORMATTED|EXTENDED it will output all the properties of the database
(e.g. OWNER and PARAMETERS). Currently we only retrieve privileges stored in
hive metastore.
This commit also implements DESCRIBE EXTENDED <table>, which is the same
as DESCRIBE FORMATTED <table> for consistency purpose.
Change-Id: I2a101ec0e3d27b344fcb521eb00e5bdbcbac8986
Reviewed-on: http://gerrit.cloudera.org:8080/804
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: Internal Jenkins
Status was ignored by the callers of CreateMerger(), which caused the
sorter to continue in a bad state, leading to this DCHECK.
Change-Id: I1aae11d142cdbf3923289768e296dbf340f049e9
Reviewed-on: http://gerrit.cloudera.org:8080/1367
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
This change adds more analysis checks to verify the location
of the table or partition specified in a "CREATE/ALTER TABLE
... CACHED IN ..." statement can actually be cached. Caching
is only supported for HDFS locations.
If table-wide caching is enabled for a table, adding a partition
at an uncacheable location will be disallowed for that table
unless the attribute "UNCACHED" is explicitly specified.
Enabling table-wide caching for a table at an uncacheable
location or a table with partitions at uncacheable locations
will also be disallowed. However, caching can still be enabled
for individual partitions whose underlying locations
support caching.
Change-Id: I2299c9285126f4b035360f2ef902147188ccd5f1
Reviewed-on: http://gerrit.cloudera.org:8080/1373
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Reviewed-by: Michael Ho <kwho@cloudera.com>
Tested-by: Internal Jenkins
There was an incorrect DCHECK in the parquet scanner. If abort_on_error
is false, the intended behaviour is to skip to the next row group, but
the DCHECK assumed that execution should have aborted if a parse error
was encountered.
This also:
- Fixes a DCHECK after an empty row group. InitColumns() would try to
create empty scan ranges for the column readers.
- Uses metadata_range_->file() instead of stream_->filename() in the
scanner. InitColumns() was using stream_->filename() in error
messages, which used to work but now stream_ is set to NULL before
calling InitColumns().
Change-Id: I8e29e4c0c268c119e1583f16bd6cf7cd59591701
Reviewed-on: http://gerrit.cloudera.org:8080/1257
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
This only enforced the defaults in Hive. Users how manually choose to
change the schema in Hive may trigger these new analysis exceptions in
this commit unnecessarily. The Hive issue tracking the length
restrictions is
https://issues.apache.org/jira/browse/HIVE-9815
Change-Id: Ia30f286193fe63e51a10f0c19f12b848c4b02f34
Reviewed-on: http://gerrit.cloudera.org:8080/721
Reviewed-by: Jim Apple <jbapple@cloudera.com>
Tested-by: Internal Jenkins
This patch implements a new built-in function
regexp_match_count. This function returns the number of
matching occurrences in input.
The regexp_match_count() function has the following syntax:
int = regexp_match_count(string input, string pattern)
int = regexp_match_count(string input, string pattern,
int start_pos, string flags)
The input value specifies the string on which the regular
expression is processed.
The pattern value specifies the regular expression.
The start_pos value specifies the character position
at which to start the search for a match. It is set
to 1 by default if it's not specified.
The flags value (if specified) dictates the behavior of
the regular expression matcher:
m: Specifies that the input data might contain more than
one line so that the '^' and the '$' matches should take
that into account.
i: Specifies that the regex matcher is case insensitive.
c: Specifies that the regex matcher is case sensitive.
n: Specifies that the '.' character matches newlines.
By default, the flag value is set to 'c'. Note that the
flags are consistent with other existing built-in functions
(e.g. regexp_like) so certain flags in IBM netezza such as
's' are not supported to avoid confusion.
Change-Id: Ib33ece0448f78e6a60bf215640f11b5049e47bb5
Reviewed-on: http://gerrit.cloudera.org:8080/1248
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
This patch extends the SHOW statement to also support
user-defined functions and user-defined aggregate functions.
The syntax of the new SHOW statements is as follows:
SHOW CREATE [AGGREGATE] FUNCTION [<db_name>.]<func_name>;
<db_name> and <func_name> are the names of the database
and udf/uda respectively.
Sample outputs of the new SHOW statements are as follows:
Query: show create function fn
+------------------------------------------------------------------+
| result |
+------------------------------------------------------------------+
| CREATE FUNCTION default.fn() |
| RETURNS INT |
| LOCATION 'hdfs://localhost:20500/test-warehouse/libTestUdfs.so' |
| SYMBOL='_Z2FnPN10impala_udf15FunctionContextE' |
| |
+------------------------------------------------------------------+
Query: show create aggregate function agg_fn
+------------------------------------------------------------------------------------------+
| result |
+------------------------------------------------------------------------------------------+
| CREATE AGGREGATE FUNCTION default.agg_fn(INT) |
| RETURNS BIGINT |
| LOCATION 'hdfs://localhost:20500/test-warehouse/libudasample.so' |
| UPDATE_FN='_Z11CountUpdatePN10impala_udf15FunctionContextERKNS_6IntValEPNS_9BigIntValE' |
| INIT_FN='_Z9CountInitPN10impala_udf15FunctionContextEPNS_9BigIntValE' |
| MERGE_FN='_Z10CountMergePN10impala_udf15FunctionContextERKNS_9BigIntValEPS2_' |
| FINALIZE_FN='_Z13CountFinalizePN10impala_udf15FunctionContextERKNS_9BigIntValE' |
| |
+------------------------------------------------------------------------------------------+
Please note that all the overloaded functions which match
the given function name and category will be printed.
This patch also extends the python test infrastructure to
support expected results which include newline characters.
A new subsection comment called 'MULTI_LINE' has been added
for the 'RESULT' section. With this comment, a test can
include its multi-line output inside [ ] and the content
inside [ ] will be treated as a single line, including the
newline character.
Change-Id: Idbe433eeaf5e24ed55c31d905fea2a6160c46011
Reviewed-on: http://gerrit.cloudera.org:8080/1271
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: Internal Jenkins
multi-line text
This patch fixes the option passed to the RE2 regex matcher
so that it will count the newline character '\n' as a valid
candidate for '.'. Previously, that option was set to false
by default, causing multi-line text to fail to match against
patterns with wildcard characters in it. This patch also adds
some tests to address these cases and fixes some typos in
like-predicate.h.
Change-Id: I25367623f587bf151e4c87cc7cb6aec3cd57e41a
Reviewed-on: http://gerrit.cloudera.org:8080/1172
Reviewed-by: Michael Ho <kwho@cloudera.com>
Tested-by: Internal Jenkins
This fixes a regression introduced with:
IMPALA-1621,2241,2271,2330,2352: Lazy switch to IO buffers to reduce
min mem needed for PAGG/PHJ
Prior to that change, as soon as any partition's stream overflowed
its small buffers, all partitions' streams would be switched
immediately to IO-buffers, which would be satisfied by the initial
buffer "reservation".
After that change, individual streams are switched to IO-buffers on
demand as they overflow their small buffers. However, that change
also made it so that Partition::Spill() would eagerly switch that
partition's streams to IO-buffers, and fail the query if the buffer
is not available. The buffer may not be available because the
reserved buffers may be in use by other partition's streams.
We don't need to fail the query if the switch to IO-buffers in
Partition::Spill() fails. Instead, we should just let the streams
switch on demand as they fill up the small buffers. When that
happens, if the IO buffer is not available, then we already have a
mechanism to pick partitions to spill until we can get the IO-buffer
(in the worst case it means working our way back down to the initial
reservation). See AppendRowStreamFull() and BuildHashTables().
The symptom of this regression was that some queries would fail at a
lower memory limit than before.
Also revert the max_block_mgr_memory values back to their originals.
Additional testing: loop custom_cluster/spilling.py. We should also
remeasure minimum memory required by queries after this change.
Change-Id: I11add15540606d42cd64f2af99f4e96140ae8bb5
Reviewed-on: http://gerrit.cloudera.org:8080/1228
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
Failure due to an issue with NULL tuples (IMPALA-2375)
where NULL tuples come from the right side of a left
outer join where the right side comes from an inline view
which produces 0 slots (e.g. the view selects a constant).
The HJ doesn't handle them correctly because the planner
inserts an IsTupleNull expr. This isn't an issue for the
PHJ because the BufferedTupleStream returns non-NULL Tuple*
ptrs even for tuples with no slots.
Per IMPALA-2375, we're going to address this after 2.3, so
moving this test case into joins-partitioned.test which only
runs on the PHJ.
Change-Id: I64cb7e8ffd60f3379aa8860135db5af8e66d686f
Reviewed-on: http://gerrit.cloudera.org:8080/1231
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: Internal Jenkins
Commit: IMPALA-2265: Sorter was not checking the returned Status of
PrepareRead
added a new test case that sets a mem_limit. The mem_limit was
calibrated against functional_parquet tables, but during exhaustive
this test is run against other formats. Other scanners will use
different amounts of memory causing this test to fail in exhaustive.
Fix the query to always run exactly as it was tuned.
Change-Id: I8140653825cb4f303ad569f087757148c756e42d
Reviewed-on: http://gerrit.cloudera.org:8080/1230
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
If an error occurs while processing an Expr, only that ExprContext
is closed and the others are still left open. This leads to a DCHECK
during teardown of the query because the destructor of ExprContext
expects it to be closed.
In this patch, we close all the ExprContexts if an error occurs in any
one.
Change-Id: Ic748bfd213511c314c59594d075048f6c6d82073
Reviewed-on: http://gerrit.cloudera.org:8080/1222
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Internal Jenkins
This query was occasionally succeeding. This patch lowers the mem
limit so it's is less likely to succeed. It also adds the specific
"Failed to allocate buffer" message to the expected error so we don't
accidentally lose coverage. This test could still become flaky in the
future (I'm not sure 4mb is guaranteed to work every time, plus it
could change in the future), but I can't think of a better solution
than to continue adjusting it or only rely on more targetted unit
tests.
Change-Id: Id585f0e3b2c77a0279efffe9dd8b8b4472225730
Reviewed-on: http://gerrit.cloudera.org:8080/1207
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Internal Jenkins
Before, Expr::IsConstant() manually specified the constant Expr
classes, but TupleIsNullPredicate and AnalyticExpr overrode
IsConstant() to always return false (which Expr::IsConstant() didn't
specify). This meant that unless the TupleIsNullPredicate was the root
expr, TupleIsNullPredicate::IsConstant() would never be called and
Expr::IsConstant() would return true. This patch changes
Expr::IsConstant() to recurse on its children, rather than having it
contain the constant logic for all expr types.
Change-Id: I756eb945e04c791eff39c33305fe78d957ec29f4
Reviewed-on: http://gerrit.cloudera.org:8080/1214
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Internal Jenkins
The sorter was dropping on the floor the returned Status of the
PrepareRead() calls. PrepareRead() tries to Pin() blocks. In some
queries with large sorts, those Pin() calls could fail with OOM,
but because the sorter was ignoring the returned Status it would
happily put the unpinned block in the vector of blocks and eventually
seg fault, because the buffer_desc_ of that block was NULL.
This patch fixes this problem and adds a test that eventually we may
want to move to the exhaustive build because it takes quite some time.
It also changes the comments of the sorter class to the doxygen style.
Change-Id: Icad48bcfbb97a68f2d51b015a37a7345ebb5e479
Reviewed-on: http://gerrit.cloudera.org:8080/1156
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
The test introduced as a part of IMPALA-2457 was flaky as the order
of the results was not enforced and could change. This patch forces
ordering and fixes this bug.
Change-Id: Ib03d2d93818f33835b267347dcd3a94062aa7475
Reviewed-on: http://gerrit.cloudera.org:8080/1189
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: Internal Jenkins
The analytic function PERCENT_RANK() is implemented as a query
rewrite in the analysis stage. An edge case where the count of the
rows returned is 1 resulted in a divide by zero which returned NaN
as the result. This patch fixes that by creating a conditional
expression.
Change-Id: Ic8d826363e4108e0246b8e844355f3382a4a3193
Reviewed-on: http://gerrit.cloudera.org:8080/1131
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Internal Jenkins
This commit fixes an issue where a predicate from the WHERE clause that
can be evaluated at a join node is incorrectly assigned to that node's
join conjuncts even if it is an outer join, thereby causing the join to
return wrong results.
Change-Id: Ibf83e4e2c7b618532b3635b312a70a2fa12a0286
Reviewed-on: http://gerrit.cloudera.org:8080/1129
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: Internal Jenkins
The bug: A query with a subplan containing a hash join with unnest nodes
on both the build and probe sides would not project the collectionn-typed
slots referenced in unnest nodes of the probe side. The reason is that
we used to first complete the hash join build before opening the probe
side. Since the build does a deep-copy those collection-typed slots
to be unnested in the probe side would not be projected.
Example query that exhibited the bug:
subplan
hash join
nested-loop join
singular row src
unnest t.c1
unnest t.c2
scan t
The tuple of 't' has two-collection typed slots, one for 't.c1', and
another for 't.c2'. If the hash join completes the build without
opening the probe side, then the 't.c2' slot would not be projected
and deep copied into the build-side hash table. That collection
would then be returned in GetNext() of the hash join.
The fix: For hash joins inside a subplan, open child(0) before doing
the build.
Change-Id: I569107b5ecafdbb75f3562707947ecc73951140c
Reviewed-on: http://gerrit.cloudera.org:8080/1128
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
The limit of 100 was determined empirically by generating deeply
nested Parquet and Avro files and then trying to run queries with
and without subplans over them (one absolute table ref vs. all relative
table refs for maximally nested subplans).
Based on those experiments we can handle up to 200 levels of nesting,
but the queries get very slow. At 300 levels, we exceed the stack space
due to the recursive implementation of the scan. Also, we decode the
rep/def levels of Parquet as uint8_t. I settled with 100 because it is
safe, future proof and reasonably high for most practical cases.
Change-Id: Iebdfa96a6dd6060387e38eaedb8ddf0f9901ac24
Reviewed-on: http://gerrit.cloudera.org:8080/905
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
These tests are failing in the old agg/join nightly. This test
references a nested types table, so we need to skip it when the old
agg/join is enabled. So, move it to a place where this already happens.
Change-Id: I09400760fd0b7506e4c127bbef92ab413d5d8615
Reviewed-on: http://gerrit.cloudera.org:8080/1143
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
Decimal tables are not generated for all file formats. The
analytic-fns test is run on some of these formats, so fails
when it cannot find the decimal_tiny table. Move the test to
the decimal test that handles this correctly.
Change-Id: Ic23b21ed90496fcc9f2f84cfd3dd92899d00498b
Impala supports reading Parquet files with multiple row groups but
with possible performance degradation due to remote reads. This patch
maximizes scan locality by allowing multiple impalads to scan the
rowgroups in their local splits. Each impalad starts a new scan range
for each split local to it if that split contains row group(s) that
need to be scanned.
Change-Id: Iaecc5fb8e89364780bc59dbfa9ae51d0d124d16e
Reviewed-on: http://gerrit.cloudera.org:8080/908
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Internal Jenkins
The commit:
IMPALA-1621,2241,2271,2330,2352: Lazy switch to IO buffers to reduce min mem needed for PAGG/PHJ
recently lowered a couple of limits from 100m to 40m, which appears to
be too aggressive based on occasionol gvm failures. Let's bump
it back up.
Change-Id: I2c3cc24841cf3a305785890329d77e4e9e74f6e5
Reviewed-on: http://gerrit.cloudera.org:8080/1125
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
The bug: During planning, when generating an EmptySetNode for a query
block (or a portion thereof) that contained relative table refs we still
populated the corresponding collection-typed slots in the parent scan,
ultimately hitting a sanity DCHECK in the BE.
The fix: Since those collection-typed slots are never used, the
corresponding parent scan should not materilaize them. When creating
an EmptySetNode we mark the appropriate collection-typed slots
as non-materialized.
Change-Id: If0b9c37c46c0e27be7f1b47c395c8c90b499e323
Reviewed-on: http://gerrit.cloudera.org:8080/1092
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
This patch changes the resolution of overloaded functions so that we
prefer functions where there is no loss of precision in argument types.
Previously, the logic would happily convert DECIMAL to FLOAT even if
there was a more suitable overload available. E.g. greatest(TINYINT,
DECIMAL) was resolved to greatest(FLOAT...) instead of greatest(DECIMAL).
This only changes behaviour when no overload exactly matches the argument
types, but the arguments can be converted with no loss of precision,
e.g. TINYINT to DECIMAL.
This patch introduces a conceptual distinction between strict and
non-strict compatibility. All contexts aside from function matching
use non-strict to support the current behavior of implicitly casting
decimals to floats/doubles.
This patch also makes resolution of overloaded functions consistent
regardless of what order functions were added to the Db - overloads are
checked in a canonical order.
Switching to this canonical order revealed further problems with
overload resolution where the correct overload was selected only because
of the order in which it was added to the database. For example, the
logic equally preferred resolving fn(STRING, TINYINT) to
fn(TIMESTAMP, INT) or fn(STRING, INT). This required changes to the
compatibility matrix.
Various cleanup and simplification of the type compatibility logic is
also included.
Change-Id: I50e657c78cdcb925b616b5b088b801510020e255
Reviewed-on: http://gerrit.cloudera.org:8080/845
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
This patch changes the Parquet scanner to check if it can't read the
full footer scan range, indicating that file has been overwritten by a
shorter file without refreshing the table metadata. Before it would
DCHECK. This patch adds a test for this case, as well as the case
where the new file is longer than the metadata states (which fails
with an existing error).
Change-Id: Ie2031ac2dc90e4f2573bd3ca8a3709db60424f07
Reviewed-on: http://gerrit.cloudera.org:8080/1084
Tested-by: Internal Jenkins
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
When a full outer join on the old (non-partitioned)
HashJoinNode, if any join fragment has 0 build rows and 0
probe rows an extra null row will be produced.
Change-Id: I75373edc4f6b3b0c23afba3c1fa363c613f23507
Reviewed-on: http://gerrit.cloudera.org:8080/1068
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
The bug was that SubplanNode::GetNext() was not explicitly setting the returned
eos to false if eos had not been reached yet. As a result, a UnionNode with
a SubplanNode as its second operand could return fewer rows than expected becasue
the eos was carried over from the previous union operand, and only a single batch
was returned from the SubplanNode.
The fix is to always set the eos return value in SubplanNode::GetNext().
Change-Id: I9f7516d7b740b9e089ea29b9fe416f3f47314e2c
Reviewed-on: http://gerrit.cloudera.org:8080/1076
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
These tests are failing intermittently for me because the "order by" clause
doesn't yield deterministic results.
Change-Id: I98eec16ec8257611a5b1419ecea401304535f374
Reviewed-on: http://gerrit.cloudera.org:8080/1067
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
The bug was that the analysis of a WITH-clause view containing a
relative table ref would register a collection-typed slot in the parent
tuple descriptor of that relative table ref. The problem is that we use
a dummy analyzer with a fresh global state for the WITH-clause analysis.
Since the descriptor table is part of the global state, it was possible
that we'd register a collection-typed slot with an item tuple id that was
the same as the parent tuple id (or another arbitrary tuple id that has
no meaning in the parent tuple). This parent/item tuple with the same id
lead to an infinite recursion in the backend.
The fix is to not register collection-typed slots in parent tuples when
analyzing a relative table ref inside a WITH-clause view. I added a new
flag to the analyzer to indicate whether it is analyzing a WITH-clause
view.
Change-Id: Ifc1bdebe4577a959aec1f6bb89a73eaa37562458
Reviewed-on: http://gerrit.cloudera.org:8080/1021
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
Fixed trivial errors: query test file was missing a required separator,
memory limit exceeded error message was wrong.
Changed nested types test to use deterministic analytic fn.
Lowered memory limit to get it failing consistently on my machine.
Changed memory limit error message to be less specific, since it can
hit different errors in some cases.
Tested by running:
py.test tests/query_test/test_nested_types.py -k tpch \
--exploration_strategy=exhaustive
Change-Id: Icf301cbeea79b825195e9f60ec250b6e167d41f3
Reviewed-on: http://gerrit.cloudera.org:8080/1029
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
It shadows the parse_status_ defined in HdfsScanner, making it so the
Parquet scanner doesn't pick up a bad parse status set in an
HdfsScanner function.
Change-Id: I1e64f1909eaf1af1b1130dda2380db7157a64c2b
Reviewed-on: http://gerrit.cloudera.org:8080/884
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Internal Jenkins
Move a very expensive semi-join test case to run only on
exhaustive so that it is not run as part of the old hj/agg
jenkins runs where it fails.
Change-Id: I4a0f915e894ceac91d86b366876e47e9cc87255a
Reviewed-on: http://gerrit.cloudera.org:8080/930
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
ExecNode::Reset() is not idempotent. It always requires a preceding
call to ExecNode::Open(). The bug was that with nested subplans the
following situation could lead to calling Reset() on the same exec
node multiple times resulting in a DCHECK or crash depending on node
being reset. The scenario is illustrated as follows.
Example Plan:
01:subplan
05:nested-loop join
02:singular row src
04:subplan
...
arbitrary exec node tree
...
03:unnest
00:scan
Sequence of calls leading to double Reset():
1. Node 04 calls Reset() on its child(1)
2. Node 04 returns from GetNext() without calling Open() on its child(1)
3. Node 01 calls Reset() on its child(1)
The problem was that SubplanNode::Reset() of node 04 used to call Reset()
on its child(1) even though child(1) had not been opened yet.
The fix is to only call Reset() on child(1) if it has been opened.
Change-Id: I1d6f5f893c9b773c05dc6cbfe99de9f74e47a888
Reviewed-on: http://gerrit.cloudera.org:8080/916
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
The bug: Several places in the sort assumed that the presence of var-len
slots in the tuple to be sorted implied having var-len blocks.
However, if all var-len slots are NULL or empty, then we can have no
var-len blocks. This case is now properly handled.
Change-Id: Ia3ad3669313e9d494ce2472af7775febfa6f247c
Reviewed-on: http://gerrit.cloudera.org:8080/913
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
