This is the first in a series of patches to clean up test_ddl.py
Summary of changes:
- Break up test_create() and corresponding .test files into:
* test_create_database()
* test_create_table()
* test_create_table_like_table()
* test_create_table_like_file()
* test_create_table_as_select()
- Merge test_nested() into the tests above
- Move a test into test_hms_integration.py
- Add a new test_ddl_base.py as base class for DDL tests.
The plan is to split up test_ddl.py into several smaller
.py files in subsequent patches.
Testing: I tested test_ddl.py and test_hms_integration.py on
exhaustive locally as well as in private builds on all filesystems.
Change-Id: I5f4c044d39e165c2535961b8d0a765c8dbbd051c
Reviewed-on: http://gerrit.cloudera.org:8080/3044
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Alex Behm <alex.behm@cloudera.com>
Adds handling and testing for a specific Parquet data corruption
scenario with plain dictionary encoded values.
The problematic scenario is when the repeat or literal count of
the RLE-encoded dictionary indexes is decoded as 0 - an invalid value.
There are several other cases of data corruption that are not yet
handled gracefully. This patch only handles one specific case.
Change-Id: Ibf406c82cdded37966f09c81e4cc1446d2b60d63
Reviewed-on: http://gerrit.cloudera.org:8080/3299
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Alex Behm <alex.behm@cloudera.com>
The Sorter's memory management logic failed to correctly manage buffers
when spilling. It would try to make use of all buffers in the system,
neglecting to account for other operators' buffer usage.
This patch adjusts the logic so that it handles contention for buffers
so long as it can get enough buffers to make progress. Instead of
precalculating the number of buffers it thinks it should be able to
pin, it just makes a best-effort attempt to pin the initial buffers
as many runs as possible, up to a limit. As long as it can pin three
runs, it can make progress.
Testing:
Added an additional test that failed before the patch without OOM.
An analytic function test that was meant to fail also started succeeding
so I had to adjust the limit there too.
Change-Id: Idfe55cc13c7f2b54cba1d05ade44cbcf6bb573c0
Reviewed-on: http://gerrit.cloudera.org:8080/2908
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Tim Armstrong <tarmstrong@cloudera.com>
Clarify relationships between classes, clean up the previous mess
where every class was friends with the other so there's an actual
distinction between public and private members. TupleIterator
is now no longer tied to TupleSorter, just Run.
Document and enforce invariants in many cases.
Factor out some functions from large functions.
Simplify and document iterator logic.
Make management of buffers when iterating over output stream more
explicitly correct: either use MarkNeedToReturn() or attach block
to the batch as appropriate. The SortedRunMerger didn't handle
resource transfer correctly, except if all the memory came from
the batch's MemPool. This patch fixes the cases when resources
are attached to the batches, but not the 'need_to_return' case.
Document that SortedRunMerger requires 'deep_copy_input' to be true
if batches can have the 'need_to_return' flag set.
Also use the atomic block exchange operation when moving between
blocks in unpinned runs to prevent pin failures at that point.
I explicitly have avoided changing the hairy block management logic
when allocating buffers for merging, that will need addressing in
a follow-up patch.
Add a SpilledRuns counter so that it's more explicit that spilling
occurred.
Testing:
Added some tests for corner cases with empty and NULL strings.
Fixed a test that previously failed with OOM but now succeeds.
Performance:
Benchmarking against old code initial revealed some regressions from
changes in inlining. Force inlining the TupleComparator::operator() and
iterator Next()/Prev() functions helped and performance seems similar or
slightly better on the targeted orderby benchmarks.
Change-Id: I9c619e81fd1b8ac50e257172c8bce101a112b52a
Reviewed-on: http://gerrit.cloudera.org:8080/2826
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Tim Armstrong <tarmstrong@cloudera.com>
With the prefetching changes, the probe expressions' local
allocations are no longer freed via QueryMaintenance() in
PHJ. Instead, they are freed explicitly in GetNext() after
an entire probe batch has been processed. Due to this
change in how we handle local allocations of probe expressions,
a DCHECK was added to verify that there is no local allocation
from the probe expression in ProcessBuildInput(). Turns out that
Expr::Open() called in ConstructBuildSide() on the probe
expressions may have caused local allocations to occur for
certain UDFs (e.g. extract()).
This change handles the situation above by freeing local
allocations of the probe expressions once before calling
ProcessBuildInput() in ConstructBuildSide(). A new regression
test is also added for this specific case.
Change-Id: I2096ca3e2093c5ab0ecc0e7ca4cd1b5f3c1ed1ed
Reviewed-on: http://gerrit.cloudera.org:8080/3253
Reviewed-by: Michael Ho <kwho@cloudera.com>
Tested-by: Internal Jenkins
This patch adds test coverage for partitioned inserts where the memory
limit will be exceeded by the table writer.
Testing:
Ran the test with exploration_strategy=exhaustive locally then ran an exhaustive
private build. Manually inspected the memory limit report to make sure
that it was behaving as expected (writer memory was being correctly
tracked, etc).
Change-Id: I8583c60d648af9eedc956315df5ac3c3d6608704
Reviewed-on: http://gerrit.cloudera.org:8080/3245
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Internal Jenkins
This change ensures that Avro tables created without column definitions
remain queryable if columns are added via ALTER TABLE. The bug was that
when synthesizing an Avro schema from the column definitions we used to
not add default values.
Change-Id: Ib86e9ba1f4329b285ae14ee299365f7291a7410e
Reviewed-on: http://gerrit.cloudera.org:8080/3219
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
Adds a new command to manually set the table-level column stats.
Syntax:
ALTER TABLE [<db_name>.]<tbl_name> SET COLUMN STATS <col_name>
('statsKey'='val','statsKey2',='val2')
Valid values for 'statsKey': numDVs, numNulls, avgSize, maxSize
The 'val' portion needs to be a number appropriate for the given stats
key (e.g., a long for numDVs, a float for avgSize).
The special value of '-1' is allowed to reset stats to 'unknown'.
The keys as well as the values are specified as string literals to be
consistent with the existing DDL for setting TBLPROPERTIES/SERDEPROPERTIES,
in particular, setting the 'numRows' table/partition property.
Testing: Ran the tests locally on exhaustive. Did private runs
on core/hdfs and core/S3.
Change-Id: I45cd8aa7241ea962788ba9ca7d0bbfd864c4304f
Reviewed-on: http://gerrit.cloudera.org:8080/3189
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
The HdfsTableSink usualy creates a HDFS connection to the filesystem
that the base table resides in. However, if we create a partition in
a FS different than that of the base table and set
S3_SKIP_INSERT_STAGING to "true", the table sink will try to write to
a different filesystem with the wrong filesystem connector.
This patch allows the table sink itself to work with different
filesystems by getting rid of a single FS connector and getting a
connector per partition.
This also reenables the multiple_filesystems test and modifies it to
use the unique_database fixture so that parallel runs on the same
bucket do not clash and end up in failures.
This patch also introduces a SECONDARY_FILESYSTEM environment variable
which will be set by the test to allow S3, Isilon and the localFS to
be used as the secondary filesystems.
All jobs with HDFS as the default filesystem need to set the
appropriate environment for S3 and Isilon, i.e. the following:
- export AWS_SECERT_ACCESS_KEY
- export AWS_ACCESS_KEY_ID
- export SECONDARY_FILESYSTEM (to whatever filesystem needs to be
tested)
TODO: SECONDARY_FILESYSTEM and FILESYSTEM_PREFIX and NAMENODE have a
lot of similarities. Need to clean them up in a following patch.
Change-Id: Ib13b610eb9efb68c83894786cea862d7eae43aa7
Reviewed-on: http://gerrit.cloudera.org:8080/3146
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Internal Jenkins
During analysis the subquery is rewritten and during that process some
previous analysis state about the Kudu columns was lost and never
repopulated. If the Kudu table has keys of different data types that
could lead to an error about incorrect data types. It may also be
possible that if the data types did match, the wrong values would be
deleted.
Change-Id: I55b6fecfd35458fbb5bc20b4be1375484d7bc3c6
Reviewed-on: http://gerrit.cloudera.org:8080/2901
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
This test had a hardcoded HDFS path which caused a failure on all
non-HDFS runs. This change simply makes the path filesystem agnostic.
Change-Id: I31e5955893f2c0d967ce6f0cab653d1020ee45b3
Reviewed-on: http://gerrit.cloudera.org:8080/3174
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
Adds a query option 'strict_mode' which treats integer and
floating pt overflows as parse errors. In the past,
overflows were ignored and the max value was returned. When
this query option is set, overflowing values are treated as if
they were completely invalid data, i.e. NULL is returned.
When abort_on_error is enabled, this means the query is
aborted.
Notes:
* DECIMAL overflow/underflow is already treated as an error.
* The handling in text-converter treats underflows the same
as overflows, so they would result in the same behavior.
However, floating point parsing never returns an underflow
today.
* We may also want to handle numeric values that are truncated
when parsing to integer types, e.g. 10.5 -> 10.
Change-Id: I7409c31ec0cb6fe0b2d9842b9f58fe1670914836
Reviewed-on: http://gerrit.cloudera.org:8080/3150
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Internal Jenkins
Bug: Impalads crash if we query an Avro table with stale metadata
Cause: This happens because avroSchema_ is not set in HdfsTable,
which is not propagated to the avro scanner and it doesn't have
appropriate checks to make sure the schema is non-null.
The patch fixes the following.
1. Avro scanner should gracefully handle the case where the avro schema
is not set. Appropriate null checks and a meaning error message have
been added.
2. This is a special case with multi-fileformat partitioned tables.
avroSchema_ should be set in HdfsTable even if any subset of the
partitions are backed by avro. Without this patch, we only set it
if the base table file format is Avro.
Change-Id: I09262d3a7b85a2263c721f3beafd0cab2a1bdf4b
Reviewed-on: http://gerrit.cloudera.org:8080/3136
Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com>
Tested-by: Internal Jenkins
- Moves the test into compute_stats.py
- Changes some test classes in compute_stats.py to inherit from
ImpalaTestSuite and not from TestComputeStats because that
will cause all tests in TestComputeStats to be run in the
subclasses again (redundantly).
- Clean up and add more coverage to testing incremental stats on
HBase which was probably broken in this commit 6b32ff06.
- Fixes a side effect that the original test had for testing
incremental stats on HBase. It computes stats on a functional
table which was not supposed to have stats.
Testing: Ran compute_stats.py on exhaustive locally in a loop 10 times.
Did a private hdfs/core run.
Change-Id: Iee8b84e30948c3c98166e08cae2666574777730c
Reviewed-on: http://gerrit.cloudera.org:8080/3074
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
Sorter can have runaway memory consumption as it never frees
local allocations made in comparator_.Less(). In addition, it
doesn't check for errors generated during expression evaluation
so it may keep sorting even after failures have occurred.
This change fixes the problem by freeing local allocations for
every n invocations of comparator_.Less() where n is the row
batch size specified in the query options. Various error checks
are also added to return early if any error is encountered.
Change-Id: I941729b4836e5dbb827d4313a0b45bc5df2fa8e1
Reviewed-on: http://gerrit.cloudera.org:8080/3116
Reviewed-by: Michael Ho <kwho@cloudera.com>
Tested-by: Internal Jenkins
With this commit runtime filters can be assigned to multiple destination
nodes (scans). For each filter, the destination nodes are determined
using equivalent classes during planning. For each filter, all its
destination nodes are in the left subtree rooted at the join node
that constructs this filter. A runtime filter may have both
local and remote targets. The backend determines how to route each
filter depending on the number and type (local, remote) of its destination
nodes.
With this commit, we enable runtime filter propagation in all the
operands of UNION [ALL|DISTINCT] nodes.
Change-Id: Iad2ce4e579a30616c469312a4e658140d317507b
Reviewed-on: http://gerrit.cloudera.org:8080/2932
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: Internal Jenkins
In 2.5 we added the ability to set per-pool default query
options. A string of key-value pairs can be specified with a
pool configuration. However, if any options fail to parse,
then all the options are ignored. We want that behavior (and
returning an error) when parsing the process-wide default
query options on startup and when parsing the options sent
from a client (e.g. in beeswax server) because an error can
be returned immediately for the triggering action at that
time (i.e. starting the impalad or submitting a query with
the options set). This behavior is bad for the pool default
query options because (a) the configuration is set by the
administrator and there's nothing we can do until a query is
submitted and (b) one invalid option shouldn't mean that
other valid options aren't set.
Change-Id: If04733b775963091b0314c65286df126fd812358
Reviewed-on: http://gerrit.cloudera.org:8080/3056
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
Before this patch, correlated exists and not exists subqueries were
rewritten as as left semi and anti joins respectively. Uncorrelated
exists subqueries were rewritten as cross joins, and uncorrelated
not-exists subqueries were not supported at all. This patch takes
advantage of the nested loop join that was recently introduced, which
allows us to rewrite both correlated and uncorrelated exists subqueries
as left semi joins and both correlated and uncorrelated not-exists
subqueries as anti joins.
Change-Id: I52ae12f116d026190f3a2a7575cda855317d11e8
Reviewed-on: http://gerrit.cloudera.org:8080/2792
Reviewed-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com>
Tested-by: Internal Jenkins
The problem: varlen data (e.g. strings) produced by aggregations is
freed by FreeLocalAllocations() after passing up the output
batch. This works for streaming operators or blocking operators that
copy their input, but results in memory corruption when the output
reaches non-copying blocking operators, e.g. SubplanNode and
NestedLoopJoinNode.
The fix: this patch makes the PartitionedAggregationNode copy out
produced string data if the node is in a subplan. Otherwise it calls
MarkNeedsToReturn() on the output batch. Marking the batch would work
in the subplan case as well, but would likely be less efficient since
it would result in many small batches coming out of the subplan.
The patch includes a test case. However, this test only exposes the
problem with an ASAN build and the --disable_mem_pools flag, which we
don't currently have automated testing for.
Change-Id: Iada891504c261ba54f4eb8c9d7e4e5223668d7b9
Reviewed-on: http://gerrit.cloudera.org:8080/2929
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
This patch adds two query options for runtime filters:
RUNTIME_FILTER_MAX_SIZE
RUNTIME_FILTER_MIN_SIZE
These options define the minimum and maximum filter sizes for a filter,
no matter what the estimates produced by the planner are. Filter sizes
are rounded up to the nearest power of two.
Change-Id: I5c13c200a0f1855f38a5da50ca34a737e741868b
Reviewed-on: http://gerrit.cloudera.org:8080/2966
Tested-by: Internal Jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
INSERTs on S3 are slower because of double buffering where we buffer
once locally and once in a staging directoy in S3 before moving the
file(s) to the final location. Also, moving the file from the staging
directory to the final location in HDFS is a quick rename which is
only a metadata operation. However, on S3, renames are not supported,
thus becoming a full file copy instead of just a metadata rename
operation.
This patch instroduces a boolean query option "s3_skip_insert_staging"
which avoids the staging step on S3 and allows the sinks to write to
the final location directly.
This trades in consistency for the sake of performance. If a node(s)
fails during the query, then we will end up with inconsistent results
in the final location.
P.S: This option is disabled for INSERT OVERWRITE queries as that
would require cleaning the destination directory before moving the
final files there. However, the coordinator is responsible for the
cleaning which takes place only after the table sinks have moved
the files to the final location. Thus, INSERT OVERWRITE queries must
still have their files moved to a staging location by the table sinks.
Performance gains:
- For non-partitioned tables, the INSERT queries run 4-4.5x faster on
S3. (Tested on a 63GB INSERT to a table)
- For heavily partitioned tables, there is considerable improvement
in the order of 4-5 minutes on queries that take ~27 minutes but
queries are still slow because of IMPALA-3482 where the catalog
takes too long to update all the metadata. (Tested with a query
that creates 2.4K partitions in a table totalling ~19GB).
Change-Id: Iff9620d41ba0d5fb1aa0c9f4abb48866fc2b0698
Reviewed-on: http://gerrit.cloudera.org:8080/2905
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Internal Jenkins
Testing: Tested the changes locally by running them in a loop
10 times. Also did a private core/hdfs run.
Change-Id: I37e1528c02e598f3fb2d673b6559d55a34bf79b4
Reviewed-on: http://gerrit.cloudera.org:8080/3002
Reviewed-by: Michael Brown <mikeb@cloudera.com>
Tested-by: Internal Jenkins
This commit fixes an issue where a GRANT ALL ON SERVER to role_name statement
followed by a REVOKE ALL ON SERVER from role_name statement would not revoke all
privileges from role_name. The problem was triggered by a specific
combination of Sentry client API calls used in Impala during
grant/revoke statements at server scope. In particular, during GRANT, Impala was using
an API call that didn't explicitly specify the privilege action (Sentry uses '*' if
no action is specified). In contrast, the corresponding REVOKE call was explicitly
specifying the privilege action to be 'ALL'. Sentry doesn't seem to
handle this case correctly, thereby failing to remove all the privileges
after a REVOKE ALL ON SERVER call. The fix from the Impala side, that
results in the correct behavior, is to always specify the privilege
action by using the appropriate API calls.
Change-Id: I6b3a0d10f5e88c6a0a10bd20f620562d2de7ab25
Reviewed-on: http://gerrit.cloudera.org:8080/2979
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: Internal Jenkins
Add missing coverage for sorting by CHAR and VARCHAR.
Add more coverage for spilling sorts.
Fix spilling tests: ensure that they actually reliably spill (many of
them had memory limits high enough that they could run entirely in
memory).
I ran this in a loop for a while to flush out flaky tests. The tests
should be fairly predictable given that they're not run concurrently
with other tests and we allocate enough block manager memory so that
each operator can obtain its reservation.
Change-Id: Ia2d2627a2c327dcdf269ea3216385b1af9dfa305
Reviewed-on: http://gerrit.cloudera.org:8080/2877
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
Now that we functionally support writes to S3 via Impala,
test_grant_revoke should not have a special case for S3, which
till this patch did the test without INSERTs.
Change-Id: Id981e7f83bf86b32d1a5b267ad3781db02337e86
Reviewed-on: http://gerrit.cloudera.org:8080/2949
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Internal Jenkins
This patch makes it a little easier to use the unique_database fixture
with .test files. The RESULTS section can now contain $DATABASE which
is replaced with the current database by the test framework.
Testing:
- ran the test locally on exhaustive
- ran the test on hdfs and the local filesystem on Jenkins
Change-Id: I8655eb769003f88c0e1ec1b254118e4ec3353b48
Reviewed-on: http://gerrit.cloudera.org:8080/2947
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
Previously Impala disallowed LOAD DATA and INSERT on S3. This patch
functionally enables LOAD DATA and INSERT on S3 without making major
changes for the sake of improving performance over S3. This patch also
enables both INSERT and LOAD DATA between file systems.
S3 does not support the rename operation, so the staged files in S3
are copied instead of renamed, which contributes to the slow
performance on S3.
The FinalizeSuccessfulInsert() function now does not make any
underlying assumptions of the filesystem it is on and works across
all supported filesystems. This is done by adding a full URI field to
the base directory for a partition in the TInsertPartitionStatus.
Also, the HdfsOp class now does not assume a single filesystem and
gets connections to the filesystems based on the URI of the file it
is operating on.
Added a python S3 client called 'boto3' to access S3 from the python
tests. A new class called S3Client is introduced which creates
wrappers around the boto3 functions and have the same function
signatures as PyWebHdfsClient by deriving from a base abstract class
BaseFileSystem so that they can be interchangeably through a
'generic_client'. test_load.py is refactored to use this generic
client. The ImpalaTestSuite setup creates a client according to the
TARGET_FILESYSTEM environment variable and assigns it to the
'generic_client'.
P.S: Currently, the test_load.py runs 4x slower on S3 than on
HDFS. Performance needs to be improved in future patches. INSERT
performance is slower than on HDFS too. This is mainly because of an
extra copy that happens between staging and the final location of a
file. However, larger INSERTs come closer to HDFS permformance than
smaller inserts.
ACLs are not taken care of for S3 in this patch. It is something
that still needs to be discussed before implementing.
Change-Id: I94e15ad67752dce21c9b7c1dced6e114905a942d
Reviewed-on: http://gerrit.cloudera.org:8080/2574
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Internal Jenkins
This change is a first step towards a more efficient Parquet scanner.
The focus is on presenting the new code flow that materializes
the table-level slots in a column-wise fashion, without going deep
into actually improving scan efficieny.
After these changes there are several obvious places that should
be optimized to realize efficiency gains.
Summary of changes
- the table-level tuples are materialized in a column-wise fashion
with new ColumnReader::ReadValueBatch() functions
- this is done by materializing a 'scratch' batch, and transferring
scratch tuples that survive filters/conjuncts to the output batch
- the tuples of nested collections are still materialized in
a row-wise fashion using the ColumnReader::ReadValue() function,
just as before
Mini benchmark
I ran the following queries on a single impalad before and after my
change using a synthetic 'huge_lineitem' table.
I modified hdfs-scan-node.cc to set the number of rows of any row
batch to 0 to focus the measurement on the scan time.
Query options:
set num_scanner_threads=1;
set disable_codegen=true;
set num_nodes=1;
select * from huge_lineitem;
Before: 22.39s
Afer: 18.50s
select * from huge_lineitem where l_linenumber < 0;
Before: 25.11s
After: 20.56s
select * from huge_lineitem where l_linenumber % 2 = 0;
Before: 26.32s
After: 21.82s
Change-Id: I72a613fa805c542e39df20588fb25c57b5f139aa
Reviewed-on: http://gerrit.cloudera.org:8080/2779
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
For IMPALA-1740 we added a test to insert.test, which creates a table and
inserts data. The table was created on HDFS by default and thus inserts with
compression enabled did not work. This change adds the required table to the
functional schema in the same way we do it for the other insert tests.
Change-Id: Ie68e7067b7a16218d27935820d5d1ce7035d2e6c
Reviewed-on: http://gerrit.cloudera.org:8080/2919
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
Instead of having a default Bloom Filter size for all runtime filters,
adjust filter size according to desired FP-rate and expected NDV from
join's build-side. Size of filter is still clipped to 4k < N < 16MB range.
If NDV estimate from planner is -1 (i.e. no stats) the default filter
size is used.
The NDV of all filters produced by the same join is currently the same
because the NDV is estimated from the cardinality of the input. In the
future, the NDV should be estimated for each filter source expr. The BE
changes anticipate this and can enable or disable individual filters if
they have differing FP rates.
Change-Id: I1fe37b8d4cfb3c52bb8e8cf0ca55e92665b87803
Reviewed-on: http://gerrit.cloudera.org:8080/2812
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: Internal Jenkins
HIVE-5795 introduced a parameter skip.header.line.count to skip header
lines from input files. This change introduces the capability to skip
an arbitrary number of header lines from csv input files on hdfs. The
size of the total file header must be smaller than
max_scan_range_length, otherwise an error will be reported. This is
necessary because scan ranges are not read in disk order, so there is
no way of identifying header lines except by counting from the start
of the first scan range.
[localhost:21000] > alter table t1 set
tblproperties('skip.header.line.count'='1');
Query: alter table t1 set tblproperties('skip.header.line.count'='1')
[localhost:21000] > select * from t1;
Query: select * from t1
+----+----+
| c1 | c2 |
+----+----+
| 1 | 1 |
| 2 | 2 |
| 3 | 3 |
+----+----+
Fetched 3 row(s) in 0.32s
[localhost:21000] > alter table t1 set
tblproperties('skip.header.line.count'='0');
Query: alter table t1 set tblproperties('skip.header.line.count'='0')
[localhost:21000] > select * from t1;
Query: select * from t1
+------+------+
| c1 | c2 |
+------+------+
| NULL | NULL |
| 1 | 1 |
| 2 | 2 |
| 3 | 3 |
+------+------+
WARNINGS: Error converting column: 0 TO INT (Data is: num1)
Error converting column: 1 TO DOUBLE (Data is: num2)
file: hdfs://localhost:20500/test-warehouse/t1/test.txt
record: num1,num2
Fetched 4 row(s) in 0.41s
Change-Id: I595f01a165d41499ca1956fe748ba3840a6eb543
Reviewed-on: http://gerrit.cloudera.org:8080/2110
Reviewed-by: Lars Volker <lv@cloudera.com>
Tested-by: Internal Jenkins
Attempting to codegen a sort where the sort expr has a CHAR type as an
intermediate result fails completely.
The problem is that ScalarFnCall checked whether its input arguments
were CHAR to disable codegen, but didn't check its output.
This patch also replaces some incorrect codegen CHAR logic that should
not be executed with DCHECKs.
Testing:
The test is a minimal reproduction of the issue. The test is executed
both by the sorter and top-n nodes so covers both cases.
Change-Id: I189073d46a10988803d572928a38f4a718690fa3
Reviewed-on: http://gerrit.cloudera.org:8080/2876
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
Filter locality was not correctly set when NUM_NODES=1 (and therefore no
distributed plan was created). The default locality should be 'local'.
This patch also fixes a bug where the initial filter routing table
wasn't printed when NUM_NODES=1.
Change-Id: I7b9a6bcc64ca6ec5fd51d63815cea25de866ef93
Reviewed-on: http://gerrit.cloudera.org:8080/2721
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: Internal Jenkins
This patch:
1) Removes JniUtil::Cleanup() and JniUtil::global_refs_. We never
called Cleanup(), and all the jobjects in global_refs_ are meant to
have the lifetime of the impalad process. This makes
JniUtil::GetGlobalClassRef() and JniUtil::LocalToGlobalRef()
thread-safe (which fixes IMPALA-3379).
2) Introduces a new JniUtil::FreeGlobalRef() method, which is a
wrapper around the JNI DeleteGlobalRef() method.
3) Change JNI users to use the JniUtil methods instead of the JNI
methods directly where appropriate. This makes error checking more
consistent, and makes it easier to find all JNI uses. This is possible
since GetGlobalClassRef() and LocalToGlobalRef() are now thread-safe
and don't leak jobjects.
4) Removes HiveUdfCall::JniContext::cl, as well as other JNI
constants, and replaces them with process-wide static singletons. It
then moves the initialization to a new HiveUdfCall::Init() method
which is once called in the main thread at the beginning of the
process. This fixes IMPALA-3378.
5) Deletes the JniContext created for each HiveUdfCall
Unfortunately I am not able to repro IMPALA-3378 so there is no test
case (I didn't attempt IMPALA-3379 but it's similar).
Change-Id: I8cd089e355d2ee2d5ace81f05b214272c05cf941
Reviewed-on: http://gerrit.cloudera.org:8080/2820
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Internal Jenkins
Impala has a hardcoded limit of 1GB in size for StringVal.
If the length of the string exceeds 1GB, Impala will simply
mark the StringVal as NULL (i.e. is_null = true). It's important
that string functions or built-in UDFs check this field before
accessing the pointer or Impala may end up doing null pointer
access, leading to crashes.
Change-Id: I55777487fff15a521818e39b4f93a8a242770ec2
Reviewed-on: http://gerrit.cloudera.org:8080/2786
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
The regex didn't match cases where the 'set' statement had whitespace
between the preceding semicolon and the 'set'. E.g. if it is not the
first statement in the block and is preceded by a newline.
The resolution by name test implicitly relied on the bug, so it needed
to be updated.
Change-Id: Ic810b31c1ad7b2bcfd29413181bb81d1a0dbcb90
Reviewed-on: http://gerrit.cloudera.org:8080/2823
Reviewed-by: Michael Ho <kwho@cloudera.com>
Tested-by: Internal Jenkins
The sorter converts string pointers to block offsets when spilling.
There was a subtle bug in the logic that assumed if the offset was
past the end of the current block, the data must necessarily be in
the next block. This is not true for zero-length strings, because
there is no backing storage so the pointer can point to the byte
after the end of the block.
This patch fixes the bug by using a simpler offset encoding scheme
that packs the block number into the upper 32 bits and the offset
within the block into the lower 32 bits.
It also slightly refactors the functions so that the method signatures
and types are more consistent with the rest of the impala codebase.
Also fix a bug with handling of multiple query options in tests.
Change-Id: I5f64593e94d367d6b6efb61a8b86e35516f18839
Reviewed-on: http://gerrit.cloudera.org:8080/2780
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
This patch changes when runtime filters are produced in the partitioned
hash-join node to allow filters to be produced even when the PHJ
spills. Filters are now produced during the level0 processing of the
PHJ's build-side input in ProcessBuildBatch().
Since this function is codegen'ed, so now is filter production. We use
constant-propagation via constant argument injection to disable filter
production at no cost when it is not needed (including in level1+
repartitioning). I inspected the IR to confirm that the constant
propagation works as expected.
This change also allows us to send filters earlier during build-side
processing. A tradeoff is that filters are still built even if the
expected FP rate is too high, although any too-permissive filters are
still not sent to the scan (see 'Performance impact' below).
The restriction that prevented filters from being computed inside a
sub-plan is removed as part of this cleanup (since the FE handles
assigning filters correctly in subplans), and a test is added to confirm
that one of the correct cases for filters in subplans works.
This patch also fixes a bug where re-partitioning beyond level0 would
not use the codegen'ed implementation of ProcessBuildBatch().
A new test is added to test_runtime_row_filters, for Parquet only, which
spills and confirms that filtering still occurs.
Finally, the legacy --enable_phj_probe_side_filtering /
--enable_probe_side_filtering flags have been deprecated, as runtime
filtering can be permanently disabled via setting
RUNTIME_FILTER_MODE=OFF. The implementation that the old flags referred
to has been removed.
Performance impact
------------------
We benchmark the performance loss due to always computing runtime
filters even when the FP-rate will turn out to be too high as follows:
select STRAIGHT_JOIN count(*) from (select id from functional.alltypes
LIMIT 1) a JOIN [BROADCAST] (select * FROM p LIMIT 100000000) b on a.id
= -b.id and b.part_col > 0
('p' is a two-column Parquet table with 1B rows).
This builds a 100M row build table (benchmarks run on one node). When
filtering is enabled, the filter is built but selects all rows from the
probe side (so that there's no benefit to having the filter, to
emphasise the cost of building the filter in the first place).
RUNTIME_FILTER_MODE Avg. time (s) over 5 runs
OFF 18.95
GLOBAL 19.55
-------------------------------
Change +3%
Change-Id: I59a2d9ee03ccea6b674392584e4c7f272233571e
Reviewed-on: http://gerrit.cloudera.org:8080/2783
Tested-by: Internal Jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
Computing stats caused the Kudu table to be reloaded in the catalog and
the column definitions ended up getting appended to the existing ones.
There was already a method to clear the column state, so now that is
called during load().
Change-Id: I9ad42338750e9d8873a3584bc22a7cd7bd465c5d
Reviewed-on: http://gerrit.cloudera.org:8080/2813
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
This change fixes two problems:
1. The query options OPTIMIZE_PARTITION_KEY_SCANS and
DISABLE_STREAMING_PREAGGREGATIONS are both boolean
so they should accept 'true' and '1' as input values.
Previously, these two options are treated as int and
value such as 'true' doesn't work with them.
2. The break statement in the case statement of the option
SCAN_NODE_CODEGEN_THRESHOLD was 'stolen' by the option
DISABLE_STREAMING_PREAGGREGATIONS when it was added.
This change adds the missing break statement back for
SCAN_NODE_CODEGEN_THRESHOLD.
Change-Id: I5c74a1e5c49e3bda15a91b40740fc7310303207b
Reviewed-on: http://gerrit.cloudera.org:8080/2776
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
For a table with location "ABC", most partitions will have locations
like "ABC/DEF=2". The "ABC" part of the location does not need to be
stored in Catalog for each partition; we can compress it down to one
int in the common case.
This is done by stripping from each partition location the last N
directories (where N is the number of clustering columns) and storing
the resulting string in a cache of partition location prefixes. In the
cache, this location prefix string is mapped to an int. Partition
locations are then stored as a tuple consisting of that int and a
suffix string; the partition location can be reconstructed as the
concatenation of the prefix string (from the cache) and the suffix.
Though this scheme was designed in the expectation that most
partitions will be stored in directories like
"/part_col_1=1.23/part_col_2=234/", it works even when that is not the
case.
TODO: Since each partition stores the literal values for the
partitioning columns, we could also elide the column names and values
when partitions are placed in directories like
"/part_col_1=1.23/part_col_2=234/"
Change-Id: I8c67b6ce0f83de2f5277a528a9ce67e47d638adb
Reviewed-on: http://gerrit.cloudera.org:8080/2355
Reviewed-by: Jim Apple <jbapple@cloudera.com>
Tested-by: Internal Jenkins
There was an observed race between TestUdfs.test_java_udfs and
TestUdfs.test_libs_with_same_filenames, because they both used the
same database name.
This patch just changes the name of the database used by
test_libs_with_same_filenames.
Change-Id: Icc38cbe720a3b9d864935416eb10612171132e17
Reviewed-on: http://gerrit.cloudera.org:8080/2767
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Internal Jenkins
This patch introduces a new query option,
PARQUET_FALLBACK_SCHEMA_RESOLUTION which allows Parquet files' schemas
to be resolved by either name or position. It's "fallback" because
eventually field IDs will be the primary schema resolution scheme, and
we don't want to create an option that we will have to change the name
of later. The default is still by position. I chose to do a query
option because it will make testing easier and also be easier to
diagnose resolution problems quickly in the field. If users want to
switch the default behavior to be by name (like Hive), they can use
the --default_query_options flag.
This patch also introduces a new test section, SHELL, which can be
used to execute shell commands in a .test file. This is useful for
copying files into test tables.
Change-Id: Id0c715ea23792b2a6872610839a40532aabbb5a6
Reviewed-on: http://gerrit.cloudera.org:8080/2384
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Internal Jenkins
