The HdfsTableSink usualy creates a HDFS connection to the filesystem
that the base table resides in. However, if we create a partition in
a FS different than that of the base table and set
S3_SKIP_INSERT_STAGING to "true", the table sink will try to write to
a different filesystem with the wrong filesystem connector.
This patch allows the table sink itself to work with different
filesystems by getting rid of a single FS connector and getting a
connector per partition.
This also reenables the multiple_filesystems test and modifies it to
use the unique_database fixture so that parallel runs on the same
bucket do not clash and end up in failures.
This patch also introduces a SECONDARY_FILESYSTEM environment variable
which will be set by the test to allow S3, Isilon and the localFS to
be used as the secondary filesystems.
All jobs with HDFS as the default filesystem need to set the
appropriate environment for S3 and Isilon, i.e. the following:
- export AWS_SECERT_ACCESS_KEY
- export AWS_ACCESS_KEY_ID
- export SECONDARY_FILESYSTEM (to whatever filesystem needs to be
tested)
TODO: SECONDARY_FILESYSTEM and FILESYSTEM_PREFIX and NAMENODE have a
lot of similarities. Need to clean them up in a following patch.
Change-Id: Ib13b610eb9efb68c83894786cea862d7eae43aa7
Reviewed-on: http://gerrit.cloudera.org:8080/3146
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Internal Jenkins
Some of our tests which are expected to fail due to low
query memory limits can fail non-deterministically with
different error messages. In addition, some tests may
throw different error messages when running with the legacy
join nodes. This change updates the test infrastructure to
allow multiple exception messages to be specified by using
adding "ANY_OF" to the "CATCH" subsection.
Change-Id: Ie6d81fd3ae601f565b575edfeefff7c5a6c07974
Reviewed-on: http://gerrit.cloudera.org:8080/3205
Reviewed-by: Michael Ho <kwho@cloudera.com>
Tested-by: Internal Jenkins
Maven has been downloading the postgres JDBC driver
all along. So, let's use the one in fe/target/dependency
instead of the one in thirdparty.
Change-Id: I76bce18fd308890e66615c8d08d5e58f02a8a132
Reviewed-on: http://gerrit.cloudera.org:8080/3232
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
During analysis the subquery is rewritten and during that process some
previous analysis state about the Kudu columns was lost and never
repopulated. If the Kudu table has keys of different data types that
could lead to an error about incorrect data types. It may also be
possible that if the data types did match, the wrong values would be
deleted.
Change-Id: I55b6fecfd35458fbb5bc20b4be1375484d7bc3c6
Reviewed-on: http://gerrit.cloudera.org:8080/2901
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
This test had a hardcoded HDFS path which caused a failure on all
non-HDFS runs. This change simply makes the path filesystem agnostic.
Change-Id: I31e5955893f2c0d967ce6f0cab653d1020ee45b3
Reviewed-on: http://gerrit.cloudera.org:8080/3174
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
PlanNode includes a 'capAtLimit()' method that can be used in
'computeStats()' on PlanNodes to ensure they do not estimate their
cardinality to be more than a pushed-down LIMIT clause.
This patch ensures that 'capAtLimit()' is used in all of the relevant
classes descending from PlanNode.
Change-Id: Ic06dcb93bbb2510c0d40151302bd817ef340b825
Reviewed-on: http://gerrit.cloudera.org:8080/3127
Reviewed-by: Jim Apple <jbapple@cloudera.com>
Tested-by: Internal Jenkins
Adds a query option 'strict_mode' which treats integer and
floating pt overflows as parse errors. In the past,
overflows were ignored and the max value was returned. When
this query option is set, overflowing values are treated as if
they were completely invalid data, i.e. NULL is returned.
When abort_on_error is enabled, this means the query is
aborted.
Notes:
* DECIMAL overflow/underflow is already treated as an error.
* The handling in text-converter treats underflows the same
as overflows, so they would result in the same behavior.
However, floating point parsing never returns an underflow
today.
* We may also want to handle numeric values that are truncated
when parsing to integer types, e.g. 10.5 -> 10.
Change-Id: I7409c31ec0cb6fe0b2d9842b9f58fe1670914836
Reviewed-on: http://gerrit.cloudera.org:8080/3150
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Internal Jenkins
This commit fixes an issue where an IllegalStateException is thrown
while generating runtime filters if a target expr of a join conjunct
is wrapped in a IF(TupleIsNull, NULL, e) expr. As this is not a valid
expr to be assigned to a scan node (target of a runtime filter), we
unwrap these exprs and replace exprs of the form IF(TupleIsNull, NULL, e)
with 'e' while producing the targer exprs for runtime filters. The original expr
of the join conjunct is not modified.
Change-Id: I2e3e207b4c8522283a1cd0d14be83d42eba58f5a
Reviewed-on: http://gerrit.cloudera.org:8080/3147
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: Internal Jenkins
Bug: Impalads crash if we query an Avro table with stale metadata
Cause: This happens because avroSchema_ is not set in HdfsTable,
which is not propagated to the avro scanner and it doesn't have
appropriate checks to make sure the schema is non-null.
The patch fixes the following.
1. Avro scanner should gracefully handle the case where the avro schema
is not set. Appropriate null checks and a meaning error message have
been added.
2. This is a special case with multi-fileformat partitioned tables.
avroSchema_ should be set in HdfsTable even if any subset of the
partitions are backed by avro. Without this patch, we only set it
if the base table file format is Avro.
Change-Id: I09262d3a7b85a2263c721f3beafd0cab2a1bdf4b
Reviewed-on: http://gerrit.cloudera.org:8080/3136
Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com>
Tested-by: Internal Jenkins
There are multiple places in the code which call
RuntimeState::SetMemLimitExceeded(). Most of them are
unnecessary as the error status constructed will eventually
be propagated up the tree of exec nodes. There is no obvious
reason to treat query memory limit exceeded differently.
In some cases such as scan-node, calling SetMemLimitExceeded()
is actually confusing as all scanner threads may pick up error
status when any thread exceeds query memory limit, causing a
lot of noise in the log.
This change replaces most calls to RuntimeState::SetMemLimitExceeded()
with MemTracker::MemLimitExceeded(). The remaining places are:
the old hash table code, the UDF framework and QueryMaintenance()
which checks for memory limit periodically. The query maintenance
case will be removed eventually once IMPALA-2399 is fixed.
Change-Id: Ic0ca128c768d1e73713866e8c513a1b75e6b4b59
Reviewed-on: http://gerrit.cloudera.org:8080/3140
Reviewed-by: Michael Ho <kwho@cloudera.com>
Tested-by: Internal Jenkins
- Moves the test into compute_stats.py
- Changes some test classes in compute_stats.py to inherit from
ImpalaTestSuite and not from TestComputeStats because that
will cause all tests in TestComputeStats to be run in the
subclasses again (redundantly).
- Clean up and add more coverage to testing incremental stats on
HBase which was probably broken in this commit 6b32ff06.
- Fixes a side effect that the original test had for testing
incremental stats on HBase. It computes stats on a functional
table which was not supposed to have stats.
Testing: Ran compute_stats.py on exhaustive locally in a loop 10 times.
Did a private hdfs/core run.
Change-Id: Iee8b84e30948c3c98166e08cae2666574777730c
Reviewed-on: http://gerrit.cloudera.org:8080/3074
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
Through emprical analysis, it was determined that setting the maximum
number of connections to S3 as 1500 was optimal for functionality and
performance. The hadoop set default of 15 connections could lead us to
have deadlocks as our parquet scanner requires that we have multiple
concurrent open connections proportional to the number of columns that
we are scanning.
Setting it to this high a value does not seem to have any negative
implications.
This has also been found to fix the Error(255): Unknown errors.
Change-Id: Ide6f1326d5155b2e5f4da3a3f23df3f3d40c5a8d
Reviewed-on: http://gerrit.cloudera.org:8080/3114
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Internal Jenkins
Sorter can have runaway memory consumption as it never frees
local allocations made in comparator_.Less(). In addition, it
doesn't check for errors generated during expression evaluation
so it may keep sorting even after failures have occurred.
This change fixes the problem by freeing local allocations for
every n invocations of comparator_.Less() where n is the row
batch size specified in the query options. Various error checks
are also added to return early if any error is encountered.
Change-Id: I941729b4836e5dbb827d4313a0b45bc5df2fa8e1
Reviewed-on: http://gerrit.cloudera.org:8080/3116
Reviewed-by: Michael Ho <kwho@cloudera.com>
Tested-by: Internal Jenkins
With this commit runtime filters can be assigned to multiple destination
nodes (scans). For each filter, the destination nodes are determined
using equivalent classes during planning. For each filter, all its
destination nodes are in the left subtree rooted at the join node
that constructs this filter. A runtime filter may have both
local and remote targets. The backend determines how to route each
filter depending on the number and type (local, remote) of its destination
nodes.
With this commit, we enable runtime filter propagation in all the
operands of UNION [ALL|DISTINCT] nodes.
Change-Id: Iad2ce4e579a30616c469312a4e658140d317507b
Reviewed-on: http://gerrit.cloudera.org:8080/2932
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: Internal Jenkins
In 2.5 we added the ability to set per-pool default query
options. A string of key-value pairs can be specified with a
pool configuration. However, if any options fail to parse,
then all the options are ignored. We want that behavior (and
returning an error) when parsing the process-wide default
query options on startup and when parsing the options sent
from a client (e.g. in beeswax server) because an error can
be returned immediately for the triggering action at that
time (i.e. starting the impalad or submitting a query with
the options set). This behavior is bad for the pool default
query options because (a) the configuration is set by the
administrator and there's nothing we can do until a query is
submitted and (b) one invalid option shouldn't mean that
other valid options aren't set.
Change-Id: If04733b775963091b0314c65286df126fd812358
Reviewed-on: http://gerrit.cloudera.org:8080/3056
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
This is needed because the workload runner required a prefix of query names to run.
Change-Id: Ica8db68141ef653b0b01a7cfa7773302717a35a2
Reviewed-on: http://gerrit.cloudera.org:8080/3021
Tested-by: Internal Jenkins
Reviewed-by: Thomas Tauber-Marshall <tmarshall@cloudera.com>
Before this patch, correlated exists and not exists subqueries were
rewritten as as left semi and anti joins respectively. Uncorrelated
exists subqueries were rewritten as cross joins, and uncorrelated
not-exists subqueries were not supported at all. This patch takes
advantage of the nested loop join that was recently introduced, which
allows us to rewrite both correlated and uncorrelated exists subqueries
as left semi joins and both correlated and uncorrelated not-exists
subqueries as anti joins.
Change-Id: I52ae12f116d026190f3a2a7575cda855317d11e8
Reviewed-on: http://gerrit.cloudera.org:8080/2792
Reviewed-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com>
Tested-by: Internal Jenkins
The problem: varlen data (e.g. strings) produced by aggregations is
freed by FreeLocalAllocations() after passing up the output
batch. This works for streaming operators or blocking operators that
copy their input, but results in memory corruption when the output
reaches non-copying blocking operators, e.g. SubplanNode and
NestedLoopJoinNode.
The fix: this patch makes the PartitionedAggregationNode copy out
produced string data if the node is in a subplan. Otherwise it calls
MarkNeedsToReturn() on the output batch. Marking the batch would work
in the subplan case as well, but would likely be less efficient since
it would result in many small batches coming out of the subplan.
The patch includes a test case. However, this test only exposes the
problem with an ASAN build and the --disable_mem_pools flag, which we
don't currently have automated testing for.
Change-Id: Iada891504c261ba54f4eb8c9d7e4e5223668d7b9
Reviewed-on: http://gerrit.cloudera.org:8080/2929
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
This patch adds two query options for runtime filters:
RUNTIME_FILTER_MAX_SIZE
RUNTIME_FILTER_MIN_SIZE
These options define the minimum and maximum filter sizes for a filter,
no matter what the estimates produced by the planner are. Filter sizes
are rounded up to the nearest power of two.
Change-Id: I5c13c200a0f1855f38a5da50ca34a737e741868b
Reviewed-on: http://gerrit.cloudera.org:8080/2966
Tested-by: Internal Jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
This patch implements a new feature to read the auth_to_local
configs from hdfs configuration files, using the parameter
hadoop.security.auth_to_local. This is done by modifying the
User#getShortName() method to use its hdfs equivalent.
This patch includes an end to end authorization test using
sentry where we add specific auth_to_local setting for a certain
user and test if the sentry authorization passes for this user
after applying these rules. Given we don't have tests that run
on a kerberized min-cluster, this patch adds a hack to load this
configuration during even on non-kerberized 'test runs'.
However this feature is disabled by default to preserve the
existing behavior. To enable it,
1. Use kerberos as authentication mechanism (by setting --principal) and
2. Add "--load_auth_to_local_rules=true" to the cluster startup args
Change-Id: I76485b83c14ba26f6fce66e5f83e8014667829e0
Reviewed-on: http://gerrit.cloudera.org:8080/2800
Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com>
Tested-by: Internal Jenkins
INSERTs on S3 are slower because of double buffering where we buffer
once locally and once in a staging directoy in S3 before moving the
file(s) to the final location. Also, moving the file from the staging
directory to the final location in HDFS is a quick rename which is
only a metadata operation. However, on S3, renames are not supported,
thus becoming a full file copy instead of just a metadata rename
operation.
This patch instroduces a boolean query option "s3_skip_insert_staging"
which avoids the staging step on S3 and allows the sinks to write to
the final location directly.
This trades in consistency for the sake of performance. If a node(s)
fails during the query, then we will end up with inconsistent results
in the final location.
P.S: This option is disabled for INSERT OVERWRITE queries as that
would require cleaning the destination directory before moving the
final files there. However, the coordinator is responsible for the
cleaning which takes place only after the table sinks have moved
the files to the final location. Thus, INSERT OVERWRITE queries must
still have their files moved to a staging location by the table sinks.
Performance gains:
- For non-partitioned tables, the INSERT queries run 4-4.5x faster on
S3. (Tested on a 63GB INSERT to a table)
- For heavily partitioned tables, there is considerable improvement
in the order of 4-5 minutes on queries that take ~27 minutes but
queries are still slow because of IMPALA-3482 where the catalog
takes too long to update all the metadata. (Tested with a query
that creates 2.4K partitions in a table totalling ~19GB).
Change-Id: Iff9620d41ba0d5fb1aa0c9f4abb48866fc2b0698
Reviewed-on: http://gerrit.cloudera.org:8080/2905
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Internal Jenkins
Testing: Tested the changes locally by running them in a loop
10 times. Also did a private core/hdfs run.
Change-Id: I37e1528c02e598f3fb2d673b6559d55a34bf79b4
Reviewed-on: http://gerrit.cloudera.org:8080/3002
Reviewed-by: Michael Brown <mikeb@cloudera.com>
Tested-by: Internal Jenkins
This commit fixes an issue where a GRANT ALL ON SERVER to role_name statement
followed by a REVOKE ALL ON SERVER from role_name statement would not revoke all
privileges from role_name. The problem was triggered by a specific
combination of Sentry client API calls used in Impala during
grant/revoke statements at server scope. In particular, during GRANT, Impala was using
an API call that didn't explicitly specify the privilege action (Sentry uses '*' if
no action is specified). In contrast, the corresponding REVOKE call was explicitly
specifying the privilege action to be 'ALL'. Sentry doesn't seem to
handle this case correctly, thereby failing to remove all the privileges
after a REVOKE ALL ON SERVER call. The fix from the Impala side, that
results in the correct behavior, is to always specify the privilege
action by using the appropriate API calls.
Change-Id: I6b3a0d10f5e88c6a0a10bd20f620562d2de7ab25
Reviewed-on: http://gerrit.cloudera.org:8080/2979
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: Internal Jenkins
New classes:
- ParallelPlanner: creates build plans, assigns plans to cohorts
- JoinBuildSink: DataSink for plan fragments that materialize build sides
- ids for plans, hash tables, plan fragments
Tests: this adds a new test file section PARALLELPLANS and augments the tpc-h/-ds tests with
those sections.
In the interest of keeping this patch small I didn't augment other test files with that
section yet (which will happen at a later date, to cover more corner cases).
Change-Id: Ic3c34dd3f9190a131e6f03d901b4bfcd164a5174
Reviewed-on: http://gerrit.cloudera.org:8080/2846
Tested-by: Internal Jenkins
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
/tmp isn't necessarily on the same filesystem as the Kudu data
directory. Fix the check so that it checks the actual Kudu directory.
Change-Id: Ic6aa27569a0650db7dcf5759952cd50c8e47f8c9
Reviewed-on: http://gerrit.cloudera.org:8080/2967
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
Add missing coverage for sorting by CHAR and VARCHAR.
Add more coverage for spilling sorts.
Fix spilling tests: ensure that they actually reliably spill (many of
them had memory limits high enough that they could run entirely in
memory).
I ran this in a loop for a while to flush out flaky tests. The tests
should be fairly predictable given that they're not run concurrently
with other tests and we allocate enough block manager memory so that
each operator can obtain its reservation.
Change-Id: Ia2d2627a2c327dcdf269ea3216385b1af9dfa305
Reviewed-on: http://gerrit.cloudera.org:8080/2877
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
Now that we functionally support writes to S3 via Impala,
test_grant_revoke should not have a special case for S3, which
till this patch did the test without INSERTs.
Change-Id: Id981e7f83bf86b32d1a5b267ad3781db02337e86
Reviewed-on: http://gerrit.cloudera.org:8080/2949
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Internal Jenkins
Added costs to all Exprs, which estimate the relative cost of evaluating
an expression and all of its children. Costs are calculated during
analysis. For now, these costs are intended as a simple way to order
expressions from cheap to expensive, not necessarily to be a precise
reflection of running times.
In general, expressions that deal with variable length types like strings
will have higher cost than those dealing with fixed length types
like numbers and booleans. Additionally, expressions with complicated
subexpressions will have higher cost than simpler expressions.
Also added PlanNode.orderConjunctsByCost, which takes a list of Exprs and
returns a new list sorted according to an estimate of the cheapest order to
evaulate the conjuncts in, based on their cost and selectivity.
The conjuncts are sorted by repeatedly iterating over them and choosing the
conjunct that would result in the least total estimated work were it to be
applied before the remaining conjuncts. Selectivities are exponentially
backed off, and Exprs without selectivity estimates are given a reasonable
default.
Change-Id: I02279a26fbc6308ac5eb819d78345fc010469034
Reviewed-on: http://gerrit.cloudera.org:8080/2598
Reviewed-by: Thomas Tauber-Marshall <tmarshall@cloudera.com>
Tested-by: Internal Jenkins
This patch makes it a little easier to use the unique_database fixture
with .test files. The RESULTS section can now contain $DATABASE which
is replaced with the current database by the test framework.
Testing:
- ran the test locally on exhaustive
- ran the test on hdfs and the local filesystem on Jenkins
Change-Id: I8655eb769003f88c0e1ec1b254118e4ec3353b48
Reviewed-on: http://gerrit.cloudera.org:8080/2947
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
Previously Impala disallowed LOAD DATA and INSERT on S3. This patch
functionally enables LOAD DATA and INSERT on S3 without making major
changes for the sake of improving performance over S3. This patch also
enables both INSERT and LOAD DATA between file systems.
S3 does not support the rename operation, so the staged files in S3
are copied instead of renamed, which contributes to the slow
performance on S3.
The FinalizeSuccessfulInsert() function now does not make any
underlying assumptions of the filesystem it is on and works across
all supported filesystems. This is done by adding a full URI field to
the base directory for a partition in the TInsertPartitionStatus.
Also, the HdfsOp class now does not assume a single filesystem and
gets connections to the filesystems based on the URI of the file it
is operating on.
Added a python S3 client called 'boto3' to access S3 from the python
tests. A new class called S3Client is introduced which creates
wrappers around the boto3 functions and have the same function
signatures as PyWebHdfsClient by deriving from a base abstract class
BaseFileSystem so that they can be interchangeably through a
'generic_client'. test_load.py is refactored to use this generic
client. The ImpalaTestSuite setup creates a client according to the
TARGET_FILESYSTEM environment variable and assigns it to the
'generic_client'.
P.S: Currently, the test_load.py runs 4x slower on S3 than on
HDFS. Performance needs to be improved in future patches. INSERT
performance is slower than on HDFS too. This is mainly because of an
extra copy that happens between staging and the final location of a
file. However, larger INSERTs come closer to HDFS permformance than
smaller inserts.
ACLs are not taken care of for S3 in this patch. It is something
that still needs to be discussed before implementing.
Change-Id: I94e15ad67752dce21c9b7c1dced6e114905a942d
Reviewed-on: http://gerrit.cloudera.org:8080/2574
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Internal Jenkins
This change is a first step towards a more efficient Parquet scanner.
The focus is on presenting the new code flow that materializes
the table-level slots in a column-wise fashion, without going deep
into actually improving scan efficieny.
After these changes there are several obvious places that should
be optimized to realize efficiency gains.
Summary of changes
- the table-level tuples are materialized in a column-wise fashion
with new ColumnReader::ReadValueBatch() functions
- this is done by materializing a 'scratch' batch, and transferring
scratch tuples that survive filters/conjuncts to the output batch
- the tuples of nested collections are still materialized in
a row-wise fashion using the ColumnReader::ReadValue() function,
just as before
Mini benchmark
I ran the following queries on a single impalad before and after my
change using a synthetic 'huge_lineitem' table.
I modified hdfs-scan-node.cc to set the number of rows of any row
batch to 0 to focus the measurement on the scan time.
Query options:
set num_scanner_threads=1;
set disable_codegen=true;
set num_nodes=1;
select * from huge_lineitem;
Before: 22.39s
Afer: 18.50s
select * from huge_lineitem where l_linenumber < 0;
Before: 25.11s
After: 20.56s
select * from huge_lineitem where l_linenumber % 2 = 0;
Before: 26.32s
After: 21.82s
Change-Id: I72a613fa805c542e39df20588fb25c57b5f139aa
Reviewed-on: http://gerrit.cloudera.org:8080/2779
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
For IMPALA-1740 we added a test to insert.test, which creates a table and
inserts data. The table was created on HDFS by default and thus inserts with
compression enabled did not work. This change adds the required table to the
functional schema in the same way we do it for the other insert tests.
Change-Id: Ie68e7067b7a16218d27935820d5d1ce7035d2e6c
Reviewed-on: http://gerrit.cloudera.org:8080/2919
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
Instead of having a default Bloom Filter size for all runtime filters,
adjust filter size according to desired FP-rate and expected NDV from
join's build-side. Size of filter is still clipped to 4k < N < 16MB range.
If NDV estimate from planner is -1 (i.e. no stats) the default filter
size is used.
The NDV of all filters produced by the same join is currently the same
because the NDV is estimated from the cardinality of the input. In the
future, the NDV should be estimated for each filter source expr. The BE
changes anticipate this and can enable or disable individual filters if
they have differing FP rates.
Change-Id: I1fe37b8d4cfb3c52bb8e8cf0ca55e92665b87803
Reviewed-on: http://gerrit.cloudera.org:8080/2812
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: Internal Jenkins
HIVE-5795 introduced a parameter skip.header.line.count to skip header
lines from input files. This change introduces the capability to skip
an arbitrary number of header lines from csv input files on hdfs. The
size of the total file header must be smaller than
max_scan_range_length, otherwise an error will be reported. This is
necessary because scan ranges are not read in disk order, so there is
no way of identifying header lines except by counting from the start
of the first scan range.
[localhost:21000] > alter table t1 set
tblproperties('skip.header.line.count'='1');
Query: alter table t1 set tblproperties('skip.header.line.count'='1')
[localhost:21000] > select * from t1;
Query: select * from t1
+----+----+
| c1 | c2 |
+----+----+
| 1 | 1 |
| 2 | 2 |
| 3 | 3 |
+----+----+
Fetched 3 row(s) in 0.32s
[localhost:21000] > alter table t1 set
tblproperties('skip.header.line.count'='0');
Query: alter table t1 set tblproperties('skip.header.line.count'='0')
[localhost:21000] > select * from t1;
Query: select * from t1
+------+------+
| c1 | c2 |
+------+------+
| NULL | NULL |
| 1 | 1 |
| 2 | 2 |
| 3 | 3 |
+------+------+
WARNINGS: Error converting column: 0 TO INT (Data is: num1)
Error converting column: 1 TO DOUBLE (Data is: num2)
file: hdfs://localhost:20500/test-warehouse/t1/test.txt
record: num1,num2
Fetched 4 row(s) in 0.41s
Change-Id: I595f01a165d41499ca1956fe748ba3840a6eb543
Reviewed-on: http://gerrit.cloudera.org:8080/2110
Reviewed-by: Lars Volker <lv@cloudera.com>
Tested-by: Internal Jenkins
Attempting to codegen a sort where the sort expr has a CHAR type as an
intermediate result fails completely.
The problem is that ScalarFnCall checked whether its input arguments
were CHAR to disable codegen, but didn't check its output.
This patch also replaces some incorrect codegen CHAR logic that should
not be executed with DCHECKs.
Testing:
The test is a minimal reproduction of the issue. The test is executed
both by the sorter and top-n nodes so covers both cases.
Change-Id: I189073d46a10988803d572928a38f4a718690fa3
Reviewed-on: http://gerrit.cloudera.org:8080/2876
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
Filter locality was not correctly set when NUM_NODES=1 (and therefore no
distributed plan was created). The default locality should be 'local'.
This patch also fixes a bug where the initial filter routing table
wasn't printed when NUM_NODES=1.
Change-Id: I7b9a6bcc64ca6ec5fd51d63815cea25de866ef93
Reviewed-on: http://gerrit.cloudera.org:8080/2721
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: Internal Jenkins
This patch:
1) Removes JniUtil::Cleanup() and JniUtil::global_refs_. We never
called Cleanup(), and all the jobjects in global_refs_ are meant to
have the lifetime of the impalad process. This makes
JniUtil::GetGlobalClassRef() and JniUtil::LocalToGlobalRef()
thread-safe (which fixes IMPALA-3379).
2) Introduces a new JniUtil::FreeGlobalRef() method, which is a
wrapper around the JNI DeleteGlobalRef() method.
3) Change JNI users to use the JniUtil methods instead of the JNI
methods directly where appropriate. This makes error checking more
consistent, and makes it easier to find all JNI uses. This is possible
since GetGlobalClassRef() and LocalToGlobalRef() are now thread-safe
and don't leak jobjects.
4) Removes HiveUdfCall::JniContext::cl, as well as other JNI
constants, and replaces them with process-wide static singletons. It
then moves the initialization to a new HiveUdfCall::Init() method
which is once called in the main thread at the beginning of the
process. This fixes IMPALA-3378.
5) Deletes the JniContext created for each HiveUdfCall
Unfortunately I am not able to repro IMPALA-3378 so there is no test
case (I didn't attempt IMPALA-3379 but it's similar).
Change-Id: I8cd089e355d2ee2d5ace81f05b214272c05cf941
Reviewed-on: http://gerrit.cloudera.org:8080/2820
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Internal Jenkins
This change whitelists the supported filesystems which can be set
as Default FS for Impala to run on.
This patch configures Impala to use S3 as the default filesystem, rather
than a secondary filesystem as before.
Change-Id: I2f45bef6c94ece634045acb906d12591587ccfed
Reviewed-on: http://gerrit.cloudera.org:8080/1121
Reviewed-by: anujphadke <aphadke@cloudera.com>
Tested-by: Internal Jenkins
Impala has a hardcoded limit of 1GB in size for StringVal.
If the length of the string exceeds 1GB, Impala will simply
mark the StringVal as NULL (i.e. is_null = true). It's important
that string functions or built-in UDFs check this field before
accessing the pointer or Impala may end up doing null pointer
access, leading to crashes.
Change-Id: I55777487fff15a521818e39b4f93a8a242770ec2
Reviewed-on: http://gerrit.cloudera.org:8080/2786
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
The regex didn't match cases where the 'set' statement had whitespace
between the preceding semicolon and the 'set'. E.g. if it is not the
first statement in the block and is preceded by a newline.
The resolution by name test implicitly relied on the bug, so it needed
to be updated.
Change-Id: Ic810b31c1ad7b2bcfd29413181bb81d1a0dbcb90
Reviewed-on: http://gerrit.cloudera.org:8080/2823
Reviewed-by: Michael Ho <kwho@cloudera.com>
Tested-by: Internal Jenkins
Changes:
1) Previously when a service would fail, the user would have to find the
the log file and open it. Now the end of the log is dumped to stdout.
2) Add start, stop, and restart commands to the "admin" script. For
example now you can run
testdata/cluster/admin restart kudu
3) Wait up to 120 seconds for services to shutdown. The timeout is the
same as for the Impala processes. If the services fail to stop an
error will be raised.
Change-Id: I537ea5656df2081d4f1f27a9f3fcef4547fdc2fe
Reviewed-on: http://gerrit.cloudera.org:8080/2751
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
If a stale snapshot is detected, the full data load proceeds even
if the option to skip data load was set. A check is added to fail
immediately if this happens for isilon or s3 because the full data
load will not work on these filesystems currently.
Change-Id: I98faaa4a66e5715bd86289a56d199599b9011f52
Reviewed-on: http://gerrit.cloudera.org:8080/2811
Reviewed-by: Harrison Sheinblatt <hs7@hotmail.com>
Tested-by: Internal Jenkins
The sorter converts string pointers to block offsets when spilling.
There was a subtle bug in the logic that assumed if the offset was
past the end of the current block, the data must necessarily be in
the next block. This is not true for zero-length strings, because
there is no backing storage so the pointer can point to the byte
after the end of the block.
This patch fixes the bug by using a simpler offset encoding scheme
that packs the block number into the upper 32 bits and the offset
within the block into the lower 32 bits.
It also slightly refactors the functions so that the method signatures
and types are more consistent with the rest of the impala codebase.
Also fix a bug with handling of multiple query options in tests.
Change-Id: I5f64593e94d367d6b6efb61a8b86e35516f18839
Reviewed-on: http://gerrit.cloudera.org:8080/2780
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
By default Kudu requires the underlying file system to support hole
punching. If support isn't there Kudu will fail to start. People using
such a file system can instead start Kudu with -block_manager=file.
Before starting Kudu in the local mini-cluster, the "fallocate"
command will be used to automatically determine if the special flag is
needed.
Note, users who need this must run bin/create-test-configuration.sh
after pulling in this commit.
This also fixes a bug in the delete_kudu_data() in the cluster admin
script. A directory name was incorrect.
Change-Id: I1ca7fedb367444c41e462b72b0b76091ee94e27c
Reviewed-on: http://gerrit.cloudera.org:8080/2750
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
Before this fix our "Waiting for something to happen" print
output would be buffered and dumped all at once when the
event we were waiting for succeeded or we hit a timeout.
After this fix the output of "print" is displayed on
the console imemdiately, as was originally intended.
Change-Id: Icf341e81d0d459504918ae7c9e88918fe5e16c59
Reviewed-on: http://gerrit.cloudera.org:8080/2810
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
The directory structure of the newer Kudu toolchain artifacts has
changed. Now the root directory is split into /release and /debug. A few
little updates are needed to the build and service scripts.
Since the toolchain no longer provides stubs for platforms that Kudu
doesn't support the stubs need to be generated. This will be done as
part of the toolchain bootstrapping.
Also this upgrades Kudu to 0.8 RC1.
Developers will need to run bin/create-test-configuration.sh after
pulling in this change. Otherwise the Kudu service will fail to start.
Change-Id: I625903bd92afece0ad819a96fc275d5812b5eb2a
Reviewed-on: http://gerrit.cloudera.org:8080/2720
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
This patch changes when runtime filters are produced in the partitioned
hash-join node to allow filters to be produced even when the PHJ
spills. Filters are now produced during the level0 processing of the
PHJ's build-side input in ProcessBuildBatch().
Since this function is codegen'ed, so now is filter production. We use
constant-propagation via constant argument injection to disable filter
production at no cost when it is not needed (including in level1+
repartitioning). I inspected the IR to confirm that the constant
propagation works as expected.
This change also allows us to send filters earlier during build-side
processing. A tradeoff is that filters are still built even if the
expected FP rate is too high, although any too-permissive filters are
still not sent to the scan (see 'Performance impact' below).
The restriction that prevented filters from being computed inside a
sub-plan is removed as part of this cleanup (since the FE handles
assigning filters correctly in subplans), and a test is added to confirm
that one of the correct cases for filters in subplans works.
This patch also fixes a bug where re-partitioning beyond level0 would
not use the codegen'ed implementation of ProcessBuildBatch().
A new test is added to test_runtime_row_filters, for Parquet only, which
spills and confirms that filtering still occurs.
Finally, the legacy --enable_phj_probe_side_filtering /
--enable_probe_side_filtering flags have been deprecated, as runtime
filtering can be permanently disabled via setting
RUNTIME_FILTER_MODE=OFF. The implementation that the old flags referred
to has been removed.
Performance impact
------------------
We benchmark the performance loss due to always computing runtime
filters even when the FP-rate will turn out to be too high as follows:
select STRAIGHT_JOIN count(*) from (select id from functional.alltypes
LIMIT 1) a JOIN [BROADCAST] (select * FROM p LIMIT 100000000) b on a.id
= -b.id and b.part_col > 0
('p' is a two-column Parquet table with 1B rows).
This builds a 100M row build table (benchmarks run on one node). When
filtering is enabled, the filter is built but selects all rows from the
probe side (so that there's no benefit to having the filter, to
emphasise the cost of building the filter in the first place).
RUNTIME_FILTER_MODE Avg. time (s) over 5 runs
OFF 18.95
GLOBAL 19.55
-------------------------------
Change +3%
Change-Id: I59a2d9ee03ccea6b674392584e4c7f272233571e
Reviewed-on: http://gerrit.cloudera.org:8080/2783
Tested-by: Internal Jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
Computing stats caused the Kudu table to be reloaded in the catalog and
the column definitions ended up getting appended to the existing ones.
There was already a method to clear the column state, so now that is
called during load().
Change-Id: I9ad42338750e9d8873a3584bc22a7cd7bd465c5d
Reviewed-on: http://gerrit.cloudera.org:8080/2813
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
I. Start HBase per directions
1. https://hbase.apache.org/book.html#_configuration_files mentions a
'regionservers' file that points to a list of hosts on which to start
HBase RegionServers. When HBase starts in our mini-cluster there are
messages printed like this:
cat: /home/mikeb/Impala/fe/src/test/resources/regionservers: No such file or directory
The presence of this file now starts a single RegionServer and takes the
place of RegionServer 1 in the "additional region servers" startup, a
separate call.
2. The additional RegionServers are started but now we only start 2 from
index 2. See https://hbase.apache.org/book.html#quickstart_pseudo
There are still 3 total RegionServers using the same ports as before. We
are simply configuring our settings as directed in the documentation.
There were mentions in testdata/bin/run-hbase.sh of a "hbase race". One
possible such bug is https://issues.apache.org/jira/browse/HBASE-5780
which has been fixed for a while. I've removed the check to wait for
that Master, though I have not removed the Python script that does the
waiting. We could remove that later after we let this patch bake.
Also, https://issues.apache.org/jira/browse/HBASE-4467 has been marked
"not a problem", so I've removed references to that.
II. Implement HBase start retry
If starting either HBase Master or additional RegionServers fails, kill
all of HBase and try again. Do this for some number of attempts.
In order to keep errexit ("set -e") happy, we expect the possibility of
some of the startup attempts failing. We use control flow in those
cases. In the last case, errexit can fail on our behalf.
There is some code duplication here, but because Bash can't give us a
stack trace on failure, and only a line number, I chose not to use
functions to handle reuse. We don't really have functions anywhere else
at the moment, either.
Testing:
It's pretty difficult to try to trigger a real "HBase fails to start"
situation. I tested my changes by faking HBase failures, both when
starting up the Master and first RegionServer, and also starting
subsequent RegionServers.
Multiple private builds have passed.
Change-Id: Ib1d055a8a9098ce24e2f31b969501b6e090eab19
Reviewed-on: http://gerrit.cloudera.org:8080/2804
Reviewed-by: Michael Brown <mikeb@cloudera.com>
Tested-by: Internal Jenkins
