Fixes a bug in generate_error_codes where the enum value specified
in generate_error_codes.py was not actually used.
Change-Id: If7e3269d12a839106c595d44da09c573a8a177f2
Reviewed-on: http://gerrit.cloudera.org:8080/1894
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
One of the error codes were removed leaveing a gap in the error code
numbers. This change just closes the gap.
Change-Id: I2e424e55439459d4c7a84dd393f55d72400dabf0
Reviewed-on: http://gerrit.cloudera.org:8080/1891
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Internal Jenkins
This patch removes the workaround that disallows SSL and Kerberos
being enabled at the same time. It was previously disallowed because
SSL and Kerberos wouldn't work together for server-server
(or daemon-daemon) communication by causing a hang.
The issue has been addressed in the following patches:
http://gerrit.cloudera.org:8080/#/c/1594/http://gerrit.cloudera.org:8080/#/c/1599/
This patch should be merged only after the above 2 are merged.
Change-Id: I63d492d1733204edd1249aff2cb3b168ec82ea92
Reviewed-on: http://gerrit.cloudera.org:8080/1772
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Internal Jenkins
FunctionContext::Allocate(), FunctionContextImpl::AllocateLocal()
and FunctionContext::Reallocate() allocate memory without taking
memory limits into account. The problem is that these functions
invoke FreePool::Allocate() which may call MemPool::Allocate()
that doesn't check against the memory limits. This patch fixes
the problem by making these FunctionContext functions check for
memory limits and set an error in the FunctionContext object if
memory limits are exceeded.
An alternative would be for these functions to call
MemPool::TryAllocate() instead and return NULL if memory limits
are exceeded. However, this may break some existing external
UDAs which don't check for allocation failures, leading to
unexpected crashes of Impala. Therefore, we stick with this
ad hoc approach until the UDF/UDA interfaces are updated in
the future releases.
Callers of these FunctionContext functions are also updated to
handle potential failed allocations instead of operating on
NULL pointers. The query status will be polled at various
locations and terminate the query.
This patch also fixes MemPool to handle the case in which malloc
may return NULL. It propagates the failure to the callers instead
of continuing to run with NULL pointers. In addition, errors during
aggregate functions' initialization are now properly propagated.
Change-Id: Icefda795cd685e5d0d8a518cbadd37f02ea5e733
Reviewed-on: http://gerrit.cloudera.org:8080/1445
Reviewed-by: Michael Ho <kwho@cloudera.com>
Tested-by: Internal Jenkins
connections
This patch does the following:
* Prevents Impala from starting if 'internal' Kerberos and SSL are
enabled at the same time.
* Changes the required configuration to enable 'internal' SSL to include
--ssl_client_ca_certificate. This allows 'external' SSL to be
configured without enabling 'internal' SSL.
Test are included for the first item. For the second, the appropriate
test is to try to connect with an internal SSL client to a non-SSL
server. However, this causes the connection to hang, which is not an
easy condition to detect in a test case.
Change-Id: I7fa545045fed57e161fb37898d5782937c710a0c
Reviewed-on: http://gerrit.cloudera.org:8080/1318
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
Reviewed-on: http://gerrit.cloudera.org:8080/1323
This makes cases like IMPALA-2402 much easier to diagnose, since this
message will mostly occur if Impala tries to read a file that's not an
Avro data file.
Change-Id: I6504e668905ecc6964b77a6fe0cfc9c7511fd5c0
Reviewed-on: http://gerrit.cloudera.org:8080/1202
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Internal Jenkins
Stream::ReadBytes() could fail by other reasons than
'stale metadata'. Adding Errorcode Check to make sure
Impala return proper error message.
It also fixes IMPALA-2488 metadata.test_stale_metadata
fails on non-hdfs filesystem.
Change-Id: I9a25df3fb49f721bf68d1b07f42a96ce170abbaa
Reviewed-on: http://gerrit.cloudera.org:8080/1166
Reviewed-by: Juan Yu <jyu@cloudera.com>
Tested-by: Internal Jenkins
This patch changes the Parquet scanner to check if it can't read the
full footer scan range, indicating that file has been overwritten by a
shorter file without refreshing the table metadata. Before it would
DCHECK. This patch adds a test for this case, as well as the case
where the new file is longer than the metadata states (which fails
with an existing error).
Change-Id: Ie2031ac2dc90e4f2573bd3ca8a3709db60424f07
Reviewed-on: http://gerrit.cloudera.org:8080/1084
Tested-by: Internal Jenkins
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
The code in resource-broker.cc that makes RPCs to Llama will
attempt to retry the RPC some number of times (which is
configurable) if the RPC returns a failure. If the RPC
throws (which thrift may do), we try to reset the connection
and then make the RPC again, but this time not guarded by a
try/catch block. If this RPC throws, the process will crash.
This fixes the issue by removing the try/catch and instead
using the ClientCache DoRpc function which handles this
already. Some additional Llama RPC calling wrappers were
removed as well.
Change-Id: Iba5add47a77fe9257e73eea5711ef4b948abe76a
Reviewed-on: http://gerrit.cloudera.org:8080/881
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
Tmp devices are blacklisted when a write error is encountered for that
device. No more scratch space will be allocated on the blacklisted
device, based on the assumption that the device is likely to be
misconfigured or failing.
This patch does not attempt to recover the query that experienced the
write error. It also does not attempt to remap any existing blocks away
from the temporary device.
This behaviour is unit tested for several failure scenarios.
This patch adds additional test infrastructure required for testing
BufferedBlockMgr behavior in the presence of faults and in
configurations with multiple tmp directories.
Adds metrics tmp-file-mgr.active-scratch-dirs and
tmp-file-mgr.active-scratch-dirs.list that track the number and set of
active scratch dirs and expose it in the Impala web UI.
Change-Id: I9d80ed3a7afad6ff8e5d739b6ea2bc0949f16746
Reviewed-on: http://gerrit.cloudera.org:8080/579
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
This patch modifies the Parquet scanner to resolve nested schemas, and
read and materialize collection types. The high-level modification is
to create a CollectionColumnReader that recursively materializes map-
and array-type slots.
This patch also adds many tests, most of which query a new table
called complextypestbl. This table contains hand-generated data that
is meant to expose edge cases in the scanner. The tests mostly test
the scanner, with a few tests of other functionality (e.g. array
serialization).
I ran a local benchmark comparing this scanner code to the original
scanner code on an expanded version of tpch_parquet.lineitem with
48009720 rows. My benchmark involved selecting different numbers of
columns with a single scanner thread, and I looked at the HDFS scan
node time in the query profiles. This code introduces a 10%-20%
regression in single-threaded scan time.
Change-Id: Id27fb728934e8346444f61752c9278d8010e5f3a
Reviewed-on: http://gerrit.cloudera.org:8080/576
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
This patch extends the deduplication of tuples in row batches to work on
non-adjacent tuples. This deduplication requires an additional data
structure (a hash table) and adds additional performance overhead (up to
3x serialization time), so it is only enabled for row batches with
compositions that are likely to blow up due to non-adjacent duplication
of large tuples. This avoids performance regression in typical cases,
while preventing size blow-ups in problematic cases, such as joining
three streams of tuples some of which contain may contain large
collections.
A test is included that ensures that adjacent deduplication is enabled.
The row batch serialize benchmark shows that deduplication does not regress
performance of serialization or deserialization.
Change-Id: I3c71ad567d1c972a0f417d19919c2b28891fb407
Reviewed-on: http://gerrit.cloudera.org:8080/573
Tested-by: Internal Jenkins
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
This patch allows administrators to configure all Impala daemons with a
password for the private key file used to negotiate connections with
clients which present the corresponding public key. This private key is
obtained by running a user-supplied shell command and using the result.
The command is supplied by setting --ssl_private_key_password_cmd. The
output of the command is truncated to a maximum of 1024 bytes (this is a
limitation of RunShellProcess(), but should not be significant for this
use case), and then all trailing whitespace is trimmed (this is to avoid
unexpected trailing newlines etc. from shell output).
If the password is incorrect clients will be unable to connect to the
server, whether or not they have the correct public key. If the command
exits with an error, the server will not start.
Change-Id: Icc13933fdf50a6170c859989626da5772fe5040d
Reviewed-on: http://gerrit.cloudera.org:8080/623
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
This patch updates AssembleRows() to have fewer exit and error paths,
as well as to explicitly distinguish between the row group being
finished and an error occurring. It functionally changes the behavior
in only two minor ways:
- The entire row group will be read regardless of how many values
the file metadata says there are. Previously it would only read up
to the number stated in the metadata, and then had extra logic for
checking if there were any values remaining.
- If abort_on_error is false and there is an error reading a row
group, subsequent row groups will still be read (except if
OOM). Before this would sometimes happen and sometimes not.
Change-Id: Id1836cfe2a507e46cb030be32b4c1553f478f639
Reviewed-on: http://gerrit.cloudera.org:8080/624
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Internal Jenkins
Most of this patch is rewriting the schema resolution logic to handle
recursive schemas. The other changes are for reading and codegening
recursive schemas.
Change-Id: I257db05e02ed99c62c8dcfd0136b9e8f392d5933
Reviewed-on: http://gerrit.cloudera.org:8080/86
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Internal Jenkins
This adds helper methods to map between Impala and Kudu
types to util/kudu-util.h.
Change-Id: Ib48d327034c5d40d67eab9e27e2fc381184536bb
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/6623
Reviewed-by: Martin Grund <mgrund@cloudera.com>
Tested-by: jenkins
The decoders aren't used yet, but will be when we materialize
arrays. In addition, setting up the repetition level decoder makes
sure the definition level encoder is initialized to the right place in
the data buffer.
Change-Id: Ic85ae812b10c747b36d884794d8dcf5976dfe74f
Reviewed-on: http://gerrit.cloudera.org:8080/405
Reviewed-by: Casey Ching <casey@cloudera.com>
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
This patch introduces the function GetConstant(), which is used by
expr compute function and UDFs to access query constants. There is a
corresponding GetIrConstant() function that returns the IR versions of
the same constants. Currently the only implemented constants are the
expr's return type and argument types, but other constants can be
easily be added to these functions. Interpreted expr functions run
normally, but cross-compiled functions can be passed to
InlineConstants(), which looks for calls to GetConstant() and replaces
them with the result of calling GetIrConstant().
I used this technique in the decimal functions that previously were
not switching on the type at all. The performance of LeastGreatest()
after this patch is the same as it was before it switched on the type.
Change-Id: I8b55744551830d894318a7bab6b6f045fb8bed41
Reviewed-on: http://gerrit.cloudera.org:8080/352
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Internal Jenkins
If a Heartbeat() RPC appears hung, the statestore should abort that RPC
so as not to hold on to a sender thread, and to trigger the failure
detector to evict the hung node.
We could just add a TCP timeout to the client cache used by the
statestore, but doing so would mean that all RPCs were subject to the
timeout, and UpdateState() typically takes *much* longer than
Heartbeat() by design, so setting a reasonable timeout would be
impossible. Instead, this patch adds a second client cache designed only
for Heartbeat() RPCs, with an aggressive timeout of 3s by
default. (Heartbeat() usually takes ~1-2ms). A timeout for UpdateState()
is also set to avoid thread starvation, but this is much less aggressive
at 300s.
This patch also adds ClientConnection::DoRpc(), which calls an RPC and
handles various failure modes, including timeout. If DoRpc() returns an
error, the statestore handles it in the usual way, including updating
the failure detector if the failed RPC is Heartbeat().
Change-Id: I2f2462278e59581937c9c10910625d2724a11efa
Reviewed-on: http://gerrit.cloudera.org:8080/206
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: Internal Jenkins
This patch introduces the concept of error codes for errors that are
recorded in Impala and are going to be presented to the client. These
error codes are used to aggregate and group incoming error / warning
messages to reduce the spill on the shell and increase the usefulness of
the messages. By splitting the message string from the implementation,
it becomes possible to edit the string independently of the code and
pave the way for internationalization.
Error messages are defined as a combination of an enum value and a
string. Both are defined in the Error.thrift file that is automatically
generated using the script in common/thrift/generate_error_codes.py. The
goal of the script is to have a central understandable repository of
error messages. Adding new messages to this file will require rebuilding
the thrift part. The proxy class ErrorMessage is responsible to
represent an error and capture the parameters that are used to format
the error message string.
When error messages are recorded they are recorded based on the
following algorithm:
- If an error message is of type GENERAL, do not aggregate this message
and simply add it to the total number of messages
- If an error messages is of specific type, record the first error
message as a sample and for all other occurrences increment the count.
- The coordinator will merge all error messages except the ones of type
GENERAL and display a count.
For example, in the case of the parquet file spanning multiple blocks
the output will look like:
Parquet files should not be split into multiple hdfs-blocks.
file=hdfs://localhost:20500/fid.parq (1 of 321 similar)
All messages are always logged to VLOG. In the coordinator error
messages are merged across all backends to retain readability in the
case of large clusters.
The current version of this patch adds these new error codes to some of
the most important error messages as a reference implementation.
Change-Id: I1f1811631836d2dd6048035ad33f7194fb71d6b8
Reviewed-on: http://gerrit.cloudera.org:8080/39
Reviewed-by: Martin Grund <mgrund@cloudera.com>
Tested-by: Internal Jenkins
