The issue: Computing the full transitive closure for all slots can be very
expensive (10s of seconds for >2k slots, minutes for >4k slots).
Queries with many views and/or unions were affected most because each
union/view adds a new tuple with slots, increasing the total number of slots.
The fix: The new algorithm exploits the sparse structure of the value transfer
graph for a significant speedup (>100x). The high-level steps are:
1. Identify complete subgraps based on bi-directional value transfers, and
coalesce the slots of each complete subgraph into a single slot.
2. Map the remaining uni-directional value transfers into the new slot domain.
3. Identify the connected components of the uni-directional value transfers.
This step partitions the value transfers into disjoint sets.
4. Compute the transitive closure of each partition from (3) in the new slot
domain separately. Hopefully, the partitions are small enough to afford
the O(N^3) complexity of the brute-force transitive closure computation.
Change-Id: I35b57295d8f04b92f00ac48c04d1ef1be4daf41b
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2360
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Kerberos, by default, uses a location for its credentials cache (where
tickets are saved to avoid having to re-authenticate on each connection)
that is unique only to the user. This means that if several programs
running as the same user try and authenticate concurrently, they may
race to access the cache, and an error might occur.
This patch causes Impala's demons to use a path that is unique to
process name and process id (e.g. /tmp/krb5_impalad_4324). The new flag
--krb_credential_cache can be used to set the location explicitly.
Any Kerberos connections initiated from Java will still use the system
default cache. The catalog service is the only such service. It's
important that the cache locations for Java and C++ are different (to
avoid the same bug being exposed in a single process, which is actually
its most likely cause). The Java location can be changed by setting the
KRB5CCNAME environment variable.
Change-Id: I4d975a3a7cf5ff6ee4c107833f5292115cd33b03
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2308
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins
(cherry picked from commit 36327bd74c76dc69207d148cc401190fe5a9a50c)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2351
This change adds the ability to refresh a local cache entry, causing
the old cache entry to be dropped and the library to be reloaded from
HDFS. This is used in ResolveSymbolLookup(), which is called by the
frontend when creating a new a function, and in ImpalaServer when
receiving a "create function" heartbeat. This change also makes sure
the FE calls into the backend for jars, so jars get refreshed as well.
Change-Id: I5fd61c1bc2e04838449335d5a68b61af8b101b01
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2286
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
(cherry picked from commit e8587794b3b82438190c91b2ebe9d1e12db73981)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2348
The hdfs-site template in CDH5 is different from the one we fine in CDH5. Specifically:
- It has entries that enable hdfs caching.
- It uses the correct parameter name for hdfs block locations timeout.
Change-Id: I0ca6bd84b074ccbb8f42243d37c5082b305f9bcf
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2338
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: jenkins
This patch simplifies the complex slot materialization logic for unions by
making the materialization independent of conjuncts assigned to MergeNodes.
When 'pushing down' predicates into union operands, we drop union operands
with constant predicates evaluating to false. Constant predicates that
evaluate to true are simply ignored.
Change-Id: I0e7ccfb206bed29db2b5d667e2bb61310980e80a
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2327
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
This should allow individual service components, such as a single nodemanager,
to be shutdown for failure testing. The mini-cluster bundled with hadoop is a
single process that does not expose the ability to control individual roles.
Now each role can be controlled and configured independently of the others.
Change-Id: Ic1d42e024226c6867e79916464d184fce886d783
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1432
Tested-by: Casey Ching <casey@cloudera.com>
Reviewed-by: Casey Ching <casey@cloudera.com>
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2297
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: Ishaan Joshi <ishaan@cloudera.com>
Adds the thrift structures for the public external data source API
and a new maven project containing the Java ExternalDataSource
interface and the generated Java thrift classes.
The ExternalDataSource.thrift structures can evolve in a backward
compatible way. The ExternalDataSource Java interface will always
contain a version number in the namespace (e.g.
com.cloudera.impala.extdatasource.v1 for V1) so we can potentially
make breaking changes to the interface in the future but still
support older versions.
A trivial implementation of the ExternalDataSource API is also
added for testing purposes.
TODO: Make the sample data source implementation realistic.
Change-Id: I827d6420a87ed7a2bce34e050362ca98ddc5dbcc
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2241
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
(cherry picked from commit f29814e9ede9d4c889f2648606fcf511feeb47ae)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2313
We should be using the short name of a Kerberos principal (e.g.
user/fully.qualified.domain@realm.com) or LDAP username (e.g. user@domain)
when checking group membership in RequestPoolService. Right now we call
UserGroupInformation.createRemoteUser() with the full user name and it
will throw an exception.
Change-Id: I39d849627cb49760807504d66109c05b7a399482
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2288
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
(cherry picked from commit 0005da9cb71f5a4a4ed6bb1dfcd74f8526cd8316)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2305
Some tests have constraints that were there only to help reduce runtime which
reduces coverage when running in exhaustive mode. The majority of the constraints
are because it adds no value to run the test across additional dimensions (or
it is invalid to run with those dimensions). Updates the tests that have
legitimate constraints to use two new helper methods for constraining the table format
dimension:
create_uncompressed_text_dimension()
create_parquet_dimension()
These will create a dimension that will produce a single test vector, either
uncompressed text or parquet respectively.
Change-Id: Id85387c1efd5d192f8059ef89934933389bfe247
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2149
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
(cherry picked from commit e02acbd469bc48c684b2089405b4a20552802481)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2290
The reported issue is that we can have redundant hash expressions in exchanges.
The underlying cause is that we fail to remove redundant join predicates.
This patch enforces slot equivalences based on our computed equivalence classes
at the lowest possible plan node by generating new equality predicates.
Each plan subtree now has a minimal set of equality predicates that express
all known equivalences between slots belonging to tuples materialized at that
plan node.
As a result, eliminating redundant join predicates becomes trivial: It is
sufficient to pick a single representative predicate of each relevant equivalence
class. All predicates beyond that are redundant.
Change-Id: I7998fe8d7bdf84cc8eb129d32c86269bedeab68e
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2177
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2278
This re-enables a subset of the stable data errors tests and updates them to
work in our test framework. This includes support for updating results via --update_results.
This also lets us remove a lot of old code that was there only to support these disabled
tests.
Change-Id: I4c40c3976d00dfc710d59f3f96c99c1ed33e7e9b
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1952
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2277
- A few places didn't have total timer at the beginning.
- Async build thread for blocking join nodes really messed things up (sum of
children was more than the time in the join node).
Change-Id: I9176ce37cf22f2bcebea21b117e45cce066dbc1d
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2276
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Previously, to produce the correct output expressions for the root plan
fragment before a table sink, InsertStmt would reorder the result
expressions for the query statement at the plan root. This had stopped
working for SelectStmts (and test coverage didn't catch that).
Now InsertStmt produces its own output expressions that can substitute
for the originals from the query statement, and the planner uses those
instead.
All query tests for column reordering have been duplicated to use SELECT
expressions.
Change-Id: Ib909fe35d27416b33ba2e5ac797aa931e1fe43f9
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2204
Tested-by: jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
(cherry picked from commit d526db7ac6274f35b6affcb7428327100026e14e)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2275
The c++ standard int128_t is exactly what we want. It is 16 bytes, stored as 2's
complement little endian (the exact extension of the native int types). It out
performs the boost library we were using (see benchmark) and looking at the assembly
for some of the operators, I doubt we can do better. This also seems like the kind
of thing hardware might be able to do natively in the future if we stuck with the
standard implementation.
This requires minimal changes to the rest of our code so the multi int library is
abstracted away.
The standard only added int128 and not 96 or any others. We still will need to use
the boost library for some cases but nothing in the hot path. We might want to revisit
implementing an int96 in the future that is of the same format to get some space
and efficiency savings but I think we can live with just int128 for a while.
Change-Id: I137ef7be812675036dd9b6e5b48dfc5c7aa9ab37
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2200
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2249
When the -fair_scheduler_allocation_path is configured with a policy that uses
the "primaryGroup" Yarn queue allocation rule, Yarn throws an error if the user
is not on the local OS. Currently the user will get an error message that says:
"java.io.IOException: No groups found for user <username>". We now return a more
helpful error message.
Change-Id: I014ac15ef607e473957752f23af94d0cc4efec0f
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2078
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
(cherry picked from commit 3cf37dc4e91afe887ada988f256b7008983580d2)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2244
FNV hash has the property that the least significant bit of the hashed value
is just the XOR of the LSBs of its input bytes. This results in poor
distribution of rows when the partition keys are duplicated -- for example,
if the partition key is (l_orderkey, l_orderkey). A recommended technique
to mitigate this is to generate a larger hash and use XOR-folding to reduce
it to the desired length.
In this patch FnvHash has been modified to use generate a 64-bit hash and
fold the result down to 32-bits. It has been renamed FnvHash64to32 to make
this explicit.
Change-Id: Ie12ad3f863fca15092803d3e4d616a654cb8d244
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2220
Reviewed-by: Srinath Shankar <sshankar@cloudera.com>
Tested-by: jenkins
Currently, the entire object is protected by one lock. Unfortunately this
lock is taken during calls into libhdfs. This means it is impossible for the
scan node to pull off ready buffers while the disk thread is reading from
this scan range. Whoops. The lock is taken while in libhdfs to facilitate cleanup
so it's very easy to split the big lock up.
Change-Id: Idbf34cdba0cf860a90f9cad016d1ec133f923d85
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2143
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2202
Partition metadata tests were marked as xfail because of IMPALA-624. Additionally, we had
to invoke hive to insert into two partitions pointing to the same location (this
limitation is now removed). This patch changes the test to use Impala exclusively,
removes the xfail tag and adds a teardown method to the test class.
Change-Id: I15fa97bef4f8714d0873a9c713627a198f3388ad
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2086
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2215
A few tests which dealt with running queries via hs2 and impala were marked as xfail as
hiveserver2 would occasionally not come up. Given that we now have a script that checks
whether hiveserver2 is up before continuining the build, it should be safe to remove the
xfail.
Change-Id: I2b5063e7259c01fc0ef8ffda86d85514c9cf959c
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2082
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2214
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
The queries in test_cancellation are currently cancelled but not closed, causing some test
queries to eventually time out because the admission controller limits are passed. This
patch ensures that all queries issued in test_cancellation are closed.
Change-Id: I65b26672155e31889bb6f43d3ac87be0f7b4eb72
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2187
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2213
This finishes up the support to use HDFS caching. The scheduler will
prefer replicas that are cached and the scan node plumbs the metadata
to the io mgr.
This is a bit hard to test without a cluster and some perf benchmarking.
I've added a basic test to make sure the path is being exercised.
Change-Id: I8762ca9ef2f88c3637113d3c5ee82f4c0ea7f1be
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2212
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
This patch modifies DelimitedTextParser and StringValue to work with
data containing null characters by using SSE instructions that take a
length, rather than expecting null-terminated strings. It also adds
some other minor changes to correctly handle data with nulls and to
faciliate testing. I checked the execution time of a count(*) and a
select(*) limit 1 query locally, and saw no difference for either text
or sequence files.
Change-Id: Ia920b35bea7048aa286f39ec83e313c2a39251d1
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2110
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Skye Wanderman-Milne <skye@cloudera.com>
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2181
The bug: Coordinator::Wait() is supposed to block until rows become available for
consumption by the client. We rely on Wait() to determine when to advance the query
status to a 'ready' state and signal to the client that rows can be fetched.
Long fetch times can trigger client timeouts at various levels (socket, app, etc.).
Coordinator::Wait() simply opens the coordinator fragment's plan tree.
For most plan nodes, Open() does work to prepare the plan tree, s.t., GetNext()
returns quickly. However, for ExchangeNodes Open() used to not wait
until rows are obtained form the underlying stream receiver.
The fix: Make ExchangeNode::Open() block until rows are available.
Change-Id: I7b197eea11d21fd732414d96c899a17b2d99631c
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2128
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2185
COMPUTE STATS is an async DDL command. When COMPUTE STATS fails it will set the
query status of the QueryExecState properly, but the original Beeswax::query() RPC
won't throw. The Impala shell sometimes did not pick up and display the
query status because no RPC actually threw. To fix this, I modified
Beeswax::get_log() to include the query status if it is not ok. The shell looks
for a special prefix to distinguish the query status from the runtime state error log.
Change-Id: I0d9dbf0801629a37de22ea4ebb6d2e5d53b836ef
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1899
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2063
This statement returns info on all partitions for the given table. It is implemented as
an alias for SHOW TABLE STATS, with some extended analysis checks (such as throwing if
the statement targets an unpartitioned table).
Change-Id: I19154a9d90314de18f86ba355aa5dbed808f147f
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2145
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: Lenni Kuff <lskuff@cloudera.com>
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2179
Tested-by: jenkins
Impala determines the location (network address) of all block replicas using
the HDFS API BlockLocation.getNames(), which returns results in IP:port format.
To find where cached replicas are located we call BlockLocation.getCachedHosts(),
which returns results as hostnames. This caused an issue where we would compare
an IP address to a hostname to determine if a replica was cached.
The fix is to resolve cached hosts by comparing against BlockLocation.getHosts(),
which returns the block replica locations by hostname. getHosts() will always return
results in the same order as getNames() and getHosts() and may contain duplicate
entries (multiple data nodes on the same host), which is what we want. This allows
the same array index to be used to convert between the two location formats.
Change-Id: I74fdc20b1dc5200d7e0e90856b8b2088f050e215
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2156
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: Lenni Kuff <lskuff@cloudera.com>
To keep the predicate assignment/propagation logic simple, we assign conjuncts
whose underlying base table exprs are constant in at least one union operand
to the evaluating MergeNode, and not in the operand(s) whose corresponding base
table exprs are constant.
The JIRA describes two different bugs:
The first bug was that the slots required for evaluating such predicates in the
MergeNode were not marked as materialized. The second bug was that predicates
'pushed' into union operands did not get re-analyzed after substituting the
predicate's exprs with the result exprs of that union operand. Missing casts
lead to a crash. The new test covers both bugs.
Change-Id: I0f5b8a366b32f7d4b2587e13793b6103cdf7e8b3
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2162
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
The standard implementation of HashTable::Equals() did not correctly
check the NULL bit when the argument row did not evaluate to NULL for a
given probe expr. In the rare circumstance that this gave rise to a
false positive (more on that below), two rows with different grouping
values would be considered equal, and one would be excluded from the
final aggregation output.
HashTable::EvalRow() fills an expression value buffer with the values of
either probe or build exprs evaluated for the argument row. These cached
values are used to determine row equality in Equals(). In order to avoid
a lot of false collisions, an 'unlikely' value is written to that buffer
for NULL values, chosen to be HashUtil::FNV_SEED. So without correct
NULL-bit checking in Equals(), two single-slot rows are considered to be
equal if one of them has NULL for its slot, and the other has a value
equal to HashUtil::FNV_SEED truncated to the size of the slot.
For tinyint columns, this value is -59. As it happens, our random
generator happened to create a table with one tinyint column and which
contained NULL and -59 as values. In order to trigger this bug, the rows
must also have been written to disk in order such that the scanners
returned -59 *first*, and then NULL to the aggregation node; the bug is
not symmetric and works in the opposite case.
Change-Id: I17d43eaeee62b2ac01b67dd599bc4346b012a074
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2130
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: jenkins
(cherry picked from commit 6e8098254280a9d5ead0b607263ca6728a3222a7)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2161
Reviewed-by: Henry Robinson <henry@cloudera.com>
We wrap certain exprs substituted from outer-joined inline view in an expr that
evaluates to NULL if the underling tuple(s) are NULL. We do this for exprs that evaluate
to non-NULL values if their slots are NULL, i.e., we must then distinguish tuples that are
NULL from slots that are NULL (otherwise evaluating an expr against a tuple that is NULL
due to the outer join may incorrectly return a non-NULL value.)
The bug: Exprs referring to an outer-joined inline view may appear in various places
in the outer query block. For example, they could appear in an On-clause or be
placed into scans/aggregates due to predicate propagation. In such cases, the underlying
tuples may not be nullable yet because they only become nullable after the outer join.
We had a DCHECK in tuple-is-null-predicate.cc requiring the tuples to be nullable.
The fix: Remove the DCHECK. The fix is not elegant but practical. It would be rather
difficult to fix the inline view expr substitution such that a TupleIsNullPredicate
never references a non-nullable tuple, esp. due to predicate propagation.
Change-Id: I180f75f14173f356abfeec751e6b2d419378a9a7
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2157
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
their parent's permissions
This patch adds --insert_inherit_permissions. If true, all
new partition directories created by INSERT will inherit their
permissions from their parent. When false, the directories are created
with the default permissions.
Change-Id: Ib2b4c251e51ea5048387169678e8dde34ecfe5f6
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1917
Tested-by: jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
Our HS2 Metadata Op implementation would not return any results if null was passed as the
table name or column name. Instead a null value should be treated the same as '%' (match
everything).
Change-Id: Ibad41e94724cd1f9c1caf40831e30a98132247d9
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2137
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
(cherry picked from commit 7020c62545397872877c03a5e101e71edf8101bf)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2142
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
The admission controller is configurable via Yarn fair scheduler allocation
and Llama configurations, but a "default pool" is used when these files are
not provided. When a pool is defined in a fair-scheduler.xml but no limits
are specified, the following Yarn/Llama default values are used: the max
number of concurrent requests is 20, the max queue size is 50, and the mem
limit is unlimited.
This changes the default values of the "default pool" limits so that the
limits are consistent with the defaults from Yarn/Llama.
Change-Id: Ic76ff550c18cc49353c72926591af46dcbe26ac7
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2006
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
(cherry picked from commit 1619d83e452e5b868d12e3934e9704fc5f16cac7)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2118
