Commit Graph

731 Commits

Author SHA1 Message Date
Alex Behm
91e1eb0789 CDH-18563: Speed up the computation of transitive value transfers.
The issue: Computing the full transitive closure for all slots can be very
expensive (10s of seconds for >2k slots, minutes for >4k slots).
Queries with many views and/or unions were affected most because each
union/view adds a new tuple with slots, increasing the total number of slots.

The fix: The new algorithm exploits the sparse structure of the value transfer
graph for a significant speedup (>100x). The high-level steps are:
1. Identify complete subgraps based on bi-directional value transfers, and
   coalesce the slots of each complete subgraph into a single slot.
2. Map the remaining uni-directional value transfers into the new slot domain.
3. Identify the connected components of the uni-directional value transfers.
   This step partitions the value transfers into disjoint sets.
4. Compute the transitive closure of each partition from (3) in the new slot
   domain separately. Hopefully, the partitions are small enough to afford
   the O(N^3) complexity of the brute-force transitive closure computation.

Change-Id: I35b57295d8f04b92f00ac48c04d1ef1be4daf41b
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2360
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
2014-04-24 23:53:28 -07:00
Skye Wanderman-Milne
bd2fc2d1d4 IMPALA-934: Refresh cached UDF library when creating a new function
This change adds the ability to refresh a local cache entry, causing
the old cache entry to be dropped and the library to be reloaded from
HDFS. This is used in ResolveSymbolLookup(), which is called by the
frontend when creating a new a function, and in ImpalaServer when
receiving a "create function" heartbeat. This change also makes sure
the FE calls into the backend for jars, so jars get refreshed as well.

Change-Id: I5fd61c1bc2e04838449335d5a68b61af8b101b01
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2286
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
(cherry picked from commit e8587794b3b82438190c91b2ebe9d1e12db73981)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2348
2014-04-24 19:39:16 -07:00
Alex Behm
121fab8fdf IMPALA-888: Drop union operands with constant conjuncts evaluating to false.
This patch simplifies the complex slot materialization logic for unions by
making the materialization independent of conjuncts assigned to MergeNodes.
When 'pushing down' predicates into union operands, we drop union operands
with constant predicates evaluating to false. Constant predicates that
evaluate to true are simply ignored.

Change-Id: I0e7ccfb206bed29db2b5d667e2bb61310980e80a
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2327
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
2014-04-23 18:25:14 -07:00
casey
2351266d0e Replace single process mini-dfs with multiple processes
This should allow individual service components, such as a single nodemanager,
to be shutdown for failure testing. The mini-cluster bundled with hadoop is a
single process that does not expose the ability to control individual roles.
Now each role can be controlled and configured independently of the others.

Change-Id: Ic1d42e024226c6867e79916464d184fce886d783
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1432
Tested-by: Casey Ching <casey@cloudera.com>
Reviewed-by: Casey Ching <casey@cloudera.com>
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2297
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: Ishaan Joshi <ishaan@cloudera.com>
2014-04-23 18:24:05 -07:00
Lenni Kuff
45a734f6cd IMPALA-951: Throw parser error if no partition spec specified in ALTER TABLE ADD/DROP PARTITION
Change-Id: I876423e39d858d602ed0fbe8369a6714c82639d8
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2295
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2320
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
2014-04-23 11:20:35 -07:00
Matthew Jacobs
b1c331fd81 IMPALA-956: RequestPoolService should use short username of principal
We should be using the short name of a Kerberos principal (e.g.
user/fully.qualified.domain@realm.com) or LDAP username (e.g. user@domain)
when checking group membership in RequestPoolService. Right now we call
UserGroupInformation.createRemoteUser() with the full user name and it
will throw an exception.

Change-Id: I39d849627cb49760807504d66109c05b7a399482
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2288
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
(cherry picked from commit 0005da9cb71f5a4a4ed6bb1dfcd74f8526cd8316)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2305
2014-04-22 13:55:39 -07:00
Victor Bittorf
c414c91931 Adding TRUNC builtin.
Includes additions to builtin UDF registration to support prepare/close.

Change-Id: I22668fa7ee033b3fa37050b7bccee935571ac453
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2243
Reviewed-by: Victor Bittorf <victor.bittorf@cloudera.com>
Tested-by: jenkins
2014-04-22 13:17:12 -07:00
Alex Behm
689870ca3a IMPALA-914: Map null type to boolean in JDBC to be compatible with Hive.
Change-Id: I5831ae7d5dcb03aecea4138d0b13487898951068
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2279
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2282
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
2014-04-21 15:00:32 -07:00
Alex Behm
c8e928119d IMPALA-912: Enforce slot equivalences at the lowest possible plan node.
The reported issue is that we can have redundant hash expressions in exchanges.
The underlying cause is that we fail to remove redundant join predicates.
This patch enforces slot equivalences based on our computed equivalence classes
at the lowest possible plan node by generating new equality predicates.
Each plan subtree now has a minimal set of equality predicates that express
all known equivalences between slots belonging to tuples materialized at that
plan node.
As a result, eliminating redundant join predicates becomes trivial: It is
sufficient to pick a single representative predicate of each relevant equivalence
class. All predicates beyond that are redundant.

Change-Id: I7998fe8d7bdf84cc8eb129d32c86269bedeab68e
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2177
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2278
2014-04-18 13:28:49 -07:00
Lenni Kuff
15327e8136 Migrate DataErrors tests to Python test framework, re-enable subset of tests
This re-enables a subset of the stable data errors tests and updates them to
work in our test framework. This includes support for updating results via --update_results.

This also lets us remove a lot of old code that was there only to support these disabled
tests.

Change-Id: I4c40c3976d00dfc710d59f3f96c99c1ed33e7e9b
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1952
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2277
2014-04-18 02:25:11 -07:00
Henry Robinson
2a69019525 IMPALA-945: Fix column reordering with SELECT expressions
Previously, to produce the correct output expressions for the root plan
fragment before a table sink, InsertStmt would reorder the result
expressions for the query statement at the plan root. This had stopped
working for SelectStmts (and test coverage didn't catch that).

Now InsertStmt produces its own output expressions that can substitute
for the originals from the query statement, and the planner uses those
instead.

All query tests for column reordering have been duplicated to use SELECT
expressions.

Change-Id: Ib909fe35d27416b33ba2e5ac797aa931e1fe43f9
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2204
Tested-by: jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
(cherry picked from commit d526db7ac6274f35b6affcb7428327100026e14e)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2275
2014-04-18 00:12:12 -07:00
Nong Li
1cab95066d Add the return type as a column for SHOW FUNCTIONS.
Also includes some misc pattern matching cleanup.

Change-Id: I6c9ec78b094a73864b4d669afbd75a48c9bf9585
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2199
Tested-by: jenkins
Reviewed-by: Nong Li <nong@cloudera.com>
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2271
2014-04-17 17:58:13 -07:00
Nong Li
44fd279f18 Decimal: switch out the boost 128_t int with the c++ standard one.
The c++ standard int128_t is exactly what we want. It is 16 bytes, stored as 2's
complement little endian (the exact extension of the native int types). It out
performs the boost library we were using (see benchmark) and looking at the assembly
for some of the operators, I doubt we can do better. This also seems like the kind
of thing hardware might be able to do natively in the future if we stuck with the
standard implementation.

This requires minimal changes to the rest of our code so the multi int library is
abstracted away.

The standard only added int128 and not 96 or any others. We still will need to use
the boost library for some cases but nothing in the hot path. We might want to revisit
implementing an int96 in the future that is of the same format to get some space
and efficiency savings but I think we can live with just int128 for a while.

Change-Id: I137ef7be812675036dd9b6e5b48dfc5c7aa9ab37
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2200
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2249
2014-04-15 23:24:46 -07:00
Matthew Jacobs
d0c353a9b4 IMPALA-922: Return helpful errors with Yarn group rules
When the -fair_scheduler_allocation_path is configured with a policy that uses
the "primaryGroup" Yarn queue allocation rule, Yarn throws an error if the user
is not on the local OS. Currently the user will get an error message that says:
"java.io.IOException: No groups found for user <username>". We now return a more
helpful error message.

Change-Id: I014ac15ef607e473957752f23af94d0cc4efec0f
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2078
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
(cherry picked from commit 3cf37dc4e91afe887ada988f256b7008983580d2)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2244
2014-04-15 15:32:05 -07:00
Nong Li
87295a4e06 Decimal implementation.
This patch implements decimal support for text based formats.

Change-Id: I8e2c9e512ed149fe965216a72cb21fffd4f18e75
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1669
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: Nong Li <nong@cloudera.com>
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2238
Tested-by: jenkins
2014-04-14 21:07:32 -07:00
Lenni Kuff
9e2dd7e049 Add support for SHOW PARTITIONS <table name>
This statement returns info on all partitions for the given table. It is implemented as
an alias for SHOW TABLE STATS, with some extended analysis checks (such as throwing if
the statement targets an unpartitioned table).

Change-Id: I19154a9d90314de18f86ba355aa5dbed808f147f
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2145
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: Lenni Kuff <lskuff@cloudera.com>
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2179
Tested-by: jenkins
2014-04-10 12:15:39 -07:00
Lenni Kuff
f1f4e99c85 [CDH5] IMP-1326: Impala assumes BlockLocation#getCachedHosts returns IP addresses
Impala determines the location (network address) of all block replicas using
the HDFS API BlockLocation.getNames(), which returns results in IP:port format.
To find where cached replicas are located we call BlockLocation.getCachedHosts(),
which returns results as hostnames. This caused an issue where we would compare
an IP address to a hostname to determine if a replica was cached.

The fix is to resolve cached hosts by comparing against BlockLocation.getHosts(),
which returns the block replica locations by hostname. getHosts() will always return
results in the same order as getNames() and getHosts() and may contain duplicate
entries (multiple data nodes on the same host), which is what we want. This allows
the same array index to be used to convert between the two location formats.

Change-Id: I74fdc20b1dc5200d7e0e90856b8b2088f050e215
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2156
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: Lenni Kuff <lskuff@cloudera.com>
2014-04-08 22:06:14 -07:00
Alex Behm
0585dfb546 IMPALA-888: Materialize union slots referenced by constant predicates.
To keep the predicate assignment/propagation logic simple, we assign conjuncts
whose underlying base table exprs are constant in at least one union operand
to the evaluating MergeNode, and not in the operand(s) whose corresponding base
table exprs are constant.
The JIRA describes two different bugs:
The first bug was that the slots required for evaluating such predicates in the
MergeNode were not marked as materialized. The second bug was that predicates
'pushed' into union operands did not get re-analyzed after substituting the
predicate's exprs with the result exprs of that union operand. Missing casts
lead to a crash. The new test covers both bugs.

Change-Id: I0f5b8a366b32f7d4b2587e13793b6103cdf7e8b3
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2162
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
2014-04-07 18:32:29 -07:00
Alex Behm
8b319f8959 IMPALA-935: Make PlanFragment.getDestFragment() return null if no destination is set.
Change-Id: I269a7f552d7ff67ff4d65e86e8c6df9c41d0fca1
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2159
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
2014-04-07 16:21:24 -07:00
Alex Behm
a85dacafe8 IMPALA-904: Make TupleIsNullPredicate work on non-nullable tuples.
We wrap certain exprs substituted from outer-joined inline view in an expr that
evaluates to NULL if the underling tuple(s) are NULL. We do this for exprs that evaluate
to non-NULL values if their slots are NULL, i.e., we must then distinguish tuples that are
NULL from slots that are NULL (otherwise evaluating an expr against a tuple that is NULL
due to the outer join may incorrectly return a non-NULL value.)

The bug: Exprs referring to an outer-joined inline view may appear in various places
in the outer query block. For example, they could appear in an On-clause or be
placed into scans/aggregates due to predicate propagation. In such cases, the underlying
tuples may not be nullable yet because they only become nullable after the outer join.
We had a DCHECK in tuple-is-null-predicate.cc requiring the tuples to be nullable.
The fix: Remove the DCHECK. The fix is not elegant but practical. It would be rather
difficult to fix the inline view expr substitution such that a TupleIsNullPredicate
never references a non-nullable tuple, esp. due to predicate propagation.

Change-Id: I180f75f14173f356abfeec751e6b2d419378a9a7
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2157
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
2014-04-07 14:18:49 -07:00
Lenni Kuff
c798b23fd9 IMPALA-925: JDBC driver returns no results from getTables()/Columns() with null name pattern
Our HS2 Metadata Op implementation would not return any results if null was passed as the
table name or column name. Instead a null value should be treated the same as '%' (match
everything).

Change-Id: Ibad41e94724cd1f9c1caf40831e30a98132247d9
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2137
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
(cherry picked from commit 7020c62545397872877c03a5e101e71edf8101bf)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2142
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
2014-04-03 17:12:25 -07:00
Matthew Jacobs
f52662c739 Fix TestRequestPoolService, do not use URL.getPath()
URL.getPath() will return a valid URL, which means that some unsafe ASCII
characters are encoded for URLs. In some test environments we had repos
with '@' in the path and this was not handled properly by this test when
we try to load test resources.

Change-Id: I74314c40d7c70b908456cd9263263e83c79e0664
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2108
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
(cherry picked from commit a7118d509bb897086966ad98e95052176246801f)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2115
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
2014-03-28 18:42:49 -07:00
Nong Li
c27bd34075 Revert "Disable decimal in analysis."
This reverts commit 695017410adf6d4f8426c4117798c93f823a4b4b.

Change-Id: I919d965e8e711d588e6c56dcdbd3c8e0d9ec7a05
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2104
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
2014-03-27 12:45:55 -07:00
Skye Wanderman-Milne
3e728f3180 Symbol mangling for UDF prepare/close functions
Change-Id: If8f1386073f467e66ada74e606fc98f3344f0733
(cherry picked from commit 32df8b3f963a2b46ec33aad86a151d4c7ecda39c)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1993
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
2014-03-19 02:15:07 -07:00
Matthew Jacobs
0492087f3e Admission controller: Fix NPE when Llama configuration is not provided
Change-Id: Ia27e1e727ad1891d05b39e16ca6eafd9a7362a00
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1985
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1992
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
2014-03-19 00:32:39 -07:00
Matthew Jacobs
c0654cf60e Admission control: Few trivial improvements for logging and clarity
Change-Id: I725c1c0845974ab73e77d0262701ddd4346e4f78
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1945
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
(cherry picked from commit c639a595bceb3bc61894f390e0a919d37b48cf0c)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1959
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
2014-03-17 22:17:52 -07:00
Skye Wanderman-Milne
44125729dc UDF/UDA memory management improvements
* AggFnEvaluator now uses the UDF mem pool (I'm planning to change
  this to per-exec node pools in the expr refactoring)
* FunctionContext::TrackAllocation()/Free() actually use the UDF's mem tracker
* Added FunctionContextImpl::Close() which sets warnings for leaked allocations

Change-Id: I792ffd49102a92b57e34df18d8ff5f5d0fd27370
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1792
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Skye Wanderman-Milne <skye@cloudera.com>
(cherry picked from commit 41a5f7cfa718789fa3b2de3a31f085411fb5000c)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1954
Tested-by: jenkins
2014-03-17 20:38:25 -07:00
Matthew Jacobs
4638505cba Admission controller: Fix default parameter to QueuePlacementPolicy.assignAppToQueue()
We should be passing YarnConfiguration.DEFAULT_QUEUE_NAME as the default value to
QueuePlacementPolicy.assignAppToQueue(). This caused a problem when the placement
policy was Specified(create=true) where the returned name was "root." because it
literally created a pool with the name "" below root.

Change-Id: I41cc2848d78bf8bd308292e80d6f6c7d06437ae9
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1942
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
(cherry picked from commit 451627ffca6fffb562adcb18290bc7388d734e7f)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1949
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
2014-03-17 18:40:01 -07:00
Henry Robinson
635dd7d289 IMPALA-875: Respect isAnalyzed_ in IntLiteral expressions
Partition column expressions are analysed twice for INSERT statements -
once to infer the type and so to add a possible cast, and once to
compute stats on the resulting expr. However, this process resulted in
an partition column expr that was a IntLiteral getting the smallest type
that would contains its value, rather than retaining the
column-compatible type that had been assigned to it.

This patch does the minimum thing, which is make IntLiteral.analyze()
idempotent. Doing the same thing to Expr and LiteralExpr unearths some
other bugs, which we will have to fix in a follow-on patch (see
IMPALA-884).

Change-Id: Ie22fc5d3f4832c735a1ebc0ef78f50d736f597fd
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1931
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins
(cherry picked from commit 1912d65ea21a5025d385948642f0d4aadad91abf)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1947
2014-03-17 17:35:12 -07:00
Alex Behm
f4cfc75544 IMPALA-860: Place outer and semi joins at a fixed position in the plan.
The bug: It is generally incorrect to re-order joins across outer/semi joins.
For example, an inner join following an outer join may reduce the cardinality,
so placing the inner join before the outer join during join re-ordering
would be incorrect because the outer join is cardinality preserving
(on one or both sides). The same argument holds for semi joins.
The fix: Place outer and semi joins at a fixed position in the plan based
on where they appeared in the original query. Inner joins to the left/right of
outer/semi joins are still re-ordered properly.

Change-Id: Idae837097b9376473d7f8124eef69b51f612b210
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1909
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1922
2014-03-15 05:18:11 -07:00
Srinath Shankar
74a975c45b IMPALA-862: count(x) may return null when a similar count(distinct x) is also used
count(x) with no distinct and no group-by expressions returns NULL on empty input
if other distinct aggs (e.g. COUNT(distinct x) are present.
This happens because the COUNT is transformed to SUM(COUNT()),
with the inner COUNT being evaluated WITH a group-by expression (e.g. x).
SUM over empty input returns NULL, but COUNT should return 0.

This patch fixes this by replacing COUNT with zeroifnull(COUNT) before AggregateInfo
is generated if there are distinct aggs and no group-bys. The logic in AggregateInfo
itself has not been modified.

Change-Id: I902e3fdd95767135b2f3fe423e8802ef57366af1
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1921
Reviewed-by: Srinath Shankar <sshankar@cloudera.com>
Tested-by: jenkins
2014-03-14 23:35:55 -07:00
Alex Behm
ce40134ad0 IMPALA-867: Fail COMPUTE STATS in analysis for Avro tables affected by HIVE-6308.
Avro tables that were not created with a column-definition list do not have
their columns properly populated in the Metastore backend DB (HIVE-6308).
For such tables COMPUTE STATS and Hive's ANALYZE TABLE cannot succeed.
This patch fails COMPUTE STATS in analysis for such broken Avro tables
and adds tests for Avro tables with mismatched a column-definition list
and Avro schema.

Change-Id: I561ecea944ae2f83d69950b7a1ab9edaa89bdcea
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1892
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1920
2014-03-14 23:24:55 -07:00
Lenni Kuff
aa0b7a35f5 IMPALA-880: COMPUTE STATS should update partitions in batches
When updating partition metadata as part of COMPUTE STATS we would previously
attempt to update all partitions at once. This could lead to HMS socket timeouts
and also could run into issues if there were > 32K partitions.

In this change we now update the partitions in batches, with a max size of 500
partitions per batch. We also compare whether the row count has changed and only
update partitions that have been modified.

Change-Id: If7bfcc30f86fc2fdd79855b981067ac29a47b5e1
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1913
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1918
2014-03-14 19:20:12 -07:00
Lenni Kuff
fd174a5e69 [CDH5] Remove duplication of network addresses in HdfsTable (updated for HDFS caching)
This is a port of 0b9134a from CDH4, but required some adjustments to work on CDH5 due to
the HDFS caching work. The differences from CDH4/CDH5 are mainly in
HdfsTable/HdfsPartition. I added a new BlockReplica class to represent a single block
with info on the host index + caching info.

This removes duplication TNetworkAddresses in the block location metadata of HdfsTable.
Each HdfsTable now contains a list of TNetworkAddress and the BlockLocations
just reference an index in this list to specify the host, rather than duplicating the
TNetworkAddress.

For a table with 100K blocks, this reduces the size of the THdfsTable struct by an additional
~50+% (on top of the duplicate file path changes). This takes the total size of the table
from:
21.1MB -> 9.4MB (file path duplication) -> 4.2MB (host duplication) = ~80% total improvement.

Change-Id: If7f11764dc0961376f9648779d253829f4cd83a2
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1367
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1887
Reviewed-by: Nong Li <nong@cloudera.com>
2014-03-14 14:30:08 -07:00
Alex Behm
01d25f5a81 Use unsetIsAnalyzed() in Analyzer.isTrueWithNullSlots() for implicit casting.
Change-Id: I99d6bf227c74c4967f5825626329c637a73e7b1a
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1896
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1904
2014-03-14 01:39:16 -07:00
Matthew Jacobs
8571c76b98 Admission controller: Fix exception message when llama-site cannot be found
We no longer look for files on the classpath, we just try to find the file
specified as it is.

Change-Id: I127f508ff6b91f671385985d32b7898b208ae2ed
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1895
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
(cherry picked from commit bc26d985a1eb9e2489d5ef0934d56016f586041a)
2014-03-13 19:21:40 -07:00
Alex Behm
15e05082c0 IMPALA-831: Distributed aggregation and top-n over unions.
Change-Id: I056e8271421008378db93e8b2393861cc9dd4b90
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1840
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1886
2014-03-13 15:42:31 -07:00
Lenni Kuff
cc1c0c61fd IMP-1291: Support "extended" ASCII characters as delimiters in text files
This fixes how we validate delimiters to be in line with Hive. A delimiter must
fit in a single byte and can be specified in the following formats, as far as I can
tell (there isn't documentation):
- A single ASCII or unicode character (ex. '|')
- An escape character in octal format (ex. \001. Stored in the metastore as a
  unicode character: \u0001).
- A signed decimal integer in the range [-128:127]. Used to support delimiters
  for ASCII character values between 128-255 (-2 maps to ASCII 254).

Previously, we were not handling the "signed integer" case so there was no way
to specify a delimiter in the "extended" ASCII range of 128-255.

To support result validation, the test infrastructure had to be updated to support
reading/writing different character encodings.

Change-Id: Ie3c4d444dc9c6e60192093ed0c0f6f151eab16bc
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1848
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1888
2014-03-13 13:00:15 -07:00
Alex Behm
d640273a3f IMPALA-861: Proper slot materialization on distinct aggs inside an inline view.
The bug: Slot materialization on distinct aggs inside an inline view did not work
if the only reference to the 2nd-phase agg-tuple slots was in a predicate from an
outer query block (e.g., Where-clause of the block with the inline view ref).
The reason was that bound predicates were fetched from the wrong tuple
(from the 1st phase agg).
The fix: Assign predicates to the top-most agg in the single-node plan that can
evaluate them, as follows: For non-distinct aggs place them in the 1st phase agg
node. For distinct aggs place them in the 2nd phase agg node.

Change-Id: I0f6ab53cf7bb0c6aed9524ad2e24a849d2dc0ec4
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1843
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1881
2014-03-13 04:39:48 -07:00
Alex Behm
7fcd7cd64e Add list of tables missing stats to explain header and mem-limit exceeded error.
Change-Id: Ibe8f329d5513ae84a8134b9ddb3645fa174d8a66
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1501
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1880
2014-03-12 21:15:22 -07:00
Nong Li
b1aeea3f0b Disable decimal in analysis.
Change-Id: I4cff1fc74ef0afeba15bee5b9eb6851abbfddbdf
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1874
Tested-by: jenkins
Reviewed-by: Nong Li <nong@cloudera.com>
2014-03-12 18:55:06 -07:00
Alex Behm
58950a52a3 IMPALA-798: Distributed execution of CTAS and explain CTAS.
Change-Id: I32004a4b31c54cf5c185169fece143a61213d12d
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1850
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1867
2014-03-12 16:51:50 -07:00
Matthew Jacobs
e817c3742c Admission controller: fix a number of TODOs
* Remove requirement that fair scheduler and Llama conf files be on the classpath if
  specified as relative paths. Now they can be specified as any relative or absolute
  path.
* Add flags to disable all per-pool max requests limits or mem limits.
* Rename RequestPoolUtils to RequestPoolService
* Make it more clear RequestPoolService is a singleton by putting it in ExecEnv
* FileWatchService: use Executors.newScheduledThreadPool instead of a thread
* Moved MEGABYTE (and related constants) to new Constants class (frontend)
* Test RequestPoolService: Removed AllocationFileLoaderServiceHelper, replaced with
  reflection

Change-Id: Iadf79cf77a7894a469c3587d0019a6d0bee7e58f
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1787
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
(cherry picked from commit b9a167f6fdb4ab2595aca6035e1f9d926b909d94)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1858
2014-03-12 14:23:54 -07:00
Nong Li
5966492a61 IMPALA-855: Improve error message for BinaryPredicate.
Change-Id: Ia31661207d84c460b3882bcd0186f3df91369f66
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1841
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
2014-03-12 13:28:08 -07:00
Lenni Kuff
08417c875f IMPALA-849: Impala does not work with boolean partition key columns
This is because in HdfsTable we call call "expr.castTo(colType)", but BooleanLiteral
(incorrectly) didn't implement "uncheckedCastTo()". This meant that instead of a
BooleanLiteral being returned we got back a CastExpr, which cannot be cast to LiteralExpr.

As part of this change it turns out Boolean partition columns are also broken in Hive. I
filed HIVE-6590 for these issues and we decided to disable INSERT into a boolean partition
column for Impala due to this bug.

Change-Id: I3e295bb96aadc08d64faf551f6393a7128a7ef27
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1755
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
2014-03-11 18:42:08 -07:00
Alex Behm
47c52ade84 IMPALA-866: Make HdfsScanNode.computeStats() idempotent with respect to totalBytes_.
Change-Id: I1c243b089db82c0544586a2a1428081aa2dbcd52
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1844
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1852
2014-03-11 18:20:15 -07:00
Alex Behm
3cdb2d9fb1 IMPALA-719: Output correct SQL in SelectStmt.toSql() if the stmt has not been analyzed.
Change-Id: I76ec5916352ace6aa9de1f90f47c4bbf6d369b93
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1172
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1820
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
2014-03-08 06:04:37 -08:00
Alex Behm
fe4f3babe5 IMPALA-824: Generalize getBoundPredicates() to handle multi-slot predicates.
The bug: Multi-slot predicates bound to a single
outer-joined tuple were not marked as materialized.
In addition, such predicates were not picked up by nodes
under the join via getBoundPredicates() even if it
would be correct to do so.
The fix: Always mark slots of predicates that must be
evaluated by a join in SelectStmt.materializeRequiredSlots(),
regardless of whether the predicate can also be safely
evaluated below the join.
This patch also generalizes getBoundPredicates() to handle
multi-slot predicates and fixes some issues with redundant
predicate assignment. Still, the new approach has several
limitations which are documented in the predicate propatation
planner test to ease future improvements.

Change-Id: If5da0354a83c00a9766fc63b7780ed4d5a9c46e5
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1717
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1819
2014-03-08 04:22:39 -08:00
Alex Behm
a615ebc549 IMPALA-822,IMP-1271: Binding predicates on an aggregation now properly trigger slot materialization.
The bug was that the number of materialized agg-tuple slots did not correspond to the number
of materialized agg functions, due to binding predicates against an AggNode causing slot
materialization after SelectStmt.materializeRequiredSlots().
This patch fixes the issue by taking binding predicates (bound to a slot in an agg tuple)
into consideration in SelectStmt.materializeRequiredSlots().
I added a new sanity check in AggregationNode.toThrift() surfaced another issue with slot
materialization that is also fixed in this patch. The ordering exprs must be marked before
the agg exprs in SelectStmt.materializeRequiredSlots() because the odering exprs may contain
agg exprs that are only referenced inside the ORDER BY clause.

Change-Id: I1bdc0466f583907bed625ce6608938e59faee83f
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1639
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1818
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
2014-03-08 00:25:26 -08:00
Alex Behm
f7c2781afe IMPALA-845: Transfer predicates to 2nd phase merge agg in some cases.
Having predicates need to be transferred to the 2nd phase merge agg
for distinct + non-distinct aggregates without group by.
For distinct + non-distinct aggregates with group by, it is correct
to evaluate the predicates at the 2nd phase (non-merge) agg.

Change-Id: I71d73c4ef92becbb81e142bc0cb5f54e790b1fb5
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1743
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1817
2014-03-07 21:45:16 -08:00