It appears that HBase sometimes ignores an admin.splitRegion() RPC,
which made our region splitting fail. As a workaround, this patch adds
another retry loop such that the split/wait sequence is attempted
multiple times.
Change-Id: I9aa8ab87bba79ea11b79c50f15328b8be844924d
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4557
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: Alex Behm <alex.behm@cloudera.com>
Defer resizing the columns_ vector until we are sure we will initialize it.
Downstream code doesn't expect any NULLs.
Change-Id: I250cceee5181428fcd3cd1a8b021edb7187ae888
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4465
Reviewed-by: Daniel Hecht <dhecht@cloudera.com>
Tested-by: jenkins
The 'less than' predicate created by AnalyticPlanner used to check if the
previous row was less than the current row is not exactly what we want
to determine when rows in RANGE windows (the default window in this case)
share the same result values. Rows get the same results when the order by
exprs evaluate equally or both null, so it's easiest (and more efficient)
to use a predicate that simply checks equality or both null. We already
create such predicates for checking for partition boundaries, so this is
a trivial change.
When we support arbitrary RANGE window offsets we will likely want to
add similar predicates that compare two tuples plus/minus the offset,
but those will be simpler because there can be only one order by expr
when specifying RANGE offsets with PRECEDING/FOLLOWING.
Change-Id: I52ff6203686832852430e498eca6ad2cc2daee98
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4474
Tested-by: jenkins
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Matthew Jacobs <mj@cloudera.com>
1) Adds BE support for RANGE windows between UNBOUNDED PRECEDING to
UNBOUNDED FOLLOWING.
2) RANGE windows with offset boundaries fail analysis because they're
not supported by the BE yet.
Change-Id: I734575eb87c909d09d24c4df028023f3b50d3cb5
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4442
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Matthew Jacobs <mj@cloudera.com>
- determine the partition group that has maximal intersection of its partition exprs with the
preceding grouping exprs
- if that intersection's expected ndv > #nodes, make that partition group the first one in the sequence
to be computed and reduce the hash partition of the preceding aggregation to that intersection
Change-Id: I612b4a260a8975deb495e5d34c32f03db4a7cca7
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4451
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: jenkins
Fixed a bug when setting the length in reading/write text files for CHAR(N).
Also added chars_tiny table for testing CHAR(N) and VARCHAR(N).
Change-Id: If5d5db30afa4b00cf03c68c6a845f182970329f4
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4415
Reviewed-by: Victor Bittorf <victor.bittorf@cloudera.com>
Tested-by: jenkins
min()/max() do not currently support windows without UNBOUNDED PRECEDING,
so this changes AnalyticExpr to detect this during analysis and throw an
AnalysisException.
Also removed some stale TODOs in the BE
Change-Id: I734b0a5d5399f9bb9d4db6ab1ddc079237b0ac03
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4431
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Like left/right outer joins, anti joins have a uni-directional value transfer.
Predicates could be pushed into anti joined plan subtrees if the condition
was inverted, but this patch does not implement this optimization.
No special consideration must be made to prevent predicate assignment
into anti-joined branches because anti-joined tuples are invisible outside
of the On-clause, and therefore, all unassigned conjuncts referencing the
invisible tuple must come from the original join's On-clause. The assignment
of such predicates is already handled correctly.
Change-Id: Ic2b94f6eb57e000ea51e253035e713288b205298
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4425
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
This patch restricts the leftmost table ref candidates of cross joins to the
very first join (like we already do for outer/semi joins). Join inversion
is still considered for cross joins.
While conceptually possible, it is tricky to reason about allowing the rhs of
arbitrary cross-join table refs as the leftmost candidate during join
re-ordering. We would have to carefully change the joinOps of all table refs
in between, and ensure to not make those changes in place to avoid "polluting"
the table refs for the next round of join re-ordering (considering a new
leftmost table ref). The safer fix is to restrict the considered orders.
Change-Id: I5fdc323e4a9c2dada06d9aec81769057f7076299
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4438
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
This commit fixes the issue (IMPALA-1215) where NOT IN subqueries return
wrong results in the presence of NULL values. The null-matching equality
operator is introduced in the front-end and the NOT IN subqueries are
rewritten using the null-aware anti-join operator.
Change-Id: I5a323357025d77c2143db86e1057999ec8a371c0
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4391
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: jenkins
- order partition and sort groups by increasing output tuple size
- the unpartitioned partition group goes last
- if the unpartitioned partition group contains an order-by, that is executed like
a regular distributed sort (with a merging exchange)
- coalesce sort groups that have a compatible partition-by clause and one is a prefix
of the other
Still missing:
- ordering comparison should take equivalence classes into account
Change-Id: Ie604c74f7804a9028f3ab59ce0c291deb0edb272
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4399
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
1) Fix ROWS following start bound where window is never fully in partition
2) Fix sum() NULL handling over sliding windows and add/consolidate tests.
sum() should return NULL when all non-NULL values are removed. Because
sum only stores the current sum as the intermediate value, we can't know
if the sum is actually 0 or if there are no non-NULL values in the window.
(avg() doesn't have this problem because it explicitly keeps the count
of the number of elements in the average as part of the intermediate state.)
Instead of changing sum() to have more intermediate state (which would
affect aggregations), we can just keep track of the number of calls to
Update() and Remove() in the FunctionContextImpl and check in SumRemove()
whether or not there are any non-null elements being summed. Added
tests (verified with Oracle).
3) Fixed a bug where the state tracking the last result tuple could be
wrong and resulting in a crash.
4) IMPALA-1269: Windows between a start offset to CURRENT ROW might could
produce wrong results between partitions.
5) IMPALA-1273: Incorrect results with very large window and small table
Tests are included for all issues.
Change-Id: I0f396c24078a1494fb977e8775f1ca8c530932eb
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4397
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
The previous code did not handle well the case where the spilling happens when
building the hash table (i.e. partitioning the build rows fit). This caused the
probe partition to be starved causing queries that should be able to run to fail
with a not enough buffers error.
Change-Id: I3a9a84e8800a72ed3ce6f5ab7ff03bc2d6eb7ad8
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4403
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: Nong Li <nong@cloudera.com>
This patch addresses:
1. Char doesn't use codegen
2. Not in-lining large CHAR(N) for N > 128
3. Parquet reader/writer for CHAR(N) and VARCHAR(N)
Change-Id: I83a29a8bd312841a3e29bfe2243884074570f247
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4280
Reviewed-by: Victor Bittorf <victor.bittorf@cloudera.com>
Tested-by: jenkins
This patch also fixes IMPALA-1164: NDV() now returns a BIGINT (and not STRING).
Change-Id: Ia2a3272204938579d61091ee4f7f2d1cbf38ed55
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4338
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Prevent predicate propagation into inline views containing analytic exprs by
a) not explicitly pushing predicates into such inline views
b) blocking the propagation path by not registering auxiliary eq predicates for the inline view
Change-Id: Ie2961d36a532f6b4603a4edbe7dd9cf0a6882d75
Adds support in the BE AnalyticEvalNode for ROWS windows with arbitrary
start bounds. If there is a start bound specified a sliding window must
be maintained. As input rows are processed they are added to the window.
As they expire from the window, they are 'removed' from the current
intermediate state of the evaluators (stored in curr_tuple_) by calling
AggFnEvaluator::Remove(). This is an initial implementation that keeps
the tuples in the window in memory. We can improve this later by using
the BufferedTupleStream with an Iterator interface supporting multiple
readers.
This also fixes IMPALA-1253: LAST_VALUE returns incorrect results
Change-Id: Id5daf6c060ab4079bb8dacf2db8992985894a820
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4335
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Matthew Jacobs <mj@cloudera.com>
Queries like;
INSERT INTO table VALUES (CAST("..." AS CHAR(N)))
Used codegen path and failed; changed to use interpreted path.
Change-Id: Id80274580df268b3f828dec19a2e0b0578061ca8
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4362
Reviewed-by: Victor Bittorf <victor.bittorf@cloudera.com>
Tested-by: jenkins
This commit fixes an issue where, for the case of a scalar agg subquery
with count, the zeroifnull fn is applied to the wrong column during
query rewrite.
This commit also fixes IMPALA-1237.
Change-Id: Ic00ae5799d6970171c007e3ed25348d7fc09d825
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4286
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: jenkins
This patch fixes two issues:
- Add API to buffered block mgr to allow an atomic Unpin and GetNewBlock. This has
the semantics of unpinning a block and giving the buffer to the new block. This
is necessary for the tuple stream to make sure another thread does not grab the
unpinned block in between.
- Buffer management reading an unpinned stream. Before moving onto a new block (and
unpinning the current), we need to make sure all the tuples returned from the
current block are returned up the operator tree.
Change-Id: I95ee58d1019dd971f6a7dc19ecafdfa54cdbf942
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4333
Tested-by: jenkins
Reviewed-by: Nong Li <nong@cloudera.com>
We weren't returning the right merge function for decimal in
GetAvgFunction(). Someday the functions will be registered in the FE
like for scalar functions.
Change-Id: I1153ef8570b0e78f0925b7d3d58ec3b0fbb2c589
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4336
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
This patch also does some clean up on agg-fn-evaluator to support this case.
Change-Id: If7e5c8663c7d371b2666acaab2966a2cd7bdccf9
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4256
Tested-by: jenkins
Reviewed-by: Nong Li <nong@cloudera.com>
Adds support for displaying all or a subset of the privileges granted to a role. Users
have privileges to execute this statement if they are already granted the role or if
they are an admin user on the Sentry Policy Service. The output includes:
* The target scope of the privilege
* The privilege level
* The target names in the object hierarchy
* Whether the privilege was granted using WITH GRANT OPTION
* The create time of the privilege
Examples:
-- Show all grants in role1
SHOW GRANT ROLE role1
-- Shows all grants in role1 on the database foo
SHOW GRANT ROLE role1 on DATABASE foo
Output looks like:
+----------+------------+-------+-----+-----------+--------------+-------------------------------+
| scope | database | table | uri | privilege | grant_option | create_time |
+----------+------------+-------+-----+-----------+--------------+-------------------------------+
| DATABASE | functional | | | ALL | false | Fri, Sep 19 2014 16:13:40.999 |
+----------+------------+-------+-----+-----------+--------------+-------------------------------+
Change-Id: I8ef1b87a4c22c8fba4228012668033d7f9d06fcb
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4389
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
This change adds support for GRANT <privilege> TO <role> WITH GRANT OPTION which allows
delegating GRANT/REVOKE authority to non-admin users. Specifically, it allows users who
have been granted the specified role to execute GRANT/REVOKE statements on all child
objects. For example, you can now do something like:
GRANT ALL ON DATABASE foo TO role1 WITH GRANT OPTION
and everyone granted role1 will be able to execute GRANT/REVOKE statements on database
foo OR any of the tables in the database.
It also adds support for REVOKE GRANT OPTION FOR <privilege> FROM <role> which allows
removing a previous WITH GRANT OPTION without actually deleting the privilege.
Similar to GRANT/REVOKE statements, the actual authorization checks on whether a user
should/should not have privileges to execute these options is done at the Sentry Service
level.
Change-Id: I8757569a3bdb68414e315ef37d6845b1859eb758
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4377
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
This patch adds the necessary changes required to authorize SHOW ROLES statements.
This is not as easy as it could be because the Sentry Service doesn't currently
expose the metadata for who is/isn't authorized to execute these statements. To authorize
the statements, we need to first make an RPC to the Sentry Service (via the
Catalog Server) and then only proceed with the SHOW statement if the check succeeds.
We should consider revisiting this approach in the future when more metadata is available
from Sentry.
Additionally, this patch adds support for SHOW CURRENT ROLES which shows all roles
that are currently granted to the current user.
Change-Id: Ia01c20d58ab081f49a85566075836d8c6e25dbd4
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4367
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
This patch adds support for V6 of the HS2 protocol, which notably
includes columnar organisation of result sets. Clients that set their
protocol version to < V6 will receive result sets in the traditional row
orientation.
The performance of fetches over HS2 goes up significantly as a result,
since the V1 protocol had some pathologies in its deserialisation
performance.
Beeswax
Row materialisation: 455ms, client processing time: 523ms
HS2 V6:
Row materialisation: 444ms, client processing time: 1.8s
HS2 V1:
Row materialisation: 585ms, client processing time: 15.9s (!)
TODO: Add support for the CHAR datatype
The following patch is also included:
Fix wait-for-hiveserver2.py when Impala moves to HS2 V6
Due to HIVE-6050, older versions of Hive are not compatible with newer
clients (even those that try to use old protocol
versions). wait-for-hiveserver2.py uses HS2 to talk to the HiveServer2
service, but picks up the newer version from V6, and fails.
This patch temporarily re-adds cli_service.thrift (renaming the Thrift
service as LegacyTCLIService) only for wait-for-hiveserver2.py to
use. As soon as Impala's thirdparty Hive moves to HS2 V6, we can get rid
of this change.
Change-Id: I2cbe884345ae7e772620b80a29b6574bd6532940
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4402
Tested-by: jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
Prior to this work, the impalad could either authenticate with
Kerberos, or authenticate with LDAP. This fixes that so that both can
co-exist in the same daemon. Prior code had both a
KerberosAuthProvider and an LdapAuthProvider; this is refactored into
a single SaslAuthProvider that potentially contains both LDAP and
Kerberos.
The terminology of "client facing" and "server facing" has been
replaced with "external" and "internal". External is for clients like
the impala shell, odbc, jdbc, etc. Internal is for daemon <-> daemon
communication.
The notion of the "auxprop" plugin is removed, as that was dead code.
The Thrift code is enhanced to pass the Realm information from the
SaslAuthProvider down to the underlying SASL library.
Change-Id: I0a0b968a107c0b25610ca37295c3fee345ecdd6d
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4051
Reviewed-by: Michael Yoder <myoder@cloudera.com>
Tested-by: jenkins
This patch adds two new hint styles:
1. Traditional commented hint: /* +hint1,hint2,hint3 */
2. End-of-line commented hint: -- +hint1,hint2,hint3\n
We now preserve hints when creating views. We always use the
end-of-line commented hint style to allow Hive to read
hinted views created by Impala. Hive does not support
traditional /* */ comments, and attempts to parse /*+ */ as
hints, failing with a parse error on unrecognized hints.
This patch also changes Impala to only issue a warning
for unrecognized hints instead of throwing an error. This
allows Impala to run against hinted views created by Hive.
Change-Id: I6e8352442e763c0029f72c17363caa087572dca0
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4235
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4361
Changes needed for PHJ to support RIGHT {SEMI, ANTI} JOINs. Codegen works as well.
Basic parser tests and minimal (end-to-end) query tests.
Need to add analyzer tests and add more query tests.
Note that in the case of right-{semi,anti} and perhaps also on {right,full}-outer we
should not be broadcasting the build side.
Change-Id: I6854ee9e4640f809f0350229bcc00811fa474f07
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4288
Reviewed-by: Ippokratis Pandis <ipandis@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4369
By including the actual FunctionCallExpr as a child of an AnalyticExpr, we would end
up substituting it if it also happened to show up as a regular/non-analytic aggregate
expr in the query. The solution is only to include the function call parameters as
children of the AnalyticExpr, not the function call itself.
Plus, fixed up partition-by/order-by less-than predicates in AnalyticPlanner.
Change-Id: Ib5d33baa2f257f7a8a21dd536332d11c55fbdbca
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4327
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4342
This change adds support for GRANT/REVOKE to Impala via the Sentry Service. This includes
support for creating and dropping roles, granting and revoking roles to/from groups,
granting/revoking privileges to/from roles, and commands to view role metadata.
The specific statements that are added in this patch are:
CREATE/DROP ROLE <roleName>
SHOW ROLES
SHOW ROLE GRANT GROUP <groupName>
GRANT/REVOKE ROLE <roleName> TO/FROM GROUP <groupName>
GRANT/REVOKE <privilegeSpec> TO/FROM <roleName
It does not include some of the fancier bulk-op syntax like support for granting multiple
roles to multiple groups in one statement.
This patch does not add support for the WITH GRANT OPTION to delegate GRANT/REVOKE
privileges to other users.
TODO:
* Authorize these statements on the client side. The current Sentry Service design makes
it difficult to authorize any GRANT/REVOKE statement on the client (Impala) side.
Privilege checks are done within the Sentry Service itself. There are a few different
options available to let Impala "fail fast" and those changes will come in a follow
on patch.
Change-Id: Ic6bd19f5939d3290255222dcc1a42ce95bd345e2
Adds agg fns for FIRST_VALUE, LAST_VALUE, LAG, LEAD. Also adds
support for ROWS windows with the end bound as unbounded following
as long as the start bound is unbounded preceding.
Change-Id: I4856ae580164d17a1bbf7d45010b61f5afa5db50
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4249
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
IMPALA-1233: Crash running query with analytic in WITH clause
IMPALA-1232: Analytic eval node crashes if cancelling query before Open()
Change-Id: I9a263775b8ef670d0f819ed53d0af1eb96edf5c7
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4313
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
Support for CHAR is implemented as a StringVal in the backend.
TODO:
1. Parquet Reader/writer
2. Codegen slot ref
3. Codegen text reader
4. Don't inline large chars
5. update impala-hs2-server.cc with CHAR support
Change-Id: Ibba2c89cea971cb740001ea7975bf3e929150471
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4075
Reviewed-by: Nong Li <nong@cloudera.com>
Reviewed-by: Victor Bittorf <victor.bittorf@cloudera.com>
Tested-by: jenkins
This commit fixes the issue where an error was thrown if a subquery was
used in either side of a between predicate. Between predicates with
subqueries are replaced by their corresponding compound predicates
during query rewrite.
Change-Id: I4315a6e91c9306c6817bf6aa6bc1d0b586a1a067
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4246
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: jenkins
This also reverts back to using CRC hash since FNV is not codegen'd
yet. The perf is not as good as the original HJ in a microbenchmark; I
haven't run a cluster run yet.
Change-Id: Ie4dc983f31631fbc78720425a0e354dd1d3342a6
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4219
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
