Commit Graph

51 Commits

Author SHA1 Message Date
Dimitris Tsirogiannis
3db5ced4ce IMPALA-3726: Add support for Kudu-specific column options
This commit adds support for Kudu-specific column options in CREATE
TABLE statements. The syntax is:
CREATE TABLE tbl_name ([col_name type [PRIMARY KEY] [option [...]]] [, ....])
where option is:
| NULL
| NOT NULL
| ENCODING encoding_val
| COMPRESSION compression_algorithm
| DEFAULT expr
| BLOCK_SIZE num

The output of the SHOW CREATE TABLE statement was altered to include all the specified
column options for Kudu tables.

Change-Id: I727b9ae1b7b2387db752b58081398dd3f3449c02
Reviewed-on: http://gerrit.cloudera.org:8080/5026
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: Internal Jenkins
2016-11-18 11:41:01 +00:00
Matthew Jacobs
77a2941a42 IMPALA-3713,IMPALA-4439: Fix Kudu DML shell reporting
Adds support in the shell to report the number of modified
rows for all DML operations, as well as the number of rows
with errors.

Testing: Added shell tests.

Change-Id: I3d3d7aa8d176e03ea58fb00f2a81fb3e34965aa1
Reviewed-on: http://gerrit.cloudera.org:8080/5103
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
2016-11-17 04:13:25 +00:00
Lars Volker
ef4c9958d0 IMPALA-4047: Remove occurrences of 'CDH'/'cdh' from repo
This change removes some of the occurrences of the strings 'CDH'/'cdh'
from the Impala repository. References to Cloudera-internal Jiras have
been replaced with upstream Jira issues on issues.cloudera.org.

For several categories of occurrences (e.g. pom.xml files,
DOWNLOAD_CDH_COMPONENTS) I also created a list of follow-up Jiras to
remove the occurrences left after this change.

Change-Id: Icb37e2ef0cd9fa0e581d359c5dd3db7812b7b2c8
Reviewed-on: http://gerrit.cloudera.org:8080/4187
Reviewed-by: Jim Apple <jbapple@cloudera.com>
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
2016-10-13 00:40:41 +00:00
Sailesh Mukil
c23bf38a20 IMPALA-3893, IMPALA-3901: impala-shell prints incorrect coordinator address, overly verbose
The webserver address was always configured as 0.0.0.0 (meaning that
the webserver could be reached on any IP for that machine) unless
otherwise specified. This is not a correct value to dispay to the
user. This patch returns the hostname of the node, when requested,
if the webserver host address is 0.0.0.0.

This patch also does not print the coordinator link for very simple
queries, as it's not necessary and is unnecessarily verbose.

This patch also does away with pinging the impalad an extra time per
query for finding the host time and webserver address. It instead
remembers the webserver address at connect time and displays client
local time for every query instead.

Change-Id: I9d167b66f2dd8629e40a7094d21ea7ce6b43d23b
Reviewed-on: http://gerrit.cloudera.org:8080/3994
Tested-by: Internal Jenkins
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Sailesh Mukil <sailesh@cloudera.com>
2016-08-23 18:25:06 +00:00
Tim Armstrong
50e21247d6 IMPALA-3992: bad shell error message when running nonexistent file
Fix the error handling code and add a test.

Change-Id: Iebcf1dc8a1a08b400a2c769a9cff38ea02c8e525
Reviewed-on: http://gerrit.cloudera.org:8080/4022
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: Internal Jenkins
2016-08-18 03:37:48 +00:00
Dan Hecht
ffa7829b70 IMPALA-3918: Remove Cloudera copyrights and add ASF license header
For files that have a Cloudera copyright (and no other copyright
notice), make changes to follow the ASF source file header policy here:

http://www.apache.org/legal/src-headers.html#headers

Specifically:
1) Remove the Cloudera copyright.
2) Modify NOTICE.txt according to
   http://www.apache.org/legal/src-headers.html#notice
   to follow that format and add a line for Cloudera.
3) Replace or add the existing ASF license text with the one given
   on the website.

Much of this change was automatically generated via:

git grep -li 'Copyright.*Cloudera' > modified_files.txt
cat modified_files.txt | xargs perl -n -i -e 'print unless m#Copyright.*Cloudera#i;'
cat modified_files_txt | xargs fix_apache_license.py [1]

Some manual fixups were performed following those steps, especially when
license text was completely missing from the file.

[1] https://gist.github.com/anonymous/ff71292094362fc5c594 with minor
    modification to ORIG_LICENSE to match Impala's license text.

Change-Id: I2e0bd8420945b953e1b806041bea4d72a3943d86
Reviewed-on: http://gerrit.cloudera.org:8080/3779
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
2016-08-09 08:19:41 +00:00
Taras Bobrovytsky
609b80410e Clean up Python test import statements
Many of our test scripts have import statements that look like
"from xxx import *". It is a good practice to explicitly name what
needs to be imported. This commit implements this practice. Also,
unused import statements are removed.

Change-Id: I6a33bb66552ae657d1725f765842f648faeb26a8
Reviewed-on: http://gerrit.cloudera.org:8080/3444
Reviewed-by: Michael Brown <mikeb@cloudera.com>
Tested-by: Internal Jenkins
2016-07-15 23:26:18 +00:00
Sailesh Mukil
900f148078 IMPALA-1671: Print time and link to coordinator web UI once query is submitted in shell
To help supportability and debugging, it's helpful to have the impala
shell print out the coordinator time and the link to the coordinator
web UI once the query is submitted.

This is done by calling the PingImpalaService() routine everytime a
query is submitted, which returns the coordinator's hostname,
webserver port and the coordinator epoch time at that moment which the
shell then formats and prints out.

Added tests to verify these new messages.

Change-Id: I704eb64546e27c367830120241311fea6091266b
Reviewed-on: http://gerrit.cloudera.org:8080/3507
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Internal Jenkins
2016-07-14 19:04:45 +00:00
Henry Robinson
0dde1c2f86 IMPALA-3628: Fix cancellation from shell when security is enabled
To cancel a query, the shell will create a separate connection inside
it's SIGINT handler, and send the cancellation RPC. However this
connection did not start a secure connection if it needed to, meaning
that the cancellation attempt would just hang.

A workaround is to kill the shell process, which I expect is what users
have been doing with this bug which has been around since 2014.

Testing:

I added a custom cluster test that starts Impala with SSL
enabled, and wrote two tests - one just to check SSL connectivity, and
the other to mimic the existing test_cancellation which sends SIGINT to
the shell process. In doing so I refactored the shell testing code a bit
so that all tests use a single ImpalaShell object, rather than rolling
their own Popen() based approaches when they needed to do something
unusual, like cancel a query.

In the cancellation test on my machine, SIGINT can take a few tries to
be effective. I'm not sure if this is a timing thing - perhaps the
Python interpreter doesn't correctly pass signals through to a handler
if it's in a blocking call, for example. The test reliably passes within
~5 tries on my machine, so the test tries 30 times, once per second.

Change-Id: If99085e75708d92a08dbecf0131a2234fedad33a
Reviewed-on: http://gerrit.cloudera.org:8080/3302
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: Henry Robinson <henry@cloudera.com>
2016-07-05 16:40:23 -07:00
David Knupp
c076f098d4 IMPALA-3491: Use unique_database fixture in test_shell_commandline.py.
Before this change, a single test database was created for the entire suite,
and each test was marked to run serially. With the addition of a test fixture
in tests/conftest.py to create a unique database per each individual method,
it's possible now to run the tests in parallel. (The tables required by
individual tests are created via local test fixtures.)

As such, any methods which had been responsible for setting up the test
database were removed. Pytest markers for running tests serially were also
removed, except in cases where interactions from running concurrency would
affect other tests.

Additional minor changes were made to improve PEP-8 compliance.

The non-serial tests were run in a loop ten times to confirm that there weren't
any unexpected failures.

Review: https://gerrit.cloudera.org/#/c/3301/

Change-Id: Icdcb04a99c0907fc1ba56baa2497fafb33b0e34e
Reviewed-on: http://gerrit.cloudera.org:8080/3301
Reviewed-by: David Knupp <dknupp@cloudera.com>
Tested-by: Internal Jenkins
2016-06-13 18:32:32 -07:00
Sailesh Mukil
ed7f5ebf53 IMPALA-1878: Support INSERT and LOAD DATA on S3 and between filesystems
Previously Impala disallowed LOAD DATA and INSERT on S3. This patch
functionally enables LOAD DATA and INSERT on S3 without making major
changes for the sake of improving performance over S3. This patch also
enables both INSERT and LOAD DATA between file systems.

S3 does not support the rename operation, so the staged files in S3
are copied instead of renamed, which contributes to the slow
performance on S3.

The FinalizeSuccessfulInsert() function now does not make any
underlying assumptions of the filesystem it is on and works across
all supported filesystems. This is done by adding a full URI field to
the base directory for a partition in the TInsertPartitionStatus.
Also, the HdfsOp class now does not assume a single filesystem and
gets connections to the filesystems based on the URI of the file it
is operating on.

Added a python S3 client called 'boto3' to access S3 from the python
tests. A new class called S3Client is introduced which creates
wrappers around the boto3 functions and have the same function
signatures as PyWebHdfsClient by deriving from a base abstract class
BaseFileSystem so that they can be interchangeably through a
'generic_client'. test_load.py is refactored to use this generic
client. The ImpalaTestSuite setup creates a client according to the
TARGET_FILESYSTEM environment variable and assigns it to the
'generic_client'.

P.S: Currently, the test_load.py runs 4x slower on S3 than on
HDFS. Performance needs to be improved in future patches. INSERT
performance is slower than on HDFS too. This is mainly because of an
extra copy that happens between staging and the final location of a
file. However, larger INSERTs come closer to HDFS permformance than
smaller inserts.

ACLs are not taken care of for S3 in this patch. It is something
that still needs to be discussed before implementing.

Change-Id: I94e15ad67752dce21c9b7c1dced6e114905a942d
Reviewed-on: http://gerrit.cloudera.org:8080/2574
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Internal Jenkins
2016-05-12 14:17:49 -07:00
Anuj Phadke
a915293109 IMPALA-1850: Allow fs.defaultFS to be set to a non-HDFS filesystem
This change whitelists the supported filesystems which can be set
as Default FS for Impala to run on.
This patch configures Impala to use S3 as the default filesystem, rather
than a secondary filesystem as before.

Change-Id: I2f45bef6c94ece634045acb906d12591587ccfed
Reviewed-on: http://gerrit.cloudera.org:8080/1121
Reviewed-by: anujphadke <aphadke@cloudera.com>
Tested-by: Internal Jenkins
2016-05-12 14:17:40 -07:00
Andre Araujo
f3733aed84 IMPALA-2180: Extend SET command to allow setting variables in Impala Shell.
The SET command has been extended with the following syntax, to allow
setting of variables in the Impala Shell:

SET VAR:<variable_name>=<value>

The UNSET command has also been modified to allow:

UNSET VAR:<variable_name>

This patch builds on the changes in IMPALA-2179. The main change for
this patch was to ensure that all SET commands are processed by the
shell, rather than being send to the front end as a query. For this
I had to modify the command sanitization function to remove comments
that happen in front of a SET command.

Comments can be a can of worms to parse, so I tried to be as strict
as possible to avoid collateral effects. Comments are only removed
if they happen right at the beginning of the line AND before a SET
command. NO other comments are touched, including comments before,
after or within queries.

Change-Id: I87e07385122187ab8d324346499896a3dfbbafe6
Reviewed-on: http://gerrit.cloudera.org:8080/679
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
2016-02-10 10:17:18 +00:00
Andre Araujo
bcce19012d IMPALA-2179: Extend Impala shell to allow passing variables through the command line
This patch adds the command line option `--var` to allow the user to set
variable to be used in commands within the shell. It does *not* implement the
setting of variables through the SET command, as Hive does. This extension will
be implemented separately on IMPALA-2180.

The syntax for specifying a parameter in the command line is --var=KEY=VAL, as
for example: --var=start_date=20150101

Variables are textually replaced by their value in the Impala shell commands.
The substitution work similarly for interactive sessions as well as for command
line queries and/or scripts (-q and -f options, respectively).

Variables can be referenced as ${VAR:VAR_NAME} (case-insensitive). The form
${HIVEVAR:VAR_NAME} can also be used for compatibility with Hive scripts.

To prevent any of the reference expressions above from being replaced you can
escape them with a backslash (e.g. \${VAR:VAR_NAME} and \${HIVEVAR:VAR_NAME}).

The Impala shell's SET command now also reports the set variables and their
values.

Change-Id: Ia491fae91256334bb60c9066d119fe9a1e9779dd
Reviewed-on: http://gerrit.cloudera.org:8080/611
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
2016-02-10 00:20:46 +00:00
Alex Behm
71694b7b42 IMPALA-2221: Fix the flaky test test_shell_commands.test_queries_closed.
The bug was that the test sometimes polled the wrong impalad for checking
metrics because ImpaladCluster() does not guarantee any particular order of
impalads, but the test was incorrectly relying on that.

The fix is to execute the shell commands on a fixed impalad, and use the
same impalad for polling metrics.

Change-Id: Iecb528be916d2900d7ec8a894bdef630250547da
Reviewed-on: http://gerrit.cloudera.org:8080/1974
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
2016-02-02 06:32:05 +00:00
Henry Robinson
cca964c3c6 IMPALA-1934: Allow shell to retrieve LDAP password from shell cmd
Adds a new option --ldap_password_cmd that takes a string which is
executed as a shell command. The stdout results are used as the LDAP
password for this shell session.

Tests are added for the negative case (where the command fails for some
reason), but without tests for successful LDAP connections we can't test
the case where the password is correct.

Change-Id: Ib0362be5e167ff752e764ad2152c4c4b679f83c2
Reviewed-on: http://gerrit.cloudera.org:8080/1542
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: Internal Jenkins
2016-01-19 23:41:25 +00:00
Casey Ching
d202d6a967 Use "impala-python" (virtualenv) instead of system python
Python tests and infra scripts will now use "python" from the virtualenv
via $IMPALA_HOME/bin/impala-python. Some scripts could be simplified now
that python 2.6 and a dependable set of third-party libraries are
available but that is not done as part of this commit.

Change-Id: If1cf96898d6350e78ea107b9026b12ba63a4162f
Reviewed-on: http://gerrit.cloudera.org:8080/603
Reviewed-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com>
Tested-by: Internal Jenkins
2015-08-06 02:09:09 +00:00
Henry Robinson
621205ebbc IMPALA-2143: Avoid sending auth credentials over insecure connections
This patch changes the behaviour of the Impala shell to refuse to
attempt an LDAP-authenticated connection to Impala unless SSL/TLS is
configured.

A new flag --auth_creds_in_clear_ok is added to suppress this
behaviour. This is similar to Impala's --ldap_passwords_in_clear_ok
flag. The shell will also now print a warning if an insecure
configuration is used.

Change-Id: Ide25d8dd881a61b9f08900112466c430da64a038
Reviewed-on: http://gerrit.cloudera.org:8080/546
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
2015-07-30 07:15:29 +00:00
Casey Ching
23eec9fc30 Simplify shell cancellation tests
The tests were doing unnecessary things. One such thing that stopped
working with the virtualenv patch was searching for the shell process to
get the pid. The search was never needed since the process was spawned
with Popen which provides the pid directly.

Change-Id: I2455e58de4fdba8fd2770f0489fac8cddf6b90a0
Reviewed-on: http://gerrit.cloudera.org:8080/555
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
2015-07-23 04:09:11 +00:00
Sailesh Mukil
42d118a8ad IMPALA-1413: Allow impala-shell -f option to read from stdin
The '-f' option in Impala shell is used to read from a file containing SQL queries.
Now, additional support is added to read from STDIN by using:

"-f -"

An additional bug was fixed in the test script test_shell_commandline.py where certain
tests(test_default_db and test_unsecure_message) would hang indefinitely due to the
subprocess(impala-shell) waiting for user input. Fixed by piping STDIN to the subprocess
which sends an implicit EOF that closes the impala-shell once the test is completed.

Change-Id: I9a2682e086a3345e089f3e9db7cc049ce3d2c19a
Reviewed-on: http://gerrit.cloudera.org:8080/479
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: Internal Jenkins
2015-06-23 23:53:52 +00:00
casey
554f1f779b Shell: Fix parsing of strings containing an escaped backslash
Upgrading sqlparse ended up trading one bug for another. The new bug is
not fixed upstream, I sent a patch. The problem is '\\' is not
considered a terminated string and we use this in the phrase "fields
escaped by '\\'" when creating tables.

Change-Id: Id57081f5a96e997afd3aa9b26dca23f627488fc3
Reviewed-on: http://gerrit.cloudera.org:8080/117
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
2015-02-27 03:17:28 +00:00
ishaan
b8f01c315a Fix the shell's runtime profile test to be more robust.
This patch uses a regex rather than a raw string to search for unique query profiles in
the shell's output.

Change-Id: I5c0c2dc30ad6d9a14e5c35177f5a494445b7cc7c
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/5381
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: jenkins
2014-11-23 21:34:55 -08:00
ishaan
b7aa5440a7 Disable the get_log_once test in the shell.
Will be re-enabled when the fix for getting the whole warning log goes in.

Change-Id: If0209dd45038d36a6a3b2d83da88662ae2cf74c5
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4592
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: Ishaan Joshi <ishaan@cloudera.com>
2014-10-06 19:49:18 -07:00
Abdullah Yousufi
f11de455a0 Fix return status of summary within impala shell
Follow up fix to IMPALA-1153. Ensure that the correct
CmdStatus is returned by the summary command. (ERROR for
invalid queries and SUCCESS even if summary is not available.)

Change-Id: Icf67164dc82f202ec15071541f6ed3b26e3ad7fb
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4089
Reviewed-by: Abdullah Yousufi <abdullah.yousufi@cloudera.com>
Tested-by: jenkins
2014-09-26 11:00:15 -07:00
Abdullah Yousufi
9853f74db1 CDH-21036: Make --quiet flag suppress warning messages in impala-shell
Change-Id: Ia57652840b3e3483a06c0bbef283c60a229db27f
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4092
Reviewed-by: Abdullah Yousufi <abdullah.yousufi@cloudera.com>
Tested-by: jenkins
2014-09-26 11:00:05 -07:00
Mike Yoder
75a97d3d7e [CDH5] Kerberize mini-cluster and Impala daemons
This is the first iteration of a kerberized development environment.
All the daemons start and use kerberos, with the sole exception of the
hive metastore.  This is sufficient to test impala authentication.

When buildall.sh is run using '-kerberize', it will stop before
loading data or attempting to run tests.

Loading data into the cluster is known to not work at this time, the
root causes being that Beeline -> HiveServer2 -> MapReduce throws
errors, and Beeline -> HiveServer2 -> HBase has problems.  These are
left for later work.

However, the impala daemons will happily authenticate using kerberos
both from clients (like the impala shell) and amongst each other.
This means that if you can get data into the mini-cluster, you could
query it.

Usage:
* Supply a '-kerberize' option to buildall.sh, or
* Supply a '-kerberize' option to create-test-configuration.sh, then
  'run-all.sh -format', re-source impala-config.sh, and then start
  impala daemons as usual.  You must reformat the cluster because
  kerberizing it will change all the ownership of all files in HDFS.

Notable changes:
* Added clean start/stop script for the llama-minikdc
* Creation of Kerberized HDFS - namenode and datanodes
* Kerberized HBase (and Zookeeper)
* Kerberized Hive (minus the MetaStore)
* Kerberized Impala
* Loading of data very nearly working

Still to go:
* Kerberize the MetaStore
* Get data loading working
* Run all tests
* The unknown unknowns
* Extensive testing

Change-Id: Iee3f56f6cc28303821fc6a3bf3ca7f5933632160
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4019
Reviewed-by: Michael Yoder <myoder@cloudera.com>
Tested-by: jenkins
2014-09-05 12:36:21 -07:00
Abdullah Yousufi
a80506ff3c Refactored impala-shell
This is a reorganization of the existing impala-shell.

The basic idea was to split up the shell into two components: one part
soley responsible for the CLI functionality, and another to represent
the impala client/connection that would interact with the Beeswax api and
execute queries, fetch results, etc.

One major change was to redo how the existing shell handled cancellation,
which was to create a thread for each rpc, so that Ctrl+C would not interrupt
the system calls and break the socket connection. In the new approach,
a new client instance is created to close the query and if the socket connection is
broken, the client reconnects. Cancellation currently works.

Change-Id: I0f371f68552c065b2317f967c6cf7483b44be3df
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3316
Reviewed-by: Abdullah Yousufi <abdullah.yousufi@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4008
2014-08-22 20:13:04 -07:00
Abdullah Yousufi
bb853c47c7 IMPALA-755: Fix shell hanging due to escaped quotes
Commands with escaped single quotes would cause
the shell to enter an infinite loop while trying
to parse the command due to shlex not escaping single
quotes correctly. Once that change was implemented,
shlex would now ignore escaped single and double quotes
outside of closed quotes, so there needed to be a check for
that as well.

ALSO, implemented testing of commands in interactive mode.
Needed this to test these inputs, as command line input
cannot span multiple lines.

Change-Id: Id67368944eeb9a73061bc3e90bd6cda73c9d9f64
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3408
Reviewed-by: Abdullah Yousufi <abdullah.yousufi@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3893
2014-08-22 15:39:01 -07:00
Abdullah Yousufi
91d262a1d7 Include summary in profile when show_profiles option is enabled
Before, when the show_profiles -p option was enabled, the runtime profile
would be printed before the query was closed, preventing the query summary
from being printed. Now the profile is printed after the query is closed.

Change-Id: Icf7b10f7612d8016736aac70aa7b77265d391a98
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3770
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3821
Reviewed-by: Abdullah Yousufi <abdullah.yousufi@cloudera.com>
2014-08-18 16:10:09 -07:00
Abdullah Yousufi
504c83fe78 IMPALA-601: Read shell configuration from a file
Optionally loads options from a file in the user's home directory, called
'.impalarc', though the path to another file can be passed in as a command-line
option. The file must have a case-sensitive [impala] header. Specifying
the option in the command line overwrites the config file's value
for the option for that instance of the shell. If an option is not
specified in the config file, its default value is used.

Change-Id: I218da2c1e10308c5b8729883fa625f0c284397a7
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2956
Reviewed-by: Abdullah Yousufi <abdullah.yousufi@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3629
2014-07-29 22:14:50 -07:00
Abdullah Yousufi
6c1e272ef7 IMPALA-1059: Make backticking -d option argument idempotent
There was an issue with the previous fix to IMPALA-1059
if the user tried to reconnect within the shell after
having passed in a database via the -d option. The
passed database would be doubly backticked. This makes
the backticking of the argument idempotent.

Change-Id: I6eaed997c2be73d8659a2a12046ce393b97ec82c
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3467
Reviewed-by: Abdullah Yousufi <abdullah.yousufi@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3502
2014-07-15 18:10:40 -07:00
Abdullah Yousufi
864ed53511 IMPALA-1059: Backtick argument passed to USE by shell -d option
If not backticked, arguments such as parquet are interpreted as
keywords, when it is possible a database by that name exists.

This could have been avoided via single quotes around backticks: -d '`parquet`'
Otherwise, -d `parquet` throws a commandline error.

In interactive mode, backticks alone (ex. use `parquet`) will pass the
name as an identifier rather than a keyword.

Change-Id: I24b43eeeb6b4bfda5388165856788a20b64bc2ba
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3307
Reviewed-by: Abdullah Yousufi <abdullah.yousufi@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3500
2014-07-15 15:43:49 -07:00
Abdullah Yousufi
f4d1afe0ce IMPALA-921: Change EXPLAIN_LEVEL value from 0 to 1 in impala-shell for SET command
Change-Id: I2bfcefb5c8143d4cb4d74157c5309cd9445bac02
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3383
Reviewed-by: Abdullah Yousufi <abdullah.yousufi@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3499
2014-07-15 12:32:43 -07:00
ishaan
f262fcea64 Support utf-8 input and out in the shell
Also add --strict_unicode option which controls whether invalid unicode
code points should be ignored on input.

Change-Id: Ice59d6dd3df4557ab3b1fc91d7ddc0e1bf03f1c7
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3218
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
2014-07-02 23:18:27 -07:00
Alex Behm
ef6705d7e0 Rename MergeNode to UnionNode.
Change-Id: I9e3675a103757db1345b04bd1d102d2719efddd0
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3128
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3154
Tested-by: Alex Behm <alex.behm@cloudera.com>
2014-06-19 12:44:21 -07:00
Skye Wanderman-Milne
c3c9365c17 Change shell to print WARNINGS instead of ERRORS
Change-Id: I8b41a2f4307e31eda970ca891adb4f12fea926bb
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3088
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Skye Wanderman-Milne <skye@cloudera.com>
(cherry picked from commit 0a655f759d5096def89d2c72be5aa9a0cb2c10b1)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3149
2014-06-19 10:42:58 -07:00
Henry Robinson
d162571211 Fix 'summary' when exch map is not set
Change-Id: I66d9987f45f6cee045a300f86de357a2761929d7
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3000
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins
(cherry picked from commit 6f82cb296d0b3f0546d4e8a26485b79f20ff8996)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/3020
Tested-by: Henry Robinson <henry@cloudera.com>
2014-06-12 22:18:04 -07:00
Henry Robinson
9a7c6d286f Add 'summary' to shell
Users can now type 'summary' in the Impala shell after a query executes
to get a breakdown of the work done by each part of the query plan.

Change-Id: Ia6a43429ffc7778f3c2c8fcbf45d83828263c2ab
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2963
Tested-by: jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
(cherry picked from commit 9b98d42acb14d43a64832767528ee572eac4979b)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2995
2014-06-12 02:59:58 -07:00
ayousufi
66e90d75ee IMPALA-286: Display set query options in default section in impala-shell
Options displayed with 'set' command. Default values distinguished
from set values by square brackets.

Change-Id: Iacf0574555aab78aa0ba2008ceb8776d372a57a5
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2913
Reviewed-by: Abdullah Yousufi <abdullah.yousufi@cloudera.com>
Tested-by: jenkins
2014-06-11 11:51:19 -07:00
Skye Wanderman-Milne
76fa721cc4 Make shell only print error/warning log once.
Without this change, the shell would always print the error log twice
for successful non-insert queries (once in __execute_query() and once
in __fetch()).

Change-Id: I0ab038230df897559b30feaea34778ea72988bc3
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2815
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
(cherry picked from commit 095dba7f395491db03daf19ff3bff2e2b4640ee4)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2908
2014-06-09 13:40:14 -07:00
Taras Bobrovytsky
46aba6149d CDH-18512: Modification to allow spaces around the = sign in SET in impala-shell
Change-Id: I3c149e9a27962ed1130b1ddbb02952f4254bd4c9
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2609
Reviewed-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2645
2014-05-21 15:34:24 -07:00
Alex Behm
91db96d903 IMPALA-762: Add the query status to Beeswax::get_log() and pick it up in the Impala shell.
COMPUTE STATS is an async DDL command. When COMPUTE STATS fails it will set the
query status of the QueryExecState properly, but the original Beeswax::query() RPC
won't throw. The Impala shell sometimes did not pick up and display the
query status because no RPC actually threw. To fix this, I modified
Beeswax::get_log() to include the query status if it is not ok. The shell looks
for a special prefix to distinguish the query status from the runtime state error log.

Change-Id: I0d9dbf0801629a37de22ea4ebb6d2e5d53b836ef
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1899
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2063
2014-04-10 15:47:06 -07:00
Lenni Kuff
70c05d4caa IMPALA-897: shell does not close queries after completion when running from a script
The problem was that we were setting a flag marking the last_query_handle as closed, but
were not resetting the flag before the next query. This caused the first query to
be closed properly, but subsequent queries would not be closed. The fix is to change
where the flag is reset to the same place as where we assign last_query_handle.

Added a test case.

Change-Id: I870a96789489bfe4f388910b808409cd0584af8a
(cherry picked from commit 1439151af5b63112b0dd631fac9c7ab4d43bba37)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1976
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
2014-03-18 18:46:54 -07:00
ishaan
17761f1f5e IMPALA-571 The shell should be able to cancel a query during an rpc.
This change makes the fetch rpc interruptable. If the user cancels the query in
the middle of a fetch, the shell reconnects to the impalad and closes the
query. It also includes some code consolidation.

Change-Id: Iaaf0dfd4cba9ce2557e4a7d0447bc9c3ffda5e29
Reviewed-on: http://gerrit.ent.cloudera.com:8080/717
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: jenkins
2014-01-08 10:54:12 -08:00
ishaan
c0129a1683 Improve the Impala shell's behavior when attempting to connect to a keberized impalad.
This change has the following additions:
- If the user's connecting to a kerberized impalad, the Impala shell will check
  whether a valid ticket exists by running 'klist -s'. If a valid ticket is not found,
  then the shell will exit with an appropriate error message on the commandline.
- If the user's connecting to a kerberized impalad without the '-k' option, the Impala
  Shell will issue a 'klist -s' to check if there are valid kerberos tickets in the
  credentials cache. If a valid ticket is found, it will retry the connection with
  kerberos enabled.
- The Impala shell encodes strings entered on the commandline as unicode. The sasl
  module expects ascii strings as arguments. Explcitly encode any string sent to the
  sasl module to ascii.

Change-Id: I1799b1e7988a19fa513b683afe1e3b66b68c1ffc
Reviewed-on: http://gerrit.ent.cloudera.com:8080/535
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: Ishaan Joshi <ishaan@cloudera.com>
2014-01-08 10:53:02 -08:00
Henry Robinson
8cee9fa138 Fix failing test_shell_commandline
Change-Id: Iea170885f740ceeb08e21e64ef88ab44584fa270
Reviewed-on: http://gerrit.ent.cloudera.com:8080/545
Tested-by: jenkins
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
2014-01-08 10:53:01 -08:00
Skye Wanderman-Milne
efac6f82fd Print errors to shell in BaseSequenceScanner.
Change-Id: I0d1b041695c0d61b8c4994833f0a703e3bfa9c6a
Reviewed-on: http://gerrit.ent.cloudera.com:8080/278
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Skye Wanderman-Milne <skye@cloudera.com>
2014-01-08 10:52:20 -08:00
Lenni Kuff
039bfa475d Update impala-shell to print errors from RuntimeState after query completion 2014-01-08 10:51:47 -08:00
ishaan
77859a5262 Add shell tests for startup display messages for connections. 2014-01-08 10:51:31 -08:00
Lenni Kuff
5946801c73 Disable shell test case for kerberos flag 2014-01-08 10:51:28 -08:00