In Thrift 0.9.3 the TSSLSocket initializer TSSLSocket.__init__ prints
warnings if positional parameters are used. Change our usage of this
initializer to use named parameters.
Follow up work on "IMPALA-8333 Remove Impala Shell warnings part 2" will
remove one further warning message.
TESTING
Ran all end-to-end tests.
Added tests for the deprecation warnings to test_client_ssl.py.
Change-Id: I31f9a0bb12ca6a1da9129eacd29ac105b883e01b
Reviewed-on: http://gerrit.cloudera.org:8080/12837
Reviewed-by: Fredy Wijaya <fwijaya@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
test_ssl has a logic that waits for the number of in-flight queries to
be 1. However, the logic for wait_for_num_in_flight_queries(1) only
waits for the condition to be true for a period of time and does not
throw an exception when the time has elapsed and the condition is not
met. In other words, the logic in test_ssl that loops while the number
of in-flight queries is 1 never gets executed. I was able to simulate
this issue by making Impala shell start much longer.
Prior to this patch, in the event that Impala shell took much longer to
start, the test started sending the commands to Impala shell even when
Impala shell was not ready to receive commands. The patch fixes the
issue by waiting until Impala shell is connected. The patch also adds
assert in other places that calls wait_for_num_in_flight_queries and
updates the default behavior for Impala shell to wait until it is
connected.
Testing:
- Ran core and exhaustive tests several times on CentOS 6 without any
issue
Change-Id: I9805269d8b806aecf5d744c219967649a041d49f
Reviewed-on: http://gerrit.cloudera.org:8080/12047
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch fixes the issue with Ctrl+C handling for cancelling a
non-running query to behave similar to Linux shell.
Before (pressing Ctrl+C does not do anything):
[localhost:21000] default> select
After (pressing Ctrl+C cancels the query and starts a new prompt):
[localhost:21000] default> select^C
[localhost:21000] default>
Testing:
- Added a new cancellation test
- Ran all shell E2E tests
Change-Id: I80d7b2c2350224d88d0bfeb1745d9ed76e83cf6d
Reviewed-on: http://gerrit.cloudera.org:8080/11990
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch reverses the revert of IMPALA-7660.
The problem with IMPALA-7660 was that urllib.urlopen added the
'context' parameter in 2.7.9, so it isn't present on rhel7, which uses
2.7.5
The fix is to switch to using the 'requests' library, which supports
ssl connections on all the platforms Impala is supported on.
This patch also adds more info to the error message printed by
start-impala-cluster.py when the debug webserver cannot be reached yet
to help with debugging these issues in the future.
Testing:
- Ran full builds on rhel7, rhel6, and ubuntu16.
Change-Id: I679469ed7f27944f75004ec4b16d513e6ea6b544
Reviewed-on: http://gerrit.cloudera.org:8080/11625
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
A recent change (IMPALA-7519) added support for ecdh ciphers for the
beeswax/hs2 server. This patch pulls in a recent change on squeasel to
extend that support to the debug webserver.
It also fixes a bug that prevented start-impala-cluster.py from
completing successfully when the webserver is launched with ssl, due
to it trying to verify the availablitiy of the webserver over http.
Testing:
- Added a custom cluster test that verifies start-impala-cluster.py
runs successfully with webserver ssl enabled.
- Adds the webserver to an existing test for ecdh ciphers.
Change-Id: I80a6b370d5860812cde13229b5bcb2977814c73c
Reviewed-on: http://gerrit.cloudera.org:8080/11585
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
The fix for IMPALA-6990 had a bug, disabling some tests erroneously.
With this change, the tests run on Ubuntu16:04 like so:
tests/custom_cluster/test_client_ssl.py::TestClientSsl::test_ssl[] PASSED
tests/custom_cluster/test_client_ssl.py::TestClientSsl::test_tls_ecdh[] PASSED
tests/custom_cluster/test_client_ssl.py::TestClientSsl::test_tls_v12[] PASSED
tests/custom_cluster/test_client_ssl.py::TestClientSsl::test_wildcard_ssl[] xfail
tests/custom_cluster/test_client_ssl.py::TestClientSsl::test_wildcard_san_ssl[] xfail
The xfails are all "Inconsistent wildcard support on target platforms".
On centos7:
custom_cluster/test_client_ssl.py::TestClientSsl::test_ssl[] PASSED
custom_cluster/test_client_ssl.py::TestClientSsl::test_tls_ecdh[] SKIPPED
custom_cluster/test_client_ssl.py::TestClientSsl::test_tls_v12[] SKIPPED
custom_cluster/test_client_ssl.py::TestClientSsl::test_wildcard_ssl[] xfail
custom_cluster/test_client_ssl.py::TestClientSsl::test_wildcard_san_ssl[] xfail
On centos6:
custom_cluster/test_client_ssl.py::TestClientSsl::test_ssl[] PASSED
custom_cluster/test_client_ssl.py::TestClientSsl::test_tls_ecdh[] SKIPPED
custom_cluster/test_client_ssl.py::TestClientSsl::test_tls_v12[] SKIPPED
custom_cluster/test_client_ssl.py::TestClientSsl::test_wildcard_ssl[] SKIPPED
custom_cluster/test_client_ssl.py::TestClientSsl::test_wildcard_san_ssl[] SKIPPED
I used "curl --silent https://.../consoleText | grep test_client_ssl | sed -e 's/\[.*\]/[]/'"
to extract these from Jenkins output.
Change-Id: I64879b8af39f967b0059797e7b36421ce0e58bed
Reviewed-on: http://gerrit.cloudera.org:8080/11530
Reviewed-by: Philip Zeyliger <philip@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This is a temporary workaround. On the CentOS 6 build that failed
test_tls_v12, test_wildcard_san_ssl and test_wildcard_ssl were
all skipped so I figured this will unblock the tests without
losing coverage on most platforms that have recent Python.
Change-Id: I94ae9d254d5fd337774a24106eb9b08585ac0b01
Reviewed-on: http://gerrit.cloudera.org:8080/11519
Reviewed-by: Thomas Marshall <thomasmarshall@cmu.edu>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Thrift's SSLSocketFactory class does not support setting ciphers that
use ecdh. This patch modifies our existing subclass of
SSLSocketFactory to override the ciphers() method and enable ECDH.
The code for this was taken from be/src/kudu/security/tls_context.cc
Testing:
- Added a custom cluster test that verifies that a cluster with only
ECDH ciphers enabled works.
Change-Id: I1666ceabec51b425e8a82be1cf519e2ac35fa5a6
Reviewed-on: http://gerrit.cloudera.org:8080/11376
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
When we upgraded to thrift-0.9.3, the TSSLSocket.py logic changed quite
a bit. Our RHEL7 machines come equipped with Python 2.7.5. Looking at
these comments, that means that we'll be unable to create a 'SSLContext'
but be able to explicitly specify ciphers:
88591e32e7/lib/py/src/transport/TSSLSocket.py (L37-L41)
# SSLContext is not available for Python < 2.7.9
_has_ssl_context = sys.hexversion >= 0x020709F0
# ciphers argument is not available for Python < 2.7.0
_has_ciphers = sys.hexversion >= 0x020700F0
If we cannot create a 'SSLContext', then we cannot use TLSv1.2 and have
to use TLSv1:
88591e32e7/lib/py/src/transport/TSSLSocket.py (L48-L49)
# For python >= 2.7.9, use latest TLS that both client and server
# supports.
# SSL 2.0 and 3.0 are disabled via ssl.OP_NO_SSLv2 and ssl.OP_NO_SSLv3.
# For python < 2.7.9, use TLS 1.0 since TLSv1_X nor OP_NO_SSLvX is
# unavailable.
_default_protocol = ssl.PROTOCOL_SSLv23 if _has_ssl_context else \
ssl.PROTOCOL_TLSv1
Our custom cluster test forces the server to use TLSv1.2 and also forces
a specific cipher:
2f22a6f67f/tests/custom_cluster/test_client_ssl.py (L118-L119)
So this combination of configuration values causes a failure in RHEL7
because we only allow a specific cipher which works with TLSv1.2, but
the client cannot use TLSv1.2 due to the Python version as mentioned above.
We've not noticed these failures on older-than-RHEL7-systems since the
OpenSSL versions on those systems don't support TLSv1.2. (< OpenSSL 1.0.1)
To fix this, we need to change the Python version on RHEL 7 to be
>= Python 2.7.9. This patch skips the test if an older version of
Python than 2.7.9 is detected.
Change-Id: I92c66ecaeb94b0c83ee6f1396c082709c21b3187
Reviewed-on: http://gerrit.cloudera.org:8080/10529
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Wildcard support is not uniform across all platforms that Impala is
tested on. This patch xfails the wildcard tests in test_client_ssl.
A follow-up change will generate certificates on a per-host basis, which
should allow compatible wildcard certs to be generated for all platforms.
Change-Id: I86148739aa1c66c817eed8b727f68cfc08c178ed
Reviewed-on: http://gerrit.cloudera.org:8080/7908
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Impala Public Jenkins
The non-wildcard certs in test_client_ssl.py require that the hostname
of the process is 'localhost' for clients to validate them. This wasn't
the case for one test, and so the cluster wouldn't actually
start. Although the test would still pass (because the shell wasn't
actually checking the certificate), it's better hygiene to have the
cluster correctly configured to make sure we're testing what we think we
are.
Testing: test continues to pass
Change-Id: Idad8bbf3b8be853d3406bcbaed24909501500ea9
Reviewed-on: http://gerrit.cloudera.org:8080/7732
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: Impala Public Jenkins
The shell uses Thrift's TSSLSocket to negotiate secure connections to
Impala. This socket uses a variable SSL_VERSION to determine which SSL
and TLS protocol versions it will connect to.
SSL_VERSION was hardcoded to be PROTOCOL_TLSv1, which only supports
TLSv1 servers and no other protocol version. Change the allowed version
to be PROTOCOL_SSLv23, which supports any TLS or SSL protocol. We rely
on the server not to allow SSLv2 or v3 connections.
Testing: Added a new custom cluster test to confirm that the shell can
connect to a TLSv1.2 cluster. Confirmed that the test is correctly
skipped on machines with an old version of OpenSSL that does not support
TLSv1.2.
Change-Id: I5487f82d110676b9c3c7a5305931da00c7f68ca0
Reviewed-on: http://gerrit.cloudera.org:8080/7675
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Impala Public Jenkins
commit 9f61397fc4 exposed a bug (one
that was latent before the commit). I am XFAILing this now just to
green the build; IMPALA-4295 can be resolved when this issue is fixed
and not just XFAILed.
Change-Id: Ie809c6c6c967447d527927ebbc6b110095e7320a
Reviewed-on: http://gerrit.cloudera.org:8080/4784
Reviewed-by: Jim Apple <jbapple@cloudera.com>
Tested-by: Internal Jenkins
The plan-root fragment instance that runs on the coordinator should be
handled like all others: started via RPC and run asynchronously. Without
this, the fragment requires special-case code throughout the
coordinator, and does not show up in system metrics etc.
This patch adds a new sink type, PlanRootSink, to the root fragment
instance so that the coordinator can pull row batches that are pushed by
the root instance. The coordinator signals completion to the fragment
instance via closing the consumer side of the sink, whereupon the
instance is free to complete.
Since the root instance now runs asynchronously wrt to the coordinator,
we add several coordination methods to allow the coordinator to wait for
a point in the instance's execution to be hit - e.g. to wait until the
instance has been opened.
Done in this patch:
* Add PlanRootSink
* Add coordination to PFE to allow coordinator to observe lifecycle
* Make FragmentMgr a singleton
* Removed dead code from Coordinator::Wait() and elsewhere.
* Moved result output exprs out of QES and into PlanRootSink.
* Remove special-case limit-based teardown of coordinator fragment, and
supporting functions in PlanFragmentExecutor.
* Simplified lifecycle of PlanFragmentExecutor by separating Open() into
Open() and Exec(), the latter of which drives the sink by reading
rows from the plan tree.
* Add child profile to PlanFragmentExecutor to measure time spent in
each lifecycle phase.
* Removed dependency between InitExecProfiles() and starting root
fragment.
* Removed mostly dead-code handling of LIMIT 0 queries.
* Ensured that SET returns a result set in all cases.
* Fix test_get_log() HS2 test. Errors are only guaranteed to be visible
after fetch calls return EOS, but test was assuming this would happen
after first fetch.
Change-Id: Ibb0064ec2f085fa3a5598ea80894fb489a01e4df
Reviewed-on: http://gerrit.cloudera.org:8080/4402
Tested-by: Internal Jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
For files that have a Cloudera copyright (and no other copyright
notice), make changes to follow the ASF source file header policy here:
http://www.apache.org/legal/src-headers.html#headers
Specifically:
1) Remove the Cloudera copyright.
2) Modify NOTICE.txt according to
http://www.apache.org/legal/src-headers.html#notice
to follow that format and add a line for Cloudera.
3) Replace or add the existing ASF license text with the one given
on the website.
Much of this change was automatically generated via:
git grep -li 'Copyright.*Cloudera' > modified_files.txt
cat modified_files.txt | xargs perl -n -i -e 'print unless m#Copyright.*Cloudera#i;'
cat modified_files_txt | xargs fix_apache_license.py [1]
Some manual fixups were performed following those steps, especially when
license text was completely missing from the file.
[1] https://gist.github.com/anonymous/ff71292094362fc5c594 with minor
modification to ORIG_LICENSE to match Impala's license text.
Change-Id: I2e0bd8420945b953e1b806041bea4d72a3943d86
Reviewed-on: http://gerrit.cloudera.org:8080/3779
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
The impala-shell could not accept wildcard or SAN certificates
previously as the thrift library it depended on did not support them.
This patch subclasses TSSLSocket and adds the logic to take care of
the above mentioned cases by introducing the new
TSSLSocketWithWildcardSAN class.
The certificate matching logic is based on the python-ssl source code.
Added custom cluster tests to test both wildcard matching and SAN
matching.
Added be/src/testutil/certificates-info.txt which contains all the
information about the certificates which are added for the tests.
This has been tested with Python2.4 and Python2.6.
Change-Id: I75e37012eeeb0bcf87a5edf875f0ff915daf8b89
Reviewed-on: http://gerrit.cloudera.org:8080/3765
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Internal Jenkins
Many of our test scripts have import statements that look like
"from xxx import *". It is a good practice to explicitly name what
needs to be imported. This commit implements this practice. Also,
unused import statements are removed.
Change-Id: I6a33bb66552ae657d1725f765842f648faeb26a8
Reviewed-on: http://gerrit.cloudera.org:8080/3444
Reviewed-by: Michael Brown <mikeb@cloudera.com>
Tested-by: Internal Jenkins
To cancel a query, the shell will create a separate connection inside
it's SIGINT handler, and send the cancellation RPC. However this
connection did not start a secure connection if it needed to, meaning
that the cancellation attempt would just hang.
A workaround is to kill the shell process, which I expect is what users
have been doing with this bug which has been around since 2014.
Testing:
I added a custom cluster test that starts Impala with SSL
enabled, and wrote two tests - one just to check SSL connectivity, and
the other to mimic the existing test_cancellation which sends SIGINT to
the shell process. In doing so I refactored the shell testing code a bit
so that all tests use a single ImpalaShell object, rather than rolling
their own Popen() based approaches when they needed to do something
unusual, like cancel a query.
In the cancellation test on my machine, SIGINT can take a few tries to
be effective. I'm not sure if this is a timing thing - perhaps the
Python interpreter doesn't correctly pass signals through to a handler
if it's in a blocking call, for example. The test reliably passes within
~5 tries on my machine, so the test tries 30 times, once per second.
Change-Id: If99085e75708d92a08dbecf0131a2234fedad33a
Reviewed-on: http://gerrit.cloudera.org:8080/3302
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: Henry Robinson <henry@cloudera.com>
