test_query_log_size_in_bytes is flaky in exhaustive builds. The expected
QueryStateRecord is off by around 100KB per query, indicating that the
test query might be too complex and lead to variability in final query
profile that is being archived.
This patch simplifies the test query and relaxes the assertion. This
patch also adds QUERY_LOG_EST_TOTAL_BYTES metric (key name
"impala-server.query-log-est-total-bytes") that is validated as well in
test_query_log_size_in_bytes. query-state-record-test.cc is modified a
bit to resolve segmentation fault issue when building
unified-be-test-validated-executable target run_clang_tidy.sh.
Testing:
- Pass test_query_log_size_in_bytes in exhaustive build.
Change-Id: Ic9b543ffa931a01c11d58e45774ea46af78b3a25
Reviewed-on: http://gerrit.cloudera.org:8080/21100
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
fixed order of "/hadoop-varz" test_web_pages.py for test "test_webserver_hide_logs_link()".
In this test for links "/hadoop-varz" was added to expected_catalog_links[]
as part of improvement IMPALA-9086 but order was conflicted with "/events"
Change-Id: I042ae0464cfc3d13b74ff28099ad52ff53e19f74
Reviewed-on: http://gerrit.cloudera.org:8080/21053
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Coordinator's /queries page is useful to show information about recently
run and completed queries. Having more entries will be helpful to
inspect queries that completed further back. The maximum entry of this
table is controlled by 'query_log_size' flag. Higher value means more
queries to keep, but it also cost more memory overhead in coordinator.
This patch increase 'query_log_size' default value from 100 to 200. This
patch also add flag 'query_log_size_in_bytes' (default to 2GB) as an
additional safeguard to evict entry from query_log_ when this limit
exceeded, preventing query_log_ total memory to grow prohibitively
large. 'query_log_size_in_bytes' is used in combination with
'query_log_size' to limit the number of QueryStateRecord to retain in
query_log_, whichever is less.
Testing:
- Pass exhaustive tests.
Change-Id: I107e2c2c7f2b239557be37360e8eecf5479e8602
Reviewed-on: http://gerrit.cloudera.org:8080/21020
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
It's a two part improvement:
1. Updated /hadoop-varz static class implementation for coordinator webUI
from: org.apache.hadoop.conf.Configuration
to : org.apache.hadoop.hive.conf.HiveConf (Inherits Configuration)
2. Added similar implementation of /hadoop-varz for catalog webUI
Now /hadoop-varz page has lot more information about HMS client
and configuration including warehouse.dir
Change-Id: I5af0eb68e71afeed64660d4d40584208ea503217
Reviewed-on: http://gerrit.cloudera.org:8080/20682
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
test_catalog_operations_with_rpc_retry uses a short timeout which could
lead to failures in the first DESCRIBE statement. What it expects is the
next REFRESH statement failed by the RPC timeout.
The DESCRIBE statement triggers a catalog RPC of PrioritizeLoad on the
table. The RPC usually finishes in 40ms. This patch bumps the RPC
timeout of the test to be 100ms so it's long enough for DESCRIBE to
succeed. Also bumps the sleep time in the REFRESH statement so its
ExecDdl RPC will always time out in 100ms.
Tests:
- Ran the test till night (2700 times) and all passed.
Change-Id: Ibbfa79d7f7530af4cfbb6f8ebc55e8267e3d3261
Reviewed-on: http://gerrit.cloudera.org:8080/20924
Reviewed-by: Wenzhe Zhou <wzhou@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
In-flight catalog operations are tracked in a map using query id as the
key. It's ok since catalog clients use 0 as the timeout by default (see
--catalog_client_rpc_timeout_ms), i.e. catalog RPCs never timeout, which
means each query will have at most one in-flight catalog RPC at a time.
However, in case catalog_client_rpc_timeout_ms is set to non-zero,
impalad could retry the catalog RPC when it's considered timed out. That
causes several in-flight catalog operations coming from the same query
(so using the same query-id as the map key).
To fix the key conflicts, this patch use the pair of (queryId, threadId)
as the key of the in-flight operations map. 'threadId' comes from the
thrift thread that handles the RPC so it's unique across different
retries.
Tests:
- Add custom-cluster test to verify all retries are shown in the
/operations page.
Change-Id: Icd94ac7532fe7f3d68028c2da82298037be706c4
Reviewed-on: http://gerrit.cloudera.org:8080/20877
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Statestore::Topic::DeleteIfVersionsMatch() is used when impalad is down.
It incorrectly bumps the metrics of total value size and total topic
size. This patch removes the codes.
In order to verify the metrics using the /topics page, this patch
changes the /topics URL to also return numeric values which can be used
in the e2e test to verify the /metrics page.
Tests
- Add e2e test
Change-Id: I3ffcfb45b7cde0b40a87c9ca410ec634cb31cefb
Reviewed-on: http://gerrit.cloudera.org:8080/20841
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch modifies the dynamic query progress reporting in impala-shell
by adding an extra query progress bar below the scan progress bar.
The query progress is calculated using the number of completed fragment
instances divided by the total number of fragment instances. Compared to
the scan progress, which is calculated based on completed scan ranges
divided by the total scan ranges, the query progress provides a more
accurate reflection of the actual completion progress of the query.
Particularly for computationally intensive queries involving complex
aggregations or sorting, such as tpcds query78, there is often
additional computation time required after the scanning is complete. In
such cases, displaying only 100% scan progress would be inaccurate.
Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Reviewed-on: http://gerrit.cloudera.org:8080/20672
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
This patch extends the /operations page in catalogd WebUI to show the
in-flight and finished catalog operations. The following fields are
shown for each operation:
- Thread ID
- Query ID
- Client IP
- Coordinator
- Type
- Target
- User
- Start Time
- End Time (only shown for finished operations)
- Duration
- Status
- Details
Finished operation records are currently kept in memory and limited by
the catalog_operation_log_size flag (defaults to 100).
To collect the above fields, this patch extends
TCatalogServiceRequestHeader to contain the coordinator hostname. Also
fix some catalog RPCs that didn't fill these fields.
Tests:
- Add e2e test in custom_cluster/test_web_pages.py
- Manually verify the web pages when running a GVO job
Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa
Reviewed-on: http://gerrit.cloudera.org:8080/20428
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
IMPALA-12053 makes the "/events" link visible in catalogd WebUI. This
causes test failures in test_webserver_hide_logs_link which checks all
the visible links. This patch fixes the test.
Tests:
- Ran test_webserver_hide_logs_link locally
Change-Id: I23cb45cf9bf2225a832ce77642d698709eebe604
Reviewed-on: http://gerrit.cloudera.org:8080/19953
Reviewed-by: Andrew Sherman <asherman@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Web queries site shows no resource pool unless it is specified with
query option. The Planner could set TQueryCtx.request_pool in
TQueryExecRequest when auto scaling is enabled. But the backend
ignores the TQueryCtx.request_pool in TQueryExecRequest when getting
resource pools for Web UI.
This patch fixes the issue in ClientRequestState::request_pool() by
checking TQueryCtx.request_pool in TQueryExecRequest. It also
removes the error path in RequestPoolService::ResolveRequestPool() if
requested_pool is empty string.
Testing:
- Updated TestExecutorGroups::test_query_cpu_count_divisor_default,
TestExecutorGroups::test_query_cpu_count_divisor_two, and
TestExecutorGroups::test_query_cpu_count_divisor_fraction to
verify resource pools on Web queries site and Web admission site.
- Updated expected error message in
TestAdmissionController::test_set_request_pool.
- Passed core test.
Change-Id: Iceacb3a8ec3bd15a8029ba05d064bbbb81e3a766
Reviewed-on: http://gerrit.cloudera.org:8080/19688
Reviewed-by: Riza Suminto <riza.suminto@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Kurt Deschler <kdeschle@cloudera.com>
Reviewed-by: Abhishek Rawat <arawat@cloudera.com>
This takes steps to make Python 2 behave like Python 3 as
a way to flush out issues with running on Python 3. Specifically,
it handles two main differences:
1. Python 3 requires absolute imports within packages. This
can be emulated via "from __future__ import absolute_import"
2. Python 3 changed division to "true" division that doesn't
round to an integer. This can be emulated via
"from __future__ import division"
This changes all Python files to add imports for absolute_import
and division. For completeness, this also includes print_function in the
import.
I scrutinized each old-division location and converted some locations
to use the integer division '//' operator if it needed an integer
result (e.g. for indices, counts of records, etc). Some code was also using
relative imports and needed to be adjusted to handle absolute_import.
This fixes all Pylint warnings about no-absolute-import and old-division,
and these warnings are now banned.
Testing:
- Ran core tests
Change-Id: Idb0fcbd11f3e8791f5951c4944be44fb580e576b
Reviewed-on: http://gerrit.cloudera.org:8080/19588
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Tested-by: Joe McDonnell <joemcdonnell@cloudera.com>
Python 3 now treats print as a function and requires
the parenthesis in invocation.
print "Hello World!"
is now:
print("Hello World!")
This fixes all locations to use the function
invocation. This is more complicated when the output
is being redirected to a file or when avoiding the
usual newline.
print >> sys.stderr , "Hello World!"
is now:
print("Hello World!", file=sys.stderr)
To support this properly and guarantee equivalent behavior
between python 2 and python 3, all files that use print
now add this import:
from __future__ import print_function
This also fixes random flake8 issues that intersect with
the changes.
Testing:
- check-python-syntax.sh shows no errors related to print
Change-Id: Ib634958369ad777a41e72d80c8053b74384ac351
Reviewed-on: http://gerrit.cloudera.org:8080/19552
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Tested-by: Michael Smith <michael.smith@cloudera.com>
Content Security Policy (CSP) is a computer security standard designed
to prevent cross-site scripting, clickjacking and other code injection
attacks. CSP provides a method for websites to declare approved origins
of content that browsers should be allowed to load on that website.
A good resource is https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP
If a page breaks the rules then the included script or css will
typically not be run by the browser.
In the Impala webui we use a CSP header to declare that all web content
comes from the impalad, with some 'unsafe' inline code.
A new server flag "--disable_content_security_policy_header=true" can be
set to disable the emission of this header in case of any compatibility
issues.
A few small changes were needed to make this CSP header work. Chart.js
was previously included via http, this was changed to being bundled
like other javascript and css we use. Some dodgy array code that
handles connection metrics was also fixed.
TESTING:
The main webui tests all now validate the CSP header is present.
A test for the new flag is also added.
Change-Id: Idc335d65b117661da0b420ddb7c9ccd80d8d76ab
Reviewed-on: http://gerrit.cloudera.org:8080/18168
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
If you start an Impala daemon with the flags
"--logtostderr=true --redirect_stdout_stderr=false"
then log files are not created. After this the webui will display an
error if you click on the "/logs" link. Fix this by not adding
the "/logs" link in this case.
TESTING
- Added a new test that validates the navbar links in the webui.
- Ran exhaustive tests
Change-Id: I65234213f32902caa1f4368181b49f012a4dbcb3
Reviewed-on: http://gerrit.cloudera.org:8080/18062
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch introduces a new startup flag
--ping_expose_webserver_url (true by default) to control whether
PingImpalaService, PingImpalaHS2Service RPC calls should expose
the debug web url to the client or not.
This is necessary as the debug web UI is not something that
end-users will necessarily have access to.
If the flag is set to false, the RPC calls will return an empty
string instead of the real url signalling that the debug web ui
is not available.
Note that if the webserver is disabled (--enable_webserver flag
is set to false) the RPC calls will behave the same and return an
empty string for the url.
Change-Id: I7ec3e92764d712b8fee63c1f45b038c31c184cfc
Reviewed-on: http://gerrit.cloudera.org:8080/16573
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
In the WebUI's query list the query statements are trimmed, but the full
query statement can be seen in the details page.
The default statement length is 250 chars and it can be adjusted by the
query_stmt_size flag that can be set when the cluster starts.
Example:
bin/start-impala-cluster.py -s1 --impalad_args --query_stmt_size=10
Testing:
-manual testing in the WebUI.
-added 'test_query_stmt()' to test_web_pages.py
-added test to the custom cluster webserver tests to check without truncate
and with custom length truncate
Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605
Reviewed-on: http://gerrit.cloudera.org:8080/15288
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch fixes two issues with --webserver_interface:
- When --webserver_interface was used start-impala-cluster.py with a
value that's different from --hostname, minicluster startup would
appear to fail as liveness is determined by checking for the webui's
availability at the address specified for --hostname.
- The value of --webserver_interface was applied correctly for the
catalogd and statestored but not for impalads, due to the way
ExecEnv constructed the Webserver.
- It is now possible to specify a hostname for webserver_interface
instead of an IP. The webserver will resolve the hostname.
This patch also upgrades our version of psutil to the latest for the
function 'net_if_addrs'. This requires a few change to our use of
psutil, mostly adding '()' to call functions that previously were
variables.
Testing:
- Added a custom cluster test that finds all available interfaces,
binds the webserver to one of them, and checks that its only
available over that interface.
Change-Id: Ic7e75908426756d73f13a0fa3cfc21fc31da164c
Reviewed-on: http://gerrit.cloudera.org:8080/14266
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This includes some optimisations and a bulk move of tests
to exhaustive.
Move a bunch of custom cluster tests to exhaustive. I selected
these partially based on runtime (i.e. I looked most carefully
at the tests that ran for over a minute) and the likelihood
of them catching a precommit bug. Regression tests for specific
edge cases and tests for parts of the code that are very stable
were prime candidates.
Remove an unnecessary cluster restart in test_breakpad.
Merge test_scheduler_error into test_failpoints to avoid an unnecessary
cluster restart.
Speed up cluster starts by ensuring that the default statestore args are
applied even when _start_impala_cluster() is called directly. This
shaves a couple of seconds off each restart. We made the default args
use a faster update frequency - see IMPALA-7185 - but they did not
take effect in all tests.
Change-Id: Ib2e3e7ebc9695baec4d69183387259958df10f62
Reviewed-on: http://gerrit.cloudera.org:8080/13967
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
In order to support the TraceEvent path handlers from Kudu (and
other functionality like pprof symbolization), the webserver needs
to support HTTP POST. This changes the signature for HTTP handler
callbacks to take a 'WebRequest' struct instead of a simple map of
arguments. This struct exposes the request type, POST data, and
unparsed query strings, enabling the callbacks to handle POSTs.
The struct itself is imported from Kudu's webserver code, so we can
start to share utility HTTP handlers (eg pprof or tracing) more
easily between Kudu and Impala.
The maximum length of a POST request is limited by the new
webserver_max_post_length_bytes parameter, which defaults to
1MB.
This patch was written by Todd Lipcon, based on code from Kudu.
Testing:
- Added a test for too big POSTs to custom_cluster/test_web_pages.py
- Verifies the existing core tests pass
Change-Id: I3bd3ecfab53263c13a4509c278569302c9901abb
Reviewed-on: http://gerrit.cloudera.org:8080/12730
Reviewed-by: Todd Lipcon <todd@apache.org>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch shows the experimental flags when set in /varz by adding
a new "Experimental" column in /varz web page. The patch also updates
the log files to show experimental flags when set.
Testing:
- Added a new test for /varz web page
- Manually looked at the log files and /varz web page with experimental
flags set
Change-Id: Ic13b2694b445059e7e52c1ae56bd7e0e925fbb0e
Reviewed-on: http://gerrit.cloudera.org:8080/12152
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
