test_event_processor_status
When hierarchical event processing is enabled, there is no info about
the current event batch shown in the /events page. Note that event
batches are dispatched and processed later in parallel. The current
event batch info is actually showing the current batch that is being
dispatched which won't take long.
This patch skips checking the current event batch info when hierarchical
event processing is enabled.
Tests
- Verified that test runs fine locally.
Change-Id: I2df24d2fd3b028a84d557e70141e68aa234908d4
Reviewed-on: http://gerrit.cloudera.org:8080/23790
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
When hierarchical event processing is enabled, there is no info about
the current event batch shown in the /events page. Note that event
batches are dispatched and processed later in parallel. The current
event batch info is actually showing the current batch that is being
dispatched which won't take long.
This patch skips checking the current event batch info when hierarchical
event processing is enabled. A new method,
is_hierarchical_event_processing_enabled(), is added in
ImpalaTestClusterProperties for the check. Also fixes
is_event_polling_enabled() to accept float values of
hms_event_polling_interval_s and adds the missing raise statement when
it fails to parse the flags.
Tests
- Ran the test locally.
Change-Id: Iffb84304a4096885492002b781199051aaa4fbb0
Reviewed-on: http://gerrit.cloudera.org:8080/23766
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Local catalog mode has been the default and works well in downstream
Impala for over 5 years. This patch turn on local catalog mode by
default (--catalog_topic_mode=minimal and --use_local_catalog=true) as
preferred mode going forward.
Implemented LocalCatalog.setIsReady() to facilitate using local catalog
mode for FE tests. Some FE tests fail due to behavior differences in
local catalog mode like IMPALA-7539. This is probably OK since Impala
now largely hand over FileSystem permission check to Apache Ranger.
The following custom cluster tests are pinned to evaluate under legacy
catalog mode because their behavior changed in local catalog mode:
TestCalcitePlanner.test_calcite_frontend
TestCoordinators.test_executor_only_lib_cache
TestMetadataReplicas
TestTupleCacheCluster
TestWorkloadManagementSQLDetailsCalcite.test_tpcds_8_decimal
At TestHBaseHmsColumnOrder.test_hbase_hms_column_order, set
--use_hms_column_order_for_hbase_tables=true flag for both impalad and
catalogd to get consistent column order in either local or legacy
catalog mode.
Changed TestCatalogRpcErrors.test_register_subscriber_rpc_error
assertions to be more fine grained by matching individual query id.
Move most of test methods from TestRangerLegacyCatalog to
TestRangerLocalCatalog, except for some that do need to run in legacy
catalog mode. Also renamed TestRangerLocalCatalog to
TestRangerDefaultCatalog. Table ownership issue in local catalog mode
remains unresolved (see IMPALA-8937).
Testing:
Pass exhaustive tests.
Change-Id: Ie303e294972d12b98f8354bf6bbc6d0cb920060f
Reviewed-on: http://gerrit.cloudera.org:8080/23080
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This is the final patch to move all Impala e2e and custom cluster tests
to use HS2 protocol by default. Only beeswax-specific test remains
testing against beeswax protocol by default. We can remove them once
Impala officially remove beeswax support.
HS2 error message formatting in impala-hs2-server.cc is adjusted a bit
to match with formatting in impala-beeswax-server.cc.
Move TestWebPageAndCloseSession from webserver/test_web_pages.py to
custom_cluster/test_web_pages.py to disable glog log buffering.
Testing:
- Pass exhaustive tests, except for some known and unrelated flaky
tests.
Change-Id: I42e9ceccbba1e6853f37e68f106265d163ccae28
Reviewed-on: http://gerrit.cloudera.org:8080/22845
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Jason Fehr <jfehr@cloudera.com>
This change adds get_workload() to ImpalaTestSuite and removes it
from all test suites that already returned 'functional-query'.
get_workload() is also removed from CustomClusterTestSuite which
used to return 'tpch'.
All other changes besides impala_test_suite.py and
custom_cluster_test_suite.py are just mass removals of
get_workload() functions.
The behavior is only changed in custom cluster tests that didn't
override get_workload(). By returning 'functional-query' instead
of 'tpch', exploration_strategy() will no longer return 'core' in
'exhaustive' test runs. See IMPALA-3947 on why workload affected
exploration_strategy. An example for affected test is
TestCatalogHMSFailures which was skipped both in core and exhaustive
runs before this change.
get_workload() functions that return a different workload than
'functional-query' are not changed - it is possible that some of
these also don't handle exploration_strategy() as expected, but
individually checking these tests is out of scope in this patch.
Change-Id: I9ec6c41ffb3a30e1ea2de773626d1485c69fe115
Reviewed-on: http://gerrit.cloudera.org:8080/22726
Reviewed-by: Riza Suminto <riza.suminto@cloudera.com>
Reviewed-by: Daniel Becker <daniel.becker@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch replace create_beeswax_client() reference to
create_hs2_client() or vector-based client creation to prepare towards
hs2 test migration.
test_session_expiration_with_queued_query is changed to use impala.dbapi
directly from Impyla due to limitation in ImpylaHS2Connection.
TestAdmissionControllerRawHS2 is migrated to use hs2 as default test
protocol.
Modify test_query_expiration.py to set query option through client
instead of SET query. test_query_expiration is slightly modified due to
behavior difference in hs2 ImpylaHS2Connection.
Remove remaining reference to BeeswaxConnection.QueryState.
Fixed a bug in ImpylaHS2Connection.wait_for_finished_timeout().
Fix some easy flake8 issues caught thorugh this command:
git show HEAD --name-only | grep '^tests.*py' \
| xargs -I {} impala-flake8 {} \
| grep -e U100 -e E111 -e E301 -e E302 -e E303 -e F...
Testing:
- Pass exhaustive tests.
Change-Id: I1d84251835d458cc87fb8fedfc20ee15aae18d51
Reviewed-on: http://gerrit.cloudera.org:8080/22700
Reviewed-by: Riza Suminto <riza.suminto@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Previously, some table metrics, such as the estimated memory usage
and the number of files, were only updated when a "FULL" Thrift object
of the table is requested. As a result, if a user ran a DESCRIBE
command on a table, and then tried to find the table on the Top-N page
of the web UI, the user would not find it.
This patch fixes the issue by updating the table metrics as soon as
an HDFS table is loaded. With this, no matter what Thrift object type of
the table is requested, the metrics will always be updated and
displayed on the web UI.
Testing:
- Added two custom cluster tests in test_web_pages.py to make sure that
table stats can be viewed on the web UI after DESCRIBE, for both
legacy and local catalog modes.
Change-Id: I6e2eb503b0f61b1e6403058bc5dc78d721e7e940
Reviewed-on: http://gerrit.cloudera.org:8080/22014
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
The /events page of catalogd shows the metrics and status of the
event-processor. This patch adds more info in this page, including
- lag info
- current event batch that's being processed
See the screenshot attached in the JIRA for how it looks like.
Also moves the error message to the top to highlight the error status.
Fixes the issue of not updating latest event id when event processor is
stopped. Also fixes the issue of error message not cleared after global
INVALIDATE METADATA.
Adds a debug action, catalogd_event_processing_delay, to inject a sleep
while processing an event. So the web page can be captured more easily.
Also adds a missing test for showing the error message of
event-processing in the /events page.
Tests:
- Add e2e test to verify the content of the page.
Change-Id: I2e7d4952c7fd04ae89b6751204499bf9dd99f57c
Reviewed-on: http://gerrit.cloudera.org:8080/20986
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
test_query_log_size_in_bytes is flaky in exhaustive builds. The expected
QueryStateRecord is off by around 100KB per query, indicating that the
test query might be too complex and lead to variability in final query
profile that is being archived.
This patch simplifies the test query and relaxes the assertion. This
patch also adds QUERY_LOG_EST_TOTAL_BYTES metric (key name
"impala-server.query-log-est-total-bytes") that is validated as well in
test_query_log_size_in_bytes. query-state-record-test.cc is modified a
bit to resolve segmentation fault issue when building
unified-be-test-validated-executable target run_clang_tidy.sh.
Testing:
- Pass test_query_log_size_in_bytes in exhaustive build.
Change-Id: Ic9b543ffa931a01c11d58e45774ea46af78b3a25
Reviewed-on: http://gerrit.cloudera.org:8080/21100
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
fixed order of "/hadoop-varz" test_web_pages.py for test "test_webserver_hide_logs_link()".
In this test for links "/hadoop-varz" was added to expected_catalog_links[]
as part of improvement IMPALA-9086 but order was conflicted with "/events"
Change-Id: I042ae0464cfc3d13b74ff28099ad52ff53e19f74
Reviewed-on: http://gerrit.cloudera.org:8080/21053
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Coordinator's /queries page is useful to show information about recently
run and completed queries. Having more entries will be helpful to
inspect queries that completed further back. The maximum entry of this
table is controlled by 'query_log_size' flag. Higher value means more
queries to keep, but it also cost more memory overhead in coordinator.
This patch increase 'query_log_size' default value from 100 to 200. This
patch also add flag 'query_log_size_in_bytes' (default to 2GB) as an
additional safeguard to evict entry from query_log_ when this limit
exceeded, preventing query_log_ total memory to grow prohibitively
large. 'query_log_size_in_bytes' is used in combination with
'query_log_size' to limit the number of QueryStateRecord to retain in
query_log_, whichever is less.
Testing:
- Pass exhaustive tests.
Change-Id: I107e2c2c7f2b239557be37360e8eecf5479e8602
Reviewed-on: http://gerrit.cloudera.org:8080/21020
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
It's a two part improvement:
1. Updated /hadoop-varz static class implementation for coordinator webUI
from: org.apache.hadoop.conf.Configuration
to : org.apache.hadoop.hive.conf.HiveConf (Inherits Configuration)
2. Added similar implementation of /hadoop-varz for catalog webUI
Now /hadoop-varz page has lot more information about HMS client
and configuration including warehouse.dir
Change-Id: I5af0eb68e71afeed64660d4d40584208ea503217
Reviewed-on: http://gerrit.cloudera.org:8080/20682
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
test_catalog_operations_with_rpc_retry uses a short timeout which could
lead to failures in the first DESCRIBE statement. What it expects is the
next REFRESH statement failed by the RPC timeout.
The DESCRIBE statement triggers a catalog RPC of PrioritizeLoad on the
table. The RPC usually finishes in 40ms. This patch bumps the RPC
timeout of the test to be 100ms so it's long enough for DESCRIBE to
succeed. Also bumps the sleep time in the REFRESH statement so its
ExecDdl RPC will always time out in 100ms.
Tests:
- Ran the test till night (2700 times) and all passed.
Change-Id: Ibbfa79d7f7530af4cfbb6f8ebc55e8267e3d3261
Reviewed-on: http://gerrit.cloudera.org:8080/20924
Reviewed-by: Wenzhe Zhou <wzhou@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
In-flight catalog operations are tracked in a map using query id as the
key. It's ok since catalog clients use 0 as the timeout by default (see
--catalog_client_rpc_timeout_ms), i.e. catalog RPCs never timeout, which
means each query will have at most one in-flight catalog RPC at a time.
However, in case catalog_client_rpc_timeout_ms is set to non-zero,
impalad could retry the catalog RPC when it's considered timed out. That
causes several in-flight catalog operations coming from the same query
(so using the same query-id as the map key).
To fix the key conflicts, this patch use the pair of (queryId, threadId)
as the key of the in-flight operations map. 'threadId' comes from the
thrift thread that handles the RPC so it's unique across different
retries.
Tests:
- Add custom-cluster test to verify all retries are shown in the
/operations page.
Change-Id: Icd94ac7532fe7f3d68028c2da82298037be706c4
Reviewed-on: http://gerrit.cloudera.org:8080/20877
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Statestore::Topic::DeleteIfVersionsMatch() is used when impalad is down.
It incorrectly bumps the metrics of total value size and total topic
size. This patch removes the codes.
In order to verify the metrics using the /topics page, this patch
changes the /topics URL to also return numeric values which can be used
in the e2e test to verify the /metrics page.
Tests
- Add e2e test
Change-Id: I3ffcfb45b7cde0b40a87c9ca410ec634cb31cefb
Reviewed-on: http://gerrit.cloudera.org:8080/20841
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch modifies the dynamic query progress reporting in impala-shell
by adding an extra query progress bar below the scan progress bar.
The query progress is calculated using the number of completed fragment
instances divided by the total number of fragment instances. Compared to
the scan progress, which is calculated based on completed scan ranges
divided by the total scan ranges, the query progress provides a more
accurate reflection of the actual completion progress of the query.
Particularly for computationally intensive queries involving complex
aggregations or sorting, such as tpcds query78, there is often
additional computation time required after the scanning is complete. In
such cases, displaying only 100% scan progress would be inaccurate.
Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Reviewed-on: http://gerrit.cloudera.org:8080/20672
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
This patch extends the /operations page in catalogd WebUI to show the
in-flight and finished catalog operations. The following fields are
shown for each operation:
- Thread ID
- Query ID
- Client IP
- Coordinator
- Type
- Target
- User
- Start Time
- End Time (only shown for finished operations)
- Duration
- Status
- Details
Finished operation records are currently kept in memory and limited by
the catalog_operation_log_size flag (defaults to 100).
To collect the above fields, this patch extends
TCatalogServiceRequestHeader to contain the coordinator hostname. Also
fix some catalog RPCs that didn't fill these fields.
Tests:
- Add e2e test in custom_cluster/test_web_pages.py
- Manually verify the web pages when running a GVO job
Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa
Reviewed-on: http://gerrit.cloudera.org:8080/20428
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
IMPALA-12053 makes the "/events" link visible in catalogd WebUI. This
causes test failures in test_webserver_hide_logs_link which checks all
the visible links. This patch fixes the test.
Tests:
- Ran test_webserver_hide_logs_link locally
Change-Id: I23cb45cf9bf2225a832ce77642d698709eebe604
Reviewed-on: http://gerrit.cloudera.org:8080/19953
Reviewed-by: Andrew Sherman <asherman@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Web queries site shows no resource pool unless it is specified with
query option. The Planner could set TQueryCtx.request_pool in
TQueryExecRequest when auto scaling is enabled. But the backend
ignores the TQueryCtx.request_pool in TQueryExecRequest when getting
resource pools for Web UI.
This patch fixes the issue in ClientRequestState::request_pool() by
checking TQueryCtx.request_pool in TQueryExecRequest. It also
removes the error path in RequestPoolService::ResolveRequestPool() if
requested_pool is empty string.
Testing:
- Updated TestExecutorGroups::test_query_cpu_count_divisor_default,
TestExecutorGroups::test_query_cpu_count_divisor_two, and
TestExecutorGroups::test_query_cpu_count_divisor_fraction to
verify resource pools on Web queries site and Web admission site.
- Updated expected error message in
TestAdmissionController::test_set_request_pool.
- Passed core test.
Change-Id: Iceacb3a8ec3bd15a8029ba05d064bbbb81e3a766
Reviewed-on: http://gerrit.cloudera.org:8080/19688
Reviewed-by: Riza Suminto <riza.suminto@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Kurt Deschler <kdeschle@cloudera.com>
Reviewed-by: Abhishek Rawat <arawat@cloudera.com>
This takes steps to make Python 2 behave like Python 3 as
a way to flush out issues with running on Python 3. Specifically,
it handles two main differences:
1. Python 3 requires absolute imports within packages. This
can be emulated via "from __future__ import absolute_import"
2. Python 3 changed division to "true" division that doesn't
round to an integer. This can be emulated via
"from __future__ import division"
This changes all Python files to add imports for absolute_import
and division. For completeness, this also includes print_function in the
import.
I scrutinized each old-division location and converted some locations
to use the integer division '//' operator if it needed an integer
result (e.g. for indices, counts of records, etc). Some code was also using
relative imports and needed to be adjusted to handle absolute_import.
This fixes all Pylint warnings about no-absolute-import and old-division,
and these warnings are now banned.
Testing:
- Ran core tests
Change-Id: Idb0fcbd11f3e8791f5951c4944be44fb580e576b
Reviewed-on: http://gerrit.cloudera.org:8080/19588
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Tested-by: Joe McDonnell <joemcdonnell@cloudera.com>
Python 3 now treats print as a function and requires
the parenthesis in invocation.
print "Hello World!"
is now:
print("Hello World!")
This fixes all locations to use the function
invocation. This is more complicated when the output
is being redirected to a file or when avoiding the
usual newline.
print >> sys.stderr , "Hello World!"
is now:
print("Hello World!", file=sys.stderr)
To support this properly and guarantee equivalent behavior
between python 2 and python 3, all files that use print
now add this import:
from __future__ import print_function
This also fixes random flake8 issues that intersect with
the changes.
Testing:
- check-python-syntax.sh shows no errors related to print
Change-Id: Ib634958369ad777a41e72d80c8053b74384ac351
Reviewed-on: http://gerrit.cloudera.org:8080/19552
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Tested-by: Michael Smith <michael.smith@cloudera.com>
Content Security Policy (CSP) is a computer security standard designed
to prevent cross-site scripting, clickjacking and other code injection
attacks. CSP provides a method for websites to declare approved origins
of content that browsers should be allowed to load on that website.
A good resource is https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP
If a page breaks the rules then the included script or css will
typically not be run by the browser.
In the Impala webui we use a CSP header to declare that all web content
comes from the impalad, with some 'unsafe' inline code.
A new server flag "--disable_content_security_policy_header=true" can be
set to disable the emission of this header in case of any compatibility
issues.
A few small changes were needed to make this CSP header work. Chart.js
was previously included via http, this was changed to being bundled
like other javascript and css we use. Some dodgy array code that
handles connection metrics was also fixed.
TESTING:
The main webui tests all now validate the CSP header is present.
A test for the new flag is also added.
Change-Id: Idc335d65b117661da0b420ddb7c9ccd80d8d76ab
Reviewed-on: http://gerrit.cloudera.org:8080/18168
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
If you start an Impala daemon with the flags
"--logtostderr=true --redirect_stdout_stderr=false"
then log files are not created. After this the webui will display an
error if you click on the "/logs" link. Fix this by not adding
the "/logs" link in this case.
TESTING
- Added a new test that validates the navbar links in the webui.
- Ran exhaustive tests
Change-Id: I65234213f32902caa1f4368181b49f012a4dbcb3
Reviewed-on: http://gerrit.cloudera.org:8080/18062
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch introduces a new startup flag
--ping_expose_webserver_url (true by default) to control whether
PingImpalaService, PingImpalaHS2Service RPC calls should expose
the debug web url to the client or not.
This is necessary as the debug web UI is not something that
end-users will necessarily have access to.
If the flag is set to false, the RPC calls will return an empty
string instead of the real url signalling that the debug web ui
is not available.
Note that if the webserver is disabled (--enable_webserver flag
is set to false) the RPC calls will behave the same and return an
empty string for the url.
Change-Id: I7ec3e92764d712b8fee63c1f45b038c31c184cfc
Reviewed-on: http://gerrit.cloudera.org:8080/16573
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
In the WebUI's query list the query statements are trimmed, but the full
query statement can be seen in the details page.
The default statement length is 250 chars and it can be adjusted by the
query_stmt_size flag that can be set when the cluster starts.
Example:
bin/start-impala-cluster.py -s1 --impalad_args --query_stmt_size=10
Testing:
-manual testing in the WebUI.
-added 'test_query_stmt()' to test_web_pages.py
-added test to the custom cluster webserver tests to check without truncate
and with custom length truncate
Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605
Reviewed-on: http://gerrit.cloudera.org:8080/15288
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch fixes two issues with --webserver_interface:
- When --webserver_interface was used start-impala-cluster.py with a
value that's different from --hostname, minicluster startup would
appear to fail as liveness is determined by checking for the webui's
availability at the address specified for --hostname.
- The value of --webserver_interface was applied correctly for the
catalogd and statestored but not for impalads, due to the way
ExecEnv constructed the Webserver.
- It is now possible to specify a hostname for webserver_interface
instead of an IP. The webserver will resolve the hostname.
This patch also upgrades our version of psutil to the latest for the
function 'net_if_addrs'. This requires a few change to our use of
psutil, mostly adding '()' to call functions that previously were
variables.
Testing:
- Added a custom cluster test that finds all available interfaces,
binds the webserver to one of them, and checks that its only
available over that interface.
Change-Id: Ic7e75908426756d73f13a0fa3cfc21fc31da164c
Reviewed-on: http://gerrit.cloudera.org:8080/14266
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This includes some optimisations and a bulk move of tests
to exhaustive.
Move a bunch of custom cluster tests to exhaustive. I selected
these partially based on runtime (i.e. I looked most carefully
at the tests that ran for over a minute) and the likelihood
of them catching a precommit bug. Regression tests for specific
edge cases and tests for parts of the code that are very stable
were prime candidates.
Remove an unnecessary cluster restart in test_breakpad.
Merge test_scheduler_error into test_failpoints to avoid an unnecessary
cluster restart.
Speed up cluster starts by ensuring that the default statestore args are
applied even when _start_impala_cluster() is called directly. This
shaves a couple of seconds off each restart. We made the default args
use a faster update frequency - see IMPALA-7185 - but they did not
take effect in all tests.
Change-Id: Ib2e3e7ebc9695baec4d69183387259958df10f62
Reviewed-on: http://gerrit.cloudera.org:8080/13967
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
In order to support the TraceEvent path handlers from Kudu (and
other functionality like pprof symbolization), the webserver needs
to support HTTP POST. This changes the signature for HTTP handler
callbacks to take a 'WebRequest' struct instead of a simple map of
arguments. This struct exposes the request type, POST data, and
unparsed query strings, enabling the callbacks to handle POSTs.
The struct itself is imported from Kudu's webserver code, so we can
start to share utility HTTP handlers (eg pprof or tracing) more
easily between Kudu and Impala.
The maximum length of a POST request is limited by the new
webserver_max_post_length_bytes parameter, which defaults to
1MB.
This patch was written by Todd Lipcon, based on code from Kudu.
Testing:
- Added a test for too big POSTs to custom_cluster/test_web_pages.py
- Verifies the existing core tests pass
Change-Id: I3bd3ecfab53263c13a4509c278569302c9901abb
Reviewed-on: http://gerrit.cloudera.org:8080/12730
Reviewed-by: Todd Lipcon <todd@apache.org>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch shows the experimental flags when set in /varz by adding
a new "Experimental" column in /varz web page. The patch also updates
the log files to show experimental flags when set.
Testing:
- Added a new test for /varz web page
- Manually looked at the log files and /varz web page with experimental
flags set
Change-Id: Ic13b2694b445059e7e52c1ae56bd7e0e925fbb0e
Reviewed-on: http://gerrit.cloudera.org:8080/12152
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
