This patch adds fragment-level metrics to the WebUI query timeline
display along with additional disk and network metrics.
The fragment's plan nodes are enlarged with an animated transition on
hovering over the fragment's row in query timeline's fragment diagram.
On clicking the plan nodes, total thread and memory usage of the parent
fragment are displayed, after accumulating memory and thread usage of
all child nodes. Thread usage is being shown on the additional Y-axis.
In this way, memory and thread usage of multiple fragments can be
compared alongside. A fragment's usage can be hidden by clicking
on any of the child plan nodes again.
These counters are available within the profile with following names.
- MemoryUsage
- ThreadUsage
Once a fragment's metrics are displayed, they are updated as they
are collected from the profile during a running query.
A grid-line is displayed along with a tooltip on hovering over the
fragment diagram, containing the instantaneous time at that position.
This grid-line also triggers tooltips and gridlines in other charts.
A warning is displayed on clicking a fragment with less number of samples
available.
RESOURCE_TRACE_RATIO query option must be set for providing periodic
metrics within the profile. This allows the following time series
counters to be displayed on the query timeline.
- HostDiskWriteThroughput
- HostDiskReadThroughput
- HostNetworkRx
- HostNetworkTx
The additional Y-axis within the utilization chart is used to represent
the average of these metrics.
The memory units in tooltips and ticks on co-ordinate axes are displayed
in human readable form such as KB, MB, GB and PB for convenience.
Both of the charts contain controls to close the chart. These charts
can also be resized until a maximum and minmum limit by dragging the
resize bar's handle.
Along with mouse wheel events, the diagrams can be horizontally
stretched by the help of buttons with horizontal zoom icons at the
top of the page. The zoom out button is disabled, when further zoom out
is not possible.
Timeticks are being autoscaled during fragment diagram's horizontal zoom.
In addition to the scrollbar, hovering on edges of the window allows
horizontal scrolling.
Test cases have been for the additional disk, network and fragment level
memory metrics parsing functions.
Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f
Reviewed-on: http://gerrit.cloudera.org:8080/20355
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
To help with testing the query timeline web page, the tests have been
integrated using JEST testing framework based on nodejs.
'run_js_tests.sh' runs JEST test suits, after it fetches nodejs binaries
and related nodejs packages, if they are not present locally.
NodeJS binaries are stored within the ${IMPALA_TOOLCHAIN} directory.
'jest-environment-jsdom' supports rendering DOM elements and
testing them without the requirement of a browser.
For implementing unit and integration tests, the script
has been divided into multiple properly functioning modules,
that are imported as ES6 modules.
Unit tests have been written for the following query profile parsing
functions.
- mapTimeseriesCounters() - Test whether the method correctly searches
and maps the profile's counter's indexes based on counter_name even
in reverse order
- accumulateTimeseriesValues() - Test whether the method correctly
accumlates values after parsing values from 'data' in
'time_series_counters' while updating 'max_samples'
- generateTimesamples() - Test whether time sample values generated
based on 'max_samples' are correct, with different 'max_samples'
- clearTimeseriesValues() - Test whether Timeseries arrays are being
properly truncated in the correct range
- initializeUtilizationMetrics() - Test whether aggregate arrays and
time sample arrays are correctly allocated based on counters and
'max_samples'
- getSvg*() - Test whether all getSvg methods are correctly setting
attributes and returning expected attributes in elements
Unit tests produce JUnitXML for integration with jenkins jobs,
these are stored in ${IMPALA_JS_TEST_LOGS_DIR}.
Change-Id: I0caf0a0beee23821f78c0b3fe1aeb7dbf92d6a3e
Reviewed-on: http://gerrit.cloudera.org:8080/20538
Reviewed-by: Wenzhe Zhou <wzhou@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch divides the query timeline script into multiple modules
for better maintainability and to help writing unit tests.
The patch also improves parsing of utilization values from the
query profile, as the order and name of counters may change within
the profile, to ensure appropriate mappings between plotted readings,
initially the profile is traversed to find the correct mapping.
Change-Id: Id9d62a799d838876509686a75ecea778b2c72cc6
Reviewed-on: http://gerrit.cloudera.org:8080/20588
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
