For large JS scripts, code-analysis and linting rules were added
to standardize changes for the webUI scripts in IMPALA-13473.
This change corrects the scripts to align with these linting
and code-analysis rules.
Change-Id: I5c8851c3d2f4e95b7eebd61f379361818afca7ea
Reviewed-on: http://gerrit.cloudera.org:8080/22950
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Riza Suminto <riza.suminto@cloudera.com>
This patch adds support for JS code analysis and linting to webUI
scripts using ESLint.
Support to enforce code style and quality is partcularly beneficial,
as the codebase for client-side scripts is consistently growing.
This has been implemented to work alongside other code style enforcement
rules present within 'critique-gerrit-review.py', which runs on the
existing jenkins job 'gerrit-auto-critic', to produce gerrit comments.
In the case of webUI scripts, ESLint's code analysis and linting checks
are performed to produce these comments.
As a shared NodeJS installation can be used for JS tests as well as
linting, a seperate common script "bin/nodejs/setup_nodejs.sh"
has been added for assiting with the NodeJS installation.
To ensure quicker run times for the jenkins job, NodeJS tarball is
cached within "${HOME}/.cache" directory, after the initial installation.
ESLint's packages and dependencies have been made to be cached
using NPM's own package management and are also cached locally.
NodeJS and ESLint dependencies are retrieved and executed, only if
there are any changes within ".js" files within the patchset,
and run with minimal overhead.
After analysis, comments are generated for all the violations according
to the specified rules.
A custom formatter has been added to extract, format and filter the
violations in JSON form.
These generated code style violations are formatted into the required
JSON form according to gerrit's REST API, similar to comments generated
by flake8. These are then posted to gerrit as comments
on the respective patchset from jenkins over SSH.
The following code style and quality rules have been added using ESLint.
- Disallow unused variables
- Enforce strict equality (=== and !==)
- Require curly braces for all control statements (if, while, etc.)
- Enforce semicolons at the end of statements
- Enforce double quotes for strings
- Set maximum line length to 90
- Disallow `var`, use `let` or `const`
- Prefer `const` where possible
- Disallow multiple empty lines
- Enforce spacing around infix operators (eg. +, =)
- Disallow the use of undeclared variables
- Require parentheses around arrow function arguments
- Require a space before blocks
- Enforce consistent spacing inside braces
- Disallow shadowing variables declared in the outer scope
- Disallow constant conditions in if statements, loops, etc
- Disallow unnecessary parentheses in expressions
- Disallow duplicate arguments in function definitions
- Disallow duplicate keys in object literals
- Disallow unreachable code after return, throw, continue, etc
- Disallow reassigning function parameters
- Require functions to always consistently return or not return at all
- Enforce consistent use of dot notation wherever possible
- Disallow multiple empty lines
- Enforce spacing around the colon in object literal properties
- Disallow optional chaining, where undefined values are not allowed
The required linting packages have been added as dependencies in the
"www/scripts" directory.
All the test scripts and related dependencies have been moved to -
$IMPALA_HOME/tests/webui/js_tests.
All the custom ESLint formatter scripts and related dependencies
have been moved to -
$IMPALA_HOME/tests/webui/linting.
A combination of NodeJS's 'prefix' argument and NODE_PATH environmental
variable is being used to seperate the dependencies and webUI scripts.
To support running the tests from a remote directory(i.e. tests/webui),
by modifying the required base paths.
The JS scripts need to be updated according to these linting rules,
as per IMPALA-13986.
Change-Id: Ieb3d0a9221738e2ac6fefd60087eaeee4366e33f
Reviewed-on: http://gerrit.cloudera.org:8080/21970
Reviewed-by: Riza Suminto <riza.suminto@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Currently, the scripts in the entire webUI contain variable and function
declarations using the ES5 standard.
In such declarations, all the variables are attached to the browser's
window object, polluting the global scope. Additionally, many unnamed
functions can be represented with cleaner and concise syntax.
This patch refactors such declarations, using the ES6 syntax.
-> Replacing 'var' declarations with 'let' and 'const'
- To improve browser's memory utilization
- For better scoping, immutability and readability
-> Replacing unnamed function declarations with arrow functions
- Better scoping by binding 'this' object to the surrounding context
These improve maintainability and browser's memory utilization.
Across many instances within the webUI scripts, no particular naming
scheme is being followed for variable and function identifiers.
Hence, they have been revised with the following naming scheme.
-> All function names have been declared using camel case.
-> All constant primitive values and strings have been declared
in uppercase.
-> All other types of variables have been declared using snake case.
This naming scheme allows easier distinction between functions,
constants and other variables, based on the identifiers.
These changes to code style are further enforced during code review
through IMPALA-13473.
Change-Id: Ie38f2c642ede14956a2c6d551a58e42538204768
Reviewed-on: http://gerrit.cloudera.org:8080/21851
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
The JSON representation of the aggregated profile or
`AggregatedRuntimeProfile` excludes some instance-level metrics
(e.g. timeseries counters, event sequences, etc) in order to limit
the profile size from growing rapidly.
In contrast, the traditional profile contains associated metrics
from root profiles(`RuntimeProfile`s) of all instances and
the aggregated profile.
The experimental profile only contains the aggregated profile.
Hence, the instance-level metrics are not present.
This patch introduces some of the aggregated instance-level metrics
to the experimental profile's JSON representation without allowing
the profile size to grow rapidly.
It also provides insights into instances with unreported or missing events.
The following attributes have been exposed to the JSON form after grouping
or aggregation.
- Aggregated Event Sequences
- Aggregated Info Strings (i.e. Table Name)
The timestamps across instances for a particular event are grouped
when the number of instances is small and aggregated otherwise,
in order to maintain the profile size and facilitate analysis.
This behavior is controlled by the json_profile_event_timestamp_limit,
which defaults to 5.
For events where the number of timestamps exceeds this limit,
they are grouped into buckets of the same size.
These buckets are spans divided evenly between the minimum and
maximum timestamps for the event. With the default limit of 5,
this results in spans of 20%.
The following aggregates are then calculated for each of these buckets,
to provide a clear and efficient summary of the data.
* Maximum timestamp
* Minimum timestamp
* Average timestamp
* Total no. of instances
The aggregate metrics are calculated with minimal overhead through
assignment to a particular division without the need for sorting,
resulting in a time complexity of O(n) with only two passes through
the entire list of timestamps.
To further optimize performance, the aggregates are computed by
circumventing the need to store each division's timestamps utilizing
only the memory required for a single value per metric, instead of
the entire range of values, while reusing previously allocated vectors.
For efficiently copying the calculated values without internally
reallocating on each insertion, memory is preallocated for each array
of metrics using RapidJSON library.
In the case of missing events, the timestamps are ordered and aligned
through the analysis of 'label_idxs'. If at least one instance contains
a complete set of events, all instances with missing timestamps are
ordered and aligned efficiently by referencing the reordering of labels.
Otherwise, the initial ordering and alignment are retained.
If any fragment instances report only a subset of events due to failure
or error, such instances are reported and only the unavailable timestamps
are skipped during the aggregate metrics calculation, while leveraging
the available timestamps.
The instances containing missing events are further recorded into the
"unreported_event_instance_idxs" field within the event sequence. These
indexes for instances are based on 'exec_params_' set during execution.
Please refer to IMPALA-13555 for further details.
All of the above logic has been encapsulated into the newly added
`ToJson` method within the `AggEventSequence` struct, prioritizing better
reuse and maintainability.
Structure of the `AggEventSequence` in JSON form -
{
profile_name : <PLAN_NODE_NAME>,
num_children : <NUM_CHILDREN>
node_metadata : <NODE_METADATA_OBJECT>
event_sequences :
[{
events : // An example event
[{
label : "Open Started""
ts_list : [ 2257887941, <other instances' timestamps> ]
// OR
ts_stat :
{
min : [ 2257887941, ...<other divisions' minimum timestamps> ],
max : [ 3257887941, ...<other divisions' maximum timestamps> ],
avg : [ 2757887941, ...<other divisions' average timestamps> ]
count : [ 2, ... <other counts of divisions' no. of instances> ]
}
}, <...other plan node's events>
],
// This field is only included, if there are unreported events
unreported_event_instance_idxs : [ 3, 5, 0 ]
}],
counters : <COUNTERS_OBJECT_ARRAY>,
child_profiles : <CHILD_PROFILES>
}
Structure of `AggInfoStrings` in JSON form.
{
profile_name : <PLAN_NODE_NAME>,
num_children : <NUM_CHILDREN>
node_metadata : <NODE_METADATA_OBJECT>
"info_strings" :
[{
"key": "<info string's key>",
"values": [<distinct info string values>]
}]
counters : <COUNTERS_OBJECT_ARRAY>,
child_profiles : <CHILD_PROFILES>
}
Note: In the above structures, unlike a plan node's profile,
a fragment's profile does not contain the 'node_metadata' field.
Added unit tests for the serialization of aggregated metrics -
- Added tests for handling info strings in aggregated JSON profiles
- Introduced AggregatedEventSequenceToJsonTest fixture to validate
event sequence serialization
- Added random profile generation for varied test conditions
- Covered scenarios for complete and missing events in both aggregated
and grouped cases
- Ensured correct JSON structure for info strings and event sequences
- Ensured proper timestamp ordering and aggregation logic in serialized
JSON profiles
Generated the latest expected JSON profile outputs from the
'impala-profile-tool' using the stored impala profile logs.
Added additional tests in tests/observability for profile v2's
JSON output, after inclusion of the new expected JSON profile
formats.
Change-Id: I49e18a7a7e1288e3e674e15b6fc86aad60a08214
Reviewed-on: http://gerrit.cloudera.org:8080/21683
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Representing phases of multiple instances in a single bar with dividers
makes it challenging to distinguish between the perimeters of each
instance's phases, hindering other instances that have already started
with the subsequent phase. Hence, the correlation between the recorded
profile metrics and the user's desired representation is less.
To improve this representation, the phases of each fragment's instances
are now being aggregated and displayed as a histogram on the timeline,
making it easier to understand the distribution of start time and
end time of different instance's phases.
When there are more than 5 instances for a phase, respective timestamps
are bucketed into 5 divisions, each spanning 20% of the difference
between maximum and minimum timestamp value. Each division's timestamps
are aggregated and then maximum value is plotted on the query timeline.
The height of each division's phase rectangle is proportional to
the number of instances in that division. A division without
any instances is not represented.
If the number of instances are less than or equal to 5, for each event,
instance timestamps are sorted in ascending order before plotting.
This results in a descending staircase form of aggregated metrics
display, where maximum timestamps within each of the 20% spanning
divisions is shown.
The precision of SVG elements' position and dimensions have been limited
and are set by the user. They will have the same precision as that of
the timeticks.
The order of rendering phases and iterating through colors for each
plan node has been reversed, in order to support this representation.
A tooltip containing the following additional details is displayed,
when hovering on a plane node's bucketed phase rectangle.
- With respect to the hovered on bucketed phase rectangle
- No. of instances
- Event's Maximum Timestamp (in seconds)
- Event's Minimum Timestamp (in seconds)
- Event's Average Timestamp (in seconds)
Outlines are drawn at the top and bottom of each plan node, according to
the timestamp of the instances' final closing phase.
The attribute 'stroke-dasharray' has been used to add separators between
the perimeters of each instance's phases, instead of an additional
SVG line element each time.
Redundant calls to 'appendChild' methods and setting fragment's ID
after each iteration have been removed.
With this approach, the number of SVG elements in the fragment diagram
is reduced substantially, resulting in very fast rendering times.
Similar to "HASH_JOIN_NODE", the rendering of 2nd and 3rd events
("Waiting for initial build" and "Initial build available")
for the node type "NESTED_LOOP_JOIN_NODE" have also been skipped.
Replaced equality operators with strict equality operators.
Manually tested with various profiles, sizes ranging from 4MB to 430MB.
Change-Id: Ied8a5966e9e4111bf7aa25aee11d23881daad7d2
Reviewed-on: http://gerrit.cloudera.org:8080/21593
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Imported query profiles are currently being stored in IndexedDB.
Although IndexedDB does not have storage limitations like other
browser storage APIs, there is a storage limit for a single
attribute / field.
For supporting larger query profiles, 'pako' compression library's
v2.1.0 has been added along with its associated license.
Before adding query profile JSON to indexedDB, it undergoes compression
using this library.
As compression and parsing profile is a long running process
that can block the main thread, this has been delegated to
a worker script running in the background. The worker script
returns parsed query attributes and compressed profile text sent to it.
The process of compression consumes time; hence, an alert message is
displayed on the queries page warning user to refrain from closing or
reloading the page. On completion, the raw total size, compressed
total size, and total processing time are logged to the browser console.
When multiple profiles are chosen, after each query profile insertion,
the subsequent one is not triggered until compression and insertion
are finished.
The inserted query profile field is decompressed before parsing on
the query plan, query profile, query statement, and query timeline page.
Added tests for the compression library methods utilized by
the worker script.
Manual testing has been done on Firefox 126.0.1 and Chrome 126.0.6478.
Change-Id: I8c4f31beb9cac89051460bf764b6d50c3933bd03
Reviewed-on: http://gerrit.cloudera.org:8080/21463
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
For query profile imports currently the following tabs are supported.
- Query Statement
- Query Timeline
- Query Text Plan
With the current patch "Query Profile" tab will also be supported.
In the "QueryProfileHandler", "query_id" is now added before verifying
its existence in the query log as in "QuerySummaryHandler" and others.
"getQueryID" function has been added to "util.js", as it is helpful
across multiple query pages for retrieving the query ID into JS scripts,
before the page loads up.
On loading the imported "Query Profile" page, query profile download
section and server's non-existing query ID alerts are removed.
All unsupported navbar tabs are removed and current tab is set to active.
The query profile is retrieved from the indexedDB's "imported_queries"
database. Then query profile is passed onto "profileToString" function,
which converts the profile into indented text for displaying on the
profile page.
Each profile and its child profiles are printed in the following order
with the right indentation(fields are skipped, if they do not exist).
Profile name:
- Info strings:
- Event sequences:
- Offset:
- Events:
- Child profile(recursive):
- Counters:
Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc
Reviewed-on: http://gerrit.cloudera.org:8080/21400
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
On horizontally scrolling the query timeline after zooming, the controls
and navigation bar at the top of the page move away from the visibility
area. This makes it difficult to use the controls while scrolling.
The controls and navigation bar now stay within the visibility area even
after scrolling horizontally, allowing for easier navigation.
Hovering on the edges to scroll horizontally is also supported as before.
Change-Id: Ib3b248554ba32766c29faf91c791f2dbbd20641e
Reviewed-on: http://gerrit.cloudera.org:8080/21173
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Fragment diagram's exchange lines were being overlapped as they
were appended after the plan node's phases.
The issue has been solved by appending the plan node's phases in order,
before exchange lines.
Change-Id: I1d37fda1d2e70e46de19b6acdf932463e3f2adbb
Reviewed-on: http://gerrit.cloudera.org:8080/21024
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
A button has been added to import query profiles on the queries page.
The button opens a system dialog to browse only "*.json" files from
client's local file system. This allows multiple JSON query profiles
to be imported.
JSON profiles are read from file system using FileReader API and then
these imported query profiles are stored in a NoSQL database using
browser's IndexedDB API.
If an error is encountered while parsing JSON profile, it is displayed
on the queries page.
IndexedDB is a large-scale NoSQL database system, that can be read
and written to asynchronously. Hence, the file size and number of JSON
profiles is only limited by the storage available on the client system.
The storage limit is browser dependent, in chromium based browsers
upto 80% of local storage may be used by IndexedDB.
The profile is parsed for the following attributes, which will be
displayed on the queries page. Number of imported queries is also
displayed.
- Query ID
- User
- Default Db
- Query Type
- Start Time
- End Time
- Bytes Read
- Bytes Sent
- Query State
- No. of rows fetched
- Resource Pool
- Query Statement
After importing or deleting query profiles, page will scroll back
to the imported query profiles section.
The queries are sorted based on descending order of the "Start Time".
A button is displayed beside the import button to easily clear all
the imported queries at once.
On opening any of the imported queries, its query timeline will be
displayed. Currently the following tabs are supported for imported
queries.
- Query Statement
- Query Timeline
- Query Text Plan
Syntax highlighting on query statement tab has also been supported.
Change-Id: Ife6eb59bf2030fd19fc92aaf134eb51c609e04d0
Reviewed-on: http://gerrit.cloudera.org:8080/20867
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch adds fragment-level metrics to the WebUI query timeline
display along with additional disk and network metrics.
The fragment's plan nodes are enlarged with an animated transition on
hovering over the fragment's row in query timeline's fragment diagram.
On clicking the plan nodes, total thread and memory usage of the parent
fragment are displayed, after accumulating memory and thread usage of
all child nodes. Thread usage is being shown on the additional Y-axis.
In this way, memory and thread usage of multiple fragments can be
compared alongside. A fragment's usage can be hidden by clicking
on any of the child plan nodes again.
These counters are available within the profile with following names.
- MemoryUsage
- ThreadUsage
Once a fragment's metrics are displayed, they are updated as they
are collected from the profile during a running query.
A grid-line is displayed along with a tooltip on hovering over the
fragment diagram, containing the instantaneous time at that position.
This grid-line also triggers tooltips and gridlines in other charts.
A warning is displayed on clicking a fragment with less number of samples
available.
RESOURCE_TRACE_RATIO query option must be set for providing periodic
metrics within the profile. This allows the following time series
counters to be displayed on the query timeline.
- HostDiskWriteThroughput
- HostDiskReadThroughput
- HostNetworkRx
- HostNetworkTx
The additional Y-axis within the utilization chart is used to represent
the average of these metrics.
The memory units in tooltips and ticks on co-ordinate axes are displayed
in human readable form such as KB, MB, GB and PB for convenience.
Both of the charts contain controls to close the chart. These charts
can also be resized until a maximum and minmum limit by dragging the
resize bar's handle.
Along with mouse wheel events, the diagrams can be horizontally
stretched by the help of buttons with horizontal zoom icons at the
top of the page. The zoom out button is disabled, when further zoom out
is not possible.
Timeticks are being autoscaled during fragment diagram's horizontal zoom.
In addition to the scrollbar, hovering on edges of the window allows
horizontal scrolling.
Test cases have been for the additional disk, network and fragment level
memory metrics parsing functions.
Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f
Reviewed-on: http://gerrit.cloudera.org:8080/20355
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
To help with testing the query timeline web page, the tests have been
integrated using JEST testing framework based on nodejs.
'run_js_tests.sh' runs JEST test suits, after it fetches nodejs binaries
and related nodejs packages, if they are not present locally.
NodeJS binaries are stored within the ${IMPALA_TOOLCHAIN} directory.
'jest-environment-jsdom' supports rendering DOM elements and
testing them without the requirement of a browser.
For implementing unit and integration tests, the script
has been divided into multiple properly functioning modules,
that are imported as ES6 modules.
Unit tests have been written for the following query profile parsing
functions.
- mapTimeseriesCounters() - Test whether the method correctly searches
and maps the profile's counter's indexes based on counter_name even
in reverse order
- accumulateTimeseriesValues() - Test whether the method correctly
accumlates values after parsing values from 'data' in
'time_series_counters' while updating 'max_samples'
- generateTimesamples() - Test whether time sample values generated
based on 'max_samples' are correct, with different 'max_samples'
- clearTimeseriesValues() - Test whether Timeseries arrays are being
properly truncated in the correct range
- initializeUtilizationMetrics() - Test whether aggregate arrays and
time sample arrays are correctly allocated based on counters and
'max_samples'
- getSvg*() - Test whether all getSvg methods are correctly setting
attributes and returning expected attributes in elements
Unit tests produce JUnitXML for integration with jenkins jobs,
these are stored in ${IMPALA_JS_TEST_LOGS_DIR}.
Change-Id: I0caf0a0beee23821f78c0b3fe1aeb7dbf92d6a3e
Reviewed-on: http://gerrit.cloudera.org:8080/20538
Reviewed-by: Wenzhe Zhou <wzhou@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch divides the query timeline script into multiple modules
for better maintainability and to help writing unit tests.
The patch also improves parsing of utilization values from the
query profile, as the order and name of counters may change within
the profile, to ensure appropriate mappings between plotted readings,
initially the profile is traversed to find the correct mapping.
Change-Id: Id9d62a799d838876509686a75ecea778b2c72cc6
Reviewed-on: http://gerrit.cloudera.org:8080/20588
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Add functions in CatalogUsageMonitor to monitor and report the
catalog usage of the tables have the longest metadata loading
time(Including maximum, median, 75th-ile, 95th-ile, 99th-ile time).
Set default tables loading metrics capacity to 100.
However, there might be a problem here because we only keep the
capacity size to 100. For example, there might be case like a
table has higher median loading time but has lower Maximum
loading time which cannot make itself to the Top-100. For now,
we will ignore case like that because we are aiming to find
the tables with maximum longest loading time.
Add the sorted table in Catalog server web-ui. The loading
time is sorted by the maximum from load_duration metrics. But
users can sort by other metrics in catalogd debug UI.
Testing:
- Add end-to-end test for webpage to verify the label and text
exist in catalog debug page. Verify all fields are in JSON response
- Launch Impala and activate some tables to see the table loading
time shown successfully on the catalog debug UI page.
Change-Id: I9305a867d7053cde9acc42dae6e47ee440f1a8bf
Reviewed-on: http://gerrit.cloudera.org:8080/14600
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch reuse the JS function introduced in IMPALA-6966,
which can convert Long size value into human readable size,
to render the "Peak mem. consumption" column in the Query
details's Backends page.
Change-Id: I04afb4091bb8b6bc9dedfeceaf9284a8c65b16a1
Reviewed-on: http://gerrit.cloudera.org:8080/10523
Reviewed-by: Philip Zeyliger <philip@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
