The MemTracker is removed from StatestoreD main and CatalogD main as
it is not used. The /memz page related metrics are refactored to a
a separate file. When the MemTracker is not available during callback
initialization it should not be part of the callback either. Therefore,
when MemTracker is not part of the callback it is not displayed.
Before/after pictures of the /memz page can be found in the Jira.
Change-Id: I1a42434b98aec1d9e0be6ae52325049378fdde23
Reviewed-on: http://gerrit.cloudera.org:8080/13670
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This is meant to help understand how the process is using memory, to
answer questions like:
* Is the Impala daemon using transparent huge pages?
* What is the system THP configuration?
* What is the RSS of the process?
* What is the virtual memory size of the process?
Most of these questions can be answered via other command line or
monitoring, except for the THP usage by the process, but adding
metrics helps consolidate the information in one place.
This commit adds a memory maintenance thread that periodically scrapes
information from the /proc and /sys filesystems and updates the
metrics.
The interfaces used are:
* /proc/<pid>/smaps, which is a documented interface to get detailed
information about mapped memory:
https://www.kernel.org/doc/Documentation/filesystems/proc.txt
* /sys/kernel/mm/ config values for transparent huge pages
Change-Id: I13873e305ba464d11dea0d7244a29ff4f332f1a9
Reviewed-on: http://gerrit.cloudera.org:8080/7472
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Impala Public Jenkins
A template parser issue means that in certain circumstances the JVM
memory metrics would not be rendered to /memz. This patch works around
the parser bug by rewriting the template not to use the problematic
nested construct.
Change-Id: Ie3f01a5cfd3c1c226532bab13895d922f05e94ee
Testing: Checked /memz on statestore, catalog, impalad.
Reviewed-on: http://gerrit.cloudera.org:8080/7106
Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com>
Tested-by: Impala Public Jenkins
Misc changes to improve usability of the profiles.
* Separate out detailed BufferPool metrics into a "Buffer pool"
sub-profile.
* Only create the limit counter if there is a limit
* Show BufferPool using in query MemTracker (it was accidentally
disabled before because there was no query-level profile).
* Reduce clutter in MemTracker dump by only showing buffer pool
reservation, not usage (the usage was misleading anyway because
it didn't include child usage).
* Remove TotalUnpinnedBytes, which had limited value - WriteIoBytes
and PeakUnpinnedBytes can answer most of the same questions - i.e.
did it unpin any pages, and how many did it need to write to disk.
* Add buffer pool metrics to /memz (if buffer pool is enabled) and
reorder /memz so more useful information is up the top.
Change-Id: I34b7f4d94c3d396ac89026c7559d6b2c6e02697c
Reviewed-on: http://gerrit.cloudera.org:8080/6690
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Impala Public Jenkins
This commit adds heap and non-heap memory usage of the embedded
JVM in the memory metrics and exposes these metrics in /memz web page of
the impalad and catalog web UI.
Change-Id: I543d4d428d7240e0f710d67973867162f2fcabc8
Reviewed-on: http://gerrit.cloudera.org:8080/5909
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: Impala Public Jenkins
The /memz page tried to add JVM metrics even when they didn't exist for
all daemons, not just Impala. This led to a crash when they tried to
access ExecEnv::GetInstance() without an initialised ExecEnv at
statestored and catalogd
To fix, changed the memz handler method to take an optional metric
group, provided by the caller. memz handler will check the existence of
the metric group.
Used C++11 lambdas rather than boost::bind to help simplify the code.
Testing: Ran locally and looked at impalad/memz, statestored/memz
and catalogd/memz
Add a test file test_web_pages.py to test sending request to /memz on
impalad / statestored / catalogd
Change-Id: If59b10f20044d1a468f27810a3029fe18fb19f29
Reviewed-on: http://gerrit.cloudera.org:8080/3998
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Internal Jenkins
For files that have a Cloudera copyright (and no other copyright
notice), make changes to follow the ASF source file header policy here:
http://www.apache.org/legal/src-headers.html#headers
Specifically:
1) Remove the Cloudera copyright.
2) Modify NOTICE.txt according to
http://www.apache.org/legal/src-headers.html#notice
to follow that format and add a line for Cloudera.
3) Replace or add the existing ASF license text with the one given
on the website.
Much of this change was automatically generated via:
git grep -li 'Copyright.*Cloudera' > modified_files.txt
cat modified_files.txt | xargs perl -n -i -e 'print unless m#Copyright.*Cloudera#i;'
cat modified_files_txt | xargs fix_apache_license.py [1]
Some manual fixups were performed following those steps, especially when
license text was completely missing from the file.
[1] https://gist.github.com/anonymous/ff71292094362fc5c594 with minor
modification to ORIG_LICENSE to match Impala's license text.
Change-Id: I2e0bd8420945b953e1b806041bea4d72a3943d86
Reviewed-on: http://gerrit.cloudera.org:8080/3779
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
Get jvm metrics by global handler ExecEnv
and filter out items with "total"
Testing: Ran locally and looked at memz page
Change-Id: Ib44c25eb5a5d70f10a6a120501eec2d50fad5ce9
Reviewed-on: http://gerrit.cloudera.org:8080/3625
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: Internal Jenkins
This patch changes all remaining non-templated webpages to use the
templating engine to produce their HTML and text output. The main
motivation for this change is to make it much harder for unescaped output
to appear in HTML pages, where it might be interpreted by the browser as
e.g. JavaScript. The template engine automatically escapes all text.
As a side-effect, moving everything to the templating engine allows us
to remove the 'HTML' path from the webserver. Every callback now
produces Json and is rendered via a template.
This patch also changes the mechanism via which pages are presented to
the browser as text, rather than HTML. Any URL that includes the
argument '?raw' will automatically be rendered as text, not HTML, but
still passes through the template process. Any callback that wishes to
ensure that is only ever rendered as text (e.g. /jsonmetrics) has only
to set the special '__raw__' key in its Json output.
Finally, this patch removes the 'is_styled' parameter to
Register*Callback(); this logic belongs in the template, not inside the
webserver itself.
Change-Id: Ia33df77c52ebfa8125c5cf1fbcf40ea4b1da5497
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3810
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins
