Since resolve_minidumps.py's call to minidump_stackwalk can go haywire
due to bad symbols in shared libraries, this adds a fallback mechanism
where it tries again with a "safe" list of shared libraries. These are
limited to the ones that make the most difference in resolving minidumps
(libc, libstdc++, and libjvm). The list of safe libraries can be
customized via the --safe_library_list.
Testing:
- Verified that this uses the fallback on Centos 7 and resolves
the minidumps successfully.
Change-Id: I6bb4c9f65f9c27bb3b86c7ff2f3a6a48e258ef01
Reviewed-on: http://gerrit.cloudera.org:8080/20863
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Tested-by: Joe McDonnell <joemcdonnell@cloudera.com>
On some platforms (Centos 7), resolve_minidumps.py's call to
minidump_stackwalk goes haywire and uses all the system memory
until it gets OOM killed. Some library must have corrupt
symbols, etc. As a workaround, this detects whether the
prlimit utility is present and uses this to run minidump_stackwalk
with a 4GB limit on virtual memory. This kills the process
earlier and avoids using all system memory.
Testing:
- Verified that bin/jenkins/finalize.sh uses resolve_minidumps.py
on a Redhat 8 Jenkins job (and it works)
- Verified that bin/jenkins/finalize.sh works properly on
my Ubuntu 20 box
- Ran a Jenkins job on Centos 7 and verified that the prlimit
code kills minidump_stackwalk when it uses 4GB of memory.
Change-Id: I4db8facb8a037327228c3714e047e0d1f0fe1d94
Reviewed-on: http://gerrit.cloudera.org:8080/20862
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Tested-by: Joe McDonnell <joemcdonnell@cloudera.com>
resolve_minidumps.py uses /bin/env. Some distributions
support that, but that path does not exist on Ubuntu 16.
This switches to use /usr/bin/env, which is available
everywhere.
Change-Id: Iecde8713bc141f2a15babf3886848eacad8910aa
Reviewed-on: http://gerrit.cloudera.org:8080/19225
Reviewed-by: Michael Smith <michael.smith@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This adds the resolve_minidumps.py script to
simplify resolving minidumps under ideal circumstances.
This is designed to handle cases where the binary
and libraries are in identical locations to when
the minidump was created. This is true for developer
environments and at the end of Jenkins jobs.
This uses Breakpad's minidump_dump utility to get a
list of the binaries/libraries that the minidump
references. It uses that list to dump all the
symbols to a temporary directory. Then it uses
the symbols to resolve the minidump.
Since it is dumping symbols for all referenced
libraries, it resolves symbols to the maximum
extent possible.
This adds a step to bin/jenkins/finalize.sh to use
this new script to resolve minidumps. The old method
can be removed in a subsequent change.
Testing:
- Ran locally on a minidump generated by sending
SIGUSR1 to local impalad
- Tested with a Centos 7 job using Python 3.6
and verified the minidump output
- Tested resolving a minidump from a binary with
compressed debug info
Change-Id: I0f8fdcb8ca89d0904dc8ec69337e3d5dfdd54adf
Reviewed-on: http://gerrit.cloudera.org:8080/18918
Reviewed-by: Wenzhe Zhou <wzhou@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
