This takes steps to make Python 2 behave like Python 3 as
a way to flush out issues with running on Python 3. Specifically,
it handles two main differences:
1. Python 3 requires absolute imports within packages. This
can be emulated via "from __future__ import absolute_import"
2. Python 3 changed division to "true" division that doesn't
round to an integer. This can be emulated via
"from __future__ import division"
This changes all Python files to add imports for absolute_import
and division. For completeness, this also includes print_function in the
import.
I scrutinized each old-division location and converted some locations
to use the integer division '//' operator if it needed an integer
result (e.g. for indices, counts of records, etc). Some code was also using
relative imports and needed to be adjusted to handle absolute_import.
This fixes all Pylint warnings about no-absolute-import and old-division,
and these warnings are now banned.
Testing:
- Ran core tests
Change-Id: Idb0fcbd11f3e8791f5951c4944be44fb580e576b
Reviewed-on: http://gerrit.cloudera.org:8080/19588
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Tested-by: Joe McDonnell <joemcdonnell@cloudera.com>
This patch adds a wrapper around ThriftDebugString method provided
in the Thrift library. The thrift's method can throw exceptions
like (bad_alloc or TProtocolException) when the object cannot be
serialized into a string representation. This exception is not
caught on the catalogd side and it crashes the catalogd.
The error was specifically seen in the catalogd's debug UI
which provides a way to display a Table object. An exception
thrown when rendering the table on the UI would have crashed
the catalogd before the patch. In order to simulate this crash a new debug
action called EXCEPTION was added. A new custom cluster test
was added which simulates a exception thrown in this method and
makes sure that fetching the table from catalogd's debug UI
does not crash the catalogd.
Tests:
1. Added a new custom cluster test which reproduces the crash.
2. Created a large table which has ~270K partitions and reduced
the memory of the catalogd to 16GB. This configuration throws
bad_alloc exception in the ThriftDebugString method and crashes
the catalogd. After the patch the crash is averted and we see
a error message on the debug UI instead. I also looped around
the catalog web UI call for more than an hour to see if there
are any other stability issues. I could not see any problems.
Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66
Reviewed-on: http://gerrit.cloudera.org:8080/17110
Reviewed-by: Vihang Karajgaonkar <vihang@cloudera.com>
Tested-by: Vihang Karajgaonkar <vihang@cloudera.com>
