StringToFloatInternal is used to parse string into float. It had logic
to ensure it is faster than standard functions like strtod in many
cases, but it was not as accurate. We are replacing it by a third
party library named fast_double_parser which is both fast and doesn't
sacrifise the accuracy for speed. On benchmarking on more than
1 million rows where string is cast to double, it is found that new
patch is on par with the earlier algorithm.
Results:
W/O library: Fetched 1222386 row(s) in 32.10s
With library: Fetched 1222386 row(s) in 31.71s
Testing:
1. Added test to check for accuracy improvement.
2. Ran existing Backend tests for correctness.
Change-Id: Ic105ad38a2fcbf2fb4e8ae8af6d9a8e251a9c141
Reviewed-on: http://gerrit.cloudera.org:8080/17389
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Original approach to convert DecimalValue(internal representation
of decimals) to double was not accurate.
It was:
static_cast<double>(value_) / pow(10.0, scale).
However only integers from −2^53 to 2^53 can be represented
accurately by double precision without any loss.
Hence, it would not work for numbers like -0.43149576573887316.
For DecimalValue representing -0.43149576573887316, value_ would be
-43149576573887316 and scale would be 17. As value_ < -2^53,
result would not be accurate. In newer approach we are using third
party library https://github.com/lemire/fast_double_parser, which
handles above scenario in a performant manner.
Testing:
1. Added End to End Tests covering following scenarios:
a. Test to show precision limitation of 16 in the write path
b. DecimalValue's value_ between -2^53 and 2^53.
b. value_ outside above range but abs(value_) < UINT64_MAX
c. abs(value_) > UINT64_MAX -covers DecimalValue<__int128_t>
2. Ran existing backend and end-to-end tests completely
Change-Id: I56f0652cb8f81a491b87d9b108a94c00ae6c99a1
Reviewed-on: http://gerrit.cloudera.org:8080/17303
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This patch reserves SQL:2016 reserved words, excluding:
1. Impala builtin function names.
2. Time unit words(year, month, etc.).
3. An exception list based on a discussion.
Some test cases are modified to avoid these words. A impalad and
catalogd startup option reserved_words_version is added. The words are
reserved if the option is set to "3.0.0".
Change-Id: If1b295e6a77e840cf1b794c2eb73e1b9d2b8ddd6
Reviewed-on: http://gerrit.cloudera.org:8080/9096
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Reviewed-by: Philip Zeyliger <philip@cloudera.com>
Tested-by: Impala Public Jenkins
In this commit we enable Decimal_V2 by default. We also update the
expected results in many of our tests.
Testing:
Ran an exhaustive test which almost passed. Updated the few failed
tests in it.
Cherry-pick: not for 2.x
Change-Id: Ibbdd05bf986b7947f106b396017faa3a0bd87fd7
Reviewed-on: http://gerrit.cloudera.org:8080/9062
Reviewed-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com>
Tested-by: Impala Public Jenkins
When multiplying double and decimal values, we used to cast all doubles
to decimals before doing the multiplication. Due to the precision of two
decimals being added during multiplication, the effective value range of
the resulting decimal type could become very small and overflows could
happen.
This change switches the behavior to cast to double precision types when
at least one of the input operands is of type float or double. In such
cases we will not have exact results in general and we assume the user
would normally not expect exact results from an inherently inexact
datatype.
Change-Id: Idd28c5471506c68a860beb0778d98c8d25825f9f
Reviewed-on: http://gerrit.cloudera.org:8080/1820
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: Internal Jenkins
Our .test file parser used to not abort tests when there
is a malformed test/section. This patch changes that behavior
to report an error and treat the test as failed.
Quite a few tests were not well-formed, and were not executed
as a result. This patch fixes those tests.
Arguably, the test file parser should be more flexible in which places
to accept comments, but this patch does not address that problem.
Change-Id: If53358eb0cb958b68e51940b071e64c1d6c3ec6f
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/5468
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Float/Doubles are lossy so using those as the default literal type
is problematic.
Change-Id: I5a619dd931d576e2e6cd7774139e9bafb9452db9
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2758
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
