Hi,
I'm working on some improvements of Calc's MOD function, but I struggle with some unexpected
results when using large integer values.
As I understand IEEE754, double precision means a mantissa of 53 bits, so the maximum correct
unsigned integer value would be 2^53.
But when I try 3^31 (which is less than 2^52) and subtract 3^31-1, 3^31-2, 3^31-3 (resp. 1, 2 and 3
less than 3^31) I don't get 1, 2 and 3 as result, but 0, 0 and 3.
This test is simply in Calc with e.g. =(3^31)-((3^31)-1).
I use a 64bit machine and master-version of LO.
What am I doing wrong here?
Winfried
Context
- IEE754 and maximum correct integer value · Winfried Donkers
Privacy Policy |
Impressum (Legal Info) |
Copyright information: Unless otherwise specified, all text and images
on this website are licensed under the
Creative Commons Attribution-Share Alike 3.0 License.
This does not include the source code of LibreOffice, which is
licensed under the Mozilla Public License (
MPLv2).
"LibreOffice" and "The Document Foundation" are
registered trademarks of their corresponding registered owners or are
in actual use as trademarks in one or more countries. Their respective
logos and icons are also subject to international copyright laws. Use
thereof is explained in our
trademark policy.