Date: prev next · Thread: first prev next last
2014 Archives by date, by thread · List index

At 06:21 03/02/2014 +0100, Erik Erlandsson wrote:
I'm seeing this "odd" behavior in a series. Around line 30 additional decimals are showing up What am I doing wrong?

At 20:29 03/02/2014 +0100, Erik Erlandsson (Nilhe AB) wrote:
I hope this works:

I'm having to guess, as you've posted a screenshot but not the spreadsheet itself. But your column appears to be calculated by an iterative formula, with each row calculated from the previous one. As you do this, rounding errors will be introduced, and these will increase in size as you double the value at each step. Since you take only the fractional part of the result, the number in each row does not generally grow in size, so the rounding errors eventually become significant compared with the values you have.

The calculation is probably being carried out internally to around fifteen significant figures. When I try this calculation, rounding errors appear at row 8, but only in the fifteenth significant figure. This will be happening in your calculation too, but becomes visible only in row 26, where the difference begins to show in the nine significant figures you chose to display. Note that the rounding will occur in the binary numbers being used in the calculation by your computer hardware, not in the decimal values being displayed. Your results all terminate after the first fractional place in decimal, but they will not do so in binary, where even 0.1 is the recurring fraction 0.0[0011] with those last four digits repeating.

You will see different results - perhaps even those you seek - if you tick Tools | Options... | LibreOffice Calc | Calculate | Precision as shown. This causes the displayed value you see to be used in each calculation instead of the true value hidden in the cell.

Of course, you could apply some mathematics and find a simpler way of calculating these values - which, after the first, simply repeat the sequence 0.2, 0.4, 0.8, and 0.6. One example is:
- but this will go awry at row 52, since 2^51 is so large (around fifteen digits) that its units digit is no longer reliable.

So here's a better version:
- which should work more or less indefinitely, though not for the (exceptional) first row.

I trust this helps.

Brian Barker

To unsubscribe e-mail to:
Posting guidelines + more:
List archive:
All messages sent to this list will be publicly archived and cannot be deleted


Privacy Policy | Impressum (Legal Info) | Copyright information: Unless otherwise specified, all text and images on this website are licensed under the Creative Commons Attribution-Share Alike 3.0 License. This does not include the source code of LibreOffice, which is licensed under the Mozilla Public License (MPLv2). "LibreOffice" and "The Document Foundation" are registered trademarks of their corresponding registered owners or are in actual use as trademarks in one or more countries. Their respective logos and icons are also subject to international copyright laws. Use thereof is explained in our trademark policy.