Re: [libreoffice-users] Re: LOcalc shows an inaccurate output when use cells autofill function

On Wed, 2011-10-05 at 15:54 -0700, Pedro wrote: 

> Steve Edmonds wrote:
> > Thats interesting. In Calc I get 16.4300000001676 and in Kcalc I get
> > 16.43 (to 20 decimal places)
>
> That is expected. Because calculators have a limited number of functions
> they use finite precision (instead of the base 2 calculations)
>
> This means that for this kind of calculations you are better off using a
> cheap calculator than your powerful PC with a spreadsheet :)

The issue is the internal representation of floating point numbers. But
often the machine errors are much less than the noise in the data. In
terms of significant figures, the correct answer is 16.43 not
16.43000-00?. The extra digits beyond the 3 are dropped for significant
figures. One should actually display only the correct significant
figures in the final answer. 

> --
> View this message in context: 
> http://nabble.documentfoundation.org/LOcalc-shows-an-inaccurate-output-when-use-cells-autofill-function-tp3397081p3398248.html
> Sent from the Users mailing list archive at Nabble.com.



-- 
Jay Lozier
jslozier@gmail.com

-- 
For unsubscribe instructions e-mail to: users+help@global.libreoffice.org
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/global/users/
All messages sent to this list will be publicly archived and cannot be deleted