Date: prev next · Thread: first prev next last
2016 Archives by date, by thread · List index

Virgil Arrington wrote:
Ken Springer wrote
I remember years ago when Intel turned out a chip that had an error in
it's math calculations.  It was a rare happening, but when they
finally admitted it publicly, trying to say it wasn't important do to
the rare occurrence, it did not go over well at all!  <G>

About 25 years ago, I was the treasurer of my children's preschool. I
created a spreadsheet to calculate paychecks, and I found that the
paycheck was consistently off by .01 (a penny). It drove me nuts. As it
turned out, one part of the calculation required the division of 28 by
7, which every third grader knows is 4. Well, my spreadsheet gave an
answer of 3.9999999999_. By itself, it wasn't a big problem, but later
in the chain of operations, the 3.99999_ produced a result that rounded
*down* to the nearest penny instead of *up*, which it would have done if
the 28/7 had resulted in 4 instead of 3.9999. I complained to a computer
friend of mine who tried to explain that the computer's answer was more
"precise" than my mental math of 28/7=4. I didn't buy it.

I wouldn't say 3.9999999... is more precise, but it sounds like your problem is related to the precision of floating-point numbers. 1/7, when represented in binary, is a recurring fraction 0.001001001... (like how 1/3 is 0.3333... in decimal), so cannot be represented precisely in binary with a finite number of bits (just as you can add as many '3's as you like to 0.3333, but it still doesn't exactly represent 1/3).

I don't think there should be a problem with calculating 28/7 = 4 as a single floating point operation, but if the calculation was done (either in the way your formula was expressed or the way the application processed it) as (1/7)*28, that may well give a slightly inaccurate result due to rounding of the (1/7). A bit like calculating 1/3 to 0.333, then multiplying that by 12 to get 3.996 instead of 4. If using a round-down function, and the result was slightly less than 4, it would round down to 3. I guess the solution was to round to the nearest integer.

I learned a valuable lesson in blindly accepting a computer's
calculation simply because it was made by a computer.

Indeed. They do exactly what they're told by a combination of hardware, software and user input - but for various reasons that might not amount to what you thought you were telling it to do.


To unsubscribe e-mail to:
Posting guidelines + more:
List archive:
All messages sent to this list will be publicly archived and cannot be deleted


Privacy Policy | Impressum (Legal Info) | Copyright information: Unless otherwise specified, all text and images on this website are licensed under the Creative Commons Attribution-Share Alike 3.0 License. This does not include the source code of LibreOffice, which is licensed under the Mozilla Public License (MPLv2). "LibreOffice" and "The Document Foundation" are registered trademarks of their corresponding registered owners or are in actual use as trademarks in one or more countries. Their respective logos and icons are also subject to international copyright laws. Use thereof is explained in our trademark policy.