Date: prev next · Thread: first prev next last
2016 Archives by date, by thread · List index

Perhaps the BASIC code was only displaying the number to a few decimal places. In C, printf defaults to 6 decimal places, which isn't enough to show the actual result as any different from the ideal result.

It's also possible that the spreadsheet was doing something more complicated internally than just 28*(1/7). Playing around with an example in C, I can see the error in representing (1/7), but multiplying that by 28 gives exactly 4:
                Decimal                           In-memory
        (1/7) = 0.142857149243354797363281250000  0x3e124925
     28*(1/7) = 4.000000000000000000000000000000  0x40800000

On the other hand, storing 1/490 in a float, then storing the result of multiplying that by 70 (should be the same as 1/7), then storing the result of multiplying that by 28, gives 3.99999976...:
                Decimal                           In-memory
      (1/490) = 0.002040816238150000572204589844  0x3b05bf37
   70*(1/490) = 0.142857134342193603515625000000  0x3e124924
28*70*(1/490) = 3.999999761581420898437500000000  0x407fffff

I'm not suggesting that's exactly what the spreadsheet was doing, but it shows that slight changes to the detail of the calculation (which wouldn't affect the result with unlimited precision) can have an effect on whether the result is equal to or slightly more or less than the expected result.


Virgil Arrington wrote:
Thanks for the explanation. Oddly enough, it makes some sense, even to
my non-techy brain. What I found interesting was that, in the
spreadsheet, I got the "wrong" 3.9999, but I wrote a simple BASIC
program on my beloved Tandy Model 100 and got the "correct" answer of 4.
I was excited to see that my little first generation laptop gave better
results than my state of the art (for the time) PC.


On 02/10/2016 06:01 PM, wrote:
Virgil Arrington wrote:
Ken Springer wrote
I remember years ago when Intel turned out a chip that had an error in
it's math calculations.  It was a rare happening, but when they
finally admitted it publicly, trying to say it wasn't important do to
the rare occurrence, it did not go over well at all! <G>

About 25 years ago, I was the treasurer of my children's preschool. I
created a spreadsheet to calculate paychecks, and I found that the
paycheck was consistently off by .01 (a penny). It drove me nuts. As it
turned out, one part of the calculation required the division of 28 by
7, which every third grader knows is 4. Well, my spreadsheet gave an
answer of 3.9999999999_. By itself, it wasn't a big problem, but later
in the chain of operations, the 3.99999_ produced a result that rounded
*down* to the nearest penny instead of *up*, which it would have done if
the 28/7 had resulted in 4 instead of 3.9999. I complained to a computer
friend of mine who tried to explain that the computer's answer was more
"precise" than my mental math of 28/7=4. I didn't buy it.

I wouldn't say 3.9999999... is more precise, but it sounds like your
problem is related to the precision of floating-point numbers. 1/7,
when represented in binary, is a recurring fraction 0.001001001...
(like how 1/3 is 0.3333... in decimal), so cannot be represented
precisely in binary with a finite number of bits (just as you can add
as many '3's as you like to 0.3333, but it still doesn't exactly
represent 1/3).

I don't think there should be a problem with calculating 28/7 = 4 as a
single floating point operation, but if the calculation was done
(either in the way your formula was expressed or the way the
application processed it) as (1/7)*28, that may well give a slightly
inaccurate result due to rounding of the (1/7). A bit like calculating
1/3 to 0.333, then multiplying that by 12 to get 3.996 instead of 4.
If using a round-down function, and the result was slightly less than
4, it would round down to 3. I guess the solution was to round to the
nearest integer.

I learned a valuable lesson in blindly accepting a computer's
calculation simply because it was made by a computer.

Indeed. They do exactly what they're told by a combination of
hardware, software and user input - but for various reasons that might
not amount to what you thought you were telling it to do.


To unsubscribe e-mail to:
Posting guidelines + more:
List archive:
All messages sent to this list will be publicly archived and cannot be deleted


Privacy Policy | Impressum (Legal Info) | Copyright information: Unless otherwise specified, all text and images on this website are licensed under the Creative Commons Attribution-Share Alike 3.0 License. This does not include the source code of LibreOffice, which is licensed under the Mozilla Public License (MPLv2). "LibreOffice" and "The Document Foundation" are registered trademarks of their corresponding registered owners or are in actual use as trademarks in one or more countries. Their respective logos and icons are also subject to international copyright laws. Use thereof is explained in our trademark policy.