Virgil Arrington wrote:
Ken Springer wrote
I remember years ago when Intel turned out a chip that had an error in
it's math calculations. It was a rare happening, but when they
finally admitted it publicly, trying to say it wasn't important do to
the rare occurrence, it did not go over well at all! <G>
About 25 years ago, I was the treasurer of my children's preschool. I
created a spreadsheet to calculate paychecks, and I found that the
paycheck was consistently off by .01 (a penny). It drove me nuts. As it
turned out, one part of the calculation required the division of 28 by
7, which every third grader knows is 4. Well, my spreadsheet gave an
answer of 3.9999999999_. By itself, it wasn't a big problem, but later
in the chain of operations, the 3.99999_ produced a result that rounded
*down* to the nearest penny instead of *up*, which it would have done if
the 28/7 had resulted in 4 instead of 3.9999. I complained to a computer
friend of mine who tried to explain that the computer's answer was more
"precise" than my mental math of 28/7=4. I didn't buy it.
I wouldn't say 3.9999999... is more precise, but it sounds like your
problem is related to the precision of floating-point numbers. 1/7,
when represented in binary, is a recurring fraction 0.001001001...
(like how 1/3 is 0.3333... in decimal), so cannot be represented
precisely in binary with a finite number of bits (just as you can add
as many '3's as you like to 0.3333, but it still doesn't exactly
represent 1/3).
I don't think there should be a problem with calculating 28/7 = 4 as a
single floating point operation, but if the calculation was done
(either in the way your formula was expressed or the way the
application processed it) as (1/7)*28, that may well give a slightly
inaccurate result due to rounding of the (1/7). A bit like calculating
1/3 to 0.333, then multiplying that by 12 to get 3.996 instead of 4.
If using a round-down function, and the result was slightly less than
4, it would round down to 3. I guess the solution was to round to the
nearest integer.
I learned a valuable lesson in blindly accepting a computer's
calculation simply because it was made by a computer.
Indeed. They do exactly what they're told by a combination of
hardware, software and user input - but for various reasons that might
not amount to what you thought you were telling it to do.
Mark.