Steve On 2016-02-11 12:14, Virgil Arrington wrote:
Thanks for the explanation. Oddly enough, it makes some sense, even to my non-techy brain. What I found interesting was that, in the spreadsheet, I got the "wrong" 3.9999, but I wrote a simple BASIC program on my beloved Tandy Model 100 and got the "correct" answer of 4. I was excited to see that my little first generation laptop gave better results than my state of the art (for the time) PC.Virgil On 02/10/2016 06:01 PM, libreoffice-ml.mbourne@spamgourmet.com wrote:Virgil Arrington wrote:Ken Springer wroteI remember years ago when Intel turned out a chip that had an error in it's math calculations. It was a rare happening, but when they finally admitted it publicly, trying to say it wasn't important do to the rare occurrence, it did not go over well at all! <G>About 25 years ago, I was the treasurer of my children's preschool. I created a spreadsheet to calculate paychecks, and I found that the paycheck was consistently off by .01 (a penny). It drove me nuts. As it turned out, one part of the calculation required the division of 28 by 7, which every third grader knows is 4. Well, my spreadsheet gave an answer of 3.9999999999_. By itself, it wasn't a big problem, but later in the chain of operations, the 3.99999_ produced a result that rounded*down* to the nearest penny instead of *up*, which it would have done if the 28/7 had resulted in 4 instead of 3.9999. I complained to a computerfriend of mine who tried to explain that the computer's answer was more "precise" than my mental math of 28/7=4. I didn't buy it.I wouldn't say 3.9999999... is more precise, but it sounds like your problem is related to the precision of floating-point numbers. 1/7, when represented in binary, is a recurring fraction 0.001001001... (like how 1/3 is 0.3333... in decimal), so cannot be represented precisely in binary with a finite number of bits (just as you can add as many '3's as you like to 0.3333, but it still doesn't exactly represent 1/3).I don't think there should be a problem with calculating 28/7 = 4 as a single floating point operation, but if the calculation was done (either in the way your formula was expressed or the way the application processed it) as (1/7)*28, that may well give a slightly inaccurate result due to rounding of the (1/7). A bit like calculating 1/3 to 0.333, then multiplying that by 12 to get 3.996 instead of 4. If using a round-down function, and the result was slightly less than 4, it would round down to 3. I guess the solution was to round to the nearest integer.I learned a valuable lesson in blindly accepting a computer's calculation simply because it was made by a computer.Indeed. They do exactly what they're told by a combination of hardware, software and user input - but for various reasons that might not amount to what you thought you were telling it to do.Mark.
-- To unsubscribe e-mail to: users+unsubscribe@global.libreoffice.org Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://listarchives.libreoffice.org/global/users/ All messages sent to this list will be publicly archived and cannot be deleted