[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [libreoffice-users] a little math problem with Calc
- Subject: Re: [libreoffice-users] a little math problem with Calc
- From: Regina Henschel <email@example.com>
- Date: Fri, 06 Jul 2012 01:27:54 +0200
- To: firstname.lastname@example.org
alex sato schrieb:
I have a problem with functions I made.
Let's say I create the function:
Function Test (n)
Test = r
Then, in a empty spreadsheet take the following steps:
-Insert the formula =TEST(E1)
-insert 11,18 in E1 and see the result in cell D1
I think the result should be 18 but I'm getting 17
Where is the error?
It is the normal behavior of data type double. 11,18 has indefinite digits in dual system. So the stored value is cut somewhere. Because there are enough binary digits to round the dual number to 15 decimal digits, you do not notice it. But when you subtract the integral part, the others digits will shift left and new digits will be generated at the end, which does not represent the original value. The calculated difference becomes a little bit to small. After multiply with 100 you do not have 18,00 but 17,99... And that results in 17. The problem becomes more visible, when the integral part is larger.
You should not use Basic functions, but use Calc functions instead. They are often tweaked to avoid such problems. You can call Calc functions inside your Macro.
To get the first two decimal digits as integer value, it seems to be more accurate to use
r(3)=(n * 100) mod 100
in your macro.
For unsubscribe instructions e-mail to: email@example.com
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/global/users/
All messages sent to this list will be publicly archived and cannot be deleted
|Re: [libreoffice-users] a little math problem with Calc||alex sato <firstname.lastname@example.org>|
|[libreoffice-users] a little math problem with Calc||alex sato <email@example.com>|
- Prev by Date: Re: [libreoffice-users] a little math problem with Calc
- Next by Date: Re: [libreoffice-users] Specialty Dictionaries
- Previous by thread: Re: [libreoffice-users] a little math problem with Calc
- Next by thread: Re: [libreoffice-users] a little math problem with Calc