2012 Archives by date, by thread · List index

# Re: [libreoffice-users] a little math problem with Calc

```Hi,

alex sato schrieb:
```
```Hi,

I have a problem with functions I made.

Let's say I create the function:

Function Test (n)
dim r(3)
r(0)=int(n)
r(1)=n-r(0)
r(2)=r(1)*100
r(3)=int(r(2))
Test = r
End Function

Then, in a empty spreadsheet take the following steps:
-select A1:D1
-press [F2]
-Insert the formula =TEST(E1)
-press [CTRL]+[SHIFT]+[ENTER]
-insert 11,18 in E1 and see the result in cell D1

I think the result should be 18 but I'm getting 17
Where is the error?
```
```
```
It is the normal behavior of data type double. 11,18 has indefinite digits in dual system. So the stored value is cut somewhere. Because there are enough binary digits to round the dual number to 15 decimal digits, you do not notice it. But when you subtract the integral part, the others digits will shift left and new digits will be generated at the end, which does not represent the original value. The calculated difference becomes a little bit to small. After multiply with 100 you do not have 18,00 but 17,99... And that results in 17. The problem becomes more visible, when the integral part is larger.
```
```
You should not use Basic functions, but use Calc functions instead. They are often tweaked to avoid such problems. You can call Calc functions inside your Macro.
```
```
To get the first two decimal digits as integer value, it seems to be more accurate to use
```        r(3)=(n * 100) mod 100

Kind regards
Regina

--
For unsubscribe instructions e-mail to: users+help@global.libreoffice.org
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/global/users/
All messages sent to this list will be publicly archived and cannot be deleted
```

### Context

Privacy Policy | Impressum (Legal Info) | Copyright information: Unless otherwise specified, all text and images on this website are licensed under the Creative Commons Attribution-Share Alike 3.0 License. This does not include the source code of LibreOffice, which is licensed under the Mozilla Public License (MPLv2). "LibreOffice" and "The Document Foundation" are registered trademarks of their corresponding registered owners or are in actual use as trademarks in one or more countries. Their respective logos and icons are also subject to international copyright laws. Use thereof is explained in our trademark policy.