Re: tests sc_array_functions_test fail due to floating point precision

Hi Eike

On December 7, 2017 at 12:40 PM Eike Rathke wrote:
> On Saturday, 2017-12-02 02:47:03 -0500, Milton Vandersloot wrote:
> > I've also opened the other test documents in
> > sc/qa/unit/data/functions/array/fods/ using LibreOffice. The documents which
> > fail are linest.fods and also logest.fods. The others succeed.
>
> If your build completed ignoring these tests (i.e. produced a
> usable office under ./instdir/), could you please open the
> failing documents and tell us the cell addresses on the second
> sheet where tests fail and the actual values they produced?

Below are the numbers. Some remarks: Many errors are in the
range of 1E-11 to 1E-12 but some are way larger >1E30. I assume
this is due some division of a very small error.

Has someone done a/some theoretical error analysis of the
algorithms involved? (e.g. numeric (forward/backward) stability
and error estimates similar to what is well-known for LU
factorization and other algorithms.)

Who/What provided the reference values -- or reformulated --
what error do these numbers have (wrt to the mathematical
solution).

Best
Milton


Below are the first three columns (A-C). They are prefixed by the
line number (e.g. 'l21'). At the end I added the formula of the
TRUE/FALSE column. I included this as some tests round to
something different than 1E-12, e.g. line 5 in linest.fods.


logest.fods Sheet 2:
--------------------

l21: 538.89039831558    538.890398315583        FALSE   (=ROUND(A21,12)=ROUND(B21,12))
off by 3E-12



linest.fods Sheet 2:
--------------------

l39: 12237.3616028624   12237.3616028623        FALSE   (=ROUND(A39,12)=ROUND(B39,12))
off by 1E-10

l46: 1.15420554120705E+030      5.5523702096735E+030    FALSE   (=ROUND(A46,12)=ROUND(B46,12))
off by 4.4E30

l66: 1.36570269395312E+030      3.73790456956461E+029   FALSE   (=ROUND(A66,12)=ROUND(B66,12))
off by 1.E30

l99: 2029957.60481663   2029957.60481605        FALSE   (=ROUND(A99,12)=ROUND(B99,12))
off by 6.E-7

l113: 1.11356775201247E+027     7.3824164159995E+026    FALSE   (=ROUND(A113,12)=ROUND(B113,12))
off by 4E26

l156: 9942283.62964539  9942283.62965002        FALSE   (=ROUND(A156,12)=ROUND(B156,12))
off by 5E-6
l157: 11577901.4177444  11577901.4177444        FALSE   (=ROUND(A157,12)=ROUND(B157,12))
off by 0. This keeps me puzzling ... (increasing the number of significant digits yields zeros on 
both entries)

l163: -108.083609022555 -108.083609022579       FALSE   (=ROUND(A163,12)=ROUND(B163,12))
off by 2.4E-11
l164: 34.8435504725302  34.8435504725221        FALSE   (=ROUND(A164,12)=ROUND(B164,12))
off by 1.2E-11

l171: 2484916.49382756  2484916.49382742        FALSE   (=ROUND(A171,12)=ROUND(B171,12))
off by 1.4E-6

l184: 2.7060018272122E-14       9.25081260404044E-13    FALSE   (=ROUND(A184,12)=ROUND(B184,12))
off by 9E-13

l186: 4.04703891013103E+033     3.46285083263994E+030   FALSE   (=ROUND(A186,12)=ROUND(B186,12))
off by 4E33

l189: 5.41123064912622E-14      1.84989825908131E-12    FALSE   (=ROUND(A189,12)=ROUND(B189,12))
off by 1.8E-12

l201: 786024.499999976  786024.50000001 FALSE   (=ROUND(A201,12)=ROUND(B201,12))
off by 4E-8

l216: 786024.499999976  786024.50000001 FALSE   (=ROUND(A216,12)=ROUND(B216,12))
off by 4E-8