D notation bug

[SMcNeill]

Code: (Select All)

Dim a As Single, b As Double

For i = 1 To 20

    a$ = "1D" + _Trim$(Str$(i))

    b$ = "1E" + _Trim$(Str$(i))

    a = Val(a$)

    b = Val(b$)

    Print a * a, b * b

Next

Run the above and watch what happens.  No matter whether you like single (E) or double (E) values, they're all still floating point numbers and as such are *always* going to behave imperfectly due to the way the values get stored and processed in binary memory.

1E13 * 1E13 = 9.9999999E25..... whereas 1D13 * 1D13 = 1D26.   <-- In this case, the 1D26 gives us the proper answer, while the 9.9999E25 is imprecise.

1D17 * 1D17 should probably give us 9.999999D33, as that's as close as it comes to calculating that value perfectly, but for whatever reason, the left is rounding to 1 (it's not truncating, as the answer then would be 9E33) and the right isn't incrementing to 34 as it ought to.

These values will never come out as perfectly precise as folks would like them to, but this does seem to be an issue with rounding screwing up somewhere in the "D" code.