Precision
#18
Out of this code:

Code: (Select All)
DIM li AS _INTEGER64
DIM lf AS _FLOAT
li = 9223372036854775806
lf = 9223372036854775806
PRINT li
PRINT lf

I get a "D+18" at the end of "lf" value. This was one less than the positive maximum value of 64-bit integer. I notice your calculations need negative numbers and that's why I tried previously with 18 trilliard and got instead "D+19" from the _FLOAT variable.

If only you could make sure you don't need a fractional part anywhere in your calculations, use _INTEGER64. This is because you said something about the astronomer and usually it means accuracy that could be offered by the "string math" solutions sprinkled on this forum. The "string math" is harder to program, indeed but gives much better accuracy for very small numbers than any simple variable storage type in any "unmanaged" programming language.
Reply


Messages In This Thread
Precision - by Chris - 11-22-2022, 08:43 PM
RE: Precision - by SMcNeill - 11-22-2022, 09:12 PM
RE: Precision - by mnrvovrfc - 11-22-2022, 09:14 PM
RE: Precision - by bplus - 11-22-2022, 09:15 PM
RE: Precision - by mnrvovrfc - 11-22-2022, 09:20 PM
RE: Precision - by Chris - 11-22-2022, 09:26 PM
RE: Precision - by Chris - 11-22-2022, 09:33 PM
RE: Precision - by Chris - 11-22-2022, 09:37 PM
RE: Precision - by Chris - 11-22-2022, 09:40 PM
RE: Precision - by mnrvovrfc - 11-22-2022, 09:41 PM
RE: Precision - by Kernelpanic - 11-23-2022, 12:58 AM
RE: Precision - by Chris - 11-23-2022, 04:22 PM
RE: Precision - by bplus - 11-23-2022, 04:30 PM
RE: Precision - by Chris - 11-23-2022, 05:28 PM
RE: Precision - by Kernelpanic - 11-23-2022, 07:45 PM
RE: Precision - by mnrvovrfc - 11-23-2022, 07:55 PM
RE: Precision - by Jack - 11-24-2022, 02:44 PM
RE: Precision - by Chris - 11-24-2022, 07:06 PM
RE: Precision - by Kernelpanic - 11-24-2022, 11:59 PM
RE: Precision - by Chris - 11-27-2022, 07:01 PM
RE: Precision - by Kernelpanic - 11-27-2022, 09:08 PM
RE: Precision - by Kernelpanic - 11-29-2022, 11:28 PM
RE: Precision - by bplus - 11-22-2022, 08:54 PM
RE: Precision - by Pete - 11-22-2022, 08:59 PM
Precision - by Chris - 11-22-2022, 09:00 PM
RE: Precision - by Chris - 11-22-2022, 09:04 PM
RE: Precision - by SMcNeill - 11-22-2022, 09:10 PM
RE: Precision - by Chris - 11-22-2022, 09:08 PM
RE: Precision - by Pete - 11-22-2022, 09:25 PM
RE: Precision - by bplus - 11-22-2022, 09:26 PM
RE: Precision - by bplus - 11-22-2022, 10:42 PM



Users browsing this thread: 10 Guest(s)