Single v Double
#17
(11-05-2022, 04:44 PM)Dimster Wrote: I'm damn sure you guys have explained this once before so if anyone could point me to the thread I'd appreciate it. Can't seem to come up with the one I kind of remember that addressed this, it could be on the other site. Anyway here's the simply code

Dim a As Double
Dim b As Single
a = .12345678
Print a
b = Abs(a * 10000000)
Print a; b


So my search is to find out what happened to the 7 in the difference between variable a and variable b





This topic is of a similar nature to an old thread: -

Possible error in INTEGER64 calculations
https://qb64forum.alephc.xyz/index.php?topic=4210.0

and my reply (last response to thread + sub-link (#183))'

Quote:Attached is the preliminary work in trying to establish the first transition point in going from Number Space Ordinary (NSO) where all positive integers are representable in _DOUBLE    TO      where the granularity is larger than 1 (i.e., resulting in "skipped" integers).


So, the above referred attachment (comparison table SINGLE vs DOUBLE vs FLOAT vs INTEGER64 etc.) among other things illustrates the "Moral": -

PRINT USING "may" give a more accurate representation of the compute result than PRINT only. 


Hope this reply here is of help explaining/illustrating.
Reply


Messages In This Thread
Single v Double - by Dimster - 11-05-2022, 04:44 PM
RE: Single v Double - by Jack - 11-05-2022, 05:14 PM
RE: Single v Double - by Pete - 11-05-2022, 05:31 PM
RE: Single v Double - by mnrvovrfc - 11-05-2022, 05:43 PM
RE: Single v Double - by Jack - 11-05-2022, 06:00 PM
RE: Single v Double - by Dimster - 11-05-2022, 06:11 PM
RE: Single v Double - by Pete - 11-05-2022, 06:32 PM
RE: Single v Double - by Jack - 11-05-2022, 07:00 PM
RE: Single v Double - by Kernelpanic - 11-05-2022, 09:56 PM
RE: Single v Double - by mnrvovrfc - 11-05-2022, 10:10 PM
RE: Single v Double - by Pete - 11-05-2022, 10:48 PM
RE: Single v Double - by Jack - 11-05-2022, 11:46 PM
RE: Single v Double - by Kernelpanic - 11-05-2022, 11:50 PM
RE: Single v Double - by Pete - 11-05-2022, 11:59 PM
RE: Single v Double - by Jack - 11-06-2022, 12:35 AM
RE: Single v Double - by Pete - 11-06-2022, 12:38 AM
RE: Single v Double - by Kernelpanic - 11-06-2022, 02:03 AM
RE: Single v Double - by Richard - 11-06-2022, 12:42 AM
RE: Single v Double - by Pete - 11-06-2022, 02:07 AM



Users browsing this thread: 13 Guest(s)