Large numbers
#6
The problem you're experiencing is that the
SINGLE
and
DOUBLE
data types are floating point numbers with a limited precision. With
SINGLE
it's about 8 digits or so (23 bits of precision), meaning that it cannot accurately represent the answer 1,731,560,447.097983 and has to round it to the nearest value that can be represented. That value is the integer result you get, and it's an integer because the integer part of the original number was already too precise to represent (that's why it rounds in the wrong direction).

When you use
DOUBLE
you get more like 15 digits of precision, so the result will be a lot closer (but still not quite right) and more importantly there will still be a fractional component in the answer since the integer component isn't too large anymore.
Reply


Messages In This Thread
Large numbers - by james2464 - 08-29-2022, 08:28 PM
RE: Large numbers - by SMcNeill - 08-29-2022, 08:31 PM
RE: Large numbers - by james2464 - 08-29-2022, 08:39 PM
RE: Large numbers - by james2464 - 08-30-2022, 01:56 AM
RE: Large numbers - by SMcNeill - 08-30-2022, 03:35 AM
RE: Large numbers - by james2464 - 08-30-2022, 12:40 PM
RE: Large numbers - by DSMan195276 - 08-30-2022, 04:20 AM
RE: Large numbers - by james2464 - 08-30-2022, 01:00 PM
RE: Large numbers - by mdijkens - 08-30-2022, 11:07 AM
RE: Large numbers - by james2464 - 08-30-2022, 12:11 PM



Users browsing this thread: 4 Guest(s)