rounding numbers and converting to string (hiding scientific notation)
#3
It would require more investigation to know for sure, but my guess is that it's an issue of .31 actually being slightly more than .31 due to the error associated with the floating point representation. You can see it if you type .31 into the calculator here: https://www.h-schmidt.net/FloatConverter/IEEE754.html

That slight positive error will likely carry over during the multiplication, and then
ceil()
will round it up. Though I think you're multiplying .31 by 1000 which would give 310, rounding to 311, so perhaps that doesn't explain everything. Still, I think the original point is correct that the error in the floating point representation for some of these decimals will mean they may not round the way you expect even though everything is working correctly.

Edit: Actually, reading your code, I think .32 happens when digits=2, not when digits=3. if that's the case then I'm pretty sure the floating point error would produce a .32 result like you're seeing. It would be interesting to see if digits=3 gives .311 though Big Grin
Reply


Messages In This Thread
RE: rounding numbers and converting to string (hiding scientific notation) - by DSMan195276 - 07-18-2022, 06:17 PM



Users browsing this thread: 4 Guest(s)