rounding numbers and converting to string (hiding scientific notation)
#10
Almost motivates me to finish that leg of my string math that requires a way to deal with repetends. I'm hoping for something as simple as...

1 / 3 = .3... so take three digits .333, multiply them by 3, and .999 rounds up to 1; so if .3... gets multiplied by 3, the results are calculated round up the last digit of 9 by 1, giving 1.

2 / 3 = .6... so take three digits .666, multiply them by 3, and 1.998 rounds up to 2; so if .6... gets multiplied by 3, the results are calculated round up the last digit of say 1.999999999998 by 2, giving 2.

I just don't know if all the nuances would hold up, like 2 / 3 = .6... and .6... *1.5 * 2 would still equal 2 by this method and it might, but I would still be concerned there could be some other instances I'm not considering that would make it so calculations on my system would not match a precision calculator with whatever algorithms they employ to handle repetends.

Pete

1 + 1 makes bigger 1.
If eggs are brain food, Biden takes his scrambled.
Reply


Messages In This Thread
RE: rounding numbers and converting to string (hiding scientific notation) - by Pete - 07-29-2022, 07:43 PM



Users browsing this thread: 5 Guest(s)