Could some Math-guru tell me what would be correct (best?) Epsilon for two decimal comparison

-
Could some Math-guru tell me what would be correct (best?) Epsilon for two decimal comparison

SameValue(999.011, 999.012, ????) .

So that would return True, largest possible numbers with max 0.01 difference...

Comments

  1. David SchwartzNovember 23, 2016 at 6:12 AM

    Depends if you're simply truncating or if you're rounding first. If you want the nearest increment, then add half of that increment first, then truncate -- so if you want 0.031..0.034 => 0.03 while 0.035..0.039 => 0.04, add 0.005 first, then truncate.

    But if all 0.03x => 0.03, then simply truncate.

    ReplyDelete
  2. Primož GabrijelčičNovember 23, 2016 at 6:21 AM

    Basically you need: Result := Round(a*100) = Round(b*100);

    ReplyDelete
  3. David HeffernanNovember 23, 2016 at 7:20 AM

    You need to decide what you actually want to do yet. Your specification is ill-formed. As a general rule, if you want to work with decimals, you should usually work with a decimal type. You are using a binary type.

    ReplyDelete

Post a Comment