subtracting decimals - what am i missing?!

[nivek7168]

Hi, i'm sure this is really simple, but i'm having trouble!

All i want to do is to subtract one decimal value from the other, this is basically my code:

Dim myDec1 as double
Dim myDec2 as double
Dim myAnser as double

mydec1= 262.672
mydec2= 262.67

myAnswer = mydec1 - mydec2

However, "myAnswer" does not equal 0.002 as i would expect. I'm getting a value of 2.00000000000955-E3 as a result. What am i doing wrong?!

[Shaggy Hiker]

Nothing, that's just how the Double type works. There are some decimal values that can't be represented precisely in binary, so you get an approximation that is slightly greater or slightly less than the true answer.

[Spoo]

Nivek

There is an explanation as to why you are effectively getting .002000000000000955,
but I'm afraid I don't know the answer... probably has something to do with how VB
deals with precision.

A possible workaround is to format the output.

    Dim myDec1 As Double
    Dim myDec2 As Double
    Dim myAnsE As Double    ' using Exponent
    Dim myAnsF As Double    ' formated
    '
    myDec1 = 262.672
    myDec2 = 262.67
    myAnsE = myDec1 - myDec2              ' 2.00000000000955-E3
    myAnsF = Format(myAnsE, "0.00000")    ' 0.002

EDIT

Shaggy beat me to it

Spoo

[si_the_geek]

Welcome to VBForums

What you are doing wrong is making an assumption which most/all of us originally made, which is that floating-point data types (in VB6, that is Double and Single) are accurate.

Unfortunately they are not accurate, because they are basically designed to be fast (working with floating-point data is surprisingly slow, so a quicker method was created).

If you want accuracy for numbers with decimal places, use data types that give the accuracy (but are slower), such as Currency and Decimal.

[Shaggy Hiker]

They are accurate to a certain level of precision, but they are not accurate to infinite precision. I think that the Currency and Decimal data types work off of something similar to fixed-point math (though I've never bothered to confirm that), so they are accurate to the level of precision they allow. Fixed point math was often used early on because it was based exclusively on integer math and was therefore faster for processors that lacked FPUs. Beginning with the 486, and getting much better with the Pentium, the FPU was built right into the chip and made some floating point math faster than integer math (division and multiplication, but not addition or subtraction), so fixed point faded out. It would still have value, though, because it was highly accurate to whatever level of precision the implementation allowed for.

[nivek7168]

Cool, thanks for the fast replies guys! Yeah, the formatting option could be an option. I've also tried (and failed!) to use the currency and decimals, not sure what i was doing wrong earlier! I was a bit wary of currency as I thought it might be currency specific (£, $, etc) The values i'm working with could be in euros, £'s or $'s.

Basically i'm getting these values back from a database in a loop and need to calculate the difference between these two decimals on each loop. I tried some formatting before, doing formatnumber(mydec1,2) + formatnumber(mydec2, 2), but i think that formatnumber function converts a number into a string, so i didn;t think i was on the right track there.

I don't have my vb6 software in front of me, but to use currency or decimal is it as simple as....

Dim myDec1 As currency (or "as decimal")
Dim myDec2 As currency (or "as decimal")

myDec1 = 262.672
myDec2 = 262.67

myAnswer = mydec1 - mydec2

The myDec's are a floating point number i'm retrieving from the database.

[si_the_geek]

I was a bit wary of currency as I thought it might be currency specific (£, $, etc) The values i'm working with could be in euros, £'s or $'s.

While the name implies that is for money, it isn't really... it is just numbers with decimal places (which is safe to use for money based values, as it is accurate).

I don't have my vb6 software in front of me, but to use currency or decimal is it as simple as....

I think it is for Currency, but have a vague feeling that Decimal is a bit more awkward... but I may have that backwards!

The myDec's are a floating point number i'm retrieving from the database.

The data type in the database may also be a problem, as the floating-point issue is not specific to VB... as such, you may still get the same issue (if it happens, it is because the database has the wrong value, and the Currency/Decimal is just replicating the value it has been given)

[nivek7168]

thanks all, it's given me a couple of ideas to try out!

[Shaggy Hiker]

MANY years ago, somebody wrote up a detailed explanation as to why certain floating point numbers couldn't be represented precisely in binary. Of course, the answer is probably a bit obvious, if you think about it, because certain floating point numbers can't be represented precisely in decimal, either. Consider the result of 1/3. No matter how you slice it, you have to leave off some amount of information. 2/3 is even worse, because you would be better off ending with a 7, the only question is on which digit. Technically, this kind of a problem exists for everything other than integers, and the data type doesn't fully matter. I think that the Decimal or Currency type may be assuming that if 1/3 is 0.3333333, then it is really 0.3333333000000...etc. In other words, it may not be dealing with repeating decimals, whereas a floating point number deals with them more, though not necessarily the way you might want.

[Lenggries]

MANY years ago, somebody wrote up a detailed explanation as to why certain floating point numbers couldn't be represented precisely in binary. Of course, the answer is probably a bit obvious, if you think about it

What is and isn't obvious depends on how many math classes someone can still remember

Here's the basic rule for floating point numbers in different bases (bear in mind we are only talking about rational numbers... irrational numbers (pi, e, square root of 2, etc) can never be accurately expressed numerically in any system):

If the denominator in your fraction consists only of prime factors that are also found in the base of your numeric system, then it can be expressed accurately, assuming you have enough space for all the digits.

Thus, 1/3 cannot be expressed in decimal format because the prime factor of 3 is 3, which is not among the prime factors of 10 (2 and 5). Similarly, 1/7, 1/11, 1/6, and 1/21 cannot be expressed accurately in decimal form, whereas 1/4, 1/5, and 1/50, can, because all of those numbers use denominators entirely comprised of the prime factors found in base 10.

When dealing with computers, you are generally dealing with base 2, which means only numbers whose denominator is a power of two itself (1/2, 3/32, etc) can be expressed accurately as floating point numbers.

Thus, even simple fractions like 1/10 cannot be expressed accurately by floating point numbers, so they just try to get as close as they can.

[dilettante]

Consider What Every Programmer Should Know About Floating-Point Arithmetic

The guy is a little anti-VB, but most of it applies. Sadly far too many people using variations on the "VB" theme are the ones who need language cheet sheets.