Is this a bug in VB5's Math??

[wengang]

Try this and let me know what you think. You only need a simple form with 3 text boxes (Text1,Text2,Text3).

In General Declarations (or Form_Load) declare these:

Dim intE As Integer
Dim dblAnswer As Double
Dim strAnswer As String

Then in Load_Form add this little loop

strAnswer = "0.000000000000001"
dblAnswer = strAnswer
intE = 0
Do Until dblAnswer >= 1
dblAnswer = (dblAnswer * 10)
intE = intE - 1
Print "dblAnswer = "; dblAnswer; " and E is "; intE
Loop
Text1.Text = dblAnswer
Text2.Text = intE

If dblAnswer = 10 Then Text3.Text = "Really 10"
***************************************************

Okay, you run this and it works fine, right?

Well, change the strAnswer by deleting 2 zeroes.
Run the program again and now dblAnswer doesn't stop at 1,
it Keeps going to 10.

1. Why didn't it stop with 12 zeroes when it did with 14?
2. The second Textbox text shows us that the application regards the value of dblAnswer as 10, so why does the third textbox not contain the number 10?

Does anybody have the answer to this one? Or, does anybody's PC not put 10 in Textbox 2 with 12 zeroes?

[kedaman]

I tested this out with a variant too but it gives a failure:
print dblAnswer - 1
-2,22044604925031E-16

The double and the variant wich is using the double data type, fails.

[kedaman]

It still fails: The double is 1-2,22044604925031E-16 when it passes the supposed "1"

[bsmith]

When I changed the Do loop to the following it works.

Do Until CInt(dblAnswer) >= 1




I am not sure what this does for you, but it should take care of your problem

[bsmith]

Another solution would be to use the string that you have and write a loop to check each character for 0 after the decimal place and accumulate the intE variable this way.

just another suggestion

[Guv]

After reading your query, I expected to give you a definitive answer with some examples. After trying to generate the examples, I discovered that I am thoroughly confused about how double & single variables appear in memory.

Temporarily ignoring my confusion, I believe that the basic answer to you query relates to conversion from decimal to binary and vice versa.

Decimal values like $1.33 or .000001 do not correspond to precise binary values. It is analogous to 1/4 being exactly .25 in decimal, but 1/3 being a never ending decimal fraction. In binary (or hex or octal), 33/100 or 1/1,000,000 are never ending binary fractions. When programming Mainframes, accounting applications always use integer pennies instead of dolars to avoid problems like this. Accountants do not like even a one penny error when dealing with millions of dollars (conversion of $5.45 to a never ending binary fraction could cause final results to be off by a penny or so).

When accepting a fractional decimal value (as an input string of characters) and converting it to a double or single, the conversion is likely to be inexact due to the reasons described above. Arithmetic on the converted data can also introduce round-of errors. When a result is converted back to a decimal, the result might be slightly different from what you would expect when comparing to a result generated using decimal arithmetic.

I had expected to provide a description and examples of double variables corresponding to various decimal values. When I investigated the appearance of some doubles, I realized that I either did not understand doubles or else my method of investigating memory was in error.

I intend to post a query relating to the appearance of data in memory. I already posted one that resulted in unsatisfactory replies due to my improper wording of the query.

I am obsessive-compulsive and intend to work on this some more. Let me know ([email protected]) if you learn more about this.

Gouverneur Cadwallader (The dinosaur from prehistoric Mainfram times).

[Frans C]

For IEEE floating point specifications (the single and double data types) you can look at:
http://www.research.microsoft.com/~h...ieeefloat.html