Results 1 to 7 of 7

Thread: calculating a wrong answer

  1. #1

    Thread Starter
    New Member
    Join Date
    Nov 2004
    Location
    Oxford, UK
    Posts
    2

    calculating a wrong answer

    Got poblem don't know how to solve it.
    If I enter code ans = 0.9/value
    when value is >60 and <80 ans is returned with a value greater than 1. The value should be less than 1.

    It such a simple bit of code I can't think why this is.

    If I code ans=0.9/70 it returns 1.29571

    The correct answer should be 0.01285

    Thanks everbody for your help, baja_yu had the answer.
    Last edited by 96172152; Nov 20th, 2004 at 09:41 PM.

  2. #2
    Addicted Member Rogier's Avatar
    Join Date
    Jun 2004
    Location
    Netherlands
    Posts
    143
    MAYBE: The answer with 0.9/70 is a HUGE number.
    No forget it! It musn't make sence!
    Sorry i dont know

  3. #3
    Banned dglienna's Avatar
    Join Date
    Jun 2004
    Location
    Center of it all
    Posts
    17,901
    it is displaying Exponential Notation, IE 1.02-E02, which is really .00102. I coded a msgbox and showed .9/70

    is this what you mean?

  4. #4
    Addicted Member Rogier's Avatar
    Join Date
    Jun 2004
    Location
    Netherlands
    Posts
    143
    i don't know........the max. value's of the stabdard vb number is something from -3337.333.........to 3333333 OR SOMETHING LIKE THAT. So I can't help you........I don't know what the problem is!
    Sorry m8 can't help you!

  5. #5
    Next Of Kin baja_yu's Avatar
    Join Date
    Aug 2002
    Location
    /dev/root
    Posts
    5,989
    It IS exponential..... you might want to try something like:

    Text1.Text = Round(0.9 / 70, 4)

  6. #6
    Fanatic Member
    Join Date
    Dec 2002
    Location
    North Carolina
    Posts
    734
    Originally posted by Rogier
    i don't know........the max. value's of the stabdard vb number is something from -3337.333.........to 3333333 OR SOMETHING LIKE THAT. So I can't help you........I don't know what the problem is!
    Sorry m8 can't help you!
    For future reference, I don't believe there is a standard variable in VB... unless you want to count Variant as that, in which case the limits of it are very large.


    "X-mas is 24.Desember you English morons.." - NoteMe

  7. #7
    Banned dglienna's Avatar
    Join Date
    Jun 2004
    Location
    Center of it all
    Posts
    17,901
    Int range is 32,768 to -32,767
    the rest are bigger.

    .5 / 50 = .01

    so if you divide a small number by a big number, the answer is alway SMALL, not HUGE!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width