Results 1 to 27 of 27

Thread: [RESOLVED] Number Precision: How to explain?

Threaded View

  1. #5

    Thread Starter
    Head Hunted anhn's Avatar
    Join Date
    Aug 2007
    Location
    Australia
    Posts
    3,669

    Re: Number Precision: How to explain?

    @szlamany, I understand what you say, but as you see:

    v=d is always TRUE. I think v was converted to Double before compared with d? Look at the digits of v and d. Hard to explain.

    w=v can be vary as seen: FALSE (my case) or TRUE (your case). But all digits of w match with those of v.

    In your case, with v=d and w=v, that implies d=w: Both d and w are Double and look at the values of d and w. How to explain?

    The picture will be clearer if someone can explain how these numbers are stored in memory bit-by-bit and how they are compared in v=d and w=v.

    Edit: @si, forget about c as it is a Decimal.
    Last edited by anhn; Sep 9th, 2008 at 07:44 AM.
    • Don't forget to use [CODE]your code here[/CODE] when posting code
    • If your question was answered please use Thread Tools to mark your thread [RESOLVED]
    • Don't forget to RATE helpful posts

    • Baby Steps a guided tour
    • IsDigits() and IsNumber() functions • Wichmann-Hill Random() function • >> and << functions for VB • CopyFileByChunk

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width