Results 1 to 4 of 4

Thread: [RESOLVED] 20.0 - 12.0 = 7.999999... ???

  1. #1

    Thread Starter
    I don't do your homework! opus's Avatar
    Join Date
    Jun 2000
    Location
    Good Old Europe
    Posts
    3,863

    Resolved [RESOLVED] 20.0 - 12.0 = 7.999999... ???

    Hi, I'm using an array of PointF variables.
    When calculating with the coordinates I something like 20.0 - 12.0 = 7.9999
    The PointF-var has a X and Y variable (both are singles), why on earth do I get such a result when using it like this:
    NewsingleVar = MyPointF(1).x -MyPointF(2).x
    You're welcome to rate this post!
    If your problem is solved, please use the Mark thread as resolved button


    Wait, I'm too old to hurry!

  2. #2
    Frenzied Member
    Join Date
    Mar 2006
    Location
    Pennsylvania
    Posts
    1,069

    Re: 20.0 - 12.0 = 7.999999... ???

    I had this problem in a game I wrote. It's because they are floats and something about accuracy loss or something like that..

    Just use the overloaded version of the ToString method to format it. Or some other format method.

  3. #3
    Super Moderator jmcilhinney's Avatar
    Join Date
    May 2005
    Location
    Sydney, Australia
    Posts
    111,221

    Re: 20.0 - 12.0 = 7.999999... ???

    This is one of the drawbacks of using floating-point numbers. If all values are whole numbers then you should use Point rather than PointF values. Otherwise you could try rounding results to a specific number of significant digits if all your original values are also to that many significant digits. If there's division involved anywhere it starts to get more complex though.
    Why is my data not saved to my database? | MSDN Data Walkthroughs
    VBForums Database Development FAQ
    My CodeBank Submissions: VB | C#
    My Blog: Data Among Multiple Forms (3 parts)
    Beginner Tutorials: VB | C# | SQL

  4. #4

    Thread Starter
    I don't do your homework! opus's Avatar
    Join Date
    Jun 2000
    Location
    Good Old Europe
    Posts
    3,863

    Re: 20.0 - 12.0 = 7.999999... ???

    Thanks for the reply.
    I need the values to be single, so there no change needed.
    I found this one only will testing and got a bit worried.
    Looks like I didn't do anything wrong, so I'm fine.
    Thanks
    You're welcome to rate this post!
    If your problem is solved, please use the Mark thread as resolved button


    Wait, I'm too old to hurry!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width