Results 1 to 7 of 7

Thread: [RESOLVED] simple Q

  1. #1

    Thread Starter
    Hyperactive Member Bearnerd's Avatar
    Join Date
    Apr 2006
    Location
    Malaysia
    Posts
    290

    Resolved [RESOLVED] simple Q

    How to get a value of scaleF = 0.10000? I always get the same 0.1 result when I calculate the following calculation in vb:

    VB Code:
    1. dim scaleF as double
    2. dim pixsize as double
    3.  
    4. pixsize = 10
    5. scaleF = 1/pixsize

  2. #2
    Super Moderator manavo11's Avatar
    Join Date
    Nov 2002
    Location
    Around the corner from si_the_geek
    Posts
    7,171

    Re: simple Q

    Well, it's the same thing... The zeros after the decimal point don't count. If you want it to make them visible anyway, you can use the format$ function.


    Has someone helped you? Then you can Rate their helpful post.

  3. #3

    Thread Starter
    Hyperactive Member Bearnerd's Avatar
    Join Date
    Apr 2006
    Location
    Malaysia
    Posts
    290

    Re: simple Q

    Thanks for the idea. But how to use the format$ function? can you please show some example?

  4. #4
    Former Admin/Moderator MartinLiss's Avatar
    Join Date
    Sep 1999
    Location
    San Jose, CA
    Posts
    33,175

    Re: simple Q

    And in case you haven't already figured out how to do that...

    VB Code:
    1. MsgBox Format$(scaleF, "0.00000")

    If that has answered your question, you can help us by pulling down the Thread Tools menu and clicking the Mark Thread Resolved button which will let everyone know that you have your answer.

  5. #5

    Thread Starter
    Hyperactive Member Bearnerd's Avatar
    Join Date
    Apr 2006
    Location
    Malaysia
    Posts
    290

    Re: simple Q

    Thanks a lot 4 your help martin! and manavo too.

  6. #6
    PowerPoster
    Join Date
    Feb 2006
    Location
    East of NYC, USA
    Posts
    5,692

    Re: [RESOLVED] simple Q

    The problem is that 1/10 isn't 0.10000, it's 0.10. "the number of significant digits in an answer should equal the least number of significant digits in any one of the numbers being multiplied, divided etc." http://www.physics.uoguelph.ca/tutor...ig/SIG_dig.htm

  7. #7
    Former Admin/Moderator MartinLiss's Avatar
    Join Date
    Sep 1999
    Location
    San Jose, CA
    Posts
    33,175

    Re: [RESOLVED] simple Q

    Quote Originally Posted by Al42
    The problem is that 1/10 isn't 0.10000, it's 0.10. "the number of significant digits in an answer should equal the least number of significant digits in any one of the numbers being multiplied, divided etc." http://www.physics.uoguelph.ca/tutor...ig/SIG_dig.htm
    I don't want to get into a long discussion about this but I believe that limiting the number of significant digits only applies with non-exact answers. For example if pixsize in the original post were 11 instead of 10 then the answer should be shown as 0.09 and not 0.09091, but since 0.10 = 1.10000 it doesn't apply.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width