1. ## [RESOLVED] simple Q

How to get a value of scaleF = 0.10000? I always get the same 0.1 result when I calculate the following calculation in vb:

VB Code:
```dim scaleF as double
dim pixsize as double
pixsize = 10
scaleF = 1/pixsize```  Reply With Quote

2. ## Re: simple Q

Well, it's the same thing... The zeros after the decimal point don't count. If you want it to make them visible anyway, you can use the format\$ function.  Reply With Quote

3. ## Re: simple Q

Thanks for the idea. But how to use the format\$ function? can you please show some example?  Reply With Quote

4. ## Re: simple Q

And in case you haven't already figured out how to do that...

VB Code:
`MsgBox Format\$(scaleF, "0.00000")`  Reply With Quote

5. ## Re: simple Q

Thanks a lot 4 your help martin! and manavo too.  Reply With Quote

6. ## Re: [RESOLVED] simple Q

The problem is that 1/10 isn't 0.10000, it's 0.10. "the number of significant digits in an answer should equal the least number of significant digits in any one of the numbers being multiplied, divided etc." http://www.physics.uoguelph.ca/tutor...ig/SIG_dig.htm  Reply With Quote

7. ## Re: [RESOLVED] simple Q Originally Posted by Al42
The problem is that 1/10 isn't 0.10000, it's 0.10. "the number of significant digits in an answer should equal the least number of significant digits in any one of the numbers being multiplied, divided etc." http://www.physics.uoguelph.ca/tutor...ig/SIG_dig.htm
I don't want to get into a long discussion about this but I believe that limiting the number of significant digits only applies with non-exact answers. For example if pixsize in the original post were 11 instead of 10 then the answer should be shown as 0.09 and not 0.09091, but since 0.10 = 1.10000 it doesn't apply.  Reply With Quote

#### Posting Permissions

• You may not post new threads
• You may not post replies
• You may not post attachments
• You may not edit your posts
•