precision question...

[MrNorth]

In my sp (sql server) Im selecting a value like this:

SELECT ROUND(WHATEVER,3) AS WHATEVER FROM MYTABLE

I want the value rounded with 3 decimals.

I execute the SP and in the returned dataset I get the value, and there it is something like "3,000,000" Even though I stated that I want 3 decimals only. I guess it has something to do with the dbtype of the column in the dataset, but where should I then perform the round?

I have another problem!!!!! In dev studio, I often get this message:

An unhandled exception of type 'System.OutOfMemoryException' occurred in system.drawing.dll

Additional information: Out of memory.

kind regards
henrik

[tal_mcmahon]

When you Round a number you are still dealing with the original properties of the original data type. That is to say if your column was a Decimal(8,4) 1234.5678 after you perform the round function you will still have the same datatype--decimal(8,4)

so

round(1234.5678 , 3)

will yield 1234.5680

you have to cast into a new datatype if you do not want the trailing '0' at the end like this

cast(round(1234.5678,3) as decimal(7,3))

this yields 1234.568

Hope that helps

Tal McMahon
Competitive Egde Software Inc.
Makers Of Report Exec© Enterprise