Floating point accuracy

[manur]

I have written a shader to show a slice of a 3dimensional volume texture.
the data loaded was .raw file. the values in the file were USHORT values.
When the outputted data was analyzed there was a mismatch between the actual value inputted and the resultant value. When I searched the net I came across an article saying that in NV40 cards odd floating point numbers cannot be shown accurately if the number is above 2048. But I am using G70 graphics card(7800 GTX). Does this card also have the same problem. Any Guyz facing the same problem

[nolxev]

Where have you read it? G70 architecture is derived from NV40, maybe they brought behind something...

[manur]

I have read it from a site called "http://www.3dcenter.org/artikel/nv40_technik/index3_e.php" . In that site it was given that NV40 series has such a problem.

[JohnyDog]

Well, floating point values cannot be represented exactly by computer hardware, it is always approximation to some degree. As for GPU, shader model 3.0 defines partial (FP16 - 16bit per component) and full (FP32 - 32 bit per component) precisions in which numbers can be stored.
Can you show your shader code and the numbers you're using/getting? If you're using some large numbers, some precision is always lost, as more bits are needed to store the part before decimal point.