Data

J

Jeff

Hi,

When I program in C++ a variabl declared as a double

x = 2.0

comes out as 2.0000001 or 1.99999999

but in excel it is 2.0 exactly why is that?

Thanks
 
D

David Biddulph

You need to ask the question (in an appropriate place, which wouldn't be
this newsgroup) as to why your C++ program got it wrong, not why Excel got
it right (assuming that your number really was 2.0).

There are problems with storing some numbers exactly if using fixed point
decimal or fixed point binary, but 2.0 can easily be stored in either of
those formats. [As examples, 1/3 can't be expressed exactly in fixed point
decimal, and 1/10 can't in fixed point binary.]
Also, if you display a number with a limited number of decimal places, then
it may of course look like a different number from what it actually is, so
if a calculation comes to 2.0000001 or 1.99999999 and you ask Excel to
display to 1 decimal place it will look like 2.0. You'll also need to
remember that precision in Excel is limited to 15 significant figures.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

Double variable value in excel 1
VBA - get result from 1
error message 1
Single vs Double 3
VBA - Function Output 2
TAN and ATAN functions ... 1
Advanced Text to Columns?? 5
Prevent Overloaded variables 2

Top