very confused....hex to decimal

  • Thread starter Thread starter Lance
  • Start date Start date
L

Lance

hi all,

in the immediate window ( typing: ? Val("&H" & "4124EC7DE6666666") and in calc, if i
convert

4124EC7DE6666666 to decimal

then i get 4.6941367371074376E+18 (or 4694136737107437158).

i know ahead of time however that the decimal value should be 685630.95000000000. why is
it that i am getting the former number in VB and calc?

lance
 
Lance,
Because "Hex" itself doesn't support decimal numbers per se, so 685630 would
be &H10BD3H

Are you thinking that 41 24 EC 7D E6 66 66 66 are the bytes for the Double
number 685630.95000000000?

If so, I would probably put the above bytes into a Byte array & use
BitConverter to convert the array of bytes to a Double.

Something like:

Dim bytes() As Byte = {&H41, &H24, &HEC, &H7D, &HE6, &H66, &H66,
&H66}
Array.Reverse(bytes) ' Get the bytes in the correct sequence
Dim d As Double = BitConverter.ToDouble(bytes, 0)

--
Hope this helps
Jay [MVP - Outlook]
..NET Application Architect, Enthusiast, & Evangelist
T.S. Bradley - http://www.tsbradley.net


| hi all,
|
| in the immediate window ( typing: ? Val("&H" & "4124EC7DE6666666") and in
calc, if i
| convert
|
| 4124EC7DE6666666 to decimal
|
| then i get 4.6941367371074376E+18 (or 4694136737107437158).
|
| i know ahead of time however that the decimal value should be
685630.95000000000. why is
| it that i am getting the former number in VB and calc?
|
| lance
|
|
|
|
 
Hi,
4124EC7DE6666666
<<

Why do you know that this should be 685630.95000000000?

A Hex number can only be converted directly to an Integer, Long, etc. It
does not represent a non-integral value -- and is a MUCH larger number (as
the conversions that you observe indicate) than your "expected" result.

You must be attempting to convert the binary representation of a floating
point number to a floating point decimal representation -- This cannot be
done with the code that you are using, which, naturally, works only with
non-floating point (integral) values.

I think that if you Google, you will find code to convert binary FP numbers
to decimal FP. The "normal" reason for such conversion is that you are
receiving a binary data stream from some foreign device, and have to convert
it for use in your program. If this isn't so, then a more detailed
explanation is required.

Dick

Dick

--
Richard Grier (Microsoft Visual Basic MVP)

See www.hardandsoftware.net for contact information.

Author of Visual Basic Programmer's Guide to Serial Communications, 4th
Edition ISBN 1-890422-28-2 (391 pages) published July 2004. See
www.mabry.com/vbpgser4 to order.
 
you nailed it on all accounts, pretty much.

by the way, the reason i know it should be 685630.9500000000 is that i can use a few other
programs (that i didn't write, <g>) that correctly extract the information. it's a UTM x
coordinate. i'm using the other programs to confirm my results.

lance
 
Out of curiosity, what is your source for this binary value?
Any particular reason you are working with the bytes themselves?

Gerald
 
it's a tif file from a filestream. i'm checking for and detecting the values of GeoTIFF
tags.

lance
 
Ah, ok. I asked because I have been working on a bunch of things to iterop
with various CAD / GIS formats. Thought maybe I might have something already
to help you out. However, I have not had a need to work with GeoTIFF's
directly, well not yet. So sadly I do not have anything to help :-|

Gerald
 
Back
Top