Getting around .Net Strings being UTF-16 encoded only

M

Mike Murray

Hi I have the following issue.

I have a character that is return by a SQL Server database "É" to be
precise, the issue is that when I store character in a .net string
variable my understanding is that it converts it to UTF-16
automatically. I've seen a lot of quotes saying "All String datatype
variables in .NET are UTF-16 encoded". The thing is that I want to
write this character to a XML file in UTF-8 encoded format instead. How
do I go about doing this?

Right now I've tried the following scenerio's (all in VB.Net) when
writing to my XmlTextWriter object:

1) Encoding the string into UTF-8 bytes, then placing the string into
a Stream Reader and then from there reading the stream into the
XmlTextWriter

Dim utf8 As New UTF8Encoding(True)
Dim fNameBytes As Byte() = utf8.GetBytes(strFName)

Dim stmFName As StreamReader = _
New StreamReader( New MemoryStream( fNameBytes ), utf8)

....

EFSXml_UserInfo.WriteAttributeString("firstname",
stmFName.ReadToEnd() )

2) I've also tried the code below:

Dim utf8 As New UTF8Encoding(True)

Dim xtrFName As XmlTextReader = _
New XmlTextReader(New StringReader(strFName))
Dim msFName As MemoryStream = New MemoryStream()

Dim xtwFName As XmlTextWriter = New XmlTextWriter(msFName, utf8)

xtwFName.WriteNode(xtrFName, false)
xtwFName.Flush()

msFName.Position = 0

Dim srFName As StreamReader = New StreamReader(msFName)

....

EFSXml_UserInfo.WriteAttributeString("firstname", srFName
..ReadToEnd() )

Neither of the 2 result in encoding the string correctly, can someone
please help me out, I've spend close to 2 days trying to figure this
stupid thing out.

Thanks,

Mike Murray
 
C

Cor Ligthert [MVP]

Mike,

Your name sounds that you are in the uni code 1252 area, wich is huge.

http://www.geocities.com/Athens/Academy/4038/graph/fontset.htm#b

You should only be busy with encoding when you know that you get characters
from outside that area.

I hope this helps,

Cor


"Mike Murray" <[email protected]> schreef in bericht
Hi I have the following issue.

I have a character that is return by a SQL Server database "É" to be
precise, the issue is that when I store character in a .net string
variable my understanding is that it converts it to UTF-16
automatically. I've seen a lot of quotes saying "All String datatype
variables in .NET are UTF-16 encoded". The thing is that I want to
write this character to a XML file in UTF-8 encoded format instead. How
do I go about doing this?

Right now I've tried the following scenerio's (all in VB.Net) when
writing to my XmlTextWriter object:

1) Encoding the string into UTF-8 bytes, then placing the string into
a Stream Reader and then from there reading the stream into the
XmlTextWriter

Dim utf8 As New UTF8Encoding(True)
Dim fNameBytes As Byte() = utf8.GetBytes(strFName)

Dim stmFName As StreamReader = _
New StreamReader( New MemoryStream( fNameBytes ), utf8)

....

EFSXml_UserInfo.WriteAttributeString("firstname",
stmFName.ReadToEnd() )

2) I've also tried the code below:

Dim utf8 As New UTF8Encoding(True)

Dim xtrFName As XmlTextReader = _
New XmlTextReader(New StringReader(strFName))
Dim msFName As MemoryStream = New MemoryStream()

Dim xtwFName As XmlTextWriter = New XmlTextWriter(msFName, utf8)

xtwFName.WriteNode(xtrFName, false)
xtwFName.Flush()

msFName.Position = 0

Dim srFName As StreamReader = New StreamReader(msFName)

....

EFSXml_UserInfo.WriteAttributeString("firstname", srFName
..ReadToEnd() )

Neither of the 2 result in encoding the string correctly, can someone
please help me out, I've spend close to 2 days trying to figure this
stupid thing out.

Thanks,

Mike Murray
 
J

Jon Skeet [C# MVP]

Mike Murray said:
I have a character that is return by a SQL Server database "É" to be
precise, the issue is that when I store character in a .net string
variable my understanding is that it converts it to UTF-16
automatically. I've seen a lot of quotes saying "All String datatype
variables in .NET are UTF-16 encoded". The thing is that I want to
write this character to a XML file in UTF-8 encoded format instead. How
do I go about doing this?
Click to expand...

Just pass the XmlTextWriter constructor a stream and Encoding.UTF8.

I suspect if that doesn't work, you aren't getting the data back
properly in the first place.

See http://www.pobox.com/~skeet/csharp/unicode.html and
http://www.pobox.com/~skeet/csharp/debuggingunicode.html
 
P

Patrice

It should be easier by using the constructor that allows to specify the
stream encoding...

I would first try with a simple text file in particular :
- paste this char into notepad and save the file as UTF-8
- create this file programmaticaly

You can then use "debug" to compare those files bytes per bytes (should be
easy with 4 bytes !). For now it could range from the character not being
stored correctly in the db to the tool used to read back the file to see if
it worked that wouldn't take into account that this is an UTF-8 encoded file
making appear it bad when it is correct etc...

--
Patrice

"Mike Murray" <[email protected]> a écrit dans le message de
Hi I have the following issue.

I have a character that is return by a SQL Server database "É" to be
precise, the issue is that when I store character in a .net string
variable my understanding is that it converts it to UTF-16
automatically. I've seen a lot of quotes saying "All String datatype
variables in .NET are UTF-16 encoded". The thing is that I want to
write this character to a XML file in UTF-8 encoded format instead. How
do I go about doing this?

Right now I've tried the following scenerio's (all in VB.Net) when
writing to my XmlTextWriter object:

1) Encoding the string into UTF-8 bytes, then placing the string into
a Stream Reader and then from there reading the stream into the
XmlTextWriter

Dim utf8 As New UTF8Encoding(True)
Dim fNameBytes As Byte() = utf8.GetBytes(strFName)

Dim stmFName As StreamReader = _
New StreamReader( New MemoryStream( fNameBytes ), utf8)

....

EFSXml_UserInfo.WriteAttributeString("firstname",
stmFName.ReadToEnd() )

2) I've also tried the code below:

Dim utf8 As New UTF8Encoding(True)

Dim xtrFName As XmlTextReader = _
New XmlTextReader(New StringReader(strFName))
Dim msFName As MemoryStream = New MemoryStream()

Dim xtwFName As XmlTextWriter = New XmlTextWriter(msFName, utf8)

xtwFName.WriteNode(xtrFName, false)
xtwFName.Flush()

msFName.Position = 0

Dim srFName As StreamReader = New StreamReader(msFName)

....

EFSXml_UserInfo.WriteAttributeString("firstname", srFName
..ReadToEnd() )

Neither of the 2 result in encoding the string correctly, can someone
please help me out, I've spend close to 2 days trying to figure this
stupid thing out.

Thanks,

Mike Murray
 
M

Mike Murray

Hi folks,

Here is some more information.

The application that is expecting the UTF-8 Encoding creates a file
using the the default that comes in from the web browser and creates
the file via vbscript, windows scripting code. Right now there is no
conversion to UTF-8 happening in the vbscript, the programmer is just
relying on what comes in through the browser to the vbscript (is there
anything funny with variables storing the incoming text as UTF-16 just
like .NET does?)

What I want to do is create the same file but from vb.NET, therefore
that is why i'm using the XmlTextWriter code. Which makes sense to me,
I just want to clarify that my code is correct, and i'm going about
this the right way.

Thanks,

Mike
 
J

Jon Skeet [C# MVP]

Mike Murray said:
Here is some more information.

The application that is expecting the UTF-8 Encoding creates a file
using the the default that comes in from the web browser and creates
the file via vbscript, windows scripting code. Right now there is no
conversion to UTF-8 happening in the vbscript, the programmer is just
relying on what comes in through the browser to the vbscript (is there
anything funny with variables storing the incoming text as UTF-16 just
like .NET does?)
Click to expand...

By the time you get the string, it should be correct Unicode.
What I want to do is create the same file but from vb.NET, therefore
that is why i'm using the XmlTextWriter code. Which makes sense to me,
I just want to clarify that my code is correct, and i'm going about
this the right way.
Click to expand...

It sounds okay to me. You shouldn't need to do anything to get
XmlTextWriter to create files correctly in UTF-8.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

This spanish character string "ñ" cause something that I don't understand 7
I need UTF-8. But I keep getting UTF-16. Why? How to fix? 1
XSLT from Dataset questions 1
HTMLEncode: low surrogate char Error 1
GetRequestStream Problem 0

Top