Extended ASCII Characters - StreamWriter

[Guest]

I am using StreamWriter's .Writeline() method to create a simple text file. I
found that ASCII characters 255 & 254 are being inserted as the first two
characters of the file which causes a 3rd-party users' parser to fail. It is
not being inserted by my data as the first line starts with "LOGIN: ", i.e.
hard-coded.

Has anyone else encountered this? Is there any filtering settings I can
apply? I looked through the documentation on

 

[Mattias Sjögren]

Has anyone else encountered this? Is there any filtering settings I can
apply? I looked through the documentation on

The first two bytes are probably the little-endian Unicode byte order
marker. Who creates the StreamWriter, and which encoding is it using?



Mattias

 

[Guest]

Not sure what you mean by who creates StreamWriter - it's provided in the C#
language. I am using the default "UTF8Encoding".

 

[Joakim Karlsson]

Depending on how you create an instance of UTF8Encoding, you will get
different results. The encoding can contain a preamble, which identifies
the encoding used together with the encoded data. The
System.Encoding.UTF8 encoding always includes a preamble.

Using the code below you will get the output:

Stream length: 14
Stream length: 17
Stream length: 17

Which indicates that the first stream does not contain any preamble, but
the other two does.

Regards,
Joakim

using System;
using System.Text;
using System.IO;

public class App
{
public static void Main()
{
// Create an encoding without the preamble
MemoryStream ms1 = new MemoryStream();
StreamWriter w1 = new StreamWriter(ms1, new UTF8Encoding(false));
w1.Write("This is a test");
w1.Flush();
Console.WriteLine("Stream length: " + ms1.Length);

// Create an encoding specifically with the preamble
MemoryStream ms2 = new MemoryStream();
StreamWriter w2 = new StreamWriter(ms2, new UTF8Encoding(true));
w2.Write("This is a test");
w2.Flush();
Console.WriteLine("Stream length: " + ms2.Length);

// Use the default UTF8 encoding, which uses the preamble
MemoryStream ms3 = new MemoryStream();
StreamWriter w3 = new StreamWriter(ms3, Encoding.UTF8);
w3.Write("This is a test");
w3.Flush();
Console.WriteLine("Stream length: " + ms3.Length);
}
}

 

[Guest]

Joakim,

Thank you - I will check into this.

Depending on how you create an instance of UTF8Encoding, you will get
different results. The encoding can contain a preamble, which identifies
the encoding used together with the encoded data. The
System.Encoding.UTF8 encoding always includes a preamble.

Using the code below you will get the output:

Stream length: 14
Stream length: 17
Stream length: 17

Which indicates that the first stream does not contain any preamble, but
the other two does.

Regards,
Joakim

using System;
using System.Text;
using System.IO;

public class App
{
public static void Main()
{
// Create an encoding without the preamble
MemoryStream ms1 = new MemoryStream();
StreamWriter w1 = new StreamWriter(ms1, new UTF8Encoding(false));
w1.Write("This is a test");
w1.Flush();
Console.WriteLine("Stream length: " + ms1.Length);

// Create an encoding specifically with the preamble
MemoryStream ms2 = new MemoryStream();
StreamWriter w2 = new StreamWriter(ms2, new UTF8Encoding(true));
w2.Write("This is a test");
w2.Flush();
Console.WriteLine("Stream length: " + ms2.Length);

// Use the default UTF8 encoding, which uses the preamble
MemoryStream ms3 = new MemoryStream();
StreamWriter w3 = new StreamWriter(ms3, Encoding.UTF8);
w3.Write("This is a test");
w3.Flush();
Console.WriteLine("Stream length: " + ms3.Length);
}
}