UTF8/UTF7/ASCII problem while reading from text file

[Lenard Gunda]

hi!

I have the following problem. I need to read data from a TXT file our
company receives.
I would use StreamReader, and process it line by line using ReadLine,
however, the following problem occurs.

The file contains characters with ASCII codes above 128. But the file is
still text (nothing like UTF7/8 or the like). It also might contain + signs.
As a result:

UTF8 encoding doesn't read characters above 128
UTF7 encoding reads everything ok, except eats the + signs, and some
characters after them
ASCII encoding reads the + sign ok, however, characters above 128 are
disappear.

Because the file arrives in this form, I do not have any control on how it
looks like. The best idea so far was to create an own ReadLine method, that
reads the file byte after byte, and converts using UTF7, while taking
special care to feed the + character (ASCII code 46) to an ASCII encoder.
This way I could build a string from a line, that contains exactly what's in
the file.

But would there be a nicer way, or just this do-it-yourself-manually?

thanx

-Lenard

 

[Jon Skeet [C# MVP]]

I have the following problem. I need to read data from a TXT file our
company receives.
I would use StreamReader, and process it line by line using ReadLine,
however, the following problem occurs.

The file contains characters with ASCII codes above 128.

No it doesn't, because there are no such things. ASCII is a 7-bit
encoding.

But the file is still text (nothing like UTF7/8 or the like).

UTF-7 and UTF-8 are text encodings - a file containing text in UTF-8
encoding is still a text file.

It also might contain + signs. As a result:

UTF8 encoding doesn't read characters above 128
UTF7 encoding reads everything ok, except eats the + signs, and some
characters after them
ASCII encoding reads the + sign ok, however, characters above 128 are
disappear.

Because the file arrives in this form, I do not have any control on how it
looks like. The best idea so far was to create an own ReadLine method, that
reads the file byte after byte, and converts using UTF7, while taking
special care to feed the + character (ASCII code 46) to an ASCII encoder.
This way I could build a string from a line, that contains exactly what's in
the file.

But would there be a nicer way, or just this do-it-yourself-manually?

It sounds like you really need to know what encoding your file is
*really* in. Have you tried Encoding.Default?

See http://www.pobox.com/~skeet/csharp/unicode.html for more
information.

 

[Lenard Gunda]

UTF-7 and UTF-8 are text encodings - a file containing text in UTF-8

encoding is still a text file.

Yup I know that. I wanted to mean plain text file, one that can be read
without conversion.

It sounds like you really need to know what encoding your file is
*really* in. Have you tried Encoding.Default?

Well, it contains ASCII characters, extended-ascii characters that are (in
this case) Finnish language characters. There's probably a code-page number
that would describe it. But ... Encoding.Default solved my problem, so it
would seem. Thanks very much!

-Lenard

 

[Jon Skeet [C# MVP]]

Yup I know that. I wanted to mean plain text file, one that can be read
without conversion.

There's *always* conversion involved. The file is binary data, and you
want text data. There's a conversion involved, even if it's ASCII.

Well, it contains ASCII characters, extended-ascii characters that are (in
this case) Finnish language characters.

"Extended-ascii" isn't a well-defined character set (there are many
character sets which are extensions of ASCII) and anything above 127 is
*not* ASCII.

There's probably a code-page number
that would describe it. But ... Encoding.Default solved my problem, so it
would seem. Thanks very much!

To find out the code page of Encoding.Default, just look at
Encoding.Default.CodePage.

I'm glad it's working for you though.

 

[Mike Schilling]

There's *always* conversion involved. The file is binary data, and you
want text data. There's a conversion involved, even if it's ASCII.


"Extended-ascii" isn't a well-defined character set (there are many
character sets which are extensions of ASCII) and anything above 127 is
*not* ASCII.


To find out the code page of Encoding.Default, just look at
Encoding.Default.CodePage.

And note that it's working because your default encoding is the one with the
Finnish characters. If you need it to work on machines where this is not
the case, take Jon's advice: look at Encoding.Default.CodePage, and add
codes that explicitly uses that encoding to read the file.

 

[Lenard Gunda]

Hi,

And note that it's working because your default encoding is the one with the
Finnish characters. If you need it to work on machines where this is not
the case, take Jon's advice: look at Encoding.Default.CodePage, and add
codes that explicitly uses that encoding to read the file.

Yup, I finally managed to understand how these Encoders work, and found it
how to create one for a particular code page. Could be useful in the future,
but because this product is supposed to run on our server, it will have the
correct settings. But good advice, anyway.

Thanks for the help.

-Lenard