convert nulls to spaces in file

[Brian Henry]

first question... I have a flat file which unfortinuatly has columns
seperated by nulls instead of spaces (a higher up company created it this
way for us) is there anyway to do a readline with this and not have it
affected by the null? because it is right now causes truncated data at wierd
places... but as soon as i manually with a hex editor change char(00) to
char(20) in the files it reads prerfectly... which leads me to my 2nd
question... if you cant do what i said in the 1st question, is there a way
to go through the file, convert the nulls to spaces and save it back then
open it as a stream reader to read it line by line? the lines are delimited
by CR+LF's thanks!

 

[Jay B. Harlow [MVP - Outlook]]

Brian,
I would simply open it with a System.IO.FileStream, read a buffer full of
bytes, change the null bytes to AscW(" "c) then write the buffer to a second
FileStream.

Something like:

Imports System.IO

Dim input As New FileStream("input.txt", FileMode.Open)
Dim output As New FileStream("output.txt", FileMode.Create)

Dim buffer(1023) As Byte
Dim length As Integer
Do
length = input.Read(buffer, 0, buffer.Length)
For index As Integer = 0 To length - 1
If buffer(index) = 0 Then
buffer(index) = AscW(" "c)
End If
Next
output.Write(buffer, 0, length)
Loop Until length < buffer.Length
input.Close()
output.Close()

Note the above may not work correctly for non Ansi 8-bit encodings, however
it should be easy enough to adapt.

Hope this helps
Jay

 

[Brian Henry]

forgot to say the test data is in the text file attached

 

[Mythran]

Hrm, how about read in the file to a string and do a split with the null
char being the delimiter? I haven't tried it and don't know if it will
work...but what the heck, worth a try eh?

Mythran

 

[Jay B. Harlow [MVP - Outlook]]

Mythran,
Brian (the original poster) stated:
Which leads me to believe something strange is going on with the encoding
when you attempt to read the file into a string (an encoding is required).
There may not be, or it may be something simple, why risk it, when I know
the FileStream won't cause problems, especially on the sample file he
provided...


Reading it as bytes with FileStream will not involve any Encoding objects. I
am reading pure bytes, changing pure bytes, writing pure bytes. Hence I
won't be taking time to track down any potential encoding problems implied
with the Brian's statement.

Hope this helps
Jay

 

[Mythran]

Yes, your way would work...maybe even better...

What I was stating is that he reads it all into a string instead of line by
line (regardless of how he reads it in....such as FileStream or whatever).
Then he just splits the results based on the Null...but my way may not work
anyways because the Split method may not accept DBNull.Value....but it may
take Chr(x). Hrm, probably would just do it your way if it was me

Mythran

 

[Jay B. Harlow [MVP - Outlook]]

Mythran,
Brian is referring to a Null Char (ChrW(0)).

If you look at the file he attached to one of his messages the file is fixed
length records. The first couple of records have 20 or 30 zero bytes at the
start of the line.

The ChrW(0) are not being used for record or field delimiters per se.

I would use the Split method if the file was using ChrW(0) for record or
field delimiters.

What I was stating is that he reads it all into a string instead of line
by line (regardless of how he reads it in....such as FileStream or
whatever).

FileStream cannot read a file into a string, it only reads Bytes. To read a
file into an System.Text.Encoding object is needed, as System.Text.Encoding
is used to convert the bytes in the file to Unicode which is what Strings
are. StreamReader uses an Encoding object to convert the bytes to a String,
if you use a FileStream you would need to create your own Encoding object &
use it to convert the bytes into a String. As I stated Brian's original
statement suggested that the Encoding object may have been having problems
with the 20 or 30 zero bytes...

Hope this helps
Jay

 

[Chris Dunaway]

What it does that it shouldnt do is instead of reading this whole line..
then stop completely for that line and move to the next line and do the same

I don't think you have an error. I think the problem is with
Debug.WriteLine. It does not properly deal with zero bytes in the string.
If you add a watch and inspect the length of the string, does it show the
correct length for the string?

I used the code below on your file and even though Debug.WriteLine could
not display the string correctly, the length of the string was correct for
the line read in and the output file was identical to the input file. I
think you are experiencing a bug in Debug.WriteLine.




Dim fs As New FileStream("test.txt", FileMode.Open, FileAccess.Read)
Dim sr As New StreamReader(fs, True)

Dim sw As New StreamWriter("testout.txt")

Dim sBuf As String

Do Until sr.Peek() = -1
sBuf = sr.ReadLine
Debug.WriteLine(sBuf)
Debug.WriteLine(vbCrLf)
Debug.WriteLine(sBuf.Length.ToString)
sw.WriteLine(sBuf)
Loop

sr.Close()
fs.Close()

sw.Close()



--
Chris

dunawayc[AT]sbcglobal_lunchmeat_[DOT]net

To send me an E-mail, remove the "[", "]", underscores ,lunchmeat, and
replace certain words in my E-Mail address.

 

[Mythran]

Ahh, didn't look at the file...but according to his first post, the columns
are separated by Null. Which I read as "ROWS" separated by NULL...so
scratch everything I've said in this thread...

Sometimes you just have to NOT listen to me at all

Mythran

 

[Jay B. Harlow [MVP - Outlook]]

Chris,
Ah! yes the Debug.Writeline doesn't like ChrW(0)...

Actually its the Marshaling class that treats the ChrW(0) in the string as a
string terminator.

Good point...

Jay

What it does that it shouldnt do is instead of reading this whole line..
then stop completely for that line and move to the next line and do the
same

I don't think you have an error. I think the problem is with
Debug.WriteLine. It does not properly deal with zero bytes in the string.
If you add a watch and inspect the length of the string, does it show the
correct length for the string?

I used the code below on your file and even though Debug.WriteLine could
not display the string correctly, the length of the string was correct for
the line read in and the output file was identical to the input file. I
think you are experiencing a bug in Debug.WriteLine.




Dim fs As New FileStream("test.txt", FileMode.Open, FileAccess.Read)
Dim sr As New StreamReader(fs, True)

Dim sw As New StreamWriter("testout.txt")

Dim sBuf As String

Do Until sr.Peek() = -1
sBuf = sr.ReadLine
Debug.WriteLine(sBuf)
Debug.WriteLine(vbCrLf)
Debug.WriteLine(sBuf.Length.ToString)
sw.WriteLine(sBuf)
Loop

sr.Close()
fs.Close()

sw.Close()



--
Chris

dunawayc[AT]sbcglobal_lunchmeat_[DOT]net

To send me an E-mail, remove the "[", "]", underscores ,lunchmeat, and
replace certain words in my E-Mail address.

 

[Jay B. Harlow [MVP - Outlook]]

Mythran,

Sometimes you just have to NOT listen to me at all

I think that can be said about everybody, me included )

Especially now that Chris just reminded me that VS.NET itself has trouble
showing strings with ChrW(0) in them, which suggests that Brian's Encoding
"problem" I suspect is more related to the debugger then to the Encoding
object I was suspecting...

Jay

Ahh, didn't look at the file...but according to his first post, the
columns are separated by Null. Which I read as "ROWS" separated by
NULL...so scratch everything I've said in this thread...

Sometimes you just have to NOT listen to me at all

Mythran

<<snip>>

 

[Brian Henry]

ah didn't even think of that...

Chris,
Ah! yes the Debug.Writeline doesn't like ChrW(0)...

Actually its the Marshaling class that treats the ChrW(0) in the string as
a string terminator.

Good point...

Jay

What it does that it shouldnt do is instead of reading this whole line..
then stop completely for that line and move to the next line and do the
same

I don't think you have an error. I think the problem is with
Debug.WriteLine. It does not properly deal with zero bytes in the
string.
If you add a watch and inspect the length of the string, does it show the
correct length for the string?

I used the code below on your file and even though Debug.WriteLine could
not display the string correctly, the length of the string was correct
for
the line read in and the output file was identical to the input file. I
think you are experiencing a bug in Debug.WriteLine.




Dim fs As New FileStream("test.txt", FileMode.Open, FileAccess.Read)
Dim sr As New StreamReader(fs, True)

Dim sw As New StreamWriter("testout.txt")

Dim sBuf As String

Do Until sr.Peek() = -1
sBuf = sr.ReadLine
Debug.WriteLine(sBuf)
Debug.WriteLine(vbCrLf)
Debug.WriteLine(sBuf.Length.ToString)
sw.WriteLine(sBuf)
Loop

sr.Close()
fs.Close()

sw.Close()



--
Chris

dunawayc[AT]sbcglobal_lunchmeat_[DOT]net

To send me an E-mail, remove the "[", "]", underscores ,lunchmeat, and
replace certain words in my E-Mail address.

 

[Peter Huang]

Hi

If you still have any concern on this issue, please feel free to post here.


Best regards,

Peter Huang
Microsoft Online Partner Support

Get Secure! - www.microsoft.com/security
This posting is provided "AS IS" with no warranties, and confers no rights.