Reading Huge Text File Problem

[WoodenSword]

Hi,

I am trying to read a huge text file (2GB) using
StreamReader.

I do this:
Read a line (readline())
I process the line a bit (split it with delimeter and
change fields a bit)
execute (using transaction) a insert query to Ms Sql server
Commit a transaction for 1000 insert queries.

For the first 10000 records speed is pretty good.However
speed gradually slows down significally ever after.What's
wrong?Do i have to use a buffer size in streamreader's
constructor?Do i have to use filestream instead? Is it Sql
server or disk performance problem ?

Thanx!

 

[Jan Tielens]

Are you building a concatenated string? If so, using the StringBuilder class
clould improve the speed.

Just an idea...

 

[Cor]

Hi WoodenSword,

I seldom say this, but I think this is typicaly a question for or a SQL
newsgroup.

Or the very active microsoft.public.dotnet.framework.AdoNet newsgroup

Or do you think that it is inside your VB.net code.

In that case we should have to see some code from you, to help you I think?

Cor

 

[Herfried K. Wagner [MVP]]

I am trying to read a huge text file (2GB) using
StreamReader.

I do this:
Read a line (readline())
I process the line a bit (split it with delimeter and
change fields a bit)
execute (using transaction) a insert query to Ms Sql server
Commit a transaction for 1000 insert queries.

For the first 10000 records speed is pretty good.However
speed gradually slows down significally ever after.What's
wrong?Do i have to use a buffer size in streamreader's
constructor?Do i have to use filestream instead? Is it Sql
server or disk performance problem ?

Post your code. Maybe you don't dispose unused objects and the
application runs out of memory.

 

[Jim Hughes]

Is the database set to Automatically Grow File?

Is the initial database free space adequate to handle all of the inserts or
is the SQL server having to pause, grow the database, insert some more
records, pause, grow the database....

 

[Jerry Ham]

You should comment out the database part and just do the file reading,
parsing piece to see if it is the file stream stuff or the database code.

Break it down into smaller chunks to focus on where the problem is.

Jerry