Importing from a text file to a Dataset

[Guest]

I have a large(1.25M), fixed field delimited, text file which I want to read into a DataSet. So, I opened a StreamReader, read a line, assigned the values to a new DataRow, and finally added the DataRow to a DataTable in the DataSet. Seemed to be a staight forward task but boy did it progress at a snail's pace!

So I tried to speed things up by first importing the data from the text file into SQL Server. To my amazement, the text import went like lightning! How can I get lightning fast text import?

Michael

 

[Miha Markic]

Hi,

See this thread
http://groups.google.com/groups?hl=...*adonet*&as_uauthors=paul%20clement&lr=&hl=en
Basically you should use OleDb provider to read from text file.
It should be faster.

--
Miha Markic - RightHand .NET consulting & software development
miha at rthand com
www.rthand.com

I have a large(1.25M), fixed field delimited, text file which I want to

read into a DataSet. So, I opened a StreamReader, read a line, assigned the
values to a new DataRow, and finally added the DataRow to a DataTable in the
DataSet. Seemed to be a staight forward task but boy did it progress at a
snail's pace!

So I tried to speed things up by first importing the data from the text

file into SQL Server. To my amazement, the text import went like lightning!
How can I get lightning fast text import?

 

[Guest]

OK! Let's see if I can get it to work.

Michael

 

[Paul Clement]

¤ OK! Let's see if I can get it to work.
¤

If you have fixed length fields (as opposed to comma or custom character delimited) you will need a
schema.ini file:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/odbc/htm/odbcjetsdk_98.asp


Paul ~~~ (e-mail address removed)
Microsoft MVP (Visual Basic)

 

[Miha Markic]

Hi Keith,

I have had similar performance problems loading a database from text

files. But the text files I am usually importing contain headings and
fields that need some kind of conversion. Would it make sense to build a
clean temporary text file and then import it by this method?

It would be faster, yes. But I don't know how faster. You'll have to measure
it by yourself.

Can this method be used on a Stream instead of a file?

I don't think so because it uses OleDb...