We did something similiar at work (Bulk Insert doing about 140,000 records)
and was running through the same issues. I was originally trying to use the
BCP program and shell out to that, but became too unreliable and couldn't
manage events that would happen (out of 140,000 rows 2 would fail, I know
you can keep going but I needed to inform the user immediatly).
What we ended up doing was using the COM object for SQL DMO (I *think*
that's the library) and this worked AWESOME! Good event model on when it
inserts rows, good documentation and you run it in process so you could keep
an eye on everything going on. 140,000 records took roughly 30 seconds
doing it this way. The whole procedure took 15 minutes, 13 of that to
generate the XML file from this proprietary language called guru, 1.5
minutes for the XML DOM parser to read it in, and 2 seconds for the XSLT
transform, and 30 for the insert.
So check out the COM object, you'll find it really easy to use and easier to
debug/work with.
HTH,
CJ
Yes, bulk insert. Originally, I was using vb6 and pulling
data from an external data source (non rdbms) and writing
each record to sql server record by record. It started
taking too long (several hours) and was becoming
unreliable, connection open too long. So with vb.net, I
read each record and write it to a series of text files
(each text file will contain at most 20,000 records). I
write close to 100 textfiles (nearly 2,000,000 records).
Then I suck em all up with DTS - takes only 2-3 minutes
with vb.net. The suggestion of using the DTS Agent is
that I have nearly 100 text files to import. I loop
through my DTS package in the vb.net app. Much easier
than using the agent. The new system takes between 1 to 1
½ hrs. Way more reliable because writing text files is a
snap with vb.net. Plus, vb.net runs the DTS package 3-4
times faster than vb6
But I understand that ado.net2 will have bulk insert
capabilities similar to DTS. I sure hope that is true. I
believe that DTS packages are hard to deal with because
they are basically com based, I mean I have to make a
reference to DTS package Object Library from the com tab
in references.