Using SqlClient.SqlBulkCopy to insert CSV into database table

R

Rod

OK, I'm completely stumped on this one.

I'm using SqlClient.SqlBulkCopy to try and bulk copy a csv file into a
database. I am getting the following error after calling the
..WriteToServer method.

"The given value of type String from the data source cannot be
converted to type bit of the specified target column."

Now I know that this means that you cannot shove a string or other
incompatible datatype into a bit field, however when examine the data
going into the .WriteToServer method, it only contains 1's or 0's.
Surely these qualify as acceptable values for a bit field!?

At any rate, the bit column accepts nulls also, however even using
nulls, it still errors out with the same messgae.

Can someone please give me some fresh ideas on this one? Here's my
code so far....

' Declare the streamreader objects and read the CSV file into
memory
Dim strFilePath As String = "c:\temp\tblDateLine.hms"
Dim sr As System.IO.StreamReader = New
System.IO.StreamReader(strFilePath)
Dim strImportCSVFile As String = sr.ReadToEnd

' Destroy the Streamreader objects
sr.Close()
sr.Dispose()

' Split into rows
Dim strDelimiter As String = "\n" & vbCrLf
Dim strInputFileRows() As String
strInputFileRows = Split(strImportCSVFile, strDelimiter)

' Split into columns
Dim dt As DataTable = New DataTable
Dim strInputFileColumns() As String

' Create the columns in the data table that correspond to the
number of columns in the input file
strInputFileColumns = Split(strInputFileRows(0), "|,") ' Just
the first row only
For Each s As String In strInputFileColumns
dt.Columns.Add(New DataColumn)
Next

' Now split the rows into columns and add them to the data
table
Dim row As DataRow
Dim finalLine As String = ""
For Each line As String In strInputFileRows
row = dt.NewRow
finalLine = line.Replace(Convert.ToString(vbCr), "")
'finalLine = line.Replace("|,|", "|,NULL")
finalLine = line.Replace("|,|", "|,")
If Microsoft.VisualBasic.Right(finalLine, 1) = "|" Then
finalLine = Microsoft.VisualBasic.Left(finalLine,
Microsoft.VisualBasic.Len(finalLine) - 1)
row.ItemArray = Split(finalLine, "|,")
dt.Rows.Add(row)
Next

' Do the bulk copy
m_strOLEDBConnectionString =
BuildOLEDBConnectionString(g_udtDatabaseSetupOpts.strBlankDatabaseName)
Dim sqlSQLServerBulkCopy As System.Data.SqlClient.SqlBulkCopy =
New System.Data.SqlClient.SqlBulkCopy(m_strOLEDBConnectionString,
SqlClient.SqlBulkCopyOptions.KeepNulls) ' Or
System.Data.SqlClient.SqlBulkCopyOptions.TableLock Or
SqlClient.SqlBulkCopyOptions.Default)

sqlSQLServerBulkCopy.BatchSize = dt.Rows.Count
sqlSQLServerBulkCopy.DestinationTableName = "tblDateLine"
sqlSQLServerBulkCopy.WriteToServer(dt)
sqlSQLServerBulkCopy.Close()


================

Here's a sample of the CSV I'm using

2004-01-01 23:59:59.000|,2004-01-01 00:00:00.000|,9.0|,2004-01-01
08:00:00.000|,2004-01-01 17:00:00.000|,0|\n

The row delimiter is |\n & crlf
The column delimeter is |,

Many thanks and best regards,

Rod
 
C

Cowboy \(Gregory A. Beamer\)

Issue #1 (check first): Check your datatable, esp. the types on the
datatable, while in debug break mode. You can set up the table to have
specific types by adding to the columns collection prior to adding rows,
which is wise (explicit programming). In this manner, the bit column becomes
a boolean and you have to translate from 0|1 to false|true, but you are
already ripping rows. Note that a strongly typed dataset with a single
datatable will uncover these errors as soon as you try to load.

I am not sure on your split, as dual delimeters can be problematic with a
split. | is adequate to separate 99.9% of all files without the ,. I am not
sure this is cause problems for you, but examining the data table will tell.
You might find that you are dealing with data, empty, data, empty, etc. now
instead of data, data, etc. If you require dual delimiters a Regex split
might be a better option.

--
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA
http://gregorybeamer.spaces.live.com

********************************************
Think outside the box!
********************************************
 
W

William \(Bill\) Vaughn

I'm a bit puzzled as to why you're creating a local data table first. Can't
you simply BCP directly from the CSV to a SQL table?

--
____________________________________
William (Bill) Vaughn
Author, Mentor, Consultant
Microsoft MVP
INETA Speaker
www.betav.com/blog/billva
www.betav.com
Please reply only to the newsgroup so that others can benefit.
This posting is provided "AS IS" with no warranties, and confers no rights.
__________________________________
Visit www.hitchhikerguides.net to get more information on my latest book:
Hitchhiker's Guide to Visual Studio and SQL Server (7th Edition)
and Hitchhiker's Guide to SQL Server 2005 Compact Edition (EBook)
-----------------------------------------------------------------------------------------------------------------------
 
R

Rod

Hi Gregory,

Thanks for having a look at my code and responding. I know what your'e
saying about creating explicit data types in the data table - I'm sure
that would help, however it's something that sounds like a really big
hassle. I'm trying to convert VB6 code that is using SQLDMO and the
bulk copy over to .net. The thing is, the DMO code just works as is -
no problems at all. The .net code seems to be so much more complex for
this type of thing - seems like a step backwards almost?

This code is used to package and deploy a database. Under the old DMO
code, all I had to do was match up the database table name with it's
corresponding CSV file and everything worked great during the bulk
copy. If I were to explicitly type the column types on the data table,
I would have to look up those values from the live database itself.

.....OK, perhaps it could be done..I'll look into it....

Regarding the delimeters, so far they seems to be working OK - I've had
no problems at all with them. Thanks anyway.
 
R

Rod

Hi Bill,

Thanks for your input here. The reason I'm using a local data table is
because I'm trying to use the .net framework to do this. In the good
old VB6 days, I simply used SQL DMO to do this and everything was
simply, elegant and sweet. It just worked going straight from a CSV to
the database. Now it seems that you cannot do this. Perhaps I've
missed something - I'm relatively new to the .net framework, and it
seems as if there are quite a few ways to connect to a database. I've
tried looking everywhere on the net for examples of this type of code
and there doesn't seem to be much at all.

Thanks again Bill. I heard you speak at VSLive in Sydney back in 2001.
Really enjoyed your style. Good stuff.

Best regards,

Rod
 
W

William \(Bill\) Vaughn

One of the SqlBulkCopy examples in my new book is a CSV import. I know it's
tricky to setup but it works.
This new 2.0 class is designed to call the SQLSMO layer underneath the
covers--that replaces SQL DMO.

--
____________________________________
William (Bill) Vaughn
Author, Mentor, Consultant
Microsoft MVP
INETA Speaker
www.betav.com/blog/billva
www.betav.com
Please reply only to the newsgroup so that others can benefit.
This posting is provided "AS IS" with no warranties, and confers no rights.
__________________________________
Visit www.hitchhikerguides.net to get more information on my latest book:
Hitchhiker's Guide to Visual Studio and SQL Server (7th Edition)
and Hitchhiker's Guide to SQL Server 2005 Compact Edition (EBook)
-----------------------------------------------------------------------------------------------------------------------
 
R

Rod

Thanks Bill - good to hear that you've got a new book out. I'll check
it out.

In the meantime, is the method that I described how you would
recommend.

Basically,

Read CSV into a data table (using Streamreader)
Bulk copy the data table to the database table (using SQLBulkCopy)

Is this about it?

Thanks,

Rod.
One of the SqlBulkCopy examples in my new book is a CSV import. I know it's
tricky to setup but it works.
This new 2.0 class is designed to call the SQLSMO layer underneath the
covers--that replaces SQL DMO.

--
____________________________________
William (Bill) Vaughn
Author, Mentor, Consultant
Microsoft MVP
INETA Speaker
www.betav.com/blog/billva
www.betav.com
Please reply only to the newsgroup so that others can benefit.
This posting is provided "AS IS" with no warranties, and confers no rights.
__________________________________
Visit www.hitchhikerguides.net to get more information on my latest book:
Hitchhiker's Guide to Visual Studio and SQL Server (7th Edition)
and Hitchhiker's Guide to SQL Server 2005 Compact Edition (EBook)
-----------------------------------------------------------------------------------------------------------------------
 
W

William \(Bill\) Vaughn

Ah, I would suggest that if you can open a DataReader against the CSV to
create a DataTable, you'll be able to pass it directly to a SQL Server
table.
--
____________________________________
William (Bill) Vaughn
Author, Mentor, Consultant
Microsoft MVP
INETA Speaker
www.betav.com/blog/billva
www.betav.com
Please reply only to the newsgroup so that others can benefit.
This posting is provided "AS IS" with no warranties, and confers no rights.
__________________________________
Visit www.hitchhikerguides.net to get more information on my latest book:
Hitchhiker's Guide to Visual Studio and SQL Server (7th Edition)
and Hitchhiker's Guide to SQL Server 2005 Compact Edition (EBook)
-----------------------------------------------------------------------------------------------------------------------
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top