Fastest way to load new table from dataset

C

Chuck

On a pocketpc 02 with sqlce
I get a dataset with 30,000 records from a webservice

I then load this dataset into a empty table that has no
indexes or constraints. The load takes 50 minutes, which
is about 49 minutes to long. Is there a faster way to do
this?

Dim cmd As New SqlCeCommand
cmd.Connection = Globals.conn
cmd.CommandText = "INSERT INTO cde_locations (LOC_cd,
LOC_DESC, CLR_REQ) VALUES (? ,?, ?)"

p0 = New SqlCeParameter("LOC_cd", SqlDbType.NVarChar, 3)
p1 = New SqlCeParameter("LOC_DESC", SqlDbType.NVarChar,
80)
p2 = New SqlCeParameter("CLR_REQ", SqlDbType.NVarChar, 1)

cmd.Parameters.Add(p0)
cmd.Parameters.Add(p1)
cmd.Parameters.Add(p2)

Dim ws As New xx.TRU_Services
ws.Timeout = -1

Dim ds As DataSet
Dim drow As DataRow
ds = ws.get_Loc_CodesDS

For Each drow In ds.Tables(0).Rows
p0.Value = drow(0)
p1.Value = drow(1)
p2.Value = drow(2)
cmd.ExecuteNonQuery()
Next
 
W

William Ryan

Chuck:

There is probably no way that I can conceive of wherein firing 30,000
updates on a PPC is not going to be slow. That's a huge dataset. You could
conceivably try to just pull down the dataset from your web service and then
serialize it using writexml....save all of your changes and send the dataset
back to the web service. You'll still have overhead with IO, but I've found
that grabbing an XML file and using ReadXML is usually faster than most
other methods when I have a big dataset.

Is using Merge Replication from your Main SQL Server an option? It still
won't be lightning fast but it should be quicker.

But the main thing I'd add is reconisder the game plan and make triple sure
that you need all 30,000 records over there. Typically you can grab much
smaller amounts of data b/c only in batch processing do you normally work
with Sets of this side. And trust me, PPC is not made to be a heavy batch
processing machine.
If you can chunk it over as needed, I think you'll be in much better shape.

HTH,

Bill
 
C

Chuck

I get the dataset from the webservice in about 40 seconds.
I don't ever need to send it back, it's just lookup data.
Our backend is Oracle. I can't use Oracles replication
because there is no .net CF driver for Oracle lite yet.
Just seems wierd that it takes the Pocket PC so long to do
inserts.

When I used Oracle lite and eVB it only took minutes for
the insert. My upgrade to latest and greatest is not
going well.
 
W

William \(Bill\) Vaughn

I think you need to figure out a different strategy. If you're updating all
30000 rows, you need to do so on the server--not on the client.

--
____________________________________
William (Bill) Vaughn
Author, Mentor, Consultant
MVP, hRD
www.betav.com
Please reply only to the newsgroup so that others can benefit.
This posting is provided "AS IS" with no warranties, and confers no rights.
__________________________________
 
C

Cor

Hi Chuck
I get the dataset from the webservice in about 40 seconds.
I don't ever need to send it back, it's just lookup data.
Our backend is Oracle.

Then I understand the webservice has nothing to do with it, it is your
updating process from your pocked PC.

But how can you get an update process if you have already 30.000 records.

Just kidding: Have you a telephone book from the whole world on it?

It makes me curious

Cor
 
C

Chuck

It's a scanner application.
The 30,000 records being downloaded show the location of
inventory items.

Users can enter/scan a product id and get the location of
the product.

On the application there needs to be an inital download of
data. This takes 50 minutes. There will periodically be
updates of this information, but that amount of data is
small and is very easy to handle/update.
 
C

Chuck

I don't need to update them, just insert them into a table
on the PocketPC sqlce database.

When using Oracle Lite and eVB on the handheld this takes
under two minutes.
Using .net and Oracle Sqlce it takes 50 minutes.
There seems to be a problem somewhere. The changes in
state are:
went from eVb to .net
went from Oracle Lite to SQLce


Unless I am doing something stupid with the ado.net
commands or there is a faster to way to load. I believe
my only other stratedgy would be to go back to Oracle Lite
and eVb. Oracle says the will have a provider in Dec. for
the compact framework. I really don't want to wait and
Oracle isn't known for having reliable releases or release
dates.
 
C

Cor

Hi Chuck,

I asked you because I did found the answer from William so very good with
the amount of information he had, did you look to his approach?

Cor
 
W

William \(Bill\) Vaughn

In this case, I would use replication, not brute force INSERTs to load the
device. It's not that hard to setup replication against any SQL Server
database (including MSDE). The process uses binary transfers so it's very
fast and trouble-free.

hth

--
____________________________________
William (Bill) Vaughn
Author, Mentor, Consultant
MVP, hRD
www.betav.com
Please reply only to the newsgroup so that others can benefit.
This posting is provided "AS IS" with no warranties, and confers no rights.
__________________________________
 
C

Chuck

SQL server is not an option. Corporate standard is Oracle.
Hmmm, maybe put msde on a web server; get a snyc request,
load msde from Oracle, then sync msde, to sqlce.

Yuck, I hate things that are complicated.
 
C

Cor

Hi Chuck,

I thougth it was not in the first place the intention in what William wrote
that you should use a SQL server.

I did read it more as make a XML dataset and download that to use on your
pocket PC

Cor
 
K

Kevin Yu [MSFT]

Hi Chuck,

I don't think it's the performance problem of SQL CE. Since SQL CE is only
a small engine on mobile device, it's not suitable for holding so many
data. I think you needn't download so many data at one time, you can only
dump what you need.

Kevin Yu
=======
"This posting is provided "AS IS" with no warranties, and confers no
rights."

--------------------
| Content-Class: urn:content-classes:message
| From: "Chuck" <[email protected]>
| Sender: "Chuck" <[email protected]>
| Subject: Fastest way to load new table from dataset
| Date: Fri, 31 Oct 2003 07:47:41 -0800
| Lines: 35
| Message-ID: <[email protected]>
| MIME-Version: 1.0
| Content-Type: text/plain;
| charset="iso-8859-1"
| Content-Transfer-Encoding: 7bit
| X-Newsreader: Microsoft CDO for Windows 2000
| X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4910.0300
| Thread-Index: AcOfxlFKAZZDYFXnTkaeBB786vXhyA==
| Newsgroups: microsoft.public.dotnet.framework.adonet
| Path: cpmsftngxa06.phx.gbl
| Xref: cpmsftngxa06.phx.gbl microsoft.public.dotnet.framework.adonet:64991
| NNTP-Posting-Host: TK2MSFTNGXA13 10.40.1.165
| X-Tomcat-NG: microsoft.public.dotnet.framework.adonet
|
| On a pocketpc 02 with sqlce
| I get a dataset with 30,000 records from a webservice
|
| I then load this dataset into a empty table that has no
| indexes or constraints. The load takes 50 minutes, which
| is about 49 minutes to long. Is there a faster way to do
| this?
|
| Dim cmd As New SqlCeCommand
| cmd.Connection = Globals.conn
| cmd.CommandText = "INSERT INTO cde_locations (LOC_cd,
| LOC_DESC, CLR_REQ) VALUES (? ,?, ?)"
|
| p0 = New SqlCeParameter("LOC_cd", SqlDbType.NVarChar, 3)
| p1 = New SqlCeParameter("LOC_DESC", SqlDbType.NVarChar,
| 80)
| p2 = New SqlCeParameter("CLR_REQ", SqlDbType.NVarChar, 1)
|
| cmd.Parameters.Add(p0)
| cmd.Parameters.Add(p1)
| cmd.Parameters.Add(p2)
|
| Dim ws As New xx.TRU_Services
| ws.Timeout = -1
|
| Dim ds As DataSet
| Dim drow As DataRow
| ds = ws.get_Loc_CodesDS
|
| For Each drow In ds.Tables(0).Rows
| p0.Value = drow(0)
| p1.Value = drow(1)
| p2.Value = drow(2)
| cmd.ExecuteNonQuery()
| Next
|
 
C

Chuck

I need all the records because they are lookup values.
CE is only inserting about 10 records/second, which is
extremely slow.
 
C

Chuck

I don't have a problem with downloading the dataset, that
only takes 40 seconds. The problem is with getting the
data inserted (50 minutes).

Do you think it would be faster to read the xml directly
using xpath or something rather than stepping through the
rows in the dataset?

Note the dataset comes down presorted by the primary key
and I use cmd.prepare
 
C

Cor

Hi Chuck,

Why do you have to insert the data if you have a dataset.

Are you merging those ore something?

And if so I think why not download the whole dataset.

In my idea the only thing you have to do is dataset.readxml("xmldataset")
(I know almost nothing from a Microsoft pda, so that is on a desktop or a
server).

Cor
 
R

Ray Beckett

Have you tried commenting out the cmd.ExecuteNonQuery to determine if the
bottleneck is the actual insertion or the looping through the records in the
dataset?
 
K

Kevin Yu [MSFT]

Hi Chuck,

The speed of adding data depends on many factors. Since the CPU of embedded
devices are always not as fast as desktop PCs', we cannot expect it to
perform as SQL server does. So I think adding 30,000 records might take a
long time for it.

Kevin Yu
=======
"This posting is provided "AS IS" with no warranties, and confers no
rights."

--------------------
| Content-Class: urn:content-classes:message
| From: "Chuck" <[email protected]>
| Sender: "Chuck" <[email protected]>
| References: <[email protected]>
<[email protected]>
| Subject: RE: Fastest way to load new table from dataset
| Date: Mon, 3 Nov 2003 06:26:38 -0800
| Lines: 88
| Message-ID: <[email protected]>
| MIME-Version: 1.0
| Content-Type: text/plain;
| charset="iso-8859-1"
| Content-Transfer-Encoding: 7bit
| X-Newsreader: Microsoft CDO for Windows 2000
| X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4910.0300
| Thread-Index: AcOiFn3MxgKpBzC8RxSlftz5C8ujNA==
| Newsgroups: microsoft.public.dotnet.framework.adonet
| Path: cpmsftngxa06.phx.gbl
| Xref: cpmsftngxa06.phx.gbl microsoft.public.dotnet.framework.adonet:65206
| NNTP-Posting-Host: TK2MSFTNGXA08 10.40.1.160
| X-Tomcat-NG: microsoft.public.dotnet.framework.adonet
|
| I need all the records because they are lookup values.
| CE is only inserting about 10 records/second, which is
| extremely slow.
|
|
| >-----Original Message-----
| >Hi Chuck,
| >
| >I don't think it's the performance problem of SQL CE.
| Since SQL CE is only
| >a small engine on mobile device, it's not suitable for
| holding so many
| >data. I think you needn't download so many data at one
| time, you can only
| >dump what you need.
| >
| >Kevin Yu
| >=======
| >"This posting is provided "AS IS" with no warranties, and
| confers no
| >rights."
| >
| >--------------------
| >| Content-Class: urn:content-classes:message
| >| From: "Chuck" <[email protected]>
| >| Sender: "Chuck" <[email protected]>
| >| Subject: Fastest way to load new table from dataset
| >| Date: Fri, 31 Oct 2003 07:47:41 -0800
| >| Lines: 35
| >| Message-ID: <[email protected]>
| >| MIME-Version: 1.0
| >| Content-Type: text/plain;
| >| charset="iso-8859-1"
| >| Content-Transfer-Encoding: 7bit
| >| X-Newsreader: Microsoft CDO for Windows 2000
| >| X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4910.0300
| >| Thread-Index: AcOfxlFKAZZDYFXnTkaeBB786vXhyA==
| >| Newsgroups: microsoft.public.dotnet.framework.adonet
| >| Path: cpmsftngxa06.phx.gbl
| >| Xref: cpmsftngxa06.phx.gbl
| microsoft.public.dotnet.framework.adonet:64991
| >| NNTP-Posting-Host: TK2MSFTNGXA13 10.40.1.165
| >| X-Tomcat-NG: microsoft.public.dotnet.framework.adonet
| >|
| >| On a pocketpc 02 with sqlce
| >| I get a dataset with 30,000 records from a webservice
| >|
| >| I then load this dataset into a empty table that has no
| >| indexes or constraints. The load takes 50 minutes,
| which
| >| is about 49 minutes to long. Is there a faster way to
| do
| >| this?
| >|
| >| Dim cmd As New SqlCeCommand
| >| cmd.Connection = Globals.conn
| >| cmd.CommandText = "INSERT INTO cde_locations (LOC_cd,
| >| LOC_DESC, CLR_REQ) VALUES (? ,?, ?)"
| >|
| >| p0 = New SqlCeParameter("LOC_cd", SqlDbType.NVarChar,
| 3)
| >| p1 = New SqlCeParameter("LOC_DESC",
| SqlDbType.NVarChar,
| >| 80)
| >| p2 = New SqlCeParameter("CLR_REQ",
| SqlDbType.NVarChar, 1)
| >|
| >| cmd.Parameters.Add(p0)
| >| cmd.Parameters.Add(p1)
| >| cmd.Parameters.Add(p2)
| >|
| >| Dim ws As New xx.TRU_Services
| >| ws.Timeout = -1
| >|
| >| Dim ds As DataSet
| >| Dim drow As DataRow
| >| ds = ws.get_Loc_CodesDS
| >|
| >| For Each drow In ds.Tables(0).Rows
| >| p0.Value = drow(0)
| >| p1.Value = drow(1)
| >| p2.Value = drow(2)
| >| cmd.ExecuteNonQuery()
| >| Next
| >|
| >
| >.
| >
|
 
C

Chuck

Ray,
You have solved it or at least gave me the hint to solve
it. I was updating another control with progress.
Walking through 30000 records set with no inserts < 1
second. Inserting with nothing else happening 100
records/sec


Removing the update got SQLce to do about 100 records/sec.

Using the cmd.prepare probably helped too. I didn't do a
before and after though.
 
R

Ray Beckett

You could probably do an update every 100 records to give the user an
indication of progress with a minimal impact on performance.

Glad to help!
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads


Top