How to send file with WebService?

[ad]

I want to send a DataSet to WebService, but the DataSet if too huge(there
about 50000 records, and 50 fields every record).
My solution is
1.save the DataSet as XML file,
2.zip the XML file.
3. send the zip file to WebService.
4. Unzip the zip file to XML.
5. Load XML file into DataSet.
6.Bulk copy the XML file to DataBase.

I can do the Steps 1,2, but I do'nt know how to set the zip file to
WebService in Step 3.

How can I do that?

 

[Christian Stueben]

Hi ad,
fiftythousand records with fifty fields each, is is a lot. But too much?
Before you try to do the xml zipping and uploading, have a look at the
webservice.timeout property. Maybee this one will help to overcome your
problems.

Here is som emor information:
http://www.kleinurl.de/?ks9sdi89

greetings from germany
Chris

 

[Ste]

Presuming you are sending the file over the wire via soap/http

Are you sending the file as a soap attachment?

If not i presume you are tyring to embed the file in the soap body - to do
this you will have to convert the file bytes into Base64 and embed the
base64 encoded zip file in the soap body.. have you done this... PS this
will make your zip file bigger.

eg..

1.save the DataSet as XML file,
2.zip the XML file.

2a base64 zip file to get a human readable file encoded
2b embedd the base64 bytes in the soap doc

3. send the zip file to WebService.
4. Unzip the zip file to XML.
5. Load XML file into DataSet.
6.Bulk copy the XML file to DataBase.

To save file size - do you really need to save the dataset as xml... have
you thought about comma delimited fields, run length encoding.. before
zip'ing etc.. The data file size can be massively reduced.. and as this data
transfer of such a big file will be your bottleneck.. the amount of time you
spend processing the file back to xml on arrival can often be offset.

Alternatively design your solution to send the data in chunks...

 

[ad]

Thanks a lot!
Could you tell me more detail?
1. What about " send the data in chunks" How to implement ?
2. ....about comma delimited fields, run length encoding.. HOw to run
lenght encoding?
3. How to embed base64 encoded into the soap body?

 

[Ste]

1) Sending in chunks.. [not talking about raw http encoding here] .. You
could make your web service send subsets of the records eg the client makes
multiple calls to your webservice sending a soap payload along the lines
of..

The server receives the data in pieces, and processes each record in
turn...More network transfer, but if things go wrong - you only need to
resend the last chunk you sent.. not the whole dataset

<recordblocktype>start</recordblocktype>
<totalrecords>10</totalrecords>
<recordstart>1</recordstart>
<recordend>5</recordend>
<datarows> record_01 data
record_02 data
record_03 data
record_04 data
record_05 data
</datarows>

<recordblocktype>chunk</recordblocktype>
...
...

<recordblocktype>end</recordblocktype>
<totalrecords>20</totalrecords>
<recordstart>6</recordstart>
<recordend>10</recordend>
<datarows>
record_06 data
record_07 data
record_08 data
record_09 data
record_10 data
</datarows>

2) You are taking xml and compressing it into zip format..dont compress the
xml version - convert to the data into another format before you compress
it.. for example..

(this trivial list of individual data rows can easily be represented in a
form other that xml)
xml version..
<data_1>somedata1</data_1><data_2>somedata2</data_2><data_3>somedata3</data_3><data_4>somedata4</data_4>
comma del.. somedata1,somedata2,somedata3,somedata4

As you can see before compression the comma del version of the data is
already smaller.. this should give you a bettter compression ration.

You will have to find a suitable compromise here between ease of use and
compression.

3) Look up the [!CDATA] tag for soap .. read this synpsis

http://webservices.xml.com/pub/a/ws/2002/08/28/endpoints.html

HTH

Steve