Increasing time with increasing size of string

[Gary Townsend]

I have been working on a way to take a datatable from a dataset and
transform it into a Geography Markup Language(GML) stream so it can be
loaded into another component. The problem i'm finding is that the building
of the string to transform it into a stream is taking longer and longer as
you add more data to the string ( as you can see from the printout below.

Any suggestions for reducing the time? Should i be writing the GML to the
memory stream right away rather than trying to build one big string and
write that to memory stream? This data table that could be coming in could
contain thousands and thousands of rows.

----------------------------------------------------------------------------
-------------------
2007-07-20 11:50:17 AM : 100
2007-07-20 11:50:20 AM : 200
2007-07-20 11:50:23 AM : 300
2007-07-20 11:50:27 AM : 400
2007-07-20 11:50:32 AM : 500
2007-07-20 11:50:37 AM : 600
2007-07-20 11:50:43 AM : 700
2007-07-20 11:50:50 AM : 800
2007-07-20 11:50:59 AM : 900
2007-07-20 11:51:08 AM : 1000
2007-07-20 11:51:17 AM : 1100
2007-07-20 11:51:27 AM : 1200
2007-07-20 11:51:39 AM : 1300
-----------------------------------------------CODE
BELOW---------------------------------------------
Public Sub LoadDataTable(ByVal dt As DataTable, ByVal xcol As String,
ByVal ycol As String, ByVal srs As String)
Dim dr As DataRow
Dim dc As DataColumn
Dim GMLString As String
Dim featureID As Integer = 1

GMLString = SetGMLHeader()

For Each dr In dt.Rows
GMLString &= " <gml:featureMember>" & vbCrLf
GMLString &= " <SML:NavPoint fid=""NavPoint." & featureID &
""">" & vbCrLf
GMLString &= ConvertRecord(dr, dt.Columns, xcol, ycol, srs)
GMLString &= " </SML:NavPoint>" & vbCrLf
GMLString &= " </gml:featureMember>" & vbCrLf
featureID += 1
If featureID Mod 100 = 0 Then
Console.WriteLine(Date.Now & " : " & featureID)
End If
Next

GMLString &= SetGMLFooter()

Dim memoryWriter As New StreamWriter(mGMLStream)
memoryWriter.Write(GMLString)
memoryWriter.Flush()
'Console.WriteLine(GMLString)
End Sub

Private Function SetGMLHeader() As String
Dim returnString As String

returnString = "<?xml version=""1.0"" encoding=""UTF-8""?>" & vbCrLf
& vbCrLf
returnString &= "<SML:FeatureCollection " & vbCrLf
returnString &= " xmlns=""http://www.spatialmapping.com/gml""" &
vbCrLf
returnString &= " xmlns:gml=""http://www.opengis.net/gml""" & vbCrLf
returnString &= " xmlns:SML=""http://www.spatialmapping.com/gml""" &
vbCrLf
returnString &= "
xmlns:xsi=""http://www.w3.org/2001/XMLSchema-instance"">" & vbCrLf & vbCrLf
& vbCrLf

Return returnString

End Function

Private Function SetGMLFooter() As String
Dim returnString As String

returnString = "</SML:FeatureCollection> " & vbCrLf
Return returnString
End Function

Private Function ConvertRecord(ByVal geoRecord As DataRow, ByVal
geoColumns As DataColumnCollection, ByVal xcol As String, ByVal ycol As
String, ByVal srs As String) As String
Dim returnString As String
For Each dc As DataColumn In geoColumns
If dc.ColumnName <> xcol And dc.ColumnName <> ycol Then
returnString &= " <SML:" & dc.ColumnName.ToUpper &
">"
returnString &= geoRecord.Item(dc.ColumnName)
returnString &= "</SML:" & dc.ColumnName.ToUpper & ">" &
vbCrLf
End If
Next
returnString &= " <SML:GEOMETRY>" & vbCrLf
returnString &= " <gmloint srsName=""" & srs & """>"
& vbCrLf
returnString &= " <gml:coordinates>" &
geoRecord.Item(xcol) & "," & geoRecord.Item(ycol) & "</gml:coordinates>" &
vbCrLf
returnString &= " </gmloint>" & vbCrLf
returnString &= " </SML:GEOMETRY>" & vbCrLf

Return returnString
End Function

 

[Gary Townsend]

Solved it and my solution was to write to the memorystream after each
iteration.
-----------------------------CODE BELOW----------------------------------
Public Sub LoadDataTable(ByVal dt As DataTable, ByVal xcol As String,
ByVal ycol As String, ByVal srs As String)
Dim dr As DataRow
Dim dc As DataColumn
Dim GMLString As String
Dim featureID As Integer = 1

GMLString = SetGMLHeader()
Dim memoryWriter As New StreamWriter(mGMLStream)
memoryWriter.Write(GMLString)
memoryWriter.Flush()

For Each dr In dt.Rows
GMLString = " <gml:featureMember>" & vbCrLf
GMLString &= " <SML:NavPoint fid=""NavPoint." & featureID &
""">" & vbCrLf
GMLString &= ConvertRecord(dr, dt.Columns, xcol, ycol, srs)
GMLString &= " </SML:NavPoint>" & vbCrLf
GMLString &= " </gml:featureMember>" & vbCrLf
featureID += 1
memoryWriter.Write(GMLString)
memoryWriter.Flush()
If featureID Mod 100 = 0 Then
Console.WriteLine(Date.Now & " : " & featureID)
End If
Next

GMLString &= SetGMLFooter()

memoryWriter.Write(GMLString)
memoryWriter.Flush()
'Console.WriteLine(GMLString)
End Sub
----------------------------------------------OUTPUT
BELOW----------------------------
2007-07-20 12:08:38 PM : 100
2007-07-20 12:08:38 PM : 200
2007-07-20 12:08:38 PM : 300
2007-07-20 12:08:38 PM : 400
2007-07-20 12:08:38 PM : 500
2007-07-20 12:08:38 PM : 600
2007-07-20 12:08:38 PM : 700
2007-07-20 12:08:38 PM : 800
2007-07-20 12:08:38 PM : 900
2007-07-20 12:08:38 PM : 1000
2007-07-20 12:08:38 PM : 1100
2007-07-20 12:08:39 PM : 1200
2007-07-20 12:08:39 PM : 1300
2007-07-20 12:08:39 PM : 1400
2007-07-20 12:08:39 PM : 1500
2007-07-20 12:08:39 PM : 1600
2007-07-20 12:08:39 PM : 1700
2007-07-20 12:08:39 PM : 1800
2007-07-20 12:08:39 PM : 1900
2007-07-20 12:08:39 PM : 2000
2007-07-20 12:08:39 PM : 2100
2007-07-20 12:08:39 PM : 2200
2007-07-20 12:08:39 PM : 2300
2007-07-20 12:08:39 PM : 2400
2007-07-20 12:08:40 PM : 2500
2007-07-20 12:08:40 PM : 2600
2007-07-20 12:08:40 PM : 2700
2007-07-20 12:08:40 PM : 2800
2007-07-20 12:08:40 PM : 2900
2007-07-20 12:08:40 PM : 3000
2007-07-20 12:08:40 PM : 3100
2007-07-20 12:08:40 PM : 3200
2007-07-20 12:08:40 PM : 3300
2007-07-20 12:08:40 PM : 3400
2007-07-20 12:08:40 PM : 3500
2007-07-20 12:08:40 PM : 3600
2007-07-20 12:08:40 PM : 3700

 

[Jack Jackson]

You should use StringBuilder instead of String when you will be
heavily modifying a string. The String class is invariant, which
means whenever you modify it a new copy is created with the changes.
If you continually append to a String, the data gets copied each time.

 

[Gary Townsend]

Interesting I didn't know that thanks for the tip i'll look into that.

You should use StringBuilder instead of String when you will be
heavily modifying a string. The String class is invariant, which
means whenever you modify it a new copy is created with the changes.
If you continually append to a String, the data gets copied each time.

-

 

[Al G]

Me too, I didn't know it either. I'll look into it. Thanks.

Al G

 

[Guest]

Me too, I didn't know it either. I'll look into it. Thanks.

Al G

StringBuilder makes a MASSIVE difference in speed - especially when you're
in the 1MB+ range