Building Byte Arrays - Speed

[Guest]

I was trying to determine the fastest way to build a byte array from
components where the size of the individual components varied depending on
the user's input. I tried three classes I built: (1) using redim arrays to
add to a normal byte array (2) using an ArrayList and finally (3) using a
memorystream. These three classes are listed below the test sub called
"TestBuildByteArray". It was interesting that using the memorystream was
twice as fast as the ArrayList and almost 2000 time as fast as using normal
byte array with redimensions. Thought some might be interested in this.

Private Sub TestBuildByteArraySpeed(ByRef timeByteArray As Long, ByRef
timeArrayList As Long, ByRef TimeArrayStream As Long)
'Testing speed off byte array building
'Notes: Add in Finalize to close and release memory
Dim starttime As Long
Dim count As Integer = 2000
Dim startsize As Integer = 500
Dim i, j As Integer
Dim byt As Byte = 125
Dim nb As Byte() = New Byte() {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25}
Dim b() As Byte

'Test ByteArray Class
ReDim b(0)
starttime = DateTime.Now.Ticks
Dim cb As ByteArray = New ByteArray(startsize)
For i = 1 To count
For j = 0 To 50 : cb.Add(byt) : Next
cb.AddRange(nb)
Next
b = cb.ToArray
timeByteArray = DateTime.Now.Ticks - starttime

'Test ArrayList Class
ReDim b(0)
starttime = DateTime.Now.Ticks
Dim cl As ArrayList = New ArrayList(startsize)
For i = 1 To count
For j = 0 To 50 : cl.Add(byt) : Next
cl.AddRange(nb)
Next
b = DirectCast(cl.ToArray(GetType(Byte)), Byte())
timeArrayList = DateTime.Now.Ticks - starttime

'Test ArrayStream Class
ReDim b(0)
starttime = DateTime.Now.Ticks
Dim cs As ArrayStream = New ArrayStream(500)
For i = 1 To count
For j = 0 To 50 : cs.Add(byt) : Next
cs.AddRange(nb)
Next
b = cs.ToArray
TimeArrayStream = DateTime.Now.Ticks - starttime

End Sub
End Class

Public Class ByteArray
Private u, c, i As Integer
Private b As Byte()
Public Sub New(ByVal size As Integer)
MyBase.New()
ReDim b(size)
u = size
c = 0
End Sub
Public ReadOnly Property ToArray() As Byte()
Get
Return b
End Get
End Property
Public Sub Add(ByVal nb As Byte)
If c > u Then ReDim Preserve b(c) : u = c
b(c) = nb
c = c + 1
End Sub
Public Sub AddRange(ByVal nb() As Byte)
i = c + UBound(nb, 1)
If i > u Then ReDim Preserve b(i)
nb.CopyTo(b, c)
c = i
End Sub
End Class

Public Class ArrayStream
Private ms As MemoryStream
Public Sub New(ByVal size As Integer)
MyBase.New()
ms = New MemoryStream(size)
End Sub
Public ReadOnly Property ToArray() As Byte()
Get
ms.Position = 0
Return ms.ToArray()
End Get
End Property
Public Sub Add(ByVal nb As Byte)
ms.WriteByte(nb)
End Sub

Public Sub AddRange(ByVal nb() As Byte)
ms.Write(nb, 0, UBound(nb) + 1)
End Sub

Protected Overrides Sub finalize()
ms.Close()
MyBase.Finalize()
End Sub
End Class

 

[Jay B. Harlow [MVP - Outlook]]

Dennis,
Your ByteArray is so slow as you are recreating the array each time you add
a byte.

Consider doing what ArrayList & MemoryStream do, over allocate the buffer.

Instead of having the buffer contain the exact number of bytes in it, have
it contain Capicity # of bytes, then have a second variable for the Length
number of bytes.

I posted the start of a ByteBuffer class that functions very similar to the
System.Text.StringBuffer class in that it maintains a byte array internally
allowing you to append byte arrays on the end, expanding the internal array
as needed.

http://groups.google.com/groups?q=B...=#[email protected]&rnum=3

Hope this helps
Jay

 

[Guest]

I did find a couple of errors in the ByteArray class but it didn't make any
difference in speed. I also tried using an initial allocation sufficently
large so the only redim needed was at the end to make the byte array the
correct size. However, in real life, one can't avoid the redim if one is
dealing with variable inputs from users. The ArrayList was still twice as
fast as the ByteArray class and the MemoryStream class was twice as fast as
the ArrayList. My main point is that the Memorystream is quite fast for
simple byte array manipulation.

Haven't had a chance to look at your ByteBuffer yet but will tonight.


Private Sub TestBuildByteArraySpeed(ByRef timeByteArray As Long, ByRef
timeArrayList As Long, ByRef TimeArrayStream As Long)
'Testing speed off byte array building
'Notes: Add in Finalize to close and release memory
Dim starttime As Long
Dim count As Integer = 1000
Dim startsize As Integer = 80000
Dim i, j As Integer
Dim byt As Byte = 125
Dim nb As Byte() = New Byte() {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25}
Dim b() As Byte

'Test ByteArray Class
ReDim b(0)
starttime = DateTime.Now.Ticks
Dim cb As ByteArray = New ByteArray(startsize)
For i = 1 To count
For j = 0 To 50 : cb.Add(byt) : Next
cb.AddRange(nb)
Next
b = cb.ToArray
timeByteArray = DateTime.Now.Ticks - starttime

'Test ArrayList Class
ReDim b(0)
starttime = DateTime.Now.Ticks
Dim cl As ArrayList = New ArrayList(startsize)
For i = 1 To count
For j = 0 To 50 : cl.Add(byt) : Next
cl.AddRange(nb)
Next
b = DirectCast(cl.ToArray(GetType(Byte)), Byte())
timeArrayList = DateTime.Now.Ticks - starttime

'Test ArrayStream Class
ReDim b(0)
starttime = DateTime.Now.Ticks
Dim cs As ArrayStream = New ArrayStream(startsize)
For i = 1 To count
For j = 0 To 50 : cs.Add(byt) : Next
cs.AddRange(nb)
Next
b = cs.ToArray
TimeArrayStream = DateTime.Now.Ticks - starttime

End Sub
End Class

Public Class ByteArray
Private u, c, i As Integer
Private b As Byte()
Public Sub New(ByVal size As Integer)
MyBase.New()
ReDim b(size)
u = size
c = 0
End Sub
Public ReadOnly Property ToArray() As Byte()
Get
ReDim Preserve b(c - 1)
Return b
End Get
End Property
Public Sub Add(ByVal nb As Byte)
If c > u Then ReDim Preserve b(c) : u = c
b(c) = nb
c = c + 1
End Sub
Public Sub AddRange(ByVal nb() As Byte)
i = c + UBound(nb, 1)
If i > u Then ReDim Preserve b(i) : u = i
nb.CopyTo(b, c)
c = i + 1
End Sub
End Class

Public Class ArrayStream
Private ms As MemoryStream
Public Sub New(ByVal size As Integer)
MyBase.New()
ms = New MemoryStream(size)
End Sub
Public ReadOnly Property ToArray() As Byte()
Get
ms.SetLength(ms.Position)
Return ms.ToArray()
End Get
End Property
Public Sub Add(ByVal nb As Byte)
ms.WriteByte(nb)
End Sub

Public Sub AddRange(ByVal nb() As Byte)
ms.Write(nb, 0, UBound(nb) + 1)
End Sub
Protected Overrides Sub finalize()
ms.Close()
MyBase.Finalize()
End Sub
End Class

 

[Cor Ligthert]

Dennis,

I think (mostly I try this, I did not because I use for this kind of things
mostly the memorystream) that when you first calculate the length of your
byteArray that you can speed that part up by avoiding the redim.

Cor

 

[Guest]

Yes, it speeds it up by hundreds of times. However, my problem was that I
don't know the final size of my byte array to begin with since I was building
it from several user's input of strings and I had no idea of the number nor
the size of the strings the users would input. I mainly was just amazed that
the memorystream was so fast.

 

[Cor Ligthert]

Dennis,

I was not clear, I did mean, first calculating the size of the array and
after that starts filling, maybe it need more time, however I think that
that will be earned not doing the redims.

Cor

 

[Jay B. Harlow [MVP - Outlook]]

Dennis,
What ArrayList & StringBuilder does is double its buffer each time the
buffer is not large enough to add any more elements. The buffer starts out
at 16 elements. I believe MemoryStream follows the same double the buffer
rule (its a good simple rule to use).

In your original code & this code you were adding 1 to the buffer each time
the buffer was not large enough, which is each time you added an element
(once past the initial size).

What I am suggesting is you start with at least a 16 element buffer, then in
your add method if the buffer does not have enough free room double it. As
you add elements you will be increasing the buffer less & less.

Its also a good idea in your class as well as MemoryStream, ArrayList &
StringBuilder to give it a good initial estimated capacity. For example, if
you expect to add 1000 elements to the class, then start with a capacity of
1000.

the ArrayList. My main point is that the Memorystream is quite fast for
simple byte array manipulation.

My main point is that the MemoryStream is holding an Array of Bytes in
memory, when it needs to reallocate this array of bytes is effectively does
an ReDim Preserve!

My second point is if the MemoryStream is doing too much you can build your
own ByteArray that will perform equally well! By "too much" I mean there are
properties & methods on MemoryStream that you don't need such as Seek.

Hope this helps
Jay