ISA Server DestinationSet Performance

[Joris Dobbelsteen]

Dear,

I've made a little tool that synchronizes my ISA Server block list with a
database. However the performance of this is very low.

I'm retreiving 41140 rows from the database, storing them in a hash table
and clearing the input tables within 1 or 2 seconds.
Getting 41140 rows from the ISA server destination set takes me an awsome
173 seconds!

Doing some processing (lookup and deletes on the hashtable) makes this
retreiving good for 379 seconds.
173 seconds for retreiving the entries from the ISA server.
206 seconds for getting the entry elements and removing the entries from the
hash table.

The objects that I'm working with and probably gives the problem is
FPCDestinationSet (and FPCDestination). These are COM objects.
I use a enumerator to work through the FPCDestinationSet (there is no other
way to do so).

My initial conclusion (though premature) would be that the COM interface is
very slow. Loading the same list in the ISA Managment console takes below 2
seconds!

What can I do to improve the performance of this application?
Should I use unsafe code for the COM parts?
A secondary problem is using over 100 MB of RAM for the application (I got
it down already from 160 MB). 50k* 200 Bytes = 10MB (50k rows with <80 byte
strings (take 120 byte for overhead). How do I get this down to a
respectible level?

In .NET the FPCDestinationSet has the function SerializeToBuffer returning a
signed byte array, but this function is not documented anywhere.

- Joris

 

[Willy Denoyette [MVP]]

Are you sure the list is completely filled (try scrolling to the bottom) in
2 seconds when running from the management console, could it be possible
that the list is filled with data taken asynchronously?
Could you try the same using a simple script (vbscript or jscript) and
check the time to load the whole 50K elements and check memory consumption?
As for the time spent to get and remove the elements from the hashtable, it
would help if you could post some code.

Willy.

 

[Joris Dobbelsteen]

Are you sure the list is completely filled (try scrolling to the bottom) in
2 seconds when running from the management console, could it be possible
that the list is filled with data taken asynchronously?

I wasn't intending to count 40000 rows
But a light look made it look like its was OK. The start was good and so was
the end. Also the list didn't scroll too fast, making it all appear it had
40.000 rows listed.
(Immediately after it started).

The FPCDestinationSet has a function called:
SerializeToBuffer (which results in a array of signed bytes).
This function works is very fast, but its completely undocumented.

Could you try the same using a simple script (vbscript or jscript) and
check the time to load the whole 50K elements and check memory

consumption?

I did it in VB6 (code at the end of the post)
Startup 19 MB.
Ending 50 MB
Delta is 30 MB for the destination array (I staticly allocated 99999
elements, 41140 are filled).

This only takes 2 seconds (oposite to 2-3 minutes!).
I would conclude that .NET-COM interop is extremely slow.

Are the any recommendations on how to increase its performance?

As for the time spent to get and remove the elements from the hashtable, it
would help if you could post some code.

The hash table isn't the problem.
The problem is getting the data from the enumerated objects (they are COM
too). And the above conclusion was that .NET - COM support is very crappy.


The hashtable works at nearly instant speeds for even 40000 elements.
The source is a DataTable that is put into it, removed and put into it
again.
Code below (all is done within a second of CPU time):
Sorry for the crappy layout (I need to drop Outlook Express some day).

*** START ***
#region "Table To Hashtable"
protected static Hashtable TableToStrings(DataTable dt)
{
Hashtable ht = new Hashtable(dt.Rows.Count, (float)0.1);

System.Diagnostics.Debug.WriteLine("* " + DateTime.Now.ToString("hh:mm:ss")
+ " filling");
foreach (DataRow dr in dt.Rows)
{
string key = ((string)dr[0]).Replace('%', '*');
if (dr[1] != null && !(dr[1] is DBNull))
key += "$" + ((string)dr[1]).Replace('%', '*');
ht.Add(key, null);
}

System.Diagnostics.Debug.WriteLine("* " + DateTime.Now.ToString("hh:mm:ss")
+ " clearing");
foreach (DataRow dr in dt.Rows)
{
string key = ((string)dr[0]).Replace('%', '*');
if (dr[1] != null && !(dr[1] is DBNull))
key += "$" + ((string)dr[1]).Replace('%', '*');
if (ht.Contains(key))
ht.Remove(key);
}

System.Diagnostics.Debug.WriteLine("* " + DateTime.Now.ToString("hh:mm:ss")
+ " " + ht.Count + " remainders, clearing rest");
ht.Clear();

System.Diagnostics.Debug.WriteLine("* " + DateTime.Now.ToString("hh:mm:ss")
+ " filling again");
foreach (DataRow dr in dt.Rows)
{
string key = ((string)dr[0]).Replace('%', '*');
if (dr[1] != null && !(dr[1] is DBNull))
key += "$" + ((string)dr[1]).Replace('%', '*');
ht.Add(key, null);
}
return ht;
}
#endregion
*** END ***

Willy.

[snip]

*** NOTES ***
Add "Microsoft Internet Securty and Acceleration Server Adminstrative
Objects" as a reference
*** START VB6 CODE FOR READING ELEMENTS ***
Private Sub Form_Load()
Dim thefpc As FPCLib.fpc
Set thefpc = CreateObject("FPC.Root")

Dim fpce As FPCLib.FPCEnterprise
Set fpce = thefpc.Enterprise

Dim polelem As FPCLib.FPCPolicyElements
Set polelem = fpce.PolicyElements

Dim destSet As FPCLib.FPCDestinationSet
Set destSet = polelem.DestinationSets.Item("Explicit Content")

Debug.Print Format(Now, "hh:mm:ss") & " navigation complete..."

Dim da() As String
ReDim da(0 To 99999) As String

Debug.Print Format(Now, "hh:mm:ss") & " allocation complete..."

Dim dest As FPCLib.FPCDestination
Dim x As Long
x = 0
For Each dest In destSet
Select Case dest.Type
Case FPCLib.FpcDestinationAddressType.fpcDestinationTypeDomain:
da(x) = dest.DomainName
Case
FPCLib.FpcDestinationAddressType.fpcDestinationTypeSingleIP:
da(x) = dest.IP_From
Case FPCLib.FpcDestinationAddressType.fpcDestinationTypeIPRange:
da(x) = dest.IP_From & "^" & dest.IP_To
End Select

If Not IsNull(dest.Path) Then
da(x) = da(x) & "$" & dest.Path
End If

x = x + 1
Next

Debug.Print Format(Now, "hh:mm:ss") & " finished with " & x & "
elements..."
MsgBox "Done..."
End Sub
*** END ***