Help on sppeding up a 15,000 line batch process

[VM]

In my Windows app, I'm running a batch process that's composed of a FOR loop
that'll run 15,000 times (datatable row count), copy cthe data of each
row -3 fields- to a struct, and send the strct to an external method (using
DLLImport). The problem is that the processing speed is very inconsistent-
it may run very FAST during certain points but then it runs VERY slow at
other moments (especially when it's getting to the end of the table: after
12,000 records). I have an idea of the speed because I have a counter being
displayed in the form (with an Application.DoEvents()). I've eliminated the
DoEvents but it's still inconsistent.

This is a new version of an application that was previously made in MS
C/C++. The only differences between my C# app and the previous version
(besides the language it was written in) is that I had to recreate the
struct in C# (which the external method receives as parameter and modifies)
and that I use the DLLImport for the method. Would this be a reason why it's
so slow? Also, the memory usage (in Task Manager) is at 100% when running.

Thanks,
VM

 

[Gary Hunt]

Can I suggest you hook up a profiler to see where the problem lies...

Chances are it's a Interop problem but difficult to tell without the source
code etc...

There are some very good profilers out there and most have a 30 day trial


cheers,

g

 

[VM]

Thanks for your reply. Would interops naturally make the system slower or
would it be something with my code? In other words, can the speed be
improved or is it a dot net limitation?

VM

 

[Gary Hunt]

I guess that depends on the Interops in question!

If you think it through then some Interop operations could certainly be
tuned - for example - if you've got a 100MB of string data and you're
passing it through Interop then the following would be true:

1. .Net stores it's string as Unicode
2. Calling an ANSI Interop function would require converting those
strings to ANSI to and from the Interop call
3. That's got to be slower than calling a UNICODE Interop function

I think the example above shows how Interop could be slower than the raw C++
but there are so many other factors involved. In my experience of calls via
Interop I haven't noticed any particular slow down so the profiler would
certainly identify which bit of code is causing the problem.

It could also be a GC issue - why don't you post the basic code and it may
be obvious (or probably won't be!)

cheers,

g

 

[VM]

The code's really short...and really easy. And this particular run takes
over 30 minutes and the speed is very inconsistent. The only long code is
the zip4Parm struct (I didn't post it) that contains over 60 fields and
they're all modified during the z4adrinq call. That's why I assumed it could
be the DLL call and the amount of time it takes to fill the struct:

[DllImport("ZIP4_W32.DLL")]
public static extern int z4adrinq([In,
Out][MarshalAs(UnmanagedType.LPStruct)] ZIP4_PARM zip4_parm);
private void btn_run_Click(object sender, System.EventArgs e)
{
......
for (int i=0;i<Datatable_AuditList.Rows.Count;i++) //count = 15,000
{
Application.DoEvents();
sFBU = Datatable_AuditList.Rows["FBU"].ToString();
sDEL = Datatable_AuditList.Rows["DEL"].ToString();
sCTY = Datatable_AuditList.Rows["CTY"].ToString();
sZ4 = Datatable_AuditList.Rows["Z4"].ToString();
ZIP4_PARM zip4Parm = new ZIP4_PARM();
zip4Parm.iprurb = sFBU;
zip4Parm.iadl1 = sDEL;
zip4Parm.ictyi = sCTY;
int iError = z4adrinq(zip4Parm); //dll call - zip4Parm's 60+
fields are modified
statusBar.Panels[0].Text = "Row Count = " + iRecCount.ToString() + "
rows ";
iRecCount++;
}
}

Thanks again,
VM

 

[Gary Hunt]

OK - a couple of good practices things to point out first

I know the compiler may pick these up but better with than without.

Accessing a row by index is probably not the fastest way so get yourself a
reference to the row and use that rather than indexing into the table each
time

e.g.
DataRow row = Datatable_AuditList.Rows;

Next one up - if you know that the row.Item[] is a string then cast is
rather than using ToString() - ToString has greater overhead so you end up
with.

sFBU = (string)row["FBU"];

And back to the first one - why not use a foreach(...) loop rather than by
index? or is there a specific requirement to go through like that?

All of the above are more than likely negligible in terms of performance -
rem out the interop call and see how long it takes - I'm guessing fractions
of a second.

Which leads us to believe that it must be the marshalling that is causing
the problem.

Could you post the zip4Parm structure as that's probably where we make the
changes - as you're passing in a LPStruct then I'm assuming that the
DllImport statement doesn't need to specify Unicode or ANSI.

cheers,

g

PS Spent the weekend doing Interop so lets get to the bottom of this one - I
got beat by .Net as the code I was up against required __cdecl callbacks via
delegates which aren't supported without an IL tweak using ILDASM... :-(

The code's really short...and really easy. And this particular run takes
over 30 minutes and the speed is very inconsistent. The only long code is
the zip4Parm struct (I didn't post it) that contains over 60 fields and
they're all modified during the z4adrinq call. That's why I assumed it could
be the DLL call and the amount of time it takes to fill the struct:

[DllImport("ZIP4_W32.DLL")]
public static extern int z4adrinq([In,
Out][MarshalAs(UnmanagedType.LPStruct)] ZIP4_PARM zip4_parm);
private void btn_run_Click(object sender, System.EventArgs e)
{
.....
for (int i=0;i<Datatable_AuditList.Rows.Count;i++) //count = 15,000
{
Application.DoEvents();
sFBU = Datatable_AuditList.Rows["FBU"].ToString();
sDEL = Datatable_AuditList.Rows["DEL"].ToString();
sCTY = Datatable_AuditList.Rows["CTY"].ToString();
sZ4 = Datatable_AuditList.Rows["Z4"].ToString();
ZIP4_PARM zip4Parm = new ZIP4_PARM();
zip4Parm.iprurb = sFBU;
zip4Parm.iadl1 = sDEL;
zip4Parm.ictyi = sCTY;
int iError = z4adrinq(zip4Parm); //dll call - zip4Parm's 60+
fields are modified
statusBar.Panels[0].Text = "Row Count = " + iRecCount.ToString() + "
rows ";
iRecCount++;
}
}

Thanks again,
VM

I guess that depends on the Interops in question!

If you think it through then some Interop operations could certainly be
tuned - for example - if you've got a 100MB of string data and you're
passing it through Interop then the following would be true:

1. .Net stores it's string as Unicode
2. Calling an ANSI Interop function would require converting those
strings to ANSI to and from the Interop call
3. That's got to be slower than calling a UNICODE Interop function

I think the example above shows how Interop could be slower than the raw C++
but there are so many other factors involved. In my experience of calls via
Interop I haven't noticed any particular slow down so the profiler would
certainly identify which bit of code is causing the problem.

It could also be a GC issue - why don't you post the basic code and it may
be obvious (or probably won't be!)

cheers,

g


a
FOR slow

in

MS reason
why

 

[VM]

Thanks for all your help. I just fixed all those details you told me about.
About the struct, here's a section of it. I'm not going to post the whole
thing because the rest is the same as this. In the C version of the
application, the struct was in a header file so they didn't have to do any
marshalling. Also, during the run I noticed that the loop may be in a
particular record for longer (maybe a second or more). Basically, the
extern method finds information based on the data I copied to the ZIP4_PARM
struct and writes it to the struct in other empty fields when it finds is.
It's similar to a database :

namespace ZipMaster
{
....
[StructLayout(LayoutKind.Sequential, CharSet=CharSet.Ansi)]
public class ZIP4_PARM
{
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=4 )]
public string rsvd0;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string iadl1;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string iadl2;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string ictyi;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=3 )]
public string istai;
public string county;
public short respn;
public char retcc;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=12 )]
public string adrkey;
public char auto_zone_ind;
public struct footer
{
public char a;
public char b;
public char c;
public char d;
public char e;
}
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=6)]
public string rsvd3;
}
}

Thanks again,
VM

OK - a couple of good practices things to point out first

I know the compiler may pick these up but better with than without.

Accessing a row by index is probably not the fastest way so get yourself a
reference to the row and use that rather than indexing into the table each
time

e.g.
DataRow row = Datatable_AuditList.Rows;

Next one up - if you know that the row.Item[] is a string then cast is
rather than using ToString() - ToString has greater overhead so you end up
with.

sFBU = (string)row["FBU"];

And back to the first one - why not use a foreach(...) loop rather than by
index? or is there a specific requirement to go through like that?

All of the above are more than likely negligible in terms of performance -
rem out the interop call and see how long it takes - I'm guessing fractions
of a second.

Which leads us to believe that it must be the marshalling that is causing
the problem.

Could you post the zip4Parm structure as that's probably where we make the
changes - as you're passing in a LPStruct then I'm assuming that the
DllImport statement doesn't need to specify Unicode or ANSI.

cheers,

g

PS Spent the weekend doing Interop so lets get to the bottom of this one - I
got beat by .Net as the code I was up against required __cdecl callbacks via
delegates which aren't supported without an IL tweak using ILDASM... :-(

The code's really short...and really easy. And this particular run takes
over 30 minutes and the speed is very inconsistent. The only long code is
the zip4Parm struct (I didn't post it) that contains over 60 fields and
they're all modified during the z4adrinq call. That's why I assumed it could
be the DLL call and the amount of time it takes to fill the struct:

[DllImport("ZIP4_W32.DLL")]
public static extern int z4adrinq([In,
Out][MarshalAs(UnmanagedType.LPStruct)] ZIP4_PARM zip4_parm);
private void btn_run_Click(object sender, System.EventArgs e)
{
.....
for (int i=0;i<Datatable_AuditList.Rows.Count;i++) //count = 15,000
{
Application.DoEvents();
sFBU = Datatable_AuditList.Rows["FBU"].ToString();
sDEL = Datatable_AuditList.Rows["DEL"].ToString();
sCTY = Datatable_AuditList.Rows["CTY"].ToString();
sZ4 = Datatable_AuditList.Rows["Z4"].ToString();
ZIP4_PARM zip4Parm = new ZIP4_PARM();
zip4Parm.iprurb = sFBU;
zip4Parm.iadl1 = sDEL;
zip4Parm.ictyi = sCTY;
int iError = z4adrinq(zip4Parm); //dll call - zip4Parm's 60+
fields are modified
statusBar.Panels[0].Text = "Row Count = " + iRecCount.ToString()

+

"
rows ";
iRecCount++;
}
}

Thanks again,
VM


raw
C++ calls
via slower
or

of

a of
each in recreate
the

 

[Gary Hunt]

Did setting ANSI vs Unicode make any differences?

g

Thanks for all your help. I just fixed all those details you told me about.
About the struct, here's a section of it. I'm not going to post the whole
thing because the rest is the same as this. In the C version of the
application, the struct was in a header file so they didn't have to do any
marshalling. Also, during the run I noticed that the loop may be in a
particular record for longer (maybe a second or more). Basically, the
extern method finds information based on the data I copied to the ZIP4_PARM
struct and writes it to the struct in other empty fields when it finds is.
It's similar to a database :

namespace ZipMaster
{
...
[StructLayout(LayoutKind.Sequential, CharSet=CharSet.Ansi)]
public class ZIP4_PARM
{
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=4 )]
public string rsvd0;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string iadl1;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string iadl2;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string ictyi;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=3 )]
public string istai;
public string county;
public short respn;
public char retcc;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=12 )]
public string adrkey;
public char auto_zone_ind;
public struct footer
{
public char a;
public char b;
public char c;
public char d;
public char e;
}
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=6)]
public string rsvd3;
}
}

Thanks again,
VM

OK - a couple of good practices things to point out first

I know the compiler may pick these up but better with than without.

Accessing a row by index is probably not the fastest way so get yourself a
reference to the row and use that rather than indexing into the table each
time

e.g.
DataRow row = Datatable_AuditList.Rows;

Next one up - if you know that the row.Item[] is a string then cast is
rather than using ToString() - ToString has greater overhead so you end up
with.

sFBU = (string)row["FBU"];

And back to the first one - why not use a foreach(...) loop rather than by
index? or is there a specific requirement to go through like that?

All of the above are more than likely negligible in terms of performance -
rem out the interop call and see how long it takes - I'm guessing fractions
of a second.

Which leads us to believe that it must be the marshalling that is causing
the problem.

Could you post the zip4Parm structure as that's probably where we make the
changes - as you're passing in a LPStruct then I'm assuming that the
DllImport statement doesn't need to specify Unicode or ANSI.

cheers,

g

PS Spent the weekend doing Interop so lets get to the bottom of this

one -
I

got beat by .Net as the code I was up against required __cdecl callbacks via
delegates which aren't supported without an IL tweak using ILDASM... :-(

code

is

the zip4Parm struct (I didn't post it) that contains over 60 fields and
they're all modified during the z4adrinq call. That's why I assumed it could
be the DLL call and the amount of time it takes to fill the struct:

[DllImport("ZIP4_W32.DLL")]
public static extern int z4adrinq([In,
Out][MarshalAs(UnmanagedType.LPStruct)] ZIP4_PARM zip4_parm);
private void btn_run_Click(object sender, System.EventArgs e)
{
.....
for (int i=0;i<Datatable_AuditList.Rows.Count;i++) //count = 15,000
{
Application.DoEvents();
sFBU = Datatable_AuditList.Rows["FBU"].ToString();
sDEL = Datatable_AuditList.Rows["DEL"].ToString();
sCTY = Datatable_AuditList.Rows["CTY"].ToString();
sZ4 = Datatable_AuditList.Rows["Z4"].ToString();
ZIP4_PARM zip4Parm = new ZIP4_PARM();
zip4Parm.iprurb = sFBU;
zip4Parm.iadl1 = sDEL;
zip4Parm.ictyi = sCTY;
int iError = z4adrinq(zip4Parm); //dll call - zip4Parm's 60+
fields are modified
statusBar.Panels[0].Text = "Row Count = " +

iRecCount.ToString()

+

wou

ld it
may of made
in

 

[Michael Giagnocavo [MVP]]

In addition to keep a reference to the row (instead of indexing each time),
you shouldn't do a string based column lookup each time either (["DEL"]).
The datatable can give you the column (or you can just use the index). Then
you can pass the column or index -- that should be faster too.

Also, you don't need to update the rowcount *every* time, do you? I'd
suggest doing this work on another thread, and then every second or so, have
your UI thread grab the completed rowcount and display it.

I think what you need to do is figure out where the slowdown is (obviously
). See if you can narrow it down to a particular set of rows. You
mentioned that near the end of the table, it goes slow. What happens if you
run it only on the last 5000 rows -- does it still go slow? A profiler
definately should tell you where the memory is going. If the memory grows,
and then hits a cieling, and things slow down, perhaps memory isn't being
freed somewhere (in unmanaged code perhaps)?

You *could* try a GC.Collect(2) each 5000 rows and see if that has any
effect (it'd only matter if the GC was getting messed up for some reason).

-mike
MVP

The code's really short...and really easy. And this particular run takes
over 30 minutes and the speed is very inconsistent. The only long code is
the zip4Parm struct (I didn't post it) that contains over 60 fields and
they're all modified during the z4adrinq call. That's why I assumed it
could
be the DLL call and the amount of time it takes to fill the struct:

[DllImport("ZIP4_W32.DLL")]
public static extern int z4adrinq([In,
Out][MarshalAs(UnmanagedType.LPStruct)] ZIP4_PARM zip4_parm);
private void btn_run_Click(object sender, System.EventArgs e)
{
.....
for (int i=0;i<Datatable_AuditList.Rows.Count;i++) //count = 15,000
{
Application.DoEvents();
sFBU = Datatable_AuditList.Rows["FBU"].ToString();
sDEL = Datatable_AuditList.Rows["DEL"].ToString();
sCTY = Datatable_AuditList.Rows["CTY"].ToString();
sZ4 = Datatable_AuditList.Rows["Z4"].ToString();
ZIP4_PARM zip4Parm = new ZIP4_PARM();
zip4Parm.iprurb = sFBU;
zip4Parm.iadl1 = sDEL;
zip4Parm.ictyi = sCTY;
int iError = z4adrinq(zip4Parm); //dll call - zip4Parm's 60+
fields are modified
statusBar.Panels[0].Text = "Row Count = " + iRecCount.ToString() +
"
rows ";
iRecCount++;
}
}

Thanks again,
VM

I guess that depends on the Interops in question!

If you think it through then some Interop operations could certainly be
tuned - for example - if you've got a 100MB of string data and you're
passing it through Interop then the following would be true:

1. .Net stores it's string as Unicode
2. Calling an ANSI Interop function would require converting those
strings to ANSI to and from the Interop call
3. That's got to be slower than calling a UNICODE Interop function

I think the example above shows how Interop could be slower than the raw C++
but there are so many other factors involved. In my experience of calls via
Interop I haven't noticed any particular slow down so the profiler would
certainly identify which bit of code is causing the problem.

It could also be a GC issue - why don't you post the basic code and it
may
be obvious (or probably won't be!)

cheers,

g


slow

 

[VM]

For 1,000 records, the difference was 66 sec. vs. 130 secs. which is pretty
good. I haven't tried it with the 15,000 records yet. But the old system did
the 15,000 records in just under 4 mins.

VM

Did setting ANSI vs Unicode make any differences?

g

Thanks for all your help. I just fixed all those details you told me about.
About the struct, here's a section of it. I'm not going to post the whole
thing because the rest is the same as this. In the C version of the
application, the struct was in a header file so they didn't have to do any
marshalling. Also, during the run I noticed that the loop may be in a
particular record for longer (maybe a second or more). Basically, the
extern method finds information based on the data I copied to the ZIP4_PARM
struct and writes it to the struct in other empty fields when it finds is.
It's similar to a database :

namespace ZipMaster
{
...
[StructLayout(LayoutKind.Sequential, CharSet=CharSet.Ansi)]
public class ZIP4_PARM
{
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=4 )]
public string rsvd0;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string iadl1;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string iadl2;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string ictyi;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=3 )]
public string istai;
public string county;
public short respn;
public char retcc;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=12 )]
public string adrkey;
public char auto_zone_ind;
public struct footer
{
public char a;
public char b;
public char c;
public char d;
public char e;
}
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=6)]
public string rsvd3;
}
}

Thanks again,
VM

OK - a couple of good practices things to point out first

I know the compiler may pick these up but better with than without.

Accessing a row by index is probably not the fastest way so get

yourself

a

reference to the row and use that rather than indexing into the table each
time

e.g.
DataRow row = Datatable_AuditList.Rows;

Next one up - if you know that the row.Item[] is a string then cast is
rather than using ToString() - ToString has greater overhead so you

end

up

with.

sFBU = (string)row["FBU"];

And back to the first one - why not use a foreach(...) loop rather

than

by

index? or is there a specific requirement to go through like that?

All of the above are more than likely negligible in terms of performance -
rem out the interop call and see how long it takes - I'm guessing fractions
of a second.

Which leads us to believe that it must be the marshalling that is causing
the problem.

Could you post the zip4Parm structure as that's probably where we make the
changes - as you're passing in a LPStruct then I'm assuming that the
DllImport statement doesn't need to specify Unicode or ANSI.

cheers,

g

PS Spent the weekend doing Interop so lets get to the bottom of this

one -
I

got beat by .Net as the code I was up against required __cdecl

callbacks
via

delegates which aren't supported without an IL tweak using ILDASM... :-(


The code's really short...and really easy. And this particular run takes
over 30 minutes and the speed is very inconsistent. The only long

code

is
the zip4Parm struct (I didn't post it) that contains over 60 fields and
they're all modified during the z4adrinq call. That's why I assumed it
could
be the DLL call and the amount of time it takes to fill the struct:

[DllImport("ZIP4_W32.DLL")]
public static extern int z4adrinq([In,
Out][MarshalAs(UnmanagedType.LPStruct)] ZIP4_PARM zip4_parm);
private void btn_run_Click(object sender, System.EventArgs e)
{
.....
for (int i=0;i<Datatable_AuditList.Rows.Count;i++) //count = 15,000
{
Application.DoEvents();
sFBU = Datatable_AuditList.Rows["FBU"].ToString();
sDEL = Datatable_AuditList.Rows["DEL"].ToString();
sCTY = Datatable_AuditList.Rows["CTY"].ToString();
sZ4 = Datatable_AuditList.Rows["Z4"].ToString();
ZIP4_PARM zip4Parm = new ZIP4_PARM();
zip4Parm.iprurb = sFBU;
zip4Parm.iadl1 = sDEL;
zip4Parm.ictyi = sCTY;
int iError = z4adrinq(zip4Parm); //dll call - zip4Parm's 60+
fields are modified
statusBar.Panels[0].Text = "Row Count = " +

iRecCount.ToString()

+
"
rows ";
iRecCount++;
}
}

Thanks again,
VM


I guess that depends on the Interops in question!

If you think it through then some Interop operations could

certainly
be

tuned - for example - if you've got a 100MB of string data and you're
passing it through Interop then the following would be true:

1. .Net stores it's string as Unicode
2. Calling an ANSI Interop function would require converting those
strings to ANSI to and from the Interop call
3. That's got to be slower than calling a UNICODE Interop function

I think the example above shows how Interop could be slower than

the
raw

C++
but there are so many other factors involved. In my experience of calls
via
Interop I haven't noticed any particular slow down so the profiler

wou

ld
certainly identify which bit of code is causing the problem.

It could also be a GC issue - why don't you post the basic code

and

it

speed

be without
the 30
day composed
of data
of parameter
and

 

[VM]

I ran it with 15,000 records and the speed difference was incredible. It
processed them in 2 mins. The only problem I seem to ba having is that the
the fields that are supposed to have data have strange characters in them.
Is it possible that this change (ansi to unicode) affects the way the
external method writes to the struct?

Thanks again,
VM

Did setting ANSI vs Unicode make any differences?

g

Thanks for all your help. I just fixed all those details you told me about.
About the struct, here's a section of it. I'm not going to post the whole
thing because the rest is the same as this. In the C version of the
application, the struct was in a header file so they didn't have to do any
marshalling. Also, during the run I noticed that the loop may be in a
particular record for longer (maybe a second or more). Basically, the
extern method finds information based on the data I copied to the ZIP4_PARM
struct and writes it to the struct in other empty fields when it finds is.
It's similar to a database :

namespace ZipMaster
{
...
[StructLayout(LayoutKind.Sequential, CharSet=CharSet.Ansi)]
public class ZIP4_PARM
{
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=4 )]
public string rsvd0;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string iadl1;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string iadl2;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string ictyi;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=3 )]
public string istai;
public string county;
public short respn;
public char retcc;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=12 )]
public string adrkey;
public char auto_zone_ind;
public struct footer
{
public char a;
public char b;
public char c;
public char d;
public char e;
}
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=6)]
public string rsvd3;
}
}

Thanks again,
VM

OK - a couple of good practices things to point out first

I know the compiler may pick these up but better with than without.

Accessing a row by index is probably not the fastest way so get

yourself

a

reference to the row and use that rather than indexing into the table each
time

e.g.
DataRow row = Datatable_AuditList.Rows;

Next one up - if you know that the row.Item[] is a string then cast is
rather than using ToString() - ToString has greater overhead so you

end

up

with.

sFBU = (string)row["FBU"];

And back to the first one - why not use a foreach(...) loop rather

than

by

index? or is there a specific requirement to go through like that?

All of the above are more than likely negligible in terms of performance -
rem out the interop call and see how long it takes - I'm guessing fractions
of a second.

Which leads us to believe that it must be the marshalling that is causing
the problem.

Could you post the zip4Parm structure as that's probably where we make the
changes - as you're passing in a LPStruct then I'm assuming that the
DllImport statement doesn't need to specify Unicode or ANSI.

cheers,

g

PS Spent the weekend doing Interop so lets get to the bottom of this

one -
I

got beat by .Net as the code I was up against required __cdecl

callbacks
via

delegates which aren't supported without an IL tweak using ILDASM... :-(


The code's really short...and really easy. And this particular run takes
over 30 minutes and the speed is very inconsistent. The only long

code

is
the zip4Parm struct (I didn't post it) that contains over 60 fields and
they're all modified during the z4adrinq call. That's why I assumed it
could
be the DLL call and the amount of time it takes to fill the struct:

[DllImport("ZIP4_W32.DLL")]
public static extern int z4adrinq([In,
Out][MarshalAs(UnmanagedType.LPStruct)] ZIP4_PARM zip4_parm);
private void btn_run_Click(object sender, System.EventArgs e)
{
.....
for (int i=0;i<Datatable_AuditList.Rows.Count;i++) //count = 15,000
{
Application.DoEvents();
sFBU = Datatable_AuditList.Rows["FBU"].ToString();
sDEL = Datatable_AuditList.Rows["DEL"].ToString();
sCTY = Datatable_AuditList.Rows["CTY"].ToString();
sZ4 = Datatable_AuditList.Rows["Z4"].ToString();
ZIP4_PARM zip4Parm = new ZIP4_PARM();
zip4Parm.iprurb = sFBU;
zip4Parm.iadl1 = sDEL;
zip4Parm.ictyi = sCTY;
int iError = z4adrinq(zip4Parm); //dll call - zip4Parm's 60+
fields are modified
statusBar.Panels[0].Text = "Row Count = " +

iRecCount.ToString()

+
"
rows ";
iRecCount++;
}
}

Thanks again,
VM


I guess that depends on the Interops in question!

If you think it through then some Interop operations could

certainly
be

tuned - for example - if you've got a 100MB of string data and you're
passing it through Interop then the following would be true:

1. .Net stores it's string as Unicode
2. Calling an ANSI Interop function would require converting those
strings to ANSI to and from the Interop call
3. That's got to be slower than calling a UNICODE Interop function

I think the example above shows how Interop could be slower than

the
raw

C++
but there are so many other factors involved. In my experience of calls
via
Interop I haven't noticed any particular slow down so the profiler

wou

ld
certainly identify which bit of code is causing the problem.

It could also be a GC issue - why don't you post the basic code

and

it

speed

be without
the 30
day composed
of data
of parameter
and

 

[=?ISO-8859-1?Q?Michel_Andr=E9?=]

I would do something like this. This is not compiled and tested.

Call the DoWork function via worker thread or spawn separate thread.

....

delegate void GUIUpdater(int recCount);

private void UpdateGUI(int recCount)
{
statusBar.Panels[0].Text = "Row Count = " + recCount.ToString() + " rows";
}

....

void DoWork()
{
DataColumn colFBU = Datatable_AuditList.Columns["FBU"];
DataColumn colDEL = Datatable_AuditList.Columns["DEL"];
DataColumn colCTY = Datatable_AuditList.Columns["CTY"];
int iError = 0;
int iCount = 0;
foreach(DataRow row in Datatable_AuditList.Rows)
{
ZIP4_PARM zip4Parm = new ZIP4_PARM();
zip4Parm.iprurb = row[colFBU] as string;
zip4Parm.iadl1 = row[colDEL] as string;
zip4Parm.ictyi = row[colCTY] as string;
iError = z4adrinq(zip4Parm); //dll call - zip4Parm's 60+
++iCount;
if (iCount % 200 == 0)
this.Invoke(new GUIUpdater(this.UpdateGUI), new object[] { iCount});
}
}

This removes the most obvious bottlenecks on the C# part of the loop.

I have found that DoEvents and updating gui can be slow at times.

Moving it to a separate thread and using the forms Invoke method with
a delegate that updates the GUI is usually the fastest way to update the
gui and the rest overhead should be in the interop part

You also could wrap the z4adrinq function in your own managed dll and
only ship the parameters needed if the marshalling copying of the 60+
output variables in the zip4Parm isn't in use. From the snippet it looks
like z4adrinq has a side effect that your after.

Well some tips that could be worth something

Regards
/Michel

The code's really short...and really easy. And this particular run takes
over 30 minutes and the speed is very inconsistent. The only long code is
the zip4Parm struct (I didn't post it) that contains over 60 fields and
they're all modified during the z4adrinq call. That's why I assumed it could
be the DLL call and the amount of time it takes to fill the struct:

[DllImport("ZIP4_W32.DLL")]
public static extern int z4adrinq([In,
Out][MarshalAs(UnmanagedType.LPStruct)] ZIP4_PARM zip4_parm);
private void btn_run_Click(object sender, System.EventArgs e)
{
.....
for (int i=0;i<Datatable_AuditList.Rows.Count;i++) //count = 15,000
{
Application.DoEvents();
sFBU = Datatable_AuditList.Rows["FBU"].ToString();
sDEL = Datatable_AuditList.Rows["DEL"].ToString();
sCTY = Datatable_AuditList.Rows["CTY"].ToString();
sZ4 = Datatable_AuditList.Rows["Z4"].ToString();
ZIP4_PARM zip4Parm = new ZIP4_PARM();
zip4Parm.iprurb = sFBU;
zip4Parm.iadl1 = sDEL;
zip4Parm.ictyi = sCTY;
int iError = z4adrinq(zip4Parm); //dll call - zip4Parm's 60+
fields are modified
statusBar.Panels[0].Text = "Row Count = " + iRecCount.ToString() + "
rows ";
iRecCount++;
}
}

Thanks again,
VM

I guess that depends on the Interops in question!

If you think it through then some Interop operations could certainly be
tuned - for example - if you've got a 100MB of string data and you're
passing it through Interop then the following would be true:

1. .Net stores it's string as Unicode
2. Calling an ANSI Interop function would require converting those
strings to ANSI to and from the Interop call
3. That's got to be slower than calling a UNICODE Interop function

I think the example above shows how Interop could be slower than the raw
C++

but there are so many other factors involved. In my experience of calls
via

Interop I haven't noticed any particular slow down so the profiler would
certainly identify which bit of code is causing the problem.

It could also be a GC issue - why don't you post the basic code and it may
be obvious (or probably won't be!)

cheers,

g



or

each

slow

counter

MS

the

 

[Gary Hunt]

Sorry - you need to find out which of ANSI vs Unicode the destination
function is written in and use the appropriate one.

(Do you know the difference? If not then ANSI is a single byte ber character
whereas Unicode takes up two bytes per character.)

cheers,

g

For 1,000 records, the difference was 66 sec. vs. 130 secs. which is pretty
good. I haven't tried it with the 15,000 records yet. But the old system did
the 15,000 records in just under 4 mins.

VM

Did setting ANSI vs Unicode make any differences?

g

Thanks for all your help. I just fixed all those details you told me about.
About the struct, here's a section of it. I'm not going to post the whole
thing because the rest is the same as this. In the C version of the
application, the struct was in a header file so they didn't have to do any
marshalling. Also, during the run I noticed that the loop may be in a
particular record for longer (maybe a second or more). Basically, the
extern method finds information based on the data I copied to the ZIP4_PARM
struct and writes it to the struct in other empty fields when it finds is.
It's similar to a database :

namespace ZipMaster
{
...
[StructLayout(LayoutKind.Sequential, CharSet=CharSet.Ansi)]
public class ZIP4_PARM
{
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=4 )]
public string rsvd0;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string iadl1;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string iadl2;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string ictyi;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=3 )]
public string istai;
public string county;
public short respn;
public char retcc;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=12 )]
public string adrkey;
public char auto_zone_ind;
public struct footer
{
public char a;
public char b;
public char c;
public char d;
public char e;
}
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=6)]
public string rsvd3;
}
}

Thanks again,
VM

OK - a couple of good practices things to point out first

I know the compiler may pick these up but better with than without.

Accessing a row by index is probably not the fastest way so get

yourself

a
reference to the row and use that rather than indexing into the

table
each

time

e.g.
DataRow row = Datatable_AuditList.Rows;

Next one up - if you know that the row.Item[] is a string then cast is
rather than using ToString() - ToString has greater overhead so you

end

up
with.

sFBU = (string)row["FBU"];

And back to the first one - why not use a foreach(...) loop rather

than

by
index? or is there a specific requirement to go through like that?

All of the above are more than likely negligible in terms of performance -
rem out the interop call and see how long it takes - I'm guessing
fractions
of a second.

Which leads us to believe that it must be the marshalling that is causing
the problem.

Could you post the zip4Parm structure as that's probably where we

make
the

changes - as you're passing in a LPStruct then I'm assuming that the
DllImport statement doesn't need to specify Unicode or ANSI.

cheers,

g

PS Spent the weekend doing Interop so lets get to the bottom of this one -
I
got beat by .Net as the code I was up against required __cdecl callbacks
via
delegates which aren't supported without an IL tweak using ILDASM... :-(


The code's really short...and really easy. And this particular run takes
over 30 minutes and the speed is very inconsistent. The only long code
is
the zip4Parm struct (I didn't post it) that contains over 60

fields
and

they're all modified during the z4adrinq call. That's why I

assumed

it

could
be the DLL call and the amount of time it takes to fill the struct:

[DllImport("ZIP4_W32.DLL")]
public static extern int z4adrinq([In,
Out][MarshalAs(UnmanagedType.LPStruct)] ZIP4_PARM zip4_parm);
private void btn_run_Click(object sender, System.EventArgs e)
{
.....
for (int i=0;i<Datatable_AuditList.Rows.Count;i++) //count = 15,000
{
Application.DoEvents();
sFBU = Datatable_AuditList.Rows["FBU"].ToString();
sDEL = Datatable_AuditList.Rows["DEL"].ToString();
sCTY = Datatable_AuditList.Rows["CTY"].ToString();
sZ4 = Datatable_AuditList.Rows["Z4"].ToString();
ZIP4_PARM zip4Parm = new ZIP4_PARM();
zip4Parm.iprurb = sFBU;
zip4Parm.iadl1 = sDEL;
zip4Parm.ictyi = sCTY;
int iError = z4adrinq(zip4Parm); //dll call - zip4Parm's 60+
fields are modified
statusBar.Panels[0].Text = "Row Count = " + iRecCount.ToString()
+
"
rows ";
iRecCount++;
}
}

Thanks again,
VM


I guess that depends on the Interops in question!

If you think it through then some Interop operations could certainly
be
tuned - for example - if you've got a 100MB of string data and you're
passing it through Interop then the following would be true:

1. .Net stores it's string as Unicode
2. Calling an ANSI Interop function would require converting those
strings to ANSI to and from the Interop call
3. That's got to be slower than calling a UNICODE Interop function

I think the example above shows how Interop could be slower than the
raw
C++
but there are so many other factors involved. In my experience of
calls
via
Interop I haven't noticed any particular slow down so the

profiler
wou

ld
certainly identify which bit of code is causing the problem.

It could also be a GC issue - why don't you post the basic code

and

it
may
be obvious (or probably won't be!)

cheers,

g


Thanks for your reply. Would interops naturally make the system
slower
or
would it be something with my code? In other words, can the

speed

be
improved or is it a dot net limitation?

VM

Can I suggest you hook up a profiler to see where the problem
lies...

Chances are it's a Interop problem but difficult to tell without
the
source
code etc...

There are some very good profilers out there and most have a 30
day
trial


cheers,

g

In my Windows app, I'm running a batch process that's composed
of
a
FOR
loop
that'll run 15,000 times (datatable row count), copy cthe data
of
each
row -3 fields- to a struct, and send the strct to an external
method
(using
DLLImport). The problem is that the processing speed is very
inconsistent-
it may run very FAST during certain points but then it

runs
VERY

slow
at
other moments (especially when it's getting to the end of the
table:
after
12,000 records). I have an idea of the speed because I

have

a previously
made

 

[VM]

Just to be sure, if I use Unicode and the struct doesn't have legible data
(after being modified), then I must use ANSI? Isn't there some way to cast
the data so I can keep on using Unicode?

VM

Sorry - you need to find out which of ANSI vs Unicode the destination
function is written in and use the appropriate one.

(Do you know the difference? If not then ANSI is a single byte ber character
whereas Unicode takes up two bytes per character.)

cheers,

g

For 1,000 records, the difference was 66 sec. vs. 130 secs. which is pretty
good. I haven't tried it with the 15,000 records yet. But the old system did
the 15,000 records in just under 4 mins.

VM

Did setting ANSI vs Unicode make any differences?

g


Thanks for all your help. I just fixed all those details you told me
about.
About the struct, here's a section of it. I'm not going to post the whole
thing because the rest is the same as this. In the C version of the
application, the struct was in a header file so they didn't have to

do
any

marshalling. Also, during the run I noticed that the loop may be in a
particular record for longer (maybe a second or more). Basically, the
extern method finds information based on the data I copied to the
ZIP4_PARM
struct and writes it to the struct in other empty fields when it

finds
is.

It's similar to a database :

namespace ZipMaster
{
...
[StructLayout(LayoutKind.Sequential, CharSet=CharSet.Ansi)]
public class ZIP4_PARM
{
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=4 )]
public string rsvd0;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string iadl1;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string iadl2;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string ictyi;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=3 )]
public string istai;
public string county;
public short respn;
public char retcc;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=12 )]
public string adrkey;
public char auto_zone_ind;
public struct footer
{
public char a;
public char b;
public char c;
public char d;
public char e;
}
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=6)]
public string rsvd3;
}
}

Thanks again,
VM

OK - a couple of good practices things to point out first

I know the compiler may pick these up but better with than without.

Accessing a row by index is probably not the fastest way so get yourself
a
reference to the row and use that rather than indexing into the table
each
time

e.g.
DataRow row = Datatable_AuditList.Rows;

Next one up - if you know that the row.Item[] is a string then

cast

is

rather than using ToString() - ToString has greater overhead so

you
end

up
with.

sFBU = (string)row["FBU"];

And back to the first one - why not use a foreach(...) loop rather than
by
index? or is there a specific requirement to go through like that?

All of the above are more than likely negligible in terms of
performance -
rem out the interop call and see how long it takes - I'm guessing
fractions
of a second.

Which leads us to believe that it must be the marshalling that is
causing
the problem.

Could you post the zip4Parm structure as that's probably where we make
the
changes - as you're passing in a LPStruct then I'm assuming that the
DllImport statement doesn't need to specify Unicode or ANSI.

cheers,

g

PS Spent the weekend doing Interop so lets get to the bottom of this
one -
I
got beat by .Net as the code I was up against required __cdecl callbacks
via
delegates which aren't supported without an IL tweak using

ILDASM...
:-(

The code's really short...and really easy. And this particular run
takes
over 30 minutes and the speed is very inconsistent. The only long
code
is
the zip4Parm struct (I didn't post it) that contains over 60 fields
and
they're all modified during the z4adrinq call. That's why I

assumed

it
could
be the DLL call and the amount of time it takes to fill the struct:

[DllImport("ZIP4_W32.DLL")]
public static extern int z4adrinq([In,
Out][MarshalAs(UnmanagedType.LPStruct)] ZIP4_PARM zip4_parm);
private void btn_run_Click(object sender, System.EventArgs e)
{
.....
for (int i=0;i<Datatable_AuditList.Rows.Count;i++) //count =
15,000
{
Application.DoEvents();
sFBU = Datatable_AuditList.Rows["FBU"].ToString();
sDEL = Datatable_AuditList.Rows["DEL"].ToString();
sCTY = Datatable_AuditList.Rows["CTY"].ToString();
sZ4 = Datatable_AuditList.Rows["Z4"].ToString();
ZIP4_PARM zip4Parm = new ZIP4_PARM();
zip4Parm.iprurb = sFBU;
zip4Parm.iadl1 = sDEL;
zip4Parm.ictyi = sCTY;
int iError = z4adrinq(zip4Parm); //dll call -

zip4Parm's
60+

fields are modified
statusBar.Panels[0].Text = "Row Count = " +
iRecCount.ToString()
+
"
rows ";
iRecCount++;
}
}

Thanks again,
VM


I guess that depends on the Interops in question!

If you think it through then some Interop operations could certainly
be
tuned - for example - if you've got a 100MB of string data and
you're
passing it through Interop then the following would be true:

1. .Net stores it's string as Unicode
2. Calling an ANSI Interop function would require converting
those
strings to ANSI to and from the Interop call
3. That's got to be slower than calling a UNICODE Interop
function

I think the example above shows how Interop could be slower

than
the

raw
C++
but there are so many other factors involved. In my experience of
calls
via
Interop I haven't noticed any particular slow down so the profiler
wou
ld
certainly identify which bit of code is causing the problem.

It could also be a GC issue - why don't you post the basic

code
and

it
may
be obvious (or probably won't be!)

cheers,

g


Thanks for your reply. Would interops naturally make the system
slower
or
would it be something with my code? In other words, can the speed
be
improved or is it a dot net limitation?

VM

Can I suggest you hook up a profiler to see where the problem
lies...

Chances are it's a Interop problem but difficult to tell without
the
source
code etc...

There are some very good profilers out there and most have

a
30

day
trial


cheers,

g

In my Windows app, I'm running a batch process that's composed
of
a
FOR
loop
that'll run 15,000 times (datatable row count), copy

cthe
data

of
each
row -3 fields- to a struct, and send the strct to an external
method
(using
DLLImport). The problem is that the processing speed is very
inconsistent-
it may run very FAST during certain points but then it runs
VERY
slow
at
other moments (especially when it's getting to the end

of
the

table:
after
12,000 records). I have an idea of the speed because I

have

a
counter
being
displayed in the form (with an Application.DoEvents()). I've
eliminated
the
DoEvents but it's still inconsistent.

This is a new version of an application that was previously
made
in
MS
C/C++. The only differences between my C# app and the
previous
version
(besides the language it was written in) is that I had to
recreate
the
struct in C# (which the external method receives as parameter
and
modifies)
and that I use the DLLImport for the method. Would this

be

a

 

[VM]

If I use Unicode and the struct doesn't have legible data (after being
modified), then I must use ANSI? Isn't there some way to cast the data so I
can keep on using Unicode? Since the DLL is proprietary, I have no way of
finding out what they used to write it.

VM

Sorry - you need to find out which of ANSI vs Unicode the destination
function is written in and use the appropriate one.

(Do you know the difference? If not then ANSI is a single byte ber character
whereas Unicode takes up two bytes per character.)

cheers,

g

For 1,000 records, the difference was 66 sec. vs. 130 secs. which is pretty
good. I haven't tried it with the 15,000 records yet. But the old system did
the 15,000 records in just under 4 mins.

VM

Did setting ANSI vs Unicode make any differences?

g


Thanks for all your help. I just fixed all those details you told me
about.
About the struct, here's a section of it. I'm not going to post the whole
thing because the rest is the same as this. In the C version of the
application, the struct was in a header file so they didn't have to

do
any

marshalling. Also, during the run I noticed that the loop may be in a
particular record for longer (maybe a second or more). Basically, the
extern method finds information based on the data I copied to the
ZIP4_PARM
struct and writes it to the struct in other empty fields when it

finds
is.

It's similar to a database :

namespace ZipMaster
{
...
[StructLayout(LayoutKind.Sequential, CharSet=CharSet.Ansi)]
public class ZIP4_PARM
{
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=4 )]
public string rsvd0;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string iadl1;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string iadl2;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string ictyi;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=3 )]
public string istai;
public string county;
public short respn;
public char retcc;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=12 )]
public string adrkey;
public char auto_zone_ind;
public struct footer
{
public char a;
public char b;
public char c;
public char d;
public char e;
}
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=6)]
public string rsvd3;
}
}

Thanks again,
VM

OK - a couple of good practices things to point out first

I know the compiler may pick these up but better with than without.

Accessing a row by index is probably not the fastest way so get yourself
a
reference to the row and use that rather than indexing into the table
each
time

e.g.
DataRow row = Datatable_AuditList.Rows;

Next one up - if you know that the row.Item[] is a string then

cast

is

rather than using ToString() - ToString has greater overhead so

you
end

up
with.

sFBU = (string)row["FBU"];

And back to the first one - why not use a foreach(...) loop rather than
by
index? or is there a specific requirement to go through like that?

All of the above are more than likely negligible in terms of
performance -
rem out the interop call and see how long it takes - I'm guessing
fractions
of a second.

Which leads us to believe that it must be the marshalling that is
causing
the problem.

Could you post the zip4Parm structure as that's probably where we make
the
changes - as you're passing in a LPStruct then I'm assuming that the
DllImport statement doesn't need to specify Unicode or ANSI.

cheers,

g

PS Spent the weekend doing Interop so lets get to the bottom of this
one -
I
got beat by .Net as the code I was up against required __cdecl callbacks
via
delegates which aren't supported without an IL tweak using

ILDASM...
:-(

The code's really short...and really easy. And this particular run
takes
over 30 minutes and the speed is very inconsistent. The only long
code
is
the zip4Parm struct (I didn't post it) that contains over 60 fields
and
they're all modified during the z4adrinq call. That's why I

assumed

it
could
be the DLL call and the amount of time it takes to fill the struct:

[DllImport("ZIP4_W32.DLL")]
public static extern int z4adrinq([In,
Out][MarshalAs(UnmanagedType.LPStruct)] ZIP4_PARM zip4_parm);
private void btn_run_Click(object sender, System.EventArgs e)
{
.....
for (int i=0;i<Datatable_AuditList.Rows.Count;i++) //count =
15,000
{
Application.DoEvents();
sFBU = Datatable_AuditList.Rows["FBU"].ToString();
sDEL = Datatable_AuditList.Rows["DEL"].ToString();
sCTY = Datatable_AuditList.Rows["CTY"].ToString();
sZ4 = Datatable_AuditList.Rows["Z4"].ToString();
ZIP4_PARM zip4Parm = new ZIP4_PARM();
zip4Parm.iprurb = sFBU;
zip4Parm.iadl1 = sDEL;
zip4Parm.ictyi = sCTY;
int iError = z4adrinq(zip4Parm); //dll call -

zip4Parm's
60+

fields are modified
statusBar.Panels[0].Text = "Row Count = " +
iRecCount.ToString()
+
"
rows ";
iRecCount++;
}
}

Thanks again,
VM


I guess that depends on the Interops in question!

If you think it through then some Interop operations could certainly
be
tuned - for example - if you've got a 100MB of string data and
you're
passing it through Interop then the following would be true:

1. .Net stores it's string as Unicode
2. Calling an ANSI Interop function would require converting
those
strings to ANSI to and from the Interop call
3. That's got to be slower than calling a UNICODE Interop
function

I think the example above shows how Interop could be slower

than
the

raw
C++
but there are so many other factors involved. In my experience of
calls
via
Interop I haven't noticed any particular slow down so the profiler
wou
ld
certainly identify which bit of code is causing the problem.

It could also be a GC issue - why don't you post the basic

code
and

it
may
be obvious (or probably won't be!)

cheers,

g


Thanks for your reply. Would interops naturally make the system
slower
or
would it be something with my code? In other words, can the speed
be
improved or is it a dot net limitation?

VM

Can I suggest you hook up a profiler to see where the problem
lies...

Chances are it's a Interop problem but difficult to tell without
the
source
code etc...

There are some very good profilers out there and most have

a
30

day
trial


cheers,

g

In my Windows app, I'm running a batch process that's composed
of
a
FOR
loop
that'll run 15,000 times (datatable row count), copy

cthe
data

of
each
row -3 fields- to a struct, and send the strct to an external
method
(using
DLLImport). The problem is that the processing speed is very
inconsistent-
it may run very FAST during certain points but then it runs
VERY
slow
at
other moments (especially when it's getting to the end

of
the

table:
after
12,000 records). I have an idea of the speed because I

have

a
counter
being
displayed in the form (with an Application.DoEvents()). I've
eliminated
the
DoEvents but it's still inconsistent.

This is a new version of an application that was previously
made
in
MS
C/C++. The only differences between my C# app and the
previous
version
(besides the language it was written in) is that I had to
recreate
the
struct in C# (which the external method receives as parameter
and
modifies)
and that I use the DLLImport for the method. Would this

be

a

 

[Michael Giagnocavo [MVP]]

If the unmanaged code is expecting ANSI, and you pass Unicode, it's just
going to mess up .
-mike
MVP

If I use Unicode and the struct doesn't have legible data (after being
modified), then I must use ANSI? Isn't there some way to cast the data so
I
can keep on using Unicode? Since the DLL is proprietary, I have no way of
finding out what they used to write it.

VM

Sorry - you need to find out which of ANSI vs Unicode the destination
function is written in and use the appropriate one.

(Do you know the difference? If not then ANSI is a single byte ber character
whereas Unicode takes up two bytes per character.)

cheers,

g

For 1,000 records, the difference was 66 sec. vs. 130 secs. which is pretty
good. I haven't tried it with the 15,000 records yet. But the old
system did
the 15,000 records in just under 4 mins.

VM

Did setting ANSI vs Unicode make any differences?

g


Thanks for all your help. I just fixed all those details you told
me
about.
About the struct, here's a section of it. I'm not going to post the
whole
thing because the rest is the same as this. In the C version of the
application, the struct was in a header file so they didn't have to do
any
marshalling. Also, during the run I noticed that the loop may be in a
particular record for longer (maybe a second or more). Basically, the
extern method finds information based on the data I copied to the
ZIP4_PARM
struct and writes it to the struct in other empty fields when it finds
is.
It's similar to a database :

namespace ZipMaster
{
...
[StructLayout(LayoutKind.Sequential, CharSet=CharSet.Ansi)]
public class ZIP4_PARM
{
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=4 )]
public string rsvd0;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string iadl1;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string iadl2;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string ictyi;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=3 )]
public string istai;
public string county;
public short respn;
public char retcc;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=12 )]
public string adrkey;
public char auto_zone_ind;
public struct footer
{
public char a;
public char b;
public char c;
public char d;
public char e;
}
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=6)]
public string rsvd3;
}
}

Thanks again,
VM

OK - a couple of good practices things to point out first

I know the compiler may pick these up but better with than without.

Accessing a row by index is probably not the fastest way so get
yourself
a
reference to the row and use that rather than indexing into the table
each
time

e.g.
DataRow row = Datatable_AuditList.Rows;

Next one up - if you know that the row.Item[] is a string then

cast

is
rather than using ToString() - ToString has greater overhead so you
end
up
with.

sFBU = (string)row["FBU"];

And back to the first one - why not use a foreach(...) loop
rather
than
by
index? or is there a specific requirement to go through like
that?

All of the above are more than likely negligible in terms of
performance -
rem out the interop call and see how long it takes - I'm guessing
fractions
of a second.

Which leads us to believe that it must be the marshalling that is
causing
the problem.

Could you post the zip4Parm structure as that's probably where we make
the
changes - as you're passing in a LPStruct then I'm assuming that the
DllImport statement doesn't need to specify Unicode or ANSI.

cheers,

g

PS Spent the weekend doing Interop so lets get to the bottom of this
one -
I
got beat by .Net as the code I was up against required __cdecl
callbacks
via
delegates which aren't supported without an IL tweak using ILDASM...
:-(


The code's really short...and really easy. And this particular run
takes
over 30 minutes and the speed is very inconsistent. The only long
code
is
the zip4Parm struct (I didn't post it) that contains over 60 fields
and
they're all modified during the z4adrinq call. That's why I assumed
it
could
be the DLL call and the amount of time it takes to fill the struct:

[DllImport("ZIP4_W32.DLL")]
public static extern int z4adrinq([In,
Out][MarshalAs(UnmanagedType.LPStruct)] ZIP4_PARM zip4_parm);
private void btn_run_Click(object sender, System.EventArgs e)
{
.....
for (int i=0;i<Datatable_AuditList.Rows.Count;i++) //count =
15,000
{
Application.DoEvents();
sFBU = Datatable_AuditList.Rows["FBU"].ToString();
sDEL = Datatable_AuditList.Rows["DEL"].ToString();
sCTY = Datatable_AuditList.Rows["CTY"].ToString();
sZ4 = Datatable_AuditList.Rows["Z4"].ToString();
ZIP4_PARM zip4Parm = new ZIP4_PARM();
zip4Parm.iprurb = sFBU;
zip4Parm.iadl1 = sDEL;
zip4Parm.ictyi = sCTY;
int iError = z4adrinq(zip4Parm); //dll call - zip4Parm's
60+
fields are modified
statusBar.Panels[0].Text = "Row Count = " +
iRecCount.ToString()
+
"
rows ";
iRecCount++;
}
}

Thanks again,
VM


I guess that depends on the Interops in question!

If you think it through then some Interop operations could
certainly
be
tuned - for example - if you've got a 100MB of string data
and
you're
passing it through Interop then the following would be true:

1. .Net stores it's string as Unicode
2. Calling an ANSI Interop function would require converting
those
strings to ANSI to and from the Interop call
3. That's got to be slower than calling a UNICODE Interop
function

I think the example above shows how Interop could be slower than
the
raw
C++
but there are so many other factors involved. In my
experience of
calls
via
Interop I haven't noticed any particular slow down so the profiler
wou
ld
certainly identify which bit of code is causing the problem.

It could also be a GC issue - why don't you post the basic code
and
it
may
be obvious (or probably won't be!)

cheers,

g


Thanks for your reply. Would interops naturally make the system
slower
or
would it be something with my code? In other words, can the
speed
be
improved or is it a dot net limitation?

VM

Can I suggest you hook up a profiler to see where the problem
lies...

Chances are it's a Interop problem but difficult to tell
without
the
source
code etc...

There are some very good profilers out there and most
have a
30
day
trial


cheers,

g

In my Windows app, I'm running a batch process that's
composed
of
a
FOR
loop
that'll run 15,000 times (datatable row count), copy cthe
data
of
each
row -3 fields- to a struct, and send the strct to an
external
method
(using
DLLImport). The problem is that the processing speed
is
very
inconsistent-
it may run very FAST during certain points but then it runs
VERY
slow
at
other moments (especially when it's getting to the end of
the
table:
after
12,000 records). I have an idea of the speed because I have
a
counter
being
displayed in the form (with an Application.DoEvents()). I've
eliminated
the
DoEvents but it's still inconsistent.

This is a new version of an application that was previously
made
in
MS
C/C++. The only differences between my C# app and the
previous
version
(besides the language it was written in) is that I had to
recreate
the
struct in C# (which the external method receives as
parameter
and
modifies)
and that I use the DLLImport for the method. Would this

be

a
reason
why
it's
so slow? Also, the memory usage (in Task Manager) is at 100%
when
running.

Thanks,
VM

 

[VM]

So assuming it just accepts ANSI, would it be possible that calling this DLL
function (made in C) from within a C application is so much faster than
converting the C-struct to C# and marshalling the data for the interop?
The previous version (made in C) did the whole process in less than 4 mins.
while the C# version takes over 30 mins.

Vaughn

If the unmanaged code is expecting ANSI, and you pass Unicode, it's just
going to mess up .
-mike
MVP

If I use Unicode and the struct doesn't have legible data (after being
modified), then I must use ANSI? Isn't there some way to cast the data so
I
can keep on using Unicode? Since the DLL is proprietary, I have no way of
finding out what they used to write it.

VM

Sorry - you need to find out which of ANSI vs Unicode the destination
function is written in and use the appropriate one.

(Do you know the difference? If not then ANSI is a single byte ber character
whereas Unicode takes up two bytes per character.)

cheers,

g


For 1,000 records, the difference was 66 sec. vs. 130 secs. which is
pretty
good. I haven't tried it with the 15,000 records yet. But the old
system
did
the 15,000 records in just under 4 mins.

VM

Did setting ANSI vs Unicode make any differences?

g


Thanks for all your help. I just fixed all those details you told
me
about.
About the struct, here's a section of it. I'm not going to post the
whole
thing because the rest is the same as this. In the C version of the
application, the struct was in a header file so they didn't have

to
do

any
marshalling. Also, during the run I noticed that the loop may be

in
a

particular record for longer (maybe a second or more).

Basically,
the

extern method finds information based on the data I copied to the
ZIP4_PARM
struct and writes it to the struct in other empty fields when it finds
is.
It's similar to a database :

namespace ZipMaster
{
...
[StructLayout(LayoutKind.Sequential, CharSet=CharSet.Ansi)]
public class ZIP4_PARM
{
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=4 )]
public string rsvd0;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string iadl1;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string iadl2;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=51 )]
public string ictyi;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=3 )]
public string istai;
public string county;
public short respn;
public char retcc;
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=12 )]
public string adrkey;
public char auto_zone_ind;
public struct footer
{
public char a;
public char b;
public char c;
public char d;
public char e;
}
[MarshalAs( UnmanagedType.ByValTStr, SizeConst=6)]
public string rsvd3;
}
}

Thanks again,
VM

OK - a couple of good practices things to point out first

I know the compiler may pick these up but better with than without.

Accessing a row by index is probably not the fastest way so get
yourself
a
reference to the row and use that rather than indexing into the
table
each
time

e.g.
DataRow row = Datatable_AuditList.Rows;

Next one up - if you know that the row.Item[] is a string then cast
is
rather than using ToString() - ToString has greater overhead so you
end
up
with.

sFBU = (string)row["FBU"];

And back to the first one - why not use a foreach(...) loop
rather
than
by
index? or is there a specific requirement to go through like
that?

All of the above are more than likely negligible in terms of
performance -
rem out the interop call and see how long it takes - I'm guessing
fractions
of a second.

Which leads us to believe that it must be the marshalling that is
causing
the problem.

Could you post the zip4Parm structure as that's probably where we
make
the
changes - as you're passing in a LPStruct then I'm assuming

that
the

DllImport statement doesn't need to specify Unicode or ANSI.

cheers,

g

PS Spent the weekend doing Interop so lets get to the bottom of this
one -
I
got beat by .Net as the code I was up against required __cdecl
callbacks
via
delegates which aren't supported without an IL tweak using ILDASM...
:-(


The code's really short...and really easy. And this

particular
run

takes
over 30 minutes and the speed is very inconsistent. The only long
code
is
the zip4Parm struct (I didn't post it) that contains over 60
fields
and
they're all modified during the z4adrinq call. That's why I
assumed
it
could
be the DLL call and the amount of time it takes to fill the
struct:

[DllImport("ZIP4_W32.DLL")]
public static extern int z4adrinq([In,
Out][MarshalAs(UnmanagedType.LPStruct)] ZIP4_PARM zip4_parm);
private void btn_run_Click(object sender, System.EventArgs e)
{
.....
for (int i=0;i<Datatable_AuditList.Rows.Count;i++)

//count
=

15,000
{
Application.DoEvents();
sFBU = Datatable_AuditList.Rows["FBU"].ToString();
sDEL = Datatable_AuditList.Rows["DEL"].ToString();
sCTY = Datatable_AuditList.Rows["CTY"].ToString();
sZ4 = Datatable_AuditList.Rows["Z4"].ToString();
ZIP4_PARM zip4Parm = new ZIP4_PARM();
zip4Parm.iprurb = sFBU;
zip4Parm.iadl1 = sDEL;
zip4Parm.ictyi = sCTY;
int iError = z4adrinq(zip4Parm); //dll call - zip4Parm's
60+
fields are modified
statusBar.Panels[0].Text = "Row Count = " +
iRecCount.ToString()
+
"
rows ";
iRecCount++;
}
}

Thanks again,
VM


I guess that depends on the Interops in question!

If you think it through then some Interop operations could
certainly
be
tuned - for example - if you've got a 100MB of string data
and
you're
passing it through Interop then the following would be true:

1. .Net stores it's string as Unicode
2. Calling an ANSI Interop function would require converting
those
strings to ANSI to and from the Interop call
3. That's got to be slower than calling a UNICODE Interop
function

I think the example above shows how Interop could be slower than
the
raw
C++
but there are so many other factors involved. In my
experience
of
calls
via
Interop I haven't noticed any particular slow down so the
profiler
wou
ld
certainly identify which bit of code is causing the problem.

It could also be a GC issue - why don't you post the basic code
and
it
may
be obvious (or probably won't be!)

cheers,

g


Thanks for your reply. Would interops naturally make the
system
slower
or
would it be something with my code? In other words, can the
speed
be
improved or is it a dot net limitation?

VM

Can I suggest you hook up a profiler to see where the
problem
lies...

Chances are it's a Interop problem but difficult to tell
without
the
source
code etc...

There are some very good profilers out there and most
have a
30
day
trial


cheers,

g

In my Windows app, I'm running a batch process that's
composed
of
a
FOR
loop
that'll run 15,000 times (datatable row count), copy cthe
data
of
each
row -3 fields- to a struct, and send the strct to an
external
method
(using
DLLImport). The problem is that the processing speed
is
very
inconsistent-
it may run very FAST during certain points but then it
runs
VERY
slow
at
other moments (especially when it's getting to the

end
of

the
table:
after
12,000 records). I have an idea of the speed because I
have
a
counter
being
displayed in the form (with an Application.DoEvents()).
I've
eliminated
the
DoEvents but it's still inconsistent.

This is a new version of an application that was
previously
made
in
MS
C/C++. The only differences between my C# app and the
previous
version
(besides the language it was written in) is that I

had
to

recreate
the
struct in C# (which the external method receives as
parameter
and
modifies)
and that I use the DLLImport for the method. Would

this
be

a
reason
why
it's
so slow? Also, the memory usage (in Task Manager) is at
100%
when
running.

Thanks,
VM

 

[Michael Giagnocavo [MVP]]

Are you only using ASCII (7-bits)? If so, perhaps you can do your own
marshalling and speed it up (assuming that the marshaller is converting
unicode to some different codepage). The reason being that going to ascii is
simply taking the first 7bits, while true encoding conversion is more
involved.

Perhaps you should summarize the problem now that you've narrowed it down
(right?) and post to ms.public.dotnet.framework.performance.

-Michael
MVP

 

[Gary Hunt]

Looking back at your original code - I think in this circumstance there may
be a "better" way.

Am I right in saying that you are taking data from an SQL server or
equivalent, putting it into a datatable and processing each row?

Well if so - and if we've identified that it is the .Net String -> ANSI
conversion that is taking the time then maybe we can get the data out of the
SQL server in ANSI format and never use unicode...

It would be possible - and stop me if I'm butchering things too much (hacker
at heart!) - to use something like the following from SQL server which would
result in binary data we could put straight into the struct....

SELECT CAST([fbu] AS varbinary) AS fbu, CAST([del] AS varbinary) AS del,
CAST([cty] AS varbinary) AS cty, CAST([z4] AS varbinary) AS z4 FROM

This will result in a datatable / sqldatareader that now has ANSI data in
the columns. They will be presented as byte arrays which is exactly what an
ANSI string is.

What do you think - too much of a fudge or an acceptable performance
compromise...

There are other items that would need to be changed as you are marshalling
as TStr's which have their length before the data itself - so the class
would look like
byte iprubLen;
byte[] iprubLen;

Dunno - just think out loud I guess...

g

 

[VM]

Thanks for your suggestion. Unfortunately, the "database" are a bunch of
proprietary files. Since everything's so prorietary, the only interface I
have to search for data is the DLL call and the struct that I use to pick up
the data.

But thanks anyway,
Vaughn

Looking back at your original code - I think in this circumstance there may
be a "better" way.

Am I right in saying that you are taking data from an SQL server or
equivalent, putting it into a datatable and processing each row?

Well if so - and if we've identified that it is the .Net String -> ANSI
conversion that is taking the time then maybe we can get the data out of the
SQL server in ANSI format and never use unicode...

It would be possible - and stop me if I'm butchering things too much (hacker
at heart!) - to use something like the following from SQL server which would
result in binary data we could put straight into the struct....

SELECT CAST([fbu] AS varbinary) AS fbu, CAST([del] AS varbinary) AS del,
CAST([cty] AS varbinary) AS cty, CAST([z4] AS varbinary) AS z4 FROM

This will result in a datatable / sqldatareader that now has ANSI data in
the columns. They will be presented as byte arrays which is exactly what an
ANSI string is.

What do you think - too much of a fudge or an acceptable performance
compromise...

There are other items that would need to be changed as you are marshalling
as TStr's which have their length before the data itself - so the class
would look like
byte iprubLen;
byte[] iprubLen;

Dunno - just think out loud I guess...

g

Are you only using ASCII (7-bits)? If so, perhaps you can do your own
marshalling and speed it up (assuming that the marshaller is converting
unicode to some different codepage). The reason being that going to

ascii

is
simply taking the first 7bits, while true encoding conversion is more
involved.

Perhaps you should summarize the problem now that you've narrowed it down
(right?) and post to ms.public.dotnet.framework.performance.

-Michael
MVP