thousands/millions of objects

[DKode]

I am re-creating an application that is outdated and originally created
in C++

This app, reads values in, creates objects for each line and populates
fields, then runs some conditions on the data setting various flags,
then outputs it all to a dbf (foxpro) database.

I want to keep the structure somewhat similar to the c++ code, and as a
result I will create a new object for every line read in, this could be
hundreds/thousands/millions of entries.

What should I keep in mind when I know I will be creating this large
number of objects, i guess foremost, uses classes not structs so they
are are not stored on the heap.

any other pointers/advice before I embark? I havent designed an app in
c# that I know will hold massive amounts of information like this, just
want to make sure i have my ducks in a row

 

[Jon Skeet [C# MVP]]

I am re-creating an application that is outdated and originally created
in C++

This app, reads values in, creates objects for each line and populates
fields, then runs some conditions on the data setting various flags,
then outputs it all to a dbf (foxpro) database.

I want to keep the structure somewhat similar to the c++ code, and as a
result I will create a new object for every line read in, this could be
hundreds/thousands/millions of entries.

What should I keep in mind when I know I will be creating this large
number of objects, i guess foremost, uses classes not structs so they
are are not stored on the heap.

any other pointers/advice before I embark? I havent designed an app in
c# that I know will hold massive amounts of information like this, just
want to make sure i have my ducks in a row

Well, I suspect you meant "structs rather than classes" but I'd
recommend using classes anyway.

There is no problem in creating millions of objects in .NET so long as
you have enough memory to handle them all. Keeping an object on the
heap only incurs a small overhead (about 8 bytes currently, I believe,
plus 4 for the reference to the object on x86).

One way to reduce the memory might be to process batches of thousands
of lines at a time rather than millions - do you actually need to read
the *whole* file in before creating/modifying the database?

 

[DKode]

I would have no problem reading in perhaps 10,000 lines first,
processing them and then sending them to the FoxPro database, then
reading the next 10,000 etc...

I am wondering if after re-designing this it will be as fast as the C++
equivalent, i've seen alot of people some specific processes in c++ are
still faster than doing the same in c#

 

[Jon Skeet [C# MVP]]

I would have no problem reading in perhaps 10,000 lines first,
processing them and then sending them to the FoxPro database, then
reading the next 10,000 etc...

In that case you should have no problems at all.

I am wondering if after re-designing this it will be as fast as the C++
equivalent, i've seen alot of people some specific processes in c++ are
still faster than doing the same in c#

I'd expect the bottleneck to be the database operations, personally.
The speed of those will depend on the driver, I imagine.

 

[DKode]

where can I find information about working with FoxPro under C#?

thanks you! You have been very helpful!

 

[Nicholas Paldino [.NET/C# MVP]]

DKode,

In order to use FoxPro in C#, you should probably use the classes in the
System.Data.Odbc namespace, and use an ODBC driver for FoxPro.

Hope this helps.

 

[Jon Skeet [C# MVP]]

where can I find information about working with FoxPro under C#?

If you search for FoxPro C# in Google you get quite a few hits. This
one seems pretty useful:

http://msdn.microsoft.com/library/default.asp?url=/library/en-
us/dv_foxhelp/html/fotskaccessingvisualfoxprodatainvisualstudionet.asp

thanks you! You have been very helpful!

My pleasure.