Sorting data in fixed file format

  • Thread starter Thread starter booksnore
  • Start date Start date
B

booksnore

I'm looking for a way to sort text files consisting of fixed file
format. The files are big, typically over 10 million records and they
consist of about 100 fields with the record being over 600 bytes in
length. I need to
sort on a combination of 8 of the fields. Has anyobe attempted a to sort
data in this way using C#? Performance and scalability are the main
factors - the input data I
have is likely to grow probably up to 40 million records. I want to
avoid loading data into a database just to sort it.

Joe
 
Hi,

IMO your best (if not unique) solution is to put everything in a database,
sort it and then export it. With this amount of data there is no a better
solution.

Importing/Exporting is very easy using DTS packages. let me know if you
need code for it
 
You have over 10 million records in a fixed file format, and you don't want
to put it into a database? Dude, that's what databases are *for*. Fixed file
formats are fine for storing static data. But filtering, selecting,
ordering, that's all stuff for databases.

--
HTH,

Kevin Spencer
Microsoft MVP
..Net Developer
You can lead a fish to a bicycle,
but it takes a very long time,
and the bicycle has to *want* to change.
 
Thanks Ignacio, what sort of code is it that you have? I would be
interested in taking a look.
regards
Joe

Joe
 
Hi,

this is what I use, I create a DTS from enterprise manager , where the data
is comnig from and where to put it, then select "create a file" or something
similar in the DTS wizard, it does create a .dts file this is the one you
will use later

Here is the code, note that I change the datasource from the code, you can
do a similar thing with the destination:
Also you need to add a reference to DTS COM library

using DTS;
using System.Data;
using System.Data.SqlClient;



void RunPackage( string packSource, string packName, string dataSource)
{
try
{
Package2Class package = new Package2Class();
object pVarPersistStgOfHost = null;

// if you need to load from file
package.LoadFromStorageFile(
packSource,
null,
null,
null,
packName,
ref pVarPersistStgOfHost);

/*
package.LoadFromSQLServer(
"",
null,
null,
DTSSQLServerStorageFlags.DTSSQLStgFlag_UseTrustedConnection,
null,
null,
null,
"Test Import Package",
ref pVarPersistStgOfHost);
*/

package._Package_Connections.Item(1).DataSource = dataSource;
package.Execute();
package.UnInitialize();

// force Release() on COM object
//
System.Runtime.InteropServices.Marshal.ReleaseComObject(package);
package = null;
}
catch(System.Runtime.InteropServices.COMException e)
{
Console.WriteLine("COMException {0}", e.ErrorCode.ToString() );
Console.WriteLine("{0}", e.Message);
Console.WriteLine("{0}", e.Source);
Console.WriteLine("Stack dump\n{0}\n", e.StackTrace);
Console.ReadLine();
}
catch(System.Exception e)
{
Console.WriteLine("Exception");
Console.WriteLine("{0}", e.Message);
Console.WriteLine("{0}", e.Source);
Console.WriteLine("Stack dump\n{0}\n", e.StackTrace);

Console.ReadLine();
}
}
 
I want to sort the data in the file - that's it nothing else. Unix has a
sort command, I wondered if there was an equivalent in Windows.

Joe
 
Back
Top