Well, you can't just drop them in a database per se, you would have to
read the data from the files and add each record to the database.
It's a bit of work to get it into the database, but once that is done,
you can very easily fetch all data associated with a user in a fraction
of the time it takes to loop through the files for it. The time needed
to get the data would typically go from several seconds to a few
milliseconds.
To the original poster...
First, I have to agree that the best solution is probably to just copy all
of the data into a relational database, one table for each file, where the
member ID (whatever you're using...name, ID number, whatever) is the key
that ties the tables together. You can then just run through all the keys
and let the database itself handle extraction of the data. It will even
be able to return all of the data as a single record, which you can just
then rewrite out to whatever new text file you want (assuming, of course,
that even once all the data is in a relational database, you really need a
new text file).
If you really don't want to do that, then to improve performance you need
to process the data differently. My first question would be: is the data
ordered in the original files in the first place? If so, then you should
not have to scan all of the files repeatedly...you should be able to read
through them in sequence, merging matching data as you find it.
If the data is not ordered in the original files, then the next best thing
would be to do what a real database would do anyway: index the files.
Scan through each of the files once, creating a new file (or in-memory
data structure, if your original data is few enough that indices for all
of your files will all fit in memory) that contains an sorted index based
on the member ID, so that you can quickly find the records of interest. A
binary search on the index will allow you to find any given record in a
very short period of time...much better than scanning each file repeatedly.
If the files you are processing are small enough (say, less than 1GB of
total data across all of the files, though this is not a hard-and-fast
limit), then you might have some success simply reading all of the files
into memory in some easily-searched data structure (eg a .NET
dictionary-list or sorted linked list). Let .NET do all the work of
looking the data up, and on a computer with enough physical RAM this
should speed things up a lot as well.
These are all general suggestions. You haven't provided much in the way
of detail of your actual data or problem, so it's hard to provide anything
more useful than that.
Pete
