merge small text tiles to a big file?

[guoqi zheng]

Dear Sir,

One of our applications needs to use millions of small text files. I think
it has some performance problem due to too many text files. I read about the
File System google are using, they use 100-200 mb file size, each files can
contains many objects(webpages).

Is it possible to archieve something like this in DotNet?

I can easily estimate the amount of small text file we needs will be over 10
millions, we will save them in many sub sub folder, each folder contain max.
10,000 files, but still performanance might be a big issue!

regards,

Guoqi Zheng
http://www.ureader.com

 

[Alex Li]

Have you think about (or have the freedom to choose) using a sql
database instead? Afterall, sql database are built for storing very
small content, and say 10 millions rows aren't too extreme...

HTH,
Alex

 

[guoqi zheng]

SQL database is not an option because our number of file will eventually
grow too big for SQL database to handle! We are actually building our own
search engines to search millions of our files, articles etc...

regards,

Guoqi Zheng
http://www.ureader.com