Reading IIS log and large files

  • Thread starter Thread starter Jmc
  • Start date Start date
J

Jmc

Hi
I'm trying to build a log viewer, its going to be part of a
contentmanagement system and needs some extra features that commersial
logviewers don´t have.

The log files (IIS log) are 200-500mb large and the problem is that it
taket to long to parse them.
I use textreader.readline and split on spases, and then I use that
infrormation.

When looping through the file (without actually doing anything) it
takes about 2-3 seconds and thats the time that a commercial log parser
takes to parse the log.
The commercial log parser also stores the data in a database, I tried
that just for fun, I used a access database and the procedure took
about 10 minutes, not to good.

My question is Can I speed up this?
I would like to have a dataset containing a number of tables,
containing Unique Useragents, ip adresses, visited pages (parsed from
querystring), visitors per hour and so on.
And I would like to do this in the time the commercial application did
it, lets be generous under 10 seconds.
Can it even be done using c#?
How would you guyes solve it?

Thanks
/Jimmy
 
"Jmc"
I'm trying to build a log viewer,
Can it even be done using c#?
Click to expand...

Use the free Microsoft Log Parser 2.2

If Log Parser does not do what you need
then you can add your own "actions" called
"COM plug-ins or input processors" to make
Log Parser 2.2 do it!

And yes Log Parser can work with C# too!

Log parser is a powerful, versatile tool
that provides universal query access to
text-based data such as log files, XML
files and CSV files, as well as key data
sources on the Windows operating system
such as the Event Log, the Registry, the
file system, and Active Directory.

Download details: Log Parser 2.2
http://www.microsoft.com/downloads/...6b-abf8-4c25-91b2-f8d975cf8c07&displaylang=en

Windows Live Search
http://www.live.com/?q=Log+Parser

Read the Log Parser Book!

Windows Live Search
http://www.live.com/?q=Log+Parser+Book

Use the Log Parser forum

Windows Live Search
http://www.live.com/?q=Log+Parser+forum

The many ways of Log Parser 2.2
will most likely do what you need! :)
 
Hi
You are right I could use Log Parser, I actually tried that as a first
resort, but to get all the information that I wanted I needed it make 5
(or so) queries so that didn't make things go very fast.
When the logfile went over 1,3GB (for one day) it just wasn't funny
anymore =).

I solved it by opening the file my self and used Hashtables to collect
the data that intereseted me.
The result was that my application parsed 1,3 gb logfile in about 40
seconds (about 10-15 of them was to pretty up User agents).
I'm "happy" and my customer is satified, so I'll leave it like it is
right now =).

For those who are thinking of writing something like this my recent
experience says: Log parser is the easiest and most nice way to read
log files, but it seems (at least when using .net) its faster to loop
through the file and collect the data.

Thanks for your help.

/Jimmy
 
"Jmc"
You are right I could use Log Parser,
Click to expand...

Glad you got it running but Log Parser
should be able to do it faster (using any
programming language)! :)

So perhaps (when you get the time) you
could go over to the LogParser forum and
see if they can help you improve your Log
Parser queries (you know the customer will
want much more "log" information next time)! :)
 
Back
Top