file i/o in multithreaded apps

[Mark]

Hi...

I've got an app that's pretty multi-threaded. While I'm debugging, I'm
trying to write a log file of events that happen, so I've got a global logger
that has
g_LogFileWriter = new StreamWriter(new FileStream(file, Append, Write.
Share.Read);

void Log2File(string format, params object[] arg)
{
lock (g_LogFileWriter)
{
g_LogFileWriter.WriteLine(format, arg);
g_LogFileWriter.Flush();
}
}

Problem is that when things happen very close together and multiple threads
call this, lines seem to be getting lost. Is there some timing limit on how
tight many threads can write to the same file stream?

Thanks
Mark

 

[Steven Cheng [MSFT]]

Hi Mark,

I agree with Pete that the code here looks quite simple and reasonable
which should work well. For the data that got lost, are they written in
some other particular threads which may has some other synchronizing
operations that may cause it to block or not writing correctly? It would be
helpful, if we can simplify it and track down what kind of data are always
lost. BTW, as Pete mentioned, you'd better not lock on the streamwriter
objecct, but use a simple Object member.

Please feel free to post here if there is anything else you found.

Sincerely,

Steven Cheng

Microsoft MSDN Online Support Lead


Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
(e-mail address removed).

==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications.

This posting is provided "AS IS" with no warranties, and confers no rights.
-----------------------------

From: =?Utf-8?B?TWFyaw==?= <[email protected]>
Subject: file i/o in multithreaded apps
Date: Mon, 7 Apr 2008 14:54:03 -0700

Hi...

I've got an app that's pretty multi-threaded. While I'm debugging, I'm
trying to write a log file of events that happen, so I've got a global logger
that has
g_LogFileWriter = new StreamWriter(new FileStream(file, Append, Write.
Share.Read);

void Log2File(string format, params object[] arg)
{
lock (g_LogFileWriter)
{
g_LogFileWriter.WriteLine(format, arg);
g_LogFileWriter.Flush();
}
}

Problem is that when things happen very close together and multiple threads
call this, lines seem to be getting lost. Is there some timing limit on how
tight many threads can write to the same file stream?

Thanks
Mark

 

[SoftLion]

"lock" won't lock if 2 simultaneous calls are made from the same thread (yes
this seems impossible but it is).
You must use a better lock. See locking in .NET framework.

 

[Troy]

This is not the problem!

The same thread will, of cause, work in a serialized manner - else nothing
would work at all... So the same thread cannot work in parallel and thereby
interrupt it self...

I agree with Peter and Steven, the problem is some where else in the code.
Maybe you are accessing the stream somewhere else?

 

[Mark]

Hi Everybody...

Thanks for answering.

I'll try to whittle this down to a small reproducible case, but the
background is this: I have an HttpListener application that takes connections
async, queues the work in the System.ThreadPool, processes the request, then
sends the result back out. The various stages log various things. I'm not
sure whether HttpListener.BeginGetContext calls get filled by ThreadPool
worker threads or I/O threads, but each request gets handled by at least 2
threads.

When I'm stress-testing my app with a tester program simulating 100
concurrent users, I'm seeing gaps in the log file using the code I posted -
log lines that every user should have just missing for some.

Interestingly, my tester program just uses Console.WriteLine then I redirect
the output to a file and that doesn't seem to be missing lines. But it's the
missing lines on the server side that are the ones I need to look at.

I didn't see anything obviously wrong with the code either; the only thing I
could think of was that there might be some low-level buffering thing that
let some output get scratched.

By the by, I've seen a lot of conflicting cross posts about the etiquette of
what to lock. If this is the only use of the stream writer anywhere, what's
the benefit of having another lock object?

Thanks
Mark

Hi Mark,

I agree with Pete that the code here looks quite simple and reasonable
which should work well. For the data that got lost, are they written in
some other particular threads which may has some other synchronizing
operations that may cause it to block or not writing correctly? It would be
helpful, if we can simplify it and track down what kind of data are always
lost. BTW, as Pete mentioned, you'd better not lock on the streamwriter
objecct, but use a simple Object member.

Please feel free to post here if there is anything else you found.

Sincerely,

Steven Cheng

Microsoft MSDN Online Support Lead


Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
(e-mail address removed).

==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications.

This posting is provided "AS IS" with no warranties, and confers no rights.
-----------------------------

From: =?Utf-8?B?TWFyaw==?= <[email protected]>
Subject: file i/o in multithreaded apps
Date: Mon, 7 Apr 2008 14:54:03 -0700

Hi...

I've got an app that's pretty multi-threaded. While I'm debugging, I'm
trying to write a log file of events that happen, so I've got a global logger
that has
g_LogFileWriter = new StreamWriter(new FileStream(file, Append, Write.
Share.Read);

void Log2File(string format, params object[] arg)
{
lock (g_LogFileWriter)
{
g_LogFileWriter.WriteLine(format, arg);
g_LogFileWriter.Flush();
}
}

Problem is that when things happen very close together and multiple threads
call this, lines seem to be getting lost. Is there some timing limit on how
tight many threads can write to the same file stream?

Thanks
Mark