Multi-Threading

[Barkingmadscot]

I am having problems with Multi-threading and write to Key Access
Database

I have a log file that and collecting infomation from

getting the info is a problem or is writing it the database, the log
files a big and it takes time to read down as it get further in to log
said, a 100,000 line in

I have changed it to mutli-threading which almost works, the problem
is when writing data to the table. I get dup keys and the whole
stops, this does not happen with the single thread app.

basically i want each thread to wait until thread before finish its
write to the database.

ie.
thread 1 waits for thread 0 to finish writing to table
thread 2 waits for thread 1 to finsih writing to table
thread 4 waits for ..........

during this each for the thread have collected the data they need to
write to the table.

Here is the line code to

Dim intcount As Integer = 1, intcount1 As Integer =
fNoOfLines(fileNm)
Dim AppThreads(4) As Thread
Dim WebData(intcount1) As DoWork

Do Until intcount > intcount1

WebData(Thread) = New DoWork

'################################

Code collecting data is here
################################

'# Setup and start new thread
AppThreads(Thread) = New Thread(AddressOf
WebData(Thread).sWriteWebUsage)
AppThreads(Thread).Start()
System.Threading.Thread.Sleep(15)
intcount = intcount + 12
Loop

I removed all the code that does the data collecting as it works
fine,

I believe my problem is at the Appthreads(thread).start()

 

[Patrice]

the log

files a big and it takes time to read down as it get further in to log

It doesn't look like a normal behavior in first place.. Reading data from
the file should take the same time regardless of the position you are in
this file. I would have checked first, if there is not something else going
wrong in the initial approach.

I get dup keys and the whole
stops, this does not happen with the single thread app.

How are created those keys ?

basically i want each thread to wait until thread before finish its
write to the database.

I don't see this in your code ?

ie.
thread 1 waits for thread 0 to finish writing to table
thread 2 waits for thread 1 to finsih writing to table
thread 4 waits for ..........

during this each for the thread have collected the data they need to
write to the table.

Here is the line code to

Dim intcount As Integer = 1, intcount1 As Integer =
fNoOfLines(fileNm)
Dim AppThreads(4) As Thread
Dim WebData(intcount1) As DoWork

Do Until intcount > intcount1

WebData(Thread) = New DoWork

'################################

Code collecting data is here
################################

'# Setup and start new thread
AppThreads(Thread) = New Thread(AddressOf
WebData(Thread).sWriteWebUsage)
AppThreads(Thread).Start()
System.Threading.Thread.Sleep(15)
intcount = intcount + 12
Loop

I removed all the code that does the data collecting as it works
fine,

I believe my problem is at the Appthreads(thread).start()

I'm not sure you have to deal with multithreading epsecially if you write
anyway to a single table and read from a single file. If what you want is to
insert a file into SQL Server you may want to try to the "bulk copy" classes
in 2.0 or you have peraps an issue in your initial code...

 

[Mr. Arnold]

I believe my problem is at the Appthreads(thread).start()

My solution to this was to set a global flag or flags that each routine in a
given thread would look at. At the top of the routine, I set the flag to
true and false when the routine is exited.

In other While loops that are working within a given thread on a
Thread.sleep(milliseconds), if the flag is false when the thread awakes
then I execute the routine in the While loop, otherwise, the routine within
thread is bypassed and the thread goes back to sleep until the sleep
interval has elapsed again. The flag is checked again when the sleep
interval has elapsed.

 

[Patrice]

Looks like Mr Arnorld better understood, sorry. I thought you already had
synchronization code that wasn't working. I didn't understood you just
wanted to know what is the missing synchronization code you should add...

 

[Guru]

My solution to this was to set a global flag or flags that each routine in
a given thread would look at. At the top of the routine, I set the flag to
true and false when the routine is exited.

In other While loops that are working within a given thread on a
Thread.sleep(milliseconds), if the flag is false when the thread awakes
then I execute the routine in the While loop, otherwise, the routine
within thread is bypassed and the thread goes back to sleep until the
sleep interval has elapsed again. The flag is checked again when the sleep
interval has elapsed.

Whereas a real programmer would fix his problem by telling him to create a
public shared definition of the DataTable and SyncLock it. And a real
programmer would set the thread sleep to TimeOut.Infinite then use
Thread.Interrupt to wake it only it was needed.

 

[Mr. Arnold]

Whereas a real programmer would fix his problem by telling him to create a
public shared definition of the DataTable and SyncLock it. And a real
programmer would set the thread sleep to TimeOut.Infinite then use
Thread.Interrupt to wake it only it was needed.

I have been a real programmer since 1980, and I am still going strong.
Whatever works is the bottom line. It makes no difference as long as it
works. What are you some kind of a lunatic that has to show the world that
you're a programmer? You have not paid your dues and I know it, otherwise,
you wouldn't have shown your John Brown behind parts like this. You are a
real slick deputy-dog.

 

[Mr. Arnold]

<snipped>

Oh, and one other thing, what if the application was using SQL Command ,
Business and Data Access Layer objects and there was no DataTable in the
solution period in your narrow you are a real programmer mindset.

Let's get your vast expertise on that, since you needed to run your mouth.
One shoe doesn't fit all situations. And as far as I am concerned, you were
totally out of line here.

 

[Guru]

<snipped>

Oh, and one other thing, what if the application was using SQL Command ,
Business and Data Access Layer objects and there was no DataTable in the
solution period in your narrow you are a real programmer mindset.

Let's get your vast expertise on that, since you needed to run your mouth.
One shoe doesn't fit all situations. And as far as I am concerned, you
were totally out of line here.

The solution is as simple as you are... fix the SQL.

Eat that, cat cornholer.

 

[Alex Clark]

This sounds a lot to me like you're using a hammer on a screw. If I
understand you correctly, you were having performance issues when reading a
100,000+ line log file and decided to use multi-threading as a solution to
this, which is seldom a solution.

Multi-threading on the desktop is useful when you're executing long running
blocking operations and want the GUI to remain responsive, running
background tasks and similar. Multi-threading on the server is useful if
you're servicing multiple requests simultaneously and perhaps want to take
full advantage of multi-core CPUs. It's nowhere near as useful for
improving performance as some people think it is; a program running a task
across 8 threads is NOT 8x faster than the same program running the same
task on just 1 thread.

Multi-threading should be used cautiously and when necessary as it can
create exactly the kinds of headaches you're having with your code. As
you've probably noticed from this thread, everybody has a different take on
how you should sync your resource access across multiple threads.

You need to investigate *why* your app is slowing down as it reads its way
through that log file. Are you allocating a new string each time you read a
line of the log file? Remember that strings are immutable, so once they're
created they won't be destroyed until the GC decides to. This could be
swamping your working set.

Is there a more efficient design pattern you could follow for reading the
file data and writing it into the database?

 

[Mr. Arnold]

The solution is as simple as you are... fix the SQL.

Eat that, cat cornholer.

Yeah right -- LOL, don't put your John Brown behind parts in my face you
young lunatic American. Now, you go crawl back into your hole that you came
out of acting like a crazy person.

And one wonders why the rest of the world hates America. Well, with
lunatics like you running around in it ready to play cowboy and attack
anything that moves with no provocation, I see the reason, and I am
American.