A brainstorming session on "pause-able" for & while loops.

W

Wazza

G'Day,

I have a re-occurring problem and wish to design a framework or
pattern to address it. I have some Ideas but I would like to do some
brainstorming with my peers before commencing.

The problem:

Several computational tasks I am running for my PhD work require
several weeks to complete and I will to be able to pause these tasks
and close the application periodically. It can be difficult to run a
machine several weeks without the windows operating system fowling up
at some point and requiring a restart... also it would need to be
paused so I can play crysis :) .

Any way these computations generally run in a while loop inside of one
of many (4) threads concurrently working on the problem..

mythread() { while(!finished) { getDataToworkOn(); doWork();
postResults();} }

sometimes the algorithm is in the pooling form..

while(!solutionfound) { ThreadPool.QueueUserWorkItem(testdati[++i]) }

I am looking to create something like a SuspendableTask class... One
of the requirements is that the "SuspendableTask" can be running many
other child "SuspendableTasks" and when the parent task is suspended
all children must also be suspended.

Here is my thought on a solution (not actual code just a quick psuedo
code and lefty out thread safety stuff), pease post your own
thoughts...

class SuspendableTask : ISuspendableTask
{
public static List<string, ISuspendableTask> masterTaskList;

bool taskPaused=false;
ITaskData data;
public volatile int workload=0;
public SuspendableTask(string name, ITaskData d)
{
data=d;
masterTaskList.add(name, this);
}

public static void pause(string name) {masterTaskList[name].paused =
true;}

public bool continue() {return !(paused || data.solved())};
}

then each thread could look something like

mythread() {
while(SuspendableTask.masterTaskList["trainNN"].continue())
{...do work ...}
}


Well to me I think there is a more elegant solution out there.... and
I'm sure someones already using it.

thanks for any sugestions,

-dm
 
C

Chris Mullins [MVP - C#]

It sounds like you've got the wrong algorithm entierly, to be honest.

You high-level description just screams "MSMQ", "Persistant Storage",
"Guaranteed Delivery", "Transactions". You'll probably want a Queue of Work
To Do, and a Queue of Results.

Your loop would, inside a Transaction:
0 - Create a Transaction
1 - Pull the work item out of the Work Queue
2 - Process it
3 - put the data into the Result queue
4 - Commit the Transaction

On some periodic schedule (one a day, once a week, etc) you would pull out
the Results and write them to long term storage (a database / file / etc).
At the same time you would populate the Work To Do queue with items that
need processing.

This is loosley coupled approach, with persistance and transactions, is
really going to be the only type of answer you can go with given your
descriptions. Anything else runs the risk of missing work items, or having
issues if the machine crashes, your process runs out of memory, or something
else unexpected happens.

--
Chris Mullins

Wazza said:
G'Day,

I have a re-occurring problem and wish to design a framework or
pattern to address it. I have some Ideas but I would like to do some
brainstorming with my peers before commencing.

The problem:

Several computational tasks I am running for my PhD work require
several weeks to complete and I will to be able to pause these tasks
and close the application periodically. It can be difficult to run a
machine several weeks without the windows operating system fowling up
at some point and requiring a restart... also it would need to be
paused so I can play crysis :) .

Any way these computations generally run in a while loop inside of one
of many (4) threads concurrently working on the problem..

mythread() { while(!finished) { getDataToworkOn(); doWork();
postResults();} }

sometimes the algorithm is in the pooling form..

while(!solutionfound) { ThreadPool.QueueUserWorkItem(testdati[++i]) }

I am looking to create something like a SuspendableTask class... One
of the requirements is that the "SuspendableTask" can be running many
other child "SuspendableTasks" and when the parent task is suspended
all children must also be suspended.

Here is my thought on a solution (not actual code just a quick psuedo
code and lefty out thread safety stuff), pease post your own
thoughts...

class SuspendableTask : ISuspendableTask
{
public static List<string, ISuspendableTask> masterTaskList;

bool taskPaused=false;
ITaskData data;
public volatile int workload=0;
public SuspendableTask(string name, ITaskData d)
{
data=d;
masterTaskList.add(name, this);
}

public static void pause(string name) {masterTaskList[name].paused =
true;}

public bool continue() {return !(paused || data.solved())};
}

then each thread could look something like

mythread() {
while(SuspendableTask.masterTaskList["trainNN"].continue())
{...do work ...}
}


Well to me I think there is a more elegant solution out there.... and
I'm sure someones already using it.

thanks for any sugestions,

-dm
Click to expand...
 
W

Wazza

Thanks Chris,

Well your idea is great unfortunately only about 1/3 of the algorithms
I am working on will fit in to a "work to do" type schedule. The rest
can't easily be rationalized as a particular set of transactions that
need to be processed.

The computations I have that are not suited to a transaction can be
characterised in 2 forms:

form 1) Time is spent in low frequency for/while loops nested up to 3
deep where the inner-most loop has periods of several minutes and the
outermost loop has periods of weeks. The inner most loop runs some
algorithmic processing and in some cases retrieves data from storage &
communicates with a matlab session

form 2) A bunch of threads(not all the same algorithm) are looking at
a central piece of data and working on it... (a kind of simulation)...
it is a threading nightmare but all the threads run on a while loops
with frequencies between 1ms and 10s. The simulation runs until a
condition is met.

The commonalty between all my algorithms is that there are points at
the end of certain loops at which it would be an ideal time to pause
and serailizse the work in order to resume another day.

Here is another Idea I am starting to think of a managed while loop
which defines 1) a procedure called in each iteration of the loop
where in the work should take place... and 2) a data set created
before the loop starts and passed to and modified by "1" as the only
data "1" can use....

The managed while loop would be given a name and a static object can
be used to allow the application to cause a managed loop (running on
another thread(s)) to serialize "2" and exit at the end of the loop.

To Further complicate things the managed loop could define a parent
loop structure as a function to be called before running itself and a
function to be called after running itself. if the after function
would return true then the before function is called again follwed by
the managed loop followed by the after function again.. This way I
could handle pausing at the innermost loop of a nested loop
structure...

ie it would be used as:

{
object data = new foobar();
MLoop w = new Mloop(new MLoopIterateDelegate( myLoopFunc), ref data);
w.addParentLoop(new ParentbeforeDelegate(before), new
parentAfterDelegate(after));

w.runAsync();
sleep (10000);

loopState ls = w.pause();
w.Dispose();
w = new MLoop.fromResumeData(ls);
w.runAsync();

}

void myLoopFunc(ref object data) {...};

void before(ref object data) { ...do stuff... };
bool after(ref object data) { return isSolved(data); };


This IS brainstorming... and i just tossing this out there (I can see
its by no means an ideal solution).

Well between this post and the last is the two thoughts I have ad on a
pause-able loop.... anyone else.

 
Top