Serialization and Enumerators

[Reuben]

Hi,
I've been trying to rewrite some code which can run as a single method so
that I can save its state - that way I can resume processing the data when I
start the program back up. The data isn't actually stored in an array, due to
memory issues, instead it's generated at runtime by an Enumerator (I used the
yield statement to generate it). The idea was that I could serialize the
Enumerator and save it, so I wouldn't have to regenerate the data. The
problem is that the generated Enumerator object isn't serializable, and
doesn't have any constructors (public or private), which means I can't
perform custom serialization, since I have no way of recreating the object. I
know that if this works, it would be of great use to many people.

Does anyone know how to serialize custom enumerators?
Please reply if you have any ideas at all - I'm really at a dead end.

 

[Marc Gravell]

An enumerator doesn't really *hold* state - it simply walks over it... I'm
not sure that this is the right approach.

First question: have you looked at XmlSerializer, BinaryFormatter, and/or
DataContractSerializer; serialization is built deeply into the framework -
you shouldn't have to (or want to) re-invent everything here...

If you can describe what problem(s) you had/have with any of the above, we
may be able to help...

Marc

 

[Jon Skeet [C# MVP]]

An enumerator doesn't really *hold* state - it simply walks over it... I'm
not sure that this is the right approach.

Well, it holds *some* state. However, that state might not be
serializable. A classic example would be my LineReader class, which
iterates over each line in a TextReader. The TextReader is the state,
which might (for example) know how far through a file it is.

If it didn't need to hold state, the C# 2 compiler wouldn't need to
build a state machine for you, which it does

 

[Lasse Vågsæther Karlsen]

Hi,
I've been trying to rewrite some code which can run as a single method so
that I can save its state - that way I can resume processing the data when I
start the program back up. The data isn't actually stored in an array, due to
memory issues, instead it's generated at runtime by an Enumerator (I used the
yield statement to generate it). The idea was that I could serialize the
Enumerator and save it, so I wouldn't have to regenerate the data. The
problem is that the generated Enumerator object isn't serializable, and
doesn't have any constructors (public or private), which means I can't
perform custom serialization, since I have no way of recreating the object. I
know that if this works, it would be of great use to many people.

Does anyone know how to serialize custom enumerators?
Please reply if you have any ideas at all - I'm really at a dead end.

Is the enumerator

(1) producing new data, or
(2) enumerating over existing data?

If (1), I would serialize out the inputs needed to re-produce the same
data. If (2) I would serialize out the data (basically this is the same
thing, the input to the enumerator).

If the production of data is without input, and is a heavy job, couldn't
you just serialize out the produced data, or will it be different data
produced the next time?

Some more information about why you want to serialize the enumerator
would be helpful.

 

[Marc Gravell]

You know what I mean ;-p

It doesn't (as a rule) hold the actual state of the underlying objects...

Marc

 

[Jon Skeet [C# MVP]]

You know what I mean ;-p

I'm still not entirely sure.

It doesn't (as a rule) hold the actual state of the underlying objects...

Right. Instead, it would hold things like "I'm at byte 200" or "I'm
looking at item 15 in the list".

 

[Reuben]

First of all, thanks for your speedy replies.

1) I have tried using a BinaryFormatter, but but it doesn't work. The
generated IEnumerator isn't serializable and gives me this exception:

System.Runtime.Serialization.SerializationException was caught
Message="Type 'WindowsApplication1.DataGenerator+<Create>d__0' in Assembly
'WindowsApplication1, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'
is not marked as serializable."

2) The data and enumerator are generated by a function using the yield
statement, so I can't serialize the source data because there isn't any. The
reason I used an enumerator is that the data takes too long to generate if
it's all done at the same time, and saving it as an array doesn't work
because the file is too large to work; loading it into memory isn't feasible,
and using a FileStream would be too slow for the access speeds I need (I'm
trying to maximize performance on the app due to the amount of data it
processes, and the speed I've gained by using the ThreadPool extensively
would be lost if the data was read directly from the hard drive).

3) My understanding of the Enumerator is that it must hold state - how else
could the yield keyword work? I used reflection against it, which revealed
the following properties: System.String __current, Int32 __state. This makes
sense, given that the Enumerator yields Strings, but how can it hold the
state of the method? If it simply generated the data before evaluating it, I
would have noticed the spike in memory usage. I know that a similar design is
used in LINQ, so someone must know how they work.

Thanks for your help.

 

[Jon Skeet [C# MVP]]

3) My understanding of the Enumerator is that it must hold state - how else
could the yield keyword work? I used reflection against it, which revealed
the following properties: System.String __current, Int32 __state. This makes
sense, given that the Enumerator yields Strings, but how can it hold the
state of the method? If it simply generated the data before evaluating it, I
would have noticed the spike in memory usage. I know that a similar design is
used in LINQ, so someone must know how they work.

There will be other fields in the type, representing the local
variables in your original method.