Well, just my luck, as soon as I posted, I found the answer - and msdn decided to redo their website so I couldn't reply until now. Apologies if anyone was working on a detailed answer.
For those interested, the following article describes the steps that the formatter follows during de-/serialization - starting on page 5:
http://msdn.microsoft.com/msdnmag/issues/02/04/net
Basically, the FormatterServices class provides the methods needed to access the members to be serialized and the means to deserialize them. I implemented these steps and now my color deserializes - and the serialization code is also shorter.
Markus