DEFRAGMENTATION OF STORAGE CARD

I have had problems with the speed of my Pocket PC application on some
Pocket PC's. It does a lot of writing to files on the storage card. I
suspected that my speed problem could have something to do with
fragmentation of the storage card. So I formatted it on one of the Pocket
PC's that had trouble. Then my application performed as expected again.

Does anyone know if there is a defragmentation API available that I can buy
and use in my application.

Thore Berntsen
VBD, Norway

Flash media physically wears out, so storage cards have internal controllers
that handle fragmentation and wear levelling. In fact it's unlikely that
the controller can be bypassed to give you raw access to the physical media.
Flash is also inherently a slow media, so if you're expecting speed, you
probably need to rethink how you use it. You'll never get performance from
flash media.

-Chris


"Thore Berntsen" <someone@microsoft,com> wrote in message
news:uBmfQt3bFHA.3932@TK2MSFTNGP12.phx.gbl...
>I have had problems with the speed of my Pocket PC application on some
>Pocket PC's. It does a lot of writing to files on the storage card. I
>suspected that my speed problem could have something to do with
>fragmentation of the storage card. So I formatted it on one of the Pocket
>PC's that had trouble. Then my application performed as expected again.
>
> Does anyone know if there is a defragmentation API available that I can
> buy and use in my application.
>
> Thore Berntsen
> VBD, Norway
>

"You'll never get performance from flash media."

That depends on what your requirements are. The OP stated the perf is
acceptable with a fresh format of the card so there ARE things that can be
done in this case. Besides fragmentation is a factor of the File system and
not the card. That being said fragmentation on a solid state media is mostly
irrelevant as there are no seek time delays. (Well, yes there, are but they
are constant time for all locations on the disk) Theoretically, you can
measure a difference if all the sectors of a file read can be read as a
contiguous chunk of physical sectors, however due to wear leveling that will
virtually never happen after the card is first used.

There is a significant and noticeable performance curve point that happens
once the solid state drive becomes 50% FULL. At this point the wear leveling
algorithm has to work harder to find a place to move sectors around to. The
more full it gets, the longer it takes.


--
Steve Maillet
EmbeddedFusion
www.EmbeddedFusion.com
smaillet at EmbeddedFusion dot com

I've done a fair amount of profiling of several controllers, software and
hardware, and have found that anything above about 25% full causes
significant performance degradation. This occurs whether you sequentially
add block-swized files to the media (meaning the controller by all righs
should have to do no load leveling, compaction or movement) or if you fill
the media with small files then delete random holes and refill. I don't
think the OP can do much about what he's seeing to guarantee any type of
performance. A larger CF card will behave better for longer, but that's
only because it's larger. Moving to a microdrive would probably be a
solution for speed, but cost and reliability suffer.

Again, we've got no idea wht he usage is, nor his accepatnce criteria, but
my opinion is that if he sees unacceptable behavior with Flash media, then
he's never going to surmount it staying with Flash media. Maybe a RAM
solution with a flash backup would work. Maybe a microdrive. At any rate,
there's no way to defrag the card.

-Chris


"Steve Maillet (eMVP)" <nospam1@EntelechyConsulting.com> wrote in message
news:uv%23$AG4bFHA.3464@tk2msftngp13.phx.gbl...
> "You'll never get performance from flash media."
>
> That depends on what your requirements are. The OP stated the perf is
> acceptable with a fresh format of the card so there ARE things that can be
> done in this case. Besides fragmentation is a factor of the File system
> and not the card. That being said fragmentation on a solid state media is
> mostly irrelevant as there are no seek time delays. (Well, yes there, are
> but they are constant time for all locations on the disk) Theoretically,
> you can measure a difference if all the sectors of a file read can be read
> as a contiguous chunk of physical sectors, however due to wear leveling
> that will virtually never happen after the card is first used.
>
> There is a significant and noticeable performance curve point that happens
> once the solid state drive becomes 50% FULL. At this point the wear
> leveling algorithm has to work harder to find a place to move sectors
> around to. The more full it gets, the longer it takes.
>
>
> --
> Steve Maillet
> EmbeddedFusion
> www.EmbeddedFusion.com
> smaillet at EmbeddedFusion dot com
>
>

I'm a little confused here. What I did was, copy the files on the storage
card to my laptop, format the card from the laptop, and then I copied the
files back to the storage card and insert it into the Pocket PC again. My
idea was that this would take care of the defragmention I suspected on the
card. My application was much faster after I did this. My conclusion was
that I got rid of fragmentation, is this conclusion wrong? I must mention
that I haved used this Pocket PC very much for automated testing the last
month.

I just downloaded a tool from this link
http://www.softwinter.com/storagetools.html I haven't tested it yet, but
it's a defragmention tool.

"<ctacke/>" <ctacke_AT_OpenNETCF_com> wrote in message
news:ex1lni4bFHA.2756@tk2msftngp13.phx.gbl...
> I've done a fair amount of profiling of several controllers, software and
> hardware, and have found that anything above about 25% full causes
> significant performance degradation. This occurs whether you sequentially
> add block-swized files to the media (meaning the controller by all righs
> should have to do no load leveling, compaction or movement) or if you fill
> the media with small files then delete random holes and refill. I don't
> think the OP can do much about what he's seeing to guarantee any type of
> performance. A larger CF card will behave better for longer, but that's
> only because it's larger. Moving to a microdrive would probably be a
> solution for speed, but cost and reliability suffer.
>
> Again, we've got no idea wht he usage is, nor his accepatnce criteria, but
> my opinion is that if he sees unacceptable behavior with Flash media, then
> he's never going to surmount it staying with Flash media. Maybe a RAM
> solution with a flash backup would work. Maybe a microdrive. At any
> rate,
> there's no way to defrag the card.
>
> -Chris
>
>
> "Steve Maillet (eMVP)" <nospam1@EntelechyConsulting.com> wrote in message
> news:uv%23$AG4bFHA.3464@tk2msftngp13.phx.gbl...
>> "You'll never get performance from flash media."
>>
>> That depends on what your requirements are. The OP stated the perf is
>> acceptable with a fresh format of the card so there ARE things that can
>> be
>> done in this case. Besides fragmentation is a factor of the File system
>> and not the card. That being said fragmentation on a solid state media is
>> mostly irrelevant as there are no seek time delays. (Well, yes there, are
>> but they are constant time for all locations on the disk) Theoretically,
>> you can measure a difference if all the sectors of a file read can be
>> read
>> as a contiguous chunk of physical sectors, however due to wear leveling
>> that will virtually never happen after the card is first used.
>>
>> There is a significant and noticeable performance curve point that
>> happens
>> once the solid state drive becomes 50% FULL. At this point the wear
>> leveling algorithm has to work harder to find a place to move sectors
>> around to. The more full it gets, the longer it takes.
>>
>>
>> --
>> Steve Maillet
>> EmbeddedFusion
>> www.EmbeddedFusion.com
>> smaillet at EmbeddedFusion dot com
>>
>>
>
>
>

"At any rate, there's no way to defrag the card."
No that's not true. fragmentation has NOTHING to do with the card. It's a
file system level thing and you can defrag a flash card if you really want
to. It has a HUGE impact on a Microdrive over time. However. fragmentation
has virtually no impact on solid state media.


--
Steve Maillet
EmbeddedFusion
www.EmbeddedFusion.com
smaillet at EmbeddedFusion dot com

Microsoft provides defragmentation of FAT formatted systems in the OS.
Although I'm not sure if that ships as part of the Pocket PC OS Builds...

--
Steve Maillet
EmbeddedFusion
www.EmbeddedFusion.com
smaillet at EmbeddedFusion dot com

> Does anyone know if there is a defragmentation API available that I can
> buy and use in my application.

Maybe you want to take a look at the FatUtil library functions
(http://msdn.microsoft.com/library/e...ryfunctions.asp).

Greetings, Christian

I must mention that the Pocket PC's I have this problem on uses Compact
Flash Disks not SD cards. Can that make a difference?

Thore Berntsen
"Steve Maillet (eMVP)" <nospam1@EntelechyConsulting.com> wrote in message
news:O1Sjsq5bFHA.3048@TK2MSFTNGP12.phx.gbl...
> "At any rate, there's no way to defrag the card."
> No that's not true. fragmentation has NOTHING to do with the card. It's a
> file system level thing and you can defrag a flash card if you really want
> to. It has a HUGE impact on a Microdrive over time. However. fragmentation
> has virtually no impact on solid state media.
>
>
> --
> Steve Maillet
> EmbeddedFusion
> www.EmbeddedFusion.com
> smaillet at EmbeddedFusion dot com
>
>

Ok, so maybe we have term mismatch here. I think of "defragmentation" as
taking pieces of files and putting them in physically sequential locations
on the media to allow the file system to access them in less number of
reads. On a device with heads this also reduces the need to reposition the
heads which is where the major performance gains comes from.

Now flash media typically has a controller, either in software or hardware,
that controls the physical location of data so it can provide wear leveling.
This is an attempt to equally use all parts of the media to prolong life.

So in order to "defrag" a flash media system, you would have to bypass the
wear-leveling piece in order to physically arrange your data. That's
typically not an option.

But as you said, with solid state media, you're not going to get much gain
even if you could because the slowness from fragemented media is probably
95% in the need to do multiple moves of the physical drive head and this
only occurs where there are moving parts (like a microdrive, which has no
load levelling and can be defragmented).

If I'm wrong I'm fine with that and I'd love to understand how flash media
could be defragmented.

And of course this all applied only to writes for flash media. Reads are
fast, so if the OP is seeing read problems it may be something altogether
different.

--
Chris Tacke
Co-founder
OpenNETCF.org
Are you using the SDF? Let's do a case study.
Email us at d c s @ o p e n n e t c f . c o m
http://www.opennetcf.org/donate


"Steve Maillet (eMVP)" <nospam1@EntelechyConsulting.com> wrote in message
news:O1Sjsq5bFHA.3048@TK2MSFTNGP12.phx.gbl...
> "At any rate, there's no way to defrag the card."
> No that's not true. fragmentation has NOTHING to do with the card. It's a
> file system level thing and you can defrag a flash card if you really want
> to. It has a HUGE impact on a Microdrive over time. However. fragmentation
> has virtually no impact on solid state media.
>
>
> --
> Steve Maillet
> EmbeddedFusion
> www.EmbeddedFusion.com
> smaillet at EmbeddedFusion dot com
>
>