How to: FASTER file access in huge project on DL DVD-ROM? [Advanced topic]

[MCSmarties]

Hello,

I have created a huge project to be run from a DL DVD-ROM.
I want to access everything directly from the drive, no installation.

Opening files and navigation is EXTREMELY slow even though
the vast majority of files are simple.


Question: Is there a way I can GREATLY improve the access time?
**********************************************************************

Please read on for details...

That project contains an involved menu system, complete offline
images of websites, videos, Powerpoint presentations, sound...

In other words, there are LOTS of files butmost of them are small
(average size probably 1 MB or less but ranges from "nothing"
to 700MB)

Running the project from a (mounted) virtual DVD is blazing fast.

Running it from the burned DL DVD-ROM is AGONIZINGLY slow.
I'm talking 1-2 minutes (not seconds) to start the project, access
any kind of file in it regardless of size, etc...

CPU and memory usage is minimal so my guess is the slow speed is
due to the drive scanning the whole 8 GB archive each time it tries
to open a new file. Note that all the necessary software to run files
is also included on the DVD-ROM (a media player, ppt viewer, etc...)

I simply burned the compilation with Nero 7 and I think that it made
a mess of my file organization (as in, clusters used)

Is there a software that can optimize the structure of a DVD-ROM
image before it is burned to the actual support?
Something like a "smart defragmenter" or indexing service?

I'm thinking of something similar to PerfectDisk (which defragments
a HDD by putting the most accessed files at the beginning of the disk,
then optimizes the rest) but specifically designed for burned media.

The file priority should be something like this:

- AUTORUN.INF which opens main menu
(I'm currently waiting 3-5 minutes for it to autostart!)
- menu system at 3 levels maximum "depth"
(made with QMB, www.quickmenubuilder.com)
- PPT presentations (using portable PPTviewer.exe)
- offline images of websites (index.html should open quickly,
ideally the following links are then pre-cached?)
- audio and video files

In other words, I'd like to force the drive to read all high-priority
files from one "cluster" at the beginning of the DVD-ROM,
then "digs deeper" only AS NEEDED. The drive currently scans
the DVD-ROM like crazy every time I make a small change, such
as navigating the main menu (takes ~ 30s to do anything at all)

Obviously one solution is to make the target system mount an
image of the project, but that kind of defeats the purpose of the
whole thing and would anyway require at least 8GB of free space.

It's difficult to find answers on google, I can't seem to find good
keywords to use (things in the vein of "file access", "speed
optimization", "slow DVD-ROM", etc... didn't yield anything useful)

Thanks a lot for any help you can give!

 

[Michael W. Ryder]

Hello,

I have created a huge project to be run from a DL DVD-ROM.
I want to access everything directly from the drive, no installation.

Opening files and navigation is EXTREMELY slow even though
the vast majority of files are simple.


Question: Is there a way I can GREATLY improve the access time?
**********************************************************************

Please read on for details...

That project contains an involved menu system, complete offline
images of websites, videos, Powerpoint presentations, sound...

In other words, there are LOTS of files butmost of them are small
(average size probably 1 MB or less but ranges from "nothing"
to 700MB)

Running the project from a (mounted) virtual DVD is blazing fast.

Running it from the burned DL DVD-ROM is AGONIZINGLY slow.
I'm talking 1-2 minutes (not seconds) to start the project, access
any kind of file in it regardless of size, etc...

CPU and memory usage is minimal so my guess is the slow speed is
due to the drive scanning the whole 8 GB archive each time it tries
to open a new file. Note that all the necessary software to run files
is also included on the DVD-ROM (a media player, ppt viewer, etc...)

I simply burned the compilation with Nero 7 and I think that it made
a mess of my file organization (as in, clusters used)

Is there a software that can optimize the structure of a DVD-ROM
image before it is burned to the actual support?
Something like a "smart defragmenter" or indexing service?

I'm thinking of something similar to PerfectDisk (which defragments
a HDD by putting the most accessed files at the beginning of the disk,
then optimizes the rest) but specifically designed for burned media.

The file priority should be something like this:

- AUTORUN.INF which opens main menu
(I'm currently waiting 3-5 minutes for it to autostart!)
- menu system at 3 levels maximum "depth"
(made with QMB, www.quickmenubuilder.com)
- PPT presentations (using portable PPTviewer.exe)
- offline images of websites (index.html should open quickly,
ideally the following links are then pre-cached?)
- audio and video files

In other words, I'd like to force the drive to read all high-priority
files from one "cluster" at the beginning of the DVD-ROM,
then "digs deeper" only AS NEEDED. The drive currently scans
the DVD-ROM like crazy every time I make a small change, such
as navigating the main menu (takes ~ 30s to do anything at all)

Obviously one solution is to make the target system mount an
image of the project, but that kind of defeats the purpose of the
whole thing and would anyway require at least 8GB of free space.

It's difficult to find answers on google, I can't seem to find good
keywords to use (things in the vein of "file access", "speed
optimization", "slow DVD-ROM", etc... didn't yield anything useful)

Thanks a lot for any help you can give!

Two things I can think of. First do you have virus/spyware checking
software turned on? Second, do the programs you are using have a lot of
pieces that need to be loaded when you do anything?
I have written my own programs in the past to avoid using graphical
interfaces because of the time they use doing anything. I can back up a
directory using XCopy many times faster than using drag-and-drop.
Similarly I have a program that will find and display one file among
over a million in a second versus the minutes that using Explorer takes.
The main thing in a project like this is to determine what is causing
the extreme delay and find an alternative.

 

[MCSmarties]

Two things I can think of. First do you have virus/spyware checking
software turned on?

Yes I do but that never slowed down anything else. And I'd rather not
have to disable it...

Second, do the programs you are using have a lot of
pieces that need to be loaded when you do anything?

Some of them do but most of them are really just tiny freeware
applications
or even batch files. And for the webpages it's just the current page -
maybe
with a few images - that needs to load.

I just really think that the problem is not loading the individual
files, it's mainly
a matter of finding them. I think that burning my project just spread
out the
files all over the place instead of sequentially adding them to
adjacent clusters.
I know it's not a good analogy but it really feels like my disk is just

hopelessly "fragmented"...

The main thing in a project like this is to determine what is causing
the extreme delay and find an alternative.

Again, I think it's the seeking for files, not loading them.
Actually when I say a "project" I don't mean a large software program
or anything, more like a huge collection of "stuff" I gathered up and
put together using a nice menu system.

Thanks for your reply anyway. Anybody else have an idea?
I'd love to hear more opinions... thanks.

 

[Michael W. Ryder]

Yes I do but that never slowed down anything else. And I'd rather not
have to disable it...

You might want to try disabling them while the computer is not connected
to a network to see if the scanning is causing a problem. I see it a
lot when opening files.

Some of them do but most of them are really just tiny freeware
applications
or even batch files. And for the webpages it's just the current page -
maybe
with a few images - that needs to load.

I just really think that the problem is not loading the individual
files, it's mainly
a matter of finding them. I think that burning my project just spread
out the
files all over the place instead of sequentially adding them to
adjacent clusters.
I know it's not a good analogy but it really feels like my disk is just

hopelessly "fragmented"...

I personally don't think this is the problem. Like I mentioned I have a
disk with over a million files on it that have been added over the
course of a couple of years and have never defragmented the drive.

Again, I think it's the seeking for files, not loading them.
Actually when I say a "project" I don't mean a large software program
or anything, more like a huge collection of "stuff" I gathered up and
put together using a nice menu system.

Thanks for your reply anyway. Anybody else have an idea?
I'd love to hear more opinions... thanks.

Are the programs looking for the files/programs using a fixed
destination or do they have to search a path looking for the desired files?

 

[MCSmarties]

You might want to try disabling them [the antivirus] (...)
to see if the scanning is causing a problem. I see it a lot when opening files.

Sorry for the delay, I was away from my computer this weekend. I have
disabled
the autoprotect feature of my antivirus scanner and ithas indeed
greatly increased
the access time. Before, it was impossible to use, now it's mainly very
slow...

Are the programs looking for the files/programs using a fixed
destination or do they have to search a path looking for the desired files?

I'm not sure I understand what you mean. The menu system essentially
features
a bunch of buttons that will launch a defined file with defined
parameters.
All paths are relative - here's a few examples:
(note that %menudir% is defined as the location of the main menu file)

%Menudir%\bsplayer.exe "..\Movies\Movie1.avi" "..\Subtitles\Movie
1.srt" -fs -eend
(launch Movie 1 with its subtitles using the software "BSPlayer" -
included - run it
full screen and exit at the end)

Note that the codecs are supplied on the CD, the player's
initialization file
locates them as needed (for XviD, MP3) in the \filters subdirectory.

%Menudir%\..\Goodies\Pictures\Gallery 1.exe
(launches a picture slideshow, complete with sound)

%Menudir%\..\Goodies\Quotes.bat quotes.txt
This one is trickier. It launches a batch file Quotes.bat which
searches for a random
quote in the file quotes.txt (which contains a dir /b /s listing of
..wav files)

The MAIN bottleneck is however the time it takes for the autorun to
start
(it still takes several minutes). The autorun.inf simply contains 2
lines, one pointing
to the menu file to execute and one to a customized icon.

 

[Michael W. Ryder]

You might want to try disabling them [the antivirus] (...)
to see if the scanning is causing a problem. I see it a lot when opening files.

Sorry for the delay, I was away from my computer this weekend. I have
disabled
the autoprotect feature of my antivirus scanner and ithas indeed
greatly increased
the access time. Before, it was impossible to use, now it's mainly very
slow...

Are the programs looking for the files/programs using a fixed
destination or do they have to search a path looking for the desired files?

I'm not sure I understand what you mean. The menu system essentially
features
a bunch of buttons that will launch a defined file with defined
parameters.
All paths are relative - here's a few examples:
(note that %menudir% is defined as the location of the main menu file)

%Menudir%\bsplayer.exe "..\Movies\Movie1.avi" "..\Subtitles\Movie
1.srt" -fs -eend
(launch Movie 1 with its subtitles using the software "BSPlayer" -
included - run it
full screen and exit at the end)

Note that the codecs are supplied on the CD, the player's
initialization file
locates them as needed (for XviD, MP3) in the \filters subdirectory.

%Menudir%\..\Goodies\Pictures\Gallery 1.exe
(launches a picture slideshow, complete with sound)

%Menudir%\..\Goodies\Quotes.bat quotes.txt
This one is trickier. It launches a batch file Quotes.bat which
searches for a random
quote in the file quotes.txt (which contains a dir /b /s listing of
.wav files)

The MAIN bottleneck is however the time it takes for the autorun to
start
(it still takes several minutes). The autorun.inf simply contains 2
lines, one pointing
to the menu file to execute and one to a customized icon.

The reason I asked about the paths is that if there are a lot of
directories with a lot of files earlier in the path it could take awhile
to search through the entire disk to find the desired files. When I
access the million plus files on one computer I feed it the exact path
to the file(s), such as G:\media\companya\clientb\fileccc.tif, and it is
very fast.
Another thing to consider is trying to make it so that the most used
players stay in memory instead of having to load and initialize each
time. I think this is where most of your sluggishness is coming from.
I know that a lot of programs have this ability, such as office
programs, but am not sure how to do this with programs not designed for
this.