I like these trans-OS discussions because they let one explore
possible ways of doing things without being chained down by the way
particular OSs actually do things
Yea it was from one HD to another. Though even if I move such data on the
same HD, I have by far less head thrashing than I do under windows.
Virtually none actually...
There's have to be head clicks after every buffer full of source
material has filled up, so this implies Linux is setting aside a
larger buffer (limited by a bit less than physical RAM) than Windows
does. The downside is less RAM left for anything running at the same
time, and maybe there's more underfootware in Windows.
Also, there will be a minimum unpageable footprint for the OS.
The only way to improve that best-case scenario would be to trade off
processor usage by compressing source material and expanding it when
writing to destrination. That's a bad idea where the source material
is already lossily compressed, as it will be with most things large
enough to matter (video, picture collections, archives, MP3s).
The superblock basically, first 1kb of the partition that defines how the
file system is structured.
Sounds as if it may be shallow as NTFS, then. "Hey, our file system
survived; your data files are your problem"; that mindset increasingly
pervades MS's approach to file system maintenance.
I don't think I have converted a file system from FAT32 to NTFS since 1995 =)
Your OEM may have, if you're still using their setup
The usual conversion killers are:
- no way to convert back from NTFS to FATxx
- if process is interrupted or fails, expect data loss
- appropriate permissions etc. are not assigned to files etc.
- 512-byte clusters because volume wasn't boundary-aligned
- loss of product activation "life" as volume serial number changed
It separates them into different category. MFT gets its own color.
OK. That's better than XP's defragger, which uses the same green for
all material that cannot be moved, forcing you to guess.
Directory files get their own color. Pagefile get its own color. Other
non-movable stuff gets its own color.
Cool! Hey, MS... did you know SVGA now supports > 16 colors?
It does a pretty nice job, I was happy with it under windows. I am still
glad I don't really need it anymore though. =)
I'd like it as a non-integrated tool, and without the underfootware
component (e.g. it could use XP's native use-tracking)
No it doesn't. A defragger has no business really modifying the file
system contents itself such as shrinking the MFT in my opinion. It should
defrag and that is it. Shrinking the MFT should be done via a separate
tool in my opinion.
OK, or perhaps a separate "Tools" thing from the defragger's UI, much
as HiJackThis hangs ADSSpy as an "extra tool".
I do, for example, expect a defragger to purge erased directory
entries, and where directories are indexed as they are in NTFS, that
would involve rebuilding those structures (at least, as I understand
it). Actually, I'd expect MFT to be cleaned up as well, especially if
the defragger boasts that it "defrags the MFT".
Defrag IS a high-risk operation that exists to make a healthy file
system more fit, and that point is not emphasised enough - i.e. it is
NOT a troubleshooting tool, any more than marathon gym training is a
diagnostic test for heart disease.
Even if a defragger "only" defrags files, the risk of disaster (crash,
corruption of what is read from disk and written back due to bad RAM,
etc.) is very much present. Doing that while avoiding the purging of
redundant structure is like jumping out of a plane, but wearing safety
glasses to protexct your eyes on landing.
Then again, this fits the "file system's OK, who cares about your
data" vendor-vision thing I mentioned earlier.
Not even sure if there is such a tool for NTFS.
Yup. MS haven't bothered to do much to maintain or recover NTFS, and
because they keep it a proprietary, moving target, no-one else can do
it either. Doing so is hi-risk the file system could change in a
blink of an SP or patch, before you've recouped your investment.
I'm not sure how well Ext2/3 does if the file system is trashed and needs
to be repaired. I do know this though, in Ext3 file undeletion is
impossible as it zeroes out the block pointers in the inode for
reliability reasons in the event of a crash.
Hm. It's one thing to consider (as MS seems to do) the user as alone,
another to consider the user's system as including techs who can fix
things. For example, you might consider disk compression to be
low-risk (as MS did) according to the first model (where if it blinks,
the user's dead anyway) but not the second.
MS talks to us techs about "partnership", but it's largely hot air
when it comes to delivering better value to end users. It only goes
as far as "partnering" to sell more stuff.
This changes in developer and professional network administration,
where genuine technical partnership is common and effective. It's
just the consumers who get left out in the cold, because the nadir of
large "royalty OEMs" is taken as the acceptable baseline.
It's pathetic.
It has major performance advantages though. Something that doesn't need to
be created doesn't take time to create it.
This is true, I guess... same thinking as defragging, really; "duty
now for the future". It's just that once yo create all those empty
file structure items, you're obliged to maintain them, if you intend
to use them without sanity-checking them first.
If you do sanity-check them first, then that's prolly the same sort of
overhead as creating them (tho noting reads are "cheaper" than writes,
which applies to HDs and flash for different reasons)
So why spend time constantly resizing the MFT, and subsequently introduce
a potential failure point in the file system, when there is no need to?
I wouldn't resize MFT on the fly, as Vista tries to do with defrag;
jars with the concept of "the OS should initiate no risky activities
unless these were initiated by the user". In that sense, I'd defrag
and purge-out MFT when defragging, and I'd defrag only when this is
explicitly initiated by the user.
Being old-school, I'd prolly do it outside of multitasking "business
as usual", either on a lock-down, defrag, shutdown basis or on a
restart, defrag, initiate multitasking basis. The latter's more
common (e.g. the way the OS does ChkDsk /F or /R for C
but in some
ways I prefer the former. Why not support both?
Heuristics
No, I was thinking of.. gah, let's find it...
http://en.wikipedia.org/wiki/Hysteresis
For example, let's say you define files < 50k to be put in one area of
disk, and > 50k to be put in another area of disk.
In that case, you'd expect thrashing if files moved across the 50k
line in either direction.
So you might deliberately make it "laggy", i.e. if < 10k it goes in
one place, and if > 100k, it goes in another place, and if it's
anywhere between, you just leave it as it is.
And/or, you can apply logic, such as "files that grow larger than the
limit are unlikely to shrink and stay below the limit". In which
case, files that grow over > 100k get moved, but once moved, they're
tagged as being not for relocation even if they do shrink below 10k
This is basically an expert system, i.e. you are trying to teach the
system to think. But *we* already think, so why not take advantage of
that? A user who knows a certain bunch of files will never change in
size and will seldom be accessed, san simply whisk these out of C: and
put them on a distant logical volume, out of the way.
It's the same sort of efficiency as pre-compiling 3D scenes on a
powerful workstation so that consumer PCs can "render" them without
having to rebuild them from first principles.
Hey, get two disks for the price of one
Yep - allowed a mix of single-sided and double-sided drives to be used
in similar ways, but it lost the efficiency of dual heads. We was po,
in those days; each CPU clock tick was palpable...
Which is why I absolutely love not having drive letters under linux. =)
I could go as far as creating a set of directories, each mounted to a
different partition, to categorize my data and impose limits of how much
of said data I want to be able to store. That way I can always guarantee
there will be X amount of space in a certain directory if I have a need
for that.
Try that under windows...you'd end up with a sea of meaningless
drive letters.
The NTFS approach would prolly be to use different volumes and then
hide this behind junctions that route automatically, as if the "drive
letter" was mounted at a point in the directory tree.
There's also a quota system of sorts, but I don't think it's flexible
in terms of what can be quota'd.
There's a fundamental "target audience" thing here. Windows could be
like Linux if it was developed purely for developers and technerati,
because we'd love to roll up our sleeves and create custom
CLSID-mediated namespace items with our choice of particular
behaviors. Then the best of these could be rolled back into consumer
space as the "canned" items we see today. Hmm.
In a way, this is how my use of partitions evolved towards the
4-volume model I currently use. Recurring themes kept coming up:
- stuff that needed fast access (C
- stuff that was important, but rarely used (F
- stuff that was small, crucial, and often used (D
- other stuff, and masses of it (E
In the above, "crucial" means "don't stick it on C: if you really want
to see it again", as C: will always be a charnel house of temp writes,
paging, and other crash-o-matics banging away. Get outta there.
Actually no, not really it doesn't. The responsibility that my multi
threaded code works rests all on me. The compiler gives me absolutely no
aid in that regard. All existing programming languages essentially lack
the ability to properly define a multi-threaded application.
Still? Wow, that's a bit retro... then again, sware always lags
behind opportunities offered by hardware. I remember reading about
hypervisons pre-emptively time-slicing apps in their protected spaces
in the closing chapers of a book on the 386... was a long time coming,
at least in Microsoft-land (I think UNIX took the lead there).
The compiler isn't even aware that my app is multi threaded. To it,
each thread is just another function.
Does it not treat each function as potentially running in parallel
with everything else? It should do... old-fashioned phrases like
"re-entrant" come to mind at this point.
The only thing that makes it multi threaded are the calls I make
to the OS to tell it which functions of my code I'd like to be
spawned on its own thread.
OK.
Moving from explicit multi-processor to implicit multi-processing
(multiple cores, out-of-order execution etc.) changes things to the
point that modern development tools should expect random pulti-core
parallelism as a given.
For example, the old NT licensing approach was; 1 core for "cheap"
licenses, multi-core for "expensive" licenses. To move from one to
the other (and to harness all cores), you'd have to do the NT
equivalent of recompiling the kernel; change the HAL.
XP Home is 1-processor, Pro is multi, from a licensing perspective.
But AFAIK, all multi-core processors are to be seen as single CPUs,
and between the PIII and P4 era, the trend switched away from
multi-processor motherboards to multi-core processors.
Microsoft's .Net Framework does give *some* aid in multi-threading during
runtime, such as modifying properties of some windows controls will throw
an exception if it's not thread-safe. But these are all runtime checks.
The compiler still merrily compiles it all without giving an error.
Hmm. By "compiler", I really mean the whole dev environment, from
static and runtime libraries to the way base classes are defined. The
actual compiler is more where processor compatibility battles are
fought, i.e. what branch weighting and cache size to optimize for, do
we use extra opcodes, are we OK for out-of-order execution, etc.
It's up to the operating system to load balance the CPU. The CPU itself
can do very little in that regard. If it gets a piece of code to execute,
it can't say I am going to execute half on one core, half on the other as
the second half may depend on the results of the first half so it cannot
be executed in parallel.
That's what I mean - this sort of on-the-fly "leave it to us"
how-it's-done logic has been moving down the line - from assemby
programmer to compiler to CPU microcode - for a while now, starting
with the original Pentium's super-scalar pipelines.
Deciding which instruction pairs to split across two pipelines
(without setbacks from cross-dependencies) is the start of the same
sort of logic that may split code between multiple cores.
In some ways, it may even be easier with true cores, as at least they
are equally capable, so you don't have to think "special case"
limitations. I grew up with a CPU that was not like this at all;
sure, there were A, Flags, B, C, H, L, IX-h, IX-l, IY-h, IY-l, SP, I
and R registers, but they were all different in terms of what they
could do. Some weren't even a full 8 bits ;-)
In the case of the Z80, you were expected to use certain registers in
certain ways, i.e. IY was supposed to be used as an address index base
register (and had opcodes optimized for this, being "expensive" to use
otherwise) etc. but if you need a bunch of separate 8-bit registers
now, you could use undocumented instructions to use the halves of
these registers as such. Useful when you want to preserve all of RAM.
Modern CPUs, as I understand it, aim to be able to do anything with
any register with equal economy - in fact, you aren't supposed tio be
concerned with what actual register is used for what. First, the
compiler took that load of your shoulders if you switched from
Assembler to C, then the processor took it over.
There are things each core does on its own to attempt to improve
performance, but load-balancing across the cores is an operating system
task. And the best the OS can do is spread active threads across the cores.
OK. Wasn't sure what point in the evolution we'd reached.
So, what's likely to be better for a budget ("I just want to do email,
write letters, look at photos") PC; a single-core 2GHz CPU, or a
dual-core 1.6GHz CPU, with Vista-32 Basic as the target OS?
That ain't a rhetorical question; next week I build such things.
Is any of that seriously an issue still though with today's processing
power on just a single core? I don't think a single letter felt "sticky"
writing this post.
It used to be a differentiator around the Win95 days, when MS Office
was said to leverage undocumented OS calls for unfair advantage.
Other word processors (Word Pervert, Lotus SmartSuite, etc.) often
felt "laggy" when typing.
If you use Word right now, and open a non-trivial document, you will
see the status bar's page and line count will be updating itself as
you work in the document, for quite a few minutes sometimes. I take
that as a sign that the pagenation is handed off to a background
thread. For another example, you can work in Eudora while it is
simultaneously pulling and sending mail from differnt email accounts
(which can give integrated av the blue heebies, heh). More threads?
Unlike some Vista betas, when you pull up a new folder and Vista's
Explorer does the silly "looking at stuff" thing, with the green
progress slug crawling behind the bread-crumbs, you can at least work
within the window. Same as when XP pulls up content from a
freshly-detected stick; you can click "Explore" from the suggestion
box without waiting for the grope to complete, at which point the
grope is cancelled. Different threads? I'd hope so.
Still doesn't really need multiple cores per se though to do it unless
maybe it has a fresh 1,000 page book to chew through I suppose.
I think if I were starting a project that I expected to take 3 years
to complete, I'd assume multi-core availability and go multi-thread,
especially if I had integrated stuff leeching off my code. I'd try
and toss those leeches onto another core, and make no assumptions
about what they haven't done to whatever they hook into.
Times like this, I'm glad I'm off that "programmer" treadmill ;-)
I used to have a Cyrix once!!
The thing that put me off the whole "trust me, it's faster even though
it's slower" is that it was always such a disclaimer'd YMMV thing.
The K5 was a low point, though. Not only was it slow as the Pentium
core it was supposed to outperform, but the complexity of the core
meant it couldn't be clocked up.
Worse, it was the only CPU where I've experienced different runtime
behaviour (e.g. "fonts look funny", some graphics appear at the wrong
size or not at all, etc.) purely because a different CPU was in the
PC. This is aside from expected "crashes because timing assumptions
were invalidated" thing that happens far more often.
In any case, AMD's own DX4-133 ate it alive, running from a cheaper
486DXn motherboard. So AMD cancelled the DX4-150 (it would have been
embarrasing; it's "ghost" remains in motherboard manuals and jumper
settings, tho), and tossed out the K5 team in favor of the NexGen team
they bought who would in turn develop the K6.
Perhaps Intel's undergoing a similar team swap, what with the mobile
team taking over desktop projects from the NetBurst folks?
Windows' internal messaging stuff is a single-threaded message
pump. Nothing more than a simple FIFO. Even in a multi threaded
application, the message pump is still a single threaded FIFO.
Hmm, I see the problem. This is what would kill OS/2's claims to
"uncrashability"; geek appologists would say "it hasn't crashed, it's
still running, it's just that you can't control it because the
keyboard/mouse input queue's fallen over.".
Hmm, why were we so NOT reassured ;-)
This becomes very evident when writing .Net based applications as many
controls will whine and moan if they are modified from a thread other than
the thread they were created on. If windows would distribute events across
multiple threads those controls would never shut up! =)
I guess the take-home here is "we aren't ready". So would you go so
far as to consider extra cores a waste of time, for general use?
When do you expect to ship?
My multi threading problem actually is not really a race condition.
OK. Those those issues find you, as a rule :-/
Currently my engine is setup as follows:
- One buffer for vertex data
- One buffer for color data
These two buffers are passed to OpenGL as vertex and color arrays
respectively as what I primarily display are just colored polygons. No
textures needed. The buffers each have a fixed size.
The rendering loop then goes to each object and queries it for its vertex
and color data, and this data is then added to the respective buffers.
When the buffers are full, they are submitted to the video card for
rendering and the engine goes to chew on the next set of object while the
video card is now busy processing the data I sent to it.
So I do have some level of multi threading here in terms of offloading
processing to the GPU.
Yup. Do you fit with the option to render in synch with frame paint?
Multi threaded rendering though would require a much different approach.
Especially since OpenGL does not like multi threading much. All OpenGL
calls have to be made from the same thread the OpenGL context was created
on.
OK. I know OpenGL's being/been depreciated in Windows as DirectX
takes off, but that won't matter if this is for Linux, and/or already
targetting hi-end PCs optized for OpenGL (tho that may be a sinking
ship in the medium-long term, i.e. if shipping years from now).
That means I'd need multiple geometry data buffers (one set of buffers per
thread) and each thread would need the ability to submit its geometry data
to the main application thread for submission to the GPU. At the same time
though, the loop in the main application thread has to be in such a way
that it doesn't consume CPU resources. It has to be able to go to sleep
while it waits on the other threads and it also has to instantly wake up
when another thread needs it to do something.
It's all doable and it's all a big pain in the butt to implement!!
There's the matter of how to distribute the load across n threads (or
"whatevers", if you abstract the problem a bit further. PICK used to
use id-hashing to distribute records across a number of pre-allocated
"data buckets", and MS may do something similar when splitting web
cache items across random-named directories.
PICK uses prime number modulos as they clump hashed items less than
the multiples of 2 that MS uses for IE's cache branches. Then again,
in the NTFS age, MS may not be hashing across random-named directories
for speed, but to break predictable paths as a malware protection.
I'd much rather try to offload more processing to the already yawning and
very bored GPU instead. But it's hard to offload memcpy, where over 90% of
my load is, to the GPU!
Is memcpy related to strcpy, as in "in null terminators we trust"?
Watch that; if you find Core 2 Duo is running things slower than the
older CPU architecture and you're due to ship to an all-Duo
marketspace, you could be performance-under-competitive
Actually it is the exact opposite way around!
Physics and AI are on such a tight leash it isn't even funny. Especially
true for multi-player games. But even in single player games, they
generally run on a very exact and tight schedule. 30 Frames per second is
a popular number I think for AI and Physics.
OK... that's interesting, and as you say, unexpected...
Graphics and sound on the other hand can be, and usually are,
asynchronous.
Or rather, locked into some sort of real-time synch?
Rendering and sound decoding / output is all handled by
hardware anyway. All the software has to do is keep the feeding the
hardware with data before it runs out of stuff to do, especially in regard
to sound. That doesn't require any overly precise timing.
OK. These days, processing is fast enough to "think" between wave
crests of sound waves, so it makes sense... in ye olde days, we'd have
to prio up for screen refresh, nowdays you can prolly cram a lot of
"thinking" between such realtime demands.
AI and Physics on the other hand, if they don't occur on a fixed framerate
then all sorts of weird and bad things happen. Especially in multiplayer
games, they absolutely have to, under all conditions, run at a fixed rate
so that all results are identical on all players systems.
Hmm... "results are identical on all players systems" could be a
saleability limitation; surely it would be good to say "the game plays
smoothly on slow PCs, but gives a tougher game on faster PCs that
allow us to run deeper AI"?
Supreme commander, just as one example, runs a complete simulation of the
entire battlefield on *every* players computer and compares them to one
another. If only one player's data does not match all other player data in
any way it's considered a desync and the game is aborted.
Ah, that's the multiplayer thing. The slow links between these
systems would also shape the way things have to be done.
Cool! Though the tradeoff may be to limit compatibility to systems
with "enough" graphic memory, maybe? Or does AGP and successors
successfully blur such limits, the way paging hides RAM limits?
I don't think there is a single soundcard these days that can't decode MP3
in hardware. =) CPU has absolutely nothing to do there hehe.
Is it? Well, that's good news. What about integrated sound? It's
very rare that I see a stand-alone sound card these days, even in PCs
built for gaming. It's only really the sound studio stuff that needs
"special" sound for ASIO performance, independent multiple inputs and
outputs, and better S/N from getting the sound guts out of the case.
Only if you're running Vista!!
Or having to "support" such titles. Interesting thought; as Linux
suggests itself for embedded/hidden use within dedicated "black box"
devices, is it tempted to pitch as controller OS for set-top players?
-------------------- ----- ---- --- -- - - - -
Tip Of The Day:
To disable the 'Tip of the Day' feature...
