Deleting many files quickly

  • Thread starter Thread starter Guest
  • Start date Start date
Pegasus said:
Multi-posted - see replies elsewhere.

Please avoid multi-posting... news servers and readers can handle crossposts
more efficiently as reading or downloading the article once in one group
makes that article downloaded or read in all groups it's posted to.
 
Kevin said:
the del command is still running. I am really looking for a utility that
may
be able to do muiltipul threads at the same time to help speed up the
process.

Thanks for your help......

Ayush said:
Well what you are doing right now is, post a question> wait for answer >
try
that answer > if it doesn't help , wait again.

Don't you think deleting files with 10 meg/min will be faster than this.

--
Ayush [ Be ''?'' Happy ]


It would take approximately 35 days @ 10mb per minute to delete 500gig.

mi

 
Reply contextualised/intertwined for clarity/brevity

Baloo said:
antioch wrote:

Don't: Munging is considered harmful. It does nothing to combat spam and
only hinders legitimate replies. It's better to use more effective
anti-spam measures like using an email server that rejects spam based on
SpamAssassin (http://spamassassin.apache.org/) scores.

Dont Munge - I am afraid your suggestion will not go down well in the
newsgroups. It is of no hinder to replies whatsoever - I fail to see what
you are talking about. How can one use antispam in the newsgroups?
If you do report spam that comes into the groups, you are told that nothing
will be done about it.
I and no doubt other users in the groups will continue to advise new users
to munge. I trust you used your real email address in this post?
It is all too easy for emails to be harvested in here - do you not realise
this.
For personal emails I am lucky enough to have an ISP who provides an
excellent and free antispam service. I might get one or two slip through in
a week - and in six months or more, I have only found one genuine email in
my mailbox spam folder.
As and when ALL ISPs provide antispam services, then some progress might be
made.

Re above - I have read the article and to me it appears the writer is in
'cloud cuckoo land' for most of it. I report a lot of spam as I feel
everyone should do their bit. I know a few people who dont bother anymore.
But nine times out of ten, nothing is done by the IPs/ISP about reports of
spam. Some appear to encourage it.
Thats where the enforcement should start - there - not at our user level.
Can you imagine the costs to these companies for setting up an effective
antispam department? Thats why they have no interest.
Rgds
Antioch
 
antioch said:
Dont Munge - I am afraid your suggestion will not go down well in the
newsgroups. It is of no hinder to replies whatsoever - I fail to see what
you are talking about.

You can't just hit reply-by-email and expect it to work without having to go
through hoops. This is not acceptable: Good anti-spam measures do not
hinder legitimate email or shift the burden to legitimate senders in any
way, shape or form. False negatives are better than false positives,
though both should be minimized as much as possible.
How can one use antispam in the newsgroups?

You can't. Do it on the receive end, not the send end of the email process.
If you do report spam that comes into the groups, you are told that
nothing will be done about it.

Depends on how and who you report it to. It takes experience to learn this.
I and no doubt other users in the groups will continue to advise new users
to munge.

That demonstrates a fundamental misunderstanding of anti-spam efforts. If
your only goal in fighting spam is mitigating it from your inbox, you might
as well just not be connected to the net. If you want to fight spam
instead of just playing lip service to the cause, find an email provider
that can do server-side filtering and reject based on filter criteria at
the time of the SMTP transaction.
I trust you used your real email address in this post?

Yes. There is no informed reason not to.
It is all too easy for emails to be harvested in here - do you not realise
this.

I fully realize this. My point is if your anti-spam strategy is effective,
you do not need to worry about address harvesters at all.
As and when ALL ISPs provide antispam services, then some progress might
be made.

ISPs are incompetent at providing anything other than an Internet
connection. ISPs should not be confused with email providers. You can get
email service from someone other than your ISP.
Re above - I have read the article and to me it appears the writer is in
'cloud cuckoo land' for most of it. I report a lot of spam as I feel
everyone should do their bit. I know a few people who dont bother
anymore. But nine times out of ten, nothing is done by the IPs/ISP about
reports of spam.

Work with your email provider to block the non-responsive sites.
Some appear to encourage it.

Work with your email provider to make sure spam havens are blocked.
Thats where the enforcement should start - there - not at our user level.

That's my point: Munging *forces* the problem onto the user level where it
doesn't belong.
Can you imagine the costs to these companies for setting up an effective
antispam department? Thats why they have no interest.

My view is if they can't provide it responsibly, they should not provide it
at all. Let someone competent do it. Fortunately, this is becoming more
common. Some ISPs, particularly discount ones, no longer offer anything
outside their core service: An Internet connection. If you want email,
news, web hosting, etc., you have to find providers for those as well.
That's the way it should be.
 
Did you remember that you multiposted your item in the
Server group too? Did you check my detailed reply there?
 
Kevin wrote:
You already found it. VFAT and NTFS are just poor performing filesystems
when it comes to dealing with a large quantity of files
True

or files of a large size,

Less true, though a harder problem applies (fixed max size limit)
and like molasses in January when it comes to dealing with a large
quantity of large files like you are.

The size of the files isn't that relevant - it's the number of
directory entries (at least doubled if LFNs are used) that hurts FATxx

You could try the RD command, with appropriate parameters, to delete
the subtree instead of one file at a time.

As you've already discovered, much of the overhead of deleting files
is from the UI shell, not the file system itself. To this,add the
effects of underfootware such as System Restore, which will be busy
logging the deletes and moving pointers to \SVI, resident av, which
may scan each file as it is touched, and indexers and thumbnailers.

Those OS overheads won't go away when you move from FATxx to NTFS, and
in fact may get worse as NTFS offers extra metadata to fiddle with.

The NTFS advantage is that directories are not linear cluster chains,
as FATxx directories are, but use a sort of B-tree indexed structure.
That allows arbitrary files to be found without trudging through the
whole directory from start to finish, as FATxx would do.
If you're looking for faster filesystem performance, I suggest making a file
server out of a Linux box using ReiserFS or XFS filesystems (though even
the default ext3 blows the doors off NTFS or VFAT in terms of performance).

The other aspect is what quality of interactive file management and
data recovery is available (if any). NTFS sucks there.


------------ ----- --- -- - - - -
Drugs are usually safe. Inject? (Y/n)
 
Jonathan wrote:
Defragging won't increase delete performance in the slightest. All you're
doing when you delete on VFAT or NTFS is removing the file handles, not the
actual file data. The actual file data doesn't go away until it's
overwritten by another file (either because something was saved to the same
blocks or a defrag moved another file into the "deleted" file's spot, etc).

Defrag last ;-)

After deleting a large number of files from a directory, Defrag is
required to rebuild that directory's cluster chain so that the deleted
entries (currently merely marked as deleted) are discarded and the
cluster chain is rebuilt (hopefully contiguously) containing only the
undeleted entries that are left.

Do the maths... number of directory entries per file (remember LFNs),
size of each entry, number of entries per cluster, number of clusters
in the directory chain, fragmentation of this chain if files have been
slowly added, length of critical period when updating the chain and
thus risk of data corruption, etc.


------------ ----- --- -- - - - -
Drugs are usually safe. Inject? (Y/n)
 
In
<snip>

Harmful ? Wow, that's rather extreme. To whom exactly is it harmful?

What about those who do not wish to receive mail? Frankly, I don't want
people replying to my posts via e-mail (and neither do most of my usenet
buddies), as the point is for all replies to stay in the group. And as far
as the few-and-far-between 'legitimate replies' are concerned, I truly hope
that any potential sender who was to examine my own munged address would
easily see how to re-enter the address properly. Perhaps it's a bit of an IQ
test. If you are not taller than this sign, you cannot ride on the Cyclone.

Now, backing away from the word "harmful," munging may not be truly
*effective* - but that's not important now.

Now, everyone go get some decaf and enjoy the funny papers. Pax!
 
Replies contextualised and intertwined
Apologies to any reader stumbling into this thread.


That sort of advice is not much use to those silly enough to NOT munge their
email address when using newsgroups then.
I do not know, and neither would you, what antispam a poster may be using if
any on their personal emails.
Munging does absolutely nothing to hinder replies in the groups.
You just can't seem to get this into your head.
To advise them not to munge in the groups, at the outset, in my opinion, is
both reckless and unforgivable.

Agreed - that is assuming a poster is aware of basic home computing
software. Not all ISPs(email providers) provide a full security service.
60% of UK users do not have this basic security on their computers.
Some posters have not even realised their ISP/email provider had on offer an
antispam filter.
You should be preeching to them - not me.

I was right :-)
You can't just hit reply-by-email and expect it to work without having to
go
through hoops. This is not acceptable: Good anti-spam measures do not
hinder legitimate email or shift the burden to legitimate senders in any
way, shape or form. False negatives are better than false positives,
though both should be minimized as much as possible.

Sorry - you lost me re the above para - but please do not bother to try and
expound on it.
You can't. Do it on the receive end, not the send end of the email
process.

Oh I am pleased you at last agree - but of course the protection should be
at the poster's end - so tell them - not me - I already have that
protection - did you not read my posts fully?
Depends on how and who you report it to. It takes experience to learn
this.

Oh does it - do tell me something new - have you ever received a report back
to say 'what' action has been taken? In two years I have had a few
acknowledgements to say a report has been received(no more than 20 I would
say)from IP owners,ISPs and email providers, whatever you want to call the
organisation to whom a report is sent. Some dont even publish abuse links.
I can report, depending on my time, up to 50 spams a week. How many do you
report.
Now this is where you might be able to teach me something - I use two
leading antispam fighting sites to parse and report. I use their experience
etc. So how do you and to whom do you report your spam? But then you would
not have to as you have such a great blocker from your email provider?
That demonstrates a fundamental misunderstanding of anti-spam efforts.

No it does not - warning them is helping them to not attract spam.


If your only goal in fighting spam is mitigating it from your inbox, you
might
as well just not be connected to the net. If you want to fight spam
instead of just playing lip service to the cause, find an email provider
that can do server-side filtering and reject based on filter criteria at
the time of the SMTP transaction.

Once again you are using your soap-box diatribe on me - go elsewhere and
preach to the unconverted, pease.
Yes. There is no informed reason not to.

Apart from giving the harvesters yet another address to sell on to the
spammers.
I fully realize this. My point is if your anti-spam strategy is
effective,
you do not need to worry about address harvesters at all.

My point is yet again - you may be protected - but what about the others who
may not be?
ISPs are incompetent at providing anything other than an Internet
connection. ISPs should not be confused with email providers. You can
get email service from someone other than your ISP.

The above does not make any sense at all. If ISPs are incompetent at
providing anything other than Internet connections I assume you are saying
they either do not provide email services or if they do provide email
service, they cannot run it properly? How stupid can that be. Mine
provides the lot, and does it very well indeed.
Work with your email provider to block the non-responsive sites.

I do already
Work with your email provider to make sure spam havens are blocked.

I do already
That's my point: Munging *forces* the problem onto the user level where
it
doesn't belong.
Rubbish


My view is if they can't provide it responsibly, they should not provide
it
at all.

So who is going to take it away from them?
Let someone competent do it. Fortunately, this is becoming more
common. Some ISPs, particularly discount ones, no longer offer anything
outside their core service: An Internet connection. If you want email,
news, web hosting, etc., you have to find providers for those as well.
That's the way it should be.

And from whom do these email providers get their allocations from????
I have yet to get any satisfactory response to reports from them either.

In conclusion - from Spamhaus-
Spam continues to plague the Internet because a small number of large
Internet Service Providers sell service knowingly to professional spammers
for profit, or do nothing to prevent spammers operating from their networks.

Although all networks claim to be anti-spam, some network executives factor
revenue made from hosting known spam gangs into corporate policy decisions
to continue to sell services to spam operations. Others simply decide that
closing the holes in their end-user broadband systems that allow spammers
access would be too costly to their bottom lines.

The majority of the world's service providers succeed in keeping spammers
off their networks and work to maintain an anti-spam reputation, but their
work is undermined daily by the few networks who, out of corporate greed or
mismanagement, choose to be part of the problem. The world's worst spam
problem networks today are:

Spam continues to plague the Internet because a small number of large
Internet Service Providers sell service knowingly to professional spammers
for profit, or do nothing to prevent spammers operating from their networks.

Although all networks claim to be anti-spam, some network executives
factor revenue made from hosting known spam gangs into corporate policy
decisions to continue to sell services to spam operations. Others simply
decide that closing the holes in their end-user broadband systems that allow
spammers access would be too costly to their bottom lines.

The majority of the world's service providers succeed in keeping
spammers off their networks and work to maintain an anti-spam reputation,
but their work is undermined daily by the few networks who, out of corporate
greed or mismanagement, choose to be part of the problem.

So please do not bother to reply to me further - I am afraid that any
further reasoning/argument from yourself will fall on deaf ears.
I do my bit - you say you do yours.

Rgds
Antioch
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Back
Top