Looking for a newsgroup about steering web crawlers

  • Thread starter Thread starter Stan Hilliard
  • Start date Start date
S

Stan Hilliard

I looked at the log files of my Frontpage web. I am not expert in
reading log files but it is obvious that robots continuously search my
site. They go where I don't think they need to go. Like hit counter
files, obsolete folders, gif files, _vti folders, etc.

I need to learn what to put initially in a robots.txt file to provide
instruction to the robots. Is this the right place to discuss that or
is there another newsgroup for it?

Advice will be appreciated,
Stan Hilliard
 
This should get you started:
http://www.robotstxt.org/wc/robots.html
--
===
Tom Willett
Microsoft MVP - FrontPage
---
FrontPage Support:
http://www.frontpagemvps.com/
===
|I looked at the log files of my Frontpage web. I am not expert in
| reading log files but it is obvious that robots continuously search my
| site. They go where I don't think they need to go. Like hit counter
| files, obsolete folders, gif files, _vti folders, etc.
|
| I need to learn what to put initially in a robots.txt file to provide
| instruction to the robots. Is this the right place to discuss that or
| is there another newsgroup for it?
|
| Advice will be appreciated,
| Stan Hilliard
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

Looking for a IPP with FP2000 extensions 2
Solved -- the incompatibility between FP discussions and FP extensions. 2
What is Best Practice with FP Discussion? 1
Problem deleting message files from discussion subweb. 6
Additional files & folders for disk based web 1
Is there a way to recover a deleted file? 3
Change Default for "My Documents" in FP 2003? 4
Open Web Site 5

Back
Top