Looking for a newsgroup about steering web crawlers

S

Stan Hilliard

I looked at the log files of my Frontpage web. I am not expert in
reading log files but it is obvious that robots continuously search my
site. They go where I don't think they need to go. Like hit counter
files, obsolete folders, gif files, _vti folders, etc.

I need to learn what to put initially in a robots.txt file to provide
instruction to the robots. Is this the right place to discuss that or
is there another newsgroup for it?

Advice will be appreciated,
Stan Hilliard
 
T

Tom Willett

This should get you started:
http://www.robotstxt.org/wc/robots.html
--
===
Tom Willett
Microsoft MVP - FrontPage
---
FrontPage Support:
http://www.frontpagemvps.com/
===
|I looked at the log files of my Frontpage web. I am not expert in
| reading log files but it is obvious that robots continuously search my
| site. They go where I don't think they need to go. Like hit counter
| files, obsolete folders, gif files, _vti folders, etc.
|
| I need to learn what to put initially in a robots.txt file to provide
| instruction to the robots. Is this the right place to discuss that or
| is there another newsgroup for it?
|
| Advice will be appreciated,
| Stan Hilliard
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

What is Best Practice with FP Discussion? 1
Solved -- the incompatibility between FP discussions and FP extensions. 2
Problem deleting message files from discussion subweb. 6
Logging(username,password) in the web automatically by web crawler 2
Open Web Site 5
no convert sub web option on right click FP2003 1
multiple domains, same "My Web Sites" folder 1
Repost: How to set up Windows XP so that FrontPage recgonises local webs? 22

Top