Robot text

  • Thread starter Thread starter Fred
  • Start date Start date
F

Fred

Could some kind person advise me what I should add in my Robot.txt.htm
folder as I want the search engines to keep checking for updates. I've
currently got this but not sure if it's correct.

User-agent: *
Disallow: /cgi-bin/
 
The content of the file is fine. However the name and extra file extension
is wrong. It needs to be robots.txt without the htm extension.
And it needs to be a simple notepad file named robots.txt

--
Steve Easton
Microsoft MVP FrontPage
95isalive
This site is best viewed............
........................with a computer
 
Note however that this won't increase the frequency at which spiders
visit your site - it just tells them what NOT to index when they get
there (in your case everything but the cgibin directory will be
indexed).

You may hear about a meta tag for "revisit-after" but again this won't
bring back the spiders any more frequently - the best it could possibly
do is tell them NOT to come back for a while and that's not usually what
people are trying to do.

Spiders visit on their own schedules. For googlebot (Google's main
spider), updating your pages frequently on a server which returns the
"last modified" information will flag your site for more frequent
visits, as will getting more links from other sites pointing to your
site (increasing PageRank).
 
Back
Top