Robot text

[Fred]

Could some kind person advise me what I should add in my Robot.txt.htm
folder as I want the search engines to keep checking for updates. I've
currently got this but not sure if it's correct.

User-agent: *
Disallow: /cgi-bin/

 

[Steve Easton]

The content of the file is fine. However the name and extra file extension
is wrong. It needs to be robots.txt without the htm extension.
And it needs to be a simple notepad file named robots.txt

--
Steve Easton
Microsoft MVP FrontPage
95isalive
This site is best viewed............
........................with a computer

 

[David Baxter]

Note however that this won't increase the frequency at which spiders
visit your site - it just tells them what NOT to index when they get
there (in your case everything but the cgibin directory will be
indexed).

You may hear about a meta tag for "revisit-after" but again this won't
bring back the spiders any more frequently - the best it could possibly
do is tell them NOT to come back for a while and that's not usually what
people are trying to do.

Spiders visit on their own schedules. For googlebot (Google's main
spider), updating your pages frequently on a server which returns the
"last modified" information will flag your site for more frequent
visits, as will getting more links from other sites pointing to your
site (increasing PageRank).