If your robots.txt file contains
nothing but:
User-agent:*
Then it tells the bots they have permission to
index the entire site.
If there's a section you don't want indexed,
then you add:
Disallow: /cgi-bin/
to keep them out of the cgi-bin,
Disallow: /images/
to keep them out of your images folder,
and so forth.
However the robots.txt file is open
"to the public" and can be read by anyone.
Example:
http://www.95isalive.com/robots.txt
--
using 2k PRO but....95isalive
This site is best viewed............
........................with a computer
Sorry for being a bit thick but the site you gave started with "This is a useful file that keeps search engines from indexing pages you do not want spidered" Obviously I want the opposite to happen and would welcome advice on 'What is - and where is it located, the root of my web' (told you I was thick) and getting even thicker, what do I place in the robots.txt file?
Yes, but you should also add
a robots.txt file to the root of your
web.
Here's a "how to."
http://www.pageresource.com/zine/robotstxt.htm
--
using 2k PRO but....95isalive
This site is best viewed............
.......................with a computer
If I add the following meta tag to my pages will it allow the robot to visit?
<meta name="robots" content="index,follow">
<meta name="revisit-after" content="7 days">
Thanks in anticipation of the usual helpful advice from this group. (e-mail address removed)
