Robots.txt

[Mike Reilly]

I have a site that has been published in FP2002 and hosted on an Apache
server. When looking over the site analysis reports I noticed that in the
"Failures" section that /robots.txt is by far the most common failure.

I would like someone to look at the code on my site to see if my "Robots"
meta tag is done correctly.

The site is www.outsidebanks.com .

I would appreciate the help if something is wrong here. Google has stopped
crawling the site for some reason. Some other spiders are still visiting
regularly.

Thanks

 

[Steve Easton]

It's not the meta tag, the search bots are looking
for a file named robots.txt which is used to set
permissions for the search engines / web crawlers.

Create one using notepad, that contains the following:
User-agent: *
and save it in your root directoryas robots.txtif you want it to exclude
your images folder then use this.User-agent: *
Disallow: /images/
Also, do a Google search for robots.txtfor more info
-- using 2k PRO but....95isaliveThis site is best
viewed...................................with a computer"Mike Reilly"

 

[Steve Easton]

Reformatted.!!

It's not the meta tag, the search bots are looking
for a file named robots.txt which is used to set
permissions for the search engines / web crawlers.

Create one using notepad, that contains the following:

User-agent: *

and save it in your root directory as robots.txt
if you want it to exclude
your images folder then use this.

User-agent: *
Disallow: /images/

Also, do a Google search for robots.txt for more info



I have a site that has been published in FP2002 and hosted on an Apache
server. When looking over the site analysis reports I noticed that in the
"Failures" section that /robots.txt is by far the most common failure.

I would like someone to look at the code on my site to see if my "Robots"
meta tag is done correctly.

The site is www.outsidebanks.com .

I would appreciate the help if something is wrong here. Google has stopped
crawling the site for some reason. Some other spiders are still visiting
regularly.

Thanks