site does NOT provide the /robots.txt file

D

Dan

After doing a SE ready check at Scrub the Web I received this message: This
site does NOT provide the /robots.txt file or the /robots.txt file is not
constructed properly. What are they talking about here? How would this tag
lay out?
 
M

Mark Fitzpatrick

A robots.txt file is basically a file that tells a search engine what files
to include/exclude kind of thing. This isn't something that FP or other
editors create for you, it's basically something you create for yourself.
Check out: http://www.robotstxt.org/wc/robots.html for some info on robots
exclusion.

Hope this helps,
Mark Fitzpatrick
Microsoft MVP - FrontPage
 
D

Dan

Does the meta tags do the same thing? If so why would you need the
robots.txt file at all?
<meta name="robots" content="Index,Follow">
or
<meta name="robots" content="All">
 
T

Thomas A. Rowe

The Robot.txt file is only for excluding content from being indexed, and
unless you have your own domain, you can use the robot.txt file.

You have more control when using the robot meta tags. As you can specific,
NoIndex,Follow which tells the search engine to not index the page, but
follow any links on the page.

But keep in mind that not all search engine read or honor the robot.txt file
or the robot meta tags.

--

==============================================
Thomas A. Rowe (Microsoft MVP - FrontPage)
WEBMASTER Resources(tm)

FrontPage Resources, Forums, WebCircle,
MS KB Quick Links, etc.
==============================================
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top