blocking robots

[Janelle]

Hello! I bought Frontpage because I am html illiterate.
Does anyone know how to put a robot.txt file on your web
site?
I`m having trouble getting an archive site to delete my
web pages that they took. I wanted to take steps so that
in the future they can`t do that anymore but I don`t know
how. My server is AOL and I`m not sure if I can put that
on my site.
Would appreciate any advice
thank you!
Janelle

 

[Jack Brewster]

Try this:
http://www.robotstxt.org/wc/robots.html

Good luck!

 

[Thomas A. Rowe]

FYI: The robot.txt file only works if a robot chooses to read it and must be
in the root of the domain.

--

==============================================
Thomas A. Rowe (Microsoft MVP - FrontPage)
WEBMASTER Resources(tm)

FrontPage Resources, Forums, WebCircle,
MS KB Quick Links, etc.
==============================================

 

[Janelle]

So that means I can`t use it on AOL?
Thanks for the reply

 

[Janelle]

Thanks for the link and the Good Luck
I`ll check it out;
thanks for your reply!
Janelle

 

[Thomas A. Rowe]

You can use robot tags in your meta tags, again this will only work for
robot's that read meta tags. The following tag would indicated that the page
is not to be index by the robot, nor should any of the hyperlinks on the
page be followed by the robot.

<meta name="robots" content="NOINDEX, NOFOLLOW">

--

==============================================
Thomas A. Rowe (Microsoft MVP - FrontPage)
WEBMASTER Resources(tm)

FrontPage Resources, Forums, WebCircle,
MS KB Quick Links, etc.
==============================================

 

[Janelle]

Thanks so much! take care;
Janelle

-----Original Message-----
You can use robot tags in your meta tags, again this will only work for
robot's that read meta tags. The following tag would indicated that the page
is not to be index by the robot, nor should any of the hyperlinks on the
page be followed by the robot.

<meta name="robots" content="NOINDEX, NOFOLLOW">

--

==============================================
Thomas A. Rowe (Microsoft MVP - FrontPage)
WEBMASTER Resources(tm)

FrontPage Resources, Forums, WebCircle,
MS KB Quick Links, etc.
==============================================
To assist you in getting the best answers for FrontPage support see:
http://www.net-sites.com/sitebuilder/newsgroups.asp




.

 

[Andre Rodziewicz]

A robots.txt file is just the job for controlling search engine
spiders, as most often these "good" spiders will honour a robots.txt
file. Another good alternative, as mentioned, is to use robot
exclusions in your meta tags. If you further want to disallow
malicious spiders, which harvest email addresses for spamming purposes
have a look at http://forums.seochat.com/showthread.php?s=&threadid=5654.

===============
Andre Rodziewicz
WimbledonVisitor.Com
The Wimbledon Business Directory
Search Engine Optimisation and Web Design
Helping your business reach the world
http://www.wimbledonvisitor.com/products

 

[Thomas A. Rowe]

FYI:

Using a robot.txt also provide a roadmap for robots sent by hackers to
determine what files/folders that are in your web site, that you don't want
seen.

--

==============================================
Thomas A. Rowe (Microsoft MVP - FrontPage)
WEBMASTER Resources(tm)

FrontPage Resources, Forums, WebCircle,
MS KB Quick Links, etc.
==============================================