blocking robots

  • Thread starter Thread starter Janelle
  • Start date Start date
J

Janelle

Hello! I bought Frontpage because I am html illiterate.
Does anyone know how to put a robot.txt file on your web
site?
I`m having trouble getting an archive site to delete my
web pages that they took. I wanted to take steps so that
in the future they can`t do that anymore but I don`t know
how. My server is AOL and I`m not sure if I can put that
on my site.
Would appreciate any advice :-)
thank you!
Janelle
 
FYI: The robot.txt file only works if a robot chooses to read it and must be
in the root of the domain.

--

==============================================
Thomas A. Rowe (Microsoft MVP - FrontPage)
WEBMASTER Resources(tm)

FrontPage Resources, Forums, WebCircle,
MS KB Quick Links, etc.
==============================================
 
You can use robot tags in your meta tags, again this will only work for
robot's that read meta tags. The following tag would indicated that the page
is not to be index by the robot, nor should any of the hyperlinks on the
page be followed by the robot.

<meta name="robots" content="NOINDEX, NOFOLLOW">

--

==============================================
Thomas A. Rowe (Microsoft MVP - FrontPage)
WEBMASTER Resources(tm)

FrontPage Resources, Forums, WebCircle,
MS KB Quick Links, etc.
==============================================
 
Thanks so much! take care;
Janelle
-----Original Message-----
You can use robot tags in your meta tags, again this will only work for
robot's that read meta tags. The following tag would indicated that the page
is not to be index by the robot, nor should any of the hyperlinks on the
page be followed by the robot.

<meta name="robots" content="NOINDEX, NOFOLLOW">

--

==============================================
Thomas A. Rowe (Microsoft MVP - FrontPage)
WEBMASTER Resources(tm)

FrontPage Resources, Forums, WebCircle,
MS KB Quick Links, etc.
==============================================
To assist you in getting the best answers for FrontPage support see:
http://www.net-sites.com/sitebuilder/newsgroups.asp




.
Click to expand...
 
A robots.txt file is just the job for controlling search engine
spiders, as most often these "good" spiders will honour a robots.txt
file. Another good alternative, as mentioned, is to use robot
exclusions in your meta tags. If you further want to disallow
malicious spiders, which harvest email addresses for spamming purposes
have a look at http://forums.seochat.com/showthread.php?s=&threadid=5654.

===============
Andre Rodziewicz
WimbledonVisitor.Com
The Wimbledon Business Directory
Search Engine Optimisation and Web Design
Helping your business reach the world
http://www.wimbledonvisitor.com/products
 
FYI:

Using a robot.txt also provide a roadmap for robots sent by hackers to
determine what files/folders that are in your web site, that you don't want
seen.

--

==============================================
Thomas A. Rowe (Microsoft MVP - FrontPage)
WEBMASTER Resources(tm)

FrontPage Resources, Forums, WebCircle,
MS KB Quick Links, etc.
==============================================
 
Back
Top