Robots.txt file.

  • Thread starter Thread starter Fred
  • Start date Start date
F

Fred

Can some kind person remind me what text/code I should place in the
Robots.txt file page so that S/E spiders will crawl and index the site. I
currently use this but unsure if this is correct.
User-agent: *
Disallow: /cgi-bin/

Fred
 
You don't actually need a robots.txt file to tell spiders to crawl your
site - you only need it to tell spider what NOT to crawl/index.

What you have here is correct, assuming you want everything indexed
EXCEPT the cgi-bin directory (which makes sense).
 
Back
Top