Robots.txt file.

F

Fred

Can some kind person remind me what text/code I should place in the
Robots.txt file page so that S/E spiders will crawl and index the site. I
currently use this but unsure if this is correct.
User-agent: *
Disallow: /cgi-bin/

Fred
 
D

David Baxter

You don't actually need a robots.txt file to tell spiders to crawl your
site - you only need it to tell spider what NOT to crawl/index.

What you have here is correct, assuming you want everything indexed
EXCEPT the cgi-bin directory (which makes sense).
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top