Did I do this right?

[Guest]

I made the robots.txt file in notepad, and saved it into my website folder
thats located on my desktop, so now it's in my list of pages in my fp 2003
program. All my pages are in alphabetical order, so its where R would be.
It has a little typed page symbol next to it, NOT a folder like _private, or
cgi-bin. Is that right?

I published, and if I look at http://mysite.com/robots.txt I see my robot
page.

Sorry, this stuff just confuses me to no end!

Thanks,
Susie

 

[Tom Willett]

It look fine, Susie. You can validate it at:
http://tool.motoricerca.info/robots-checker.phtml

 

[Guest]

Thanks Tom!!! The validator said "No errors found in this robots.txt file"
YEA!!!!!! I did it!!!!

I did mine like this, because I read that it should have at least one
disallow. What do you think, I don't have a password on the cgi-bin file tho.

User-agent: *
Disallow: /cgi-bin/

Thanks again!!!
Susie

 

[Tom Willett]

The whole purpose of a robots.txt file is to disallow files. So, if you
didn't have at least one, why would you need it?

 

[Guest]

I read that a robot file is good to have for search engines. Can you tell me
if I have put it in the root directory? I was very confused by how to get to
the root directory.
Thanks,
Susie

 

[Tom Willett]

The only reason you would need to have one for the search engines is to
prevent 404 errors in your logs when they attempt to search it. And, an
empty robots.txt file would accomplish that (which is what I use).

The robots will visit your site whether you have one or not. Many will
simply not index pages that are denied in your file.

Yes, the file must be in the *root* of the main root web, and not in a sub
site.

 

[Guest]

Thanks Tom, but I have to say that now I am confused, I read this comment by
Steve Raston, (he was replying to someone that wanted to make sure people
found certain things.) Steve said, *And If* you have your robots.txt file
set up to allow search engine spiders to index your web.

So I thought I had to have one. And this site
http://www.seoconsultants.com/robots-text-file/ says in important notes, that
it needs to disallow at least one.

 

[Tom Willett]

Susie:

Don't worry about it. You did just fine ;-)
Move on to your next project.