How can I tell if a visitor is a search engine bot?

[Scott Allen]

You'll want to look at the Agent header coming in with the request.

Here are some reference sources:
http://www.google.com/search?hl=en&q=search+engine+agent+strings

HTH!

 

[jongalloway]

You can check the Request.UserAgent property. Each major search bot
uses a different user agent, so you'd need to check against a list. The
biggest ones are googlebot, msnbot, and slurp (Yahoo).

Here's a partial list:
http://webmaster-resources.web-mastery.net/spider-identification.html

If you just want to prevent search engines from spidering all or
portions of your site, you can do this with a robots.txt file at the
root of your site:
http://www.searchengineworld.com/robots/robots_tutorial.htm

Keep in mind that it is possible for users can change their user agent
(for instance via the Firefox User Agent Switcher extension), so if you
have pages of your site that you want to allow only subscribers or
search engines, you'd need to check against an IP whitelist to keep
people out.

- Jon
http://weblogs.asp.net/jgalloway

 

[Mark]

Hi, I am writing some code to log vistors to my website.

I notice in some statistics pacakages that they are able to differentiate
between normal visitors and search bots. Does anyone know how that is done?

I thought that they might be looking at the IP address of the remote visitor
and then if the IP range is associated with Google or whatever than that
"person" is a bot...

Any ideas welcome
thanks
Mark

 

[Mark]

Hi, thanks for your help
Regards
Mark

 

[bruce barker]

note: following these bot rules is strictly up to the bot. the major engines
are polite and do, but not all.

-- bruce (sqlwork.com)

| You can check the Request.UserAgent property. Each major search bot
| uses a different user agent, so you'd need to check against a list. The
| biggest ones are googlebot, msnbot, and slurp (Yahoo).
|
| Here's a partial list:
| http://webmaster-resources.web-mastery.net/spider-identification.html
|
| If you just want to prevent search engines from spidering all or
| portions of your site, you can do this with a robots.txt file at the
| root of your site:
| http://www.searchengineworld.com/robots/robots_tutorial.htm
|
| Keep in mind that it is possible for users can change their user agent
| (for instance via the Firefox User Agent Switcher extension), so if you
| have pages of your site that you want to allow only subscribers or
| search engines, you'd need to check against an IP whitelist to keep
| people out.
|
| - Jon
| http://weblogs.asp.net/jgalloway
|