Spiders/Bots can be friendly- how do you think google, msn,
etc get all of their information?
Get the name of the bot and search for info on it. Many
times the useragent section in the site log will have the
url for more info on the spider/bot
Todays bots by me-
IRLbot/1.0+(+
http://irl.cs.tamu.edu/crawler) ---------------------
Indifferent - A research project by Texas A and M
Baiduspider+(+
http://www.baidu.com/search/spider.htm) --------
Unknown to me spider
Googlebot/2.1+(+
http://www.googlebot.com/bot.html)-----------
Google
msnbot/1.0+(+
http://search.msn.com/msnbot.htm) ---------------
MSN
appie+1.1+(
http://www.walhello.com) --------------------------Unknown
to me spider
Baidu is the only one I ned to keep an eye AFAIK. The IP
origin was China and from what I could find they deal with
some porn on their networks. If there are others I should be
concerned about, please let me know
message
: Thanks Thom. I will try to make some sense of it and
respond accordingly.
: Wasn't aware there could be"friendly" robots.
:
: "Thomas A. Rowe" wrote:
:
: > You need to first determine if the robots are friendly,
before you consider blocking them. You need
: > to review the raw log files, because a lot of detail is
not available in the host provide stat
: > applications.
: >
: > --
: > ==============================================
: > Thomas A. Rowe (Microsoft MVP - FrontPage)
: > WEBMASTER Resources(tm)
: >
: > FrontPage Resources, WebCircle, MS KB Quick Links, etc.
: > ==============================================
: > To assist you in getting the best answers for FrontPage
support see:
: >
http://www.net-sites.com/sitebuilder/newsgroups.asp
: >
message
: >
: > >I was aware of that but when I read my usage report my
web host provides
: > > there are a few IP address that show up all the time
and at odd hours in the
: > > morning. The report calls them robots or something
similar and I assume it
: > > involves spam or something not desirable.
: > >
: > > "Kevin Spencer" wrote:
: > >
: > >> You should also consider that most popel (and their
computers) don't have
: > >> permanent IP addresses. They change every couple of
days.
: > >>
: > >> --
: > >> HTH,
: > >>
: > >> Kevin Spencer
: > >> Microsoft MVP
: > >> ..Net Developer
: > >> What You Seek Is What You Get.
: > >>
in message
: > >>
: > >> > Not exactly the answer I was hoping for, I was
thinking it could be
: > >> > something
: > >> > simple. Thanks.
: > >> >
: > >> > "Thomas A. Rowe" wrote:
: > >> >
: > >> >> FP doesn't have any function to accomplish this
directly. Your host/admin
: > >> >> would have to do this
: > >> >> directly on server via the IIS MMC or you would
have to write custom
: > >> >> ASP/VBScript and rename all
: > >> >> pages to have a .asp extensions.
: > >> >>
: > >> >> --
: > >> >> ==============================================
: > >> >> Thomas A. Rowe (Microsoft MVP - FrontPage)
: > >> >> WEBMASTER Resources(tm)
: > >> >>
: > >> >> FrontPage Resources, WebCircle, MS KB Quick Links,
etc.
: > >> >> ==============================================
: > >> >> To assist you in getting the best answers for
FrontPage support see:
: > >> >>
http://www.net-sites.com/sitebuilder/newsgroups.asp
: > >> >>
: > >> >> "Don Hale" <
[email protected]>
wrote in message
: > >> >>
: > >> >> > Is there a way to ban IP addresses in FP 2003
publishing to a Windows
: > >> >> > 2003
: > >> >> > server?
: > >> >> > All the info and script examples I find are for
non-windows servers.
: > >> >>
: > >> >>
: > >> >>
: > >>
: > >>
: > >>
: >
: >
: >