Stop HttpWebRequest on my site

[Not4u]

Hello,

Someone is using HttpWebRequest to automaticly post datas or retrieve
datas from my site.
How can i stop pages to be get by the HttpWebRequest method ?

I know that pages get with HttpWebRequest have is no referer.
So for me the solution is to check the referer of the calling page, if
the referer is my domain, ok i can send the page, if not i can redirect
to an error page.
Is it a good idea ?
If you have other solution ?

Thanks

 

[Bruce Barker]

you can add a referer header to a HttpWebRequest. there is really no way to
detect someone useing HttpWebRequest (or like) from a browser. also many
proxy servers strip the referer header, so you may block valid browser
users.

-- bruce (sqlwork.com)

 

[Not4u]

So there is no solution to block the use of HttpWebRequest ?.

 

[tdavisjr]

For posting data to your site, you probably would have to implement
some sort of human verification scheme to detect that a actual person
is actually doing the posting and not a scrip of some sort. You see
this all the time where website display a distored image with letters
and numbers and you have to type what is displayed in the image to
continue. This will trap scrips from posting data.

For reading, you probably have to protect those pages you don't want
them to read and require the person wanting to read to login first.

 

[Not4u]

Yes i'll use the solution of random images.
My problem is it's a price comparison bot that grab info from my site
and use it to compare prices.

 

[Joerg Jooss]

Thus wrote Not4u,

Hello,

Someone is using HttpWebRequest to automaticly post datas or retrieve
datas from my site.
How can i stop pages to be get by the HttpWebRequest method ?
I know that pages get with HttpWebRequest have is no referer.

That's not true. There's no way of knowing whether a request was sent by
a browser or some application simply sending the very same HTTP headers (unless
you could tell from certain implementation specific behaviors at TCP level...).

So for me the solution is to check the referer of the calling page, if
the referer is my domain, ok i can send the page, if not i can
redirect
to an error page.
Is it a good idea ?
If you have other solution ?

If you think that you're victimized by a robot, you may try to add a robots.txt
(see http://www.robotstxt.org/wc/faq.html#robotstxt) to your web site.

Another way is to look for suspicious access patterns in your web server
logs that belong to certain source IP addresses, and block these addresses
at a firewall.

Cheers,