I am making use of a proxy to refresh certain areas of a webpage every
second. From the point of view of the website in question, this is not
a problem, however I feel it must put a significant load on my ISPs
DNS server, making requests every second to refresh.websitename.com
(sometimes www.websitename.com, or cache.websitename.com when grabbing
required graphics etc).

Will this be putting a load on the server that could lead to my IP
address being blocked (can they even do that)? i.e. does the host get
resolved on every request?

What is the best way to minimise the load? Use of the Windows hosts
file?

My ISP DNS server does not seem that reliable and I regularly get
"unable to resolve hostname" errors. I switched to a public DNS server
yesterday and those errors disappeared, however this morning I was no
longer able to use the public DNS server. Could they have blocked my
IP address due to excessive usage?

I understand, and agree, that I should not be using a DNS server other
than that provided by my ISP, it was simply for test purposes to prove
the root cause. I am now in the process of switching ISPs.

--
Iain Simpson
(E-Mail Removed)
(remove 'GetRidOfThis' from address)

On Tue, 06 Apr 2004 12:47:44 -0400, Iain Simpson
<(E-Mail Removed)> wrote:

>I am making use of a proxy to refresh certain areas of a webpage every
>second. From the point of view of the website in question, this is not
>a problem, however I feel it must put a significant load on my ISPs
>DNS server, making requests every second to refresh.websitename.com
>(sometimes www.websitename.com, or cache.websitename.com when grabbing
>required graphics etc).
>
>Will this be putting a load on the server that could lead to my IP
>address being blocked (can they even do that)? i.e. does the host get
>resolved on every request?

The amount of load depends on the DNS record and zone TTL for the
hosts at websitename.com. You can find out the TTL by performing an
nslookup from the command prompt in Windows. Yes, someone could block
you from using their DNS server, with varying degrees of
effectiveness.

This doc might help:
http://www.microsoft.com/resources/d...owDnsWorks.asp

(http://tinyurl.com/22nou)

>What is the best way to minimise the load? Use of the Windows hosts
>file?

A Windows Hosts file might be a good way to solve the 'host not found'
error.

>
>My ISP DNS server does not seem that reliable and I regularly get
>"unable to resolve hostname" errors. I switched to a public DNS server
>yesterday and those errors disappeared, however this morning I was no
>longer able to use the public DNS server. Could they have blocked my
>IP address due to excessive usage?

You might try nslookup using that server as the default just to see if
you're blocked.

Here's a doc on nslookup if you're not familar with it:
http://support.microsoft.com/default...b;en-us;200525

Steve

>The amount of load depends on the DNS record and zone TTL for the
>hosts at websitename.com. You can find out the TTL by performing an
>nslookup from the command prompt in Windows. Yes, someone could block
>you from using their DNS server, with varying degrees of
>effectiveness.

Thanks for your reply Steve.

It looks like, for whatever reason, the public DNS server I was using
yesterday is no longer available (or at least temporarily
unavailable), as even using online ping services, it is unreachable.

>>What is the best way to minimise the load? Use of the Windows hosts
>>file?
>
>A Windows Hosts file might be a good way to solve the 'host not found'
>error.

But will creating entries in my hosts file in any way lessen the
burden on the DNS server I specify in my router? So, if my proxy is
making a request every second to refresh.betfair.com, will making an
entry for refresh.betfair.com in my hosts file along with it's IP
address, lessen the demands on the primary DNS server I have in my
settings?

--
Iain Simpson
(E-Mail Removed)
(remove 'GetRidOfThis' from address)

On Tue, 06 Apr 2004 12:47:44 -0400, Iain Simpson
<(E-Mail Removed)> wrote:

>I am making use of a proxy to refresh certain areas of a webpage every
>second. From the point of view of the website in question, this is not
>a problem, however I feel it must put a significant load on my ISPs
>DNS server, making requests every second to refresh.websitename.com
>(sometimes www.websitename.com, or cache.websitename.com when grabbing
>required graphics etc).
>
>Will this be putting a load on the server that could lead to my IP
>address being blocked (can they even do that)? i.e. does the host get
>resolved on every request?

No. It's cached the first time you look it up.

>What is the best way to minimise the load? Use of the Windows hosts
>file?

That's one way, but first determine if you even have a load. I don't
know a way to do this outside of a sniffer or protocol analyzer, but
there may be other methods.

>My ISP DNS server does not seem that reliable and I regularly get
>"unable to resolve hostname" errors. I switched to a public DNS server
>yesterday and those errors disappeared, however this morning I was no
>longer able to use the public DNS server. Could they have blocked my
>IP address due to excessive usage?
>
>I understand, and agree, that I should not be using a DNS server other
>than that provided by my ISP, it was simply for test purposes to prove
>the root cause. I am now in the process of switching ISPs.

You could resolve all of this by not using your "refresh" method. I
doubt you have a load on DNS, but it is generating traffic.

Jeff

On Wed, 07 Apr 2004 01:21:38 -0400, Iain Simpson
<(E-Mail Removed)> wrote:

>
>>The amount of load depends on the DNS record and zone TTL for the
>>hosts at websitename.com. You can find out the TTL by performing an
>>nslookup from the command prompt in Windows. Yes, someone could block
>>you from using their DNS server, with varying degrees of
>>effectiveness.
>
>Thanks for your reply Steve.
>
>It looks like, for whatever reason, the public DNS server I was using
>yesterday is no longer available (or at least temporarily
>unavailable), as even using online ping services, it is unreachable.
>
>>>What is the best way to minimise the load? Use of the Windows hosts
>>>file?
>>
>>A Windows Hosts file might be a good way to solve the 'host not found'
>>error.
>
>But will creating entries in my hosts file in any way lessen the
>burden on the DNS server I specify in my router? So, if my proxy is
>making a request every second to refresh.betfair.com, will making an
>entry for refresh.betfair.com in my hosts file along with it's IP
>address, lessen the demands on the primary DNS server I have in my
>settings?

You always use the same address. Once you look it up the first time,
it's cached and you don't look it up again until the TTL expires or
your cache flushes. Whether you look it up in DNS or Hosts makes no
difference here.

Hosts will be queried before DNS. If the name is resolved there,
there is no DNS request made.

Jeff

On Wed, 07 Apr 2004 13:06:58 GMT, (E-Mail Removed) (Jeff
Cochran) wrote:

>You always use the same address. Once you look it up the first time,
>it's cached and you don't look it up again until the TTL expires or
>your cache flushes. Whether you look it up in DNS or Hosts makes no
>difference here.
>
>Hosts will be queried before DNS. If the name is resolved there,
>there is no DNS request made.

Thanks for your replies Jeff.

This makes sense on one level, but not on another.

I use a piece of software called Privoxy to refresh the relevant
section of the webpage every second. Privoxy creates a log file giving
details of each request made. Generally that stream involves requests
to refresh.betfair.com. As you can see these requests are being
successful but suddenly a request to the same domain
(refresh.betfair.com) fails with a "could not resolve hostname" error.
From what you have said, that would suggest the TTL for
refresh.betfair.com must be zero, i.e. a DNS lookup is required every
request? Otherwise how could it fail having been cached? If the TTL is
zero, would a hosts entry for refresh.betfair.com be a big help in
reducing the requests to the DNS server?

Apr 02 15:41:03 Privoxy(02220) Request:
refresh.betfair.com/BetMgr_Refresh.aspx?ei=1693681&typ=O&iid=0&ts=1080938463000&init=0&sb=0
Apr 02 15:41:04 Privoxy(02220) Request:
refresh.betfair.com/BetMgr_Refresh.aspx?ei=1693681&typ=O&iid=0&ts=1080938464000&init=0&sb=0
Apr 02 15:41:05 Privoxy(02220) Request:
refresh.betfair.com/BetMgr_Refresh.aspx?ei=1693681&typ=O&iid=0&ts=1080938465000&init=0&sb=0
Apr 02 15:41:06 Privoxy(02220) Request:
refresh.betfair.com/BetMgr_Refresh.aspx?ei=1693681&typ=O&iid=0&ts=1080938466000&init=0&sb=0
Apr 02 15:41:07 Privoxy(02220) Request:
refresh.betfair.com/BetMgr_Refresh.aspx?ei=1693681&typ=O&iid=0&ts=1080938467000&init=0&sb=0
Apr 02 15:41:07 Privoxy(02220) Error: could not resolve hostname
refresh.betfair.com
Apr 02 15:41:08 Privoxy(02220) Request:
refresh.betfair.com/BetMgr_Refresh.aspx?ei=1693681&typ=O&iid=0&ts=1080938468000&init=0&sb=0
Apr 02 15:41:08 Privoxy(02220) Error: could not resolve hostname
refresh.betfair.com

--
Iain Simpson
(E-Mail Removed)
(remove 'GetRidOfThis' from address)

On Wed, 07 Apr 2004 13:04:38 GMT, (E-Mail Removed) (Jeff
Cochran) wrote:

>You could resolve all of this by not using your "refresh" method. I
>doubt you have a load on DNS, but it is generating traffic.

It is indeed generating traffic, but no more than say listening to
music, a radio station, or downloading a movie over the 'net does.
That's what I pay my ISP for.

The need for the regular refresh is to gain a stream of prices, so no
different in terms of traffic than someone signed up to receive
streaming NASDAQ quotes, or whatever. Just a different way of
achieving the same kind of real-time data. I am creating more "up"
traffic than they are, but that's about it.

--
Iain Simpson
(E-Mail Removed)
(remove 'GetRidOfThis' from address)

On Wed, 07 Apr 2004 09:43:03 -0400, Iain Simpson
<(E-Mail Removed)> wrote:

>On Wed, 07 Apr 2004 13:06:58 GMT, (E-Mail Removed) (Jeff
>Cochran) wrote:
>
>>You always use the same address. Once you look it up the first time,
>>it's cached and you don't look it up again until the TTL expires or
>>your cache flushes. Whether you look it up in DNS or Hosts makes no
>>difference here.
>>
>>Hosts will be queried before DNS. If the name is resolved there,
>>there is no DNS request made.
>
>Thanks for your replies Jeff.
>
>This makes sense on one level, but not on another.
>
>I use a piece of software called Privoxy to refresh the relevant
>section of the webpage every second. Privoxy creates a log file giving
>details of each request made. Generally that stream involves requests
>to refresh.betfair.com. As you can see these requests are being
>successful but suddenly a request to the same domain
>(refresh.betfair.com) fails with a "could not resolve hostname" error.
>From what you have said, that would suggest the TTL for
>refresh.betfair.com must be zero, i.e. a DNS lookup is required every
>request? Otherwise how could it fail having been cached? If the TTL is
>zero, would a hosts entry for refresh.betfair.com be a big help in
>reducing the requests to the DNS server?

Just because it's cached doesn't mean something won't prevent it from
reading the cache. Nor does it prevent another type of error that
your app reports as being unable to resolve the name.

Jeff