web query with Workbooks.Open("http:// .....

[Alex]

I query a web page multiple times in a loop with a parameter to get different
pages. QueryTable does not get all the info I like. Furthermore, it does
not work unless one put in a delay in order for excel to wait for the result.


Found the OPEN method in a book; Workbooks.Open("http:// .....
This method is more robust, does not need a delay. I use .Find to parse
the page to get the data.
Just in case, I added a loop delay of about 10 seconds to avoid "issues"
related to server loading. However, eventually got message in the opened
excel book which indicates that the server stopped me. Once I got 100 hits
without problem. Last time I got 7 when the special message showed up for
the first time. Have not tested to see whether the server allows me again or
not.

Does the Workbooks.Open method behave different than a regular browser
click, apparently. Is there a method where one can download the entire web
page without looking different than a regular web access (from the server's
perspective)? I do not mind to have a long loop delay, e.g. 1 minute or more.

regards
Alex

 

[Tim Williams]

What message did you get?
Typically web sites will not (understandably) like being "scraped" and may
put measures in place to prevent people accessing too many pages in quick
succession. These may apply both to browsers or to other tools such as
Excel.

Tim

 

[Alex]

Hi Tim. thanks for your intrest

The message in the Workbook sheet that is opened as a result of the OPEN
method was something like; A message was sent to the IT dep.".

This is far from the exact message but the point is that they must have
understood that I accessed the web page in a different manner. I could use
the excel function; =HYPERLINK(CONCATENATE("http:// ... with a parameter for
the particular web page to get the web page in the browser automatically and
this would be undetectable that it comes under program control if I have a
sufficiently large loop delay (fixed + random) but then I do not know how to
access the browser text from EXCEL ..... I can only program EXCEL, never
done any html or browser programing