Command-line file downloader

[Rich Pasco]

I am looking for a Win32 command-line (console) utility, suitable for
being driven from a BAT script, which accepts as input a URL and then
downloads the referenced web page (or other file) to my local disk.

The effect that I want to produce is exactly that I would get by right-
clicking on a link in a web browser and choosing from the popup menu
"save link target as..." except this one would be driven entirely from
the command line, without graphics, popups, or point-and-click.

I began to write my own, but it's harder than it looks; it involves
parsing the URL from the command line, to get the host name, issuing a
DNS query to get its IP address, opening a socket and speaking HTTP
to request the file, then interpreting the HTTP reply to reconstruct
the file and save it locally.

Might be easier if someone else has already done it.

Any suggestions?

- Rich

 

[foxidrive]

I am looking for a Win32 command-line (console) utility, suitable for
being driven from a BAT script, which accepts as input a URL and then
downloads the referenced web page (or other file) to my local disk.

WGET

 

[Olof Lagerkvist]

WGET

Download: http://UnxUtils.sourceforge.net

 

[Rich Pasco]

WGET

Thank you! I found it at http://www.gnu.org/software/wget/wget.html

- Rich

 

[Stefan Kanthak]

I am looking for a Win32 command-line (console) utility, suitable for
being driven from a BAT script, which accepts as input a URL and then
downloads the referenced web page (or other file) to my local disk.

Microsoft's BITSAdmin.exe controls the built-in BITS, but is restricted
to http.
If you want a more versatile utility, then get cURL http://curl.haxx.se/;
it's "better" than WGET.

Stefan

 

[Mark V]

Thank you! I found it at
http://www.gnu.org/software/wget/wget.html

Here are two more links to round it out.

http://xoomer.alice.it/hherold/

http://users.ugent.be/~bpuype/wget/ (win32 only)

 

[Rich Pasco]

Microsoft's BITSAdmin.exe controls the built-in BITS, but is restricted
to http.

I found information about it here:
http://msdn2.microsoft.com/en-us/library/aa362813.aspx

If you want a more versatile utility, then get cURL http://curl.haxx.se/;
it's "better" than WGET.

In what way is it "better"? WGET is already working perfectly in my script.

- Rich

 

[Mark Blain]

In what way is it "better"? WGET is already working perfectly in my
script.

There's a feature comparison of several command-line downloading tools
including cURL and wget at
http://curl.haxx.se/docs/comparison-table.html

 

[Jason Gurtz]

There's a feature comparison of several command-line downloading tools
including cURL and wget at
http://curl.haxx.se/docs/comparison-table.html

The lack of recursive downloading is a pretty significant hit against
curl. FWIW, the comparison table also barely scratches the features of
wget.

It appears the main advantage of curl is support for a small handful of
additional protocols. IMO to issue a blanket statement that it is carte
blanch better than wget is pretty bold. That said, curl is a pretty
great choice.

Maybe more important is the related curl library for programming projects

It also might be worth taking a look at the ncftp package. In addition
to the interactive commandline ftp client, there is also the scriptable
ncftpput and ncftpget included (of course it is ftp only).

Good luck in all automated file transfer projects...

~Jason

 

[Mark Blain]

It appears the main advantage of curl is support for a small handful
of additional protocols. IMO to issue a blanket statement that it is
carte blanch better than wget is pretty bold...

For the record Stefan said that, not me. You're correct that the
comparison table I suggested is incomplete, but it's a good place to start.

 

[Stefan Kanthak]

Address munging IS HARMFUL!

The lack of recursive downloading is a pretty significant hit against
curl. FWIW, the comparison table also barely scratches the features of
wget.

It barely scratches the features of curl too.-)
While curl itself doesn't perform recursive up/downloads, there are libcurl
based programs which have this capability.
And besides downloads, curl/libcurl does uploads too.
Has wget an accompanying libwget?!
Does it support large files?
OK, thats now rhetoric, but LFS support was introduced just recently.
Look at the history of both tools: wget was written to fetch web pages,
curl was written to "grok URLs".

It appears the main advantage of curl is support for a small handful of
additional protocols.

And uploads too, and a library, ...

IMO to issue a blanket statement that it is carte
blanch better than wget is pretty bold. That said, curl is a pretty
great choice.

I wrote "better" not without reason enclosed in quotes.

Maybe more important is the related curl library for programming projects
Right.

It also might be worth taking a look at the ncftp package. In addition
to the interactive commandline ftp client, there is also the scriptable
ncftpput and ncftpget included (of course it is ftp only).

Good luck in all automated file transfer projects...

But on the first hand: why should I "install" a third party solution at
all and not use BITS instead (which has the advantage to use the Internet
settings of Windows). Unfortunately BITSAdmin.exe needs to be installed
from the support tools package too.

Stefan

 

[Jason Gurtz]

Address munging IS HARMFUL!

Concrete reasons for NOT munging on public groups (particularly ones
mirrored on the web) will be entertained if anyone cares to offer them.

Religious rants....well, I'm a spambot hating atheist sorry

~Jason

--