Anybody know how to download a whole website?

[Jourdy288]

http://www.craveonline.com/gaming/articles/179039-gamepro-magazine-shuts-down


Looks like GamePro is about to be shut down- and with it a TON of articles about the gaming industry! Before they shut down the site, anybody know how to save a copy of it? I want it all.

 

[Mr Rob]

Downloading 'all' of a website is impossible, as you do not know the location of every file on the server. About the only approach you could do is use a web crawler to get the first page, identify all the links, follow all of those links to other web pages, grab the links on those pages, and repeat.


There are probably solutions for this already online somewhere, or you could even build one with a little bit of scripting.

 

[mindlord]

httrack is a free tool for mirroring websites

 

[diligentcircle]

Is it not archived on the wayback machine?

 

[Blue Protoman]

Backups of every page are not guaranteed.

 

[Caine]

On linux, type (b.t.w. this is not very friendly to the website owner):

wget -m -k -K -E http://www.gamepro.com

On windows, check out httrack.


On MAC, browse to a website that sells non-mac products

 

[Prometheus]

On linux, type (b.t.w. this is not very friendly to the website owner):

wget -m -k -K -E http://www.gamepro.com

On windows, check out httrack.


On MAC, browse to a website that sells non-mac products

Or just use curl, which is equivalent to wget...

 

[Jourdy288]

Thanks guys, I'll try these! I wonder how big it'll be...

 

[Mr Rob]

Holy Hannah, apparently wget has a -r option, too. For recursion. I love man pages.

 

[Caine]

Holy Hannah, apparently wget has a -r option, too. For recursion. I love man pages.

You've got to love that mentality of doing only one well-defined thing and do it REALLY well.