Re: wget guru needed

Jim Showalter <jshowalter@xxxxxxxxx> writes:

GoDaddy doesn't allow the use of ssh on shared hosted accounts, so I'm
backing-up my small website using wget with the following options:

"wget --recursive --wait=3 --tries=5 --mirror --output-file=wget.log
--passive-ftp --page-requisites ftp://uname:pas...@xxxxxxxxxxxxxxxx";

I have three questions:

1. The three second delay between retrievals makes this very slow -
2.5 hrs. to backup 17 MB! Could I reduce 'wait' to 1 or 2 seconds
without being a jerk?

2. Do I have the correct options to give reliable backups of my website?

3. Are any of these options unnecessary?


The underlying problem with doing it this was is that you're using a
command line http tool to access an ftp URL. wget spiders web pages
using links. you might get some very interesting results depending on
how the links are constructed in your files (absolute vs relative),
and it's not clear to me whether when scripting files are invoked by
the web server whether they'd even be grabbed by your command above.

It's suggest abandoning wget for this task.

If you would like commandline (and there are many good reasons to like it), look into rsync.

mkdir mysitebackup
cd mysitebackup
rsync -P -v -r ftp.site.com::/ .


Otherwise a proper GUI ftp client for whatever platform you are using
would also be good. Obiously, command line ftp isn't terribly
feasible for such a thing, but there are a number of good gui ftp
clients out there that have an understanding of pulling down entire
directory structures. For windows, winscp is decent.



--
Todd H.
http://www.toddh.net/
.

Relevant Pages

  • testing a website from python
    ... I just want to test that a given website is up or not from a python script. ... I thought of using wget as an os command. ...
    (comp.lang.python)
  • Re: testing a website from python
    ... I just want to test that a given website is up or not from a python script. ... I thought of using wget as an os command. ...
    (comp.lang.python)
  • Re: [SLE] wget confusion...
    ... >>I'm a bit confused by the behavior of wget... ... I've issued the command ... >, it will retrieve that file ... Anders ...
    (SuSE)
  • Re: [SLE] wget confusion...
    ... > I'm a bit confused by the behavior of wget... ... I've issued the command ... (later mod time than the previous download), it will retrieve that file ...
    (SuSE)
  • Re: Thoughts about finding viruses in email inboxes
    ... I don't know why they are using wget to get it, but it worked for me as ... and had no problems with the install. ... Tried the wget command and got: ... your browser appears to be trying to go through a proxy server on ...
    (Ubuntu)