Re: Searching for Robust Download Automation

On Apr 9, 1:40 am, g...@xxxxxxxxx (Greg Andrews) wrote:
UUCP? Was that a joke?

I was half joking, since the original poster didn't mention a need
to execute a command upon successful transfer.

However I was half serious.  I can't count the number of times I've
seen people post questions in this group about scripting automatic
file transfers that will detect when the transfer is completed (not
just interrupted) and perform some action on the transferred file
that usually involves running a Unix script/command/program on it.

The usual replies seem to involve shell/perl scripts to glue a non-
retrying transport program (e.g. ftp/scp/rsync) together with an
ad hoc methods for detecting corrupted/incomplete files, executing
commands on the successful ones, logging any errors from executing
those commands, and cleaning out the transferred files.  All of which
is reinventing the wheel that UUCP has had for 20+ years.

UUCP has done the things I list above for 20+ years, has had very
fast transfer protocols for 10+ years, and the config steps to run
it over ssh aren't harder than for other transports like rsync.

The two downsides to UUCP are its config files are more arcane, having
come out of the Bronze Age of Unix utility software development, and
there are fewer websites offering config tips.

However, for the average person asking for this kind of automatic
transfer system, those hurdles are much easier to handle than the large,
complex programming task required to implement the usual suggestions.

Don't get me wrong.  I'm not saying UUCP is a be-all, end-all.  I'm
just saying it's overlooked in some situations where it's the right
tool for the job.  Then it's scoffed at, as you did.

I can't use UUCP because some of the data source nodes are not under
my control.