Re: scp'ing 250,000 files

Michael Heiming <michael+USENET@xxxxxxxxxxxxxx> wrote:
In comp.unix.shell Digital Puer <digital_puer@xxxxxxxxxxx>:
Hi, I have a huge set of 250,000 files (each about 15KB) that I need
to transfer from one machine to another. I would prefer to use scp.

These files have a .dat extension, but there are others (executables,
.txt files, etc.) interspersed between them. I would like just do
"scp *.dat ..." but the command-line is saying that there are too
many arguments. I would prefer not to scp the whole directory,
since there are too many non-.dat files to be transferred.

The host machine is Windows with Cygwin running tcsh. The
destinatoin machine is Linux.

The general solution is to use find, pipe though tar, pipe to ssh
on the remote machine, change dir and use tar to unpack on the
remote system in a single line.

find /dir -name '*.dat" | tar -cvzf - -T - | ssh remotehost "cd /; tar -xvzf -"

While this should work flawlessly on unix or alike, dunno about
cygwin + doze?

I didn't know stdin to ssh is sent over to remote machine. I've doing
it using netcat.

--
William Park <opengeometry@xxxxxxxx>, Toronto, Canada
ThinFlash: Linux thin-client on USB key (flash) drive
http://home.eol.ca/~parkw/thinflash.html
BashDiff: Super Bash shell
http://freshmeat.net/projects/bashdiff/
.

Relevant Pages

  • Re: Is SSH worth it??
    ... > We would be using SSH and SCP. ... SCP for automated scripts. ... > client will not be prompted for a password. ... remote machine, but imho it is better to swap client+server and give ...
    (Security-Basics)
  • Re: scp data while preserving file modification times?
    ... I can copy the data to my local machine using scp -r. ... *could* tar everything up on the remote side and just download the tar ... file, but the remote machine is shared, and I don't want to hose the ...
    (comp.security.ssh)
  • Re: scping 250,000 files
    ... I would prefer to use scp. ... The general solution is to use find, pipe though tar, pipe to ssh ... on the remote machine, change dir and use tar to unpack on the ...
    (comp.unix.shell)
  • Re: FTP and SSH access question
    ... > remote machine. ... > access via SSH. ... I figure the combination between root FTP ... like 'scp' which can do secure file-copying. ...
    (comp.os.linux.networking)
  • Re: FTP and SSH access question
    ... > remote machine. ... > access via SSH. ... I figure the combination between root FTP ... like 'scp' which can do secure file-copying. ...
    (comp.os.linux.security)