Re: scp more perfectly fills the pipe than NFS/TCP

In the last episode (Dec 21), Zaphod Beeblebrox said:
On Sun, Dec 20, 2009 at 12:27 AM, Dan Nelson <dnelson@xxxxxxxxxxxxxxx> wrote:
In the last episode (Dec 19), Zaphod Beeblebrox said:
Here's an interesting conundrum.  I don't know what's different between
the TCP that scp uses from the TCP that NFS uses, but given the same
two FreeBSD machines, SCP fills the pipe with packets better.

Examine the following graphic: http://www.eicat.ca/~dgilbert/example-mrtg.png

The system doing the scp and the NFS server is FreeBSD-7.2-p1.  The
system receiving the scp and the NFS client is FreeBSD-8.0-p1

The scp transfer is the left hand side of the graph and the NFS
transfer is on the right.

The NFS is mounted with "-3 -T -b -l -i" and no other options.  Files
are being moved over NFS with the system "mv" command.  The files in
each case are large (50 to 500 meg files).

If you increase the NFS blocksize (-r 32768 for example) you will get
slightly better performance, but you will likely never match the scp
results.  They're doing two different things under the hood: scp is
streaming the entire file in one operation, while NFS is performing many
"read 8k at offset 0", "read 8k at offset 8k", etc requests one after
another, so a high-latency connection will take a performance hit due to
the latency in issuing each command.  According to the mount_nfs
manpage, it looks like there is some prefetching that can be enabled
with the "-a ##" option.  It doesn't say what the default is, though.

While the link is slow, it is really directly connected with a latency
of 10ms or so. Isn't mv mmap()'ing large enough regions to cause
there to be a reasonable queue to transfer?

I've never been impressed with FreeBSD's ability to detect sequential read
patterns and prefetch blocks ahead of time, even on local ufs filesystems.

--
Dan Nelson
dnelson@xxxxxxxxxxxxxxx
_______________________________________________
freebsd-hackers@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: NFS write() calls lead to read() calls?
    ... If I run the scp again, I can see X MB/s going out from HostA, 2*X ... MB/s coming in on HostB and X MB/s out plus X MB/s in on HostC. ... If I rmthe file on the NFS share, then the first scpwill not ... MAXBSIZE = 64K for large regular files and all non-regular files. ...
    (freebsd-net)
  • Re: NFS write() calls lead to read() calls?
    ... If I run the scp again, I can see X MB/s going out from HostA, 2*X ... MB/s coming in on HostB and X MB/s out plus X MB/s in on HostC. ... If I rmthe file on the NFS share, then the first scpwill not ... MAXBSIZE = 64K for large regular files and all non-regular files. ...
    (freebsd-current)
  • scp more perfectly fills the pipe than NFS/TCP
    ... between the TCP that scp uses from the TCP that NFS uses, ... SCP fills the pipe with packets better. ... The system doing the scp and the NFS server is FreeBSD-7.2-p1. ... The connection is a DSL that terminates on the local lan near the ...
    (freebsd-hackers)
  • Re: NFS Buffering
    ... The problem is that after scp reports that it's 100% complete the ... It looks like the nfs daemon is caching write data ... sitting in a write buffer. ... buffering is done by the nfsiod. ...
    (Fedora)
  • Re: scp more perfectly fills the pipe than NFS/TCP
    ... the TCP that scp uses from the TCP that NFS uses, ... SCP fills the pipe with packets better. ... The system doing the scp and the NFS server is FreeBSD-7.2-p1. ... being moved over NFS with the system "mv" command. ...
    (freebsd-hackers)