Network pipes

From: Diomidis Spinellis (dds_at_aueb.gr)
Date: 07/24/03

  • Next message: Luigi Rizzo: "Re: Network pipes" 
    Date: Thu, 24 Jul 2003 11:19:49 +0300
To: freebsd-hackers@freebsd.org

    I am currently testing a set of modifications to /bin/sh that allow a
    user to create a pipeline over the network using a socket as its
    endpoints. Currently a command like

    tar cvf - / | ssh remotehost dd of=/dev/st0 bs=32k

    has tar sending each block through a pipe to a local ssh process, ssh
    communicating through a socket with a remote ssh daemon and dd
    communicating with sshd through a pipe again. The changed shell allows
    you to write

    tar cvf - / |@ ssh remotehost -- dd of=/dev/st0 bs=32k | :

    The effect of the above command is that a socket is created between the
    local and the remote host with the standard output of tar and the
    standard input of dd redirected to that socket. Authentication is still
    performed using ssh (or any other remote login mechanism you specify
    before the -- argument), but the flow between the two processes is from
    then on not protected in terms of integrity and privacy. Thus the
    method will mostly be useful within the context of a LAN or a VPN.

    The authentication design requires the users to have a special command
    in their path on the remote host, but does not require an additional
    privileged server or the reservation of special ports.

    By eliminating two processes, the associated context switches, the data
    copying, and (in the case of ssh) encryption performance is markedly
    improved:

    dd if=/dev/zero bs=4k count=8192 | ssh remotehost -- dd of=/dev/null
    33554432 bytes transferred in 17.118648 secs (1960110 bytes/sec)
    dd if=/dev/zero bs=4k count=8192 |@ ssh remotehost -- dd of=/dev/null |
    :
    33554432 bytes transferred in 4.452980 secs (7535276 bytes/sec)

    Even eliminating the encryption overhead by using rsh you can still see

    dd if=/dev/zero bs=4k count=8192 | rsh remotehost -- dd of=/dev/null
    33554432 bytes transferred in 131.907130 secs (254379 bytes/sec)
    dd if=/dev/zero bs=4k count=8192 |@ rsh remotehost -- dd of=/dev/null |
    :
    33554432 bytes transferred in 86.545385 secs (387709 bytes/sec)

    My questions are:

    1. How do you feel about integrating these changes to the /bin/sh in
    -CURRENT? Note that network pipes are a different process plumbing
    mechanism, so they really do belong to a shell; implementing them
    through a separate command would be inelegant.

    2. Do you see any problems with the new syntax introduced?

    3. After the remote process starts running standard error output is
    lost. Do find this a significant problem?

    4. Both sides of the remote process are communication endpoints and have
    to be connected to other local processes via pipes. Would it be enough
    to document this behaviour or should it be hidden from the user by means
    of forked read/write processes?

    Diomidis - http://www.spinellis.gr
    _______________________________________________
    freebsd-hackers@freebsd.org mailing list
    http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
    To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

  • Next message: Luigi Rizzo: "Re: Network pipes"

    Relevant Pages