Re: automatic compression over communication channel ?

From: Pascal Bourguignon (spam_at_thalassa.informatimago.com)
Date: 01/29/04


Date: 29 Jan 2004 21:10:37 +0100


"Vu Pham" <vu@sivell.com> writes:

> I am sorry if this question is not appropriate for this forum.
>
> In my client/server app, I compress the data at one end and the other will
> decompress it. To do this, I must implement the zlib api at the
> "application" level : I know in advance how long this transaction will be,
> then I compress it, add a flag and some parameters to notify the other end
> about this compression ...For apps that currently do not have the
> compression, I need to change a lot of codes..
>
> I wonder if there is any way I can implement the compression at a lower
> level - somewhere below my application level and above the tcp level - so
> that I just need to call, say zsend() or zrecv() and these functions will do
> the compression/decompression. stuff.

An easy way to add compression is to use a ppp/ssh tunnel.

To establish such a ppp/ssh tunnel you may use a script including this:
(Edit it to inhib encryption and tune the compression level to fit your needs).

REMOTE_HOST=example.com
REMOTE_USER=example
LOCAL_USER=example

# proxyarp Does not work.
COMMON_OPT="
    local
    nocrtscts
    asyncmap 0
    bsdcomp 12,12
    nodeflate
    noauth
    noipdefault
    proxyarp ktune
"
CALLER_OPT="
    persist
    defaultroute
"
PROTO_OPT="
    lcp-echo-failure 4
    lcp-echo-interval 45
    lcp-max-configure 20
    lcp-max-failure 10
    lcp-max-terminate 3
    lcp-restart 3
    ipcp-accept-local
    ipcp-accept-remote
    ipcp-max-configure 10
    ipcp-max-failure 10
    ipcp-max-terminate 3
    ipcp-restart 3
"
PPP_OPTIONS="$COMMON_OPT $CALLER_OPT $PROTO_OPT"

SSH_OPTIONS=$( echo "ssh $SSH_DEBUG -2 -t -x -a -q -C \
    -i ~${LOCAL_USER}/.ssh/id_dsa \
    -e none \
    -o 'CompressionLevel 2' \
    -o 'Batchmode yes' \
    -c blowfish \
    -l ${REMOTE_USER} ${REMOTE_HOST}" | sed -e 's/ */ /g' )

pppd $PPP_DEBUG $PPP_OPTIONS connect-delay 20000 pty "$SSH_OPTIONS"

On the remote host, you must set the REMOTE_USER account such:

example:x:0:1600:VPN:/home/example:/local/sbin/vpn-example

with the vpn-example program:
---(pppsh.c)------------------------------------------------------------
/******************************************************************************
FILE: pppsh.c
LANGUAGE: ANSI-C
SYSTEM: UNIX
USER-INTERFACE: None.
DESCRIPTION

    This program encapsulate pppd passing it a compiled-in option file
    argument.

USAGE

    Compile with -DPPP_OPTION_FILE=\"/etc/ppp/peers/NAME\" \
                 -o /usr/local/sbin/ppp-NAME

    Add an entry in /etc/passwd :
      NAME:password:uid:gid:PPP for NAME:/home/NAME:/usr/local/sbin/ppp-NAME

AUTHORS
    <PJB> Pascal J. Bourguignon
MODIFICATIONS
    2001-12-19 <PJB> Creation.
BUGS
LEGAL
    GPL
    Copyright Pascal J. Bourguignon 2001 - 2001

    This program is free software; you can redistribute it and/or
    modify it under the terms of the GNU General Public License as
    published by the Free Software Foundation; either version 2 of the
    License, or (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
    General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with this program; see the file COPYING; if not, write to
    the Free Software Foundation, Inc., 59 Temple Place, Suite 330,
    Boston, MA 02111-1307 USA

******************************************************************************/
#include <sysexits.h>
#include <unistd.h>

    static const char rcsid[]="$Id$";
    
#ifndef PPPD
#define PPPD "/usr/sbin/pppd"
#endif

#ifndef PPP_OPTION_FILE
#error "Please define PPP_OPTION_FILE"
#endif

int main(int argc,char** argv)
{
    char** environment[]={0};
    execle(PPPD,PPPD,"file",PPP_OPTION_FILE,0,environment);
    perror("pppsh");
    exit(EX_OSFILE);
    return(EX_OSFILE);
}/*main*/

----(makefile)----------------------------------------------------------
all:
        @[ `hostname -s` = host1 ] && localip=192.168.0.1 ./make-peers || true
        @[ `hostname -s` = host2 ] && localip=192.168.0.2 ./make-peers || true

users:
        ./make-users

install:all
        @[ `hostname -s` = host1 ] && localip=192.168.0.1 ./make-peers install || true
        @[ `hostname -s` = host2 ] && localip=192.168.0.2 ./make-peers install || true
clean:
        rm -f vpn-* *.o *~
---(make-peers)---------------------------------------------------------
#!/bin/bash
install=0
for arg ; do
    case "$arg" in
    install|--install|-i) install=1 ;;
    *) ;;
    esac
done

PPPD=/usr/sbin/pppd

for ip_peer in \
    192.168.0.1-host1 \
    192.168.0.2-host2 \
do
    remoteip=${ip_peer/-*}
    peer=${ip_peer/*-}
    PPP_OPTION_FILE=/etc/ppp/peers/vpn-${peer}
    if [ "$remoteip" = "$localip" ] ; then
        continue
    fi
    if [ $install -eq 0 ] ; then
        echo "Generating VPN link from $localip to $remoteip ($peer)."
        cat > vpn-${peer}.ppp <<EOF
ipparam vpn-${peer}
${localip}:${remoteip}
netmask 255.255.255.255
proxyarp
noauth
asyncmap 0
nocrtscts
passive
local
bsdcomp 12,12
deflate 12,12
lcp-echo-interval 15
lcp-echo-failure 8
EOF
        gcc \
            -o vpn-${peer} pppsh.c \
            -DPPP_OPTION_FILE=\"${PPP_OPTION_FILE}\" \
            -DPPPD=\"${PPPD}\"
    else
        echo "Installing VPN link to $remoteip ($peer)."
        install -g vpnusers -m 644 vpn-${peer}.ppp ${PPP_OPTION_FILE}
        install -g vpnusers -m 750 vpn-${peer} /local/sbin/vpn-${peer}
    fi
done
---(/etc/ppp/ip-up.local)-----------------------------------------------
#!/bin/bash
BASENAME=${0##*/}
INTERFACE=$1
DEVICE=$2
SPEED=$3
LOCALIP=$4
REMOTEIP=$5

echo "$@" >> /tmp/ip-up.local.log

case "$INTERFACE" in
ppp*)
    route add -net 195.114.85.128/25 metric 3 "$INTERFACE"
    route add -net 10.0.0.0/8 metric 3 "$INTERFACE"
    ;;
esac
------------------------------------------------------------------------

-- 
__Pascal_Bourguignon__                     http://www.informatimago.com/
There is no worse tyranny than to force a man to pay for what he doesn't
want merely because you think it would be good for him.--Robert Heinlein
http://www.theadvocates.org/


Relevant Pages

  • Re: automatic compression over communication channel ?
    ... > In my client/server app, I compress the data at one end and the other will ... > compression, I need to change a lot of codes.. ... AFAIK there's no standard protocol for this. ... could design an architecture analogous to SSL. ...
    (comp.unix.programmer)
  • automatic compression over communication channel ?
    ... I am sorry if this question is not appropriate for this forum. ... In my client/server app, I compress the data at one end and the other will ... compression, I need to change a lot of codes.. ... that I just need to call, say zsendor zrecv() and these functions will do ...
    (comp.unix.programmer)
  • Re: [RFC] LZO de/compression support - take 3
    ... On 5/23/07, Nitin Gupta wrote: ... This file is part of the LZO real-time data compression library. ... The LZO library is free software; ... GNU General Public License for more details. ...
    (Linux-Kernel)
  • [RFC] LZO de/compression support - take 3
    ... This file is part of the LZO real-time data compression library. ... The LZO library is free software; ... GNU General Public License for more details. ...
    (Linux-Kernel)
  • [RFC] LZO de/compression support - take 6
    ... Following compares this kernel port vs original miniLZO code: ... Compression: 41.8412 usec ... The LZO library is free software; ... GNU General Public License for more details. ...
    (Linux-Kernel)