Channel bonding.

From: Sean (seancody_at_gmail.com)
Date: 04/22/05

  • Next message: Eric Anderson: "Re: Channel bonding." 
    Date: Fri, 22 Apr 2005 10:32:14 -0500
To: freebsd-performance@freebsd.org

    I've been experimenting with the idea of doing channel bonding as a
    means of improving the performance of some heavily used file servers.
    Currently I am using a single Intel 1000MT interface on each file
    server and it has rather lack luster performance.

    I've set two ports of my switch to 'shared' (an Extreme
    BlackDiamond 6800) and am using an Intel 1000MT Dual Port for
    the bonding interfaces.

    The performance increase with I see is marginally better than
    just the one interface (70MB/s [bonded] vs 60MB/s [single]) which
    is slightly disappointing. I am using ifstat and iostat (for disk
    throughput, 30MB/s on a 3ware 7500-12 yet again disappointing) to
    monitor and a variant of tcpblast to generate traffic. I'm using
    4 other machines (on the same blade on the switch) to generate the
    traffice to the bonded interface all are similar hardware with
    varrying versions of FreeBSD. In order to get the numbers as high
    as I have I've enabled polling (some stability issues being
    used under SMP).

    Before I dropped everything and moved over to trying out ng_fec
    I wanted to get a few opinions on other things I can check or try.
    These servers typically have anywhere between 20-100 clients reading
    and writing many large files as fast as they can. So far the machines
    only perform well when there are fewer than 20 clients. The whole
    point of the experiment is increase performance of our current
    resources instead of buying more servers. I really don't know
    what to expect (in terms of performance) from this but just based on
    the 'ratings' on the individual parts this machine is not preforming
    very well.

    In case anyone has any ideas I've included the 'specs' of the hardware
    below.

    Hardware:
            Dual Intel Xeon CPU 2.66GHz
            Intel Server SE7501BR2 Motherboard
            2X 512 MB Registered ECC DDR RAM
            3ware 7500-12 (12x120GB, RAID-5)
            Intel PRO/1000 MT Dual Port (em0,1)
            Intel PRO/1000 MT (On board) (em2)
            
    Switch:
            Extreme Black Diamond 6800
            Gigabit Blade: G24T^3 51052

    Kernel:
    FreeBSD phoenix 5.3-RELEASE FreeBSD 5.3-RELEASE #1: Wed Apr 20
    13:33:09 CDT 2005
    root@phoenix.franticfilms.com:/usr/src/sys/i386/compile/SMP i386

    Channel Bonding commands used:
    ifconfig em0 up
    ifconfig em1 up
    kldload ng_ether.ko
    ngctl mkpeer em0: one2many upper one
    ngctl connect em0: em0:upper lower many0
    ngctl connect em1: em0:upper lower many1
    echo Allow em1 to xmit/recv em0 frames
    ngctl msg em1: setpromisc 1
    ngctl msg em1: setautosrc 0
    ngctl msg em0:upper setconfig "{ xmitAlg=1 failAlg=1 enabledLinks=[ 1 1 ] }"
    ifconfig em0 A.B.C.D netmask 255.255.255.0

    Contents of /etc/sysctl.conf:
    net.inet.tcp.inflight_enable=1
    net.inet.tcp.sendspace=32767
    net.inet.tcp.recvspace=32767
    net.inet.tcp.delayed_ack=0
    vfs.hirunningspace=10485760
    vfs.lorunningspace=10485760
    net.inet.tcp.local_slowstart_flightsize=32767
    net.inet.tcp.rfc1323=1
    kern.maxfilesperproc=2048
    vfs.vmiodirenable=1
    kern.ipc.somaxconn=4096
    kern.maxfiles=65536
    kern.polling.enable=1 

    -- 
Sean
_______________________________________________
freebsd-performance@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"
  • Next message: Eric Anderson: "Re: Channel bonding."