Re: Throughput problems with dummynet and high delays

Andy:

Hi, you must be working with Injong :-)

At one point I was playing with dummynet in satellite networks.
One of the KEY problems I found is that the number of
packets you can have in queue, limited to 100, was not
nearly enough. This was (at the time) a hardcoded parameter
inside both the ipfw code as well as the dummynet code.

What I did is changed this to be a #define inside the
ip_dummynet.h file.. What this then allowed was to set
the value up a bit higher... I used 1200 for my 550ms+
sat network...

With just rough math you need about 89,478 - 1500 bytes
packets per second to get a gig a bit.

So thats about 894 per 100ms.. but I would want more than that I am
thinking... Of course you might be able to reduce that
with 9000 byte mtus... but even at 9000 byte mtu
100 packets will NOT cut it...

My old patches were from way back in the 4.10 erra.. I could
dig around and see if I can find something.. but I would
be able to only give you something for:

6.1 or 7.0... I don't have a 6.0 machine around anywhere ..
and soon I will be moving my 6.1 -> 6.2 :-)

Let me know if you want me to poke around.. or of course
you could too.. its not a hard change to make :-)

R



Andy Jones wrote:
Hi,

I'm a researcher at the University of North Carolina trying to simulate
certain link characteristics using dummynet pipes in ipfw. Our end goal is
to thoroughly test high speed TCP variants in our experimental network in a
wide range of situations (which includes varying the delay from 1ms to
200ms).

I have two Dell PowerEdge 2850 servers connected to each other using an
Intel Gigabit ethernet card (although I'm not sure of the exact model). They
both run FreeBSD 6.0. I'm using iperf to push as many bits through the wire
as possible.

Without dummynet, sustained throughput is as expected, close to 1Gbps
[ 3] 0.0-180.0 sec 19.2 GBytes 918 Mbits/sec

When dummynet is used to add delay (100ms in my case) to the network, the
machines have problems sustaining high throughput.

Here are the setup on the receiver end
% sysctl kern.ipc.maxsockbuf=16M
% sysctl net.inet.tcp.recvspace=12MB
% iperf -s

and on the sender end
% sysctl kern.ipc.maxsockbuf=16M
% sysctl net.inet.tcp.sendspace=12MB
% ipfw pipe 1 config delay 100
% ipfw add 10 pipe 1 ip from any to any out
% iperf -c <machine> [args ...]

kern.ipc.nmbclusters has also been tuned to 65536 at boot time. Our kernel
is also has HZ=1000. The ipfw rule is added such that it is the first rule
in the chain. 12MB is about the right size send buffer for the
bandwidth-delay product (1Gbps * 0.1 RTT / 8bits/byte). We're also using an
MTU of roughly 9000 bytes.

What happens is as the TCP window grows larger (about 3-4MB), the sender
spends most of its time processing interrupts (80-90% as reported by top)
and throughput peaks at about 300Mbps. I've dug into the dummynet code and
I've found that a large amount of time is spent in the routine
transmit_event(struct dn_pipe *p) which dequeues packets from a pipe and
calls ip_output. It appears that ip_output is the culprit, but what it is
doing with its time, I'm not sure. Packet drops are not being lost according
to TCP and dummynet. I suspect either pfil_run_hooks(...) or (*
ifp->if_output) (...) calls in ip_output are taking too much time, but I'm
not sure.

Any suggestions on what could be happening would be appreciated!

-Andy Jones
_______________________________________________
freebsd-net@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@xxxxxxxxxxx"



--
Randall Stewart
NSSTG - Cisco Systems Inc.
803-345-0369 <or> 803-317-4952 (cell)
_______________________________________________
freebsd-net@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: Best way to do traffic shaping?
    ... Dummynet is a tool that has been a BSD staple for a long time now. ... Originally it was developed to allow for simulation of various network ... into a way to control the packets flowing through a server. ...
    (Ubuntu)
  • IPFILTER byte/packet counting
    ... Using IPFILTER with a bridge, can ipf count packets and bytes going ... I see things like dummynet (which only works with ... ipfw?). ...
    (FreeBSD-Security)
  • Re: Dummynet fragmenting packets
    ... pass the packets between the two NICs without routing through a dummynet ... so the packet must be broken up in order to pass it on to the network ... Dummynet has the wrong information about what the MTUs are on your ...
    (freebsd-questions)
  • dummynet setting ifp pointer in mbuf?
    ... I am having problems combining ipf's ipnat rules with dummynet. ... outgoing packets as well). ... queue 2 tcp from any 22 to any out xmit if0 ... So it seems the queued packets interface is set to the outgoing interface. ...
    (freebsd-net)
  • Incompatibility between dummynet and PF rdr.
    ... Are there any known compatibility problems between dummynet and PF rdr rules? ... When I try to combine both, the packets seem to simply disappear. ... | pfctl -f - ... kldload ipfw; ipfw add 65000 allow ip from any to any ...
    (freebsd-net)