Re: Traffic Shaping not working correctly after ipfw coverted to use pfil_hooks API

• Previous message: Giorgos Keramidas: "Re: error at usr.bin/ftp when building 5.3RC1 world"
• In reply to: Vincent Poy: "Re: Traffic Shaping not working correctly after ipfw coverted to use pfil_hooks API"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Date: Thu, 28 Oct 2004 19:07:05 -0700
To: Brian Fundakowski Feldman <green@freebsd.org>

On Thu, 28 Oct 2004 17:19:43 -0700, Vincent Poy <vincepoy@gmail.com> wrote:
> On Tue, 26 Oct 2004 19:17:46 -0400, Brian Fundakowski Feldman
> <green@freebsd.org> wrote:
> > On Tue, Oct 26, 2004 at 01:30:43PM -0700, Luigi Rizzo wrote:
> >
> >
> > > On Sun, Oct 24, 2004 at 04:37:32PM +0200, Andre Oppermann wrote:
> > > > [bouncing over to Luigi]
> > > >
> > > > Luigi, do you have any idea what might be going wrong here?
> > >
> > > no, sorry... I have to say the ipfw/natd/dummynet configuration is rather
> > > convoluted here so it is a bit hard to tell whether the
> > > problem is in dummynet calls or divert sockets.
> > >
> > > I am also confused by the numbers in the initial report:
> > >
> > > > > > >>Vincent Poy wrote:
> > > > > > >>
> > > > > > >>>However, after the latest -CURRENT upgrade, it will do 200KB/sec down
> > > > > > >>>and 52KB/sec up. If I only download only, then it does show
> > > > > > >>>650KB/sec. Normally, when I change the bandwidth to a number lower
> > > > > > >>>than 480Kbps for the pipe, the download speeds would go up when
> > > > > > >>>downloading. However, I have tried in 10kbps steps down to 350kbps
> > > > > > >>>but it still did not top 200KB/sec in downloading.
> > >
> > > there is a mix of two different notations, Kbps and KB/sec, and
> > > i cannot make sense of them.
> > > Finally, I am curious as to why one would mix the upload and download
> > > traffic, i believe *DSL data rates are independent in the two
> > > directions unlike analog modems...
> >
> > In ip_divert.c the behavior may have changed when I implemented diverted
> > rules, in divert_output(): the cookie (ipfw rule number) can be added
> > and set to sin->sin_port if the sockaddr_in used for hinting the
> > interface has a cleared sin->sin_addr field, but has not cleared the
> > sin->sin_port. This specifically means for diverted output, and not
> > loopback, packets could be skipping rules when they shouldn't be.
> >
> > Could you try modifying that function in src/sys/netinet/ip_divert.c
> > so that "dt->cookie = sin->sin_port;" changes to
> > if (sin->sin_addr.s_addr != 0)
> > dt->cookie = sin->sin_port;
> > and see if that changes things?
>
> Before I did the modifying of the function, I updated to the latest
> -CURRENT of 1AM PDT October 27, 2004 and this is what I did:
>
> *** /usr/src/sys/netinet/ip_divert.c.old Wed Oct 27 00:30:59 2004
> --- /usr/src/sys/netinet/ip_divert.c Thu Oct 28 15:46:25 2004
> ***************
> *** 297,303 ****
> if (sin) {
> int i;
>
> ! dt->cookie = sin->sin_port;
> /*
> * Find receive interface with the given name, stuffed
> * (if it exists) in the sin_zero[] field.
> --- 297,304 ----
> if (sin) {
> int i;
>
> ! if (sin->sin_addr.s_addr != 0)
> ! dt->cookie = sin->sin_port;
> /*
> * Find receive interface with the given name, stuffed
> * (if it exists) in the sin_zero[] field.
>
> Recompiled kernel and rebooted.
>
> As I mentioned before, the ADSL circuit is 6016Kbps/608Kbps on SBC's
> ATM Network so there is a 13% overhead due to ATM making it
> 5000Kbps/520Kbps effective.
>
> ipfw rules are as follows as it's been simplified, I am only using
> ipfw2/dummynet so that ACK packets will have priority if I am
> downloading while uploading. I'm using 400Kbit/s as the upload limit
> so that downloads will be at their maximum while uploading.
>
> ${fwcmd} enable one_pass
> # Define our upload pipe
> ${fwcmd} pipe 1 config bw 400Kbit/s
> ${fwcmd} add 49 skipto 100 ip from 208.201.244.224/29 to any
> ${fwcmd} add 50 divert natd all from any to any via ${natd_interface}
> ${fwcmd} add 100 pass all from any to any via lo0
> ${fwcmd} add 200 deny all from any to 127.0.0.0/8
> ${fwcmd} add 300 deny ip from 127.0.0.0/8 to any
> # Route LAN and RFC1918 networks without Traffic Shaping
> ${fwcmd} add 63000 allow all from any to 10.0.0.0/8 out
> ${fwcmd} add 63001 allow all from any to 172.16.0.0/12 out
> ${fwcmd} add 63002 allow all from any to 192.168.0.0/16 out
> ${fwcmd} add 63003 allow all from any to 208.201.244.224/29 out
> # Assign outgoing empty/small ACK packets to the high-priority queue
> ${fwcmd} add 63004 set 0 queue 1 tcp from any to any tcpflags ack out
> # Assign outgoing UDP (DNS/gaming) and SSH traffic to the
> medium-high-priority queue
> ${fwcmd} add 63005 set 0 queue 2 tcp from any to any 22,23 out
> ${fwcmd} add 63006 set 0 queue 2 udp from any to any not 80,443 out
> # Assign outgoing HTTP/HTTPS WEB traffic to the medium-low-priority queue
> ${fwcmd} add 63007 set 0 queue 3 all from any to any 80,443 out
> # Assign all other outgoing traffic to the low-priority queue
> ${fwcmd} add 63008 set 0 queue 4 all from any to any out
> And ofcourse the default open rules of:
> ${fwcmd} add 65000 allow ip from any to any
> ${fwcmd} add 65535 deny ip from any to any
>
> The ftp tests I am doing as follows is directly between me and my
> ISP's ftp server:
>
> traceroute to ftp.sonic.net (208.201.249.234), 64 hops max, 40 byte packets
> 1 adsl-208-201-244-1.sonic.net (208.201.244.1) 8.201 ms 8.162 ms 8.260 ms
> 2 fast0-1.dist2-1.sr.sonic.net (208.201.224.160) 8.873 ms 8.749 ms 9.525 ms
> 3 eth0.a.ftp.sonic.net (208.201.249.234) 8.625 ms 9.287 ms 8.104 ms
>
> The results were the same with and without the change that Brian had
> mentioned above.
>
> This is what the maximum speed of the download/upload pipes. The upload pipe
> is limited at 400Kbit/sec due to the ipfw pipe config.
>
> Upload only:
> 10485760 bytes sent in 03:40 (46.37 KB/s)
> Download only:
> 10485760 bytes received in 00:16 (616.48 KB/s)
>
> Here is where the problem is if I do a upload and download at the same time:
>
> Upload:
> 10485760 bytes sent in 04:01 (42.41 KB/s)
> Download:
> 10485760 bytes received in 00:59 (172.91 KB/s)
>
> It appears that somehow the ACKs were not sent with priority on the
> upload pipe while downloading which is slowing the download side to
> less than 33% speed of the pipe.
>
> Here are the ipfw pipe and queue's:
>
> root@bigbang [5:11pm][/home/vince] >> ipfw pipe show
> 00001: 400.000 Kbit/s 0 ms 50 sl. 0 queues (1 buckets) droptail
> mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
> q00001: weight 100 pipe 1 50 sl. 1 queues (1 buckets) droptail
> mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
> BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
> 0 tcp 208.201.244.226/1449 64.12.185.119/80 96816 57368081 39 13603 311
> q00002: weight 66 pipe 1 50 sl. 1 queues (1 buckets) droptail
> mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
> BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
> 0 udp 208.201.244.225/1026 192.35.51.30/53 1092 98426 0 0 0
> q00003: weight 33 pipe 1 50 sl. 1 queues (1 buckets) droptail
> mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
> BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
> 0 tcp 208.201.244.226/1449 64.12.185.119/80 396 20592 0 0 0
> q00004: weight 1 pipe 1 50 sl. 1 queues (1 buckets) droptail
> mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
> BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
> 0 icmp 208.201.244.225/0 80.138.73.42/0 235 15900 0 0 0
>
> root@bigbang [5:14pm][/home/vince] >> ipfw queue show
> q00001: weight 100 pipe 1 50 sl. 1 queues (1 buckets) droptail
> mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
> BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
> 0 tcp 208.201.244.226/1449 64.12.185.119/80 98271 58076455 29 35976 324
> q00002: weight 66 pipe 1 50 sl. 1 queues (1 buckets) droptail
> mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
> BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
> 0 udp 208.201.244.225/1026 192.35.51.30/53 1105 99511 0 0 0
> q00003: weight 33 pipe 1 50 sl. 1 queues (1 buckets) droptail
> mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
> BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
> 0 tcp 208.201.244.226/1449 64.12.185.119/80 396 20592 0 0 0
> q00004: weight 1 pipe 1 50 sl. 1 queues (1 buckets) droptail
> mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
> BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
> 0 icmp 208.201.244.225/0 80.138.73.42/0 237 16045 0 0 0
>
> Which seems like it's sending everything through queue 1 when
> uploading/downloading instead of ACKs only on queue 1 and the other
> data on queue 4.
>
> Hope this explains things better for Luigi.

I just tried this again.... and it seems like the queues are not
working at all while it did on the March 6, 2004 -CURRENT since the
queue show output appears to not update the source and destination IPs
for all of the queues.

root@bigbang [6:44pm][/home/vince] >> ipfw queue show
q00001: weight 100 pipe 1 50 sl. 1 queues (1 buckets) droptail
    mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 tcp 208.201.244.226/1449 64.12.185.119/80 181047 91968211 0 0 421
q00002: weight 66 pipe 1 50 sl. 1 queues (1 buckets) droptail
    mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 udp 208.201.244.225/1026 192.35.51.30/53 2727 238253 0 0 0
q00003: weight 33 pipe 1 50 sl. 1 queues (1 buckets) droptail
    mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 tcp 208.201.244.226/1449 64.12.185.119/80 2211 114696 0 0 0
q00004: weight 1 pipe 1 50 sl. 1 queues (1 buckets) droptail
    mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 icmp 208.201.244.225/0 80.138.73.42/0 1847 148609 0 0 0

Queue 1 I tested by doing a ftp receive from 208.201.249.234 to
208.201.244.224 which is updating the total pkt/bytes counter but the
Source IP and Destination IP didn't change.

Queue 2 I tested by querying my DNS server from an outside source.
The results were similar with 1 as the total pkt/bytes counter did
update but the Source IP and Destination IP didn't change.

Queue 3 I tested by downloading a file via http from an outside
source. None of the information changed at all as the Source IP and
Destination IP didn't change. The total pkt/bytes counter didn't
change at all.

Queue 4 I tested by sending a file via ftp. The counters change but
the protocol
shows icmp and with the same Source IP and Destination IP as shown above.

Hope this provides more information.

Cheers,
Vince
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

• Previous message: Giorgos Keramidas: "Re: error at usr.bin/ftp when building 5.3RC1 world"
• In reply to: Vincent Poy: "Re: Traffic Shaping not working correctly after ipfw coverted to use pfil_hooks API"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]