1168 octets payload and bad TCP checksums

From: Kelly Yancey (kelly_at_nttmcl.com)
Date: 01/03/04

  • Next message: Gleb Smirnoff: "Re: <name_comp> inetd[<some_num>]: warning: can't get client address: Connection reset by peer" 
    Date: Fri, 2 Jan 2004 19:27:08 -0800 (PST)
To: net@freebsd.org

      We've got Broadcom BCM5701 cards configured for vlan tagging on a
    FreeBSD 4.7 based router; a vlan switch then terminates the trunked
    segment and splits it into separate physical subnets. It turns out that
    hosts on those segments cannot receive TCP packets with precisely 1168
    octets of payload (ethernet frame size 1222 octets) as the checksum is
    always incorrect. We've manually backported all of the bge driver updates
    from 4-stable, but to no avail.
      What is particularly odd is that the checksums are always wrong by the
    same amount: 0xAC48 (the dump below only shows retries of the same
    packet, but the difference is the same even for other packets).
    Furthermore, it appears only TCP packets with 1168 octets of data are
    affected. I cannot easily create an environment without the vlans to
    determine whether or not tagging is related. Note also, that the IP
    checksum is correct.

      Has anyone else experienced similar problems? Does anyone have a clue
    where to begin to track down the problem? Currently I'm looking at the
    tcp checksum calculation (tcp_fillheaders), but I don't really see how
    that could be the culprit as if such a bug existed there, it would affect
    all interfaces and surely would have been noticed by now. At the same
    time, I don't see anywhere else offhand the problem could be. Again, if
    anyone has any advice, I would greatly appreciate it. Thanks,

      Kelly

    bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1504
            options=3<rxcsum,txcsum>
            inet 10.30.3.254 netmask 0xfffffff8 broadcast 10.30.3.255
            ether 00:00:5e:00:01:4b
            media: Ethernet autoselect (100baseTX <full-duplex>)
            status: active
    vlan9: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
            inet 10.30.3.1 netmask 0xfffffff8 broadcast 10.30.3.7
            ether 00:00:5e:00:01:4b
            vlan: 9 parent interface: bge0
    vlan10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
            inet 10.30.3.9 netmask 0xfffffff8 broadcast 10.30.3.15
            ether 00:00:5e:00:01:4b
            vlan: 10 parent interface: bge0

      Extract from tcpdump -vvv taken on host 216.69.90.56 connected to
    FreeBSD router via vlan10 interface:

    11:38:55.665425 216.69.68.198.22 > 216.69.90.56.3335: . [tcp sum ok] 561:2021(1460) ack 432 win 14352 (DF) [tos 0x10] (ttl 59, id 57881, len 1500)
    11:38:55.666782 216.69.68.198.22 > 216.69.90.56.3335: P [tcp sum ok] 2021:2049(28) ack 432 win 14352 (DF) [tos 0x10] (ttl 59, id 57882, len 68)
    11:38:55.666839 216.69.90.56.3335 > 216.69.68.198.22: . [tcp sum ok] 432:432(0) ack 2049 win 17520 (DF) (ttl 128, id 57057, len 40)
    11:38:55.668899 216.69.68.198.22 > 216.69.90.56.3335: P [bad tcp cksum 1de3!] 2049:3217(1168) ack 432 win 14352 (DF) [tos 0x10] (ttl 59, id 57883, len 1208)
    11:38:55.920110 216.69.68.198.22 > 216.69.90.56.3335: P [bad tcp cksum 1de3!] 2049:3217(1168) ack 432 win 14352 (DF) [tos 0x10] (ttl 59, id 57884, len 1208)
    11:38:56.419788 216.69.68.198.22 > 216.69.90.56.3335: P [bad tcp cksum 1de3!] 2049:3217(1168) ack 432 win 14352 (DF) [tos 0x10] (ttl 59, id 57885, len 1208)
    11:38:56.442824 216.69.224.134 > 216.69.90.56: icmp: echo request (ttl 108, id 24195, len 92)
    11:38:57.419622 216.69.68.198.22 > 216.69.90.56.3335: P [bad tcp cksum 1de3!] 2049:3217(1168) ack 432 win 14352 (DF) [tos 0x10] (ttl 59, id 57886, len 1208)
    11:38:58.098535 216.69.90.56.3337 > 216.69.68.197.53: [udp sum ok] 12575+ PTR? 56.90.69.216.in-addr.arpa. (43) (ttl 128, id 57060, len 71)
    11:38:58.098868 216.69.90.56.3337 > 216.69.68.197.53: [udp sum ok] 12576+ PTR? 1.90.69.216.in-addr.arpa. (42) (ttl 128, id 57061, len 70)
    11:38:58.102453 216.69.68.197.53 > 216.69.90.56.3337: [udp sum ok] 12575 NXDomain* q: PTR? 56.90.69.216.in-addr.arpa. 0/1/0 ns: 90.69.216.in-addr.arpa. SOA ns.nttmcl.com. hostmaster.nttmcl.com. 2002111000 7200 3600 1209600 432000 (103) (ttl 59, id 43147, len 131)
    11:38:58.103689 216.69.68.197.53 > 216.69.90.56.3337: [udp sum ok] 12576 NXDomain* q: PTR? 1.90.69.216.in-addr.arpa. 0/1/0 ns: 90.69.216.in-addr.arpa. SOA ns.nttmcl.com. hostmaster.nttmcl.com. 2002111000 7200 3600 1209600 432000 (102) (ttl 59, id 63562, len 130)
    11:38:59.419902 216.69.68.198.22 > 216.69.90.56.3335: P [bad tcp cksum 1de3!] 2049:3217(1168) ack 432 win 14352 (DF) [tos 0x10] (ttl 59, id 57887, len 1208)
    11:39:03.419776 216.69.68.198.22 > 216.69.90.56.3335: P [bad tcp cksum 1de3!] 2049:3217(1168) ack 432 win 14352 (DF) [tos 0x10] (ttl 59, id 57888, len 1208)
    11:39:06.305954 216.69.90.56.3335 > 216.69.68.198.22: P [tcp sum ok] 432:480(48) ack 2049 win 17520 (DF) (ttl 128, id 57062, len 88)
    11:39:06.344820 216.69.68.198.22 > 216.69.90.56.3335: . [tcp sum ok] 3217:3217(0) ack 480 win 14352 (DF) [tos 0x10] (ttl 59, id 57889, len 40)
    11:39:07.031807 216.69.90.56.3335 > 216.69.68.198.22: P [tcp sum ok] 480:528(48) ack 2049 win 17520 (DF) (ttl 128, id 57065, len 88)
    11:39:07.035322 216.69.68.198.22 > 216.69.90.56.3335: . [tcp sum ok] 3217:3217(0) ack 528 win 14352 (DF) [tos 0x10] (ttl 59, id 57890, len 40)

    _______________________________________________
    freebsd-net@freebsd.org mailing list
    http://lists.freebsd.org/mailman/listinfo/freebsd-net
    To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"

  • Next message: Gleb Smirnoff: "Re: <name_comp> inetd[<some_num>]: warning: can't get client address: Connection reset by peer"

    Relevant Pages

    • bge data corruption bug (was: 1168 octets payload and bad TCPchecksums)
      ... a vlan switch then terminates the trunked ... > hosts on those segments cannot receive TCP packets with precisely 1168 ... > checksum is correct. ... after vlan untagging is 1222 octets; the sent frame includes a tag ...
      (freebsd-net)
    • RE: VLAN as a DMZ
      ... VLAN-Hopping is a potential using only VLAN security to isolate an insecure ... DMZ segment is a bad idea? ...
      (Security-Basics)
    • Re: NLBS in VLANs environment
      ... computer, enabled it for NLB, configured cluster parameters the same for each one, and so on... ... And after doing this, network stops responding on both computers, because arp resolution fails on every VLAN adapter, until disabling NLB. ... A VLAN in a switch is a group of ports that are segmented with a new IP segment so that all members in the segment are on the same subnet. ...
      (microsoft.public.windows.server.clustering)
    • Re: VLAN as a DMZ
      ... Subject: VLAN as a DMZ ... VLAN to communicate with the switch (to change the VLAN memberships - Telnet ... It'd be just an isolated segment. ...
      (Security-Basics)
    • Re: VLAN interfaces on FreeBSD; performance issues
      ... >> The reason for wanting VLAN tagging is the machine has once NIC ... My goal is to make this machine a gateway for several servers that I ... need to segment that will be on different IP subnets. ... layer-2 separation for security. ...
      (freebsd-isp)