Re: Routing outbound IP packets on multihomed box

On Fri, Jun 15, 2007 at 05:35:33PM -0700, Julian Elischer wrote:
Christopher Cowart wrote:
On Fri, Jun 15, 2007 at 06:30:23PM -0400, Boris Kochergin wrote:
Christopher Cowart wrote:
I have a server with two NICs:

em0: 169.229.79.139/25
vlan526: 169.229.126.9/24

The default gateway is 169.229.79.129. The router for the 126 subnet is
169.229.126.1.

netstat -rn:
| Destination Gateway Flags Refs Use Netif
Expire
| default 169.229.79.129 UGS 0 102537 em0
| 127.0.0.1 127.0.0.1 UH 0 217 lo0
| 169.229.79.128/25 link#1 UC 0 0 em0
| 169.229.79.129 00:15:c7:b9:f4:80 UHLW 2 4 em0
1193
| 169.229.79.139 00:11:25:ab:42:70 UHLW 1 589 lo0
| 169.229.126/24 link#9 UC 0 0 vlan52
| 169.229.126.1 00:15:c7:b9:f4:80 UHLW 1 34 vlan52
1200
| 169.229.126.9 00:18:f8:09:d3:a5 UHLW 1 8 lo0

The IP address on em0 works exactly as one would expect. I have full IP
connectivity to it from other subnets.

The problem is I can't get 2-way connectivity with the IP address on
vlan526.

Using my workstation on a third subnet (169.229.127.38/24), I cannot
ping 169.229.126.9. I leave the ping running and do some tcpdumps on
the server.

$ sudo tcpdump -ni vlan526 host 169.229.127.38
| 14:14:37.002920 IP 169.229.127.38 > 169.229.126.9: ICMP echo
| request, id 15733, seq 35, length 64
| 14:14:38.003037 IP 169.229.127.38 > 169.229.126.9: ICMP echo
| request, id 15733, seq 36, length 64

Notice there are no echo replies. That's because they're being sent
here:

$ sudo tcpdump -ni em0 host 169.229.127.38
| 14:15:42.006997 IP 169.229.126.9 > 169.229.127.38: ICMP echo reply,
| id 15733, seq 100, length 64
| 14:15:43.007118 IP 169.229.126.9 > 169.229.127.38: ICMP echo reply,
| id 15733, seq 101, length 64

I repeated this last snoop with a -w and loaded it into ethereal. The
echo replies being sent out on em0 indeed have a source address of
169.229.126.9. The router (169.229.79.139) drops these packets on the
floor, because their source address isn't routable on that interface.

Because routing is based on destination, not source address, I'm not
sure how to get packets sourced from the 126 subnet to the router on the
126 subnet. I tried the following ipfw rule right after allow loopback
traffic (my second rule):

fwd 169.229.126.1 ip from 169.229.126.9 to not 169.229.126.0/24

Still no luck. Has anyone set up a multihomed box on *different* subnets
before without routing them through the FreeBSD box? Does anyone have
any pointers or things I should be looking at?
Hi. I've come across this problem but solved it with a PF rule of this
form, if that's an option for you:

pass out route-to (vlan256 169.229.126.1) from 169.229.126.9 to any

This tells PF to send all packets sent from 169.229.126.9 through the
vlan256 interface with a next-hop address of 169.229.126.1.

Unfortunately, I don't think we can use pf. The rest of our
infrastructure is ipfw and we don't particularly want this to be a
one-off. I was under the impression that my ipfw rule did exactly this,
by sending the packets to the 126 router as their next hop.

Anyone have any ideas on whether an ipfw fwd rule can be used in a
similar way to this pf rule?

Thanks again,

he ipfw rule should work, assuming you have the IPFIREWALL_FORWARDING option
(and it's not the couple of versions of the OS where you also needed the
IPFIREWALL_FORWARD_EXTENDED option as well..

I had forwarding enabled, but not the EXTENDED flag. Apparently
bypassing the routing table in the firewall is an "unsafe" practice. Now
I can source traffic from the vlan526 interface correctly. Thanks!

also, you need to make it an 'out' rule..i.e.
fwd 169.229.126.1 ip from 169.229.126.9 to not 169.229.126.0/24 out

This doesn't appear to be necessary...

The next step, and here I'm having more problems, is I want to NAT
traffic using the vlan526 interface.

Using the same configuration as above, I've added:
vlan80: 10.80.1.1/16

When a NAT'd client pings off the 126 subnet (say, 169.229.127.38),
the packets go out em0 again.

My ipfw ruleset looks like this:
allow all from any to any via lo0
divert natd ip from not me to any via 169.229.126.9
fwd 169.229.126.1 ip from 169.229.126.9 to not 169.229.126.0/24
allow icmp from any to any

My understanding of divert is that after natd munges the packets,
they'll return to the ipfw rules immediately following rule 13. In this
case, traffic appearing from the NAT ip address should be forwarded to
the 126 router on the next-hop.

When I do `tcpdump -ni em0 host 169.229.127.38`, I notice the outbound
ICMP echo requests have src-addr of 10.80.x.x (IP address of the nat'd
client). Does this mean that as ipfw continues processing the packet,
even after natd has munged it, that as far as ipfw is concerned, the
src-addr of the packet hasn't changed?

In other words, would I need to write a complicated set of rules that do
something like:
divert natd ip from not me to any via 169.229.126.9
fwd 169.229.126.1 ip from 169.229.126.9 to not 169.229.126.0/24
fwd 169.229.126.1 ip from 10.80.0.0/16 to not 169.229.126.0/24

Any recommendations on how to solve the NATing extension of this routing
problem?

Thanks again,

--
Chris Cowart
Lead Systems Administrator
Network Infrastructure, RSSP-IT
UC Berkeley

Attachment: signature.asc
Description: Digital signature

Relevant Pages

  • Re: VPN and Routing in one box
    ... Packets originate in Subnet 1, ... The VPN is the first hop. ... When packets arrive via the VPN at Subnet 2, they have to be routed to a particular router / IP address on Subnet 2, which is the next hop in order to be futher routed to Subnet 3. ...
    (comp.dcom.vpn)
  • Re: route newbie question
    ... Subnet IP and other sit: ... That there is a router somewhere with the address 10.10.4.1 that knows ... computers needs to know what *IT* should do with packets bound for ... Since it can't send packets ...
    (comp.os.linux.networking)
  • Re: nat and ipfw
    ... packets to be translated. ... the packets to natd is one thing, ... > dsl and the other for an internal subnet. ... > ipfw configured and running. ...
    (freebsd-questions)
  • Re: Routing outbound IP packets on multihomed box
    ... The router for the 126 subnet is ... Using my workstation on a third subnet, ... sure how to get packets sourced from the 126 subnet to the router on the ... I was under the impression that my ipfw rule did exactly this, ...
    (freebsd-net)
  • Re: Routing outbound IP packets on multihomed box
    ... The router for the 126 subnet is ... Using my workstation on a third subnet, ... sure how to get packets sourced from the 126 subnet to the router on the ... I was under the impression that my ipfw rule did exactly this, ...
    (freebsd-net)