Regression with jails/IPv6/pf


So, I tried to do a routine update to the latest stable/9 yesterday
(r238771), and I found that access to the jail on my server had stopped
working. Everything else seemed to be fine, and reverting to the
previous system (r237456 from 2012-06-22 (Boot Environments FTW)) bought
it all back to life.

After spending most of today bisecting versions and compiling kernels,
I found:

r238177 worked absolutely fine

r238236 accessing the jail worked, but everything was slow, as if
DNS queries were timing out.

r238246 lots of network timeouts everywhere: accessing the jail
failed, but then so did accessing the main host. So much
so that svn couldn't update properly.

r238256 worked fine for accessing the main host, but failed when
trying to access the jail.

Looks like this seems to have been introduced in a batch of commits
MFC'd by bz@ (CC'd) around then.

Now, this jail is set up in an unusual way, which is why I guess I'm the
first person to be affected. For starters, it only has IPv6
connectivity, and secondly, because I'm running some daemons there I
don't want listening on an external network socket, it's bound to the
loopback and I use firewall redirection to send traffic to it.

The jail config in /etc/rc.conf looks like this:

jail_interface="lo1"
jail_devfs_enable="YES"
jail_devfs_ruleset="devfsrules_jail_zfs"
jail_fdescfs_enable="YES"
jail_procfs_enable="YES"
jail_set_hostname_allow="NO"
jail_socket_unixiproute_only="YES"
jail_sysvipc_allow="NO"
jail_parallel_start="NO"

jail_xenophobe_hostname="xenophobe.infracaninophile.co.uk"
jail_xenophobe_rootdir="/jail/xenophobe"
jail_xenophobe_ip="fd87:cd50:2103:1:54f9:9484:e8b0:12d1"
jail_xenophobe_mount_enable="YES"
jail_xenophobe_zfs="zroot/jail/xenophobe zroot/jail/xenophobe/TimeMachine"
jail_xenophobe_params="enforce_statfs=1"

I've cloned a second loopback I/F and given the jail an address from the
IPv6 private address range (RFC4193). Cloning the interface
isn't absolutely necessary -- exactly the same symptoms occur if I use
an alias address on lo0 -- but it makes it easier to see only jail
traffic when using tcpdump.

Then I've enabled access via the network using nat+rdr in PF, like so:

table <localnets> const { 2001:8b0:151:1::/64, \
81.187.76.160/29, \
fd87:cd50:2103:1::/64 }

xenophobe_int="fd87:cd50:2103:1:54f9:9484:e8b0:12d1"
xenophobe_ext="2001:8b0:151:1:54f9:9484:e8b0:12d1"

[...]

nat on $ext_if_plus from $xenophobe_int to any -> $xenophobe_ext
rdr inet6 proto tcp from <localnets> to $xenophobe_ext \
port { 22, 80, 443, 548, 4700 } -> $xenophobe_int

When trying to ssh into the jail with a kernel exhibiting this problem,
tcpdump showed that traffic was reaching the sshd in the jail and
responses were being generated, but they didn't make it out onto the net.

Cheers,

Matthew

--
Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard
Flat 3
PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate
JID: matthew@xxxxxxxxxxxxxxxxxxxxxx Kent, CT11 9PW

Attachment: signature.asc
Description: OpenPGP digital signature

Relevant Pages

  • Re: Closing information leaks in jails?
    ... > restricted devfs in the jail (devfsrules_jail for example from ... but the primary IP address of the interfaces. ... > - some interesting information about the network related stuff via netstat ...
    (FreeBSD-Security)
  • Re: ipnat.conf - map and rdr wont work!
    ... Use tcpdump, you should see if your rdr/map rules work as expected. ... Can't help you more, really, you need to investigate where packets are ... if i disable ipnat then map or rdr wont work as they simply disabled ... nothing gets to jail there for no logs inside of jail ...
    (freebsd-questions)
  • Re: jail source address selection doesnt work?
    ... establish the connection with netstat. ... Given the destination network does not match any directly connected ... network and, based on your previous email, you don't have an route going out a gateway of either of those two networks to 192.168.72.3 it's doing ... the fallback to the "primary" jail IP, ...
    (freebsd-net)
  • Re: My planned work on networking stack (vimage)
    ... > this isnt exactly what jail was designed to do, ... > providing a clone-able network stack within jails. ... The fact that the virtualization patches are highly disruptive by their ... could have any future in the official FreeBSD tree... ...
    (freebsd-net)
  • Re: Point-to-point connection between jails?
    ... jail A with address 10.1.1.1 and for which I can configure ... network interface as the means by which two jails can ... like with epair and VIMAGE? ...
    (freebsd-net)