Re: crash when bpf is used heavily

From: Robert Watson (rwatson_at_freebsd.org)
Date: 05/28/04

  • Next message: Nate Lawson: "Re: Inspiron 8000 and acpiconf -s3" 
    Date: Fri, 28 May 2004 12:30:22 -0400 (EDT)
To: Sergey Lyubka <devnull@uptsoft.com>

    On Fri, 28 May 2004, Sergey Lyubka wrote:

    > > bump up your kernel virtual memory - see the handbook looking for the
    > > "KVA" setting.
    >
    > Yeah, did that. I had crashes when 20 simultaneous BPF apps where
    > running. Now it crashes at 30.
    >
    > Guys, these are not the solutions. vm allocator should check for memory
    > availability, imho, of course. I see it as a seriuos bug in -current.
    > Such unstable kernel no way can be put on production environment.

    Does seem like a non-ideal failure mode, doesn't it...

    It looks like the BPF code is written to handle the case where allocation
    fails, but that it passes flags to the memory allocator that prevent the
    memory allocator from returning a failure. Specifically,
    src/sys/net/bpf.c:bpf_allocbufs() passes M_WAITOK into malloc(). Try
    changing that flag (in both instances) to M_NOWAIT. This will still
    permit BPF to consume large quantities of memory if the maximum buffer
    size is set so large, but it will cause BPF itself not to cause a direct
    panic if address space is exhausted. I'm a little surprised M_NOWAIT
    isn't already the setting there, actually.

    The system will still be running in a low address space scenario which
    might cause other parts of the system to bump into the failure, however.
    Unfortunately, balancing multiple consumers of address space is a "hard
    problem". With the mbuf allocator, we make use of a separate address
    space map of bounded size to prevent the total address space consumed by
    packet buffers exceeding a certain size. It might be interesting to
    experiment with allocating BPF space from the same map, as it would change
    the trade-off from "panic if there's no room" to "stall the network stack
    if there's no room". The other common solution is to use smaller buffers,
    making the trade-off become "If the packets come too fast, we drop them".
    I realize that is the problem you're trying to solve... :-) On systems
    I've worked with that need to do processing of many high speed packet
    streams, we've generally tried to combine all the processing into modules
    in a single process, as this has a number of benefits:

    (1) It avoids storing the same packet many times in many buffers for
        different consumers.

    (2) It reduces the over-all memory overhead of buffering in the kernel.

    (3) It reduces the number of memory copies by avoiding copying the same
        packet many times (in particular, between user and kernel space)

    (4) It avoids performing additional context switches during high speed
        tracing, which can substantially impact available CPU resources for
        packet copying and monitoring.

    I realize, of course, that that approach cannot apply in all environments.

    Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
    robert@fledge.watson.org Senior Research Scientist, McAfee Research

    _______________________________________________
    freebsd-current@freebsd.org mailing list
    http://lists.freebsd.org/mailman/listinfo/freebsd-current
    To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

  • Next message: Nate Lawson: "Re: Inspiron 8000 and acpiconf -s3"

    Relevant Pages

    • Re: crash when bpf is used heavily
      ... > It looks like the BPF code is written to handle the case where allocation ... > memory allocator from returning a failure. ... The other common solution is to use smaller buffers, ...
      (freebsd-current)
    • Re: [PATCH 1/1] network memory allocator.
      ... Kevent network AIO uses usual alloc_skb, naio is called when packet is ... data and main system can work with that free memory. ... You do not see the point of network tree allocator. ...
      (Linux-Kernel)
    • Re: libpcap perf improvement? latest ideas?
      ... performance (with comparison of linux and freebsd) I searched freebsd resources for pcap improvements. ... I agree that a reference model can be used to reduce the number of copies done currently for BPF. ... In PF_PACKET you are forced to do a system call per-acquisition and another system for receiving the time-stamp of the last packet read for example. ... I proposed a model that allows for dynamic ring buffer size and signaling for soft and hard-limits to allow application buffering to handle potential drops. ...
      (freebsd-hackers)
    • Re: network related benchmark
      ... I believe that there is a FreeBSD developer working on an updated version of BPF that will perform even better than it does already. ... We're still working on refining and measuring performance of the implementation, but we're already seeing marked performance improvement as a result of reduced memory copies. ... Just to clarify the copying point: right now the FreeBSD BPF implementation performs a minimum of two memory copies per packet sniffed: one copy from the mbufs/mbuf clusters to the BPF buffer, and then one copy from the BPF buffer to user space. ...
      (freebsd-net)
    • Re: Implementation of Sampling for BPF
      ... BPF was never intended to be reliable every-packet solution. ... every packet, I am looking at attempting to define which packets it discards so that there is an equal chance of sampling something that happens, rather then an unknown/unpredictable chance. ... Putting as many servers as needed does scale well if you need only sampled data - just put an appropriate sampler/load balancer before them. ...
      (freebsd-net)