Re: Network Stack Locking

From: Matthew Dillon (dillon_at_apollo.backplane.com)
Date: 05/25/04

  • Next message: Luigi Rizzo: "Re: Network Stack Locking" 
    Date: Tue, 25 May 2004 11:29:45 -0700 (PDT)
To: Alan Cox <alc@cs.rice.edu>

    :>:Sounds a lot like a lot of the Mach IPC optimizations, including their use
    :>:of continuations during IPC to avoid a full context switch.
    :>:
    :>:Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
    :>:robert@fledge.watson.org Senior Research Scientist, McAfee Research
    :>
    :> Well, I like the performance aspects of a continuation mechanism, but
    :> I really dislike the memory overhead. Even a minimal stack is
    :> expensive when you multiply it by potentially hundreds of thousands
    :> of 'blocking' entities such as PCBs.. say, a TCP output stream.
    :> Because of this the overhead and cache pollution generated by the
    :> continuation mechanism increases as system load increases rather
    :> then decreases.
    :
    :When the explicit continuation mechanism was used, the thread's stack
    :was freed when the thread blocked and a new stack allocated when the
    :thread was restarted. Here is a URL:
    :http://citeseer.ist.psu.edu/draves91using.html. Notice the mention of
    :space reduction in the abstract. It's worth reading.
    :
    :Alan

        Ah, now I understand. The continuation is an exit from the procedure.

        This sounds very similar to an interrupt mechanism I designed about
        a decade ago. Instead of saving and restoring the interrupt
        context for the interrupt thread, the thread switch was special-cased
        to basically create a context (by pushing a procedure call on the stack)
        on switch-in and to throw it away on switch-out (which was always at
        the end of the interrupt routine). I extended the same mechanism down
        into 'userland' by creating a 'waitforever()' system call which basically
        did nothing but wait for and dispatch signal vectors (most of the programs
        were event oriented and had no main loop). The nice thing about this was
        that no context had to be saved while the program was sitting in
        waitforever(), or restored when the program returned from a signal handler.
        This more then doubled scheduler performance (which on a 10MHz 68000 was
        important).

        In many ways, the continuation mechanism and the message queue mechanism
        appear to be nearly identical. If an explicit exit from a procedure
        is required to optimize the stack with the continuation mechanism, then
        that isn't much different then moving the message to an event queue
        and returning to the message processing loop. Neither case allows
        you to save stack context or to save the current procedural stacking
        level, and both mechanisms allow you to reuse your current stack to
        handle multiple messages/continuations.

                                            -Matt
                                            Matthew Dillon
                                            <dillon@backplane.com>
    _______________________________________________
    freebsd-arch@freebsd.org mailing list
    http://lists.freebsd.org/mailman/listinfo/freebsd-arch
    To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org"

  • Next message: Luigi Rizzo: "Re: Network Stack Locking"

    Relevant Pages

    • Re: Network Stack Locking
      ... >:When the explicit continuation mechanism was used, the thread's stack ... > to basically create a context ... the continuation mechanism and the message queue mechanism ...
      (freebsd-arch)
    • Re: Miltu-core CPUs, threads vs AST driven approaches
      ... The way to do that is to run the several 'worker' threads at normal (not AST) level, and use the AST routines only to process I/O completion and as part of that completion move the continuation context for the relevant activity onto a 'to do' queue, which each thread visits whenever it has no processing to do. ... This preserves VMS's guarantee that no more than one AST will be active at a time, and the extremely brief processing required at AST level (compared with the processing required in each thread between interruptions) ensures that this will not impede progress even if many processors are working in parallel in the single process. ... The main gotcha here is that *all* the context for each operation must be held in that queued context structure, rather than conveniently on a thread's stack - since the thread's stack gets unwound before each such interruption. ...
      (comp.os.vms)
    • Re: interrupt routine and application pages
      ... application stack in the interrupt context. ... are still in the context of interrupted thread. ... your code runs at raised IRQL, Memory Manager just had no chance to ...
      (microsoft.public.development.device.drivers)
    • Re: Odd call stack
      ... they solved it in WinCE version 1.0. ... This is sort of a "light" context switch with no ... marshaling (just take parameters off the same stack, ... >> address space of the server process and a little magic. ...
      (microsoft.public.windowsce.platbuilder)
    • Re: Odd call stack
      ... will save and restore the context of 'callee process', ... > Michael Salamone ... >>I was debugging my own audio device driver and get follwoing call ... >> stack log after stopped at some breakpoint inside the driver code. ...
      (microsoft.public.windowsce.platbuilder)