HEADSUP: New i386 interrupt and SMP code..

From: John Baldwin (jhb_at_FreeBSD.org)
Date: 10/30/03

  • Next message: Brooks Davis: "Re: HEADSUP: New i386 interrupt and SMP code.." 
    Date: Thu, 30 Oct 2003 17:34:47 -0500 (EST)
To: arch@FreeBSD.org

    Coming very soon to a CVS tree near you are some very large changes to
    the i386 interrupt and SMP code. New features include:

    - Runtime selection of using the I/O APICs or the AT PICs to route
      interrupts.
    - I/O APICs can be used in a UP kernel or on a UP system that
      supplies either an MP Table or ACPI APIC Table.
    - An SMP kernel can run on a UP machine. This means that SMP
      can now be enabled in GENERIC and the SMP kernel config can die.
    - The ACPI MADT table can be used to enumerate CPUs instead of
      the MP Table if ACPI is enabled. This will add true HT support
      in that we will finally support the BIOS setting for HT.
    - I/O APIC interrupts are now longer forced into 8 IRQs. Thus,
      when using APICs, each PCI interrupt really gets its own IRQ
      and isn't shared with anyone else.
    - Multiple fast interrupt handlers can be attached to a given
      interrupt source provided that all of the handlers are fast.
      (Note: at this point, fast is a poor name, INTR_DIRECT might
      be a better name.)
    - Logical APIC IDs are used to route APIC interrupts from the
      I/O APICs to CPUs. In theory the APIC interrupt code can
      now support 60 CPUs. The hardware is still limited to 16
      however.
    - We now correctly route PCI interrupts when using APICs
      using the PCI interrupt routing infrastructure instead of
      a gross hack in pci_cfgregread(). This means that we can
      route interrupts across bridges, support mp tables that
      only list interrupts for chassis devices, etc. We also
      correctly route PCI interrupts when using APICs and ACPI.
    - The new interrupt source abstraction should make it substantially
      easier to add support for MSI interrupts.
    - We properly support mixed mode by EOI'ing the AT PIC and
      not EOI'ing the local APIC for mixed mode interrupts (just
      irq 0: clk right now).
    - This code can largely be pulled over to amd64 to support
      APICs and SMP on that arch.

    Some implementation details include:

    - APIC interrupt entry points only use one entry point per 32
      vectors and use the APIC ISR registers to determine which
      interrupt triggered in that range. This means that the APIC
      code only has to provide 5 entry points instead of 159.
    - Because we now support up to 159 different IRQs, the critical
      section optimization code no longer scales well. Especially
      since the new APIC code does not use a separate entry point
      for each IRQ. Thus, for the time being at least, critical
      sections have been reverted back to disabling interrupts for
      now. I do have a WIP for optimizing critical sections using
      a more scalable algorithm should the need arise.
    - Each IRQ is actually a cookie tied to an interrupt source.
      Each interrupt source is tied to a PIC driver. The PIC driver
      supports several operations on each interrupt source including
      disabling the source, enabling it for the first time, etc.
      Each PIC driver is free to store private per-source data with
      each source and private per-pic data with each PIC.
    - APICs (both I/O and local (CPUs)) are enumerated by APIC
      enumerator drivers of which 2 are provided: one to use
      the ACPI MADT table and one to use the MP Table.
    - The SMP code no longer knows anything specific to the MP
      table. Instead, the APIC enumerators inform the SMP code
      of CPUs via a simple cpu_add() interface and the SMP code
      takes it from there. The SMP code is now much easier to
      read. Also, all of the APIC code has been split out into
      separate IO and local APIC files aiding in the cleanup.
    - Almost all of the interrupt dispatch code now happens in C
      rather than assembly. Notably, fast interrupt handlers no
      longer have a separate entry point.

    Downsides:

    - ACPI will no longer work as a module for know. The reason
      for this is that ACPI's APIC enumerator needs to be able
      to hook into a SI_SUB_TUNABLES - 1 SYSINIT() due to existing
      code that wants to know the available CPUs in the system
      very early (specifically, UMA). However, code in kernel
      modules cannot be executed until SI_SUB_KLD, which is much
      too late. This might be able to be addressed later with
      some creative hacking.
    - I haven't ported the changes over to PC98 yet.

    Code:

    The code lives in p4 under //depot/user/jhb/acpipci/...
    Note that several files have moved around so you might want to
    check the 'notes' file and 'setup.sh' file. If you want to
    try it out you can check out the tree using p4 and build a
    kernel. Just be sure to:

     1) Run setup.sh first to create needed symlinks for moved
        files.
     2) Use 'device apic' instead of 'options APIC_IO'.

    I'm sure there's more details that I've forgotten, but that's
    a start at least. 

    -- 
John Baldwin <jhb@FreeBSD.org>  <><  http://www.FreeBSD.org/~jhb/
"Power Users Use the Power to Serve!"  -  http://www.FreeBSD.org/
_______________________________________________
freebsd-arch@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org"
  • Next message: Brooks Davis: "Re: HEADSUP: New i386 interrupt and SMP code.."

    Relevant Pages

    • Re: HEADSUP: New i386 interrupt and SMP code..
      ... > when using APICs, each PCI interrupt really gets its own IRQ ... > interrupt source provided that all of the handlers are fast. ... > - The SMP code no longer knows anything specific to the MP ...
      (freebsd-arch)
    • Re: can someone explain...[ PCI interrupts]
      ... interrupts for PCI devices are roughly split up into two categories: INTx interrupt lines and MSI interrupts. ... Each processor contains a local APIC that can receive messages from other APICs and send messages to other local APICs. ... Basically, an IRQ value is a cookie useful for binding a device interrupt tuple or an ISA IRQ) to a x86 interrupt tuple. ...
      (freebsd-current)
    • Re: hardwired VMI crap
      ... In terms of interrupt gates for the ... It's not rocket science to implement an abstract interrupt controller, ... would ever enable the APICs. ... backend to the generic IRQ code. ...
      (Linux-Kernel)
    • Re: Potential source of interrupt aliasing
      ... > that the device is asserting the interrupt on the WRONG PIN. ... allocated consecutively across APICs. ... > * The bridge chips have a limited number of IRQ pins. ...
      (freebsd-current)
    • Linux-2.6.3 : [eth0: Too much work at interrupt, status=0x00000001.]
      ... Using 1 I/O APICs ... number of MP IRQ sources: ... eth0: Setting full-duplex based on MII #1 link partner capability of 41e1. ... Too much work at interrupt, ...
      (Linux-Kernel)