Re: AMD64 NUMA-awareness?

From: Jeff Roberson (jroberson_at_chesapeake.net)
Date: 05/20/05

  • Next message: Patty: "Dude check out this sweet site!" 
    Date: Fri, 20 May 2005 04:36:32 -0400 (EDT)
To: Scott Long <scottl@samsco.org>

    On Thu, 19 May 2005, Scott Long wrote:

    > Jung-uk Kim wrote:
    >
    > > ULE scheduler paper
    > > (http://www.usenix.org/publications/library/proceedings/bsdcon03/tech/roberson.html)
    > > says:
    > >
    > > 'SMT introduces a concept of non-uniform processors into ULE which
    > > could be extended to support NUMA. The concept of expressing the
    > > penalty for migration through the use of separate queues could be
    > > further developed to include a local and global load-balancing
    > > policy. At the time of this writing, however, FreeBSD does not
    > > support any true NUMA capable machines and so this is left until such
    > > time that it does.'
    > >
    > > I am not sure about the meaning of 'true NUMA capable machines' but
    > > AMD64 is ccNUMA unless I am completely mistaken, and FreeBSD/amd64 is
    > > well-supported. Even multicore processors are available now and
    > > Intel is going to release dual-core processors with HTT to make
    > > matters worse.
    > >
    >
    > Even HTT by itself presents some interesting scheduling challenges that
    > need to be accounted for.
    >
    > > Is there anybody working on this?
    >
    > Jeff stated recently that both ULE and 4BSD have some very primitive
    > awareness of topology, but that much more work is needed. NUMA would
    > also benefit from the UMA memory allocator understanding memory topology
    > and working with the scheduler to keep processes on CPUs that are
    > closest to their memory. Unfortunately, there doesn't appear to be
    > much work going on, so if anyone is interested, it's a great project and
    > I encourage all to help.

    I have put a great deal of thought into the issues in both UMA and
    schedulers for NUMA. I have not put any code in as of yet, however. I
    think that linux has had limited success with numa support on AMD64. The
    remote penalty doesn't seem to be great enough to really warrant much more
    complicated algorithms. Interested parties are certainly welcome to
    explore this more. I believe remote accesses are in the range of 33-50%
    more expensive than local. We might benefit by making a copy of the
    kernel text for each processor. That would be a good starting point to
    see if any gains are observed in the best case.

    >
    > Scott
    >
    > _______________________________________________
    > freebsd-arch@freebsd.org mailing list
    > http://lists.freebsd.org/mailman/listinfo/freebsd-arch
    > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org"
    >
    _______________________________________________
    freebsd-arch@freebsd.org mailing list
    http://lists.freebsd.org/mailman/listinfo/freebsd-arch
    To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org"

  • Next message: Patty: "Dude check out this sweet site!"

    Relevant Pages

    • Re: 2.6.9-rc1-mm1
      ... >> that it seems to slow some workloads on NUMA. ... but I have no benchmarks that give any useful numbers. ... Actually, with the current scheduler, updatedb really sucks. ... > subjective reports, due to placebo effect. ...
      (Linux-Kernel)
    • Re: 2.6.4-mm1
      ... > For SMT it is a less complex than shared runqueues, ... > It is also more flexible than shared runqueues in that you can still ... > For Opteron type NUMA, it actually balances much more aggressively ... > than the default NUMA scheduler, especially when a CPU is idle. ...
      (Linux-Kernel)
    • Re: [patch] scheduler fix for 1cpu/node case
      ... I really feel there's no point in a NUMA scheduler for the Hammer ... The interesting thing is probably whether we want balance on exec ... Especially in the corner case of one CPU per node this is ...
      (Linux-Kernel)
    • [patch] scheduler fix for 1cpu/node case
      ... after talking to several people at OLS about the current NUMA ... Fact is that the current separation of local and global balancing, ... Especially in the corner case of one CPU per node this is ... scheduler while keeping NUMA memory management on. ...
      (Linux-Kernel)
    • Re: [RFC] generalise scheduling classes
      ... >that time) was looking at two execution units on a single memory path. ... I don't think I said new, but I guess they (SMT, NUMA, CMP) are newish ... but the scheduler apparently doesn't work so well for atypical ... send the line "unsubscribe linux-kernel" in ...
      (Linux-Kernel)