Re: New cpu_switch() and cpu_throw().

On Tue, 5 Jun 2007, John Baldwin wrote:

On Tuesday 05 June 2007 01:32:46 am Jeff Roberson wrote:
For every architecture we need to support a new features in cpu_switch()
and cpu_throw() before they can support per-cpu schedlock. I'll describe
those below. I'm soliciting help or advice in implementing these on
platforms other than x86, and amd64, especially on ia64 where things are
implemented in C!

I checked in the new version of cpu_switch() for amd64 today after
threadlock went in. Basically, we have to release a thread's lock when
it's switched out and acquire a lock when it's switched in.

The release must happen after we're totally done with the stack and
vmspace of the thread to be switched out. On amd64 this meant after we
clear the active bits for tlb shootdown. The release actually makes use
of a new 'mtx' argument to cpu_switch() and sets the td_lock pointer to
this argument rather than unlocking a real lock. td_lock has previously
been set to the blocked lock, which is always blocked. Threads
spinning in thread_lock() will notice the td_lock pointer change and
acquire the new lock. So this is simple, just a non-atomic store with a
pointer passed as an argument. On amd64:

movq %rdx, TD_LOCK(%rdi) /* Release the old thread */

The acquire part is slightly more complicated and involves a little loop.
We don't actually have to spin trying to lock the thread. We just spin
until it's no longer set to the blocked lock. The switching thread
already owns the per-cpu scheduler lock for the current cpu. If we're
switching into a thread that is set to the blocked_lock another cpu is
about to set it to our current cpu's lock via the mtx argument mentioned
above. On amd64 we have:

/* Wait for the new thread to become unblocked */
movq $blocked_lock, %rdx
1:
movq TD_LOCK(%rsi),%rcx
cmpq %rcx, %rdx
je 1b

If this is to handle a thread migrating from one CPU to the next (and there's
no interlock to control migration, otherwise you wouldn't have to spin here)
then you will need memory barriers on the first write (i.e. the first write
above should be an atomic_store_rel()) and the equivalent of an _acq barrier
here.

So, thanks for pointing this out. Attilio also mentions that on x86 and amd64 we need a pause in the wait loop. As we discussed, we can just use sfence rather than atomics on amd64, however, x86 will need atomics since you can't rely on the presence of *fence. Other architectures will have to ensure memory ordering as appropriate.

Jeff


--
John Baldwin

_______________________________________________
freebsd-arch@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: New cpu_switch() and cpu_throw().
    ... I checked in the new version of cpu_switchfor amd64 today after ... it's switched out and acquire a lock when it's switched in. ... switching into a thread that is set to the blocked_lock another cpu is ... As we discussed, we can just use sfence rather than atomics on amd64, however, x86 will need atomics since you can't rely on the presence of *fence. ...
    (freebsd-arch)
  • New cpu_switch() and cpu_throw().
    ... For every architecture we need to support a new features in cpu_switchand cpu_throwbefore they can support per-cpu schedlock. ... I'm soliciting help or advice in implementing these on platforms other than x86, and amd64, especially on ia64 where things are implemented in C! ... we have to release a thread's lock when it's switched out and acquire a lock when it's switched in. ... If we're switching into a thread that is set to the blocked_lock another cpu is about to set it to our current cpu's lock via the mtx argument mentioned above. ...
    (freebsd-arch)
  • Re: New cpu_switch() and cpu_throw().
    ... and cpu_throwbefore they can support per-cpu schedlock. ... I checked in the new version of cpu_switchfor amd64 today after ... it's switched out and acquire a lock when it's switched in. ... switching into a thread that is set to the blocked_lock another cpu is ...
    (freebsd-arch)
  • Re: 2.6 kernel causing X to hang?
    ... >> not showing the results of me doing anything to it. ... I can echo similar problems since switching to 2.6.4 and now 2.6.5. ... lock-up when GDM is running and no users logged it. ... And to make it more interesting, sometimes X will lock ...
    (Debian-User)
  • Re: Ranging and Pioneer
    ... Don't think about switching the transmitter on and off. ... command the spacecraft to continuously downlink the contents of some ... lock on to that modulating waveform. ...
    (sci.astro.research)