Re: cpuset and affinity implementation

On Mon, 25 Feb 2008, Daniel Eischen wrote:

On Mon, 25 Feb 2008, Jeff Roberson wrote:

On Mon, 25 Feb 2008, Daniel Eischen wrote:

On Mon, 25 Feb 2008, Jeff Roberson wrote:

On Mon, 25 Feb 2008, Alfred Perlstein wrote:

Jeff, this is very cool. I do have one issue though:

+ * A thread may not be assigned to a a group seperate from other threads in
+ * the process. This is to remove ambiguity when the setid is queried with
+ * a pid argument. There is no other technical limitation.

Am I understanding things correctly such that within a process there
can only be one "set"?

If so this restricts some of the benifits you get with sets and
binding.

An example would be some sort of system with multiple CPUs where some
are assigned specifically for pseudo-realtime processing and others are for
general control things such as cli, stats, etc.

In our case we would like to be able to run some threads on specific
cpu sets, and other threads to be run anywhere on the control CPUs.

Can this be done with this API?

Individual threads can be bound to any cpu or group of cpus within the set. So if you just make a set that includes all cpus in the system you can then bind your realtime threads to specific cpus and the other threads to the remainder. You will have to specifically bind each thread however.

The reason individual threads can't be assigned to groups is because cpuset_getid() for a pid wouldn't make sense then and I expect administrators to be mostly interested in managing groups of processes.

If the administrator sets up a set of CPUs specifically for
real-time and another set for non-real-time, you may want to
bind some threads to the real-time set, and leave other threads
unbound (or even bound to the non-real-time set). In this
case, I think cpuset_getid() should either return the default
cpuset of all cpus in the system, or the last cpuset to
which the process was bound.

But regardless, I think binding a thread to a different
processor set should be allowed and should override its
inherent binding of the process' processor set.
Hmm, I guess in this case, a subsequent binding of the
process to a processor set should probably override any
per-thread bindings.

I think we're getting into complex corner cases here which will only confuse the api and administrators. I don't expect administrators will want to set groups to individual threads. How would he even identify the individual thread? And if he did, he could just as easily set masks on that thread along with others in the process.

I'm already a little nervous about how complicated this will be for programmers. If we allowed each thread in a pid to be in its own set, we'd have to make cpuset_getid() return an array of ids. I definitely don't want to do that.

Solaris does seem to allow this BTW.

Solaris also doesn't allow a processor to be in more than one set. It doesn't allow a thread to bind to a processor that's in a processor set. It also doesn't seem provide a mechanism to query the set that a thread is in, so there is no ambiguity for the querying. However, when you modify you have the option of retrieving the old set. They must simply return the first one discovered. We could do that but it doesn't seem very attractive.

Would people be in favor of binding threads to sets if it meant getting the id from a pid was not always 100% accurate? Even though a thread may already restrict its set?

From the pset_assign man page:

"Processors with LWPs bound to them using processor_bind(2) cannot be assigned to a new processor set. If this is attempted, pset_assign() will fail and set errno to EBUSY."

My cpuset design seems to be a lot more flexible.

Cheers,
Jeff


--
DE

_______________________________________________
freebsd-arch@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: cpuset and affinity implementation
    ... An example would be some sort of system with multiple CPUs where some ... The reason individual threads can't be assigned to groups is because cpuset_getidfor a pid wouldn't make sense then and I expect administrators to be mostly interested in managing groups of processes. ... cpuset of all cpus in the system, ... I think binding a thread to a different ...
    (freebsd-arch)
  • Re: cpuset and affinity implementation
    ... An example would be some sort of system with multiple CPUs where some ... The reason individual threads can't be assigned to groups is because cpuset_getidfor a pid wouldn't make sense then and I expect administrators to be mostly interested in managing groups of processes. ... I think binding a thread to a different ...
    (freebsd-arch)
  • Re: cpuset and affinity implementation
    ... An example would be some sort of system with multiple CPUs where some ... The reason individual threads can't be assigned to groups is because cpuset_getidfor a pid wouldn't make sense then and I expect administrators to be mostly interested in managing groups of processes. ... I think binding a thread to a different ...
    (freebsd-arch)
  • Re: cpuset and affinity implementation
    ... An example would be some sort of system with multiple CPUs where some ... So if you just make a set that includes all cpus in the system you can then bind your realtime threads to specific cpus and the other threads to the remainder. ... I think binding a thread to a different ... processor set should be allowed and should override its ...
    (freebsd-arch)