Re: getaffinity/setaffinity and cpu sets.

On Wed, 20 Feb 2008, Jeff Roberson wrote:

I also have a 'cpuset' command which can run a new program with a given cpu set, view and modify sets of arbitrary pids. This is all working and I can supply patches if anyone is interested. I have to implement 4BSD support before I can commit.

I have a proposal for solaris style processor sets which I think is simple and sufficient for most cases. It involves the following new syscalls:

int cpuset(void); int setcpuset(pid_t pid, int setid); int getcpuset(pid_t pid);

The notion would be that you can create a new numbered cpuset with cpuset(). You can modify or inspect its affinity with get/setaffinity above and the CPU_WHICH_SET argument. The cpuset exists as long as there are members of the set. Sort of like a process group or session. The {get,set}cpuset calls can inspect or modify the state.

This set would not be modifiable by user processes or by processes in a jail. It would create the restriction that differs between 'avail' and 'sys' above. Processors would be able to directly bind to any processor within the set. Changing the set would apply to all processes in the set. The cpuset would be per-process while the mask is per-thread. Sets involvement is inherited on fork().

In solaris sets can be named and have a more complete management api. I'm not really interested in implementing all of that but I believe what I have outlined here would be subset of this and no code/syscalls would be wasted.

Comments? Objections? I'm fairly pleased with this arrangement now.

Just to put a few notes from our conversation on IRC in e-mail:

- I think I'd prefer int cpuset(cpuset_t *set), int getcpuset(pid_t, cpuset_t
*) so that we don't mix up ID's and return values. More recent interfaces
tend to do this, I believe, and it means that the prototype, even if not the
ABI, remains the same if the set identifier changes in the future.

- You don't mention what happens if a process's cpu set changes to preclude a
CPU the process has a thread with affinity for. Online, you suggested
SIGKILL, and I thought maybe a new SIGCPUGONE with a default SIGKILL action
might be a friendlier model. We should see what Solaris and others do here
though. I like the idea that the affinity is a guarantee in userspace
because it means that you can rely on it; I'm OK with the idea that your
thread always runs on the CPUs you have affinity for unless in the
SIGCPUGONE handler :-).

- It would be nice to be able to use CPU sets in jail as well, suggesting a
hierarchal model with some sort of tagging so you know what CPU sets were
created in a jail such that you know whether they can be changed in a jail.
While I recognize this makes things a lot more tricky, I think we should
basically be planning more carefully with respect to virtualization when we
add new interfaces, since it's a widely used feature, and the current set of
"stragglers" unsupported in Jail is growing rather than shrinking.

- There's still no way to specify an affinity policy rather than explicit
affinity, but if our CPU set model is sufficiently general, that might be a
vehicle to do that. I.e., cpuset_setpolicy() rather than setting a mask.

- In the interests of boring API changes, recent APIs tend to prefix the
method on the object name. Have you thought about cpuset_create(),
cpuset_foo(), etc? That reduces the chances of interfering with application
namespaces. I think, anyway. :-).

I need to ponder the proposal a little more, ideally over a hot beverage this morning, and will follow up if I have further thoughts. Thanks for working on this, BTW -- affinity is well-overdue for FreeBSD.

Robert N M Watson
Computer Laboratory
University of Cambridge
_______________________________________________
freebsd-arch@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@xxxxxxxxxxx"

Relevant Pages
Quantcast