Re: [RFC][PATCH 4/4] Rename lock_cpu_hotplug/unlock_cpu_hotplug

Paul E. McKenney
2006-08-29T18:04:30Z

On Sun, Aug 27, 2006 at 12:59:44AM -0700, Paul Jackson wrote:
> Gautham wrote:
> > Which is why I did not expose the locking(at least the write side of it)
> > outside. We don't want too many lazy programmers anyway!
>
> Good thing you did that ;). This lazy programmer was already
> having fantasies of changing the cpuset locks (in kernel/cpuset.c)
> manage_mutex and callback_mutex to these writer and reader "unfair
> rwsem" locks, respectively.
>
> Essentially these cpuset locks need to guard the cpuset hierarchy, just
> as your locks need to guard cpu_online_map.
>
> The change agents (such as a system admin changing something in
> the /dev/cpuset hierarchy) are big slow mammoths that appear rarely,
> and need to single thread their entire operation, preventing anyone
> else from changing the cpuset hierarchy for an extended period of time,
> while they validate the request and setup to make the requested change
> or changes.
>
> The inhibitors are a swarm of locusts, that change nothing, and need
> quick, safe access, free of change during a brief critical section.
>
> Finally the mammoths must not trample the locusts (change what the
> locusts see during their critical sections.)
>
> The cpuset change agents (mammoths) take manage_mutex for their entire
> operation, locking each other out. They also take callback_mutex when
> making the actual changes that might momentarilly make the cpuset
> structures inconsistent.

Can the locusts reasonably take a return value from the acquisition
primitive and feed it to the release primitive?

Thanx, Paul

> The cpuset readers (locusts) take callback_mutex for the brief critical
> section during which they read out a value or two from the cpuset
> structures.
>
> If I understand your unfair rwsem proposal, the cpuset locks differ
> from your proposal in these ways:
> 1) The cpuset locks are crafted from existing mutex mechanisms.
> 2) The 'reader' (change inhibitor) side, aka the callback_mutex,
> is single threaded. No nesting or sharing of that lock allowed.
> This is ok for inhibiting changes to the cpuset structures, as
> that is not done on any kernel hot code paths (the cpuset
> 'locusts' are actually few in number, with tricks such as
> the cpuset generation number used to suppress the locust
> population.) It would obviously not be ok to single thread
> reading cpu_online_map.
> 3) The 'writer' side (the mammoths), after taking the big
> manage_mutex for its entire operation, *also* has to take the
> small callback_mutex briefly around any actual changes, to lock
> out readers.
> 4) Frequently accessed cpuset data, such as the cpus and memory
> nodes allowed to a task, are cached in the task struct, to
> keep from accessing the tasks cpuset frequently (more locust
> population suppression.)
>
> Differences (2) and (4) are compromises, imposed by difference (1).
>
> The day might come when there are too many cpuset locusts -- too many
> tasks taking the cpuset callback_mutex, and then something like your
> unfair rwsem's could become enticing.
>
> --
> I won't rest till it's the best ...
> Programmer, Linux Scalability
> Paul Jackson <pj@sgi.com> 1.925.600.0401
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>