Re: [PATCH] adjust root-domain->online span in response to hotplugevent

["Gregory Haskins" <ghaskins@novell.com>]

>>> On Sat, Mar 8, 2008 at  3:41 AM, in message <20080308084118.GA24552@elte.hu>,
Ingo Molnar <mingo@elte.hu> wrote: 

> * Gregory Haskins <ghaskins@novell.com> wrote:
> 
>>    Unfortunately, this patch will introduce its own set of bugs. 
>> However, your analysis was spot-on.  I think I see the problem now.  
>> It was introduced when I put a hack in to "fix" s2ram problems in -mm 
>> as a result of the new root-domain logic.  I think the following patch 
>> will fix both issues:
>> 
>> (I verified that I could take a cpu offline/online, but I don't have 
>> an s2ram compatible machine handy.  Andrew, I believe you could 
>> reproduce the s2ram problem a few months ago when that issue popped 
>> up.  So if you could, please verify that s2ram also works with this 
>> patch applied, in addition to the hotplug problem.
> 
> thanks Gregory, i've queued up your fix. If it passes all tests over the 
> day i'll send it to Linus in about 12 hours.
> 


After thinking about it some more, I am not sure if I got this fix quite right. The first two hunks are technically fine and should ultimately go in.  The last hunk is questionable.

Ultimately, I believe the root cause of these reported issues is that cpu_online_map and rd->online can get out of sync.  I fear that while the current patch may fix the hotplug/s2ram case, it potentially breaks the disjoint cpuset topology change case.  I will run a few cpuset tests ASAP to confirm, though I may not have a chance until Monday.

What I probably need is to tie the code that sets/clears cpu_online_map to the root-domain rd->online map somehow.  However, a cursory cscope inspection has failed to reveal the location that this cpu_online_map manipulation occurs.  If anyone knows where cpu_online_map is actually updated, please let me know.  Otherwise, let me think about this some more and get back to you.

In the meantime, cpu_and'ing "task->cpus_allowed & cpu_online_map & rd->online" in the place that Suresh identified should technically fix the hotplug issue without adding more bugs, at the expense of requiring two calls to cpu_and().  The expense should be more or less negligible on mainstream kernels, but may hurt our big-iron (e.g. SGI) brethren.  Hopefully I will come up with a real fix shortly, before it becomes a real issue.

Regards,
-Greg