[PATCH] generic ipi function calls: wait on alloc failure fallback

[PATCH] generic ipi function calls: wait on alloc failure fallback

[Jeremy Fitzhardinge]

When a GFP_ATOMIC allocation fails, smp_call_function_mask falls back
to allocating the data on the stack and converting it to a waiting call.

Make sure we actually wait in this case.

Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
---
 kernel/smp.c |    1 +
 1 file changed, 1 insertion(+)

===================================================================
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -312,6 +312,7 @@ int smp_call_function_mask(cpumask_t mas
 	if (!data) {
 		data = &d;
 		data->csd.flags = CSD_FLAG_WAIT;
+		wait = 1;
 	}
 
 	spin_lock_init(&data->lock);

Re: [PATCH] generic ipi function calls: wait on alloc failure fallback

[Jeremy Fitzhardinge]

Jeremy Fitzhardinge wrote:
> When a GFP_ATOMIC allocation fails, smp_call_function_mask falls back
> to allocating the data on the stack and converting it to a waiting call.
>
> Make sure we actually wait in this case. 

Unfortunately this doesn't solve my crash, though it may account for 
some of them.

The oops I'm looking at at the moment is a NULL pointer on ->next of the 
rcu list in generic_smp_call_function_interrupt()...

    J

[PATCH] generic ipi function calls: wait on alloc failure fallback

[Jeremy Fitzhardinge]

When a GFP_ATOMIC allocation fails, it falls back to allocating the
data on the stack and converting it to a waiting call.

Make sure we actually wait in this case.

Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
---
 kernel/smp.c |    1 +
 1 file changed, 1 insertion(+)

===================================================================
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -312,6 +312,7 @@ int smp_call_function_mask(cpumask_t mas
 	if (!data) {
 		data = &d;
 		data->csd.flags = CSD_FLAG_WAIT;
+		wait = 1;
 	}
 
 	spin_lock_init(&data->lock);

Re: [PATCH] generic ipi function calls: wait on alloc failure fallback

[Ingo Molnar]

* Jeremy Fitzhardinge <jeremy@goop.org> wrote:

> When a GFP_ATOMIC allocation fails, it falls back to allocating the 
> data on the stack and converting it to a waiting call.
>
> Make sure we actually wait in this case.

cool, thanks!

does this explain the xen64 weirdnesses you've been seeing?

	Ingo

Re: [PATCH] generic ipi function calls: wait on alloc failure fallback

[Jeremy Fitzhardinge]

Ingo Molnar wrote:
> * Jeremy Fitzhardinge <jeremy@goop.org> wrote:
>
>   
>> When a GFP_ATOMIC allocation fails, it falls back to allocating the 
>> data on the stack and converting it to a waiting call.
>>
>> Make sure we actually wait in this case.
>>     
>
> cool, thanks!
>
> does this explain the xen64 weirdnesses you've been seeing?
>   

No, but I haven't seen it lately.  I think the other RCU fixes may have 
helped.  But it's all a bit of a worry: I didn't have a good theory 
about what was going wrong, the RCU patches didn't look like they'd fix 
the symptoms I was seeing.

I've seen it with 32 and 64-bit Xen, but there's nothing about the 
problem which makes me think it's really Xen specific.  If it were, I'd 
expect to see failures all over the place, rather than in just in this 
one specific place.

I'm concerned there's a lurking bug, particularly if it's a generic race 
or something that happens to be triggered when running under Xen because 
of the timing changes.  I've tried reproducing it in a hvm Xen domain 
(so it's running the normal x86 kernel fully virtualized, but with the 
Xen scheduler, etc).  I didn't see a problem, but it isn't a very 
convincing test one way or the other.

    J

Re: [PATCH] generic ipi function calls: wait on alloc failure fallback

[Ingo Molnar]

* Jeremy Fitzhardinge <jeremy@goop.org> wrote:

>> does this explain the xen64 weirdnesses you've been seeing?
>>   
>
> No, but I haven't seen it lately.  I think the other RCU fixes may 
> have helped.  But it's all a bit of a worry: I didn't have a good 
> theory about what was going wrong, the RCU patches didn't look like 
> they'd fix the symptoms I was seeing.
>
> I've seen it with 32 and 64-bit Xen, but there's nothing about the 
> problem which makes me think it's really Xen specific.  If it were, 
> I'd expect to see failures all over the place, rather than in just in 
> this one specific place.
>
> I'm concerned there's a lurking bug, particularly if it's a generic 
> race or something that happens to be triggered when running under Xen 
> because of the timing changes.  I've tried reproducing it in a hvm Xen 
> domain (so it's running the normal x86 kernel fully virtualized, but 
> with the Xen scheduler, etc).  I didn't see a problem, but it isn't a 
> very convincing test one way or the other.

ok. I doubt there's much we can do at this stage - the code looks fine. 
If it's some recently added core kernel problem sooner or later some 
workload or hw will come about that shows it in a more debuggable 
manner.

	Ingo

Re: [PATCH] generic ipi function calls: wait on alloc failure fallback

[Jeremy Fitzhardinge]

Ingo Molnar wrote:
> ok. I doubt there's much we can do at this stage - the code looks fine. 
> If it's some recently added core kernel problem sooner or later some 
> workload or hw will come about that shows it in a more debuggable 
> manner.

Yep.  It has been rock solid for me lately, even doing traditionally 
bug-inducing things like save/restore.

    J