vmalloc_sync_all() patch problems?

vmalloc_sync_all() patch problems?

[Christopher S. Aker]

4.1.2-rc3-pre @ 23159
3.0.4 + vmalloc_sync_all() patch

While running my Xen test suite:

BUG: unable to handle kernel paging request at bffffd68
IP: [<c102bb51>] vmalloc_sync_all+0x141/0x1e0
*pdpt = 0000000027d71027 *pde = 0000000000000000
Oops: 0000 [#1] SMP
Modules linked in: ebt_arp ip6t_rt ebt_mark ebt_limit ip6table_mangle 
xt_mark ebtable_nat ebtable_filter
Pid: 39, comm: xenwatch Not tainted 3.0.4-1 #1 Supermicro X8DTU/X8DTU
EIP: 0061:[<c102bb51>] EFLAGS: 00010283 CPU: 3
EIP is at vmalloc_sync_all+0x141/0x1e0
EAX: bffffd68 EBX: efe5a720 ECX: 00000008 EDX: 00000001
ESI: ea082a34 EDI: c1a73d68 EBP: eb1d9e80 ESP: eb1d9e4c
  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
Process xenwatch (pid: 39, ti=eb1d8000 task=eb11b0c0 task.ti=eb1d8000)
Stack:
  2b048000 00000000 2b048000 00000000 fffff001 00000008 c0000d68 7bf85067
  f5a00000 00000018 ea254480 eb1d9f14 eb1d9f54 eb1d9e94 c10e5212 c10e35a0
  00000000 e7d3d600 eb1d9ee4 c1446275 eb1d9ea8 c13932c0 e7d3d600 00000011
Call Trace:
  [<c10e5212>] alloc_vm_area+0x42/0x60
  [<c10e35a0>] ? is_vmalloc_or_module_addr+0x50/0x50
  [<c1446275>] xen_blkif_map+0x35/0x1e0
  [<c13932c0>] ? xen_evtchn_do_upcall+0x20/0x30
  [<c1446a61>] frontend_changed+0x261/0x2d0
  [<c139852d>] xenbus_otherend_changed+0x7d/0x90
  [<c1398732>] frontend_changed+0x12/0x20
  [<c1396aa5>] xenwatch_thread+0x85/0x130
  [<c10625d0>] ? wake_up_bit+0x60/0x60
  [<c1396a20>] ? split+0xd0/0xd0
  [<c10621e4>] kthread+0x74/0x80
  [<c1062170>] ? kthread_worker_fn+0x160/0x160
  [<c16ca2b6>] kernel_thread_helper+0x6/0x10
Code: c1 89 c7 81 e7 00 f0 ff ff 03 7d e4 8b 17 89 55 e8 8b 4f 04 83 e2 
01 89 4d e0 0f 84 68 ff ff ff 8b 45 dc 25 00 f0 ff ff 03 45 e4 <8b> 08 
8b 50 04 f6 c1 01 89 55 dc 74 7a 8b 55 dc 89 c8 ff 15 14
EIP: [<c102bb51>] vmalloc_sync_all+0x141/0x1e0 SS:ESP 0069:eb1d9e4c
CR2: 00000000bffffd68
---[ end trace 486c192808e46938 ]---
INFO: rcu_sched_state detected stall on CPU 7 (t=60000 jiffies)
INFO: rcu_sched_state detected stall on CPU 7 (t=240030 jiffies)
... and so on

-Chris

Re: vmalloc_sync_all() patch problems?

[David Vrabel]

On 28/09/11 15:28, Christopher S. Aker wrote:
> 4.1.2-rc3-pre @ 23159
> 3.0.4 + vmalloc_sync_all() patch

This is surprising as that vmalloc_sync_all() patch is a revert of a
recent change only present in 3.0 and others have reported the patch
works for them.

You're going to have to provide more information on your system and
tests I think.

David

> While running my Xen test suite:
> 
> BUG: unable to handle kernel paging request at bffffd68
> IP: [<c102bb51>] vmalloc_sync_all+0x141/0x1e0
> *pdpt = 0000000027d71027 *pde = 0000000000000000
> Oops: 0000 [#1] SMP
> Modules linked in: ebt_arp ip6t_rt ebt_mark ebt_limit ip6table_mangle
> xt_mark ebtable_nat ebtable_filter
> Pid: 39, comm: xenwatch Not tainted 3.0.4-1 #1 Supermicro X8DTU/X8DTU
> EIP: 0061:[<c102bb51>] EFLAGS: 00010283 CPU: 3
> EIP is at vmalloc_sync_all+0x141/0x1e0
> EAX: bffffd68 EBX: efe5a720 ECX: 00000008 EDX: 00000001
> ESI: ea082a34 EDI: c1a73d68 EBP: eb1d9e80 ESP: eb1d9e4c
>  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
> Process xenwatch (pid: 39, ti=eb1d8000 task=eb11b0c0 task.ti=eb1d8000)
> Stack:
>  2b048000 00000000 2b048000 00000000 fffff001 00000008 c0000d68 7bf85067
>  f5a00000 00000018 ea254480 eb1d9f14 eb1d9f54 eb1d9e94 c10e5212 c10e35a0
>  00000000 e7d3d600 eb1d9ee4 c1446275 eb1d9ea8 c13932c0 e7d3d600 00000011
> Call Trace:
>  [<c10e5212>] alloc_vm_area+0x42/0x60
>  [<c10e35a0>] ? is_vmalloc_or_module_addr+0x50/0x50
>  [<c1446275>] xen_blkif_map+0x35/0x1e0
>  [<c13932c0>] ? xen_evtchn_do_upcall+0x20/0x30
>  [<c1446a61>] frontend_changed+0x261/0x2d0
>  [<c139852d>] xenbus_otherend_changed+0x7d/0x90
>  [<c1398732>] frontend_changed+0x12/0x20
>  [<c1396aa5>] xenwatch_thread+0x85/0x130
>  [<c10625d0>] ? wake_up_bit+0x60/0x60
>  [<c1396a20>] ? split+0xd0/0xd0
>  [<c10621e4>] kthread+0x74/0x80
>  [<c1062170>] ? kthread_worker_fn+0x160/0x160
>  [<c16ca2b6>] kernel_thread_helper+0x6/0x10
> Code: c1 89 c7 81 e7 00 f0 ff ff 03 7d e4 8b 17 89 55 e8 8b 4f 04 83 e2
> 01 89 4d e0 0f 84 68 ff ff ff 8b 45 dc 25 00 f0 ff ff 03 45 e4 <8b> 08
> 8b 50 04 f6 c1 01 89 55 dc 74 7a 8b 55 dc 89 c8 ff 15 14
> EIP: [<c102bb51>] vmalloc_sync_all+0x141/0x1e0 SS:ESP 0069:eb1d9e4c
> CR2: 00000000bffffd68
> ---[ end trace 486c192808e46938 ]---
> INFO: rcu_sched_state detected stall on CPU 7 (t=60000 jiffies)
> INFO: rcu_sched_state detected stall on CPU 7 (t=240030 jiffies)
> ... and so on
> 
> -Chris
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: vmalloc_sync_all() patch problems?

[Christopher S. Aker]

On 9/28/11 10:45 AM, David Vrabel wrote:
> You're going to have to provide more information on your system and
> tests I think.

Nothing crazy.  64 bit Xen, 32 bit dom0, my test suite creates many 
domUs (in this case about 40) each with a root image and swap image. 
Some swap thrash, some spin cpu, some are repeatedly shut down or xm 
destroyed.  No networking.  This particular box has about 32G in it, and 
itself and many other boxes identical to it have no problem with our old 
stack (xen 3.4, 2.6.18 dom0).

I've restarted the tests to see if I can reproduce, but I'm certain that 
if it happened once, it'll happen again.

Can I provide anything else?

-Chris

Re: vmalloc_sync_all() patch problems?

[David Vrabel]

On 28/09/11 17:02, Christopher S. Aker wrote:
> On 9/28/11 10:45 AM, David Vrabel wrote:
>> You're going to have to provide more information on your system and
>> tests I think.
> 
> Nothing crazy.  64 bit Xen, 32 bit dom0, my test suite creates many
> domUs (in this case about 40) each with a root image and swap image.
> Some swap thrash, some spin cpu, some are repeatedly shut down or xm
> destroyed.  No networking.  This particular box has about 32G in it, and
> itself and many other boxes identical to it have no problem with our old
> stack (xen 3.4, 2.6.18 dom0).
> 
> I've restarted the tests to see if I can reproduce, but I'm certain that
> if it happened once, it'll happen again.

Instead of the vmalloc_sync_all() patch you could try this series instead.

http://lists.xensource.com/archives/html/xen-devel/2011-09/msg01343.html

David

Re: vmalloc_sync_all() patch problems?

[Christopher S. Aker]

On 9/28/11 1:04 PM, David Vrabel wrote:
> On 28/09/11 17:02, Christopher S. Aker wrote:
>> I've restarted the tests to see if I can reproduce, but I'm certain that
>> if it happened once, it'll happen again.

I was able to reproduce it.

> Instead of the vmalloc_sync_all() patch you could try this series instead.
>
> http://lists.xensource.com/archives/html/xen-devel/2011-09/msg01343.html

I'll rebuild using this series and reset the tests.  Will let you know!

Thanks,
-Chris

Re: vmalloc_sync_all() patch problems?

[Konrad Rzeszutek Wilk]

On Fri, Sep 30, 2011 at 11:38:23AM -0400, Christopher S. Aker wrote:
> On 9/28/11 1:04 PM, David Vrabel wrote:
> >On 28/09/11 17:02, Christopher S. Aker wrote:
> >>I've restarted the tests to see if I can reproduce, but I'm certain that
> >>if it happened once, it'll happen again.
> 
> I was able to reproduce it.
> 
> >Instead of the vmalloc_sync_all() patch you could try this series instead.
> >
> >http://lists.xensource.com/archives/html/xen-devel/2011-09/msg01343.html
> 
> I'll rebuild using this series and reset the tests.  Will let you know!

ping?

Re: vmalloc_sync_all() patch problems?

[Christopher S. Aker]

On Oct 10, 2011, at 11:51 AM, Konrad Rzeszutek Wilk wrote:
> On Fri, Sep 30, 2011 at 11:38:23AM -0400, Christopher S. Aker wrote:
>> On 9/28/11 1:04 PM, David Vrabel wrote:
>>> On 28/09/11 17:02, Christopher S. Aker wrote:
>>>> I've restarted the tests to see if I can reproduce, but I'm certain that
>>>> if it happened once, it'll happen again.
>> 
>> I was able to reproduce it.
>> 
>>> Instead of the vmalloc_sync_all() patch you could try this series instead.
>>> 
>>> http://lists.xensource.com/archives/html/xen-devel/2011-09/msg01343.html
>> 
>> I'll rebuild using this series and reset the tests.  Will let you know!
> 
> ping?

With that patchset applied: 6 days of thrashing, 30000 domains cycled, and no crashes.  I'd say it worked.

Cheers,
-Chris

Re: vmalloc_sync_all() patch problems?

[Konrad Rzeszutek Wilk]

On Tue, Oct 11, 2011 at 11:18:12AM -0400, Christopher S. Aker wrote:
> On Oct 10, 2011, at 11:51 AM, Konrad Rzeszutek Wilk wrote:
> > On Fri, Sep 30, 2011 at 11:38:23AM -0400, Christopher S. Aker wrote:
> >> On 9/28/11 1:04 PM, David Vrabel wrote:
> >>> On 28/09/11 17:02, Christopher S. Aker wrote:
> >>>> I've restarted the tests to see if I can reproduce, but I'm certain that
> >>>> if it happened once, it'll happen again.
> >> 
> >> I was able to reproduce it.
> >> 
> >>> Instead of the vmalloc_sync_all() patch you could try this series instead.
> >>> 
> >>> http://lists.xensource.com/archives/html/xen-devel/2011-09/msg01343.html
> >> 
> >> I'll rebuild using this series and reset the tests.  Will let you know!
> > 
> > ping?
> 
> With that patchset applied: 6 days of thrashing, 30000 domains cycled, and no crashes.  I'd say it worked.

Excellent. You got any more bugs that we should be aware off ? (hopefully not).

Re: vmalloc_sync_all() patch problems?

[Christopher S. Aker]

On 10/11/11 1:28 PM, Konrad Rzeszutek Wilk wrote:
>> With that patchset applied: 6 days of thrashing, 30000 domains
>> cycled, and no crashes.  I'd say it worked.
>
> Excellent. You got any more bugs that we should be aware off ?
> (hopefully not).

Always! I'll gather the info and author posts over the next couple of days.

-Chris