Re: __purge_vmap_area_lazy crash with CONFIG_PREEMPT_RCU=y

[Marcelo Tosatti <mtosatti@redhat.com>]

On Tue, Dec 30, 2008 at 02:53:36PM +1100, Nick Piggin wrote:
> >  RSP <ffff88011f4c7be8>
> > ---[ end trace 31811279a2e983e8 ]---
> > note: qemu-system-x86[4440] exited with preempt_count 2
> >
> >
> > (gdb) l *(__purge_vmap_area_lazy + 0x12c)
> > 0xffffffff80289ca2 is in __purge_vmap_area_lazy (mm/vmalloc.c:516).
> > 511     if (nr || force_flush)
> > 512         flush_tlb_kernel_range(*start, *end);
> > 513
> > 514     if (nr) {
> > 515         spin_lock(&vmap_area_lock);
> > 516         list_for_each_entry(va, &valist, purge_list)
> > 517             __free_vmap_area(va);
> > 518         spin_unlock(&vmap_area_lock);
> > 519     }
> > 520     spin_unlock(&purge_lock);
> >
> > 0xffffffff80289c9a <__purge_vmap_area_lazy+292>:    mov 0x40(%rbx),%rax
> > 0xffffffff80289c9e <__purge_vmap_area_lazy+296>:    lea -0x40(%rax),%rbx
> > 0xffffffff80289ca2 <__purge_vmap_area_lazy+300>:    mov 0x40(%rbx),%rax
> >                                                     ^^^^^^^^^^^^^^^^^^^

Note:

RAX: 6b6b6b6b6b6b6b6b RBX: 6b6b6b6b6b6b6b2b

> > 0xffffffff80289ca6 <__purge_vmap_area_lazy+304>:    prefetcht0 (%rax)
> >
> >
> > Which vanishes once PREEMPT_RCU is disabled.
> >
> > Nick? KVM does not make direct use of RCU. Same issue happens if the
> > entire __purge_vmap_area_lazy runs with vmap_area_lock held.

Hum, mmu_notifiers does.

> The thing is that the valist and va->purge_list is protected by
> purge_lock in that function. 

Which disables preemption...

> I can't easily see how that could get corrupted.

Perhaps the corruption happens before the freeing pass.

> Is it easy to reproduce?

Yes. Enable CONFIG_PREEMPT, CONFIG_PREEMPT_RCU, and DEBUG_PREEMPT. Start
a Linux guest, wait for boot to finish, and shut it down. Sometimes
it happens even before guest shutdown, in the complete_pio path as
reported.

> Can you try putting preempt_disable around rcu_read_lock?

Tried it before, does not help. Even tried to protect all
rcu_read_lock/unlock pairs with preempt_disable/enable in vmalloc.c.