How to avoid spurious lockdep warnings?

How to avoid spurious lockdep warnings?

[Jeremy Fitzhardinge]

In a Xen system, when a new pagetable is about to be put in use it is 
"pinned", meaning that each page in the pagetable is registered with the 
hypervisor.  This is done in arch/x86/xen/mmu.c:pin_page().

In order to make this efficient, the hypercalls for pinning are batched, 
so that multiple pages are submitted at once in a single multicall.  
While a page is batched pending the hypercall, its corresponding 
pte_lock is held.

This means that the code can end up holding multiple pte locks at once, 
though it is guaranteed to never try to hold the same lock at once.  
However, because these locks are in the same lock class, I get a 
spurious warning from lockdep.  Is there some way I can get rid of this 
warning?

=============================================
[ INFO: possible recursive locking detected ]
2.6.25-rc6-x86-latest.git-dirty #297
---------------------------------------------
init/1 is trying to acquire lock:
 (__pte_lockptr(page)){--..}, at: [<c0105038>] pin_page+0x6a/0x167

but task is already holding lock:
 (__pte_lockptr(page)){--..}, at: [<c0105038>] pin_page+0x6a/0x167

other info that might help us debug this:
4 locks held by init/1:
 #0:  (&mm->mmap_sem){----}, at: [<c012645c>] copy_process+0x97e/0x122f
 #1:  (&mm->mmap_sem/1){--..}, at: [<c012646c>] copy_process+0x98e/0x122f
 #2:  (&mm->page_table_lock){--..}, at: [<c0104bf5>] xen_dup_mmap+0x11/0x24
 #3:  (__pte_lockptr(page)){--..}, at: [<c0105038>] pin_page+0x6a/0x167

stack backtrace:
Pid: 1, comm: init Not tainted 2.6.25-rc6-x86-latest.git-dirty #297
 [<c0144049>] __lock_acquire+0x821/0xb50
 [<c01443ee>] lock_acquire+0x76/0x9d
 [<c0105038>] ? pin_page+0x6a/0x167
 [<c0454537>] _spin_lock+0x23/0x32
 [<c0105038>] ? pin_page+0x6a/0x167
 [<c0105038>] pin_page+0x6a/0x167
 [<c0104520>] pgd_walk+0x18f/0x1e1
 [<c0104fce>] ? pin_page+0x0/0x167
 [<c0104b20>] xen_pgd_pin+0x46/0x10a
 [<c0104bfd>] xen_dup_mmap+0x19/0x24
 [<c0126619>] copy_process+0xb3b/0x122f
 [<c0126e6d>] do_fork+0xab/0x1dd
 [<c017341b>] ? vfs_write+0xf1/0x108
 [<c010877a>] ? sysenter_past_esp+0xba/0xc8
 [<c0106cd5>] sys_clone+0x1f/0x21
 [<c0108731>] sysenter_past_esp+0x71/0xc8
 =======================


Thanks,
    J

Re: How to avoid spurious lockdep warnings?

[Peter Zijlstra]

On Thu, 2008-03-20 at 16:02 -0700, Jeremy Fitzhardinge wrote:
> In a Xen system, when a new pagetable is about to be put in use it is 
> "pinned", meaning that each page in the pagetable is registered with the 
> hypervisor.  This is done in arch/x86/xen/mmu.c:pin_page().
> 
> In order to make this efficient, the hypercalls for pinning are batched, 
> so that multiple pages are submitted at once in a single multicall.  
> While a page is batched pending the hypercall, its corresponding 
> pte_lock is held.
> 
> This means that the code can end up holding multiple pte locks at once, 
> though it is guaranteed to never try to hold the same lock at once.  
> However, because these locks are in the same lock class, I get a 
> spurious warning from lockdep.  Is there some way I can get rid of this 
> warning?

How many locks at once?

Re: How to avoid spurious lockdep warnings?

[Jeremy Fitzhardinge]

Peter Zijlstra wrote:
> On Thu, 2008-03-20 at 16:02 -0700, Jeremy Fitzhardinge wrote:
>   
>> In a Xen system, when a new pagetable is about to be put in use it is 
>> "pinned", meaning that each page in the pagetable is registered with the 
>> hypervisor.  This is done in arch/x86/xen/mmu.c:pin_page().
>>
>> In order to make this efficient, the hypercalls for pinning are batched, 
>> so that multiple pages are submitted at once in a single multicall.  
>> While a page is batched pending the hypercall, its corresponding 
>> pte_lock is held.
>>
>> This means that the code can end up holding multiple pte locks at once, 
>> though it is guaranteed to never try to hold the same lock at once.  
>> However, because these locks are in the same lock class, I get a 
>> spurious warning from lockdep.  Is there some way I can get rid of this 
>> warning?
>>     
>
> How many locks at once?

(We discussed this, but for the record...)

The main limit is the batch size, which is currently 32.  There's 
nothing magic about this number, so it may change (I can't imagine it 
getting much larger however, since a 32x mitigation of hypercall 
overhead is already pretty good).

    J

Re: How to avoid spurious lockdep warnings?

[Arjan van de Ven]

On Thu, 20 Mar 2008 16:02:11 -0700
Jeremy Fitzhardinge <jeremy@goop.org> wrote:

> In a Xen system, when a new pagetable is about to be put in use it is 
> "pinned", meaning that each page in the pagetable is registered with
> the hypervisor.  This is done in arch/x86/xen/mmu.c:pin_page().
> 
> In order to make this efficient, the hypercalls for pinning are
> batched, so that multiple pages are submitted at once in a single
> multicall. While a page is batched pending the hypercall, its
> corresponding pte_lock is held.
> 
> This means that the code can end up holding multiple pte locks at
> once, though it is guaranteed to never try to hold the same lock at
> once. However, because these locks are in the same lock class, I get
> a spurious warning from lockdep.  Is there some way I can get rid of
> this warning?


what's the ordering guarantee between these locks ?


-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: How to avoid spurious lockdep warnings?

[Jeremy Fitzhardinge]

Arjan van de Ven wrote:
> On Thu, 20 Mar 2008 16:02:11 -0700
> Jeremy Fitzhardinge <jeremy@goop.org> wrote:
>
>   
>> In a Xen system, when a new pagetable is about to be put in use it is 
>> "pinned", meaning that each page in the pagetable is registered with
>> the hypervisor.  This is done in arch/x86/xen/mmu.c:pin_page().
>>
>> In order to make this efficient, the hypercalls for pinning are
>> batched, so that multiple pages are submitted at once in a single
>> multicall. While a page is batched pending the hypercall, its
>> corresponding pte_lock is held.
>>
>> This means that the code can end up holding multiple pte locks at
>> once, though it is guaranteed to never try to hold the same lock at
>> once. However, because these locks are in the same lock class, I get
>> a spurious warning from lockdep.  Is there some way I can get rid of
>> this warning?
>>     
>
>
> what's the ordering guarantee between these locks ?
>   

Pagetable virtual address order.  move_ptes can also lock two ptes 
within one pagetable, without regard to order, but they're protected by 
mmap_sem, which is also held during pinning.

    J