Re: Lockups with btrfs on 3.16-rc1 - bisected

[Chris Mason <clm@fb.com>]

On 06/18/2014 07:19 PM, Waiman Long wrote:
> On 06/18/2014 07:10 PM, Josef Bacik wrote:
>>
>>
>> On 06/18/2014 03:47 PM, Waiman Long wrote:
>>> On 06/18/2014 06:27 PM, Josef Bacik wrote:
>>>>
>>>>
>>>> On 06/18/2014 03:17 PM, Waiman Long wrote:
>>>>> On 06/18/2014 04:57 PM, Marc Dionne wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I've been seeing very reproducible soft lockups with 3.16-rc1 similar
>>>>>> to what is reported here:
>>>>>> https://urldefense.proofpoint.com/v1/url?u=http://marc.info/?l%3Dlinux-btrfs%26m%3D140290088532203%26w%3D2&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=cKCbChRKsMpTX8ybrSkonQ%3D%3D%0A&m=aoagvtZMwVb16gh1HApZZL00I7eP50GurBpuEo3l%2B5g%3D%0A&s=c62558feb60a480bbb52802093de8c97b5e1f23d4100265b6120c8065bd99565
>>>>>>
>>>>>>
>>>>>> , along with the
>>>>>> occasional hard lockup, making it impossible to complete a parallel
>>>>>> build on a btrfs filesystem for the package I work on.  This was
>>>>>> working fine just a few days before rc1.
>>>>>>
>>>>>> Bisecting brought me to the following commit:
>>>>>>
>>>>>>    commit bd01ec1a13f9a327950c8e3080096446c7804753
>>>>>>    Author: Waiman Long<Waiman.Long@hp.com>
>>>>>>    Date:   Mon Feb 3 13:18:57 2014 +0100
>>>>>>
>>>>>>        x86, locking/rwlocks: Enable qrwlocks on x86
>>>>>>
>>>>>> And sure enough if I revert that commit on top of current mainline,
>>>>>> I'm unable to reproduce the soft lockups and hangs.
>>>>>>
>>>>>> Marc
>>>>>
>>>>> The queue rwlock is fair. As a result, recursive read_lock is not
>>>>> allowed unless the task is in an interrupt context. Doing recursive
>>>>> read_lock will hang the process when a write_lock happens somewhere in
>>>>> between. Are recursive read_lock being done in the btrfs code?
>>>>>
>>>>
>>>> We walk down a tree and read lock each node as we walk down, is that
>>>> what you mean?  Or do you mean read_lock multiple times on the same
>>>> lock in the same process, cause we definitely don't do that.  Thanks,
>>>>
>>>> Josef
>>>
>>> I meant recursively read_lock the same lock in a process.
>>
>> I take it back, we do actually do this in some cases.  Thanks,
>>
>> Josef
> 
> This is what I thought when I looked at the looking code in btrfs. The
> unlock code doesn't clear the lock_owner pid, this may cause the
> lock_nested to be set incorrectly.
> 
> Anyway, are you going to do something about it?

Thanks for reporting this, we shouldn't be actually taking the lock
recursively.  Could you please try with lockdep enabled?  If the problem
goes away with lockdep on, I think I know what's causing it.  Otherwise,
lockdep should clue us in.

-chris