Re: Lockups with btrfs on 3.16-rc1 - bisected

[Chris Mason <clm@fb.com>]

On 06/19/2014 01:52 PM, Waiman Long wrote:
> On 06/19/2014 12:51 PM, Chris Mason wrote:
>> On 06/18/2014 11:21 PM, Waiman Long wrote:
>>> On 06/18/2014 10:11 PM, Chris Mason wrote:
>>>> On 06/18/2014 10:03 PM, Marc Dionne wrote:
>>>>> On Wed, Jun 18, 2014 at 8:41 PM, Marc
>>>>> Dionne<marc.c.dionne@gmail.com>   wrote:
>>>>>> On Wed, Jun 18, 2014 at 8:08 PM, Waiman Long<waiman.long@hp.com>
>>>>>> wrote:
>>>>>>> On 06/18/2014 08:03 PM, Marc Dionne wrote:
>>>>> And for an additional data point, just removing those
>>>>> CONFIG_DEBUG_LOCK_ALLOC ifdefs looks like it's sufficient to prevent
>>>>> the symptoms when lockdep is not enabled.
>>>> Ok, somehow we've added a lock inversion here that wasn't here before.
>>>> Thanks for confirming, I'll nail it down.
>>>>
>>>> -chris
>>>>
>>> I am pretty sure that the hangup is caused by the following kind of code
>>> fragment in the locking.c file:
>>>
>>>   if (eb->lock_nested) {
>>>                  read_lock(&eb->lock);
>>>                  if (eb->lock_nested&&  current->pid ==
>>> eb->lock_owner) {
>>>
>>> Is it possible to do the check without taking the read_lock?
>> I think you're right, we haven't added any new recursive takers of the
>> lock.  The path where we are deadlocking has an extent buffer that isn't
>> in the path yet locked.  I think we're taking the read lock while that
>> one is write locked.
>>
>> Reworking the nesting a big here.
>>
>> -chris
> 
> I would like to take back my comments. I took out the read_lock, but the
> process still hang while doing file activities on btrfs filesystem. So
> the problem is trickier than I thought. Below are the stack backtraces
> of some of the relevant processes.
> 

You weren't wrong, but it was also the tree trylock code.  Our trylocks
only back off if the blocking lock is held.  btrfs_next_leaf needs it to
be a true trylock.  The confusing part is this hasn't really changed,
but one of the callers must be a spinner where we used to have a blocker.

-chris