Re: kernel BUG at linux-4.2.0/fs/btrfs/extent-tree.c:1833 on rebalance

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

在 2015年09月20日 18:35, Stéphane Lesimple 写道:
> Le 2015-09-20 03:22, Qu Wenruo a écrit :
>>>> The mentioned steps are as follows :
>>>>
>>>> 0) Rsync data from the next ext4 "snapshot" to the subvolume
>>>> 1) Do 'sync; btrfs qgroup show -⁠prce -⁠-⁠raw' and save the output <==
>>>> 2) Create the needed readonly snapshot on btrfs
>>>> 3) Do 'sync; btrfs qgroup show -⁠prce -⁠-⁠raw' and save the output <==
>>>> 4) Avoid doing IO if possible until step 6)
>>>> 5) Do 'btrfs quota rescan -⁠w' and save it <==
>>>> 6) Do 'sync; btrfs qgroup show -⁠prce -⁠-⁠raw' and save the output <==
>>>>
>>>> The resulting files are available here:
>>>> http://speed47.net/tmp2/qgroup.tar.gz
>>>> The run2 is the more complete one, during run1 the machine crashed
>>>> even faster.
>>>> It's interesting to note, however, that it seems to have crashed the
>>>> same way and at the same step in the process.
>>
>> Your data really helps a lot!!
>>
>> And the good news is, the qgroup accouting part is working as expected.
>> Although I only checked about 1/3/6 of about 5 snaps, they are all OK.
>>
>> I can make a script to cross check them, but from the last few result,
>> I think qgroup works fine.
>>
>> I'm more confident about the minus number, which should be a result of
>> deleted subvolume, and the real problem is, such qgroup is not handled
>> well with qgroup rescan.
>
> I agree with your analysis, this matches what I observed.
>
>> I'll try to add a hot fix for such case if needed.
>> But right now, I don't have a good idea for it until Mark's work of
>> rescan subtree.
>>
>> Maybe I can add a new option for btrfs-progs to automatically remove
>> the qgroup and trigger a rescan?
>
> Until this is properly fixed in the kernel code, and this is good news to
> know Mark and you are working on it, this would be a good workaround yes!
>
>>> [ 5738.172879] Call Trace:
>>> [ 5738.172887]  [<ffffffffc031565b>] btrfs_start_transaction+0x1b/0x20
>>> [btrfs]
>>> [ 5738.172896]  [<ffffffffc0378038>]
>>> btrfs_qgroup_rescan_worker+0x388/0x5a0 [btrfs]
>>
>> Your netconsole backtrace is also of greate value.
>> This one implies that, my rework also caused some stupid bug.
>> (Yeah, I always make such bugs) or some existing unexposed rescan bug.
>>
>> Would you please use gdb to show the codes of
>> "btrfs_qgroup_rescan_worker+0x388" ?
>> (Need kernel debuginfo)
>>
>> My guess is the following line:(pretty sure, but not 100% sure)
>> ------
>> /*
>>          * only update status, since the previous part has alreay
>> updated the
>>          * qgroup info.
>>          */
>>         trans = btrfs_start_transaction(fs_info->quota_root, 1); <<<<<
>>         if (IS_ERR(trans)) {
>>                 err = PTR_ERR(trans);
>>                 btrfs_err(fs_info,
>>                           "fail to start transaction for status
>> update: %d\n",
>>                           err);
>>                 goto done;
>>         }
>> ------
>
> The kernel and modules were already compiled with debuginfo.
> However for some reason, I couldn't get gdb disassembly of /proc/kcore
> properly
> aligned with the source I compiled: the asm code doesn't match the C
> code shown
> by gdb. In any case, watching the source of this function, this is the
> only place
> btrfs_start_transaction is called, so we can be 100% sure it's where the
> crash
> happens indeed.

Yep, that's the only caller.

Here is some useful small hint to locate the code, if you are 
interestied in kernel development.

# Not sure about whether ubuntu gzipped modules, at least Arch does
# compress it
$ cp <kernel modules dir>/kernel/fs/btrfs/btrfs.ko.gz /tmp/
$ gunzip /tmp/btrfs.ko.gz
$ gdb /tmp/btrfs.ko
# Make sure gdb read all the needed debuginfo
$ gdb list *(btrfs_qgroup_rescan_worker+0x388)

And gdb will find the code position for you.
Quite easy one, only backtrace info is needed.

Another hint is about how to collect the kernel crash info.
Your netconsole setup would be definitely one good practice.

Another one I use to collect crash info is kdump.
Ubuntu should have a good wiki on it.

>
>> But that means, at rescan time, fs_info->quota_root is still NULL,
>> which is quite wired.
>> I can add extra check to avoid such NULL pointer for now, but it's
>> better to review the existing rescan workflow, as I think there is
>> some race for it to init quota_root.
>>
>> You can also try the following hotfix patch to see if it works:
>> http://pastebin.com/966GQXPk
>>
>> My concern is, this may cause qgroup rescan to exit without updating
>> its accounting info...
>>
>> So still need your help.
>> Or I can use your reproducer script to test it next Monday.
>
> Compiling with your patch, just amended of a little printk to know if
> the execution
> flow enters the added if condition. Will let you know about the results.
>
>>>> But I'm pretty sure I can get that (u64)-1 value again by deleting
>>>> snapshots. Shall I ? Or do you have something else for me to run
>>>> before that ?
>>
>> You have already done a great job in helping maturing qgroups.
>> The minus number and 0 excl is somewhat expected for deleted snapshots.
>>
>> Good news is, 1) it doesn't affect valid(non-orphan) qgroup.
>> 2) Mark is already working on it.
>>
>> I'll try to add a btrfs-progs hotfix for you to delete and rescan
>> qgroups to avoid such problem.
>
> That would be good !
>
>>>> So, as a quick summary of this big thread, it seems I've been hitting
>>>> 3 bugs, all reproductible :
>>>> - kernel BUG on balance (this original thread)
>>
>> For this, I can't provide much help, as extent backref bug is quite
>> hard to debug, unless a developer is interested in it and find a
>> stable way to reproduce it.
>
> Yes, unfortunately as it looks so much like a race condition, I know I can
> reproduce it with my worflow, but it can take between 1 minute and 12
> hours,
> so I wouldn't call it a "stable way" to reproduce it unfortunately :(
>
> Still if any dev is interested in it, I can reproduce it, with a patched
> kernel if needed.

Maybe you are already doing it, you can only compile the btrfs modules, 
which will be far more faster than compile the whole kernel, if and only 
if the compiled module can be loaded.

Thanks,
Qu
>
>> The rest two are explained or have hot fix mentioned above.
>
> And thanks for that, will keep you posted.
>