btrfs issues in 3.14

btrfs issues in 3.14

[Kenny MacDermid]

[-- Attachment #1: Type: text/plain, Size: 1107 bytes --]

Hello,

I've been having a number of issues with processes hanging due to
btrfs using 3.14 kernels. This seems pretty new as it has been working
fine before. I also rebuilt the filesystem and am still receiving
hangs.

The filesystem is running on dmcrypt which is running on lvm2 which is
running on an SSD (SAMSUNG MZMTD256HAGM-000L1).

When the issue occurs the process is unable to be killed and the
system will not fully shutdown.

$ uname -a
Linux orange 3.14.2-1-ARCH #1 SMP PREEMPT Sun Apr 27 11:28:44 CEST
2014 x86_64 GNU/Linux

$ btrfs --version
Btrfs v3.14.1

$ btrfs fi show
Btrfs v3.14.1

$ btrfs fi df /home
Data, single: total=71.01GiB, used=68.72GiB
System, DUP: total=8.00MiB, used=16.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, DUP: total=1.50GiB, used=863.33MiB
Metadata, single: total=8.00MiB, used=0.00

I opened bugs 75181 and 75191 and I'll include the relevant journalctl
entries. The kernel was upgraded from 3.14.1-1 to 3.14.2-1 during this
time, and the filesystem was rebuilt after the orphan issue.

I'm not on this list so please CC me on replies.

Thanks,

Kenny

[-- Attachment #2: journal.txt.gz --]
[-- Type: application/x-gzip, Size: 8132 bytes --]

Re: btrfs issues in 3.14

[Liu Bo]

On Tue, May 06, 2014 at 08:49:04PM -0300, Kenny MacDermid wrote:
> Hello,
> 
> I've been having a number of issues with processes hanging due to
> btrfs using 3.14 kernels. This seems pretty new as it has been working
> fine before. I also rebuilt the filesystem and am still receiving
> hangs.
> 
> The filesystem is running on dmcrypt which is running on lvm2 which is
> running on an SSD (SAMSUNG MZMTD256HAGM-000L1).
> 
> When the issue occurs the process is unable to be killed and the
> system will not fully shutdown.
> 
> $ uname -a
> Linux orange 3.14.2-1-ARCH #1 SMP PREEMPT Sun Apr 27 11:28:44 CEST
> 2014 x86_64 GNU/Linux
> 
> $ btrfs --version
> Btrfs v3.14.1
> 
> $ btrfs fi show
> Btrfs v3.14.1
> 
> $ btrfs fi df /home
> Data, single: total=71.01GiB, used=68.72GiB
> System, DUP: total=8.00MiB, used=16.00KiB
> System, single: total=4.00MiB, used=0.00
> Metadata, DUP: total=1.50GiB, used=863.33MiB
> Metadata, single: total=8.00MiB, used=0.00
> 
> I opened bugs 75181 and 75191 and I'll include the relevant journalctl
> entries. The kernel was upgraded from 3.14.1-1 to 3.14.2-1 during this
> time, and the filesystem was rebuilt after the orphan issue.
> 
> I'm not on this list so please CC me on replies.

What does sysrq+w say when the hang happens?

-liubo

Re: btrfs issues in 3.14

[Kenny MacDermid]

[-- Attachment #1: Type: text/plain, Size: 444 bytes --]

On Tue, May 6, 2014 at 11:22 PM, Liu Bo <bo.li.liu@oracle.com> wrote:
>
> What does sysrq+w say when the hang happens?

The whole system isn't hung, I may have explained that wrong. The
system will hang if I try to shutdown, and the process will hang if I
try to kill -9 it.

It looks like the browser is in this state currently so I did an 'echo
w >/proc/sysrq-trigger' and have attached the full dmesg with the
browser issues and the output.

[-- Attachment #2: dmesg.txt.gz --]
[-- Type: application/x-gzip, Size: 44184 bytes --]

Re: btrfs issues in 3.14

[Kenny MacDermid]

On Wed, May 7, 2014 at 9:35 AM, Kenny MacDermid
<kenny.macdermid@gmail.com> wrote:
> On Tue, May 6, 2014 at 11:22 PM, Liu Bo <bo.li.liu@oracle.com> wrote:
>>
>> What does sysrq+w say when the hang happens?
>
> The whole system isn't hung, I may have explained that wrong. The
> system will hang if I try to shutdown, and the process will hang if I
> try to kill -9 it.
>
> It looks like the browser is in this state currently so I did an 'echo
> w >/proc/sysrq-trigger' and have attached the full dmesg with the
> browser issues and the output.

I had to hard reboot to clear that issue, and I decided to do another
'btrfs check' while /home was unmounted. It generated the following
output:

checking extents
checking free space cache
Wanted bytes 45056, found 32768 for off 63805808640
Wanted bytes 111190016, found 32768 for off 63805808640
cache appears valid but isnt 62843256832
Checking filesystem on //dev/mapper/home
UUID: 9a60a25f-eeb4-494c-b1af-ebd8e4f79b6b
found 13672418478 bytes used err is -22
total csum bytes: 72089212
total tree bytes: 906100736
total fs tree bytes: 808370176
total extent tree bytes: 18153472
btree space waste bytes: 116247440
file data blocks allocated: 101046853632
 referenced 73680674816
Btrfs v3.14.1

This is on the new filesystem. I redid the dmcrypt and the lvm lv when
I recreated the filesystem as well, so it's less than a week old.
Before rebuilding the old was was telling me:

Checking filesystem on /dev/mapper/home
UUID: 4f5d7a10-d003-48a7-a901-bf22d534888f
free space inode generation (0) did not match free space cache
generation (115200)
found 29963117667 bytes used err is 1
total csum bytes: 63740440
total tree bytes: 745504768
total fs tree bytes: 624951296
total extent tree bytes: 36749312
btree space waste bytes: 119018687
file data blocks allocated: 181026942976
 referenced 73759866880
Btrfs v0.20-rc1-358-g194aa4a-dirty

and

checking extents
checking free space cache
checking fs roots
root 257 inode 29647 errors 200, dir isize wrong
root 257 inode 391917 errors 200, dir isize wrong
root 257 inode 497392 errors 410, odd dir item, nbytes wrong
Checking filesystem on /dev/mapper/home
UUID: 4f5d7a10-d003-48a7-a901-bf22d534888f
free space inode generation (0) did not match free space cache
generation (115200)
found 31310902624 bytes used err is 1
total csum bytes: 63579480
total tree bytes: 743342080
total fs tree bytes: 623198208
total extent tree bytes: 36601856
btree space waste bytes: 118906643
file data blocks allocated: 180831965184
 referenced 73631731712
Btrfs v3.14

Re: btrfs issues in 3.14

[Liu Bo]

On Wed, May 07, 2014 at 09:35:06AM -0300, Kenny MacDermid wrote:
> On Tue, May 6, 2014 at 11:22 PM, Liu Bo <bo.li.liu@oracle.com> wrote:
> >
> > What does sysrq+w say when the hang happens?
> 
> The whole system isn't hung, I may have explained that wrong. The
> system will hang if I try to shutdown, and the process will hang if I
> try to kill -9 it.
> 
> It looks like the browser is in this state currently so I did an 'echo
> w >/proc/sysrq-trigger' and have attached the full dmesg with the
> browser issues and the output.

Those stacks show the blocked tasks are waiting for a page's writeback, but
they don't show what blocks the endio process of that page.

I'd recommand you to try the lastest 3.15.0-rc4 or btrfs-next, as many fixes
are merged during this period.

thanks,
-liubo

Re: btrfs issues in 3.14

[Kenny MacDermid]

On Wed, May 7, 2014 at 11:48 PM, Liu Bo <bo.li.liu@oracle.com> wrote:
>
> On Wed, May 07, 2014 at 09:35:06AM -0300, Kenny MacDermid wrote:
> > On Tue, May 6, 2014 at 11:22 PM, Liu Bo <bo.li.liu@oracle.com> wrote:
> > >
> > > What does sysrq+w say when the hang happens?
> >
> > The whole system isn't hung, I may have explained that wrong. The
> > system will hang if I try to shutdown, and the process will hang if I
> > try to kill -9 it.
> >
> > It looks like the browser is in this state currently so I did an 'echo
> > w >/proc/sysrq-trigger' and have attached the full dmesg with the
> > browser issues and the output.
>
> Those stacks show the blocked tasks are waiting for a page's writeback, but
> they don't show what blocks the endio process of that page.
>
> I'd recommand you to try the lastest 3.15.0-rc4 or btrfs-next, as many fixes
> are merged during this period.
>

Thank you, I upgraded to the Arch package for 3.15.0-1-mainline (it's
rc4) and I'll let you know if the errors reoccur.

Should the filesystem be rebuilt again?

A 'btrfs check' of it returned:

checking extents
checking free space cache
checking fs roots
checking csums
checking root refs
Checking filesystem on /dev/mapper/home
UUID: 9a60a25f-eeb4-494c-b1af-ebd8e4f79b6b
free space inode generation (0) did not match free space cache generation (6409)
free space inode generation (0) did not match free space cache generation (6397)
found 41686685877 bytes used err is 0
total csum bytes: 74074632
total tree bytes: 907673600
total fs tree bytes: 807567360
total extent tree bytes: 18251776
btree space waste bytes: 116552179
file data blocks allocated: 112191107072
 referenced 75535110144
Btrfs v3.14.1

Re: btrfs issues in 3.14

[Liu Bo]

On Thu, May 08, 2014 at 10:51:03AM -0300, Kenny MacDermid wrote:
> On Wed, May 7, 2014 at 11:48 PM, Liu Bo <bo.li.liu@oracle.com> wrote:
> >
> > On Wed, May 07, 2014 at 09:35:06AM -0300, Kenny MacDermid wrote:
> > > On Tue, May 6, 2014 at 11:22 PM, Liu Bo <bo.li.liu@oracle.com> wrote:
> > > >
> > > > What does sysrq+w say when the hang happens?
> > >
> > > The whole system isn't hung, I may have explained that wrong. The
> > > system will hang if I try to shutdown, and the process will hang if I
> > > try to kill -9 it.
> > >
> > > It looks like the browser is in this state currently so I did an 'echo
> > > w >/proc/sysrq-trigger' and have attached the full dmesg with the
> > > browser issues and the output.
> >
> > Those stacks show the blocked tasks are waiting for a page's writeback, but
> > they don't show what blocks the endio process of that page.
> >
> > I'd recommand you to try the lastest 3.15.0-rc4 or btrfs-next, as many fixes
> > are merged during this period.
> >
> 
> Thank you, I upgraded to the Arch package for 3.15.0-1-mainline (it's
> rc4) and I'll let you know if the errors reoccur.

FYI, this patch seems to address your problem.

Btrfs: fix hang on error (such as ENOSPC) when writing extent pages
https://patchwork.kernel.org/patch/4139971/

-liubo

> 
> Should the filesystem be rebuilt again?
> 
> A 'btrfs check' of it returned:
> 
> checking extents
> checking free space cache
> checking fs roots
> checking csums
> checking root refs
> Checking filesystem on /dev/mapper/home
> UUID: 9a60a25f-eeb4-494c-b1af-ebd8e4f79b6b
> free space inode generation (0) did not match free space cache generation (6409)
> free space inode generation (0) did not match free space cache generation (6397)
> found 41686685877 bytes used err is 0
> total csum bytes: 74074632
> total tree bytes: 907673600
> total fs tree bytes: 807567360
> total extent tree bytes: 18251776
> btree space waste bytes: 116552179
> file data blocks allocated: 112191107072
>  referenced 75535110144
> Btrfs v3.14.1