kernel >= 3.16 saturates I/O endlessly and without reason

kernel >= 3.16 saturates I/O endlessly and without reason

[Julien Muchembled]

[-- Attachment #1: Type: text/plain, Size: 1651 bytes --]

Hello,

I upgraded my kernel from 3.15.10 to 3.17.2 and after a while, btrfs kworkers started to read/write data at maximum speed for hours whereas I was doing nothing. So I rebooted with kernel 3.15.10 and hopefully everything went back to normal. Later attempts to upgrade to 3.16.7 or 3.17.2 showed the same issue, immediately at boot or after a while.

In order to investigate a little, I copied the partition and mounted it in qemu (snapshot mode, with cow file in tmpfs, debian kernel 3.16.7-2). I could see that btrfs always wrote the same ~8MB in loop. qemu used all cpu and when I stopped it, it wrote 190G and the generation number raised from 625955 to 853922.

I dumped the FS with btrfs-image. The restored FS has the same issue so I can send the image (78M) to anyone who would like to debug.

$ btrfs fi df /
Data, single: total=13.62GiB, used=12.60GiB
System, single: total=4.00MiB, used=16.00KiB
Metadata, single: total=1.00GiB, used=747.48MiB
GlobalReserve, single: total=256.00MiB, used=0.00B

$ btrfs fi show --mounted
Label: none  uuid: f5a13768-b968-49c7-8a09-fdad88b83ea8
        Total devices 1 FS bytes used 13.33GiB
        devid    1 size 14.62GiB used 14.62GiB path /dev/sdb2

$ btrfs sub list -a /
ID 257 gen 626025 top level 5 path <FS_TREE>/@
ID 269 gen 28 top level 257 path @/usr/src/linux-sysrcd
ID 278 gen 599690 top level 257 path @/usr/src/linux-3.15-1
ID 279 gen 13003 top level 257 path @/usr/src/python3.3
ID 281 gen 13036 top level 257 path @/usr/src/python3.4
ID 282 gen 614068 top level 257 path @/usr/src/linux-3.16-1
ID 283 gen 599642 top level 257 path @/usr/src/linux-3.17-1


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: kernel >= 3.16 saturates I/O endlessly and without reason

[Chris Mason]

On Tue, Nov 11, 2014 at 3:10 PM, Julien Muchembled <jm@jmuchemb.eu> 
wrote:
> Hello,
> 
> I upgraded my kernel from 3.15.10 to 3.17.2 and after a while, btrfs 
> kworkers started to read/write data at maximum speed for hours 
> whereas I was doing nothing. So I rebooted with kernel 3.15.10 and 
> hopefully everything went back to normal. Later attempts to upgrade 
> to 3.16.7 or 3.17.2 showed the same issue, immediately at boot or 
> after a while.
> 
> In order to investigate a little, I copied the partition and mounted 
> it in qemu (snapshot mode, with cow file in tmpfs, debian kernel 
> 3.16.7-2). I could see that btrfs always wrote the same ~8MB in loop. 
> qemu used all cpu and when I stopped it, it wrote 190G and the 
> generation number raised from 625955 to 853922.
> 
> I dumped the FS with btrfs-image. The restored FS has the same issue 
> so I can send the image (78M) to anyone who would like to debug.

Which mount options are you using?  This sounds like something in 
autodefrag...

-chris

Re: kernel >= 3.16 saturates I/O endlessly and without reason

[Julien Muchembled]

Le 11/11/14 21:30, Chris Mason a écrit :
> On Tue, Nov 11, 2014 at 3:10 PM, Julien Muchembled <jm@jmuchemb.eu> wrote:
>> I upgraded my kernel from 3.15.10 to 3.17.2 and after a while, btrfs kworkers started to read/write data at maximum speed for hours whereas I was doing nothing. So I rebooted with kernel 3.15.10 and hopefully everything went back to normal. Later attempts to upgrade to 3.16.7 or 3.17.2 showed the same issue, immediately at boot or after a while.
>>
>> In order to investigate a little, I copied the partition and mounted it in qemu (snapshot mode, with cow file in tmpfs, debian kernel 3.16.7-2). I could see that btrfs always wrote the same ~8MB in loop. qemu used all cpu and when I stopped it, it wrote 190G and the generation number raised from 625955 to 853922.
>>
>> I dumped the FS with btrfs-image. The restored FS has the same issue so I can send the image (78M) to anyone who would like to debug.
> 
> Which mount options are you using?  This sounds like something in autodefrag...

I've never used autodefrag.
In qemu, I did:

  mount -t btrfs -o noatime /dev/sda /mnt/sda