6+ MiB/s constant usage on a btrfs volume with kernel 3.16

6+ MiB/s constant usage on a btrfs volume with kernel 3.16

[Oon-Ee Ng]

Hi,

I'm using 64-bit Arch Linux. Since update to kernel versions 3.16 and
3.16.1 I'm getting a constant 6+ MiB/s write on my root. Root does not
seem to fill up though. Has run for 2 hours straight, and obviously
slows everything else down to a crawl.

Downgrading to 3.15.8 (last working version for me) solves the
symptoms. I'm unsure how to check for root causes, and my distro's
mailing list suggested starting here.

btrfs info as shown below, and this dropbox link[1] is a png
printscreen of my iotop and perf top. Please advise.

Btrfs v3.14.2
% btrfs fi show
Label: 'DATA'  uuid: 762a0d46-d88b-4d2a-a4ff-ba9de5ebe993
        Total devices 1 FS bytes used 380.17GiB
        devid    1 size 439.83GiB used 439.83GiB path /dev/sda9

Label: 'HOME'  uuid: 00f91282-ab65-429d-b0f6-2439924755a7
        Total devices 1 FS bytes used 73.03GiB
        devid    1 size 97.66GiB used 97.66GiB path /dev/sda7

Label: 'ROOT'  uuid: 4c860350-eced-4224-a4ec-657576fc54e8
        Total devices 1 FS bytes used 25.49GiB
        devid    1 size 34.18GiB used 34.18GiB path /dev/sda6

Btrfs v3.14.2
% btrfs fi df /
Data, single: total=31.90GiB, used=24.66GiB
System, DUP: total=8.00MiB, used=12.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, DUP: total=1.12GiB, used=853.64MiB
Metadata, single: total=8.00MiB, used=0.00
unknown, single: total=288.00MiB, used=0.00

[1] - https://db.tt/TSh5piq6

Re: 6+ MiB/s constant usage on a btrfs volume with kernel 3.16

[Duncan]

Oon-Ee Ng posted on Sun, 24 Aug 2014 17:55:32 +0800 as excerpted:

> I'm using 64-bit Arch Linux. Since update to kernel versions 3.16 and
> 3.16.1 I'm getting a constant 6+ MiB/s write on my root. Root does not
> seem to fill up though. Has run for 2 hours straight, and obviously
> slows everything else down to a crawl.
> 
> Downgrading to 3.15.8 (last working version for me) solves the symptoms.
> I'm unsure how to check for root causes, and my distro's mailing list
> suggested starting here.

I've seen that a couple of times here, but it always goes away when I 
reboot (and / is ro by default here, so it'd be on one of the other btrfs 
partitions, probably /home or /var/log).

Meanwhile, if you use the compress mount option, you might wish to return 
to 3.14.x temporarily.  There's a bug that can trigger lockups on 3.15+, 
altho a fix should make it to a later 3.16 stable and to 3.17.  The bug 
was a regression for 3.15 so doesn't affect 3.14, and is only a problem 
if you're using (AFAIK or have used) compression.  It's also not as big 
an issue here (on fast ssds with smaller than usual partitions) as it is 
on some people's systems, tho I'm not sure why and I think it has it me a 
time or two, but I've not bothered downgrading as it hasn't been that big 
an issue, here.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: 6+ MiB/s constant usage on a btrfs volume with kernel 3.16

[Oon-Ee Ng]

On Sun, Aug 24, 2014 at 8:57 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> Oon-Ee Ng posted on Sun, 24 Aug 2014 17:55:32 +0800 as excerpted:
>
>> I'm using 64-bit Arch Linux. Since update to kernel versions 3.16 and
>> 3.16.1 I'm getting a constant 6+ MiB/s write on my root. Root does not
>> seem to fill up though. Has run for 2 hours straight, and obviously
>> slows everything else down to a crawl.
>>
>> Downgrading to 3.15.8 (last working version for me) solves the symptoms.
>> I'm unsure how to check for root causes, and my distro's mailing list
>> suggested starting here.
>
> I've seen that a couple of times here, but it always goes away when I
> reboot (and / is ro by default here, so it'd be on one of the other btrfs
> partitions, probably /home or /var/log).

Happens a 100% of the time here, annoyingly. As mentioned, 3.15 was
working for me, and still does. Multiple reboots and it happens
immediately on boot even before gdm comes up.
>
> Meanwhile, if you use the compress mount option, you might wish to return
> to 3.14.x temporarily.  There's a bug that can trigger lockups on 3.15+,
> altho a fix should make it to a later 3.16 stable and to 3.17.  The bug
> was a regression for 3.15 so doesn't affect 3.14, and is only a problem
> if you're using (AFAIK or have used) compression.  It's also not as big
> an issue here (on fast ssds with smaller than usual partitions) as it is
> on some people's systems, tho I'm not sure why and I think it has it me a
> time or two, but I've not bothered downgrading as it hasn't been that big
> an issue, here.

I have never used compression and haven't experienced any lockups.

Re: 6+ MiB/s constant usage on a btrfs volume with kernel 3.16

[Flash ROM]

> Happens a 100% of the time here, annoyingly. As mentioned, 3.15 was
> working for me, and still does. Multiple reboots and it happens
> immediately on boot even before gdm comes up.
Would be logical to do block-level I/O tracing to get idea WHAT is this IO, right? 

Try something like echo 1 > /proc/sys/vm/block_dump and see dmesg.

Beware: it can flood your logs and then dumping logs to HDD can also cause I/O ( which is in turn logged). 

So take care and do not forget to disable it by doing echo 0 > /proc/sys/vm/block_dump

Re: 6+ MiB/s constant usage on a btrfs volume with kernel 3.16

[Holger Hoffstätte]

On Sun, 24 Aug 2014 19:46:41 +0200, Flash ROM wrote:

>> Happens a 100% of the time here, annoyingly. As mentioned, 3.15 was
>> working for me, and still does. Multiple reboots and it happens
>> immediately on boot even before gdm comes up.
> Would be logical to do block-level I/O tracing to get idea WHAT is this
> IO, right?

Or you could just use proper tools for this:

iotop:
http://guichaz.free.fr/iotop/

iosnoop:
http://www.brendangregg.com/blog/2014-07-16/iosnoop-for-linux.html

Probably either a continuing balance or autodefrag vs. systemd's logging.

-h

Re: 6+ MiB/s constant usage on a btrfs volume with kernel 3.16

[Oon-Ee Ng]

On Mon, Aug 25, 2014 at 2:08 AM, Holger Hoffstätte
<holger.hoffstaette@googlemail.com> wrote:
> On Sun, 24 Aug 2014 19:46:41 +0200, Flash ROM wrote:
>
>>> Happens a 100% of the time here, annoyingly. As mentioned, 3.15 was
>>> working for me, and still does. Multiple reboots and it happens
>>> immediately on boot even before gdm comes up.
>> Would be logical to do block-level I/O tracing to get idea WHAT is this
>> IO, right?
>
> Or you could just use proper tools for this:
>
> iotop:
> http://guichaz.free.fr/iotop/

iotop already done, showed in the attached picture. I understand not
all would want to load a pic, so the summary for iotop is that all the
I/O is allocated to a kworker thread. The same attached picture also
shows perf top for that pid, which I do not know how to interpret.
https://db.tt/TSh5piq6
>
> iosnoop:
> http://www.brendangregg.com/blog/2014-07-16/iosnoop-for-linux.html
>
> Probably either a continuing balance or autodefrag vs. systemd's logging.

Thank you, I'll try that out.

Re: 6+ MiB/s constant usage on a btrfs volume with kernel 3.16

[Oon-Ee Ng]

On Mon, Aug 25, 2014 at 8:05 AM, Oon-Ee Ng <ngoonee.talk@gmail.com> wrote:
> On Mon, Aug 25, 2014 at 2:08 AM, Holger Hoffstätte
> <holger.hoffstaette@googlemail.com> wrote:
>>
>> iosnoop:
>> http://www.brendangregg.com/blog/2014-07-16/iosnoop-for-linux.html
>>
>> Probably either a continuing balance or autodefrag vs. systemd's logging.
>
> Thank you, I'll try that out.
http://tny.cz/da4d5338 shows the problem, and for comparison this is a
30 second iosnoop log from 3.15.8 where I do not experience this issue
http://pastebin.com/8EjKiAuf