Hard drive hangs after excessive I/O

Hard drive hangs after excessive I/O

[Dave]

I'm currently running kernel 3.13.3 with Chris's for-linus merged in
(up to commit 93de4ba86480a9e0d1062cb1d535fa97fb81af48).  After about
a day of heavy IO I'll start to see the following in my kernel log:

[22223.454967] INFO: task qemu-system-x86:27673 blocked for more than
120 seconds.
[22223.454970]       Not tainted 3.13.3-2-dzk #1
[22223.454972] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[22223.454974] qemu-system-x86 D ffff8804039d7800     0 27673      1 0x00000000
[22223.454978]  ffff8800d1d6bda0 0000000000000082 ffff8800096aec00
ffff8800d1d6bfd8
[22223.454983]  0000000000014440 0000000000014440 ffff8800096aec00
ffffffffa04aafa0
[22223.454988]  000000000000018b ffff8802e8870000 0000000000003000
0000000000001000
[22223.454993] Call Trace:
[22223.455015]  [<ffffffffa04aafa0>] ? btrfs_set_token_64+0x60/0xf0 [btrfs]
[22223.455036]  [<ffffffffa04ae8fb>] ? release_extent_buffer+0x2b/0xd0 [btrfs]
[22223.455041]  [<ffffffff8151d429>] schedule+0x29/0x70
[22223.455061]  [<ffffffffa04ce282>] wait_log_commit.isra.18+0x102/0x130 [btrfs]
[22223.455066]  [<ffffffff810a2bf0>] ? __wake_up_sync+0x20/0x20
[22223.455085]  [<ffffffffa04d1465>] btrfs_sync_log+0x3e5/0x6b0 [btrfs]
[22223.455091]  [<ffffffff811ba0a0>] ? dput+0x20/0x100
[22223.455112]  [<ffffffffa04a4681>] btrfs_sync_file+0x2c1/0x2f0 [btrfs]
[22223.455117]  [<ffffffff811d2541>] do_fsync+0x51/0x80
[22223.455122]  [<ffffffff811d2823>] SyS_fdatasync+0x13/0x20
[22223.455127]  [<ffffffff81528d6d>] system_call_fastpath+0x1a/0x1f

During this time, all IO totally freezes.  Atop reports no disk
activity, while my desktop is totally hung.  After a few minutes
everything comes back to life, only to freeze again after a few more
minutes.

A reboot clears everything up for a day or so but the problem
invariably comes back.  Can anybody shed some light on this?

PS, the above qemu process is accessing image files on a btrfs volume.
 These files were created as such:
touch winxp.img
chattr +C winxp.img
fallocate -l20G winxp.img

I'm actually quite pleased with VM performance on top of btrfs (until
the above problem starts occurring).
-- 
-=[dave]=-

Entropy isn't what it used to be.

Re: Hard drive hangs after excessive I/O

[Duncan]

Dave posted on Fri, 21 Feb 2014 10:21:38 -0500 as excerpted:

> During this time, all IO totally freezes.  Atop reports no disk
> activity, while my desktop is totally hung.  After a few minutes
> everything comes back to life, only to freeze again after a few more
> minutes.
> 
> A reboot clears everything up for a day or so but the problem invariably
> comes back.  Can anybody shed some light on this?
> 
> PS, the above qemu process is accessing image files on a btrfs volume.
>  These files were created as such:
> touch winxp.img
> chattr +C winxp.img
> fallocate -l20G winxp.img

That procedure should indeed make those VM images properly NOCOW.
Thanks for being so explicit. =:^)

One other question:  Are you, or perhaps your distro via likely automated 
script, doing any btrfs snapshotting of the VM-image containing btrfs 
(sub)volume?

Because snapshotting on a an actively being rewritten NOCOW file is one 
known exception to that NOCOW.  The snapshot takes a copy of the file at 
that moment, and further writes to it must therefore be COWed -- tho only 
the first time a block is written to after the snapshot, further writes 
to the same block will remain in-place written until the next snapshot 
freezes that in-place, triggering yet another COW on first write once 
again.

So depending on the presumably scripted snapshotting schedule and the 
activity on the VM, fragmentation probably won't be as bad on a 
snapshotted NOCOW image as on a standard COW image, but it will still 
happen.

So if you have a snapshotting script running on that subvolume, that 
might be /part/ of it, tho to my knowledge that doesn't fully explain why 
it'd be better for a day or so after a reboot, since the fragmentation 
should be the same as before the reboot.

If you're not doing snapshotting, given that the VMs should indeed be 
NOCOW and as posted you've handled that correctly, I've no idea.  But 
I'll certainly be following the thread, as this is one aspect I've been 
particularly interested in. I don't have any specific personal 
involvement in it at this point, but it's a general enough problem I very 
well could in the future.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Hard drive hangs after excessive I/O

[Dave]

On Fri, Feb 21, 2014 at 11:09 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> One other question:  Are you, or perhaps your distro via likely automated
> script, doing any btrfs snapshotting of the VM-image containing btrfs
> (sub)volume?

I haven't done any snapshotting on this machine.

This problem is pretty easy to reproduce, I spawn about five new
virtual machines and begin installing Windows XP on them.  The host
machine obviously slows down quite a bit but the hard drive chatters
throughout the process.

The next day I wipe those five machines, spawn five new ones, and
begin the concurrent installations again.  This time I see the IO
spurts that I described in my first email; that is, drive thrashes for
a while, then hangs for a couple minutes...  repeat until host reboot.
-- 
-=[dave]=-

Entropy isn't what it used to be.

Re: Hard drive hangs after excessive I/O

[Chris Murphy]

On Feb 21, 2014, at 10:16 AM, Dave <dave@thekilempire.com> wrote:

> On Fri, Feb 21, 2014 at 11:09 AM, Duncan <1i5t5.duncan@cox.net> wrote:
>> One other question:  Are you, or perhaps your distro via likely automated
>> script, doing any btrfs snapshotting of the VM-image containing btrfs
>> (sub)volume?
> 
> I haven't done any snapshotting on this machine.
> 
> This problem is pretty easy to reproduce, I spawn about five new
> virtual machines and begin installing Windows XP on them.  The host
> machine obviously slows down quite a bit but the hard drive chatters
> throughout the process.
> 
> The next day I wipe those five machines, spawn five new ones, and
> begin the concurrent installations again.  This time I see the IO
> spurts that I described in my first email; that is, drive thrashes for
> a while, then hangs for a couple minutes…  repeat until host reboot.

Five concurrent VMs for one spinning disk. It sounds like it's seeking to death. The trace shows fsync and datasync so the storage stack is maybe being compelled to write to disk a lot, and not take as much advantage of Btrfs delayed allocation. Is the VM drive a virtio device? What write caching is it configured for? While risky in the face of a crash, I've had much better results with unsafe caching.


Chris Murphy

Re: Hard drive hangs after excessive I/O

[Dave]

On Fri, Feb 21, 2014 at 12:59 PM, Chris Murphy <lists@colorremedies.com> wrote:
> Five concurrent VMs for one spinning disk. It sounds like it's seeking to death.

That's my reproducible test, not typical use.

"Seeking to death" would indicate drive activity.  When the problem
occurs, drive activity stops for minutes at a time.

I suppose I shouldn't have even mentioned VMs because that seems to be
muddying the discussion.  The problem persists after all the VMs are
shut down.  For instance, right now I'm not running any VMs but I'm
experiencing the problem (though I was performing the above test).  I
can observe it by doing a local git clone of the linux kernel:
git clone /usr/src/linux

If I run this from a fresh boot, it causes the hard drive to chatter
but doesn't cause IO to hang.  After running the above 5-VM test for a
day, even after shutting down the VMs, my machine is still in this
state that causes IO to periodically hang.
-- 
-=[dave]=-

Entropy isn't what it used to be.

Re: Hard drive hangs after excessive I/O

[Chris Murphy]

On Feb 21, 2014, at 12:27 PM, Dave <dave@thekilempire.com> wrote:

> On Fri, Feb 21, 2014 at 12:59 PM, Chris Murphy <lists@colorremedies.com> wrote:
>> Five concurrent VMs for one spinning disk. It sounds like it's seeking to death.
> 
> That's my reproducible test, not typical use.
> 
> "Seeking to death" would indicate drive activity.  When the problem
> occurs, drive activity stops for minutes at a time.
> 
> I suppose I shouldn't have even mentioned VMs because that seems to be
> muddying the discussion.  The problem persists after all the VMs are
> shut down.  For instance, right now I'm not running any VMs but I'm
> experiencing the problem (though I was performing the above test).  I
> can observe it by doing a local git clone of the linux kernel:
> git clone /usr/src/linux
> 
> If I run this from a fresh boot, it causes the hard drive to chatter
> but doesn't cause IO to hang.  After running the above 5-VM test for a
> day, even after shutting down the VMs, my machine is still in this
> state that causes IO to periodically hang.

I am not a Btrfs developer, but when the problem isn't already known, usually comes the request to reproduce the problem and issue sysrq+w, and then grab a full dmesg and post that.


Chris Murphy