From: andre.przywara@arm.com (Andre Przywara)
To: linux-arm-kernel@lists.infradead.org
Subject: block layer bug with 4.4-rc3+
Date: Tue, 15 Dec 2015 11:05:17 +0000 [thread overview]
Message-ID: <566FF3ED.1000503@arm.com> (raw)
Hi,
I've been experiencing issues with at least 4.4-rc3 (including current
HEAD) on a Calxeda Midway (4*ARM Cortex-A15 (32-bit), 8GB RAM, SATA
spinning disk or SSD).
After some disk I/O load (kernel compile with -j6) I see the kernel
screaming:
[ 103.736982] ata1.00: exception Emask 0x0 SAct 0x3ffff0 SErr 0x0
action 0x6 frozen
[ 103.744476] ata1.00: failed command: WRITE FPDMA QUEUED
[ 103.749707] ata1.00: cmd 61/00:20:48:6b:41/08:00:0a:00:00/40 tag 4
ncq 1048576 out
[ 103.749707] res 40/00:00:00:00:00/00:00:00:00:00/00 Emask
0x4 (timeout)
[ 103.764659] ata1.00: status: { DRDY }
[ 103.768321] ata1.00: failed command: WRITE FPDMA QUEUED
[ 103.773547] ata1.00: cmd 61/98:28:48:73:41/42:00:0a:00:00/40 tag 5
ncq 8728576 out
[ 103.773547] res 40/00:00:00:00:00/00:00:00:00:00/00 Emask
0x4 (timeout)
< repeated with increasing tag numbers>
This repeats for a while, but then seems to recover later, though I
haven't checked if there are more issues and rebooted instead to avoid
filesystem damage.
While I agree that this looks like a disk error on the first glance, I
never saw this before 4.4-rc2, had the very same error on different
nodes (with another spinning disk and even an SSD) and I can make it
vanish by reverting the commit I identified after bisection:
commit 578270bfbd2803dc7b0b03fbc2ac119efbc73195
Author: Ming Lei <ming.lei@canonical.com>
Date: Tue Nov 24 10:35:29 2015 +0800
block: fix segment split
...
I understand that this fix seems sane, but actually reverting it fixes
the issue for me: 4.4-rc5 crashed within some minutes with the above
log, 4.4-rc5 with 578270bfbd reverted survived 19 hours of continuous
kernel compiles without issues.
Looking at the git history of that file I see quite some recent changes
there, but it's beyond my understanding of the code to spot the real
culprit.
Can anyone point me to a change in blk-merge.c I could try to revert to
identify the real root cause? I can run tests quickly, though a real
positive case would need some hours of runtime to be sure it's fine.
Many thanks!
Cheers,
Andre.
WARNING: multiple messages have this Message-ID (diff)
From: Andre Przywara <andre.przywara@arm.com>
To: Jens Axboe <axboe@kernel.dk>, Ming Lei <ming.lei@canonical.com>
Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
Rob Herring <rob.herring@linaro.org>,
Eric Auger <eric.auger@linaro.org>,
"linux-arm-kernel@lists.infradead.org"
<linux-arm-kernel@lists.infradead.org>
Subject: block layer bug with 4.4-rc3+
Date: Tue, 15 Dec 2015 11:05:17 +0000 [thread overview]
Message-ID: <566FF3ED.1000503@arm.com> (raw)
Hi,
I've been experiencing issues with at least 4.4-rc3 (including current
HEAD) on a Calxeda Midway (4*ARM Cortex-A15 (32-bit), 8GB RAM, SATA
spinning disk or SSD).
After some disk I/O load (kernel compile with -j6) I see the kernel
screaming:
[ 103.736982] ata1.00: exception Emask 0x0 SAct 0x3ffff0 SErr 0x0
action 0x6 frozen
[ 103.744476] ata1.00: failed command: WRITE FPDMA QUEUED
[ 103.749707] ata1.00: cmd 61/00:20:48:6b:41/08:00:0a:00:00/40 tag 4
ncq 1048576 out
[ 103.749707] res 40/00:00:00:00:00/00:00:00:00:00/00 Emask
0x4 (timeout)
[ 103.764659] ata1.00: status: { DRDY }
[ 103.768321] ata1.00: failed command: WRITE FPDMA QUEUED
[ 103.773547] ata1.00: cmd 61/98:28:48:73:41/42:00:0a:00:00/40 tag 5
ncq 8728576 out
[ 103.773547] res 40/00:00:00:00:00/00:00:00:00:00/00 Emask
0x4 (timeout)
< repeated with increasing tag numbers>
This repeats for a while, but then seems to recover later, though I
haven't checked if there are more issues and rebooted instead to avoid
filesystem damage.
While I agree that this looks like a disk error on the first glance, I
never saw this before 4.4-rc2, had the very same error on different
nodes (with another spinning disk and even an SSD) and I can make it
vanish by reverting the commit I identified after bisection:
commit 578270bfbd2803dc7b0b03fbc2ac119efbc73195
Author: Ming Lei <ming.lei@canonical.com>
Date: Tue Nov 24 10:35:29 2015 +0800
block: fix segment split
...
I understand that this fix seems sane, but actually reverting it fixes
the issue for me: 4.4-rc5 crashed within some minutes with the above
log, 4.4-rc5 with 578270bfbd reverted survived 19 hours of continuous
kernel compiles without issues.
Looking at the git history of that file I see quite some recent changes
there, but it's beyond my understanding of the code to spot the real
culprit.
Can anyone point me to a change in blk-merge.c I could try to revert to
identify the real root cause? I can run tests quickly, though a real
positive case would need some hours of runtime to be sure it's fine.
Many thanks!
Cheers,
Andre.
next reply other threads:[~2015-12-15 11:05 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-12-15 11:05 Andre Przywara [this message]
2015-12-15 11:05 ` block layer bug with 4.4-rc3+ Andre Przywara
2015-12-15 11:54 ` Ming Lei
2015-12-15 11:54 ` Ming Lei
2015-12-15 12:23 ` Andre Przywara
2015-12-15 12:23 ` Andre Przywara
2015-12-15 13:39 ` Ming Lei
2015-12-15 13:39 ` Ming Lei
2015-12-16 14:55 ` Andre Przywara
2015-12-16 14:55 ` Andre Przywara
2015-12-16 15:43 ` Arnd Bergmann
2015-12-16 15:43 ` Arnd Bergmann
2015-12-17 3:55 ` Ming Lei
2015-12-17 3:55 ` Ming Lei
2015-12-17 9:28 ` Andre Przywara
2015-12-17 9:28 ` Andre Przywara
2015-12-17 9:46 ` Arnd Bergmann
2015-12-17 9:46 ` Arnd Bergmann
2015-12-17 3:52 ` Ming Lei
2015-12-17 3:52 ` Ming Lei
2015-12-17 12:32 ` Andre Przywara
2015-12-17 12:32 ` Andre Przywara
2015-12-17 12:33 ` Andre Przywara
2015-12-17 12:33 ` Andre Przywara
2015-12-18 8:36 ` Ming Lei
2015-12-18 8:36 ` Ming Lei
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=566FF3ED.1000503@arm.com \
--to=andre.przywara@arm.com \
--cc=linux-arm-kernel@lists.infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.