From: christoffer.dall@linaro.org (Christoffer Dall)
To: linux-arm-kernel@lists.infradead.org
Subject: [BUG] Read-Only THP causes stalls (commit 10359213d)
Date: Mon, 25 May 2015 12:19:44 +0200 [thread overview]
Message-ID: <20150525101944.GA14095@cbox> (raw)
In-Reply-To: <20150525100515.GA8275@node.dhcp.inet.fi>
On Mon, May 25, 2015 at 01:05:15PM +0300, Kirill A. Shutemov wrote:
> On Sun, May 24, 2015 at 09:34:04PM +0200, Christoffer Dall wrote:
> > Hi all,
> >
> > I noticed a regression on my arm64 APM X-Gene system a couple
> > of weeks back. I would occassionally see the system lock up and see RCU
> > stalls during the caching phase of kernbench. I then wrote a small
> > script that does nothing but cache the files
> > (http://paste.ubuntu.com/11324767/) and ran that in a loop. On a known
> > bad commit (v4.1-rc2), out of 25 boots, I never saw it get past 21
> > iterations of the loop. I have since tried to run a bisect from v3.19 to
> > v4.0 using 100 iterations as my criteria for a good commit.
> >
> > This resulted in the following first bad commit:
> >
> > 10359213d05acf804558bda7cc9b8422a828d1cd
> > (mm: incorporate read-only pages into transparent huge pages, 2015-02-11)
> >
> > Indeed, running the workload on v4.1-rc4 still produced the behavior,
> > but reverting the above commit gets me through 100 iterations of the
> > loop.
> >
> > I have not tried to reproduce on an x86 system. Turning on a bunch
> > of kernel debugging features *seems* to hide the problem. My config for
> > the XGene system is defconfig + CONFIG_BRIDGE and
> > CONFIG_POWER_RESET_XGENE.
> >
> > Please let me know if I can help test patches or other things I can
> > do to help. I'm afraid that by simply reading the patch I didn't see
> > anything obviously wrong with it which would cause this behavior.
>
> I don't see the problem on x86.
I'm wondering if we could have some weird combination of how the
specific architecture works along with these patches...
>
> Some backtraces could help to track it down.
>
I don't really get backtraces as the sytem just locks up. But here are
some of the RCU stalls as I've observed them on the console:
http://paste.ubuntu.com/11014701/
http://paste.ubuntu.com/11023143/
http://paste.ubuntu.com/11023261/
Occasionally, I also get this error from the SATA system at the moment
when I power off the device, but not sure if it is related:
ata1: exception Emask 0x10 SAct 0x0 SErr 0x180000 action 0xe frozen
ata1: irq_stat 0x00400000, PHY RDY changed
ata1: SError: { 10B8B Dispar }
Thanks,
-Christoffer
WARNING: multiple messages have this Message-ID (diff)
From: Christoffer Dall <christoffer.dall@linaro.org>
To: "Kirill A. Shutemov" <kirill@shutemov.name>
Cc: linux-mm@kvack.org, ebru.akagunduz@gmail.com,
akpm@linux-foundation.org, linux-kernel@vger.kernel.org,
kirill.shutemov@linux.intel.com, riel@redhat.com, vbabka@suse.cz,
zhangyanfei@cn.fujitsu.com, aarcange@redhat.com,
Will Deacon <will.deacon@arm.com>,
Andre Przywara <andre.przywara@arm.com>,
Marc Zyngier <marc.zyngier@arm.com>,
linux-arm-kernel@lists.infradead.org
Subject: Re: [BUG] Read-Only THP causes stalls (commit 10359213d)
Date: Mon, 25 May 2015 12:19:44 +0200 [thread overview]
Message-ID: <20150525101944.GA14095@cbox> (raw)
In-Reply-To: <20150525100515.GA8275@node.dhcp.inet.fi>
On Mon, May 25, 2015 at 01:05:15PM +0300, Kirill A. Shutemov wrote:
> On Sun, May 24, 2015 at 09:34:04PM +0200, Christoffer Dall wrote:
> > Hi all,
> >
> > I noticed a regression on my arm64 APM X-Gene system a couple
> > of weeks back. I would occassionally see the system lock up and see RCU
> > stalls during the caching phase of kernbench. I then wrote a small
> > script that does nothing but cache the files
> > (http://paste.ubuntu.com/11324767/) and ran that in a loop. On a known
> > bad commit (v4.1-rc2), out of 25 boots, I never saw it get past 21
> > iterations of the loop. I have since tried to run a bisect from v3.19 to
> > v4.0 using 100 iterations as my criteria for a good commit.
> >
> > This resulted in the following first bad commit:
> >
> > 10359213d05acf804558bda7cc9b8422a828d1cd
> > (mm: incorporate read-only pages into transparent huge pages, 2015-02-11)
> >
> > Indeed, running the workload on v4.1-rc4 still produced the behavior,
> > but reverting the above commit gets me through 100 iterations of the
> > loop.
> >
> > I have not tried to reproduce on an x86 system. Turning on a bunch
> > of kernel debugging features *seems* to hide the problem. My config for
> > the XGene system is defconfig + CONFIG_BRIDGE and
> > CONFIG_POWER_RESET_XGENE.
> >
> > Please let me know if I can help test patches or other things I can
> > do to help. I'm afraid that by simply reading the patch I didn't see
> > anything obviously wrong with it which would cause this behavior.
>
> I don't see the problem on x86.
I'm wondering if we could have some weird combination of how the
specific architecture works along with these patches...
>
> Some backtraces could help to track it down.
>
I don't really get backtraces as the sytem just locks up. But here are
some of the RCU stalls as I've observed them on the console:
http://paste.ubuntu.com/11014701/
http://paste.ubuntu.com/11023143/
http://paste.ubuntu.com/11023261/
Occasionally, I also get this error from the SATA system at the moment
when I power off the device, but not sure if it is related:
ata1: exception Emask 0x10 SAct 0x0 SErr 0x180000 action 0xe frozen
ata1: irq_stat 0x00400000, PHY RDY changed
ata1: SError: { 10B8B Dispar }
Thanks,
-Christoffer
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
WARNING: multiple messages have this Message-ID (diff)
From: Christoffer Dall <christoffer.dall@linaro.org>
To: "Kirill A. Shutemov" <kirill@shutemov.name>
Cc: linux-mm@kvack.org, ebru.akagunduz@gmail.com,
akpm@linux-foundation.org, linux-kernel@vger.kernel.org,
kirill.shutemov@linux.intel.com, riel@redhat.com, vbabka@suse.cz,
zhangyanfei@cn.fujitsu.com, aarcange@redhat.com,
Will Deacon <will.deacon@arm.com>,
Andre Przywara <andre.przywara@arm.com>,
Marc Zyngier <marc.zyngier@arm.com>,
linux-arm-kernel@lists.infradead.org
Subject: Re: [BUG] Read-Only THP causes stalls (commit 10359213d)
Date: Mon, 25 May 2015 12:19:44 +0200 [thread overview]
Message-ID: <20150525101944.GA14095@cbox> (raw)
In-Reply-To: <20150525100515.GA8275@node.dhcp.inet.fi>
On Mon, May 25, 2015 at 01:05:15PM +0300, Kirill A. Shutemov wrote:
> On Sun, May 24, 2015 at 09:34:04PM +0200, Christoffer Dall wrote:
> > Hi all,
> >
> > I noticed a regression on my arm64 APM X-Gene system a couple
> > of weeks back. I would occassionally see the system lock up and see RCU
> > stalls during the caching phase of kernbench. I then wrote a small
> > script that does nothing but cache the files
> > (http://paste.ubuntu.com/11324767/) and ran that in a loop. On a known
> > bad commit (v4.1-rc2), out of 25 boots, I never saw it get past 21
> > iterations of the loop. I have since tried to run a bisect from v3.19 to
> > v4.0 using 100 iterations as my criteria for a good commit.
> >
> > This resulted in the following first bad commit:
> >
> > 10359213d05acf804558bda7cc9b8422a828d1cd
> > (mm: incorporate read-only pages into transparent huge pages, 2015-02-11)
> >
> > Indeed, running the workload on v4.1-rc4 still produced the behavior,
> > but reverting the above commit gets me through 100 iterations of the
> > loop.
> >
> > I have not tried to reproduce on an x86 system. Turning on a bunch
> > of kernel debugging features *seems* to hide the problem. My config for
> > the XGene system is defconfig + CONFIG_BRIDGE and
> > CONFIG_POWER_RESET_XGENE.
> >
> > Please let me know if I can help test patches or other things I can
> > do to help. I'm afraid that by simply reading the patch I didn't see
> > anything obviously wrong with it which would cause this behavior.
>
> I don't see the problem on x86.
I'm wondering if we could have some weird combination of how the
specific architecture works along with these patches...
>
> Some backtraces could help to track it down.
>
I don't really get backtraces as the sytem just locks up. But here are
some of the RCU stalls as I've observed them on the console:
http://paste.ubuntu.com/11014701/
http://paste.ubuntu.com/11023143/
http://paste.ubuntu.com/11023261/
Occasionally, I also get this error from the SATA system at the moment
when I power off the device, but not sure if it is related:
ata1: exception Emask 0x10 SAct 0x0 SErr 0x180000 action 0xe frozen
ata1: irq_stat 0x00400000, PHY RDY changed
ata1: SError: { 10B8B Dispar }
Thanks,
-Christoffer
next prev parent reply other threads:[~2015-05-25 10:19 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-05-24 19:34 [BUG] Read-Only THP causes stalls (commit 10359213d) Christoffer Dall
2015-05-24 19:34 ` Christoffer Dall
2015-05-24 19:34 ` Christoffer Dall
2015-05-25 10:05 ` Kirill A. Shutemov
2015-05-25 10:05 ` Kirill A. Shutemov
2015-05-25 10:05 ` Kirill A. Shutemov
2015-05-25 10:19 ` Christoffer Dall [this message]
2015-05-25 10:19 ` Christoffer Dall
2015-05-25 10:19 ` Christoffer Dall
2015-05-25 14:15 ` Andrea Arcangeli
2015-05-25 14:15 ` Andrea Arcangeli
2015-05-25 14:15 ` Andrea Arcangeli
2015-05-26 8:08 ` Christoffer Dall
2015-05-26 8:08 ` Christoffer Dall
2015-05-26 8:08 ` Christoffer Dall
2015-05-26 10:57 ` Marc Zyngier
2015-05-26 10:57 ` Marc Zyngier
2015-05-26 10:57 ` Marc Zyngier
2015-05-26 14:24 ` Steve Capper
2015-05-26 14:24 ` Steve Capper
2015-05-26 14:24 ` Steve Capper
2015-05-26 14:35 ` Christoffer Dall
2015-05-26 14:35 ` Christoffer Dall
2015-05-26 14:35 ` Christoffer Dall
2015-05-26 14:48 ` Steve Capper
2015-05-26 14:48 ` Steve Capper
2015-05-26 14:48 ` Steve Capper
2015-05-26 14:58 ` Andrea Arcangeli
2015-05-26 14:58 ` Andrea Arcangeli
2015-05-26 14:58 ` Andrea Arcangeli
2015-05-26 14:42 ` Andrea Arcangeli
2015-05-26 14:42 ` Andrea Arcangeli
2015-05-26 14:42 ` Andrea Arcangeli
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20150525101944.GA14095@cbox \
--to=christoffer.dall@linaro.org \
--cc=linux-arm-kernel@lists.infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.