From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lb0-f177.google.com (mail-lb0-f177.google.com [209.85.217.177]) by kanga.kvack.org (Postfix) with ESMTP id 5CE1D6B00D8 for ; Sun, 24 May 2015 15:33:40 -0400 (EDT) Received: by lbbuc2 with SMTP id uc2so40688166lbb.2 for ; Sun, 24 May 2015 12:33:39 -0700 (PDT) Received: from mail-la0-f54.google.com (mail-la0-f54.google.com. [209.85.215.54]) by mx.google.com with ESMTPS id d4si6629350lbb.6.2015.05.24.12.33.37 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 24 May 2015 12:33:38 -0700 (PDT) Received: by labbd9 with SMTP id bd9so38710204lab.2 for ; Sun, 24 May 2015 12:33:37 -0700 (PDT) Date: Sun, 24 May 2015 21:34:04 +0200 From: Christoffer Dall Subject: [BUG] Read-Only THP causes stalls (commit 10359213d) Message-ID: <20150524193404.GD16910@cbox> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-linux-mm@kvack.org List-ID: To: linux-mm@kvack.org, ebru.akagunduz@gmail.com Cc: akpm@linux-foundation.org, linux-kernel@vger.kernel.org, kirill.shutemov@linux.intel.com, riel@redhat.com, vbabka@suse.cz, zhangyanfei@cn.fujitsu.com, aarcange@redhat.com, Will Deacon , Andre Przywara , Marc Zyngier , linux-arm-kernel@lists.infradead.org Hi all, I noticed a regression on my arm64 APM X-Gene system a couple of weeks back. I would occassionally see the system lock up and see RCU stalls during the caching phase of kernbench. I then wrote a small script that does nothing but cache the files (http://paste.ubuntu.com/11324767/) and ran that in a loop. On a known bad commit (v4.1-rc2), out of 25 boots, I never saw it get past 21 iterations of the loop. I have since tried to run a bisect from v3.19 to v4.0 using 100 iterations as my criteria for a good commit. This resulted in the following first bad commit: 10359213d05acf804558bda7cc9b8422a828d1cd (mm: incorporate read-only pages into transparent huge pages, 2015-02-11) Indeed, running the workload on v4.1-rc4 still produced the behavior, but reverting the above commit gets me through 100 iterations of the loop. I have not tried to reproduce on an x86 system. Turning on a bunch of kernel debugging features *seems* to hide the problem. My config for the XGene system is defconfig + CONFIG_BRIDGE and CONFIG_POWER_RESET_XGENE. Please let me know if I can help test patches or other things I can do to help. I'm afraid that by simply reading the patch I didn't see anything obviously wrong with it which would cause this behavior. Thanks, -Christoffer -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-f179.google.com (mail-wi0-f179.google.com [209.85.212.179]) by kanga.kvack.org (Postfix) with ESMTP id 377046B00AC for ; Mon, 25 May 2015 06:05:30 -0400 (EDT) Received: by wizk4 with SMTP id k4so43495173wiz.1 for ; Mon, 25 May 2015 03:05:29 -0700 (PDT) Received: from jenni1.inet.fi (mta-out1.inet.fi. [62.71.2.195]) by mx.google.com with ESMTP id em3si11788633wib.90.2015.05.25.03.05.28 for ; Mon, 25 May 2015 03:05:28 -0700 (PDT) Date: Mon, 25 May 2015 13:05:15 +0300 From: "Kirill A. Shutemov" Subject: Re: [BUG] Read-Only THP causes stalls (commit 10359213d) Message-ID: <20150525100515.GA8275@node.dhcp.inet.fi> References: <20150524193404.GD16910@cbox> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150524193404.GD16910@cbox> Sender: owner-linux-mm@kvack.org List-ID: To: Christoffer Dall Cc: linux-mm@kvack.org, ebru.akagunduz@gmail.com, akpm@linux-foundation.org, linux-kernel@vger.kernel.org, kirill.shutemov@linux.intel.com, riel@redhat.com, vbabka@suse.cz, zhangyanfei@cn.fujitsu.com, aarcange@redhat.com, Will Deacon , Andre Przywara , Marc Zyngier , linux-arm-kernel@lists.infradead.org On Sun, May 24, 2015 at 09:34:04PM +0200, Christoffer Dall wrote: > Hi all, > > I noticed a regression on my arm64 APM X-Gene system a couple > of weeks back. I would occassionally see the system lock up and see RCU > stalls during the caching phase of kernbench. I then wrote a small > script that does nothing but cache the files > (http://paste.ubuntu.com/11324767/) and ran that in a loop. On a known > bad commit (v4.1-rc2), out of 25 boots, I never saw it get past 21 > iterations of the loop. I have since tried to run a bisect from v3.19 to > v4.0 using 100 iterations as my criteria for a good commit. > > This resulted in the following first bad commit: > > 10359213d05acf804558bda7cc9b8422a828d1cd > (mm: incorporate read-only pages into transparent huge pages, 2015-02-11) > > Indeed, running the workload on v4.1-rc4 still produced the behavior, > but reverting the above commit gets me through 100 iterations of the > loop. > > I have not tried to reproduce on an x86 system. Turning on a bunch > of kernel debugging features *seems* to hide the problem. My config for > the XGene system is defconfig + CONFIG_BRIDGE and > CONFIG_POWER_RESET_XGENE. > > Please let me know if I can help test patches or other things I can > do to help. I'm afraid that by simply reading the patch I didn't see > anything obviously wrong with it which would cause this behavior. I don't see the problem on x86. Some backtraces could help to track it down. -- Kirill A. Shutemov -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-la0-f43.google.com (mail-la0-f43.google.com [209.85.215.43]) by kanga.kvack.org (Postfix) with ESMTP id 19F976B00AC for ; Mon, 25 May 2015 06:19:20 -0400 (EDT) Received: by labbd9 with SMTP id bd9so46781902lab.2 for ; Mon, 25 May 2015 03:19:19 -0700 (PDT) Received: from mail-la0-f46.google.com (mail-la0-f46.google.com. [209.85.215.46]) by mx.google.com with ESMTPS id lp10si5509418lbb.1.2015.05.25.03.19.17 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 May 2015 03:19:18 -0700 (PDT) Received: by labbd9 with SMTP id bd9so46781499lab.2 for ; Mon, 25 May 2015 03:19:17 -0700 (PDT) Date: Mon, 25 May 2015 12:19:44 +0200 From: Christoffer Dall Subject: Re: [BUG] Read-Only THP causes stalls (commit 10359213d) Message-ID: <20150525101944.GA14095@cbox> References: <20150524193404.GD16910@cbox> <20150525100515.GA8275@node.dhcp.inet.fi> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150525100515.GA8275@node.dhcp.inet.fi> Sender: owner-linux-mm@kvack.org List-ID: To: "Kirill A. Shutemov" Cc: linux-mm@kvack.org, ebru.akagunduz@gmail.com, akpm@linux-foundation.org, linux-kernel@vger.kernel.org, kirill.shutemov@linux.intel.com, riel@redhat.com, vbabka@suse.cz, zhangyanfei@cn.fujitsu.com, aarcange@redhat.com, Will Deacon , Andre Przywara , Marc Zyngier , linux-arm-kernel@lists.infradead.org On Mon, May 25, 2015 at 01:05:15PM +0300, Kirill A. Shutemov wrote: > On Sun, May 24, 2015 at 09:34:04PM +0200, Christoffer Dall wrote: > > Hi all, > > > > I noticed a regression on my arm64 APM X-Gene system a couple > > of weeks back. I would occassionally see the system lock up and see RCU > > stalls during the caching phase of kernbench. I then wrote a small > > script that does nothing but cache the files > > (http://paste.ubuntu.com/11324767/) and ran that in a loop. On a known > > bad commit (v4.1-rc2), out of 25 boots, I never saw it get past 21 > > iterations of the loop. I have since tried to run a bisect from v3.19 to > > v4.0 using 100 iterations as my criteria for a good commit. > > > > This resulted in the following first bad commit: > > > > 10359213d05acf804558bda7cc9b8422a828d1cd > > (mm: incorporate read-only pages into transparent huge pages, 2015-02-11) > > > > Indeed, running the workload on v4.1-rc4 still produced the behavior, > > but reverting the above commit gets me through 100 iterations of the > > loop. > > > > I have not tried to reproduce on an x86 system. Turning on a bunch > > of kernel debugging features *seems* to hide the problem. My config for > > the XGene system is defconfig + CONFIG_BRIDGE and > > CONFIG_POWER_RESET_XGENE. > > > > Please let me know if I can help test patches or other things I can > > do to help. I'm afraid that by simply reading the patch I didn't see > > anything obviously wrong with it which would cause this behavior. > > I don't see the problem on x86. I'm wondering if we could have some weird combination of how the specific architecture works along with these patches... > > Some backtraces could help to track it down. > I don't really get backtraces as the sytem just locks up. But here are some of the RCU stalls as I've observed them on the console: http://paste.ubuntu.com/11014701/ http://paste.ubuntu.com/11023143/ http://paste.ubuntu.com/11023261/ Occasionally, I also get this error from the SATA system at the moment when I power off the device, but not sure if it is related: ata1: exception Emask 0x10 SAct 0x0 SErr 0x180000 action 0xe frozen ata1: irq_stat 0x00400000, PHY RDY changed ata1: SError: { 10B8B Dispar } Thanks, -Christoffer -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qk0-f171.google.com (mail-qk0-f171.google.com [209.85.220.171]) by kanga.kvack.org (Postfix) with ESMTP id 9B7376B0032 for ; Mon, 25 May 2015 10:16:00 -0400 (EDT) Received: by qkdn188 with SMTP id n188so65498009qkd.2 for ; Mon, 25 May 2015 07:16:00 -0700 (PDT) Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id p74si11421298qha.68.2015.05.25.07.15.58 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 May 2015 07:15:59 -0700 (PDT) Date: Mon, 25 May 2015 16:15:25 +0200 From: Andrea Arcangeli Subject: Re: [BUG] Read-Only THP causes stalls (commit 10359213d) Message-ID: <20150525141525.GB26958@redhat.com> References: <20150524193404.GD16910@cbox> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150524193404.GD16910@cbox> Sender: owner-linux-mm@kvack.org List-ID: To: Christoffer Dall Cc: linux-mm@kvack.org, ebru.akagunduz@gmail.com, akpm@linux-foundation.org, linux-kernel@vger.kernel.org, kirill.shutemov@linux.intel.com, riel@redhat.com, vbabka@suse.cz, zhangyanfei@cn.fujitsu.com, Will Deacon , Andre Przywara , Marc Zyngier , linux-arm-kernel@lists.infradead.org Hello Christoffer, On Sun, May 24, 2015 at 09:34:04PM +0200, Christoffer Dall wrote: > Hi all, > > I noticed a regression on my arm64 APM X-Gene system a couple > of weeks back. I would occassionally see the system lock up and see RCU > stalls during the caching phase of kernbench. I then wrote a small > script that does nothing but cache the files > (http://paste.ubuntu.com/11324767/) and ran that in a loop. On a known > bad commit (v4.1-rc2), out of 25 boots, I never saw it get past 21 > iterations of the loop. I have since tried to run a bisect from v3.19 to > v4.0 using 100 iterations as my criteria for a good commit. > > This resulted in the following first bad commit: > > 10359213d05acf804558bda7cc9b8422a828d1cd > (mm: incorporate read-only pages into transparent huge pages, 2015-02-11) > > Indeed, running the workload on v4.1-rc4 still produced the behavior, > but reverting the above commit gets me through 100 iterations of the > loop. > > I have not tried to reproduce on an x86 system. Turning on a bunch > of kernel debugging features *seems* to hide the problem. My config for > the XGene system is defconfig + CONFIG_BRIDGE and > CONFIG_POWER_RESET_XGENE. > > Please let me know if I can help test patches or other things I can > do to help. I'm afraid that by simply reading the patch I didn't see > anything obviously wrong with it which would cause this behavior. As further confirmation, could you try: echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan and verify the problem goes away without having to revert the patch? Accordingly you should reproduce much eaiser this way (setting $largevalue to 8192 or something, it doesn't matter). echo $largevalue > /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs Then push the system into swap with some memhog -r1000 xG. The patch just allows readonly anon pages to be collapsed along with read-write ones, the vma permissions allows it, so they have to be swapcache pages, this is why swap shall be required. Perhaps there's some arch detail that needs fixing but it'll be easier to track it down once you have a way to reproduce fast. Thanks! Andrea -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lb0-f174.google.com (mail-lb0-f174.google.com [209.85.217.174]) by kanga.kvack.org (Postfix) with ESMTP id 2801A6B0185 for ; Tue, 26 May 2015 04:08:22 -0400 (EDT) Received: by lbbzk7 with SMTP id zk7so64795877lbb.0 for ; Tue, 26 May 2015 01:08:21 -0700 (PDT) Received: from mail-la0-f50.google.com (mail-la0-f50.google.com. [209.85.215.50]) by mx.google.com with ESMTPS id m14si10359794laa.111.2015.05.26.01.08.20 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 May 2015 01:08:20 -0700 (PDT) Received: by labbd9 with SMTP id bd9so62128348lab.2 for ; Tue, 26 May 2015 01:08:20 -0700 (PDT) Date: Tue, 26 May 2015 10:08:48 +0200 From: Christoffer Dall Subject: Re: [BUG] Read-Only THP causes stalls (commit 10359213d) Message-ID: <20150526080848.GA27075@cbox> References: <20150524193404.GD16910@cbox> <20150525141525.GB26958@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150525141525.GB26958@redhat.com> Sender: owner-linux-mm@kvack.org List-ID: To: Andrea Arcangeli Cc: linux-mm@kvack.org, ebru.akagunduz@gmail.com, akpm@linux-foundation.org, linux-kernel@vger.kernel.org, kirill.shutemov@linux.intel.com, riel@redhat.com, vbabka@suse.cz, zhangyanfei@cn.fujitsu.com, Will Deacon , Andre Przywara , Marc Zyngier , linux-arm-kernel@lists.infradead.org Hi Andrea, On Mon, May 25, 2015 at 04:15:25PM +0200, Andrea Arcangeli wrote: > Hello Christoffer, > > On Sun, May 24, 2015 at 09:34:04PM +0200, Christoffer Dall wrote: > > Hi all, > > > > I noticed a regression on my arm64 APM X-Gene system a couple > > of weeks back. I would occassionally see the system lock up and see RCU > > stalls during the caching phase of kernbench. I then wrote a small > > script that does nothing but cache the files > > (http://paste.ubuntu.com/11324767/) and ran that in a loop. On a known > > bad commit (v4.1-rc2), out of 25 boots, I never saw it get past 21 > > iterations of the loop. I have since tried to run a bisect from v3.19 to > > v4.0 using 100 iterations as my criteria for a good commit. > > > > This resulted in the following first bad commit: > > > > 10359213d05acf804558bda7cc9b8422a828d1cd > > (mm: incorporate read-only pages into transparent huge pages, 2015-02-11) > > > > Indeed, running the workload on v4.1-rc4 still produced the behavior, > > but reverting the above commit gets me through 100 iterations of the > > loop. > > > > I have not tried to reproduce on an x86 system. Turning on a bunch > > of kernel debugging features *seems* to hide the problem. My config for > > the XGene system is defconfig + CONFIG_BRIDGE and > > CONFIG_POWER_RESET_XGENE. > > > > Please let me know if I can help test patches or other things I can > > do to help. I'm afraid that by simply reading the patch I didn't see > > anything obviously wrong with it which would cause this behavior. > > As further confirmation, could you try: > > echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan this returns -EINVAL. But I'm trying now with: echo never > /sys/kernel/mm/transparent_hugepage/enabled > > and verify the problem goes away without having to revert the patch? will let you know, so far so good... > > Accordingly you should reproduce much eaiser this way (setting > $largevalue to 8192 or something, it doesn't matter). > > echo $largevalue > /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan > echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs > echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs > > Then push the system into swap with some memhog -r1000 xG. what is memhog? I couldn't find the utility in Google... I did try with the above settings and just push a bunch of data into ramfs and tmpfs and indeed the sytem died very quickly (on v4.0-rc4). > > The patch just allows readonly anon pages to be collapsed along with > read-write ones, the vma permissions allows it, so they have to be > swapcache pages, this is why swap shall be required. > > Perhaps there's some arch detail that needs fixing but it'll be easier > to track it down once you have a way to reproduce fast. > Yes, would be great to be able to reproduce quickly. Thanks, -Christoffer -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pd0-f170.google.com (mail-pd0-f170.google.com [209.85.192.170]) by kanga.kvack.org (Postfix) with ESMTP id 72AEF6B012C for ; Tue, 26 May 2015 06:58:04 -0400 (EDT) Received: by pdbqa5 with SMTP id qa5so88342531pdb.0 for ; Tue, 26 May 2015 03:58:04 -0700 (PDT) Received: from foss.arm.com (foss.arm.com. [217.140.101.70]) by mx.google.com with ESMTP id px1si20347416pbb.240.2015.05.26.03.58.03 for ; Tue, 26 May 2015 03:58:03 -0700 (PDT) Message-ID: <556451B6.6080303@arm.com> Date: Tue, 26 May 2015 11:57:58 +0100 From: Marc Zyngier MIME-Version: 1.0 Subject: Re: [BUG] Read-Only THP causes stalls (commit 10359213d) References: <20150524193404.GD16910@cbox> <20150525141525.GB26958@redhat.com> <20150526080848.GA27075@cbox> In-Reply-To: <20150526080848.GA27075@cbox> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Christoffer Dall , Andrea Arcangeli Cc: "linux-mm@kvack.org" , "ebru.akagunduz@gmail.com" , "akpm@linux-foundation.org" , "linux-kernel@vger.kernel.org" , "kirill.shutemov@linux.intel.com" , "riel@redhat.com" , "vbabka@suse.cz" , "zhangyanfei@cn.fujitsu.com" , Will Deacon , Andre Przywara , "linux-arm-kernel@lists.infradead.org" On 26/05/15 09:08, Christoffer Dall wrote: [...] >> Then push the system into swap with some memhog -r1000 xG. > > what is memhog? I couldn't find the utility in Google... This looks to be part of the numactl suite, though Debian doesn't seem to include it in its numactl package... Thanks, M. -- Jazz is not dead. It just smells funny... -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ie0-f179.google.com (mail-ie0-f179.google.com [209.85.223.179]) by kanga.kvack.org (Postfix) with ESMTP id 9D4696B0121 for ; Tue, 26 May 2015 10:24:21 -0400 (EDT) Received: by iebgx4 with SMTP id gx4so93058093ieb.0 for ; Tue, 26 May 2015 07:24:21 -0700 (PDT) Received: from mail-ie0-f179.google.com (mail-ie0-f179.google.com. [209.85.223.179]) by mx.google.com with ESMTPS id sb9si8643087igb.22.2015.05.26.07.24.21 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 May 2015 07:24:21 -0700 (PDT) Received: by iesa3 with SMTP id a3so93194754ies.2 for ; Tue, 26 May 2015 07:24:21 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <20150526080848.GA27075@cbox> References: <20150524193404.GD16910@cbox> <20150525141525.GB26958@redhat.com> <20150526080848.GA27075@cbox> Date: Tue, 26 May 2015 15:24:20 +0100 Message-ID: Subject: Re: [BUG] Read-Only THP causes stalls (commit 10359213d) From: Steve Capper Content-Type: text/plain; charset=UTF-8 Sender: owner-linux-mm@kvack.org List-ID: To: Christoffer Dall Cc: Andrea Arcangeli , "linux-mm@kvack.org" , ebru.akagunduz@gmail.com, "akpm@linux-foundation.org" , "linux-kernel@vger.kernel.org" , kirill.shutemov@linux.intel.com, Rik van Riel , vbabka@suse.cz, Zhang Yanfei , Will Deacon , Andre Przywara , Marc Zyngier , "linux-arm-kernel@lists.infradead.org" On 26 May 2015 at 09:08, Christoffer Dall wrote: > Hi Andrea, > > On Mon, May 25, 2015 at 04:15:25PM +0200, Andrea Arcangeli wrote: >> Hello Christoffer, >> >> On Sun, May 24, 2015 at 09:34:04PM +0200, Christoffer Dall wrote: >> > Hi all, >> > >> > I noticed a regression on my arm64 APM X-Gene system a couple >> > of weeks back. I would occassionally see the system lock up and see RCU >> > stalls during the caching phase of kernbench. I then wrote a small >> > script that does nothing but cache the files >> > (http://paste.ubuntu.com/11324767/) and ran that in a loop. On a known >> > bad commit (v4.1-rc2), out of 25 boots, I never saw it get past 21 >> > iterations of the loop. I have since tried to run a bisect from v3.19 to >> > v4.0 using 100 iterations as my criteria for a good commit. >> > >> > This resulted in the following first bad commit: >> > >> > 10359213d05acf804558bda7cc9b8422a828d1cd >> > (mm: incorporate read-only pages into transparent huge pages, 2015-02-11) >> > >> > Indeed, running the workload on v4.1-rc4 still produced the behavior, >> > but reverting the above commit gets me through 100 iterations of the >> > loop. >> > >> > I have not tried to reproduce on an x86 system. Turning on a bunch >> > of kernel debugging features *seems* to hide the problem. My config for >> > the XGene system is defconfig + CONFIG_BRIDGE and >> > CONFIG_POWER_RESET_XGENE. >> > >> > Please let me know if I can help test patches or other things I can >> > do to help. I'm afraid that by simply reading the patch I didn't see >> > anything obviously wrong with it which would cause this behavior. >> >> As further confirmation, could you try: >> >> echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan > > this returns -EINVAL. > > But I'm trying now with: > > echo never > /sys/kernel/mm/transparent_hugepage/enabled > >> >> and verify the problem goes away without having to revert the patch? > > will let you know, so far so good... > >> >> Accordingly you should reproduce much eaiser this way (setting >> $largevalue to 8192 or something, it doesn't matter). >> >> echo $largevalue > /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan >> echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs >> echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs >> >> Then push the system into swap with some memhog -r1000 xG. > > what is memhog? I couldn't find the utility in Google... > > I did try with the above settings and just push a bunch of data into > ramfs and tmpfs and indeed the sytem died very quickly (on v4.0-rc4). > >> >> The patch just allows readonly anon pages to be collapsed along with >> read-write ones, the vma permissions allows it, so they have to be >> swapcache pages, this is why swap shall be required. >> >> Perhaps there's some arch detail that needs fixing but it'll be easier >> to track it down once you have a way to reproduce fast. >> > Yes, would be great to be able to reproduce quickly. > > Thanks, > -Christoffer > Hi Christoffer, I'm trying to reproduce this on hardware here; but have been unable to thus far with 4.1-rc2 on a Xgene and Seattle systems. Also, I tried the memhog + pages_to_scan suggestion from Andrea. Maybe a silly question, where is your root filesystem located? Is there anything network mounted? Cheers, -- Steve -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by kanga.kvack.org (Postfix) with ESMTP id CE15F6B0073 for ; Tue, 26 May 2015 10:35:20 -0400 (EDT) Received: by lbbzk7 with SMTP id zk7so72076692lbb.0 for ; Tue, 26 May 2015 07:35:20 -0700 (PDT) Received: from mail-lb0-f175.google.com (mail-lb0-f175.google.com. [209.85.217.175]) by mx.google.com with ESMTPS id wm10si11093029lbb.149.2015.05.26.07.35.18 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 May 2015 07:35:19 -0700 (PDT) Received: by lbbqq2 with SMTP id qq2so71846659lbb.3 for ; Tue, 26 May 2015 07:35:18 -0700 (PDT) Date: Tue, 26 May 2015 16:35:47 +0200 From: Christoffer Dall Subject: Re: [BUG] Read-Only THP causes stalls (commit 10359213d) Message-ID: <20150526143547.GA22363@cbox> References: <20150524193404.GD16910@cbox> <20150525141525.GB26958@redhat.com> <20150526080848.GA27075@cbox> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-ID: To: Steve Capper Cc: Andrea Arcangeli , "linux-mm@kvack.org" , ebru.akagunduz@gmail.com, "akpm@linux-foundation.org" , "linux-kernel@vger.kernel.org" , kirill.shutemov@linux.intel.com, Rik van Riel , vbabka@suse.cz, Zhang Yanfei , Will Deacon , Andre Przywara , Marc Zyngier , "linux-arm-kernel@lists.infradead.org" Hi Steve, On Tue, May 26, 2015 at 03:24:20PM +0100, Steve Capper wrote: > >> On Sun, May 24, 2015 at 09:34:04PM +0200, Christoffer Dall wrote: > >> > Hi all, > >> > > >> > I noticed a regression on my arm64 APM X-Gene system a couple > >> > of weeks back. I would occassionally see the system lock up and see RCU > >> > stalls during the caching phase of kernbench. I then wrote a small > >> > script that does nothing but cache the files > >> > (http://paste.ubuntu.com/11324767/) and ran that in a loop. On a known > >> > bad commit (v4.1-rc2), out of 25 boots, I never saw it get past 21 > >> > iterations of the loop. I have since tried to run a bisect from v3.19 to > >> > v4.0 using 100 iterations as my criteria for a good commit. > >> > > >> > This resulted in the following first bad commit: > >> > > >> > 10359213d05acf804558bda7cc9b8422a828d1cd > >> > (mm: incorporate read-only pages into transparent huge pages, 2015-02-11) > >> > > >> > Indeed, running the workload on v4.1-rc4 still produced the behavior, > >> > but reverting the above commit gets me through 100 iterations of the > >> > loop. > >> > > >> > I have not tried to reproduce on an x86 system. Turning on a bunch > >> > of kernel debugging features *seems* to hide the problem. My config for > >> > the XGene system is defconfig + CONFIG_BRIDGE and > >> > CONFIG_POWER_RESET_XGENE. > >> > > >> > Please let me know if I can help test patches or other things I can > >> > do to help. I'm afraid that by simply reading the patch I didn't see > >> > anything obviously wrong with it which would cause this behavior. > >> > >> As further confirmation, could you try: > >> > >> echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan > > > > this returns -EINVAL. > > > > But I'm trying now with: > > > > echo never > /sys/kernel/mm/transparent_hugepage/enabled > > > >> > >> and verify the problem goes away without having to revert the patch? > > > > will let you know, so far so good... > > > >> > >> Accordingly you should reproduce much eaiser this way (setting > >> $largevalue to 8192 or something, it doesn't matter). > >> > >> echo $largevalue > /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan > >> echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs > >> echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs > >> > >> Then push the system into swap with some memhog -r1000 xG. > > > > what is memhog? I couldn't find the utility in Google... > > > > I did try with the above settings and just push a bunch of data into > > ramfs and tmpfs and indeed the sytem died very quickly (on v4.0-rc4). > > > >> > >> The patch just allows readonly anon pages to be collapsed along with > >> read-write ones, the vma permissions allows it, so they have to be > >> swapcache pages, this is why swap shall be required. > >> > >> Perhaps there's some arch detail that needs fixing but it'll be easier > >> to track it down once you have a way to reproduce fast. > >> > > Yes, would be great to be able to reproduce quickly. > > > I'm trying to reproduce this on hardware here; but have been unable to > thus far with 4.1-rc2 on a Xgene and Seattle systems. Really? That's concerning. I think Andre mentioned he could reproduce... How many iterations have you run the caching loop for? Are you using defconfig? I noticed that turning on debugging features was hiding the problem. > Also, I tried the memhog + pages_to_scan suggestion from Andrea. Any chance you could send me the memhog tool? > > Maybe a silly question, where is your root filesystem located? Is > there anything network mounted? > It's a regular ext4 on the local SATA disk. Ubuntu Trusty. Thanks, -Christoffer -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qk0-f174.google.com (mail-qk0-f174.google.com [209.85.220.174]) by kanga.kvack.org (Postfix) with ESMTP id 565E96B0073 for ; Tue, 26 May 2015 10:42:52 -0400 (EDT) Received: by qkgx75 with SMTP id x75so90569424qkg.1 for ; Tue, 26 May 2015 07:42:52 -0700 (PDT) Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id 30si9270642qks.30.2015.05.26.07.42.50 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 May 2015 07:42:51 -0700 (PDT) Date: Tue, 26 May 2015 16:42:47 +0200 From: Andrea Arcangeli Subject: Re: [BUG] Read-Only THP causes stalls (commit 10359213d) Message-ID: <20150526144247.GJ26958@redhat.com> References: <20150524193404.GD16910@cbox> <20150525141525.GB26958@redhat.com> <20150526080848.GA27075@cbox> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150526080848.GA27075@cbox> Sender: owner-linux-mm@kvack.org List-ID: To: Christoffer Dall Cc: linux-mm@kvack.org, ebru.akagunduz@gmail.com, akpm@linux-foundation.org, linux-kernel@vger.kernel.org, kirill.shutemov@linux.intel.com, riel@redhat.com, vbabka@suse.cz, zhangyanfei@cn.fujitsu.com, Will Deacon , Andre Przywara , Marc Zyngier , linux-arm-kernel@lists.infradead.org On Tue, May 26, 2015 at 10:08:48AM +0200, Christoffer Dall wrote: > > echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan > > this returns -EINVAL. > Oops sorry, I haven't re-read the code, pages_to_scan 0 does not make sense, it would only be useful for debugging purposes because it doesn't shut off khugepaged entirely, so it is ok that it returns -EINVAL, just it won't allow this debug tweak... > But I'm trying now with: > > echo never > /sys/kernel/mm/transparent_hugepage/enabled > > > > > and verify the problem goes away without having to revert the patch? > > will let you know, so far so good... I only intended to disable khugepaged, to validate the theory it was that patch that made the difference. Increasing the sleep time is equivalent to set pages_to_scan to 0, so you can use this instead: echo 3600000 >/sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs echo 3600000 >/sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs In addition to knowing if it still happens with THP disables, it's interesting to know also if it happens with THP enabled but khugepaged disabled. > what is memhog? I couldn't find the utility in Google... Somebody answered, yes it's from numactl. > I did try with the above settings and just push a bunch of data into > ramfs and tmpfs and indeed the sytem died very quickly (on v4.0-rc4). That's fine, memhog just was a way to hit swap. tmpfs pages aren't candidate for khugepaged THP collapsing, so it'd be perhaps quicker to reproduce with something like memhog that uses anonymous memory but it still happens, as long as you hit swap it's ok. If other arm don't exhibit this problem, perhaps it has to do with some difference in THP, I recall there were two models for arm. Thanks, Andrea -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ie0-f180.google.com (mail-ie0-f180.google.com [209.85.223.180]) by kanga.kvack.org (Postfix) with ESMTP id 523DB6B0121 for ; Tue, 26 May 2015 10:48:47 -0400 (EDT) Received: by iebgx4 with SMTP id gx4so93666739ieb.0 for ; Tue, 26 May 2015 07:48:47 -0700 (PDT) Received: from mail-ie0-f181.google.com (mail-ie0-f181.google.com. [209.85.223.181]) by mx.google.com with ESMTPS id yp6si10542027icb.65.2015.05.26.07.48.46 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 May 2015 07:48:46 -0700 (PDT) Received: by iesa3 with SMTP id a3so93807375ies.2 for ; Tue, 26 May 2015 07:48:46 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <20150526143547.GA22363@cbox> References: <20150524193404.GD16910@cbox> <20150525141525.GB26958@redhat.com> <20150526080848.GA27075@cbox> <20150526143547.GA22363@cbox> Date: Tue, 26 May 2015 15:48:46 +0100 Message-ID: Subject: Re: [BUG] Read-Only THP causes stalls (commit 10359213d) From: Steve Capper Content-Type: text/plain; charset=UTF-8 Sender: owner-linux-mm@kvack.org List-ID: To: Christoffer Dall Cc: Andrea Arcangeli , "linux-mm@kvack.org" , ebru.akagunduz@gmail.com, "akpm@linux-foundation.org" , "linux-kernel@vger.kernel.org" , kirill.shutemov@linux.intel.com, Rik van Riel , vbabka@suse.cz, Zhang Yanfei , Will Deacon , Andre Przywara , Marc Zyngier , "linux-arm-kernel@lists.infradead.org" On 26 May 2015 at 15:35, Christoffer Dall wrote: > Hi Steve, > > On Tue, May 26, 2015 at 03:24:20PM +0100, Steve Capper wrote: >> >> On Sun, May 24, 2015 at 09:34:04PM +0200, Christoffer Dall wrote: >> >> > Hi all, >> >> > >> >> > I noticed a regression on my arm64 APM X-Gene system a couple >> >> > of weeks back. I would occassionally see the system lock up and see RCU >> >> > stalls during the caching phase of kernbench. I then wrote a small >> >> > script that does nothing but cache the files >> >> > (http://paste.ubuntu.com/11324767/) and ran that in a loop. On a known >> >> > bad commit (v4.1-rc2), out of 25 boots, I never saw it get past 21 >> >> > iterations of the loop. I have since tried to run a bisect from v3.19 to >> >> > v4.0 using 100 iterations as my criteria for a good commit. >> >> > >> >> > This resulted in the following first bad commit: >> >> > >> >> > 10359213d05acf804558bda7cc9b8422a828d1cd >> >> > (mm: incorporate read-only pages into transparent huge pages, 2015-02-11) >> >> > >> >> > Indeed, running the workload on v4.1-rc4 still produced the behavior, >> >> > but reverting the above commit gets me through 100 iterations of the >> >> > loop. >> >> > >> >> > I have not tried to reproduce on an x86 system. Turning on a bunch >> >> > of kernel debugging features *seems* to hide the problem. My config for >> >> > the XGene system is defconfig + CONFIG_BRIDGE and >> >> > CONFIG_POWER_RESET_XGENE. >> >> > >> >> > Please let me know if I can help test patches or other things I can >> >> > do to help. I'm afraid that by simply reading the patch I didn't see >> >> > anything obviously wrong with it which would cause this behavior. >> >> >> >> As further confirmation, could you try: >> >> >> >> echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan >> > >> > this returns -EINVAL. >> > >> > But I'm trying now with: >> > >> > echo never > /sys/kernel/mm/transparent_hugepage/enabled >> > >> >> >> >> and verify the problem goes away without having to revert the patch? >> > >> > will let you know, so far so good... >> > >> >> >> >> Accordingly you should reproduce much eaiser this way (setting >> >> $largevalue to 8192 or something, it doesn't matter). >> >> >> >> echo $largevalue > /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan >> >> echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs >> >> echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs >> >> >> >> Then push the system into swap with some memhog -r1000 xG. >> > >> > what is memhog? I couldn't find the utility in Google... >> > >> > I did try with the above settings and just push a bunch of data into >> > ramfs and tmpfs and indeed the sytem died very quickly (on v4.0-rc4). >> > >> >> >> >> The patch just allows readonly anon pages to be collapsed along with >> >> read-write ones, the vma permissions allows it, so they have to be >> >> swapcache pages, this is why swap shall be required. >> >> >> >> Perhaps there's some arch detail that needs fixing but it'll be easier >> >> to track it down once you have a way to reproduce fast. >> >> >> > Yes, would be great to be able to reproduce quickly. >> > > >> I'm trying to reproduce this on hardware here; but have been unable to >> thus far with 4.1-rc2 on a Xgene and Seattle systems. > > Really? That's concerning. I think Andre mentioned he could > reproduce... > > How many iterations have you run the caching loop for? > > Are you using defconfig? I noticed that turning on debugging features > was hiding the problem. > >> Also, I tried the memhog + pages_to_scan suggestion from Andrea. > > Any chance you could send me the memhog tool? > >> >> Maybe a silly question, where is your root filesystem located? Is >> there anything network mounted? >> > It's a regular ext4 on the local SATA disk. Ubuntu Trusty. > > Thanks, > -Christoffer Sending an email to lakml appears to have been enough to make it hang on the Xgene :-). The system is completely frozen, not even the serial port works. On Seattle, I've hit 100 iterations multiple times without any problems. Investigating... Cheers, -- Steve -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-f169.google.com (mail-wi0-f169.google.com [209.85.212.169]) by kanga.kvack.org (Postfix) with ESMTP id 18ADE6B0121 for ; Tue, 26 May 2015 10:59:06 -0400 (EDT) Received: by wicmx19 with SMTP id mx19so67552154wic.0 for ; Tue, 26 May 2015 07:59:05 -0700 (PDT) Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id p3si18578603wia.63.2015.05.26.07.59.03 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 May 2015 07:59:04 -0700 (PDT) Date: Tue, 26 May 2015 16:58:46 +0200 From: Andrea Arcangeli Subject: Re: [BUG] Read-Only THP causes stalls (commit 10359213d) Message-ID: <20150526145846.GK26958@redhat.com> References: <20150524193404.GD16910@cbox> <20150525141525.GB26958@redhat.com> <20150526080848.GA27075@cbox> <20150526143547.GA22363@cbox> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150526143547.GA22363@cbox> Sender: owner-linux-mm@kvack.org List-ID: To: Christoffer Dall Cc: Steve Capper , "linux-mm@kvack.org" , ebru.akagunduz@gmail.com, "akpm@linux-foundation.org" , "linux-kernel@vger.kernel.org" , kirill.shutemov@linux.intel.com, Rik van Riel , vbabka@suse.cz, Zhang Yanfei , Will Deacon , Andre Przywara , Marc Zyngier , "linux-arm-kernel@lists.infradead.org" On Tue, May 26, 2015 at 04:35:47PM +0200, Christoffer Dall wrote: > Any chance you could send me the memhog tool? memhog is just the first that come to mind because I got it preinstalled everywhere (I only miss it on cyanogenmod as there's no numactl there... yet). Anything else would do as well, as long as you allocate lots of anonymous memory (malloc(); bzero() or just write 1 byte every 4k). The tmpfs trick was fine as well as you'd end up swapping the anonymous memory allocated by the running apps. This would be the python version which I actually used sometime if I couldn't find something preinstalled and I didn't want to install packages. echo 1 >/proc/sys/vm/overcommit_memory python a = "a" while True: a += a This is the more polished way, I just happen to have it installed everywhere (except the cellphone) so I tend to use it, I think it's simpler to install the numactl package. https://github.com/numactl/numactl/blob/master/memhog.c -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org