From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 891ECC001B3 for ; Mon, 3 Jul 2023 18:45:15 +0000 (UTC) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linuxfoundation.org header.i=@linuxfoundation.org header.a=rsa-sha256 header.s=korg header.b=Z7K9Fath; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4QvvxQ0N2Tz3c2B for ; Tue, 4 Jul 2023 04:45:14 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=linuxfoundation.org header.i=@linuxfoundation.org header.a=rsa-sha256 header.s=korg header.b=Z7K9Fath; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linuxfoundation.org (client-ip=139.178.84.217; helo=dfw.source.kernel.org; envelope-from=gregkh@linuxfoundation.org; receiver=lists.ozlabs.org) Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4QvvwR4DKDz30dw for ; Tue, 4 Jul 2023 04:44:22 +1000 (AEST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id A99D16100B; Mon, 3 Jul 2023 18:44:20 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C0D2AC433C7; Mon, 3 Jul 2023 18:44:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1688409860; bh=2uAqICqoHfr9JtNMBA+xJ3OMXBg414ERFpRhYQggwas=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=Z7K9FathaHPI6/wxy5SOzbw+Mn3X/luBzM657rDx4HJNm/6VjummZyAFF/3sfzVYF LxMRVIqsVH39SZ+5khv50NT8+BLJktP3dug43pWtYN8QAKJhSueo7i4IHOa7osISsg 6L983hQun2qcLOjWngQMIZXxjvWbVIiOoK+4uq0w= Date: Mon, 3 Jul 2023 20:44:17 +0200 From: Greg KH To: Suren Baghdasaryan Subject: Re: Fwd: Memory corruption in multithreaded user space program while calling fork Message-ID: <2023070359-evasive-regroup-f3b8@gregkh> References: <5c7455db-4ed8-b54f-e2d5-d2811908123d@leemhuis.info> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jacob Young , Linux regressions mailing list , Linux Kernel Mailing List , Linux Memory Management , Bagas Sanjaya , Andrew Morton , Laurent Dufour , Linux PowerPC , Linux ARM Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Mon, Jul 03, 2023 at 11:27:19AM -0700, Suren Baghdasaryan wrote: > On Mon, Jul 3, 2023 at 11:08 AM Suren Baghdasaryan wrote: > > > > On Mon, Jul 3, 2023 at 2:53 AM Linux regression tracking (Thorsten > > Leemhuis) wrote: > > > > > > On 02.07.23 14:27, Bagas Sanjaya wrote: > > > > I notice a regression report on Bugzilla [1]. Quoting from it: > > > > > > > >> After upgrading to kernel version 6.4.0 from 6.3.9, I noticed frequent but random crashes in a user space program. After a lot of reduction, I have come up with the following reproducer program: > > > > [...] > > > >> After tuning the various parameters for my computer, exit code 2, which indicates that memory corruption was detected, occurs approximately 99% of the time. Exit code 1, which occurs approximately 1% of the time, means it ran out of statically-allocated memory before reproducing the issue, and increasing the memory usage any more only leads to diminishing returns. There is also something like a 0.1% chance that it segfaults due to memory corruption elsewhere than in the statically-allocated buffer. > > > >> > > > >> With this reproducer in hand, I was able to perform the following bisection: > > > > [...] > > > > > > > > See Bugzilla for the full thread. > > > > > > Additional details from > > > https://bugzilla.kernel.org/show_bug.cgi?id=217624#c5 : > > > > > > ``` > > > I can confirm that v6.4 with 0bff0aaea03e2a3ed6bfa302155cca8a432a1829 > > > reverted no longer causes any memory corruption with either my > > > reproducer or the original program. > > > ``` > > > > > > FWIW: 0bff0aaea03 ("x86/mm: try VMA lock-based page fault handling > > > first") [merged for v6.4-rc1, authored by Suren Baghdasaryan [already CCed]] > > > > > > That's the same commit that causes build problems with go: > > > > > > https://lore.kernel.org/all/dbdef34c-3a07-5951-e1ae-e9c6e3cdf51b@kernel.org/ > > > > Thanks! I'll investigate this later today. After discussing with > > Andrew, we would like to disable CONFIG_PER_VMA_LOCK by default until > > the issue is fixed. I'll post a patch shortly. > > Posted at: https://lore.kernel.org/all/20230703182150.2193578-1-surenb@google.com/ As that change fixes something in 6.4, why not cc: stable on it as well? thanks, greg k-h From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A1AC1259D for ; Mon, 3 Jul 2023 18:44:20 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C0D2AC433C7; Mon, 3 Jul 2023 18:44:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1688409860; bh=2uAqICqoHfr9JtNMBA+xJ3OMXBg414ERFpRhYQggwas=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=Z7K9FathaHPI6/wxy5SOzbw+Mn3X/luBzM657rDx4HJNm/6VjummZyAFF/3sfzVYF LxMRVIqsVH39SZ+5khv50NT8+BLJktP3dug43pWtYN8QAKJhSueo7i4IHOa7osISsg 6L983hQun2qcLOjWngQMIZXxjvWbVIiOoK+4uq0w= Date: Mon, 3 Jul 2023 20:44:17 +0200 From: Greg KH To: Suren Baghdasaryan Cc: Linux regressions mailing list , Bagas Sanjaya , Andrew Morton , Jacob Young , Laurent Dufour , Linux Kernel Mailing List , Linux Memory Management , Linux PowerPC , Linux ARM Subject: Re: Fwd: Memory corruption in multithreaded user space program while calling fork Message-ID: <2023070359-evasive-regroup-f3b8@gregkh> References: <5c7455db-4ed8-b54f-e2d5-d2811908123d@leemhuis.info> Precedence: bulk X-Mailing-List: regressions@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: On Mon, Jul 03, 2023 at 11:27:19AM -0700, Suren Baghdasaryan wrote: > On Mon, Jul 3, 2023 at 11:08 AM Suren Baghdasaryan wrote: > > > > On Mon, Jul 3, 2023 at 2:53 AM Linux regression tracking (Thorsten > > Leemhuis) wrote: > > > > > > On 02.07.23 14:27, Bagas Sanjaya wrote: > > > > I notice a regression report on Bugzilla [1]. Quoting from it: > > > > > > > >> After upgrading to kernel version 6.4.0 from 6.3.9, I noticed frequent but random crashes in a user space program. After a lot of reduction, I have come up with the following reproducer program: > > > > [...] > > > >> After tuning the various parameters for my computer, exit code 2, which indicates that memory corruption was detected, occurs approximately 99% of the time. Exit code 1, which occurs approximately 1% of the time, means it ran out of statically-allocated memory before reproducing the issue, and increasing the memory usage any more only leads to diminishing returns. There is also something like a 0.1% chance that it segfaults due to memory corruption elsewhere than in the statically-allocated buffer. > > > >> > > > >> With this reproducer in hand, I was able to perform the following bisection: > > > > [...] > > > > > > > > See Bugzilla for the full thread. > > > > > > Additional details from > > > https://bugzilla.kernel.org/show_bug.cgi?id=217624#c5 : > > > > > > ``` > > > I can confirm that v6.4 with 0bff0aaea03e2a3ed6bfa302155cca8a432a1829 > > > reverted no longer causes any memory corruption with either my > > > reproducer or the original program. > > > ``` > > > > > > FWIW: 0bff0aaea03 ("x86/mm: try VMA lock-based page fault handling > > > first") [merged for v6.4-rc1, authored by Suren Baghdasaryan [already CCed]] > > > > > > That's the same commit that causes build problems with go: > > > > > > https://lore.kernel.org/all/dbdef34c-3a07-5951-e1ae-e9c6e3cdf51b@kernel.org/ > > > > Thanks! I'll investigate this later today. After discussing with > > Andrew, we would like to disable CONFIG_PER_VMA_LOCK by default until > > the issue is fixed. I'll post a patch shortly. > > Posted at: https://lore.kernel.org/all/20230703182150.2193578-1-surenb@google.com/ As that change fixes something in 6.4, why not cc: stable on it as well? thanks, greg k-h From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 13416C001B3 for ; Mon, 3 Jul 2023 18:44:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=I33v1Upwfpmkx/BiKqs+kJrvP96h5VZ9YH3sLk5CiPU=; b=DOgJBWzPYvJurg s8e6hrxLpNmLSRxz3lx4EKDFVRTicADX3tg3Bo72ZjPtzFxtEdWyHtbt2qS6dlwgtOQtx/QrIYe7w y7ltIToaOXPTD1/Rmluj1cyD591BEZvtMzvw+au2G1cdf/+73fYn2pi7X2mJDzfi2gpwaYn0KRbeB FT4PHRsOpAEof/ZMAbwvqi3/UsBYGoOqpRsLlailO/clEDlC/VpPKT4xrOOzJDcyses32gb825kya 8fe6gYEQRJXyTNhdQbsUnGDMik0WgEK6jMSreS774ZYZcmh2BbZI8OjDJBlJrVtXL8STzG6ALyz1f TyY9engfYHIucG12xkDA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qGOXA-00BEH4-1T; Mon, 03 Jul 2023 18:44:24 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qGOX7-00BEGg-0h for linux-arm-kernel@lists.infradead.org; Mon, 03 Jul 2023 18:44:22 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id A99D16100B; Mon, 3 Jul 2023 18:44:20 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C0D2AC433C7; Mon, 3 Jul 2023 18:44:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1688409860; bh=2uAqICqoHfr9JtNMBA+xJ3OMXBg414ERFpRhYQggwas=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=Z7K9FathaHPI6/wxy5SOzbw+Mn3X/luBzM657rDx4HJNm/6VjummZyAFF/3sfzVYF LxMRVIqsVH39SZ+5khv50NT8+BLJktP3dug43pWtYN8QAKJhSueo7i4IHOa7osISsg 6L983hQun2qcLOjWngQMIZXxjvWbVIiOoK+4uq0w= Date: Mon, 3 Jul 2023 20:44:17 +0200 From: Greg KH To: Suren Baghdasaryan Cc: Linux regressions mailing list , Bagas Sanjaya , Andrew Morton , Jacob Young , Laurent Dufour , Linux Kernel Mailing List , Linux Memory Management , Linux PowerPC , Linux ARM Subject: Re: Fwd: Memory corruption in multithreaded user space program while calling fork Message-ID: <2023070359-evasive-regroup-f3b8@gregkh> References: <5c7455db-4ed8-b54f-e2d5-d2811908123d@leemhuis.info> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230703_114421_345671_3A03D918 X-CRM114-Status: GOOD ( 22.50 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gTW9uLCBKdWwgMDMsIDIwMjMgYXQgMTE6Mjc6MTlBTSAtMDcwMCwgU3VyZW4gQmFnaGRhc2Fy eWFuIHdyb3RlOgo+IE9uIE1vbiwgSnVsIDMsIDIwMjMgYXQgMTE6MDjigK9BTSBTdXJlbiBCYWdo ZGFzYXJ5YW4gPHN1cmVuYkBnb29nbGUuY29tPiB3cm90ZToKPiA+Cj4gPiBPbiBNb24sIEp1bCAz LCAyMDIzIGF0IDI6NTPigK9BTSBMaW51eCByZWdyZXNzaW9uIHRyYWNraW5nIChUaG9yc3Rlbgo+ ID4gTGVlbWh1aXMpIDxyZWdyZXNzaW9uc0BsZWVtaHVpcy5pbmZvPiB3cm90ZToKPiA+ID4KPiA+ ID4gT24gMDIuMDcuMjMgMTQ6MjcsIEJhZ2FzIFNhbmpheWEgd3JvdGU6Cj4gPiA+ID4gSSBub3Rp Y2UgYSByZWdyZXNzaW9uIHJlcG9ydCBvbiBCdWd6aWxsYSBbMV0uIFF1b3RpbmcgZnJvbSBpdDoK PiA+ID4gPgo+ID4gPiA+PiBBZnRlciB1cGdyYWRpbmcgdG8ga2VybmVsIHZlcnNpb24gNi40LjAg ZnJvbSA2LjMuOSwgSSBub3RpY2VkIGZyZXF1ZW50IGJ1dCByYW5kb20gY3Jhc2hlcyBpbiBhIHVz ZXIgc3BhY2UgcHJvZ3JhbS4gIEFmdGVyIGEgbG90IG9mIHJlZHVjdGlvbiwgSSBoYXZlIGNvbWUg dXAgd2l0aCB0aGUgZm9sbG93aW5nIHJlcHJvZHVjZXIgcHJvZ3JhbToKPiA+ID4gPiBbLi4uXQo+ ID4gPiA+PiBBZnRlciB0dW5pbmcgdGhlIHZhcmlvdXMgcGFyYW1ldGVycyBmb3IgbXkgY29tcHV0 ZXIsIGV4aXQgY29kZSAyLCB3aGljaCBpbmRpY2F0ZXMgdGhhdCBtZW1vcnkgY29ycnVwdGlvbiB3 YXMgZGV0ZWN0ZWQsIG9jY3VycyBhcHByb3hpbWF0ZWx5IDk5JSBvZiB0aGUgdGltZS4gIEV4aXQg Y29kZSAxLCB3aGljaCBvY2N1cnMgYXBwcm94aW1hdGVseSAxJSBvZiB0aGUgdGltZSwgbWVhbnMg aXQgcmFuIG91dCBvZiBzdGF0aWNhbGx5LWFsbG9jYXRlZCBtZW1vcnkgYmVmb3JlIHJlcHJvZHVj aW5nIHRoZSBpc3N1ZSwgYW5kIGluY3JlYXNpbmcgdGhlIG1lbW9yeSB1c2FnZSBhbnkgbW9yZSBv bmx5IGxlYWRzIHRvIGRpbWluaXNoaW5nIHJldHVybnMuICBUaGVyZSBpcyBhbHNvIHNvbWV0aGlu ZyBsaWtlIGEgMC4xJSBjaGFuY2UgdGhhdCBpdCBzZWdmYXVsdHMgZHVlIHRvIG1lbW9yeSBjb3Jy dXB0aW9uIGVsc2V3aGVyZSB0aGFuIGluIHRoZSBzdGF0aWNhbGx5LWFsbG9jYXRlZCBidWZmZXIu Cj4gPiA+ID4+Cj4gPiA+ID4+IFdpdGggdGhpcyByZXByb2R1Y2VyIGluIGhhbmQsIEkgd2FzIGFi bGUgdG8gcGVyZm9ybSB0aGUgZm9sbG93aW5nIGJpc2VjdGlvbjoKPiA+ID4gPiBbLi4uXQo+ID4g PiA+Cj4gPiA+ID4gU2VlIEJ1Z3ppbGxhIGZvciB0aGUgZnVsbCB0aHJlYWQuCj4gPiA+Cj4gPiA+ IEFkZGl0aW9uYWwgZGV0YWlscyBmcm9tCj4gPiA+IGh0dHBzOi8vYnVnemlsbGEua2VybmVsLm9y Zy9zaG93X2J1Zy5jZ2k/aWQ9MjE3NjI0I2M1IDoKPiA+ID4KPiA+ID4gYGBgCj4gPiA+IEkgY2Fu IGNvbmZpcm0gdGhhdCB2Ni40IHdpdGggMGJmZjBhYWVhMDNlMmEzZWQ2YmZhMzAyMTU1Y2NhOGE0 MzJhMTgyOQo+ID4gPiByZXZlcnRlZCBubyBsb25nZXIgY2F1c2VzIGFueSBtZW1vcnkgY29ycnVw dGlvbiB3aXRoIGVpdGhlciBteQo+ID4gPiByZXByb2R1Y2VyIG9yIHRoZSBvcmlnaW5hbCBwcm9n cmFtLgo+ID4gPiBgYGAKPiA+ID4KPiA+ID4gRldJVzogMGJmZjBhYWVhMDMgKCJ4ODYvbW06IHRy eSBWTUEgbG9jay1iYXNlZCBwYWdlIGZhdWx0IGhhbmRsaW5nCj4gPiA+IGZpcnN0IikgW21lcmdl ZCBmb3IgdjYuNC1yYzEsIGF1dGhvcmVkIGJ5IFN1cmVuIEJhZ2hkYXNhcnlhbiBbYWxyZWFkeSBD Q2VkXV0KPiA+ID4KPiA+ID4gVGhhdCdzIHRoZSBzYW1lIGNvbW1pdCB0aGF0IGNhdXNlcyBidWls ZCBwcm9ibGVtcyB3aXRoIGdvOgo+ID4gPgo+ID4gPiBodHRwczovL2xvcmUua2VybmVsLm9yZy9h bGwvZGJkZWYzNGMtM2EwNy01OTUxLWUxYWUtZTljNmUzY2RmNTFiQGtlcm5lbC5vcmcvCj4gPgo+ ID4gVGhhbmtzISBJJ2xsIGludmVzdGlnYXRlIHRoaXMgbGF0ZXIgdG9kYXkuIEFmdGVyIGRpc2N1 c3Npbmcgd2l0aAo+ID4gQW5kcmV3LCB3ZSB3b3VsZCBsaWtlIHRvIGRpc2FibGUgQ09ORklHX1BF Ul9WTUFfTE9DSyBieSBkZWZhdWx0IHVudGlsCj4gPiB0aGUgaXNzdWUgaXMgZml4ZWQuIEknbGwg cG9zdCBhIHBhdGNoIHNob3J0bHkuCj4gCj4gUG9zdGVkIGF0OiBodHRwczovL2xvcmUua2VybmVs Lm9yZy9hbGwvMjAyMzA3MDMxODIxNTAuMjE5MzU3OC0xLXN1cmVuYkBnb29nbGUuY29tLwoKQXMg dGhhdCBjaGFuZ2UgZml4ZXMgc29tZXRoaW5nIGluIDYuNCwgd2h5IG5vdCBjYzogc3RhYmxlIG9u IGl0IGFzIHdlbGw/Cgp0aGFua3MsCgpncmVnIGstaAoKX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX18KbGludXgtYXJtLWtlcm5lbCBtYWlsaW5nIGxpc3QKbGlu dXgtYXJtLWtlcm5lbEBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5pbmZyYWRlYWQu b3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtYXJtLWtlcm5lbAo=