From mboxrd@z Thu Jan 1 00:00:00 1970 Date: Tue, 16 Nov 2021 20:06:38 +0000 From: "Russell King (Oracle)" Subject: Re: [PATCH v3 7/7] ARM: implement support for vmap'ed stacks Message-ID: References: <20211115111816.3911213-1-ardb@kernel.org> <20211115111816.3911213-8-ardb@kernel.org> MIME-Version: 1.0 In-Reply-To: Sender: Russell King (Oracle) Content-Type: text/plain; charset=us-ascii Content-Disposition: inline List-ID: To: Ard Biesheuvel Cc: Guillaume Tucker , Tony Lindgren , linux-omap , Linux ARM , Nicolas Pitre , Arnd Bergmann , Kees Cook , Keith Packard , Linus Walleij , Nick Desaulniers , "kernelci@groups.io" On Tue, Nov 16, 2021 at 08:28:02PM +0100, Ard Biesheuvel wrote: > (+ Tony and linux-omap@) > > On Tue, 16 Nov 2021 at 10:23, Guillaume Tucker > wrote: > > > > Hi Ard, > > > > Please see the bisection report below about a boot failure on > > omap4-panda which is pointing to this patch. > > > > Reports aren't automatically sent to the public while we're > > trialing new bisection features on kernelci.org but this one > > looks valid. > > > > Some more details can be found here: > > > > https://linux.kernelci.org/test/case/id/6191b1b97c175a5ade335948/ > > > > It seems like the kernel just froze after about 3 seconds without > > any obvious errors in the log. > > > > Please let us know if you need any help debugging this issue or > > if you have a fix to try. > > > > Thanks for the report. > > I wonder if this might be related to low level platform code running > off a different stack (maybe in SRAM?) when an interrupt is taken? Or > using a different set of page tables that are out of sync in terms of > VMALLOC space mappings? > > Could anyone who speaks OMAP please take a look at the linked boot > log, and hopefully make sense of it? > > For background, this series enables vmap'ed stacks support for ARMv7, > which means that the entry code checks whether the stack pointer may > be pointing into the guard region before the vmalloc'ed stack, and > kills the task if it looks like the kernel stack overflowed. > > Here's another instance: > https://linux.kernelci.org/build/id/6193fa5c6c4e1d02bd3358ff/ > > Everything builds and boots happily, but odd things happen on OMAP > based devices: Panda just gives up right after discovering the USB > controller, and Beagle-XM just starts showing all kinds of weird > crashes at roughly the same point in the boot. I haven't looked at the logs yet... but there may be a more fundamental reason that it may be stalling. vmalloc space is lazily mapped to process page tables that the allocation did not happen inside - specifically the L1 entries. When a new thread is created, you're vmalloc()ing a kernel stack. This is done in the parent task for the child task. If the child task doesn't contain the L1 entry for its vmalloc'd stack, then the first stack access by the child will fault. The fault processing will be done in the child's context, so we immediately try to save the state to the child's kernel stack, which is not yet mapped. The result is another fault, which triggers yet another fault, etc. -- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last! From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 509A0C433F5 for ; Tue, 16 Nov 2021 20:08:34 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1F6B561BAA for ; Tue, 16 Nov 2021 20:08:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 1F6B561BAA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=armlinux.org.uk Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ncIxbbLgeitqwqXq3eG+8wrAaFdVetn2DGHILXnlCMo=; b=px9k3MZuK1T/Hx bypwbp/C/2qhlp0O9baLqIZErmSWPqEAkzjoU+HMcHCmOd/mEL8q3xqxRyNQ7mKmQ5pDpcpadjsp3 FH0eul8UwdhzZ4Hq4o0IpQSnlMnW9BIENcFS1IUHZEa8mbKTngHNE3pSMcZd5aSYzHUL5NG0CGob+ F9Hk6u8EHT7eb0BG/wC55urhqarrgk47fOoiYuvyWkot45G2Asg30WnbEdI39S5097fZrPEM0E46a ShZckw37ARXDg+3cGy1F25wH858cZ9WDpDF2vlD73vmVBAxvA0xgEBRFKEIh6e9GbPz4nL4OmMsDd Djf2BBNCGdBgMl2z4dzg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mn4jJ-002g1z-5V; Tue, 16 Nov 2021 20:06:57 +0000 Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mn4jE-002g1G-Lf for linux-arm-kernel@lists.infradead.org; Tue, 16 Nov 2021 20:06:54 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Sender:In-Reply-To:Content-Type: MIME-Version:References:Message-ID:Subject:Cc:To:From:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=6UtycufX3/huPXihxdrXM9vDKWY/pC3UaEVeBhqAA4Y=; b=D5nzpdrhn2H/ITqWlJqbzT+KcK +ta7U9rPuc3zLHafS3gZxKdP3IomKOMdu3lcZYb9ex7XIYKZxj5Leu5a7S1s6JwsfTRFC7zJhn4nW F9Z+EGqKzwpznfCb08V0FbxoTG0g2110JS9N73W8j8NcwstyyhKSu4zBNgI6DIuwp6ypwxqk7vb+q KF8N8vs97Up+2Y7/q7va/VHMpg8WsGysWR7xiNmpaUpkwc8cfzi2MaLf3Bc5Vzc1Dzo0Z2jsJQi/o utRw47Vkn38+tX2wlGqMmWG7KMBxwIsmN7/+fDmiJ7QNxHgTO9jr9KxNDS1OVScfojnHeRJeOPsbW JZUdZCEA==; Received: from shell.armlinux.org.uk ([fd8f:7570:feb6:1:5054:ff:fe00:4ec]:55666) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mn4j5-00011N-Eh; Tue, 16 Nov 2021 20:06:43 +0000 Received: from linux by shell.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1mn4j0-0002FT-Nk; Tue, 16 Nov 2021 20:06:38 +0000 Date: Tue, 16 Nov 2021 20:06:38 +0000 From: "Russell King (Oracle)" To: Ard Biesheuvel Cc: Guillaume Tucker , Tony Lindgren , linux-omap , Linux ARM , Nicolas Pitre , Arnd Bergmann , Kees Cook , Keith Packard , Linus Walleij , Nick Desaulniers , "kernelci@groups.io" Subject: Re: [PATCH v3 7/7] ARM: implement support for vmap'ed stacks Message-ID: References: <20211115111816.3911213-1-ardb@kernel.org> <20211115111816.3911213-8-ardb@kernel.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211116_120652_734344_0F591307 X-CRM114-Status: GOOD ( 31.99 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Nov 16, 2021 at 08:28:02PM +0100, Ard Biesheuvel wrote: > (+ Tony and linux-omap@) > > On Tue, 16 Nov 2021 at 10:23, Guillaume Tucker > wrote: > > > > Hi Ard, > > > > Please see the bisection report below about a boot failure on > > omap4-panda which is pointing to this patch. > > > > Reports aren't automatically sent to the public while we're > > trialing new bisection features on kernelci.org but this one > > looks valid. > > > > Some more details can be found here: > > > > https://linux.kernelci.org/test/case/id/6191b1b97c175a5ade335948/ > > > > It seems like the kernel just froze after about 3 seconds without > > any obvious errors in the log. > > > > Please let us know if you need any help debugging this issue or > > if you have a fix to try. > > > > Thanks for the report. > > I wonder if this might be related to low level platform code running > off a different stack (maybe in SRAM?) when an interrupt is taken? Or > using a different set of page tables that are out of sync in terms of > VMALLOC space mappings? > > Could anyone who speaks OMAP please take a look at the linked boot > log, and hopefully make sense of it? > > For background, this series enables vmap'ed stacks support for ARMv7, > which means that the entry code checks whether the stack pointer may > be pointing into the guard region before the vmalloc'ed stack, and > kills the task if it looks like the kernel stack overflowed. > > Here's another instance: > https://linux.kernelci.org/build/id/6193fa5c6c4e1d02bd3358ff/ > > Everything builds and boots happily, but odd things happen on OMAP > based devices: Panda just gives up right after discovering the USB > controller, and Beagle-XM just starts showing all kinds of weird > crashes at roughly the same point in the boot. I haven't looked at the logs yet... but there may be a more fundamental reason that it may be stalling. vmalloc space is lazily mapped to process page tables that the allocation did not happen inside - specifically the L1 entries. When a new thread is created, you're vmalloc()ing a kernel stack. This is done in the parent task for the child task. If the child task doesn't contain the L1 entry for its vmalloc'd stack, then the first stack access by the child will fault. The fault processing will be done in the child's context, so we immediately try to save the state to the child's kernel stack, which is not yet mapped. The result is another fault, which triggers yet another fault, etc. -- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last! _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel