From mboxrd@z Thu Jan 1 00:00:00 1970 From: Yu Zhao Subject: [PATCH v3 3/3] arm64: mm: enable per pmd page table lock Date: Sat, 9 Mar 2019 18:19:06 -0700 Message-ID: <20190310011906.254635-3-yuzhao@google.com> References: <20190218231319.178224-1-yuzhao@google.com> <20190310011906.254635-1-yuzhao@google.com> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Return-path: In-Reply-To: <20190310011906.254635-1-yuzhao@google.com> Sender: linux-kernel-owner@vger.kernel.org To: Catalin Marinas , Will Deacon , Mark Rutland Cc: "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Joel Fernandes , "Kirill A . Shutemov" , Ard Biesheuvel , Chintan Pandya , Jun Yao , Laura Abbott , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, Yu Zhao List-Id: linux-arch.vger.kernel.org Switch from per mm_struct to per pmd page table lock by enabling ARCH_ENABLE_SPLIT_PMD_PTLOCK. This provides better granularity for large system. I'm not sure if there is contention on mm->page_table_lock. Given the option comes at no cost (apart from initializing more spin locks), why not enable it now. We only do so when pmd is not folded, so we don't mistakenly call pgtable_pmd_page_ctor() on pud or p4d in pgd_pgtable_alloc(). (We check shift against PMD_SHIFT, which is same as PUD_SHIFT when pmd is folded). Signed-off-by: Yu Zhao --- arch/arm64/Kconfig | 3 +++ arch/arm64/include/asm/pgalloc.h | 12 +++++++++++- arch/arm64/include/asm/tlb.h | 5 ++++- 3 files changed, 18 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index cfbf307d6dc4..a3b1b789f766 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -872,6 +872,9 @@ config ARCH_WANT_HUGE_PMD_SHARE config ARCH_HAS_CACHE_LINE_SIZE def_bool y +config ARCH_ENABLE_SPLIT_PMD_PTLOCK + def_bool y if PGTABLE_LEVELS > 2 + config SECCOMP bool "Enable seccomp to safely compute untrusted bytecode" ---help--- diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 52fa47c73bf0..dabba4b2c61f 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -33,12 +33,22 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) { - return (pmd_t *)__get_free_page(PGALLOC_GFP); + struct page *page; + + page = alloc_page(PGALLOC_GFP); + if (!page) + return NULL; + if (!pgtable_pmd_page_ctor(page)) { + __free_page(page); + return NULL; + } + return page_address(page); } static inline void pmd_free(struct mm_struct *mm, pmd_t *pmdp) { BUG_ON((unsigned long)pmdp & (PAGE_SIZE-1)); + pgtable_pmd_page_dtor(virt_to_page(pmdp)); free_page((unsigned long)pmdp); } diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 106fdc951b6e..4e3becfed387 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -62,7 +62,10 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) { - tlb_remove_table(tlb, virt_to_page(pmdp)); + struct page *page = virt_to_page(pmdp); + + pgtable_pmd_page_dtor(page); + tlb_remove_table(tlb, page); } #endif -- 2.21.0.360.g471c308f928-goog From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.0 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D1CFC43381 for ; Sun, 10 Mar 2019 01:19:49 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D09CA206BA for ; Sun, 10 Mar 2019 01:19:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="eyieBnqZ"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="ne3qu3Fx" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D09CA206BA Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=zA7in20WoFzQOw3ejEpfzzWItfZpc0S3oBp9nm8MRy0=; b=eyieBnqZfkhyRz 83pn+hI3pVLEbWY7aLi/NRomE7mC/Mk0nNYJgosIZKwlqzWKigEOj3pbhe/Kygm7KAEXFiOVUmPBa fTb/fDdfSkRpNwg98mnIswQJ9jS3hTV9NQIC3UrJ+8tMLCey5smOKi8PKm+VQUCXpkADbgLMr5VRd xjB5OP/U5mNl9pqy39CJHdNJM8X3Zva7LDN27+6wf72L/5x2i/FI5Ts96WD0F9Nv5kPOlxIooMLhu /0jgTZk+7634zenIHOtQQDvfnS27YgL14L78r7XxhxGlGURot05enHC2b11tXyYSeJ5mlNsCkuLzg oTAf4MWXktHlno1zp++A==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1h2n7p-00023Z-71; Sun, 10 Mar 2019 01:19:37 +0000 Received: from mail-qt1-x84a.google.com ([2607:f8b0:4864:20::84a]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1h2n7S-0001Zr-VR for linux-arm-kernel@lists.infradead.org; Sun, 10 Mar 2019 01:19:16 +0000 Received: by mail-qt1-x84a.google.com with SMTP id 35so1694044qtq.5 for ; Sat, 09 Mar 2019 17:19:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=+4D1c6JWSWE8QT+yoi5CUv0ZKRIbRAhMzGEdnkssg9A=; b=ne3qu3FxuNnaxTHNaR0KvnSI+dxHillTroSbc2a23I9NeDzowB1PzvJmROLLgBDQkQ 4isyzzDBrnqmSk42wEbalVZQRdtGPCAh3G/B2qqSQtX3/BY5STPBC/0sf8VymLZxZnqv TFyb/EiMA0LE7HOd2jjMWMn2DLhBGtYXdC/SxMBWDrB+9bVk7z2HS1cqhsOLbpOYAqnq EWEylhDMVOFJ8n8MACWSvz57cSQhFCTOFpDGhLPApw5kwjO/4JcLuQBWDIBdVRtBa8MN 6AY17se4aFWkWszozRMszsYdPmcuPedaKhntA1DBHOnvx8MI+7RGhbs5ZQZ6UykgKoku kghg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=+4D1c6JWSWE8QT+yoi5CUv0ZKRIbRAhMzGEdnkssg9A=; b=VONMwBhwpQT4njxwMvKKS4SI9z9lrqZ2EKAuSOq/dMxCM3uhcw5etSbP151qojHSXn E38DO2HrDUXW2j+LsJhUBkle4feNqmSA17htI/aLlFhEOjHCqqS52vGUgY/b8mvWizww F+7IJ2Q4ZIidjOv81RkIfSEITGq1WlsA7M0SqL4jbvb83eMjOOtxZok+MidqEpOKhaCV mHerHNJaSgTinySsWpRcUNvwYEk6qTjf9AEJXuK9AXEBE9VTFehBgQ+D2uIt/y/LcFZv 8XaOyBb3BwD/QrOfKFVOxnn+CYJiU/MWY7XC2kCCfNTkR2Qr7B/5zPjJTqunmuJH8+BM tD/g== X-Gm-Message-State: APjAAAUp/poTYZKD6XUwFG2T9atEaIXmFHD8wgG0u4RuvcOPejxLpIhU uGVrZfFmr9djAKmuIUJm4kEHEijKomw= X-Google-Smtp-Source: APXvYqzdCq/8JWJo7I6dgNrFQli0gtIFsccTph2MG2IS/kBQfoPBV4rNembCgH+TWILl3RUeNtEXGWzHp7A= X-Received: by 2002:ac8:1997:: with SMTP id u23mr15120422qtj.11.1552180753103; Sat, 09 Mar 2019 17:19:13 -0800 (PST) Date: Sat, 9 Mar 2019 18:19:06 -0700 In-Reply-To: <20190310011906.254635-1-yuzhao@google.com> Message-Id: <20190310011906.254635-3-yuzhao@google.com> Mime-Version: 1.0 References: <20190218231319.178224-1-yuzhao@google.com> <20190310011906.254635-1-yuzhao@google.com> X-Mailer: git-send-email 2.21.0.360.g471c308f928-goog Subject: [PATCH v3 3/3] arm64: mm: enable per pmd page table lock From: Yu Zhao To: Catalin Marinas , Will Deacon , Mark Rutland X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190309_171915_008864_3E7B38E0 X-CRM114-Status: GOOD ( 15.10 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Yu Zhao , Ard Biesheuvel , Peter Zijlstra , linux-kernel@vger.kernel.org, Nick Piggin , Jun Yao , linux-mm@kvack.org, "Aneesh Kumar K . V" , Chintan Pandya , Joel Fernandes , "Kirill A . Shutemov" , Andrew Morton , Laura Abbott , linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org Switch from per mm_struct to per pmd page table lock by enabling ARCH_ENABLE_SPLIT_PMD_PTLOCK. This provides better granularity for large system. I'm not sure if there is contention on mm->page_table_lock. Given the option comes at no cost (apart from initializing more spin locks), why not enable it now. We only do so when pmd is not folded, so we don't mistakenly call pgtable_pmd_page_ctor() on pud or p4d in pgd_pgtable_alloc(). (We check shift against PMD_SHIFT, which is same as PUD_SHIFT when pmd is folded). Signed-off-by: Yu Zhao --- arch/arm64/Kconfig | 3 +++ arch/arm64/include/asm/pgalloc.h | 12 +++++++++++- arch/arm64/include/asm/tlb.h | 5 ++++- 3 files changed, 18 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index cfbf307d6dc4..a3b1b789f766 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -872,6 +872,9 @@ config ARCH_WANT_HUGE_PMD_SHARE config ARCH_HAS_CACHE_LINE_SIZE def_bool y +config ARCH_ENABLE_SPLIT_PMD_PTLOCK + def_bool y if PGTABLE_LEVELS > 2 + config SECCOMP bool "Enable seccomp to safely compute untrusted bytecode" ---help--- diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 52fa47c73bf0..dabba4b2c61f 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -33,12 +33,22 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) { - return (pmd_t *)__get_free_page(PGALLOC_GFP); + struct page *page; + + page = alloc_page(PGALLOC_GFP); + if (!page) + return NULL; + if (!pgtable_pmd_page_ctor(page)) { + __free_page(page); + return NULL; + } + return page_address(page); } static inline void pmd_free(struct mm_struct *mm, pmd_t *pmdp) { BUG_ON((unsigned long)pmdp & (PAGE_SIZE-1)); + pgtable_pmd_page_dtor(virt_to_page(pmdp)); free_page((unsigned long)pmdp); } diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 106fdc951b6e..4e3becfed387 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -62,7 +62,10 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) { - tlb_remove_table(tlb, virt_to_page(pmdp)); + struct page *page = virt_to_page(pmdp); + + pgtable_pmd_page_dtor(page); + tlb_remove_table(tlb, page); } #endif -- 2.21.0.360.g471c308f928-goog _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel