From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752609AbdFMKRE (ORCPT ); Tue, 13 Jun 2017 06:17:04 -0400 Received: from terminus.zytor.com ([65.50.211.136]:52707 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751996AbdFMKRC (ORCPT ); Tue, 13 Jun 2017 06:17:02 -0400 Date: Tue, 13 Jun 2017 03:10:08 -0700 From: "tip-bot for Kirill A. Shutemov" Message-ID: Cc: luto@amacapital.net, hpa@zytor.com, linux-kernel@vger.kernel.org, dave.hansen@intel.com, peterz@infradead.org, luto@kernel.org, bp@alien8.de, mingo@kernel.org, tglx@linutronix.de, kirill.shutemov@linux.intel.com, jpoimboe@redhat.com, akpm@linux-foundation.org, dvlasenk@redhat.com, brgerst@gmail.com, torvalds@linux-foundation.org Reply-To: akpm@linux-foundation.org, dvlasenk@redhat.com, jpoimboe@redhat.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, mingo@kernel.org, torvalds@linux-foundation.org, brgerst@gmail.com, bp@alien8.de, luto@kernel.org, hpa@zytor.com, luto@amacapital.net, dave.hansen@intel.com, peterz@infradead.org, linux-kernel@vger.kernel.org In-Reply-To: <20170606113133.22974-12-kirill.shutemov@linux.intel.com> References: <20170606113133.22974-12-kirill.shutemov@linux.intel.com> To: linux-tip-commits@vger.kernel.org Subject: [tip:x86/mm] x86/mm: Make kernel_physical_mapping_init() support 5-level paging Git-Commit-ID: 7e82ea946ae4d056859b19fcdec66425878395eb X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Commit-ID: 7e82ea946ae4d056859b19fcdec66425878395eb Gitweb: http://git.kernel.org/tip/7e82ea946ae4d056859b19fcdec66425878395eb Author: Kirill A. Shutemov AuthorDate: Tue, 6 Jun 2017 14:31:30 +0300 Committer: Ingo Molnar CommitDate: Tue, 13 Jun 2017 08:56:57 +0200 x86/mm: Make kernel_physical_mapping_init() support 5-level paging Populate additional page table level if CONFIG_X86_5LEVEL is enabled. Signed-off-by: Kirill A. Shutemov Cc: Andrew Morton Cc: Andy Lutomirski Cc: Andy Lutomirski Cc: Borislav Petkov Cc: Brian Gerst Cc: Dave Hansen Cc: Denys Vlasenko Cc: H. Peter Anvin Cc: Josh Poimboeuf Cc: Linus Torvalds Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: linux-arch@vger.kernel.org Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/20170606113133.22974-12-kirill.shutemov@linux.intel.com Signed-off-by: Ingo Molnar --- arch/x86/mm/init_64.c | 69 ++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 60 insertions(+), 9 deletions(-) diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 7a9497a..b863d14 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -624,6 +624,57 @@ phys_pud_init(pud_t *pud_page, unsigned long paddr, unsigned long paddr_end, return paddr_last; } +static unsigned long __meminit +phys_p4d_init(p4d_t *p4d_page, unsigned long paddr, unsigned long paddr_end, + unsigned long page_size_mask) +{ + unsigned long paddr_next, paddr_last = paddr_end; + unsigned long vaddr = (unsigned long)__va(paddr); + int i = p4d_index(vaddr); + + if (!IS_ENABLED(CONFIG_X86_5LEVEL)) + return phys_pud_init((pud_t *) p4d_page, paddr, paddr_end, page_size_mask); + + for (; i < PTRS_PER_P4D; i++, paddr = paddr_next) { + p4d_t *p4d; + pud_t *pud; + + vaddr = (unsigned long)__va(paddr); + p4d = p4d_page + p4d_index(vaddr); + paddr_next = (paddr & P4D_MASK) + P4D_SIZE; + + if (paddr >= paddr_end) { + if (!after_bootmem && + !e820__mapped_any(paddr & P4D_MASK, paddr_next, + E820_TYPE_RAM) && + !e820__mapped_any(paddr & P4D_MASK, paddr_next, + E820_TYPE_RESERVED_KERN)) + set_p4d(p4d, __p4d(0)); + continue; + } + + if (!p4d_none(*p4d)) { + pud = pud_offset(p4d, 0); + paddr_last = phys_pud_init(pud, paddr, + paddr_end, + page_size_mask); + __flush_tlb_all(); + continue; + } + + pud = alloc_low_page(); + paddr_last = phys_pud_init(pud, paddr, paddr_end, + page_size_mask); + + spin_lock(&init_mm.page_table_lock); + p4d_populate(&init_mm, p4d, pud); + spin_unlock(&init_mm.page_table_lock); + } + __flush_tlb_all(); + + return paddr_last; +} + /* * Create page table mapping for the physical memory for specific physical * addresses. The virtual and physical addresses have to be aligned on PMD level @@ -645,26 +696,26 @@ kernel_physical_mapping_init(unsigned long paddr_start, for (; vaddr < vaddr_end; vaddr = vaddr_next) { pgd_t *pgd = pgd_offset_k(vaddr); p4d_t *p4d; - pud_t *pud; vaddr_next = (vaddr & PGDIR_MASK) + PGDIR_SIZE; - BUILD_BUG_ON(pgd_none(*pgd)); - p4d = p4d_offset(pgd, vaddr); - if (p4d_val(*p4d)) { - pud = (pud_t *)p4d_page_vaddr(*p4d); - paddr_last = phys_pud_init(pud, __pa(vaddr), + if (pgd_val(*pgd)) { + p4d = (p4d_t *)pgd_page_vaddr(*pgd); + paddr_last = phys_p4d_init(p4d, __pa(vaddr), __pa(vaddr_end), page_size_mask); continue; } - pud = alloc_low_page(); - paddr_last = phys_pud_init(pud, __pa(vaddr), __pa(vaddr_end), + p4d = alloc_low_page(); + paddr_last = phys_p4d_init(p4d, __pa(vaddr), __pa(vaddr_end), page_size_mask); spin_lock(&init_mm.page_table_lock); - p4d_populate(&init_mm, p4d, pud); + if (IS_ENABLED(CONFIG_X86_5LEVEL)) + pgd_populate(&init_mm, pgd, p4d); + else + p4d_populate(&init_mm, p4d_offset(pgd, vaddr), (pud_t *) p4d); spin_unlock(&init_mm.page_table_lock); pgd_changed = true; }