From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9FC5C636CC for ; Sat, 11 Feb 2023 03:40:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 716D06B0075; Fri, 10 Feb 2023 22:40:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6C6AD6B0078; Fri, 10 Feb 2023 22:40:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 58E646B007B; Fri, 10 Feb 2023 22:40:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 48A806B0075 for ; Fri, 10 Feb 2023 22:40:02 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 0F9321A12A0 for ; Sat, 11 Feb 2023 03:40:02 +0000 (UTC) X-FDA: 80453607444.08.8A238E7 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf08.hostedemail.com (Postfix) with ESMTP id 2CAEF16000D for ; Sat, 11 Feb 2023 03:39:59 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="v/XKKIln"; spf=none (imf08.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676086800; a=rsa-sha256; cv=none; b=QVT3gJz086Df79mVLHragXmqihS2/zXCG0wJSX2p1mrdikpRJ1Wak43SXXEA0bi9FcK4PA S61vlfhDAGsRuRAR538nlEYVnphdBoXEdo0XOBV+rs6/7iBLAMwyAw3Nv7eh25M7YSTNvl BfYFPxFRvvxkcBuQNKE5xzoqH/a2a1w= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="v/XKKIln"; spf=none (imf08.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676086800; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=i952xWUVOOOc/GL6UVWoLXB53jR4WQOcBwrrQqvGx28=; b=b6qcmfhIEUQfyDlqVPn68YReinVOeGM9ThfoHXdusL8B8F/M2nE1YEEyeLr/Peyvc3HC7p mN2t91B1gN3nmZ2rWwh4xlY7NKxhx66TMYnfBxh79Fk2Tvy25Bj4LwNLK9781RV/NsgOON f5sEut8jYzAzBPT2B/MbiM8GvzGQQT4= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=i952xWUVOOOc/GL6UVWoLXB53jR4WQOcBwrrQqvGx28=; b=v/XKKIlnabnQGORcG2wgH9ln+J Y/CLdqdbpm8aMCkCYi4WighWAkihKQVTeO/YSVvBG+acDXYcZ8ZQTZEvh/yevqOMHGDbJND5ERYH4 E6fhpNnELquRd1tlrRwHNg0VoLqXq8/ru81p98rLiOVTlb3RyH2ppv9A4L7FB8SRU66NtlXX95fn7 hsKUcx3zqfzaAXS0SbRy4S9sh6RebtYE9fzSk4VpfWF6ujE6fg+J4QsjeA7u0H+sJRqeAvpQsOSCp YXc07TgZZKPt4OHMQZ81R0QAqUPWIs5MFYf0ix5PLHaX9bS93LEP7KKAEpvmqF7i0QEKnj1lfoPH/ PpQ5aFTg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pQgjv-003k2q-H9; Sat, 11 Feb 2023 03:39:51 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org, linux-arch@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 6/7] arc: Implement the new page table range API Date: Sat, 11 Feb 2023 03:39:47 +0000 Message-Id: <20230211033948.891959-7-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230211033948.891959-1-willy@infradead.org> References: <20230211033948.891959-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Queue-Id: 2CAEF16000D X-Rspamd-Server: rspam01 X-Stat-Signature: hux9yya89bx8e561g8ix8f1ogdha8mex X-HE-Tag: 1676086799-105698 X-HE-Meta: U2FsdGVkX19uFFjT8oV89IkM7XQYn8H1w+ASGPF7hVosY+k2gDTZCY89SvUPnxIdp0ggiW5I0vAkzSN4WPocQF3t7i7XY0uCalU9kePX8YLlOAgV91nfAxmHivuLBv2Nk0J9Ru49GlejIRJfEtwHpD5bVMdasUWAyJMzfdXN0UxpNl35JcULXtZSRr73qYzEqvdjtiBvtmGW8kum5Fj2uY0EF8+9D8wSfmG9l2Fp4tUtq0sIMXweP2WhMyJNFfNK8jPAq/dd3V4hgksdVA5h3Gi5oyvN5uTD9ICLggfk31c9oli9ECi4bQL3Ydt7gFDn1PRoHTDf2f1ayi6mdvRbBvhZvtFa7pHmrOsDqM77DRO/7qS7zBLjT34vlSqA1b96lF22bx0WGvlp0hr413/yY3pCutvRX0HUtD98onpnRkMCmNe9n6hITYlE3uuXhMdSPM77f/C+XBLUm3N59e5HPZhZ/hs+/VXAXJW8gGI+wPtmGmQz8H1EBWXXZzkfckkOS+tFRRSngfcCzoRW4VF8SseU+11I2VDA4xhEyHJa5Aj4D83t+r7GFY6kGjWrYHrW85Z2lwT+dHrUP9nnWNCfaQ9YkEFcNBIRu/27iT2ja1+Kn/LJ3BkMvWj9PMCsXO+fKEUGh9GOFeWH4Hnzp18qFQ8btdY6m7j/BETKCSRdx18hyuCXlyE09ymm8ZYc73W6ITXMFoC8NB8pNaIGU5lUfEmoAODoyn3BYoK00wFAVOcrPczq5aIl9Tu484j3Ni/Oi5rE/33yAvHBoQzon8UHx7JnfFCXwjkB9h4khVsRSgN7xyAuteqaftoliO7dIprlo0TkskYixhqRhETF0ecigz79TeRh27AmVbsR6V6yL9bKRY3eWnYddgUBBjNdcJHDfZz89OqnLxTCq4ZIzExB1BOVwhjRt9kzq6IWResJgxK4yJOI2rxPYlsrwtMtjEsnpMPJD5XlQCdB/VHKrHD rKr1dSrw O0Zgx2Nvncoblunib2Hk7UbFHKXtDhO/VYLOoA56RKwWw8C++p1ML2gUbWSfwzn1O6MhP+GGahS2Y72wWvKrFvlABQtbumSDVlgDIzBQV3ppjca9JXGr1Zra9mP8jbq8nejRLuMuKpsuPhFQgY/nv4ZhQv2SZcF0Bv0Een9HyBQItQ8YlQ7xO2Io8Vv9nQ2bMEtt4WPoM7jnjphU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add set_ptes(), update_mmu_cache_range(), flush_dcache_folio() and flush_icache_pages(). This is a fairly deep change. The PG_dc_clean flag changes from being a per-page bit to being a per-folio bit (which means it cannot always be set as we don't know that all pages in this folio were cleaned). The internal flush routines are enhanced to take the number of pages to flush. Signed-off-by: Matthew Wilcox (Oracle) --- arch/arc/include/asm/cacheflush.h | 7 ++- arch/arc/include/asm/pgtable-bits-arcv2.h | 20 ++++++-- arch/arc/mm/cache.c | 61 ++++++++++++++--------- arch/arc/mm/tlb.c | 18 ++++--- 4 files changed, 68 insertions(+), 38 deletions(-) diff --git a/arch/arc/include/asm/cacheflush.h b/arch/arc/include/asm/cacheflush.h index e201b4b1655a..04f65f588510 100644 --- a/arch/arc/include/asm/cacheflush.h +++ b/arch/arc/include/asm/cacheflush.h @@ -25,17 +25,20 @@ * in update_mmu_cache() */ #define flush_icache_page(vma, page) +#define flush_icache_pages(vma, page, nr) void flush_cache_all(void); void flush_icache_range(unsigned long kstart, unsigned long kend); void __sync_icache_dcache(phys_addr_t paddr, unsigned long vaddr, int len); -void __inv_icache_page(phys_addr_t paddr, unsigned long vaddr); -void __flush_dcache_page(phys_addr_t paddr, unsigned long vaddr); +void __inv_icache_pages(phys_addr_t paddr, unsigned long vaddr, unsigned nr); +void __flush_dcache_pages(phys_addr_t paddr, unsigned long vaddr, unsigned nr); #define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 void flush_dcache_page(struct page *page); +void flush_dcache_folio(struct folio *folio); +#define flush_dcache_folio flush_dcache_folio void dma_cache_wback_inv(phys_addr_t start, unsigned long sz); void dma_cache_inv(phys_addr_t start, unsigned long sz); diff --git a/arch/arc/include/asm/pgtable-bits-arcv2.h b/arch/arc/include/asm/pgtable-bits-arcv2.h index 6e9f8ca6d6a1..4a1b2ce204c6 100644 --- a/arch/arc/include/asm/pgtable-bits-arcv2.h +++ b/arch/arc/include/asm/pgtable-bits-arcv2.h @@ -100,14 +100,24 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) return __pte((pte_val(pte) & _PAGE_CHG_MASK) | pgprot_val(newprot)); } -static inline void set_pte_at(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pteval) +static inline void set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr) { - set_pte(ptep, pteval); + for (;;) { + set_pte(ptep, pte); + if (--nr == 0) + break; + ptep++; + pte_val(pte) += PAGE_SIZE; + } } +#define set_pte_at(mm, addr, ptep, pte) set_ptes(mm, addr, ptep, pte, 1) -void update_mmu_cache(struct vm_area_struct *vma, unsigned long address, - pte_t *ptep); +void update_mmu_cache_range(struct vm_area_struct *vma, unsigned long address, + pte_t *ptep, unsigned int nr); + +#define update_mmu_cache(vma, addr, ptep) \ + update_mmu_cache_range(vma, addr, ptep, 1) /* * Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that diff --git a/arch/arc/mm/cache.c b/arch/arc/mm/cache.c index 55c6de138eae..3c16ee942a5c 100644 --- a/arch/arc/mm/cache.c +++ b/arch/arc/mm/cache.c @@ -752,17 +752,17 @@ static inline void arc_slc_enable(void) * There's a corollary case, where kernel READs from a userspace mapped page. * If the U-mapping is not congruent to K-mapping, former needs flushing. */ -void flush_dcache_page(struct page *page) +void flush_dcache_folio(struct folio *folio) { struct address_space *mapping; if (!cache_is_vipt_aliasing()) { - clear_bit(PG_dc_clean, &page->flags); + clear_bit(PG_dc_clean, &folio->flags); return; } /* don't handle anon pages here */ - mapping = page_mapping_file(page); + mapping = folio_flush_mapping(folio); if (!mapping) return; @@ -771,17 +771,27 @@ void flush_dcache_page(struct page *page) * Make a note that K-mapping is dirty */ if (!mapping_mapped(mapping)) { - clear_bit(PG_dc_clean, &page->flags); - } else if (page_mapcount(page)) { - + clear_bit(PG_dc_clean, &folio->flags); + } else if (folio_mapped(folio)) { /* kernel reading from page with U-mapping */ - phys_addr_t paddr = (unsigned long)page_address(page); - unsigned long vaddr = page->index << PAGE_SHIFT; + phys_addr_t paddr = (unsigned long)folio_address(folio); + unsigned long vaddr = folio_pos(folio); + /* + * vaddr is not actually the virtual address, but is + * congruent to every user mapping. + */ if (addr_not_cache_congruent(paddr, vaddr)) - __flush_dcache_page(paddr, vaddr); + __flush_dcache_pages(paddr, vaddr, + folio_nr_pages(folio)); } } +EXPORT_SYMBOL(flush_dcache_folio); + +void flush_dcache_page(struct page *page) +{ + return flush_dcache_folio(page_folio(page)); +} EXPORT_SYMBOL(flush_dcache_page); /* @@ -921,18 +931,18 @@ void __sync_icache_dcache(phys_addr_t paddr, unsigned long vaddr, int len) } /* wrapper to compile time eliminate alignment checks in flush loop */ -void __inv_icache_page(phys_addr_t paddr, unsigned long vaddr) +void __inv_icache_pages(phys_addr_t paddr, unsigned long vaddr, unsigned nr) { - __ic_line_inv_vaddr(paddr, vaddr, PAGE_SIZE); + __ic_line_inv_vaddr(paddr, vaddr, nr * PAGE_SIZE); } /* * wrapper to clearout kernel or userspace mappings of a page * For kernel mappings @vaddr == @paddr */ -void __flush_dcache_page(phys_addr_t paddr, unsigned long vaddr) +void __flush_dcache_pages(phys_addr_t paddr, unsigned long vaddr, unsigned nr) { - __dc_line_op(paddr, vaddr & PAGE_MASK, PAGE_SIZE, OP_FLUSH_N_INV); + __dc_line_op(paddr, vaddr & PAGE_MASK, nr * PAGE_SIZE, OP_FLUSH_N_INV); } noinline void flush_cache_all(void) @@ -962,10 +972,10 @@ void flush_cache_page(struct vm_area_struct *vma, unsigned long u_vaddr, u_vaddr &= PAGE_MASK; - __flush_dcache_page(paddr, u_vaddr); + __flush_dcache_pages(paddr, u_vaddr, 1); if (vma->vm_flags & VM_EXEC) - __inv_icache_page(paddr, u_vaddr); + __inv_icache_pages(paddr, u_vaddr, 1); } void flush_cache_range(struct vm_area_struct *vma, unsigned long start, @@ -978,9 +988,9 @@ void flush_anon_page(struct vm_area_struct *vma, struct page *page, unsigned long u_vaddr) { /* TBD: do we really need to clear the kernel mapping */ - __flush_dcache_page((phys_addr_t)page_address(page), u_vaddr); - __flush_dcache_page((phys_addr_t)page_address(page), - (phys_addr_t)page_address(page)); + __flush_dcache_pages((phys_addr_t)page_address(page), u_vaddr, 1); + __flush_dcache_pages((phys_addr_t)page_address(page), + (phys_addr_t)page_address(page), 1); } @@ -989,6 +999,8 @@ void flush_anon_page(struct vm_area_struct *vma, struct page *page, void copy_user_highpage(struct page *to, struct page *from, unsigned long u_vaddr, struct vm_area_struct *vma) { + struct folio *src = page_folio(from); + struct folio *dst = page_folio(to); void *kfrom = kmap_atomic(from); void *kto = kmap_atomic(to); int clean_src_k_mappings = 0; @@ -1005,7 +1017,7 @@ void copy_user_highpage(struct page *to, struct page *from, * addr_not_cache_congruent() is 0 */ if (page_mapcount(from) && addr_not_cache_congruent(kfrom, u_vaddr)) { - __flush_dcache_page((unsigned long)kfrom, u_vaddr); + __flush_dcache_pages((unsigned long)kfrom, u_vaddr, 1); clean_src_k_mappings = 1; } @@ -1019,17 +1031,17 @@ void copy_user_highpage(struct page *to, struct page *from, * non copied user pages (e.g. read faults which wire in pagecache page * directly). */ - clear_bit(PG_dc_clean, &to->flags); + clear_bit(PG_dc_clean, &dst->flags); /* * if SRC was already usermapped and non-congruent to kernel mapping * sync the kernel mapping back to physical page */ if (clean_src_k_mappings) { - __flush_dcache_page((unsigned long)kfrom, (unsigned long)kfrom); - set_bit(PG_dc_clean, &from->flags); + __flush_dcache_pages((unsigned long)kfrom, + (unsigned long)kfrom, 1); } else { - clear_bit(PG_dc_clean, &from->flags); + clear_bit(PG_dc_clean, &src->flags); } kunmap_atomic(kto); @@ -1038,8 +1050,9 @@ void copy_user_highpage(struct page *to, struct page *from, void clear_user_page(void *to, unsigned long u_vaddr, struct page *page) { + struct folio *folio = page_folio(page); clear_page(to); - clear_bit(PG_dc_clean, &page->flags); + clear_bit(PG_dc_clean, &folio->flags); } EXPORT_SYMBOL(clear_user_page); diff --git a/arch/arc/mm/tlb.c b/arch/arc/mm/tlb.c index 5f71445f26bd..0a996b65bb4e 100644 --- a/arch/arc/mm/tlb.c +++ b/arch/arc/mm/tlb.c @@ -467,8 +467,8 @@ void create_tlb(struct vm_area_struct *vma, unsigned long vaddr, pte_t *ptep) * Note that flush (when done) involves both WBACK - so physical page is * in sync as well as INV - so any non-congruent aliases don't remain */ -void update_mmu_cache(struct vm_area_struct *vma, unsigned long vaddr_unaligned, - pte_t *ptep) +void update_mmu_cache_range(struct vm_area_struct *vma, + unsigned long vaddr_unaligned, pte_t *ptep, unsigned int nr) { unsigned long vaddr = vaddr_unaligned & PAGE_MASK; phys_addr_t paddr = pte_val(*ptep) & PAGE_MASK_PHYS; @@ -491,15 +491,19 @@ void update_mmu_cache(struct vm_area_struct *vma, unsigned long vaddr_unaligned, */ if ((vma->vm_flags & VM_EXEC) || addr_not_cache_congruent(paddr, vaddr)) { - - int dirty = !test_and_set_bit(PG_dc_clean, &page->flags); + struct folio *folio = page_folio(page); + int dirty = !test_and_set_bit(PG_dc_clean, &folio->flags); if (dirty) { + unsigned long offset = offset_in_folio(folio, paddr); + nr = folio_nr_pages(folio); + paddr -= offset; + vaddr -= offset; /* wback + inv dcache lines (K-mapping) */ - __flush_dcache_page(paddr, paddr); + __flush_dcache_pages(paddr, paddr, nr); /* invalidate any existing icache lines (U-mapping) */ if (vma->vm_flags & VM_EXEC) - __inv_icache_page(paddr, vaddr); + __inv_icache_pages(paddr, vaddr, nr); } } } @@ -531,7 +535,7 @@ void update_mmu_cache_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd) { pte_t pte = __pte(pmd_val(*pmd)); - update_mmu_cache(vma, addr, &pte); + update_mmu_cache_range(vma, addr, &pte, HPAGE_PMD_NR); } void local_flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, -- 2.39.1