[PATCH 0/2] riscv: Enable percpu page first chunk allocator

[PATCH 0/2] riscv: Enable percpu page first chunk allocator

[Alexandre Ghiti]

While working with pcpu variables, I noticed that riscv did not support
first chunk allocation in the vmalloc area which may be needed as a fallback
in case of a sparse NUMA configuration.

patch 1 starts by introducing a new function flush_cache_vmap_early() which
is needed since a new vmalloc mapping is established and directly accessed:
on riscv, this would likely fail in case of a reordered access or if the
uarch caches invalid entries in TLB.

patch 2 simply enables the page percpu first chunk allocator in riscv.

Alexandre Ghiti (2):
  mm: Introduce flush_cache_vmap_early() and its riscv implementation
  riscv: Enable pcpu page first chunk allocator

 arch/riscv/Kconfig                  | 2 ++
 arch/riscv/include/asm/cacheflush.h | 3 ++-
 arch/riscv/include/asm/tlbflush.h   | 2 ++
 arch/riscv/mm/kasan_init.c          | 8 ++++++++
 arch/riscv/mm/tlbflush.c            | 5 +++++
 include/asm-generic/cacheflush.h    | 6 ++++++
 mm/percpu.c                         | 8 +-------
 7 files changed, 26 insertions(+), 8 deletions(-)

-- 
2.39.2


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[PATCH 1/2] mm: Introduce flush_cache_vmap_early() and its riscv implementation

[Alexandre Ghiti]

The pcpu setup when using the page allocator sets up a new vmalloc
mapping very early in the boot process, so early that it cannot use the
flush_cache_vmap() function which may depend on structures not yet
initialized (for example in riscv, we currently send an IPI to flush
other cpus TLB).

But on some architectures, we must call flush_cache_vmap(): for example,
in riscv, some uarchs can cache invalid TLB entries so we need to flush
the new established mapping to avoid taking an exception.

So fix this by introducing a new function flush_cache_vmap_early() which
is called right after setting the new page table entry and before
accessing this new mapping. This new function implements a local flush
tlb on riscv and is no-op for other architectures (same as today).

Signed-off-by: Alexandre Ghiti <alexghiti@rivosinc.com>
---
 arch/riscv/include/asm/cacheflush.h | 3 ++-
 arch/riscv/include/asm/tlbflush.h   | 2 ++
 arch/riscv/mm/tlbflush.c            | 5 +++++
 include/asm-generic/cacheflush.h    | 6 ++++++
 mm/percpu.c                         | 8 +-------
 5 files changed, 16 insertions(+), 8 deletions(-)

diff --git a/arch/riscv/include/asm/cacheflush.h b/arch/riscv/include/asm/cacheflush.h
index 3cb53c4df27c..a129dac4521d 100644
--- a/arch/riscv/include/asm/cacheflush.h
+++ b/arch/riscv/include/asm/cacheflush.h
@@ -37,7 +37,8 @@ static inline void flush_dcache_page(struct page *page)
 	flush_icache_mm(vma->vm_mm, 0)
 
 #ifdef CONFIG_64BIT
-#define flush_cache_vmap(start, end)	flush_tlb_kernel_range(start, end)
+#define flush_cache_vmap(start, end)		flush_tlb_kernel_range(start, end)
+#define flush_cache_vmap_early(start, end)	local_flush_tlb_kernel_range(start, end)
 #endif
 
 #ifndef CONFIG_SMP
diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h
index 8f3418c5f172..f0d6328076b6 100644
--- a/arch/riscv/include/asm/tlbflush.h
+++ b/arch/riscv/include/asm/tlbflush.h
@@ -41,6 +41,7 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr);
 void flush_tlb_range(struct vm_area_struct *vma, unsigned long start,
 		     unsigned long end);
 void flush_tlb_kernel_range(unsigned long start, unsigned long end);
+void local_flush_tlb_kernel_range(unsigned long start, unsigned long end);
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 #define __HAVE_ARCH_FLUSH_PMD_TLB_RANGE
 void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start,
@@ -64,6 +65,7 @@ static inline void flush_tlb_kernel_range(unsigned long start,
 	local_flush_tlb_all();
 }
 
+#define local_flush_tlb_kernel_range(start, end) flush_tlb_kernel_range(start, end)
 #define flush_tlb_mm(mm) flush_tlb_all()
 #define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all()
 #endif /* !CONFIG_SMP || !CONFIG_MMU */
diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c
index e6659d7368b3..8aadc5f71c93 100644
--- a/arch/riscv/mm/tlbflush.c
+++ b/arch/riscv/mm/tlbflush.c
@@ -66,6 +66,11 @@ static inline void local_flush_tlb_range_asid(unsigned long start,
 		local_flush_tlb_range_threshold_asid(start, size, stride, asid);
 }
 
+void local_flush_tlb_kernel_range(unsigned long start, unsigned long end)
+{
+	local_flush_tlb_range_asid(start, end, PAGE_SIZE, FLUSH_TLB_NO_ASID);
+}
+
 static void __ipi_flush_tlb_all(void *info)
 {
 	local_flush_tlb_all();
diff --git a/include/asm-generic/cacheflush.h b/include/asm-generic/cacheflush.h
index 84ec53ccc450..7ee8a179d103 100644
--- a/include/asm-generic/cacheflush.h
+++ b/include/asm-generic/cacheflush.h
@@ -91,6 +91,12 @@ static inline void flush_cache_vmap(unsigned long start, unsigned long end)
 }
 #endif
 
+#ifndef flush_cache_vmap_early
+static inline void flush_cache_vmap_early(unsigned long start, unsigned long end)
+{
+}
+#endif
+
 #ifndef flush_cache_vunmap
 static inline void flush_cache_vunmap(unsigned long start, unsigned long end)
 {
diff --git a/mm/percpu.c b/mm/percpu.c
index a7665de8485f..d287cebd58ca 100644
--- a/mm/percpu.c
+++ b/mm/percpu.c
@@ -3306,13 +3306,7 @@ int __init pcpu_page_first_chunk(size_t reserved_size, pcpu_fc_cpu_to_node_fn_t
 		if (rc < 0)
 			panic("failed to map percpu area, err=%d\n", rc);
 
-		/*
-		 * FIXME: Archs with virtual cache should flush local
-		 * cache for the linear mapping here - something
-		 * equivalent to flush_cache_vmap() on the local cpu.
-		 * flush_cache_vmap() can't be used as most supporting
-		 * data structures are not set up yet.
-		 */
+		flush_cache_vmap_early(unit_addr, unit_addr + ai->unit_size);
 
 		/* copy static data */
 		memcpy((void *)unit_addr, __per_cpu_load, ai->static_size);
-- 
2.39.2


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[PATCH 2/2] riscv: Enable pcpu page first chunk allocator

[Alexandre Ghiti]

As explained in commit 6ea529a2037c ("percpu: make embedding first chunk
allocator check vmalloc space size"), the embedding first chunk allocator
needs the vmalloc space to be larger than the maximum distance between
units which are grouped into NUMA nodes.

On a very sparse NUMA configurations and a small vmalloc area (for example,
it is 64GB in sv39), the allocation of dynamic percpu data in the vmalloc
area could fail.

So provide the pcpu page allocator as a fallback in case we fall into
such a sparse configuration (which happened in arm64 as shown by
commit 09cea6195073 ("arm64: support page mapping percpu first chunk
allocator")).

Signed-off-by: Alexandre Ghiti <alexghiti@rivosinc.com>
---
 arch/riscv/Kconfig         | 2 ++
 arch/riscv/mm/kasan_init.c | 8 ++++++++
 2 files changed, 10 insertions(+)

diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index 5b1e61aca6cf..7b82d8301e42 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -416,7 +416,9 @@ config NUMA
 	depends on SMP && MMU
 	select ARCH_SUPPORTS_NUMA_BALANCING
 	select GENERIC_ARCH_NUMA
+	select HAVE_SETUP_PER_CPU_AREA
 	select NEED_PER_CPU_EMBED_FIRST_CHUNK
+	select NEED_PER_CPU_PAGE_FIRST_CHUNK
 	select OF_NUMA
 	select USE_PERCPU_NUMA_NODE_ID
 	help
diff --git a/arch/riscv/mm/kasan_init.c b/arch/riscv/mm/kasan_init.c
index 5e39dcf23fdb..4c9a2c527f08 100644
--- a/arch/riscv/mm/kasan_init.c
+++ b/arch/riscv/mm/kasan_init.c
@@ -438,6 +438,14 @@ static void __init kasan_shallow_populate(void *start, void *end)
 	kasan_shallow_populate_pgd(vaddr, vend);
 }
 
+#ifdef CONFIG_KASAN_VMALLOC
+void __init kasan_populate_early_vm_area_shadow(void *start, unsigned long size)
+{
+	kasan_populate(kasan_mem_to_shadow(start),
+		       kasan_mem_to_shadow(start + size));
+}
+#endif
+
 static void __init create_tmp_mapping(void)
 {
 	void *ptr;
-- 
2.39.2


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH 0/2] riscv: Enable percpu page first chunk allocator

[Alexandre Ghiti]

Hi Tejun,

On 10/11/2023 15:07, Alexandre Ghiti wrote:
> While working with pcpu variables, I noticed that riscv did not support
> first chunk allocation in the vmalloc area which may be needed as a fallback
> in case of a sparse NUMA configuration.
>
> patch 1 starts by introducing a new function flush_cache_vmap_early() which
> is needed since a new vmalloc mapping is established and directly accessed:
> on riscv, this would likely fail in case of a reordered access or if the
> uarch caches invalid entries in TLB.
>
> patch 2 simply enables the page percpu first chunk allocator in riscv.
>
> Alexandre Ghiti (2):
>    mm: Introduce flush_cache_vmap_early() and its riscv implementation
>    riscv: Enable pcpu page first chunk allocator
>
>   arch/riscv/Kconfig                  | 2 ++
>   arch/riscv/include/asm/cacheflush.h | 3 ++-
>   arch/riscv/include/asm/tlbflush.h   | 2 ++
>   arch/riscv/mm/kasan_init.c          | 8 ++++++++
>   arch/riscv/mm/tlbflush.c            | 5 +++++
>   include/asm-generic/cacheflush.h    | 6 ++++++
>   mm/percpu.c                         | 8 +-------
>   7 files changed, 26 insertions(+), 8 deletions(-)
>

Any feedback regarding this?

Thanks,

Alex


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH 0/2] riscv: Enable percpu page first chunk allocator

[Tejun Heo]

On Wed, Dec 06, 2023 at 11:08:20AM +0100, Alexandre Ghiti wrote:
> Hi Tejun,
> 
> On 10/11/2023 15:07, Alexandre Ghiti wrote:
> > While working with pcpu variables, I noticed that riscv did not support
> > first chunk allocation in the vmalloc area which may be needed as a fallback
> > in case of a sparse NUMA configuration.
> > 
> > patch 1 starts by introducing a new function flush_cache_vmap_early() which
> > is needed since a new vmalloc mapping is established and directly accessed:
> > on riscv, this would likely fail in case of a reordered access or if the
> > uarch caches invalid entries in TLB.
> > 
> > patch 2 simply enables the page percpu first chunk allocator in riscv.
> > 
> > Alexandre Ghiti (2):
> >    mm: Introduce flush_cache_vmap_early() and its riscv implementation
> >    riscv: Enable pcpu page first chunk allocator
> > 
> >   arch/riscv/Kconfig                  | 2 ++
> >   arch/riscv/include/asm/cacheflush.h | 3 ++-
> >   arch/riscv/include/asm/tlbflush.h   | 2 ++
> >   arch/riscv/mm/kasan_init.c          | 8 ++++++++
> >   arch/riscv/mm/tlbflush.c            | 5 +++++
> >   include/asm-generic/cacheflush.h    | 6 ++++++
> >   mm/percpu.c                         | 8 +-------
> >   7 files changed, 26 insertions(+), 8 deletions(-)
> > 
> 
> Any feedback regarding this?

On cursory look, it looked fine to me but Dennis is maintaining the percpu
tree now. Dennis?

Thanks.

-- 
tejun

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH 0/2] riscv: Enable percpu page first chunk allocator

[Dennis Zhou]

Hello,

On Wed, Dec 06, 2023 at 09:00:27AM -1000, Tejun Heo wrote:
> On Wed, Dec 06, 2023 at 11:08:20AM +0100, Alexandre Ghiti wrote:
> > Hi Tejun,
> > 
> > On 10/11/2023 15:07, Alexandre Ghiti wrote:
> > > While working with pcpu variables, I noticed that riscv did not support
> > > first chunk allocation in the vmalloc area which may be needed as a fallback
> > > in case of a sparse NUMA configuration.
> > > 
> > > patch 1 starts by introducing a new function flush_cache_vmap_early() which
> > > is needed since a new vmalloc mapping is established and directly accessed:
> > > on riscv, this would likely fail in case of a reordered access or if the
> > > uarch caches invalid entries in TLB.
> > > 
> > > patch 2 simply enables the page percpu first chunk allocator in riscv.
> > > 
> > > Alexandre Ghiti (2):
> > >    mm: Introduce flush_cache_vmap_early() and its riscv implementation
> > >    riscv: Enable pcpu page first chunk allocator
> > > 
> > >   arch/riscv/Kconfig                  | 2 ++
> > >   arch/riscv/include/asm/cacheflush.h | 3 ++-
> > >   arch/riscv/include/asm/tlbflush.h   | 2 ++
> > >   arch/riscv/mm/kasan_init.c          | 8 ++++++++
> > >   arch/riscv/mm/tlbflush.c            | 5 +++++
> > >   include/asm-generic/cacheflush.h    | 6 ++++++
> > >   mm/percpu.c                         | 8 +-------
> > >   7 files changed, 26 insertions(+), 8 deletions(-)
> > > 
> > 
> > Any feedback regarding this?
> 
> On cursory look, it looked fine to me but Dennis is maintaining the percpu
> tree now. Dennis?
> 

Ah I wasn't sure at the time if we needed this to go through percpu vs
risc v. I need to poke tglx and potentially pull some more stuff so I
can take it.

I regrettably got both the covid and flu vaccines today and feel like a
truck hit me. I'll review this tomorrow and make sure it's taken care
of for the next merge window.

Thanks,
Dennis

> Thanks.
> 
> -- 
> tejun

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH 0/2] riscv: Enable percpu page first chunk allocator

[Dennis Zhou]

Hello,

On Fri, Nov 10, 2023 at 03:07:19PM +0100, Alexandre Ghiti wrote:
> While working with pcpu variables, I noticed that riscv did not support
> first chunk allocation in the vmalloc area which may be needed as a fallback
> in case of a sparse NUMA configuration.
> 
> patch 1 starts by introducing a new function flush_cache_vmap_early() which
> is needed since a new vmalloc mapping is established and directly accessed:
> on riscv, this would likely fail in case of a reordered access or if the
> uarch caches invalid entries in TLB.
> 
> patch 2 simply enables the page percpu first chunk allocator in riscv.
> 
> Alexandre Ghiti (2):
>   mm: Introduce flush_cache_vmap_early() and its riscv implementation
>   riscv: Enable pcpu page first chunk allocator
> 
>  arch/riscv/Kconfig                  | 2 ++
>  arch/riscv/include/asm/cacheflush.h | 3 ++-
>  arch/riscv/include/asm/tlbflush.h   | 2 ++
>  arch/riscv/mm/kasan_init.c          | 8 ++++++++
>  arch/riscv/mm/tlbflush.c            | 5 +++++
>  include/asm-generic/cacheflush.h    | 6 ++++++
>  mm/percpu.c                         | 8 +-------
>  7 files changed, 26 insertions(+), 8 deletions(-)
> 
> -- 
> 2.39.2
> 

I've applied this to percpu#for-6.8.

Thanks,
Dennis

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv