[ARM]Fixmap mapping region is not enough for system of 14+ CPUs.

[ARM]Fixmap mapping region is not enough for system of 14+ CPUs.

[Nicolas Pitre]

On Wed, 12 Mar 2014, Liu hua wrote:

> Hi Russell, Will or Nicolas,
> 
> (In this mail, we only discuss ARM 32-bit linux.)
> 
> As we know, the region (0xfff00000-0xfffdffff) is reserved as fixmap
> mapping region.
> 
> The function "kmap_atomic" maps highmem pages to this region referring
> to CPUID and per-cpu variable "__kmap_atomic_idx" via
> 
>   idx = type + KM_TYPE_NR * smp_processor_id();
>   vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
> 
> Size of region used by one cpu is 0x10000 (KM_TYPE_NR << PAGE_SHIFT).
> And the total size of the fixmap mapping region is 0xe0000.
> (only support 14 CPUs).
> 
> So in a system of more than 14 CPUs, this region is not large enough.
> should we change the memory layout on ARM Linux to support 14+ cpu system ?
> Or can we do anything else to support that ?

How many CPUs do you have?

What about the following patch?  If this doesn't work for you then more 
intrusive changes will be needed.

diff --git a/arch/arm/include/asm/fixmap.h b/arch/arm/include/asm/fixmap.h
index 68ea615c2a..254f2df08d 100644
--- a/arch/arm/include/asm/fixmap.h
+++ b/arch/arm/include/asm/fixmap.h
@@ -14,7 +14,17 @@
  */
 
 #define FIXADDR_START		0xfff00000UL
+
+#if !defined(CONFIG_HAVE_TCM) && !defined(CONFIG_CPU_XSCALE)
+/*
+ * If no TCM nor on on a XScale then enlarge the fixmap area to
+ * accommodate up to 30 CPUs.
+ */
+#define FIXADDR_END		0xffff0000UL
+#else
 #define FIXADDR_END		0xfffe0000UL
+#endif
+
 #define FIXADDR_TOP		(FIXADDR_END - PAGE_SIZE)
 
 enum fixed_addresses {


Nicolas

[ARM]Fixmap mapping region is not enough for system of 14+ CPUs.

[Liu hua]

On 2014/3/13 1:08, Nicolas Pitre wrote:
> On Wed, 12 Mar 2014, Liu hua wrote:
> 
>> Hi Russell, Will or Nicolas,
>>
>> (In this mail, we only discuss ARM 32-bit linux.)
>>
>> As we know, the region (0xfff00000-0xfffdffff) is reserved as fixmap
>> mapping region.
>>
>> The function "kmap_atomic" maps highmem pages to this region referring
>> to CPUID and per-cpu variable "__kmap_atomic_idx" via
>>
>>   idx = type + KM_TYPE_NR * smp_processor_id();
>>   vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
>>
>> Size of region used by one cpu is 0x10000 (KM_TYPE_NR << PAGE_SHIFT).
>> And the total size of the fixmap mapping region is 0xe0000.
>> (only support 14 CPUs).
>>
>> So in a system of more than 14 CPUs, this region is not large enough.
>> should we change the memory layout on ARM Linux to support 14+ cpu system ?
>> Or can we do anything else to support that ?
> 
> How many CPUs do you have?
> 
> What about the following patch?  If this doesn't work for you then more 
> intrusive changes will be needed.
> 
> diff --git a/arch/arm/include/asm/fixmap.h b/arch/arm/include/asm/fixmap.h
> index 68ea615c2a..254f2df08d 100644
> --- a/arch/arm/include/asm/fixmap.h
> +++ b/arch/arm/include/asm/fixmap.h
> @@ -14,7 +14,17 @@
>   */
>  
>  #define FIXADDR_START		0xfff00000UL
> +
> +#if !defined(CONFIG_HAVE_TCM) && !defined(CONFIG_CPU_XSCALE)
> +/*
> + * If no TCM nor on on a XScale then enlarge the fixmap area to
> + * accommodate up to 30 CPUs.
> + */
> +#define FIXADDR_END		0xffff0000UL
> +#else
>  #define FIXADDR_END		0xfffe0000UL
One cpu need 0x10000-size region; So this patch can only support
uo to 15 CPUs
> +#endif
> +
>  #define FIXADDR_TOP		(FIXADDR_END - PAGE_SIZE)
>  
>  enum fixed_addresses {
> 
> 
> Nicolas
> 

We have 16 CPUs in our system. And your patch cannot work if
the CPU number exceeds 15.

In addition, we can configure NR_CPUS up to 32. So should we
solve this problem completely?

There is an 1M-size hole between DMA mapping region and fixmap
mapping region(0xffe00000-0xfff00000).if this region are belonged
to fixmap mapping region, it can only support 30 CPUs.

Since we have alloced a second level page table to coverring
0xffe00000 - 0xffffffff(2M). And 0xffff0000 upwards used as
vector pages. So if the left region is all used as fixmap mapping
region(size : 0x1f0000). Up to 31 CPUs is supported.


The following patch can solve our problem(up to 30 CPUs), But it
reduce the DMA region. What do you think about it ?
Is is necessory to solve this problem completely(up to 32 CPUs)
?


Liu Hua

-----------------
diff --git a/arch/arm/include/asm/fixmap.h b/arch/arm/include/asm/fixmap.h
index 68ea615..8379891 100644
--- a/arch/arm/include/asm/fixmap.h
+++ b/arch/arm/include/asm/fixmap.h
@@ -13,7 +13,7 @@
  * 0xfffe0000 and 0xfffeffff.
  */

-#define FIXADDR_START              0xfff00000UL
+#define FIXADDR_START                0xffe00000UL
 #define FIXADDR_END            0xfffe0000UL
 #define FIXADDR_TOP            (FIXADDR_END - PAGE_SIZE)

diff --git a/arch/arm/include/asm/memory.h b/arch/arm/include/asm/memory.h
index 4afb376..0c40674 100644
--- a/arch/arm/include/asm/memory.h
+++ b/arch/arm/include/asm/memory.h
@@ -83,7 +83,7 @@
  */
 #define IOREMAP_MAX_ORDER      24

-#define CONSISTENT_END             (0xffe00000UL)
+#define CONSISTENT_END               (0xffd00000UL)

 #else /* CONFIG_MMU */

[ARM]Fixmap mapping region is not enough for system of 14+ CPUs.

[Nicolas Pitre]

On Thu, 13 Mar 2014, Liu hua wrote:

> On 2014/3/13 1:08, Nicolas Pitre wrote:
> > On Wed, 12 Mar 2014, Liu hua wrote:
> > 
> >> Hi Russell, Will or Nicolas,
> >>
> >> (In this mail, we only discuss ARM 32-bit linux.)
> >>
> >> As we know, the region (0xfff00000-0xfffdffff) is reserved as fixmap
> >> mapping region.
> >>
> >> The function "kmap_atomic" maps highmem pages to this region referring
> >> to CPUID and per-cpu variable "__kmap_atomic_idx" via
> >>
> >>   idx = type + KM_TYPE_NR * smp_processor_id();
> >>   vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
> >>
> >> Size of region used by one cpu is 0x10000 (KM_TYPE_NR << PAGE_SHIFT).
> >> And the total size of the fixmap mapping region is 0xe0000.
> >> (only support 14 CPUs).
> >>
> >> So in a system of more than 14 CPUs, this region is not large enough.
> >> should we change the memory layout on ARM Linux to support 14+ cpu system ?
> >> Or can we do anything else to support that ?
> > 
> > How many CPUs do you have?
> > 
> > What about the following patch?  If this doesn't work for you then more 
> > intrusive changes will be needed.
> > 
> > diff --git a/arch/arm/include/asm/fixmap.h b/arch/arm/include/asm/fixmap.h
> > index 68ea615c2a..254f2df08d 100644
> > --- a/arch/arm/include/asm/fixmap.h
> > +++ b/arch/arm/include/asm/fixmap.h
> > @@ -14,7 +14,17 @@
> >   */
> >  
> >  #define FIXADDR_START		0xfff00000UL
> > +
> > +#if !defined(CONFIG_HAVE_TCM) && !defined(CONFIG_CPU_XSCALE)
> > +/*
> > + * If no TCM nor on on a XScale then enlarge the fixmap area to
> > + * accommodate up to 30 CPUs.
> > + */
> > +#define FIXADDR_END		0xffff0000UL
> > +#else
> >  #define FIXADDR_END		0xfffe0000UL
> One cpu need 0x10000-size region; So this patch can only support
> uo to 15 CPUs
Hmmm... You're right of course.

> We have 16 CPUs in our system. And your patch cannot work if
> the CPU number exceeds 15.
> 
> In addition, we can configure NR_CPUS up to 32. So should we
> solve this problem completely?
> 
> There is an 1M-size hole between DMA mapping region and fixmap
> mapping region(0xffe00000-0xfff00000).if this region are belonged
> to fixmap mapping region, it can only support 30 CPUs.
> 
> Since we have alloced a second level page table to coverring
> 0xffe00000 - 0xffffffff(2M). And 0xffff0000 upwards used as
> vector pages. So if the left region is all used as fixmap mapping
> region(size : 0x1f0000). Up to 31 CPUs is supported.
> 
> 
> The following patch can solve our problem(up to 30 CPUs), But it
> reduce the DMA region. What do you think about it ?
> Is is necessory to solve this problem completely(up to 32 CPUs)
> ?
> 
> 
> Liu Hua
> 
> -----------------
> diff --git a/arch/arm/include/asm/fixmap.h b/arch/arm/include/asm/fixmap.h
> index 68ea615..8379891 100644
> --- a/arch/arm/include/asm/fixmap.h
> +++ b/arch/arm/include/asm/fixmap.h
> @@ -13,7 +13,7 @@
>   * 0xfffe0000 and 0xfffeffff.
>   */
> 
> -#define FIXADDR_START              0xfff00000UL
> +#define FIXADDR_START                0xffe00000UL
>  #define FIXADDR_END            0xfffe0000UL
>  #define FIXADDR_TOP            (FIXADDR_END - PAGE_SIZE)
> 
> diff --git a/arch/arm/include/asm/memory.h b/arch/arm/include/asm/memory.h
> index 4afb376..0c40674 100644
> --- a/arch/arm/include/asm/memory.h
> +++ b/arch/arm/include/asm/memory.h
> @@ -83,7 +83,7 @@
>   */
>  #define IOREMAP_MAX_ORDER      24
> 
> -#define CONSISTENT_END             (0xffe00000UL)
> +#define CONSISTENT_END               (0xffd00000UL)

Some systems might possibly have issues with a smaller DMA region. 
I don't recall which though.

you should also update Documentation/arm/memory.txt in your patch.
Then I'd suggest you can repost it more formally for comments.


Nicolas