nocache/highmem mapping

nocache/highmem mapping

[Konrad Eisele]

Hi,
We encountered something strange for out Sparc-Leon arch,
and would like to ask weather someone can clarify:
When running on a board with 512 MB we encountered
difficulties in highmem (256MB works ok). We could only
solve this problem by redefining the strange SRMMU_MAX_NOCACHE_PAGES
to a multiple of PMD_SIZE (0x400000)
-#define SRMMU_MAX_NOCACHE_PAGES	(1280)
+#define SRMMU_MAX_NOCACHE_PAGES	(1024)
So we would like to ask why is this the case only for Leon-Sparc,
Obviously other sun machines work well (maybe I'm wrong...)

Here is what we digged out:
The virtual address regions with 512 MB ram and
#define SRMMU_MAX_NOCACHE_PAGES	(1280)
are:

0xfc0000000 - 0xfc5000000 : nocache mem
0xfc5000000 - 0xfc9000000 : highmem

however srmmu_nocache_init()->srmmu_early_allocate_ptable_skeleton()
initializes the the page skelleton in steps of:
                 ...
	while(start < end) {
                 ...
		start = (start + PMD_SIZE) & PMD_MASK;
         }
                 ...

PMD_SIZE is 0x4000000, that means it will will initilize

0xfc0000000 - 0xfc4000000 and
0xfc4000000 - 0xfc8000000
before it breaks.

But this leaves highmem's front region
0xfc5000000 - 0xfc8000000 allocated with a
skelleton (!).
The effect is that  srmmu_paging_init()'s:
         ...
         pgd = pgd_offset_k(PKMAP_BASE);
	pmd = srmmu_pmd_offset(pgd, PKMAP_BASE);
	pte = srmmu_pte_offset(pmd, PKMAP_BASE);
	pkmap_page_table = pte;
         ...

pkmap_page_table will end up with the pte-ptd-set that nocache
has allocated, highmem's  mm/highmem.c:map_new_virtual()
               ...
               set_pte_at(&init_mm, vaddr,
		   &(pkmap_page_table[last_pkmap_nr]), mk_pte(page, kmap_prot));
               ...
will fail starting from v-address 0xfc800000.

The root of evil is  vaddrs.h's strange:
...
#define SRMMU_MAX_NOCACHE_PAGES	(1280)
...

We had to change it to
#define SRMMU_MAX_NOCACHE_PAGES	(1024) (PMD_SIZE size multiple (0x400000))

I'm not shure why this is the case only on our machines.

Maybe I am missing something.
-- Konrad

Re: nocache/highmem mapping

[David Miller]

From: Konrad Eisele <konrad@gaisler.com>
Date: Tue, 06 Oct 2009 17:35:24 +0200

> Here is what we digged out:
> The virtual address regions with 512 MB ram and
> #define SRMMU_MAX_NOCACHE_PAGES	(1280)
> are:
> 
> 0xfc0000000 - 0xfc5000000 : nocache mem
> 0xfc5000000 - 0xfc9000000 : highmem
> 
> however srmmu_nocache_init()->srmmu_early_allocate_ptable_skeleton()
> initializes the the page skelleton in steps of:
>                 ...
> 	while(start < end) {
>                 ...
> 		start = (start + PMD_SIZE) & PMD_MASK;
>         }
>                 ...
> 
> PMD_SIZE is 0x4000000, that means it will will initilize
> 
> 0xfc0000000 - 0xfc4000000 and
> 0xfc4000000 - 0xfc8000000
> before it breaks.

I think you have found a genuine bug and maybe it explains
bugs others have run into over the past few years :-)

I'll do some more research on this tomorrow and apply some
kind of fix.

Thanks!

Re: nocache/highmem mapping

[David Miller]

From: Konrad Eisele <konrad@gaisler.com>
Date: Tue, 06 Oct 2009 17:35:24 +0200

I looked over this in detail and I can't see what the problem
is.

> Here is what we digged out:
> The virtual address regions with 512 MB ram and
> #define SRMMU_MAX_NOCACHE_PAGES	(1280)
> are:
> 
> 0xfc0000000 - 0xfc5000000 : nocache mem
> 0xfc5000000 - 0xfc9000000 : highmem
> 
> however srmmu_nocache_init()->srmmu_early_allocate_ptable_skeleton()
> initializes the the page skelleton in steps of:
>                 ...
> 	while(start < end) {
>                 ...
> 		start = (start + PMD_SIZE) & PMD_MASK;
>         }
>                 ...
> 
> PMD_SIZE is 0x4000000, that means it will will initilize
> 
> 0xfc0000000 - 0xfc4000000 and
> 0xfc4000000 - 0xfc8000000
> before it breaks.
> 
> But this leaves highmem's front region
> 0xfc5000000 - 0xfc8000000 allocated with a
> skelleton (!).

That's perfectly fine, these page tables can be shared by
multiple region users.  And in fact you calculate of the
highmem front region is flawed, it's not 0xfc500000, in
fact it's a constant and not at all related to how much of
a nocache region we decide to use.

> The effect is that  srmmu_paging_init()'s:
>         ...
>         pgd = pgd_offset_k(PKMAP_BASE);
> 	pmd = srmmu_pmd_offset(pgd, PKMAP_BASE);
> 	pte = srmmu_pte_offset(pmd, PKMAP_BASE);
> 	pkmap_page_table = pte;
>         ...
> 
> pkmap_page_table will end up with the pte-ptd-set that nocache
> has allocated, highmem's  mm/highmem.c:map_new_virtual()
>               ...
>               set_pte_at(&init_mm, vaddr,
> 		   &(pkmap_page_table[last_pkmap_nr]), mk_pte(page,
> 		   kmap_prot));
>               ...
> will fail starting from v-address 0xfc800000.

The only real requirement is that PKMAP_BASE is aligned to a single
PTE chunk, so that pkmap_page_table highmem thing works.

Since I see no real reason to lower the NOCACHE area size, it's better
to just properly align-up the PKMAP_BASE.

And wait.... that's exactly what PKMAP_BASE is defined to do:

#define PKMAP_SIZE (LAST_PKMAP << PAGE_SHIFT)
#define PKMAP_BASE PMD_ALIGN(SRMMU_NOCACHE_VADDR + (SRMMU_MAX_NOCACHE_PAGES << PAGE_SHIFT))

PKMAP_BASE is defined to be PMD_ALIGN'd _exactly_ because of the
requirements created by pkmap_page_table and the highmem code.

PKMAP_BASE should evaluate always to:

	PMD_ALIGN(KERNBASE + SRMMU_MAXMEM + (1280 << PAGE_SHIFT))
	|
	PMD_ALIGN(0xf0000000 + 0x0c000000 + (1280 << PAGE_SHIFT))
	|
	PMD_ALIGN(0xfc000000 + (1280 << PAGE_SHIFT))
	|
	PMD_ALIGN(0xfc000000 + 0x500000)
	|
	PMD_ALIGN(0xfc500000)

So what's PMD_ALIGN?

#define PMD_SHIFT		SUN4C_PMD_SHIFT
#define PMD_SIZE        	(1UL << PMD_SHIFT)
#define PMD_MASK        	(~(PMD_SIZE-1))
#define PMD_ALIGN(__addr) 	(((__addr) + ~PMD_MASK) & PMD_MASK)

This value here is the "abstract" PMD size we use in the sparc32 port,
which we happen to just use the sun4c PMD size.

And since that is larger than the SRMMU's PMD size, this should
still align PKMAP_BASE sufficiently.

#define PMD_SHIFT		22
#define PMD_SIZE        	(1UL << 22)
#define PMD_MASK        	(~(PMD_SIZE-1))
#define PMD_ALIGN(__addr) 	(((__addr) + ~PMD_MASK) & PMD_MASK)

Thus:

	PMD_ALIGN(0xfc500000)
	|
	0xfc800000

So PKMAP_BASE is 0xfc800000, and your bug must be something else.

Re: nocache/highmem mapping

[David Miller]

From: Konrad Eisele <konrad@gaisler.com>
Date: Mon, 02 Nov 2009 13:33:05 +0100

> David Miller wrote:
>> From: Konrad Eisele <konrad@gaisler.com>
>> Date: Tue, 06 Oct 2009 17:35:24 +0200
>> I looked over this in detail and I can't see what the problem
>> is.
> 
> You are right. Now that you say that PKMAP_BASE should be
> 0xfc800000 instead of 0xfc500000 I went over my local version
> of the code and saw what went wrong on my machine:
> A earalier patch was aligning PKMAP_BASE to an odd value,
> this was done because I was patching my local version of the
> sourcecode to support a hardware-srmmu-configuration with
> pagesize > 4k. Thanks that you pointed it out. Sorry for
> the confusion.

No problem, I'm just glad this is resolved.

Re: nocache/highmem mapping

[Konrad Eisele]

David Miller wrote:
> From: Konrad Eisele <konrad@gaisler.com>
> Date: Tue, 06 Oct 2009 17:35:24 +0200
> 
> I looked over this in detail and I can't see what the problem
> is.

You are right. Now that you say that PKMAP_BASE should be
0xfc800000 instead of 0xfc500000 I went over my local version
of the code and saw what went wrong on my machine:
A earalier patch was aligning PKMAP_BASE to an odd value,
this was done because I was patching my local version of the
sourcecode to support a hardware-srmmu-configuration with
pagesize > 4k. Thanks that you pointed it out. Sorry for
the confusion.
-- Konrad

> 
>> Here is what we digged out:
>> The virtual address regions with 512 MB ram and
>> #define SRMMU_MAX_NOCACHE_PAGES	(1280)
>> are:
>>
>> 0xfc0000000 - 0xfc5000000 : nocache mem
>> 0xfc5000000 - 0xfc9000000 : highmem
>>
>> however srmmu_nocache_init()->srmmu_early_allocate_ptable_skeleton()
>> initializes the the page skelleton in steps of:
>>                 ...
>> 	while(start < end) {
>>                 ...
>> 		start = (start + PMD_SIZE) & PMD_MASK;
>>         }
>>                 ...
>>
>> PMD_SIZE is 0x4000000, that means it will will initilize
>>
>> 0xfc0000000 - 0xfc4000000 and
>> 0xfc4000000 - 0xfc8000000
>> before it breaks.
>>
>> But this leaves highmem's front region
>> 0xfc5000000 - 0xfc8000000 allocated with a
>> skelleton (!).
> 
> That's perfectly fine, these page tables can be shared by
> multiple region users.  And in fact you calculate of the
> highmem front region is flawed, it's not 0xfc500000, in
> fact it's a constant and not at all related to how much of
> a nocache region we decide to use.
> 
>> The effect is that  srmmu_paging_init()'s:
>>         ...
>>         pgd = pgd_offset_k(PKMAP_BASE);
>> 	pmd = srmmu_pmd_offset(pgd, PKMAP_BASE);
>> 	pte = srmmu_pte_offset(pmd, PKMAP_BASE);
>> 	pkmap_page_table = pte;
>>         ...
>>
>> pkmap_page_table will end up with the pte-ptd-set that nocache
>> has allocated, highmem's  mm/highmem.c:map_new_virtual()
>>               ...
>>               set_pte_at(&init_mm, vaddr,
>> 		   &(pkmap_page_table[last_pkmap_nr]), mk_pte(page,
>> 		   kmap_prot));
>>               ...
>> will fail starting from v-address 0xfc800000.
> 
> The only real requirement is that PKMAP_BASE is aligned to a single
> PTE chunk, so that pkmap_page_table highmem thing works.
> 
> Since I see no real reason to lower the NOCACHE area size, it's better
> to just properly align-up the PKMAP_BASE.
> 
> And wait.... that's exactly what PKMAP_BASE is defined to do:
> 
> #define PKMAP_SIZE (LAST_PKMAP << PAGE_SHIFT)
> #define PKMAP_BASE PMD_ALIGN(SRMMU_NOCACHE_VADDR + (SRMMU_MAX_NOCACHE_PAGES << PAGE_SHIFT))
> 
> PKMAP_BASE is defined to be PMD_ALIGN'd _exactly_ because of the
> requirements created by pkmap_page_table and the highmem code.
> 
> PKMAP_BASE should evaluate always to:
> 
> 	PMD_ALIGN(KERNBASE + SRMMU_MAXMEM + (1280 << PAGE_SHIFT))
> 	|
> 	PMD_ALIGN(0xf0000000 + 0x0c000000 + (1280 << PAGE_SHIFT))
> 	|
> 	PMD_ALIGN(0xfc000000 + (1280 << PAGE_SHIFT))
> 	|
> 	PMD_ALIGN(0xfc000000 + 0x500000)
> 	|
> 	PMD_ALIGN(0xfc500000)
> 
> So what's PMD_ALIGN?
> 
> #define PMD_SHIFT		SUN4C_PMD_SHIFT
> #define PMD_SIZE        	(1UL << PMD_SHIFT)
> #define PMD_MASK        	(~(PMD_SIZE-1))
> #define PMD_ALIGN(__addr) 	(((__addr) + ~PMD_MASK) & PMD_MASK)
> 
> This value here is the "abstract" PMD size we use in the sparc32 port,
> which we happen to just use the sun4c PMD size.
> 
> And since that is larger than the SRMMU's PMD size, this should
> still align PKMAP_BASE sufficiently.
> 
> #define PMD_SHIFT		22
> #define PMD_SIZE        	(1UL << 22)
> #define PMD_MASK        	(~(PMD_SIZE-1))
> #define PMD_ALIGN(__addr) 	(((__addr) + ~PMD_MASK) & PMD_MASK)
> 
> Thus:
> 
> 	PMD_ALIGN(0xfc500000)
> 	|
> 	0xfc800000
> 
> So PKMAP_BASE is 0xfc800000, and your bug must be something else.
> 
>