Re: The impact to change page size to 16k for cache alias

Re: The impact to change page size to 16k for cache alias

[Peter Watkins]

Hi Colin,

When we went to 64K pages we had to get a firmware upgrade for a myrinet card. As I recall it was already OK for 16K pages. 

For mellanox cards we had to set a .ini parameter, log2_uar_bar_megabytes. Send me a note if you want more details.

Other than that, the drivers we tried just worked (qla24xxx, e1000, tg3, sata sil, sil24).

--Peter

The impact to change page size to 16k for cache alias

[林建安]

Hi all,
We are willing to use 16k page size to avoid cache alias problem.
The Linux version we use is 2.6.12. If we just upgrade mm system to support 
16k page size, what else problems will happen?
There is already one thing we know that applications of ELF format 
applications should be transformed to be 16k alignment.
Another one, we think, highly suspected to be problematic is that many 
drivers will be ok for 4k page size but fails for 16k.
That is because 4k page size had been seemed to be natural for a very long 
long time.
Any other problem that shall happen for 16k page size?

Regards,
Colin 

The impact to change page size to 16k for cache alias

[林建安]

Hi all,
We are willing to use 16k page size to avoid cache alias problem.
The Linux version we use is 2.6.12. If we just upgrade mm system to support 
16k page size, what else problems will happen?
There is already one thing we know that applications of ELF format 
applications should be transformed to be 16k alignment.
Another one, we think, highly suspected to be problematic is that many 
drivers will be ok for 4k page size but fails for 16k.
That is because 4k page size had been seemed to be natural for a very long 
long time.
Any other problem that shall happen for 16k page size?

Regards,
Colin 

Re: The impact to change page size to 16k for cache alias

[Zhang Le]

On 12:06 Mon 30 Mar     , 林建安 wrote:
>
> Hi all,
> We are willing to use 16k page size to avoid cache alias problem.
> The Linux version we use is 2.6.12. If we just upgrade mm system to 
> support 16k page size, what else problems will happen?
> There is already one thing we know that applications of ELF format  
> applications should be transformed to be 16k alignment.
> Another one, we think, highly suspected to be problematic is that many  
> drivers will be ok for 4k page size but fails for 16k.
> That is because 4k page size had been seemed to be natural for a very 
> long long time.
> Any other problem that shall happen for 16k page size?

Linux on Loongson 2E and 2F uses 16k page size to avoid cache alias problem, too.
However, I haven't encountered any problem on Linux kernel itself due to 16k page
size.

Anyway, I am not 100% familiar with Loongson patches, so I am not sure whether
the page size problem is already been taken care of in the patch. If you are
interested to find out yourself, you can get the whole source here:
http://repo.or.cz/w/linux-2.6/linux-loongson.git

Zhang, Le
http://zhangle.is-a-geek.org

Re: The impact to change page size to 16k for cache alias

[Ralf Baechle]

On Mon, Mar 30, 2009 at 04:24:15PM +0800, Zhang Le wrote:

> > Hi all,
> > We are willing to use 16k page size to avoid cache alias problem.
> > The Linux version we use is 2.6.12. If we just upgrade mm system to 
> > support 16k page size, what else problems will happen?
> > There is already one thing we know that applications of ELF format  
> > applications should be transformed to be 16k alignment.
> > Another one, we think, highly suspected to be problematic is that many  
> > drivers will be ok for 4k page size but fails for 16k.
> > That is because 4k page size had been seemed to be natural for a very 
> > long long time.
> > Any other problem that shall happen for 16k page size?
> 
> Linux on Loongson 2E and 2F uses 16k page size to avoid cache alias problem, too.
> However, I haven't encountered any problem on Linux kernel itself due to 16k page
> size.
> 
> Anyway, I am not 100% familiar with Loongson patches, so I am not sure whether
> the page size problem is already been taken care of in the patch. If you are
> interested to find out yourself, you can get the whole source here:
> http://repo.or.cz/w/linux-2.6/linux-loongson.git

I've got a report that Fulong is currently only working with 16k pages.  So
4k is no longer the bullet proof choice for all cases :)

  Ralf

Re: The impact to change page size to 16k for cache alias

[Zhang Le]

On 10:11 Tue 31 Mar     , Ralf Baechle wrote:
> On Mon, Mar 30, 2009 at 04:24:15PM +0800, Zhang Le wrote:
> 
> > > Hi all,
> > > We are willing to use 16k page size to avoid cache alias problem.
> > > The Linux version we use is 2.6.12. If we just upgrade mm system to 
> > > support 16k page size, what else problems will happen?
> > > There is already one thing we know that applications of ELF format  
> > > applications should be transformed to be 16k alignment.
> > > Another one, we think, highly suspected to be problematic is that many  
> > > drivers will be ok for 4k page size but fails for 16k.
> > > That is because 4k page size had been seemed to be natural for a very 
> > > long long time.
> > > Any other problem that shall happen for 16k page size?
> > 
> > Linux on Loongson 2E and 2F uses 16k page size to avoid cache alias problem, too.
> > However, I haven't encountered any problem on Linux kernel itself due to 16k page
> > size.
> > 
> > Anyway, I am not 100% familiar with Loongson patches, so I am not sure whether
> > the page size problem is already been taken care of in the patch. If you are
> > interested to find out yourself, you can get the whole source here:
> > http://repo.or.cz/w/linux-2.6/linux-loongson.git
> 
> I've got a report that Fulong is currently only working with 16k pages.  So
> 4k is no longer the bullet proof choice for all cases :)

Yes, at least from what I can tell.
I have tried 4k before, because I heard someone told me the aliasing problem
already can be taken care by software, namely Linux. But as it turned out, 16k
is still necessary for the Loongson boxes to function properly.

And btw, lemote intended to call the box fuloong, as can be seen here:
http://www.lemote.com/english/fuloong.html

Zhang, Le
http://zhangle.is-a-geek.org

Re: The impact to change page size to 16k for cache alias

[Ralf Baechle]

On Wed, Apr 01, 2009 at 12:54:12AM +0800, Zhang Le wrote:

> > > Linux on Loongson 2E and 2F uses 16k page size to avoid cache alias problem, too.
> > > However, I haven't encountered any problem on Linux kernel itself due to 16k page
> > > size.
> > > 
> > > Anyway, I am not 100% familiar with Loongson patches, so I am not sure whether
> > > the page size problem is already been taken care of in the patch. If you are
> > > interested to find out yourself, you can get the whole source here:
> > > http://repo.or.cz/w/linux-2.6/linux-loongson.git
> > 
> > I've got a report that Fulong is currently only working with 16k pages.  So
> > 4k is no longer the bullet proof choice for all cases :)
> 
> Yes, at least from what I can tell.
> I have tried 4k before, because I heard someone told me the aliasing problem
> already can be taken care by software, namely Linux. But as it turned out, 16k
> is still necessary for the Loongson boxes to function properly.

Linux knows how to handle aliasing caches - the vast majority of MIPS
systems have aliasing caches and are running at 4k page size.  So the
issue is Loongson-specific.  This might be due to processor detection
not detecting the cache size, cache line size, number of ways correctly
or because loongson's caches has some unusual cache properties which the
Linux kernel is not designed to handle yet.

Btw, I just noticed that there is no
arch/mips/include/asm/mach-lemote/cpu-feature-overrides.h which is going
to impact code size and performance.

  Ralf

Re: The impact to change page size to 16k for cache alias

[Zhang Le]

On 19:12 Tue 31 Mar     , Ralf Baechle wrote:
> On Wed, Apr 01, 2009 at 12:54:12AM +0800, Zhang Le wrote:
> 
> > > > Linux on Loongson 2E and 2F uses 16k page size to avoid cache alias problem, too.
> > > > However, I haven't encountered any problem on Linux kernel itself due to 16k page
> > > > size.
> > > > 
> > > > Anyway, I am not 100% familiar with Loongson patches, so I am not sure whether
> > > > the page size problem is already been taken care of in the patch. If you are
> > > > interested to find out yourself, you can get the whole source here:
> > > > http://repo.or.cz/w/linux-2.6/linux-loongson.git
> > > 
> > > I've got a report that Fulong is currently only working with 16k pages.  So
> > > 4k is no longer the bullet proof choice for all cases :)
> > 
> > Yes, at least from what I can tell.
> > I have tried 4k before, because I heard someone told me the aliasing problem
> > already can be taken care by software, namely Linux. But as it turned out, 16k
> > is still necessary for the Loongson boxes to function properly.
> 
> Linux knows how to handle aliasing caches - the vast majority of MIPS
> systems have aliasing caches and are running at 4k page size.  So the
> issue is Loongson-specific.  This might be due to processor detection
> not detecting the cache size, cache line size, number of ways correctly
> or because loongson's caches has some unusual cache properties which the
> Linux kernel is not designed to handle yet.
> 
> Btw, I just noticed that there is no
> arch/mips/include/asm/mach-lemote/cpu-feature-overrides.h which is going
> to impact code size and performance.

Thanks for the reminder. I didn't know it before. I am gonna take a look.

Zhang, Le
http://zhangle.is-a-geek.org

Re: The impact to change page size to 16k for cache alias

[Ralf Baechle]

On Mon, Mar 30, 2009 at 12:06:51PM +0800, 林建安 wrote:

> Hi all,
> We are willing to use 16k page size to avoid cache alias problem.
> The Linux version we use is 2.6.12. If we just upgrade mm system to 
> support 16k page size, what else problems will happen?
> There is already one thing we know that applications of ELF format  
> applications should be transformed to be 16k alignment.
> Another one, we think, highly suspected to be problematic is that many  
> drivers will be ok for 4k page size but fails for 16k.
> That is because 4k page size had been seemed to be natural for a very 
> long long time.
> Any other problem that shall happen for 16k page size?

Most drivers are agnostic of the page size so no problems due to increased
page size are to be expected.

You will have to upgrade your kernel though.  There were some important
fixes for larger page size applied to later kernels.

  Ralf