Cache aliasing issues using 4K pages.

Cache aliasing issues using 4K pages.

[The Engineer]

We are working with a 2.6.12 kernel on a dual-core mips architecture.
In this dual-core system, one core is running the linux kernel and the
other is used for some real-time handling (not directly controlled by
Linux)
We had different stability issues, which could be pinpointed to be
related with cache aliasing problems.
Cache aliasing happens when the same physical memory can be cached
twice as it is accessed by two different virtual addresses.
Indeed, for the index to select the correct cache line the virtual
address is used. If some bits of the virtual page address are used in
the cache index, aliasing can occur.


As there is no hardware solution in the mips to recover from this
(which would provide some cache coherency, even for one core), the
only intrinsic safe solution is to enlarge the page size, so that
cache indexing is only done by the offset address in the page (thus
the physical part of the address).
Another solution is to flush the cache if a page is being remapped to
an aliased address (but in our case linux does not has control on the
second core, which can cause issues with shared data between both
cores).
Currently the second solution is used in the kernel, but we found
different issues with it (for instance: we had to merge more recent
mips kernels, to get a reliable copy-on-write behaviour after
forks...).

Therefore some questions:
- Are there still some known issues with cache aliasing in the MIPS kernel?
- Are there known issues when using 16KB pages (8KB pages seems not be
possible due to tlb issues).

Thanks in advance,
Luc

Re: [SPAM] Cache aliasing issues using 4K pages.

[Markus Gothe]

[-- Attachment #1.1: Type: text/plain, Size: 2253 bytes --]

Lemme guess, it's not a dual-core as in SMP but a sub-CPU.
This usually involves that the memory differs, for example does it  
have a TLB. Which vendor and which core is it? I suppose you use some  
sort of firmware from the vendor to access the sub-CPU, right?

//Markus

On 14 Jan 2008, at 21:25, The Engineer wrote:

> We are working with a 2.6.12 kernel on a dual-core mips architecture.
> In this dual-core system, one core is running the linux kernel and the
> other is used for some real-time handling (not directly controlled by
> Linux)
> We had different stability issues, which could be pinpointed to be
> related with cache aliasing problems.
> Cache aliasing happens when the same physical memory can be cached
> twice as it is accessed by two different virtual addresses.
> Indeed, for the index to select the correct cache line the virtual
> address is used. If some bits of the virtual page address are used in
> the cache index, aliasing can occur.
>
>
> As there is no hardware solution in the mips to recover from this
> (which would provide some cache coherency, even for one core), the
> only intrinsic safe solution is to enlarge the page size, so that
> cache indexing is only done by the offset address in the page (thus
> the physical part of the address).
> Another solution is to flush the cache if a page is being remapped to
> an aliased address (but in our case linux does not has control on the
> second core, which can cause issues with shared data between both
> cores).
> Currently the second solution is used in the kernel, but we found
> different issues with it (for instance: we had to merge more recent
> mips kernels, to get a reliable copy-on-write behaviour after
> forks...).
>
> Therefore some questions:
> - Are there still some known issues with cache aliasing in the MIPS  
> kernel?
> - Are there known issues when using 16KB pages (8KB pages seems not be
> possible due to tlb issues).
>
> Thanks in advance,
> Luc
>

_______________________________________

Mr Markus Gothe
Software Engineer

Phone: +46 (0)13 21 81 20 (ext. 1046)
Fax: +46 (0)13 21 21 15
Mobile: +46 (0)70 348 44 35
Diskettgatan 11, SE-583 35 Linköping, Sweden
www.27m.com




[-- Attachment #1.2: Type: text/html, Size: 4929 bytes --]

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: Cache aliasing issues using 4K pages.

[Thiemo Seufer]

The Engineer wrote:
> We are working with a 2.6.12 kernel on a dual-core mips architecture.
> In this dual-core system, one core is running the linux kernel and the
> other is used for some real-time handling (not directly controlled by
> Linux)
> We had different stability issues, which could be pinpointed to be
> related with cache aliasing problems.
> Cache aliasing happens when the same physical memory can be cached
> twice as it is accessed by two different virtual addresses.
> Indeed, for the index to select the correct cache line the virtual
> address is used. If some bits of the virtual page address are used in
> the cache index, aliasing can occur.
> 
> 
> As there is no hardware solution in the mips to recover from this
> (which would provide some cache coherency, even for one core), the
> only intrinsic safe solution is to enlarge the page size, so that
> cache indexing is only done by the offset address in the page (thus
> the physical part of the address).
> Another solution is to flush the cache if a page is being remapped to
> an aliased address (but in our case linux does not has control on the
> second core, which can cause issues with shared data between both
> cores).
> Currently the second solution is used in the kernel, but we found
> different issues with it (for instance: we had to merge more recent
> mips kernels, to get a reliable copy-on-write behaviour after
> forks...).
> 
> Therefore some questions:
> - Are there still some known issues with cache aliasing in the MIPS kernel?
> - Are there known issues when using 16KB pages (8KB pages seems not be
> possible due to tlb issues).

With recent kernels and toolchains 16k pages work ok IME. With a 2.6.12
kernel however you'll have to backport a serious amount of bugfixes
before 16k pages can work. Upgrading the kernel is probably less work.


Thiemo

Re: [SPAM] Cache aliasing issues using 4K pages.

[Luc Perneel]

[-- Attachment #1: Type: text/plain, Size: 2738 bytes --]

It is a real SMP system (you could run linux on it using both cores just
like on a dual core x86 desktop).
Never the less, only one of the cores is running linux. the other runs the
real-time part.
We use a kernel module and shared memory to communicate from the linux
applications to the real-time part on the second core.
Thus the HW is SMP, the software is not...
Luc

On Jan 14, 2008 11:34 PM, Markus Gothe <markus.gothe@27m.se> wrote:

> Lemme guess, it's not a dual-core as in SMP but a sub-CPU.This usually
> involves that the memory differs, for example does it have a TLB. Which
> vendor and which core is it? I suppose you use some sort of firmware from
> the vendor to access the sub-CPU, right?
>
> //Markus
>
> On 14 Jan 2008, at 21:25, The Engineer wrote:
>
> We are working with a 2.6.12 kernel on a dual-core mips architecture.
> In this dual-core system, one core is running the linux kernel and the
> other is used for some real-time handling (not directly controlled by
> Linux)
> We had different stability issues, which could be pinpointed to be
> related with cache aliasing problems.
> Cache aliasing happens when the same physical memory can be cached
> twice as it is accessed by two different virtual addresses.
> Indeed, for the index to select the correct cache line the virtual
> address is used. If some bits of the virtual page address are used in
> the cache index, aliasing can occur.
>
>
> As there is no hardware solution in the mips to recover from this
> (which would provide some cache coherency, even for one core), the
> only intrinsic safe solution is to enlarge the page size, so that
> cache indexing is only done by the offset address in the page (thus
> the physical part of the address).
> Another solution is to flush the cache if a page is being remapped to
> an aliased address (but in our case linux does not has control on the
> second core, which can cause issues with shared data between both
> cores).
> Currently the second solution is used in the kernel, but we found
> different issues with it (for instance: we had to merge more recent
> mips kernels, to get a reliable copy-on-write behaviour after
> forks...).
>
> Therefore some questions:
> - Are there still some known issues with cache aliasing in the MIPS
> kernel?
> - Are there known issues when using 16KB pages (8KB pages seems not be
> possible due to tlb issues).
>
> Thanks in advance,
> Luc
>
>
> _______________________________________
>
> Mr Markus Gothe
> Software Engineer
>
> Phone: +46 (0)13 21 81 20 (ext. 1046)
> Fax: +46 (0)13 21 21 15
> Mobile: +46 (0)70 348 44 35
> Diskettgatan 11, SE-583 35 Linköping, Sweden
> www.27m.com
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 4232 bytes --]

Re: Cache aliasing issues using 4K pages.

[Luc Perneel]

[-- Attachment #1: Type: text/plain, Size: 2062 bytes --]

On Jan 15, 2008 2:58 AM, Thiemo Seufer <ths@networkno.de> wrote:

> The Engineer wrote:
> > We are working with a 2.6.12 kernel on a dual-core mips architecture.
> > In this dual-core system, one core is running the linux kernel and the
> > other is used for some real-time handling (not directly controlled by
> > Linux)
> > We had different stability issues, which could be pinpointed to be
> > related with cache aliasing problems.
> > Cache aliasing happens when the same physical memory can be cached
> > twice as it is accessed by two different virtual addresses.
> > Indeed, for the index to select the correct cache line the virtual
> > address is used. If some bits of the virtual page address are used in
> > the cache index, aliasing can occur.
> >
> >
> > As there is no hardware solution in the mips to recover from this
> > (which would provide some cache coherency, even for one core), the
> > only intrinsic safe solution is to enlarge the page size, so that
> > cache indexing is only done by the offset address in the page (thus
> > the physical part of the address).
> > Another solution is to flush the cache if a page is being remapped to
> > an aliased address (but in our case linux does not has control on the
> > second core, which can cause issues with shared data between both
> > cores).
> > Currently the second solution is used in the kernel, but we found
> > different issues with it (for instance: we had to merge more recent
> > mips kernels, to get a reliable copy-on-write behaviour after
> > forks...).
> >
> > Therefore some questions:
> > - Are there still some known issues with cache aliasing in the MIPS
> kernel?
> > - Are there known issues when using 16KB pages (8KB pages seems not be
> > possible due to tlb issues).
>
> With recent kernels and toolchains 16k pages work ok IME. With a 2.6.12
> kernel however you'll have to backport a serious amount of bugfixes
> before 16k pages can work. Upgrading the kernel is probably less work.
>
> Thiemo
>
Thanks for the info, any idea from which kernel this works ok?
Luc

[-- Attachment #2: Type: text/html, Size: 2509 bytes --]

Re: Cache aliasing issues using 4K pages.

[Thiemo Seufer]

Luc Perneel wrote:
> On Jan 15, 2008 2:58 AM, Thiemo Seufer <ths@networkno.de> wrote:
> 
> > The Engineer wrote:
> > > We are working with a 2.6.12 kernel on a dual-core mips architecture.
> > > In this dual-core system, one core is running the linux kernel and the
> > > other is used for some real-time handling (not directly controlled by
> > > Linux)
> > > We had different stability issues, which could be pinpointed to be
> > > related with cache aliasing problems.
> > > Cache aliasing happens when the same physical memory can be cached
> > > twice as it is accessed by two different virtual addresses.
> > > Indeed, for the index to select the correct cache line the virtual
> > > address is used. If some bits of the virtual page address are used in
> > > the cache index, aliasing can occur.
> > >
> > >
> > > As there is no hardware solution in the mips to recover from this
> > > (which would provide some cache coherency, even for one core), the
> > > only intrinsic safe solution is to enlarge the page size, so that
> > > cache indexing is only done by the offset address in the page (thus
> > > the physical part of the address).
> > > Another solution is to flush the cache if a page is being remapped to
> > > an aliased address (but in our case linux does not has control on the
> > > second core, which can cause issues with shared data between both
> > > cores).
> > > Currently the second solution is used in the kernel, but we found
> > > different issues with it (for instance: we had to merge more recent
> > > mips kernels, to get a reliable copy-on-write behaviour after
> > > forks...).
> > >
> > > Therefore some questions:
> > > - Are there still some known issues with cache aliasing in the MIPS
> > kernel?
> > > - Are there known issues when using 16KB pages (8KB pages seems not be
> > > possible due to tlb issues).
> >
> > With recent kernels and toolchains 16k pages work ok IME. With a 2.6.12
> > kernel however you'll have to backport a serious amount of bugfixes
> > before 16k pages can work. Upgrading the kernel is probably less work.
> >
> > Thiemo
> >
> Thanks for the info, any idea from which kernel this works ok?

I believe the linux-2.6.22-stable branch from www.linux-mips.org is the
best version to use. I don't recall if the earlier branches got all the
necessary fixes.


Thiemo