Anyone noticed that there are a lot of cache flushes after kunmap/kunmap_atomic is called?

Anyone noticed that there are a lot of cache flushes after kunmap/kunmap_atomic is called?

[David VomLehn]

On the MIPS processor, cache flushing is done based on virtual addresses. 
However, in the Linux kernel, there are a lot of places where memory is mapped 
with kmap or kmap_atomic, then unmapped with the corresponding kunmap or 
kunmap_atomic and only *then* is the cache flushed. In other words, we only flush 
the cache after we have dropped the mapping of memory into a virtual address. I 
think this is generally wrong.

This may really only affect those of us who have enabled high memory, but it's 
pretty prevalent in kernel code. We noted this before, but have apparently just 
been bitten by it. Is it just me or is there a fairly widespread problem for 
processors that flush the cache using virtual addresses?




     - - - - -                              Cisco                            - - - - -         
This e-mail and any attachments may contain information which is confidential, 
proprietary, privileged or otherwise protected by law. The information is solely 
intended for the named addressee (or a person responsible for delivering it to 
the addressee). If you are not the intended recipient of this message, you are 
not authorized to read, print, retain, copy or disseminate this message or any 
part of it. If you have received this e-mail in error, please notify the sender 
immediately by return e-mail and delete it from your computer.

Re: Anyone noticed that there are a lot of cache flushes after kunmap/kunmap_atomic is called?

[Ralf Baechle]

On Wed, Aug 06, 2008 at 07:09:57PM -0700, David VomLehn wrote:

> On the MIPS processor, cache flushing is done based on virtual addresses. 
> However, in the Linux kernel, there are a lot of places where memory is 
> mapped with kmap or kmap_atomic, then unmapped with the corresponding 
> kunmap or kunmap_atomic and only *then* is the cache flushed. In other 
> words, we only flush the cache after we have dropped the mapping of 
> memory into a virtual address. I think this is generally wrong.
>
> This may really only affect those of us who have enabled high memory, but 
> it's pretty prevalent in kernel code. We noted this before, but have 
> apparently just been bitten by it. Is it just me or is there a fairly 
> widespread problem for processors that flush the cache using virtual 
> addresses?

Not all MIPS processors have virtually indexed caches; some have physically
indexed caches or caches which behave effectivly like physically indexed
and MIPS highmem only tries to support the latter kinds of caches.  Which
means most of the cache flushes turn into nops anyway.

  Ralf

Re: Anyone noticed that there are a lot of cache flushes after kunmap/kunmap_atomic is called?

[Jon Fraser]

David,

  I'm battling this now.  Our mips 24k has a virtually indexed cache.
We're definitely seeing issues where the cache hasn't been flushed
for highmem pages.  I thought I had this fixed, but I'm still seeing
some problems.

Jon


On Fri, 2008-08-08 at 09:24 +0100, Ralf Baechle wrote:
> On Wed, Aug 06, 2008 at 07:09:57PM -0700, David VomLehn wrote:
> 
> > On the MIPS processor, cache flushing is done based on virtual addresses. 
> > However, in the Linux kernel, there are a lot of places where memory is 
> > mapped with kmap or kmap_atomic, then unmapped with the corresponding 
> > kunmap or kunmap_atomic and only *then* is the cache flushed. In other 
> > words, we only flush the cache after we have dropped the mapping of 
> > memory into a virtual address. I think this is generally wrong.
> >
> > This may really only affect those of us who have enabled high memory, but 
> > it's pretty prevalent in kernel code. We noted this before, but have 
> > apparently just been bitten by it. Is it just me or is there a fairly 
> > widespread problem for processors that flush the cache using virtual 
> > addresses?
> 
> Not all MIPS processors have virtually indexed caches; some have physically
> indexed caches or caches which behave effectivly like physically indexed
> and MIPS highmem only tries to support the latter kinds of caches.  Which
> means most of the cache flushes turn into nops anyway.
> 
>   Ralf
> 
> 

Re: Anyone noticed that there are a lot of cache flushes after kunmap/kunmap_atomic is called?

[David VomLehn]

Jon Fraser wrote:
> David,
> 
>   I'm battling this now.  Our mips 24k has a virtually indexed cache.
> We're definitely seeing issues where the cache hasn't been flushed
> for highmem pages.  I thought I had this fixed, but I'm still seeing
> some problems.

As I understand it, you have the most difficult combination of things:
o You are using high memory
o You have a virtually indexed cache
o You have data cache aliases

Fortunately, we have only the first two of those in our system. We should 
probably put together a coherent set of patches for people who want high memory. 
So far as I can tell, the 32-bit MIPS architecture has a long way to go before it 
runs out of steam in the embedded world. I expect more people will need MIPS 
highmem support in the next few years.

David

Re: Anyone noticed that there are a lot of cache flushes after kunmap/kunmap_atomic is called?

[Jon Fraser]

David,

I agree.  It's my number one work item this week.  Well,
so far, anyway :-)

Jon


On Mon, 2008-08-11 at 16:19 -0700, David VomLehn wrote:
> Jon Fraser wrote:
> > David,
> > 
> >   I'm battling this now.  Our mips 24k has a virtually indexed cache.
> > We're definitely seeing issues where the cache hasn't been flushed
> > for highmem pages.  I thought I had this fixed, but I'm still seeing
> > some problems.
> 
> As I understand it, you have the most difficult combination of things:
> o You are using high memory
> o You have a virtually indexed cache
> o You have data cache aliases
> 
> Fortunately, we have only the first two of those in our system. We should 
> probably put together a coherent set of patches for people who want high memory. 
> So far as I can tell, the 32-bit MIPS architecture has a long way to go before it 
> runs out of steam in the embedded world. I expect more people will need MIPS 
> highmem support in the next few years.
> 
> David
> 
> 

Re: Anyone noticed that there are a lot of cache flushes after kunmap/kunmap_atomic is called?

[Kevin D. Kissell]

I remember dealing with analgous situations in System V many many years 
ago.  If the cache flushes are virtual, then either there must be 
bullet-proof protection (hardware or software) against virtual aliasing, 
or they must be flushed on deallocation, before the mapping is 
destroyed.  The good news is that, provided that DMA buffer management 
is done correctly, one should be able to use hit-invalidate instead of 
hit-writeback-invalidate on the deallocated pages.  That won't save that 
many CPU cycles, but it will save some number of memory cycles and 
milliwatts.

          Regards,

          Kevin K.

David VomLehn wrote:
> On the MIPS processor, cache flushing is done based on virtual 
> addresses. However, in the Linux kernel, there are a lot of places 
> where memory is mapped with kmap or kmap_atomic, then unmapped with 
> the corresponding kunmap or kunmap_atomic and only *then* is the cache 
> flushed. In other words, we only flush the cache after we have dropped 
> the mapping of memory into a virtual address. I think this is 
> generally wrong.
>
> This may really only affect those of us who have enabled high memory, 
> but it's pretty prevalent in kernel code. We noted this before, but 
> have apparently just been bitten by it. Is it just me or is there a 
> fairly widespread problem for processors that flush the cache using 
> virtual addresses?
>
>
>
>
>     - - - - -                              
> Cisco                            - - - - -         This e-mail and any 
> attachments may contain information which is confidential, 
> proprietary, privileged or otherwise protected by law. The information 
> is solely intended for the named addressee (or a person responsible 
> for delivering it to the addressee). If you are not the intended 
> recipient of this message, you are not authorized to read, print, 
> retain, copy or disseminate this message or any part of it. If you 
> have received this e-mail in error, please notify the sender 
> immediately by return e-mail and delete it from your computer.
>
>
>