O_DIRECT file access and cache aliasing...

O_DIRECT file access and cache aliasing...

[David Daney]

We have a system based on a Sigam Designs SMP8634 processor (MIPS 4Kec). 
  The caches are reported as:

Primary instruction cache 16kB, physically tagged, 2-way, linesize 16 bytes.
Primary data cache 16kB, 2-way, linesize 16 bytes.

Configured with CONFIG_DMA_NONCOHERENT.

When we write files that were opened with O_DIRECT set, we observe that 
there are many 16 byte chunks of data in the files that contain all 
zeros instead of the correct data.

My understanding is that the cache is virtually indexed.  So I think 
what is happening is that when data is written to memory by a user 
application that does an O_DIRECT write, the IDE driver is given a list 
of pages to transfer to the disk.  The driver then does a 
dma_cache_wback() on the KSEG0 address of the pages before initiating 
the DMA operation.  Since the KSEG0 address and the USEG address of the 
physical memory are different, the data is never flushed to memory 
resulting in incorrect data being written to disk.

Two questions:

1) Does this analysis seem plausible?

2) How do I fix it given that I cannot change the hardware?

Several possibilities come to mind:

A) Don't use O_DIRECT mode.

B) Hack up sys_read and sys_write to flush the USEG addresses when 
CONFIG_DMA_NONCOHERENT *and* O_DIRECT are in effect.

Any helpful advice would be welcome,
David Daney

Re: O_DIRECT file access and cache aliasing...

[Atsushi Nemoto]

On Tue, 28 Aug 2007 18:04:28 -0700, David Daney <ddaney@avtrex.com> wrote:
> When we write files that were opened with O_DIRECT set, we observe that 
> there are many 16 byte chunks of data in the files that contain all 
> zeros instead of the correct data.
> 
> My understanding is that the cache is virtually indexed.  So I think 
> what is happening is that when data is written to memory by a user 
> application that does an O_DIRECT write, the IDE driver is given a list 
> of pages to transfer to the disk.  The driver then does a 
> dma_cache_wback() on the KSEG0 address of the pages before initiating 
> the DMA operation.  Since the KSEG0 address and the USEG address of the 
> physical memory are different, the data is never flushed to memory 
> resulting in incorrect data being written to disk.

I think get_user_pages() should flush user data for O_DIRECT.

The get_user_pages() uses flush_anon_page() to do it, and MIPS
flush_anon_page() was added on Mar 2007.  Does your kernel have this
fix?

---
Atsushi Nemoto

Re: O_DIRECT file access and cache aliasing...

[David Daney]

YH Lin wrote:
> Hi There,
>
> There's system call "cacheflush" which is specific to MIPS Linux for
> flushing cache in the user level.
>
> $ man cacheflush
>
> should be able to give out more information.
>   
Yes, we know of the cacheflush system call, it was one of the tools we 
used to diagnose the cache aliasing defect.

Ideally the kernel we are using (2.6.15 + vendor BSP) would present 
standard Linux semantics.  That way we would not have to conditionally 
sprinkle cacheflush calls throughout out code to achieve the correct 
behavior obtained on other systems.

As Atsushi Nemoto noted in the other message, the problem has probably 
been corrected in the kernel mainline.  Perhaps an upgrade is in order.

Thanks,
David Daney

> Regards,
> YH Lin
>
> -----Original Message-----
> From: linux-mips-bounce@linux-mips.org
> [mailto:linux-mips-bounce@linux-mips.org] On Behalf Of David Daney
> Sent: Tuesday, August 28, 2007 6:04 PM
> To: linux-mips@linux-mips.org
> Cc: Johannes Schmidt; Steve Francis
> Subject: O_DIRECT file access and cache aliasing...
>
> We have a system based on a Sigam Designs SMP8634 processor (MIPS 4Kec).
>
>   The caches are reported as:
>
> Primary instruction cache 16kB, physically tagged, 2-way, linesize 16
> bytes.
> Primary data cache 16kB, 2-way, linesize 16 bytes.
>
> Configured with CONFIG_DMA_NONCOHERENT.
>
> When we write files that were opened with O_DIRECT set, we observe that 
> there are many 16 byte chunks of data in the files that contain all 
> zeros instead of the correct data.
>
> My understanding is that the cache is virtually indexed.  So I think 
> what is happening is that when data is written to memory by a user 
> application that does an O_DIRECT write, the IDE driver is given a list 
> of pages to transfer to the disk.  The driver then does a 
> dma_cache_wback() on the KSEG0 address of the pages before initiating 
> the DMA operation.  Since the KSEG0 address and the USEG address of the 
> physical memory are different, the data is never flushed to memory 
> resulting in incorrect data being written to disk.
>
> Two questions:
>
> 1) Does this analysis seem plausible?
>
> 2) How do I fix it given that I cannot change the hardware?
>
> Several possibilities come to mind:
>
> A) Don't use O_DIRECT mode.
>
> B) Hack up sys_read and sys_write to flush the USEG addresses when 
> CONFIG_DMA_NONCOHERENT *and* O_DIRECT are in effect.
>
> Any helpful advice would be welcome,
> David Daney
>
>
>
>   

Re: O_DIRECT file access and cache aliasing...

[David Daney]

Atsushi Nemoto wrote:
> On Tue, 28 Aug 2007 18:04:28 -0700, David Daney <ddaney@avtrex.com> wrote:
>   
>> When we write files that were opened with O_DIRECT set, we observe that 
>> there are many 16 byte chunks of data in the files that contain all 
>> zeros instead of the correct data.
>>
>> My understanding is that the cache is virtually indexed.  So I think 
>> what is happening is that when data is written to memory by a user 
>> application that does an O_DIRECT write, the IDE driver is given a list 
>> of pages to transfer to the disk.  The driver then does a 
>> dma_cache_wback() on the KSEG0 address of the pages before initiating 
>> the DMA operation.  Since the KSEG0 address and the USEG address of the 
>> physical memory are different, the data is never flushed to memory 
>> resulting in incorrect data being written to disk.
>>     
>
> I think get_user_pages() should flush user data for O_DIRECT.
>
> The get_user_pages() uses flush_anon_page() to do it, and MIPS
> flush_anon_page() was added on Mar 2007.  Does your kernel have this
> fix?
>   
I doubt it.  We are currently using 2.6.15.

We will look at either moving to a recent kernel or back-porting the 
patch you mention.

Thanks for the help,
David Daney