file corruption with highmem kernel

file corruption with highmem kernel

[Anoop P.A.]

List,

I am running 2.6.18 (highmem) kernel in RM9000 based SOC with 2 Gig RAM.
I have observed file corruption and system hangs (Easily reproducible on
remounting file system) when doing file copy to SATA disk / USB disk (
SATA/ USB controller over PCI) . How ever when I limit memory with
option mem=512M from command line everything seems to be working fine.

Issue is reproducible with 2.6.18-stable lmo git sources .

I have modified dma-noncoherent.c as follows and I am no more
experiencing system hang. But file's are getting corrupted (observed bus
error / segmentation fault / illegal instruction error few times)
occasionally.  One more observation I have made is file corruption is
more if I use root file system from onboard USB flash, than running a
NFS root mount.


--- arch/mips/mm/dma-noncoherent.c.orig	2010-08-02 23:53:17.000000000
+0530
+++ arch/mips/mm/dma-noncoherent.c	2010-08-02 23:56:19.000000000
+0530
@@ -132,12 +132,13 @@
 	for (i = 0; i < nents; i++, sg++) {
 		unsigned long addr;
 
-		addr = (unsigned long) page_address(sg->page);
-		if (addr) {
-			__dma_sync(addr + sg->offset, sg->length,
direction);
-			sg->dma_address =
(dma_addr_t)page_to_phys(sg->page)
-					  + sg->offset;
-		}
+		addr = (unsigned long) page_address(sg->page) +
sg->offset;
+		if (addr) 
+			__dma_sync(addr, sg->length, direction);
+		
+		sg->dma_address = (dma_addr_t)page_to_phys(sg->page)
+					+ sg->offset;
+		
 	}
 
 	return nents;
@@ -187,9 +188,9 @@
 		return;
 
 	for (i = 0; i < nhwentries; i++, sg++) {
-		addr = (unsigned long) page_address(sg->page);
+		addr = (unsigned long) page_address(sg->page) +
sg->offset;
 		if (addr)
-			__dma_sync(addr + sg->offset, sg->length,
direction);
+			__dma_sync(addr , sg->length, direction);
 	}
 }


It will be great if any body can give me some pointers / help to fix the
issue.

Thanks
Anoop

RE: file corruption with highmem kernel

[Anoop P.A.]

List,

With a slightly modified patched (copied below) I have reached a point
where I am no more seeing errors like segmentation fault, bus error
(which was due to memory corruption I believe).
How ever I am still seeing some kind of file corruption. 

I believe this file corruption happening because cache is not getting
invalidated before a highmem dma. I am not sure which routine to call to
invalidate cache for a highmem address. 

Hope to see response from linux-mips gurus

Thank you,
Anoop   

--- arch/mips/mm/dma-noncoherent.c.orig	2010-08-02 23:53:17.000000000
+0530
+++ arch/mips/mm/dma-noncoherent.c	2010-08-06 00:17:21.000000000
+0530
@@ -131,13 +131,14 @@
 
 	for (i = 0; i < nents; i++, sg++) {
 		unsigned long addr;
-
-		addr = (unsigned long) page_address(sg->page);
-		if (addr) {
-			__dma_sync(addr + sg->offset, sg->length,
direction);
-			sg->dma_address =
(dma_addr_t)page_to_phys(sg->page)
-					  + sg->offset;
+		if (!PageHighMem(sg->page)){
+			addr = (unsigned long)page_address(sg->page) +
sg->offset;
+			__dma_sync(addr , sg->length, direction);
 		}
+
+		sg->dma_address = (dma_addr_t)page_to_phys(sg->page)
+					+ sg->offset;
+		
 	}
 
 	return nents;
@@ -187,9 +188,10 @@
 		return;
 
 	for (i = 0; i < nhwentries; i++, sg++) {
-		addr = (unsigned long) page_address(sg->page);
-		if (addr)
-			__dma_sync(addr + sg->offset, sg->length,
direction);
+		if (!PageHighMem(sg->page)){
+			addr = (unsigned long)page_address(sg->page) +
sg->offset;
+			__dma_sync(addr , sg->length, direction);
+		}
 	}
 }

Re: file corruption with highmem kernel

[Ralf Baechle]

On Thu, Aug 05, 2010 at 06:43:50AM -0700, Anoop P.A. wrote:

> With a slightly modified patched (copied below) I have reached a point
> where I am no more seeing errors like segmentation fault, bus error
> (which was due to memory corruption I believe).
> How ever I am still seeing some kind of file corruption. 
> 
> I believe this file corruption happening because cache is not getting
> invalidated before a highmem dma. I am not sure which routine to call to
> invalidate cache for a highmem address. 

Since you're running on an RM9000 class CPU, why don't just use a 64-bit
kernel?  Highmem is just so f*cking insane that you want to avoid it like
French kisses from a zombie suffering ebola.  If you have DMA restrictions
then you may consider reusing ZONE_DMA.

That said, a word on the history of the MIPS highmem code.  It was written
for a company who in the early stages of the 64-bit kernel didn't want to
be the first through the minefield in 2001.  That CPU had full coherency
and no cache aliases so arch/mips/mm/dma-*.c did not need any code to
support it at all and for many years after that everybody either had a
small 32-bit system that didn't need highmem or went 64-bit right away so
the gaps in the code while well known were never fixed up ...

I'm a bit surprised that the patch posted actually made things work better
for you since it entirely avoids flushing of highmem pages.  The code as it
was originally written using page_address() will perform cacheflushes
for highmem pages as well - but only for highmem pages are actually are
mapped.  That is your code will flush less pages than the existing code.

  Ralf

RE: file corruption with highmem kernel

[Anoop P.A.]

> 
> Since you're running on an RM9000 class CPU, why don't just use a
64-bit
> kernel?  Highmem is just so f*cking insane that you want to avoid it
like
> French kisses from a zombie suffering ebola.  If you have DMA
restrictions
> then you may consider reusing ZONE_DMA.
[Anoop P.A.] I understand that I am working on 64 bit kernel in
parallel, DMA issue made me concentrate on highmem support as immediate
option.
 
> small 32-bit system that didn't need highmem or went 64-bit right away
so
> the gaps in the code while well known were never fixed up ...

[Anoop P.A.] I presume with decreased value of RAM, more 32 bit machines
will come with bulk of memory pretty soon to the world of highmem

> 
> I'm a bit surprised that the patch posted actually made things work
better
> for you since it entirely avoids flushing of highmem pages.  The code
as
> it
> was originally written using page_address() will perform cacheflushes
> for highmem pages as well - but only for highmem pages are actually
are
> mapped.  That is your code will flush less pages than the existing
code.

[Anoop P.A.] I confirmed it, calling __dma_sync for highmem map address
occasionally causes memory corruption ( reports bus error , segmentation
fault etc..) .

Blowing entire scache (I understand it is pain) for highmem pages fixed
my file corruption issue

--- linux-2.6.18/arch/mips/mm/dma-noncoherent.c	2006-08-23
23:16:07.000000000 +0530
+++ linux-2.6.18/arch/mips/mm/dma-noncoherent.c	2010-08-06
20:48:36.000000000 +0530
@@ -15,6 +15,7 @@
 
 #include <asm/cache.h>
 #include <asm/io.h>
+#include <asm/r4kcache.h>
 
 /*
  * Warning on the terminology - Linux calls an uncached area coherent;
@@ -131,13 +132,21 @@
 
 	for (i = 0; i < nents; i++, sg++) {
 		unsigned long addr;
+		if (!PageHighMem(sg->page))
+		{
+               		addr = (unsigned
long)page_address(sg->page) + sg->offset;
+               		__dma_sync(addr, sg->length, direction);
+		}
+		else
+		{ 
+		/*blasting entire cache for all	the highmem page's might
be over head 
+		  But __dma_sync is failing for mapped highmem pages, so
this is the 
+		  easiest solution for time being*/
+			blast_scache32(); /*RM9000 sc_line=32 */ 
+		}
 
-		addr = (unsigned long) page_address(sg->page);
-		if (addr) {
-			__dma_sync(addr + sg->offset, sg->length,
direction);
-			sg->dma_address =
(dma_addr_t)page_to_phys(sg->page)
+		sg->dma_address = (dma_addr_t)page_to_phys(sg->page)
 					  + sg->offset;
-		}
 	}
 
 	return nents;


Thanks
Anoop