Re: [Fwd: [parisc-linux] Problems with raw interface.]

Re: [Fwd: [parisc-linux] Problems with raw interface.]

[Stephen C. Tweedie]

Hi,

On Thu, 2003-09-18 at 22:08, Alan Cox wrote:
> Might interest you since this is possibly showing raw has cache
> coherency issues with HPPA (which is basically software coherency only)

> -----Forwarded Message-----
> > From: "SUDARSANAN,RANJITH (HP-India,ex2)" <ranjith_sudarsanan@hp.com>
> > To: parisc-linux@lists.parisc-linux.org
> > Subject: [parisc-linux] Problems with raw interface.
> > Date: Fri, 19 Sep 2003 00:36:55 +0530

> >         I am developing a scsi disk exerciser and I am blocked on a
> > strange problem with the raw interface. When I use the block device
> > /dev/sda directly I am able to read and write from the disk properly.
> > But when I use the raw interface I get garbage. For instance please
> > look at the output below. I am reading the 1st sector of the disk,
> > which is the boot sector, When I use the block device /dev/sda I get
> > my expected output, whereas when I use the raw interface I get
> > garbage. Can any one explain? The machine I am using is an L Class. 

Do you _ever_ get valid output, or is raw always failing?  There may
well be a cache coherency point being missed.

The key places are rw_raw_dev(), where we set up the virtual address to
physical page mappings, and brw_kiovec(), where the actual physical page
IO is done.

--Stephen

Re: [Fwd: [parisc-linux] Problems with raw interface.]

[Grant Grundler]

On Fri, Sep 19, 2003 at 05:34:04PM +0100, Stephen C. Tweedie wrote:
> > > But when I use the raw interface I get garbage. For instance please
> > > look at the output below. I am reading the 1st sector of the disk,
> > > which is the boot sector, When I use the block device /dev/sda I get
> > > my expected output, whereas when I use the raw interface I get
> > > garbage. Can any one explain? The machine I am using is an L Class. 
> 
> Do you _ever_ get valid output, or is raw always failing?  There may
> well be a cache coherency point being missed.

I was able to consistently reproduce this problem on a c3000 (400Mhz
PA8500). That's running 2.4.22 kernel.

> The key places are rw_raw_dev(), where we set up the virtual address to
> physical page mappings, and brw_kiovec(), where the actual physical page
> IO is done.

ok - I should try again with 2.6 kernel and look again.
But I'm certainly no vm/cache expert.

grant

Re: [Fwd: [parisc-linux] Problems with raw interface.]

[Grant Grundler]

On Fri, Sep 19, 2003 at 05:34:04PM +0100, Stephen C. Tweedie wrote:
> Do you _ever_ get valid output, or is raw always failing?
> There may well be a cache coherency point being missed.

2.6.0-test5 (-pa9) produced incorrect output 3 for 3.
But it's NOT identical for all three runs.
I think you are correct.

BTW, kernel and user space will alias to different cachelines
for the same 32-bit "offset" because of "Space Registers"
(form of segmented addressing).

I grabbed 128k data off the beginning of the disk.
Reading 8 blocks from raw1 was only showing all zeros.

I'll guess 2.4.22 has same bug since beginning out "output"
was also all zero's. Getting longer runs from 2.4.22
would be useful...something for tomorrow or monday.

I used:
  ion:/tmp# dd if=/dev/sda of=/tmp/sda.block1 count=512
and
  ion:/tmp# dd if=/dev/raw/raw1 of=/tmp/sda.raw1 count=512

grundler@ion:/tmp$ cksum *
1803524181 262144 sda.block1
1803524181 262144 sda.block2
1803524181 262144 sda.block3
3975907619 262144 sda.raw1
3975907619 262144 sda.raw2
2757932476 262144 sda.raw3

/ is on /dev/sdh

Output files are on ftp://gsyprf10.external.hp.com/pub/raw-pa9/

I just added 2MB files (2MB since 1.5MB dcache) which show more
of the same.

od -Ax -tx /tmp/sda.raw3 | less
000000 00000000 00000000 00000000 00000000
*
02b290 30000000 70000000 00000000 00000500
02b2a0 56000000 18000100 2d000000 00000100
02b2b0 fa040f98 0127c101 082c1698 0127c101
02b2c0 00000000 00000000 00000000 00000000
*
02b490 30000000 70000000 00000000 00000500
02b4a0 56000000 18000100 2d000000 00000100
02b4b0 fa040f98 0127c101 082c1698 0127c101
02b4c0 00000000 00000000 00000000 00000000
*
...

I think sda contains random data from a previous life in a test environment.

ion:/tmp# od -Ax -tx /tmp/sda.block1 | less
000000 33c08ed0 bc007cfb 5007501f fcbe1b7c
000010 bf1b0650 57b9e501 f3a4cbbd be07b104
000020 386e007c 09751383 c510e2f4 cd188bf5
000030 83c61049 7419382c 74f6a0b5 07b4078b
000040 f0ac3c00 74fcbb07 00b40ecd 10ebf288
000050 4e10e846 00732afe 4610807e 040b740b
000060 807e040c 7405a0b6 0775d280 46020683
...


> The key places are rw_raw_dev(), where we set up the virtual address to
> physical page mappings, and brw_kiovec(), where the actual physical page
> IO is done.

I can look, but willy or jejb have much better chance of finding
something...

cheers,
grant

Re: [Fwd: [parisc-linux] Problems with raw interface.]

[Matthew Wilcox]

On Sat, Sep 20, 2003 at 11:35:55PM -0600, Grant Grundler wrote:
> BTW, kernel and user space will alias to different cachelines
> for the same 32-bit "offset" because of "Space Registers"
> (form of segmented addressing).

Rubbish, they alias to the same cachelines because of the 4MB get-out
clause.  What you may have meant is that the same page accessed through
user and kernel mappings will alias to different cachelines because
their addresses *aren't* congruent modulo 4MB.

> > The key places are rw_raw_dev(), where we set up the virtual address to
> > physical page mappings, and brw_kiovec(), where the actual physical page
> > IO is done.
> 
> I can look, but willy or jejb have much better chance of finding
> something...

Those functions don't seem to exist in 2.6.  The only reference is:
./Documentation/block/biodoc.txt: of data, so brw_kiovec() invokes ll_rw_kio for each kiobuf in a kiovec.
which seems to be an orphaned comment.

I'll reluctantly take a look at 2.4.

-- 
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

Re: [Fwd: [parisc-linux] Problems with raw interface.]

[Grant Grundler]

On Sun, Sep 21, 2003 at 03:25:04PM +0100, Matthew Wilcox wrote:
> > BTW, kernel and user space will alias to different cachelines
> > for the same 32-bit "offset" because of "Space Registers"
> > (form of segmented addressing).
> 
> Rubbish, they alias to the same cachelines because of the 4MB get-out
> clause.  What you may have meant is that the same page accessed through
> user and kernel mappings will alias to different cachelines because
> their addresses *aren't* congruent modulo 4MB.

yes - that's what I meant...I was thinking at least part of the reason
they aren't congruent is because they are in different "segments" (ie use
a different Space ID). Maybe I'm just confused by past experience where
Space ID hashing was enabled (HPUX); parisc-linux has never had space ID
hashing enabled (at least not intentionally).

BTW, I suspect the following code in 2.4.22 mm/memory.c:map_user_kiobuf()
deals with this problem:
	...
        while (pgcount--) {
	                /* FIXME: flush superflous for rw==READ,
			*  probably wrong function for rw==WRITE
			*/
			flush_dcache_page(iobuf->maplist[pgcount]);
	}
	...

[ I got here because rw_raw_dev() calls map_user_kiobuf().]

But for "READ" (inbound data), I think I need a call that will
invalidate cachelines for userspace addresses. Preferable *after*
the DMA has completed in order to avoid issues with data prefetching
by the CPU.  ie memory has the most recent copy and any CPU holding
cachelines for the kernel address range will be coherent.


> Those functions don't seem to exist in 2.6.  The only reference is:
> ./Documentation/block/biodoc.txt: of data, so brw_kiovec() invokes
> ll_rw_kio for each kiobuf in a kiovec.
> which seems to be an orphaned comment.
> 
> I'll reluctantly take a look at 2.4.

Well, the problem exists in 2.6 and is easy to reproduce.
Perhaps find the equivalent calls there and I'll backport the fix to 2.4?
I'm assuming what's broken is obvious once one finds the right code.

thanks,
grant

Re: [Fwd: [parisc-linux] Problems with raw interface.]

[Stephen C. Tweedie]

Hi,

On Mon, 2003-09-22 at 06:00, Grant Grundler wrote:

> BTW, I suspect the following code in 2.4.22 mm/memory.c:map_user_kiobuf()
> deals with this problem:
> 	...
>         while (pgcount--) {
> 	                /* FIXME: flush superflous for rw==READ,
> 			*  probably wrong function for rw==WRITE
> 			*/
> 			flush_dcache_page(iobuf->maplist[pgcount]);
> 	}
> 	...
> 
> [ I got here because rw_raw_dev() calls map_user_kiobuf().]

Well, it might do.  It depends on the architecture, and I know zip about
the parisc cache architecture.  

For writes, we may need to flush cache writes to ram before the IO to
ensure ram is up-to-date.  For reads. we may need to flush pending
writeback before the IO (to ensure that prior cache operations don't end
up being committed on top of the new IO), plus we need to flush the
entire region out of cache after the IO in order to make the new
contents visible.

Where that needs to be done will depend on whether your cache flush
instructions work from physical or virtual addresses.  brw_koivec() is
the place to deal with physical cache flushes and invalidations, as
that's where we do the low-level IO on the struct page.  But you'll need
to do it at a higher level within the raw.c driver itself if you need
access to the virtual addresses.

--Stephen