* Re: [Fwd: [parisc-linux] Problems with raw interface.] [not found] <1063919293.16536.2.camel@dhcp23.swansea.linux.org.uk> @ 2003-09-19 16:34 ` Stephen C. Tweedie 2003-09-21 0:27 ` Grant Grundler 2003-09-21 5:35 ` Grant Grundler 0 siblings, 2 replies; 6+ messages in thread From: Stephen C. Tweedie @ 2003-09-19 16:34 UTC (permalink / raw) To: Alan Cox, Ranjith Sudarsanan, parisc-linux@lists.parisc-linux.org Cc: Stephen Tweedie Hi, On Thu, 2003-09-18 at 22:08, Alan Cox wrote: > Might interest you since this is possibly showing raw has cache > coherency issues with HPPA (which is basically software coherency only) > -----Forwarded Message----- > > From: "SUDARSANAN,RANJITH (HP-India,ex2)" <ranjith_sudarsanan@hp.com> > > To: parisc-linux@lists.parisc-linux.org > > Subject: [parisc-linux] Problems with raw interface. > > Date: Fri, 19 Sep 2003 00:36:55 +0530 > > I am developing a scsi disk exerciser and I am blocked on a > > strange problem with the raw interface. When I use the block device > > /dev/sda directly I am able to read and write from the disk properly. > > But when I use the raw interface I get garbage. For instance please > > look at the output below. I am reading the 1st sector of the disk, > > which is the boot sector, When I use the block device /dev/sda I get > > my expected output, whereas when I use the raw interface I get > > garbage. Can any one explain? The machine I am using is an L Class. Do you _ever_ get valid output, or is raw always failing? There may well be a cache coherency point being missed. The key places are rw_raw_dev(), where we set up the virtual address to physical page mappings, and brw_kiovec(), where the actual physical page IO is done. --Stephen ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Fwd: [parisc-linux] Problems with raw interface.] 2003-09-19 16:34 ` [Fwd: [parisc-linux] Problems with raw interface.] Stephen C. Tweedie @ 2003-09-21 0:27 ` Grant Grundler 2003-09-21 5:35 ` Grant Grundler 1 sibling, 0 replies; 6+ messages in thread From: Grant Grundler @ 2003-09-21 0:27 UTC (permalink / raw) To: Stephen C. Tweedie Cc: Alan Cox, Ranjith Sudarsanan, parisc-linux@lists.parisc-linux.org On Fri, Sep 19, 2003 at 05:34:04PM +0100, Stephen C. Tweedie wrote: > > > But when I use the raw interface I get garbage. For instance please > > > look at the output below. I am reading the 1st sector of the disk, > > > which is the boot sector, When I use the block device /dev/sda I get > > > my expected output, whereas when I use the raw interface I get > > > garbage. Can any one explain? The machine I am using is an L Class. > > Do you _ever_ get valid output, or is raw always failing? There may > well be a cache coherency point being missed. I was able to consistently reproduce this problem on a c3000 (400Mhz PA8500). That's running 2.4.22 kernel. > The key places are rw_raw_dev(), where we set up the virtual address to > physical page mappings, and brw_kiovec(), where the actual physical page > IO is done. ok - I should try again with 2.6 kernel and look again. But I'm certainly no vm/cache expert. grant ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Fwd: [parisc-linux] Problems with raw interface.] 2003-09-19 16:34 ` [Fwd: [parisc-linux] Problems with raw interface.] Stephen C. Tweedie 2003-09-21 0:27 ` Grant Grundler @ 2003-09-21 5:35 ` Grant Grundler 2003-09-21 14:25 ` Matthew Wilcox 1 sibling, 1 reply; 6+ messages in thread From: Grant Grundler @ 2003-09-21 5:35 UTC (permalink / raw) To: Stephen C. Tweedie Cc: Alan Cox, Ranjith Sudarsanan, parisc-linux@lists.parisc-linux.org On Fri, Sep 19, 2003 at 05:34:04PM +0100, Stephen C. Tweedie wrote: > Do you _ever_ get valid output, or is raw always failing? > There may well be a cache coherency point being missed. 2.6.0-test5 (-pa9) produced incorrect output 3 for 3. But it's NOT identical for all three runs. I think you are correct. BTW, kernel and user space will alias to different cachelines for the same 32-bit "offset" because of "Space Registers" (form of segmented addressing). I grabbed 128k data off the beginning of the disk. Reading 8 blocks from raw1 was only showing all zeros. I'll guess 2.4.22 has same bug since beginning out "output" was also all zero's. Getting longer runs from 2.4.22 would be useful...something for tomorrow or monday. I used: ion:/tmp# dd if=/dev/sda of=/tmp/sda.block1 count=512 and ion:/tmp# dd if=/dev/raw/raw1 of=/tmp/sda.raw1 count=512 grundler@ion:/tmp$ cksum * 1803524181 262144 sda.block1 1803524181 262144 sda.block2 1803524181 262144 sda.block3 3975907619 262144 sda.raw1 3975907619 262144 sda.raw2 2757932476 262144 sda.raw3 / is on /dev/sdh Output files are on ftp://gsyprf10.external.hp.com/pub/raw-pa9/ I just added 2MB files (2MB since 1.5MB dcache) which show more of the same. od -Ax -tx /tmp/sda.raw3 | less 000000 00000000 00000000 00000000 00000000 * 02b290 30000000 70000000 00000000 00000500 02b2a0 56000000 18000100 2d000000 00000100 02b2b0 fa040f98 0127c101 082c1698 0127c101 02b2c0 00000000 00000000 00000000 00000000 * 02b490 30000000 70000000 00000000 00000500 02b4a0 56000000 18000100 2d000000 00000100 02b4b0 fa040f98 0127c101 082c1698 0127c101 02b4c0 00000000 00000000 00000000 00000000 * ... I think sda contains random data from a previous life in a test environment. ion:/tmp# od -Ax -tx /tmp/sda.block1 | less 000000 33c08ed0 bc007cfb 5007501f fcbe1b7c 000010 bf1b0650 57b9e501 f3a4cbbd be07b104 000020 386e007c 09751383 c510e2f4 cd188bf5 000030 83c61049 7419382c 74f6a0b5 07b4078b 000040 f0ac3c00 74fcbb07 00b40ecd 10ebf288 000050 4e10e846 00732afe 4610807e 040b740b 000060 807e040c 7405a0b6 0775d280 46020683 ... > The key places are rw_raw_dev(), where we set up the virtual address to > physical page mappings, and brw_kiovec(), where the actual physical page > IO is done. I can look, but willy or jejb have much better chance of finding something... cheers, grant ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Fwd: [parisc-linux] Problems with raw interface.] 2003-09-21 5:35 ` Grant Grundler @ 2003-09-21 14:25 ` Matthew Wilcox 2003-09-22 5:00 ` Grant Grundler 0 siblings, 1 reply; 6+ messages in thread From: Matthew Wilcox @ 2003-09-21 14:25 UTC (permalink / raw) To: Grant Grundler Cc: Stephen C. Tweedie, Alan Cox, Ranjith Sudarsanan, parisc-linux@lists.parisc-linux.org On Sat, Sep 20, 2003 at 11:35:55PM -0600, Grant Grundler wrote: > BTW, kernel and user space will alias to different cachelines > for the same 32-bit "offset" because of "Space Registers" > (form of segmented addressing). Rubbish, they alias to the same cachelines because of the 4MB get-out clause. What you may have meant is that the same page accessed through user and kernel mappings will alias to different cachelines because their addresses *aren't* congruent modulo 4MB. > > The key places are rw_raw_dev(), where we set up the virtual address to > > physical page mappings, and brw_kiovec(), where the actual physical page > > IO is done. > > I can look, but willy or jejb have much better chance of finding > something... Those functions don't seem to exist in 2.6. The only reference is: ./Documentation/block/biodoc.txt: of data, so brw_kiovec() invokes ll_rw_kio for each kiobuf in a kiovec. which seems to be an orphaned comment. I'll reluctantly take a look at 2.4. -- "It's not Hollywood. War is real, war is primarily not about defeat or victory, it is about death. I've seen thousands and thousands of dead bodies. Do you think I want to have an academic debate on this subject?" -- Robert Fisk ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Fwd: [parisc-linux] Problems with raw interface.] 2003-09-21 14:25 ` Matthew Wilcox @ 2003-09-22 5:00 ` Grant Grundler 2003-09-22 10:47 ` Stephen C. Tweedie 0 siblings, 1 reply; 6+ messages in thread From: Grant Grundler @ 2003-09-22 5:00 UTC (permalink / raw) To: Matthew Wilcox Cc: Stephen C. Tweedie, Alan Cox, Ranjith Sudarsanan, parisc-linux@lists.parisc-linux.org On Sun, Sep 21, 2003 at 03:25:04PM +0100, Matthew Wilcox wrote: > > BTW, kernel and user space will alias to different cachelines > > for the same 32-bit "offset" because of "Space Registers" > > (form of segmented addressing). > > Rubbish, they alias to the same cachelines because of the 4MB get-out > clause. What you may have meant is that the same page accessed through > user and kernel mappings will alias to different cachelines because > their addresses *aren't* congruent modulo 4MB. yes - that's what I meant...I was thinking at least part of the reason they aren't congruent is because they are in different "segments" (ie use a different Space ID). Maybe I'm just confused by past experience where Space ID hashing was enabled (HPUX); parisc-linux has never had space ID hashing enabled (at least not intentionally). BTW, I suspect the following code in 2.4.22 mm/memory.c:map_user_kiobuf() deals with this problem: ... while (pgcount--) { /* FIXME: flush superflous for rw==READ, * probably wrong function for rw==WRITE */ flush_dcache_page(iobuf->maplist[pgcount]); } ... [ I got here because rw_raw_dev() calls map_user_kiobuf().] But for "READ" (inbound data), I think I need a call that will invalidate cachelines for userspace addresses. Preferable *after* the DMA has completed in order to avoid issues with data prefetching by the CPU. ie memory has the most recent copy and any CPU holding cachelines for the kernel address range will be coherent. > Those functions don't seem to exist in 2.6. The only reference is: > ./Documentation/block/biodoc.txt: of data, so brw_kiovec() invokes > ll_rw_kio for each kiobuf in a kiovec. > which seems to be an orphaned comment. > > I'll reluctantly take a look at 2.4. Well, the problem exists in 2.6 and is easy to reproduce. Perhaps find the equivalent calls there and I'll backport the fix to 2.4? I'm assuming what's broken is obvious once one finds the right code. thanks, grant ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Fwd: [parisc-linux] Problems with raw interface.] 2003-09-22 5:00 ` Grant Grundler @ 2003-09-22 10:47 ` Stephen C. Tweedie 0 siblings, 0 replies; 6+ messages in thread From: Stephen C. Tweedie @ 2003-09-22 10:47 UTC (permalink / raw) To: Grant Grundler Cc: Matthew Wilcox, Alan Cox, Ranjith Sudarsanan, parisc-linux@lists.parisc-linux.org, Stephen Tweedie Hi, On Mon, 2003-09-22 at 06:00, Grant Grundler wrote: > BTW, I suspect the following code in 2.4.22 mm/memory.c:map_user_kiobuf() > deals with this problem: > ... > while (pgcount--) { > /* FIXME: flush superflous for rw==READ, > * probably wrong function for rw==WRITE > */ > flush_dcache_page(iobuf->maplist[pgcount]); > } > ... > > [ I got here because rw_raw_dev() calls map_user_kiobuf().] Well, it might do. It depends on the architecture, and I know zip about the parisc cache architecture. For writes, we may need to flush cache writes to ram before the IO to ensure ram is up-to-date. For reads. we may need to flush pending writeback before the IO (to ensure that prior cache operations don't end up being committed on top of the new IO), plus we need to flush the entire region out of cache after the IO in order to make the new contents visible. Where that needs to be done will depend on whether your cache flush instructions work from physical or virtual addresses. brw_koivec() is the place to deal with physical cache flushes and invalidations, as that's where we do the low-level IO on the struct page. But you'll need to do it at a higher level within the raw.c driver itself if you need access to the virtual addresses. --Stephen ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2003-09-22 10:47 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <1063919293.16536.2.camel@dhcp23.swansea.linux.org.uk>
2003-09-19 16:34 ` [Fwd: [parisc-linux] Problems with raw interface.] Stephen C. Tweedie
2003-09-21 0:27 ` Grant Grundler
2003-09-21 5:35 ` Grant Grundler
2003-09-21 14:25 ` Matthew Wilcox
2003-09-22 5:00 ` Grant Grundler
2003-09-22 10:47 ` Stephen C. Tweedie
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox