[parisc-linux] stop the buffer bouncing

[parisc-linux] stop the buffer bouncing

[Grant Grundler]

James,
I think I found the reason parisc is bouncing buffers on the a500
when it shouldn't (iommu is present):

include/asm/pci.h defines:
#define PCI_DMA_BUS_IS_PHYS     (1)

and drivers/scsi/scsi_lib.c uses it like this:
        if (PCI_DMA_BUS_IS_PHYS && host_dev && host_dev->dma_mask)
                return *host_dev->dma_mask;

Which means anything > ~0U will get bounced.
Sym2 driver uses a 32-bit DMA mask.
And we established this a500 has RAM above :
model 9000/800/A500-6X
Memory Ranges:
 0) Start 0x0000000000000000 End 0x00000000efffffff Size   3840 Mb
 1) Start 0x00000010f0000000 End 0x00000010ffffffff Size    256 Mb

If someone else can review/test and/or commit, that would be great.
Otherwise I can commit this tomorrow. (Assuming CVS is still alive and well)

Patch below is partially inspired by include/asm-ia64/pci.h.
Though it might be better to define/declare parisc_bus_is_phys
elsewhere if EISA support also eventually needs it.

Ah, almost forgot the good news: a500 is still running multiple
instances of wipe (14 or 15, I forgot). Check riot's console
the morning...

Log Message:
	Fix the PCI_DMA_BUS_IS_PHYS so parisc-linux never bounces buffers
	on systems with an IOMMU.

Signed-of-by: Grant Grundler <grundler@parisc-linux.org>

thanks,
grant


Index: arch/parisc/kernel/setup.c
===================================================================
RCS file: /var/cvs/linux-2.6/arch/parisc/kernel/setup.c,v
retrieving revision 1.9
diff -u -p -r1.9 setup.c
--- arch/parisc/kernel/setup.c	12 Oct 2004 01:07:00 -0000	1.9
+++ arch/parisc/kernel/setup.c	16 Nov 2004 06:08:02 -0000
@@ -53,6 +53,10 @@ struct proc_dir_entry * proc_runway_root
 struct proc_dir_entry * proc_gsc_root = NULL;
 struct proc_dir_entry * proc_mckinley_root = NULL;
 
+#ifndef CONFIG_PA20
+int parisc_bus_is_phys = TRUE;	/* Assume no IOMMU is present */
+EXPORT_SYMBOL(parisc_bus_is_phys);
+#endif
 
 void __init setup_cmdline(char **cmdline_p)
 {
Index: drivers/parisc/ccio-dma.c
===================================================================
RCS file: /var/cvs/linux-2.6/drivers/parisc/ccio-dma.c,v
retrieving revision 1.16
diff -u -p -r1.16 ccio-dma.c
--- drivers/parisc/ccio-dma.c	26 Oct 2004 19:52:46 -0000	1.16
+++ drivers/parisc/ccio-dma.c	16 Nov 2004 06:08:03 -0000
@@ -1560,6 +1560,10 @@ static int ccio_probe(struct parisc_devi
 	parisc_vmerge_boundary = IOVP_SIZE;
 	parisc_vmerge_max_size = BITS_PER_LONG * IOVP_SIZE;
 	ioc_count++;
+
+#ifndef CONFIG_PA20
+	parisc_bus_is_phys = FALSE;
+#endif
 	return 0;
 }
 
Index: drivers/parisc/sba_iommu.c
===================================================================
RCS file: /var/cvs/linux-2.6/drivers/parisc/sba_iommu.c,v
retrieving revision 1.21
diff -u -p -r1.21 sba_iommu.c
--- drivers/parisc/sba_iommu.c	9 Nov 2004 20:28:33 -0000	1.21
+++ drivers/parisc/sba_iommu.c	16 Nov 2004 06:08:03 -0000
@@ -2114,6 +2116,10 @@ sba_driver_callback(struct parisc_device
 	parisc_vmerge_boundary = IOVP_SIZE;
 	parisc_vmerge_max_size = IOVP_SIZE * BITS_PER_LONG;
 
+#ifndef CONFIG_PA20
+	parisc_bus_is_phys = FALSE;
+#endif
+
 	return 0;
 }
 
Index: include/asm-parisc/pci.h
===================================================================
RCS file: /var/cvs/linux-2.6/include/asm-parisc/pci.h,v
retrieving revision 1.12
diff -u -p -r1.12 pci.h
--- include/asm-parisc/pci.h	3 Nov 2004 00:57:50 -0000	1.12
+++ include/asm-parisc/pci.h	16 Nov 2004 06:08:04 -0000
@@ -106,11 +106,22 @@ static __inline__  int pci_is_lmmio(stru
 struct pci_bus;
 struct pci_dev;
 
-/* The PCI address space does equal the physical memory
- * address space.  The networking and block device layers use
- * this boolean for bounce buffer decisions.
+#ifdef CONFIG_PA20
+/* All PA-2.0 machines have an IOMMU. */
+#define PCI_DMA_BUS_IS_PHYS     (0)
+#else
+/* _some_ PA1.1 boxes have an IOMMU (pa7200)...but we can run these bits on
+ * a PA-2.0 box in narrow mode as well. *sigh* 
+ * FIXME: WAX EISA controller mangles DMA addressing too.
+ */
+
+/* Set to TRUE if PCI address space equals the physical memory address space.
+ * The networking and block device layers use this boolean for bounce buffer
+ * decisions.
  */
-#define PCI_DMA_BUS_IS_PHYS     (1)
+extern unsigned int parisc_bus_is_phys;
+#define PCI_DMA_BUS_IS_PHYS     (parisc_bus_is_phys)
+#endif
 
 /*
 ** Most PCI devices (eg Tulip, NCR720) also export the same registers
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] stop the buffer bouncing

[James Bottomley]

On Tue, 2004-11-16 at 00:37, Grant Grundler wrote:
> If someone else can review/test and/or commit, that would be great.
> Otherwise I can commit this tomorrow. (Assuming CVS is still alive and well)

It looks fine to me ... I didn't realise this was broken ...

> Patch below is partially inspired by include/asm-ia64/pci.h.
> Though it might be better to define/declare parisc_bus_is_phys
> elsewhere if EISA support also eventually needs it.

Unfortunately, EISA support needs a per bus check for the iommu (since
WAX on some of the machines has its own IOMMU but the rest of the system
doesn't).  I was going to look at introducing this abstraction, but we
never managed to get an unencumbered copy of the WAX specs to get it all
working.

Thanks for debugging this.

James


_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] stop the buffer bouncing

[Grant Grundler]

On Tue, Nov 16, 2004 at 03:07:48PM +0000, Matthew Wilcox wrote:
> Er, what?  Eh?  sym2 only uses a 32-bit DMA mask if you configure it to.
> And it appears we do tell it to ;-(

yeah...I'm probably responsible for that setting.
It worked the best. :^/
The 39/40 bit or full 64-bit support don't help on parisc-linux anyway.

> All machines include setup.c ... that's fine.

cool

> > +#ifndef CONFIG_PA20
> > +	parisc_bus_is_phys = FALSE;
> > +#endif
>
> Ifdefs suck.  Try this ...
> 
> +	parisc_has_iommu();

Yes - that's a good idea.

> /*
>  * If the PCI device's view of memory is the same as the CPU's view of memory,
>  * PCI_DMA_BUS_IS_PHYS is true.  The networking and block device layers use
>  * this boolean for bounce buffer decisions.
>  */
> #ifdef CONFIG_PA20
> /* All PA-2.0 machines have an IOMMU. */
> #define PCI_DMA_BUS_IS_PHYS	0
> #define parisc_has_iommu()	do { } while (0)
> #else
> #define PCI_DMA_BUS_IS_PHYS	parisc_bus_is_phys
> #define parisc_has_iommu()	do { parisc_bus_is_phys = 1; } while (0)
> #endif

This looks way better to me too.

> I'm tempted to define a couple of new config symbols rather than use
> CONFIG_PA20 -- if PCI_LBA and CCIO are both deselected (eg 712 build),
> we know the answer is 1.

Yes, that's true.

> Equally, if GSC is deselected, this must be
> a pure PCI machine (if it boots ;-) and they all have an IOMMU, though
> I guess you'd always select PA20 for that case.  In any case, I think
> a couple of new symbols are in order.

Hrm....I was trying to avoid new config symbols.
We have three cases where I only saw two before:
1) PA20=y, always have IOMMU (SBA or CCIO _MUST_ be enabled).
2) PA11=y, if SBA=n and CCIO=n, never IOMMU.
3) PA11=y, if SBA=y _or_ CCIO=y, requires runtime check

I think this is straight forward enough to code up in pci.h.

Further:
CCIO depends on GSC
SBA  depends on PCI

ergo Enabling PA20 and GSC but not PCI means CCIO *MUST* be enabled.
ergo Enabling PA20 and PCI but not GSC means SBA  *MUST* be enabled.

Could someone else look at Kconfig files to verify the above two
rules are enforced?

thanks,
grant
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] stop the buffer bouncing

[Matthew Wilcox]

On Tue, Nov 16, 2004 at 09:09:39AM -0700, Grant Grundler wrote:
> yeah...I'm probably responsible for that setting.
> It worked the best. :^/
> The 39/40 bit or full 64-bit support don't help on parisc-linux anyway.

Why not?  40 bits of addressing supports up to 1TB, which exceeds the
highest physical address on any PA machine I know about -- oh, but we
can't bypass the IOMMU, right?

> > Equally, if GSC is deselected, this must be
> > a pure PCI machine (if it boots ;-) and they all have an IOMMU, though
> > I guess you'd always select PA20 for that case.  In any case, I think
> > a couple of new symbols are in order.
> 
> Hrm....I was trying to avoid new config symbols.
> We have three cases where I only saw two before:
> 1) PA20=y, always have IOMMU (SBA or CCIO _MUST_ be enabled).
> 2) PA11=y, if SBA=n and CCIO=n, never IOMMU.
> 3) PA11=y, if SBA=y _or_ CCIO=y, requires runtime check
> 
> I think this is straight forward enough to code up in pci.h.
> 
> Further:
> CCIO depends on GSC
> SBA  depends on PCI
> 
> ergo Enabling PA20 and GSC but not PCI means CCIO *MUST* be enabled.
> ergo Enabling PA20 and PCI but not GSC means SBA  *MUST* be enabled.
> 
> Could someone else look at Kconfig files to verify the above two
> rules are enforced?

Ah, "depends on" doesn't mean what you think it does.  It means "if
GSC isn't selected, CCIO must be N.  if GSC is M, CCIO can be N or M.
if GSC is Y, CCIO can be Y, N or M."  OK, GSC can't be modular (nor can PCI),
but SCSI can, so:

config SCSI_SYM53C8XX_2
        tristate "SYM53C8XX Version 2 SCSI support"
        depends on PCI && SCSI

PCI ->	Y	N
SCSI v
Y	YMN	N
M	MN	N
N	N	N

The values in the table show the permitted values that SCSI_SYM53C8XX_2
can take on.  If there's only one possible answer, it doesn't even ask
the question ;-)  Think of 'depends on' as 'limit to' and it might be more
clear to you.

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] stop the buffer bouncing

[Grant Grundler]

On Tue, Nov 16, 2004 at 04:34:03PM +0000, Matthew Wilcox wrote:
> On Tue, Nov 16, 2004 at 09:09:39AM -0700, Grant Grundler wrote:
> > yeah...I'm probably responsible for that setting.
> > It worked the best. :^/
> > The 39/40 bit or full 64-bit support don't help on parisc-linux anyway.
> 
> Why not?  40 bits of addressing supports up to 1TB, which exceeds the
> highest physical address on any PA machine I know about -- oh, but we
> can't bypass the IOMMU, right?

exactly.

Certainly not for CCIO.
On SBA, someone could try writing DVI mode support but I'm not going to.
I think DVI mode adds more complexity than performance improvement.
But clearly some workloads could benefit. Anything that helps avoid
IO TLB thrashing is a good thing.

On ZX1 (also SBA), I suppose it's possible to fully bypass using
non-coherent support.  But that seems like a step backwards for general use.

...
> > ergo Enabling PA20 and GSC but not PCI means CCIO *MUST* be enabled.
> > ergo Enabling PA20 and PCI but not GSC means SBA  *MUST* be enabled.
> > 
> > Could someone else look at Kconfig files to verify the above two
> > rules are enforced?
> 
> Ah, "depends on" doesn't mean what you think it does.  It means "if
> GSC isn't selected, CCIO must be N.

Oh. You are right...but YKWIM.

> if GSC is M, CCIO can be N or M.
> if GSC is Y, CCIO can be Y, N or M."
...
> The values in the table show the permitted values that SCSI_SYM53C8XX_2
> can take on.  If there's only one possible answer, it doesn't even ask
> the question ;-)  Think of 'depends on' as 'limit to' and it might be more
> clear to you.

*nod*
I'll have to re-read this and the original docs...but I suspected
I didn't fully understand Kconfig anyway.  That's contributing to why
I'm "defering" it. :^)

thanks,
grant
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] stop the buffer bouncing

[Grant Grundler]

On Tue, Nov 16, 2004 at 04:34:03PM +0000, Matthew Wilcox wrote:
> > Further:
> > CCIO depends on GSC
> > SBA  depends on PCI
> > 
> > ergo Enabling PA20 and GSC but not PCI means CCIO *MUST* be enabled.
> > ergo Enabling PA20 and PCI but not GSC means SBA  *MUST* be enabled.
> > 
> > Could someone else look at Kconfig files to verify the above two
> > rules are enforced?
> 
> Ah, "depends on" doesn't mean what you think it does.  It means "if
> GSC isn't selected, CCIO must be N.

Actually, in this particular case, this is what I was expecting. 
I didn't understand some of the other subtleties.
ie I'm still now sure how to encode "depends" statements
that involve PA20 and PCI/GSC vis a vis SBA/CCIO support.
Seems like it should be possible though.

thanks,
grant
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] stop the buffer bouncing

[Grant Grundler]

On Sat, Nov 20, 2004 at 10:40:04PM +0000, Joel Soete wrote:
> It seems to also help c110 pb with ncr53c710:
> a big apt-get upgrade + 2 tar in //

I'm guessing the problem is the ncr53c8xx driver does NOT call
dma_set_mask(). PCI devices by default get a 32-bit mask.
I'll commit a patch so that's true for "parisc" (ie GSC) devices
as well. EISA device drivers will need to set an appropriate dma mask.
-pa17 should be ready shortly.

grant
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] stop the buffer bouncing

[Matthew Wilcox]

On Sat, Nov 20, 2004 at 04:42:01PM -0700, Grant Grundler wrote:
> I'm guessing the problem is the ncr53c8xx driver does NOT call
> dma_set_mask(). PCI devices by default get a 32-bit mask.
> I'll commit a patch so that's true for "parisc" (ie GSC) devices
> as well. EISA device drivers will need to set an appropriate dma mask.
> -pa17 should be ready shortly.

EISA devices already get this set:

drivers/eisa/eisa-bus.c:        edev->dma_mask = root->dma_mask; /* Default DMA mask */
drivers/eisa/virtual_root.c:    .dma_mask      = 0xffffffff,

It's purely a parisc_device setup problem.  Nice spotting, Grant.

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] stop the buffer bouncing

[Matthew Wilcox]

On Sat, Nov 20, 2004 at 05:02:56PM -0700, Grant Grundler wrote:
> Ah good. But don't most EISA devices only support 24-bits?
> Oh wait...maybe that's just an ISA bus limitation that was
> carried along into EISA?

AFAIK, that's purely an ISA card limitation.  If you plug an ISA card into
an EISA slot, it'll only be able to DMA to 24-bit addresses.  But an
ISA card in an EISA slot don't get an eisa_device.

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux