DMA transfers in 2.5.67

DMA transfers in 2.5.67

[Måns Rullgård]

What do I need to do in a driver before doing DMA transfers to a PCI
card?  Using a driver that worked in 2.4 gives a throughput of only 10
MB/s in 2.5.67.  Is there some magic initialization that I have
missed?

-- 
Måns Rullgård
mru@users.sf.net

Re: DMA transfers in 2.5.67

[Alan Cox]

On Maw, 2003-04-15 at 21:01, Måns Rullgård wrote:
> What do I need to do in a driver before doing DMA transfers to a PCI
> card?  Using a driver that worked in 2.4 gives a throughput of only 10
> MB/s in 2.5.67.  Is there some magic initialization that I have
> missed?

Assuming your driver uses the new PCI api for DMA in 2.4/2.5 then there
isnt really anything to watch. Is this on a box with > 800Mb of memory
however ?

Re: DMA transfers in 2.5.67

[Alan Cox]

On Maw, 2003-04-15 at 22:38, Måns Rullgård wrote:
> It's an Alpha with 768 MB.  Is it the pci_alloc_* functions you are
> referring to?  I don't think they are used currently. How much memory
> can these allocate?  I need chunks of up to 1 MB, not necessarily
> phycically continuous.
> 
> What do those functions do that normal memory allocation does not?
> Apart from setting up sg mappings, that is.

A normal memory allocation might not be visible from the device, however
pci_map_sg() deals with such things. What I really meant was are you
using the pci_ DMA functionality

Re: DMA transfers in 2.5.67

[Måns Rullgård]

Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

> > What do I need to do in a driver before doing DMA transfers to a PCI
> > card?  Using a driver that worked in 2.4 gives a throughput of only 10
> > MB/s in 2.5.67.  Is there some magic initialization that I have
> > missed?
> 
> Assuming your driver uses the new PCI api for DMA in 2.4/2.5 then there
> isnt really anything to watch. Is this on a box with > 800Mb of memory
> however ?

It's an Alpha with 768 MB.  Is it the pci_alloc_* functions you are
referring to?  I don't think they are used currently. How much memory
can these allocate?  I need chunks of up to 1 MB, not necessarily
phycically continuous.

What do those functions do that normal memory allocation does not?
Apart from setting up sg mappings, that is.

-- 
Måns Rullgård
mru@users.sf.net

Re: DMA transfers in 2.5.67

[Måns Rullgård]

Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

> > It's an Alpha with 768 MB.  Is it the pci_alloc_* functions you are
> > referring to?  I don't think they are used currently. How much memory
> > can these allocate?  I need chunks of up to 1 MB, not necessarily
> > phycically continuous.
> > 
> > What do those functions do that normal memory allocation does not?
> > Apart from setting up sg mappings, that is.
> 
> A normal memory allocation might not be visible from the device, however
> pci_map_sg() deals with such things. What I really meant was are you
> using the pci_ DMA functionality

Those functions are not used at the moment, but I could change that.
The question remains why DMA transfers are so slow.  The memory is
clearly visible from the bus.

Btw, I just noticed that hard disk throughput is much lower with 2.5
than 2.4.  With 2.4.21-pre5 I get ~40 MB/s, but with 2.5.67 the speed
drops to 25-30 MB/s.  Everything according to hdparm.  Is it possible
that DMA is generally slow for some reason?

-- 
Måns Rullgård
mru@users.sf.net

Re: DMA transfers in 2.5.67

[Ivan Kokshaysky]

On Wed, Apr 16, 2003 at 12:09:00AM +0200, MЕns RullgЕrd wrote:
> Btw, I just noticed that hard disk throughput is much lower with 2.5
> than 2.4.  With 2.4.21-pre5 I get ~40 MB/s, but with 2.5.67 the speed
> drops to 25-30 MB/s.  Everything according to hdparm.  Is it possible
> that DMA is generally slow for some reason?

Possible reason is that in 2.4 we've forced reasonable latency timer
value for all PCI devices, while in 2.5 we haven't as yet.

Ivan.

Re: DMA transfers in 2.5.67

[Måns Rullgård]

Ivan Kokshaysky <ink@jurassic.park.msu.ru> writes:

> On Wed, Apr 16, 2003 at 12:09:00AM +0200, Måns Rullgård wrote:
> > Btw, I just noticed that hard disk throughput is much lower with 2.5
> > than 2.4.  With 2.4.21-pre5 I get ~40 MB/s, but with 2.5.67 the speed
> > drops to 25-30 MB/s.  Everything according to hdparm.  Is it possible
> > that DMA is generally slow for some reason?
> 
> Possible reason is that in 2.4 we've forced reasonable latency timer
> value for all PCI devices, while in 2.5 we haven't as yet.

Do you mean whatever causes this message (for a 3com NIC)?

PCI: Setting latency timer of device 00:05.0 to 64

Would that also explain why my hard disks are slow under 2.5?  There
is no corresponding message for the ide controller (htp374).

-- 
Måns Rullgård
mru@users.sf.net

Re: DMA transfers in 2.5.67

[Måns Rullgård]

mru@users.sourceforge.net (Måns Rullgård) writes:

> > > Btw, I just noticed that hard disk throughput is much lower with 2.5
> > > than 2.4.  With 2.4.21-pre5 I get ~40 MB/s, but with 2.5.67 the speed
> > > drops to 25-30 MB/s.  Everything according to hdparm.  Is it possible
> > > that DMA is generally slow for some reason?
> > 
> > Possible reason is that in 2.4 we've forced reasonable latency timer
> > value for all PCI devices, while in 2.5 we haven't as yet.
> 
> Do you mean whatever causes this message (for a 3com NIC)?
> 
> PCI: Setting latency timer of device 00:05.0 to 64
> 
> Would that also explain why my hard disks are slow under 2.5?  There
> is no corresponding message for the ide controller (htp374).

I just checked the troublesome board.  Here's what lspci has to say:

00:06.0 Display controller: 3DLabs GLINT R3 (rev 01)
	Subsystem: 3DLabs: Unknown device 0121
	Flags: bus master, 66Mhz, medium devsel, latency 0, IRQ 27
	Memory at 0000000009020000 (32-bit, non-prefetchable) [size=128K]
	Memory at 000000000c000000 (32-bit, prefetchable) [size=64M]
	Memory at 0000000010000000 (32-bit, prefetchable) [size=64M]
	Expansion ROM at 0000000009060000 [disabled] [size=64K]
	Capabilities: <available only to root>

The latency 0 doesn't look too good.  What should I do to change it?

-- 
Måns Rullgård
mru@users.sf.net

Re: DMA transfers in 2.5.67

[Måns Rullgård]

mru@users.sourceforge.net (Måns Rullgård) writes:

> > > > Btw, I just noticed that hard disk throughput is much lower with 2.5
> > > > than 2.4.  With 2.4.21-pre5 I get ~40 MB/s, but with 2.5.67 the speed
> > > > drops to 25-30 MB/s.  Everything according to hdparm.  Is it possible
> > > > that DMA is generally slow for some reason?
> > > 
> > > Possible reason is that in 2.4 we've forced reasonable latency timer
> > > value for all PCI devices, while in 2.5 we haven't as yet.
> > 
> > Do you mean whatever causes this message (for a 3com NIC)?
> > 
> > PCI: Setting latency timer of device 00:05.0 to 64
> > 
> > Would that also explain why my hard disks are slow under 2.5?  There
> > is no corresponding message for the ide controller (htp374).
> 
> I just checked the troublesome board.  Here's what lspci has to say:
> 
> 00:06.0 Display controller: 3DLabs GLINT R3 (rev 01)
> 	Subsystem: 3DLabs: Unknown device 0121
> 	Flags: bus master, 66Mhz, medium devsel, latency 0, IRQ 27
> 	Memory at 0000000009020000 (32-bit, non-prefetchable) [size=128K]
> 	Memory at 000000000c000000 (32-bit, prefetchable) [size=64M]
> 	Memory at 0000000010000000 (32-bit, prefetchable) [size=64M]
> 	Expansion ROM at 0000000009060000 [disabled] [size=64K]
> 	Capabilities: <available only to root>
> 
> The latency 0 doesn't look too good.  What should I do to change it?

I set the latency to 128 using setpci and now I get 66 MB/s.  Other
values give slower transfer rates.  Is there some other setting that
could improve it even more?

-- 
Måns Rullgård
mru@users.sf.net

Re: DMA transfers in 2.5.67

[Chuck Ebbert]

> Btw, I just noticed that hard disk throughput is much lower with 2.5
> than 2.4.  With 2.4.21-pre5 I get ~40 MB/s, but with 2.5.67 the speed
> drops to 25-30 MB/s.  Everything according to hdparm.  Is it possible
> that DMA is generally slow for some reason?


  I am seeing that too, with IDE hardware.

  And 2.4.20aa1 is even faster:



# mount /ext3_fs
# time dd if=/ext3_fs/100MiB_file of=/dev/null bs=32k



  2.4.20aa1 : 3.3 sec (exactly what I expect to see)
  2.5.66    : 6.6 sec


--
 Chuck

Re: DMA transfers in 2.5.67

[Ivan Kokshaysky]

On Wed, Apr 16, 2003 at 11:30:43AM +0200, MЕns RullgЕrd wrote:
> I set the latency to 128 using setpci and now I get 66 MB/s.  Other
> values give slower transfer rates.  Is there some other setting that
> could improve it even more?

Interesting. Did you set latency 128 for all devices or only for
3Dlabs card?

Ivan.

Re: DMA transfers in 2.5.67

[Måns Rullgård]

Ivan Kokshaysky <ink@jurassic.park.msu.ru> writes:

> > I set the latency to 128 using setpci and now I get 66 MB/s.  Other
> > values give slower transfer rates.  Is there some other setting that
> > could improve it even more?
> 
> Interesting. Did you set latency 128 for all devices or only for
> 3Dlabs card?

Only the 3Dlabs.  The remaining cards are configured like this:

00:05.0 Ethernet controller: 3Com Corporation 3c905C-TX [Fast Etherlink] (rev 78)
	Subsystem: 3Com Corporation 3C905C-TX Fast Etherlink for PC Management NIC
	Flags: bus master, medium devsel, latency 64, IRQ 25

00:06.0 Display controller: 3DLabs GLINT R3 (rev 01)
	Subsystem: 3DLabs: Unknown device 0121
	Flags: bus master, 66Mhz, medium devsel, latency 128, IRQ 27

00:07.0 RAID bus controller: Triones Technologies, Inc.: Unknown device 0008 (rev 07)
	Subsystem: Triones Technologies, Inc.: Unknown device 0001
	Flags: bus master, 66Mhz, medium devsel, latency 120, IRQ 26

00:07.1 RAID bus controller: Triones Technologies, Inc.: Unknown device 0008 (rev 07)
	Subsystem: Triones Technologies, Inc.: Unknown device 0001
	Flags: bus master, 66Mhz, medium devsel, latency 120, IRQ 26

00:08.1 IDE interface: Contaq Microsystems 82c693 (prog-if 80 [Master])
	Flags: bus master, medium devsel, latency 0

00:08.2 IDE interface: Contaq Microsystems 82c693 (prog-if 00 [])
	Flags: bus master, medium devsel, latency 0

00:08.3 USB Controller: Contaq Microsystems 82c693 (prog-if 10 [OHCI])
	Flags: bus master, medium devsel, latency 0

00:09.0 USB Controller: NEC Corporation USB (rev 41) (prog-if 10 [OHCI])
	Subsystem: Adaptec: Unknown device 0035
	Flags: bus master, medium devsel, latency 64, IRQ 24

00:09.1 USB Controller: NEC Corporation USB (rev 41) (prog-if 10 [OHCI])
	Subsystem: Adaptec: Unknown device 0035
	Flags: bus master, medium devsel, latency 64, IRQ 28

00:09.2 USB Controller: NEC Corporation: Unknown device 00e0 (rev 02) (prog-if 20)
	Subsystem: Adaptec: Unknown device 00e0
	Flags: bus master, medium devsel, latency 68, IRQ 32

As you can see, the good IDE controller (Triones/Highpoint hpt374) has
a latency of 120.  Disk transfer rates are still only half of what I
get with a 2.4 kernel.  What could be the cause of this?


-- 
Måns Rullgård
mru@users.sf.net

Re: DMA transfers in 2.5.67

[Andrew Morton]

Chuck Ebbert <76306.1226@compuserve.com> wrote:
>
> # mount /ext3_fs
> # time dd if=/ext3_fs/100MiB_file of=/dev/null bs=32k
> 
>   2.4.20aa1 : 3.3 sec (exactly what I expect to see)
>   2.5.66    : 6.6 sec

With this test 2.4 will leave a lot more unwritten dirty data in memory.

You should include a `sync' in the timings.

Re: DMA transfers in 2.5.67

[Andrew Morton]

Andrew Morton <akpm@digeo.com> wrote:
>
> Chuck Ebbert <76306.1226@compuserve.com> wrote:
> >
> > # mount /ext3_fs
> > # time dd if=/ext3_fs/100MiB_file of=/dev/null bs=32k
> > 
> >   2.4.20aa1 : 3.3 sec (exactly what I expect to see)
> >   2.5.66    : 6.6 sec
> 
> With this test 2.4 will leave a lot more unwritten dirty data in memory.
> 
> You should include a `sync' in the timings.

Well you should include the sync if you're writing to disk ;)

doh.

Re: DMA transfers in 2.5.67

[Måns Rullgård]

Andrew Morton <akpm@digeo.com> writes:

> > # mount /ext3_fs
> > # time dd if=/ext3_fs/100MiB_file of=/dev/null bs=32k
> > 
> >   2.4.20aa1 : 3.3 sec (exactly what I expect to see)
> >   2.5.66    : 6.6 sec
> 
> With this test 2.4 will leave a lot more unwritten dirty data in memory.
> 
> You should include a `sync' in the timings.

That was reading a file discarding that data.  A sync shouldn't make
any difference.

-- 
Måns Rullgård
mru@users.sf.net

Re: DMA transfers in 2.5.67

[Chuck Ebbert]

Andrew Morton wrote:


> > Chuck Ebbert <76306.1226@compuserve.com> wrote:
> > >
> > > # mount /ext3_fs
> > > # time dd if=/ext3_fs/100MiB_file of=/dev/null bs=32k
> > > 
> > >   2.4.20aa1 : 3.3 sec (exactly what I expect to see)
> > >   2.5.66    : 6.6 sec
> > 
> > With this test 2.4 will leave a lot more unwritten dirty data in memory.
> > 
> > You should include a `sync' in the timings.
> 
> Well you should include the sync if you're writing to disk ;)


  :)

  All is not beer and skittles here with Andrea's kernel, though.
Sometimes instead of 31MB/sec I get this with 1 sequential stream:


 1  0  0      0   1232   1120  47988   0   0 14928     0 3834  7534   1  44  55
 1  0  0      0   1372   1120  47848   0   0 14704     0 3778  7446   0  32  68
 1  0  0      0   1464   1064  47816   0   0 14880     0 3822  7501   1  43  56
 1  0  0      0   1336   1064  47944   0   0 14844     0 3813  7493   0  29  71
 1  0  0      0   1432   1064  47848   0   0 14748    32 3800  7467   0  41  59
 1  0  0      0   1532   1064  47748   0   0 13976     0 3596  7045   1  33  66
 


  Pretty high context switch and interrupt rates for a PPro 200, huh?

  And I can't reliably reproduce it (so far...)

------
 Chuck