Re: 1000xf bus problem

Re: 1000xf bus problem

[Robert Hancock]

Greg.Chandler@wellsfargo.com wrote:
> I'm running a e1000xf adapter in a 64-bit/100Mhz PCI slot.  The intel
> site shows this is a supported config for the card, but linux is pulling
> this info:
> 
> ed:02.0 Ethernet controller: Intel Corporation 82544EI Gigabit Ethernet
> Controller (Fiber) (rev 02)
> 	Subsystem: Intel Corporation PRO/1000 XF Server Adapter
> 	Flags: bus master, 66Mhz, medium devsel, latency 64, IRQ 18
> 	Memory at f7fe0000 (64-bit, non-prefetchable) [size=128K]
>       Memory at f7fc0000 (64-bit, non-prefetchable) [size=128K]
>       I/O ports at 7000 [size=32]
>       [virtual] Expansion ROM at f10a0000 [disabled] [size=128K]
>       Capabilities: [dc] Power Management version 2
>       Capabilities: [e4] PCI-X non-bridge device
>       Capabilities: [f0] Message Signalled Interrupts: 64bit+ Queue=0/0
> Enable-
> 
> My thoughput tests show it is definitely not running at the 100Mhz bus
> rate is should be capable of.

How are you determining this?

 > Any ideas on how to make it work at full speed?

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: 1000xf bus problem

[Kok, Auke]

Robert Hancock wrote:
> Greg.Chandler@wellsfargo.com wrote:
>> I'm running a e1000xf adapter in a 64-bit/100Mhz PCI slot.  The intel
>> site shows this is a supported config for the card, but linux is pulling
>> this info:
>>
>> ed:02.0 Ethernet controller: Intel Corporation 82544EI Gigabit Ethernet
>> Controller (Fiber) (rev 02)
>> 	Subsystem: Intel Corporation PRO/1000 XF Server Adapter
>> 	Flags: bus master, 66Mhz, medium devsel, latency 64, IRQ 18
>> 	Memory at f7fe0000 (64-bit, non-prefetchable) [size=128K]
>>       Memory at f7fc0000 (64-bit, non-prefetchable) [size=128K]
>>       I/O ports at 7000 [size=32]
>>       [virtual] Expansion ROM at f10a0000 [disabled] [size=128K]
>>       Capabilities: [dc] Power Management version 2
>>       Capabilities: [e4] PCI-X non-bridge device
>>       Capabilities: [f0] Message Signalled Interrupts: 64bit+ Queue=0/0
>> Enable-
>>
>> My thoughput tests show it is definitely not running at the 100Mhz bus
>> rate is should be capable of.
> 
> How are you determining this?
> 
>  > Any ideas on how to make it work at full speed?

what is the dmesh output of e1000 ? it should show you what the card itself 
detects (at least the newer drivers since 7.0.x all do). This may provide some 
clues as to what the card has detected.

Auke

RE: 1000xf bus problem

[Greg.Chandler]

If you mean dmesg it says this:
e1000: 0000:0d:02.0: e1000_probe: (PCI-X:100MHz:64-bit)  {macaddress}

That's weird... dmesg shows one thing, lspci shows another, and my data
transfers seem to point to the lspci info...

Any idea which I should trust?


-----Original Message-----
From: Kok, Auke [mailto:auke-jan.h.kok@intel.com] 
Sent: Sunday, March 18, 2007 3:52 PM
To: Robert Hancock
Cc: Chandler, Greg; linux-kernel; e1000-devel@lists.sourceforge.net
Subject: Re: 1000xf bus problem

Robert Hancock wrote:
> Greg.Chandler@wellsfargo.com wrote:
>> I'm running a e1000xf adapter in a 64-bit/100Mhz PCI slot.  The intel

>> site shows this is a supported config for the card, but linux is 
>> pulling this info:
>>
>> ed:02.0 Ethernet controller: Intel Corporation 82544EI Gigabit 
>> Ethernet Controller (Fiber) (rev 02)
>> 	Subsystem: Intel Corporation PRO/1000 XF Server Adapter
>> 	Flags: bus master, 66Mhz, medium devsel, latency 64, IRQ 18
>> 	Memory at f7fe0000 (64-bit, non-prefetchable) [size=128K]
>>       Memory at f7fc0000 (64-bit, non-prefetchable) [size=128K]
>>       I/O ports at 7000 [size=32]
>>       [virtual] Expansion ROM at f10a0000 [disabled] [size=128K]
>>       Capabilities: [dc] Power Management version 2
>>       Capabilities: [e4] PCI-X non-bridge device
>>       Capabilities: [f0] Message Signalled Interrupts: 64bit+ 
>> Queue=0/0
>> Enable-
>>
>> My thoughput tests show it is definitely not running at the 100Mhz 
>> bus rate is should be capable of.
> 
> How are you determining this?
> 
>  > Any ideas on how to make it work at full speed?

what is the dmesh output of e1000 ? it should show you what the card
itself detects (at least the newer drivers since 7.0.x all do). This may
provide some clues as to what the card has detected.

Auke

Re: 1000xf bus problem

[Kok, Auke]

Greg.Chandler@wellsfargo.com wrote:
> If you mean dmesg it says this:
> e1000: 0000:0d:02.0: e1000_probe: (PCI-X:100MHz:64-bit)  {macaddress}
> 
> That's weird... dmesg shows one thing, lspci shows another, and my data
> transfers seem to point to the lspci info...
> 
> Any idea which I should trust?


Both, the e1000 driver asks the card what it sees from it's side of the 
connection, and lspci tells you what the cpu side of it is connected to.

Since stuff like pci bridges exist, both could very well be correct!

I highly suspect that that is exactly the case.


Auke

RE: 1000xf bus problem

[Greg.Chandler]

That would completely and uttly suck if it were the case.
So in theory, the card should be talking at full speed, but since the
bridge is 66 then it would be the bottleneck right?

There are 3 pci busses in the server I am working on, so I know there is
at least _a_ bridge there.....
 

-----Original Message-----
From: Kok, Auke [mailto:auke-jan.h.kok@intel.com] 
Sent: Sunday, March 18, 2007 10:40 PM
To: Chandler, Greg
Cc: auke-jan.h.kok@intel.com; hancockr@shaw.ca;
linux-kernel@vger.kernel.org; e1000-devel@lists.sourceforge.net
Subject: Re: 1000xf bus problem

Greg.Chandler@wellsfargo.com wrote:
> If you mean dmesg it says this:
> e1000: 0000:0d:02.0: e1000_probe: (PCI-X:100MHz:64-bit)  {macaddress}
> 
> That's weird... dmesg shows one thing, lspci shows another, and my 
> data transfers seem to point to the lspci info...
> 
> Any idea which I should trust?


Both, the e1000 driver asks the card what it sees from it's side of the
connection, and lspci tells you what the cpu side of it is connected to.

Since stuff like pci bridges exist, both could very well be correct!

I highly suspect that that is exactly the case.


Auke

RE: 1000xf bus problem

[Greg.Chandler]

lspci -v shows the message below, and I am moving files between systems,
{from RAMdisk to RAMdisk} on idle machines.
The transfer rate is concurrent with just under the max throughput
capable on a 64-bit/66Mhz PCI socket.
 

-----Original Message-----
From: Robert Hancock [mailto:hancockr@shaw.ca] 
Sent: Sunday, March 18, 2007 2:41 PM
To: Chandler, Greg; linux-kernel
Cc: e1000-devel@lists.sourceforge.net
Subject: Re: 1000xf bus problem

Greg.Chandler@wellsfargo.com wrote:
> I'm running a e1000xf adapter in a 64-bit/100Mhz PCI slot.  The intel 
> site shows this is a supported config for the card, but linux is 
> pulling this info:
> 
> ed:02.0 Ethernet controller: Intel Corporation 82544EI Gigabit 
> Ethernet Controller (Fiber) (rev 02)
> 	Subsystem: Intel Corporation PRO/1000 XF Server Adapter
> 	Flags: bus master, 66Mhz, medium devsel, latency 64, IRQ 18
> 	Memory at f7fe0000 (64-bit, non-prefetchable) [size=128K]
>       Memory at f7fc0000 (64-bit, non-prefetchable) [size=128K]
>       I/O ports at 7000 [size=32]
>       [virtual] Expansion ROM at f10a0000 [disabled] [size=128K]
>       Capabilities: [dc] Power Management version 2
>       Capabilities: [e4] PCI-X non-bridge device
>       Capabilities: [f0] Message Signalled Interrupts: 64bit+ 
> Queue=0/0
> Enable-
> 
> My thoughput tests show it is definitely not running at the 100Mhz bus

> rate is should be capable of.

How are you determining this?

 > Any ideas on how to make it work at full speed?

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca Home Page:
http://www.roberthancock.com/

Re: 1000xf bus problem

[Robert Hancock]

Greg.Chandler@wellsfargo.com wrote:
> lspci -v shows the message below, and I am moving files between systems,
> {from RAMdisk to RAMdisk} on idle machines.
> The transfer rate is concurrent with just under the max throughput
> capable on a 64-bit/66Mhz PCI socket.

I think you miscalculate, that bus can transfer 532 MB/sec, Gigabit 
Ethernet tops out at 125 MB/sec at absolute maximum and it's difficult 
to achieve that in practice.

And what lspci reports is merely that the device supports 66MHz bus 
speed. That doesn't mean it doesn't support higher speeds and that 
doesn't mean that's what it's running at.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: 1000xf bus problem

[Willy Tarreau]

On Sun, Mar 18, 2007 at 11:20:09PM -0600, Robert Hancock wrote:
> Greg.Chandler@wellsfargo.com wrote:
> >lspci -v shows the message below, and I am moving files between systems,
> >{from RAMdisk to RAMdisk} on idle machines.
> >The transfer rate is concurrent with just under the max throughput
> >capable on a 64-bit/66Mhz PCI socket.
> 
> I think you miscalculate, that bus can transfer 532 MB/sec, Gigabit 
> Ethernet tops out at 125 MB/sec at absolute maximum and it's difficult 
> to achieve that in practice.

On TCP payload, you should achieve 118.66 * 10^6 bytes/s ~= 113 MB/s.
This is what you should observe with FTP or netcat for instance. On
local networks, it is perfectly attainable, I do this every day. If you
are far from this, check both sides link status with ethtool, and ensure
that you do not have wiring problems. It is usually very easy to fill the
wire with an e1000.

Also, it would be interesting to check the other side. What card does it
have, what bus, what driver ? And what software or tests are you using
to conclude that you're limited by the bus ?

Regards,
Willy

Re: [E1000-devel] 1000xf bus problem

[Robin Humble]

On Mon, Mar 19, 2007 at 06:31:51AM +0100, Willy Tarreau wrote:
>On Sun, Mar 18, 2007 at 11:20:09PM -0600, Robert Hancock wrote:
>> Greg.Chandler@wellsfargo.com wrote:
>> >lspci -v shows the message below, and I am moving files between systems,
>> >{from RAMdisk to RAMdisk} on idle machines.
>> >The transfer rate is concurrent with just under the max throughput
>> >capable on a 64-bit/66Mhz PCI socket.
>> 
>> I think you miscalculate, that bus can transfer 532 MB/sec, Gigabit 
>> Ethernet tops out at 125 MB/sec at absolute maximum and it's difficult 
>> to achieve that in practice.
>
>On TCP payload, you should achieve 118.66 * 10^6 bytes/s ~= 113 MB/s.

times 2 for full duplex.
so a 32bit/33MHz bus = 132MB/s isn't enough for fdx, but anything more
(64bit or 66MHz or both) should be fine.

cheers,
robin

>This is what you should observe with FTP or netcat for instance. On
>local networks, it is perfectly attainable, I do this every day. If you
>are far from this, check both sides link status with ethtool, and ensure
>that you do not have wiring problems. It is usually very easy to fill the
>wire with an e1000.
>
>Also, it would be interesting to check the other side. What card does it
>have, what bus, what driver ? And what software or tests are you using
>to conclude that you're limited by the bus ?
>
>Regards,
>Willy
>
>
>-------------------------------------------------------------------------
>Take Surveys. Earn Cash. Influence the Future of IT
>Join SourceForge.net's Techsay panel and you'll get the chance to share your
>opinions on IT & business topics through brief surveys-and earn cash
>http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
>_______________________________________________
>E1000-devel mailing list
>E1000-devel@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/e1000-devel

Re: [E1000-devel] 1000xf bus problem

[Willy Tarreau]

On Mon, Mar 19, 2007 at 01:59:32AM -0400, Robin Humble wrote:
> On Mon, Mar 19, 2007 at 06:31:51AM +0100, Willy Tarreau wrote:
> >On Sun, Mar 18, 2007 at 11:20:09PM -0600, Robert Hancock wrote:
> >> Greg.Chandler@wellsfargo.com wrote:
> >> >lspci -v shows the message below, and I am moving files between systems,
> >> >{from RAMdisk to RAMdisk} on idle machines.
> >> >The transfer rate is concurrent with just under the max throughput
> >> >capable on a 64-bit/66Mhz PCI socket.
> >> 
> >> I think you miscalculate, that bus can transfer 532 MB/sec, Gigabit 
> >> Ethernet tops out at 125 MB/sec at absolute maximum and it's difficult 
> >> to achieve that in practice.
> >
> >On TCP payload, you should achieve 118.66 * 10^6 bytes/s ~= 113 MB/s.
> 
> times 2 for full duplex.

perfectly, I was talking about something easy to measure (ftp...)

> so a 32bit/33MHz bus = 132MB/s isn't enough for fdx, but anything more
> (64bit or 66MHz or both) should be fine.

Not even, in fact, you have to subtract the PCI overhead. It is very hard to
go beyond 800 Mbps half duplex on PCI/33/32bits, so if you have either 64bits
or 66 MHz, you won't go past 1.6 Gbps FD, or a little less than 200 MB/s.

66*64 is really required to simultaneously achieve 1 Gbps in both directions.

Regards,
Willy