From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Apparao, Padmashree K" Subject: PCI-E networking Date: Tue, 10 Oct 2006 23:26:03 -0700 Message-ID: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0302532895==" Return-path: Content-class: urn:content-classes:message List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Mime-version: 1.0 Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. --===============0302532895== Content-class: urn:content-classes:message Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C6ECFE.20D196B1" This is a multi-part message in MIME format. ------_=_NextPart_001_01C6ECFE.20D196B1 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable Hi, =20 I see a weird problem with networking under Xen. I am using 2 dual ports PCI-e nics in my platform and a single PCI-X nic. Each PCI-E nic shares an irq. Each port is on a different subnet. =20 In native linux I can get line rate in each port. In Xen (custome xen-unstable) however, a ping on each subnet is very slooow. A ping takes about 3000ms or even more.=20 =20 In fact I downloaded the Xen 3.0.2-3 testing build of xen and I see the problem there too. Has anyone seen this kind of behavior? Is this related to the interrupt sharing of the ports? =20 Suggestions =20 Thanks - Padma =20 =20 =20 =20 =20 ------_=_NextPart_001_01C6ECFE.20D196B1 Content-Type: text/html; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable

Hi,

 

I see a weird problem with networking under Xen. I am = using 2 dual ports PCI-e nics in my platform and a single PCI-X nic. Each = PCI-E nic shares an irq. Each port is on a different = subnet.

 

In native linux I can get line rate in each = port.

In Xen (custome xen-unstable)  however, a ping = on each subnet is very slooow. A ping takes about 3000ms or even more. =

 

In fact I downloaded the Xen 3.0.2-3 testing build of = xen and I see the problem there too. Has anyone seen this kind of behavior? = Is this related to the interrupt sharing of the = ports?

 

Suggestions

 

Thanks

-          Padma

 

 

 

 

 

------_=_NextPart_001_01C6ECFE.20D196B1-- --===============0302532895== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============0302532895==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "James Harper" Subject: RE: PCI-E networking Date: Wed, 11 Oct 2006 17:00:26 +1000 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Return-path: Content-class: urn:content-classes:message In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "Apparao, Padmashree K" , xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org > In Xen (custome xen-unstable) however, a ping on each subnet is very > slooow. A ping takes about 3000ms or even more. Is that 3000ms for the first reply or every reply? Make sure that you ping the IP address and use the -n option to make sure that DNS is not causing any issues. Then start a ping and do a tcpdump of icmp and arp packets and send the results of the first 3 pings or so here. "tcpdump -I eth0 icmp or arp" should do the trick. James From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Ian Pratt" Subject: RE: PCI-E networking Date: Wed, 11 Oct 2006 10:28:30 +0100 Message-ID: <3AAA99889D105740BE010EB6D5A5A3B20505FA@paddington.ad.cl.cam.ac.uk> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Return-path: Content-class: urn:content-classes:message List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "Apparao, Padmashree K" , xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org > I see a weird problem with networking under Xen. I am using 2 dual ports > PCI-e nics in my platform and a single PCI-X nic. Each PCI-E nic shares an > irq. Each port is on a different subnet. > =20 > In native linux I can get line rate in each port. >=20 > In Xen (custome xen-unstable) however, a ping on each subnet is very > slooow. A ping takes about 3000ms or even more. >=20 > In fact I downloaded the Xen 3.0.2-3 testing build of xen and I see the > problem there too. Has anyone seen this kind of behavior? Is this related > to the interrupt sharing of the ports? Odd. First thing to try is switching to xen-unstable or 3.0.3-testing there have been some interrupt changes, but still no MSI support, I'm afraid. Ian From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Apparao, Padmashree K" Subject: RE: PCI-E networking Date: Wed, 11 Oct 2006 09:56:46 -0700 Message-ID: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1673818241==" Return-path: Content-class: urn:content-classes:message List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Mime-version: 1.0 Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: James Harper , xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. --===============1673818241== Content-class: urn:content-classes:message Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C6ED56.3CDC3847" This is a multi-part message in MIME format. ------_=_NextPart_001_01C6ED56.3CDC3847 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable =20 Thanks. Here is the output. BTW these cards are PCI-MSI, if that makes a difference. Thanks- Padma =20 Ping -n Stats >>>> ping -n 192.168.133.2 PING 192.168.133.2 (192.168.133.2) 56(84) bytes of data. 64 bytes from 192.168.133.2: icmp_seq=3D0 ttl=3D64 time=3D2400 ms 64 bytes from 192.168.133.2: icmp_seq=3D1 ttl=3D64 time=3D1400 ms 64 bytes from 192.168.133.2: icmp_seq=3D15 ttl=3D64 time=3D2410 ms 64 bytes from 192.168.133.2: icmp_seq=3D16 ttl=3D64 time=3D1410 ms 64 bytes from 192.168.133.2: icmp_seq=3D17 ttl=3D64 time=3D410 ms 64 bytes from 192.168.133.2: icmp_seq=3D18 ttl=3D64 time=3D7070 ms 64 bytes from 192.168.133.2: icmp_seq=3D19 ttl=3D64 time=3D6070 ms =20 Ping with tcpdump with icmp =20 >>>>ping 192.168.133.2 PING 192.168.133.2 (192.168.133.2) 56(84) bytes of data. >>From 192.168.133.1 icmp_seq=3D0 Destination Host Unreachable >>From 192.168.133.1 icmp_seq=3D1 Destination Host Unreachable >>From 192.168.133.1 icmp_seq=3D2 Destination Host Unreachable 64 bytes from 192.168.133.2: icmp_seq=3D3 ttl=3D64 time=3D1990 ms 64 bytes from 192.168.133.2: icmp_seq=3D4 ttl=3D64 time=3D990 ms 64 bytes from 192.168.133.2: icmp_seq=3D5 ttl=3D64 time=3D15.6 ms 64 bytes from 192.168.133.2: icmp_seq=3D6 ttl=3D64 time=3D4010 ms 64 bytes from 192.168.133.2: icmp_seq=3D7 ttl=3D64 time=3D3010 ms 64 bytes from 192.168.133.2: icmp_seq=3D8 ttl=3D64 time=3D2010 ms 64 bytes from 192.168.133.2: icmp_seq=3D9 ttl=3D64 time=3D1010 ms 64 bytes from 192.168.133.2: icmp_seq=3D10 ttl=3D64 time=3D10.7 ms --- 192.168.133.2 ping statistics --- 22 packets transmitted, 8 received, +3 errors, 63% packet loss, time 21021ms rtt min/avg/max/mdev =3D 10.743/1631.395/4010.772/1316.715 ms, pipe 6 =20 >>> tcpdump -i eth7 icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth7, link-type EN10MB (Ethernet), capture size 96 bytes 23:43:32.021469 IP 192.168.133.1 > 192.168.133.2: icmp 64: echo request seq3 23:43:32.021473 IP 192.168.133.1 > 192.168.133.2: icmp 64: echo request seq4 23:43:32.021751 IP 192.168.133.2 > 192.168.133.1: icmp 64: echo reply seq 3 23:43:32.021754 IP 192.168.133.2 > 192.168.133.1: icmp 64: echo reply seq 4 23:43:32.030887 IP 192.168.133.1 > 192.168.133.2: icmp 64: echo request seq5 23:43:32.046498 IP 192.168.133.2 > 192.168.133.1: icmp 64: echo reply seq 5 23:43:33.030840 IP 192.168.133.1 > 192.168.133.2: icmp 64: echo request seq6 38 packets captured 38 packets received by filter 0 packets dropped by kernel =20 Ping with tcpdump arp >>> ping 192.168.133.2 PING 192.168.133.2 (192.168.133.2) 56(84) bytes of data. 64 bytes from 192.168.133.2: icmp_seq=3D0 ttl=3D64 time=3D2896 ms 64 bytes from 192.168.133.2: icmp_seq=3D1 ttl=3D64 time=3D1890 ms 64 bytes from 192.168.133.2: icmp_seq=3D2 ttl=3D64 time=3D890 ms 64 bytes from 192.168.133.2: icmp_seq=3D3 ttl=3D64 time=3D28390 m34 = packets transmitted, 32 received, 5% packet loss, time 33015ms rtt min/avg/max/mdev =3D 380.263/13214.810/28390.176/8765.006 ms, pipe = 30s =20 =20 >>> tcpdump -i eth7 arp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth7, link-type EN10MB (Ethernet), capture size 96 bytes 0 packets captured 8 packets received by filter 0 packets dropped by kernel =20 =20 =20 -----Original Message----- From: James Harper [mailto:james.harper@bendigoit.com.au]=20 Sent: Wednesday, October 11, 2006 12:00 AM To: Apparao, Padmashree K; xen-devel@lists.xensource.com Subject: RE: [Xen-devel] PCI-E networking =20 > In Xen (custome xen-unstable) however, a ping on each subnet is very > slooow. A ping takes about 3000ms or even more. =20 Is that 3000ms for the first reply or every reply? =20 Make sure that you ping the IP address and use the -n option to make sure that DNS is not causing any issues. =20 Then start a ping and do a tcpdump of icmp and arp packets and send the results of the first 3 pings or so here. "tcpdump -I eth0 icmp or arp" should do the trick. =20 James ------_=_NextPart_001_01C6ED56.3CDC3847 Content-Type: text/html; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable

 

Thanks.


Here is the output. BTW these cards are PCI-MSI, if that makes a = difference.

Thanks- Padma

 

Ping –n Stats

>>>> ping -n = 192.168.133.2

PING 192.168.133.2 (192.168.133.2) 56(84) bytes of data.

64 bytes from 192.168.133.2: icmp_seq=3D0 = ttl=3D64 time=3D2400 ms

64 bytes from 192.168.133.2: icmp_seq=3D1 = ttl=3D64 time=3D1400 ms

64 bytes from 192.168.133.2: icmp_seq=3D15 = ttl=3D64 time=3D2410 ms

64 bytes from 192.168.133.2: icmp_seq=3D16 = ttl=3D64 time=3D1410 ms

64 bytes from 192.168.133.2: icmp_seq=3D17 = ttl=3D64 time=3D410 ms

64 bytes from 192.168.133.2: icmp_seq=3D18 = ttl=3D64 time=3D7070 ms

64 bytes from 192.168.133.2: icmp_seq=3D19 = ttl=3D64 time=3D6070 ms

 

Ping with tcpdump with = icmp

 

>>>>ping = 192.168.133.2

PING 192.168.133.2 (192.168.133.2) 56(84) bytes of data.

From 192.168.133.1 icmp_seq=3D0 Destination = Host Unreachable

From 192.168.133.1 icmp_seq=3D1 Destination = Host Unreachable

From 192.168.133.1 icmp_seq=3D2 Destination = Host Unreachable

64 bytes from 192.168.133.2: icmp_seq=3D3 = ttl=3D64 time=3D1990 ms

64 bytes from 192.168.133.2: icmp_seq=3D4 = ttl=3D64 time=3D990 ms

64 bytes from 192.168.133.2: icmp_seq=3D5 = ttl=3D64 time=3D15.6 ms

64 bytes from 192.168.133.2: icmp_seq=3D6 = ttl=3D64 time=3D4010 ms

64 bytes from 192.168.133.2: icmp_seq=3D7 = ttl=3D64 time=3D3010 ms

64 bytes from 192.168.133.2: icmp_seq=3D8 = ttl=3D64 time=3D2010 ms

64 bytes from 192.168.133.2: icmp_seq=3D9 = ttl=3D64 time=3D1010 ms

64 bytes from 192.168.133.2: icmp_seq=3D10 = ttl=3D64 time=3D10.7 ms

--- 192.168.133.2 ping statistics = ---

22 packets transmitted, 8 received, +3 = errors, 63% packet loss, time 21021ms

rtt min/avg/max/mdev =3D 10.743/1631.395/4010.772/1316.715 ms, pipe = 6

 

>>> tcpdump -i eth7 = icmp

tcpdump: verbose output suppressed, use -v or = -vv for full protocol decode

listening on eth7, link-type EN10MB = (Ethernet), capture size 96 bytes

23:43:32.021469 IP 192.168.133.1 > = 192.168.133.2: icmp 64: echo request seq3

23:43:32.021473 IP 192.168.133.1 > = 192.168.133.2: icmp 64: echo request seq4

23:43:32.021751 IP 192.168.133.2 > = 192.168.133.1: icmp 64: echo reply seq 3

23:43:32.021754 IP 192.168.133.2 > = 192.168.133.1: icmp 64: echo reply seq 4

23:43:32.030887 IP 192.168.133.1 > = 192.168.133.2: icmp 64: echo request seq5

23:43:32.046498 IP 192.168.133.2 > = 192.168.133.1: icmp 64: echo reply seq 5

23:43:33.030840 IP 192.168.133.1 > = 192.168.133.2: icmp 64: echo request seq6

38 packets = captured

38 packets received by = filter

0 packets dropped by = kernel

 

Ping with tcpdump arp

>>> ping = 192.168.133.2

PING 192.168.133.2 (192.168.133.2) 56(84) bytes of data.

64 bytes from 192.168.133.2: icmp_seq=3D0 = ttl=3D64 time=3D2896 ms

64 bytes from 192.168.133.2: icmp_seq=3D1 = ttl=3D64 time=3D1890 ms

64 bytes from 192.168.133.2: icmp_seq=3D2 = ttl=3D64 time=3D890 ms

64 bytes from 192.168.133.2: icmp_seq=3D3 = ttl=3D64 time=3D28390 m34 packets transmitted, 32 received, 5% packet loss, time = 33015ms

rtt min/avg/max/mdev =3D = 380.263/13214.810/28390.176/8765.006 ms, pipe 30s

 

 

>>> tcpdump -i eth7 = arp

tcpdump: verbose output suppressed, use -v or = -vv for full protocol decode

listening on eth7, link-type EN10MB = (Ethernet), capture size 96 bytes

0 packets = captured

8 packets received by = filter

0 packets dropped by = kernel

 

 

 

-----Original Message-----
From: James Harper [mailto:james.harper@bendigoit.com.au]
Sent: Wednesday, October 11, 2006 12:00 AM
To: Apparao, Padmashree K; xen-devel@lists.xensource.com
Subject: RE: [Xen-devel] PCI-E networking

 

> In Xen (custome xen-unstable)  however, a ping on each = subnet is very

> slooow. A ping takes about 3000ms or even = more.

 

Is that 3000ms for the first reply or every = reply?

 

Make sure that you ping the IP address and use the -n option to = make

sure that DNS is not causing any = issues.

 

Then start a ping and do a tcpdump of icmp and arp packets and = send the

results of the first 3 pings or so here. "tcpdump -I eth0 = icmp or arp"

should do the trick.

 

James

------_=_NextPart_001_01C6ED56.3CDC3847-- --===============1673818241== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============1673818241==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Christopher G. Stach II" Subject: Re: PCI-E networking Date: Wed, 11 Oct 2006 13:22:39 -0500 Message-ID: <452D366F.10307@ldsys.net> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "Apparao, Padmashree K" Cc: James Harper , xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org Apparao, Padmashree K wrote: > > > Thanks. > > > Here is the output. BTW these cards are PCI-MSI, if that makes a difference. > > Thanks- Padma > > * * > 1. Try a crossover directly to the other host. 2. Pay attention to ``netstat -s'' 3. DNS wouldn't affect a ping RTT calculation 4. Try different cables, network hardware, destination host (with a different kind of NIC) -- Christopher G. Stach II