network hang trigger

network hang trigger

[James Harper]

I have found that simply doing a ping with packetsize > 1500 (maybe
mtu???) causes the network to hang for a short time. This is from xenU
and xen0 on the same machine.

Can someone else _please_ test this? I am able to make the network hang
by saying from domU:
ping -s 1473 <dom0 ip>
(1473 + 28 byte header = 1501 byte packet)

I'll do more testing tomorrow.

thanks

James


-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m

RE: network hang trigger

[James Harper]

Lowering the MTU introduces the problem at a lower packetsize as
expected, so I'd suspect the problem is fragmentation.

I did a test running tcpdumps in both domains, and when I do a ping from
domU to dom0, I see the correct packets on domU, but dom0 only gets the
first fragment then nothing.

James

> -----Original Message-----
> From: xen-devel-admin@lists.sourceforge.net [mailto:xen-devel-
> admin@lists.sourceforge.net] On Behalf Of James Harper
> Sent: Wednesday, 15 September 2004 17:24
> To: xen-devel@lists.sourceforge.net
> Subject: [Xen-devel] network hang trigger
> 
> I have found that simply doing a ping with packetsize > 1500 (maybe
> mtu???) causes the network to hang for a short time. This is from xenU
> and xen0 on the same machine.
> 
> Can someone else _please_ test this? I am able to make the network
hang
> by saying from domU:
> ping -s 1473 <dom0 ip>
> (1473 + 28 byte header = 1501 byte packet)
> 
> I'll do more testing tomorrow.
> 
> thanks
> 
> James
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
> Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
> Camcorder. More prizes in the weekly Lunch Hour Challenge.
> Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m

Re: network hang trigger

[Peri Hankey]

James

I can confirm that your problem occurs in xen-2.0-20040914. In this 
example a4.local is the xen0 machine (2.6.8.1-xen0), a30 is one of 
several xenU guests on a4  (all 2.6.8.1-xenU). First a simple ping, 
which works OK. Then your special poison ping: nasty pause, and possibly 
useful messages.

    [administrator@a30 administrator]$ ping a4.local
    PING a4.local (192.168.0.4) 56(84) bytes of data.
    64 bytes from a4.local (192.168.0.4): icmp_seq=1 ttl=64 time=5.72 ms
    64 bytes from a4.local (192.168.0.4): icmp_seq=2 ttl=64 time=0.150 ms

    --- a4.local ping statistics ---
    2 packets transmitted, 2 received, 0% packet loss, time 1010ms
    rtt min/avg/max/mdev = 0.150/2.937/5.724/2.787 ms
    [administrator@a30 administrator]$ ping -s 1473 a4.local
    PING a4.local (192.168.0.4) 1473(1501) bytes of data.

    ping: sendmsg: No buffer space available
    ping: sendmsg: No buffer space available
    ping: sendmsg: No buffer space available
    ping: sendmsg: No buffer space available
    ping: sendmsg: No buffer space available
    ping: sendmsg: No buffer space available
    ping: sendmsg: No buffer space available
     From 192.168.0.4 icmp_seq=1 Frag reassembly time exceeded
    1481 bytes from 192.168.0.4: icmp_seq=2 ttl=64 time=28980 ms
    1481 bytes from 192.168.0.4: icmp_seq=3 ttl=64 time=27980 ms
    1481 bytes from a4.local (192.168.0.4): icmp_seq=4 ttl=64 time=26980 ms

    --- a4.local ping statistics ---
    11 packets transmitted, 3 received, +1 errors, 72% packet loss, time
    29287ms
    rtt min/avg/max/mdev = 26980.569/27980.589/28980.612/816.525 ms, pipe 11

I hope that helps.
 
Peri

James Harper wrote:

>I have found that simply doing a ping with packetsize > 1500 (maybe
>mtu???) causes the network to hang for a short time. This is from xenU
>and xen0 on the same machine.
>
>Can someone else _please_ test this? I am able to make the network hang
>by saying from domU:
>ping -s 1473 <dom0 ip>
>(1473 + 28 byte header = 1501 byte packet)
>
>I'll do more testing tomorrow.
>
>thanks
>
>James
>
>
>-------------------------------------------------------
>This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
>Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
>Camcorder. More prizes in the weekly Lunch Hour Challenge.
>Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/xen-devel
>
>
>  
>



-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m

Re: network hang trigger

[Keir Fraser]

I can't reproduce this. e.g.,
 [root@druid-0 root]# ping -s 2473 128.232.39.40
 PING 128.232.39.40 (128.232.39.40) 2473(2501) bytes of data.
 2481 bytes from 128.232.39.40: icmp_seq=0 ttl=64 time=0.074 ms
 2481 bytes from 128.232.39.40: icmp_seq=1 ttl=64 time=0.057 ms
 2481 bytes from 128.232.39.40: icmp_seq=2 ttl=64 time=0.055 ms
 2481 bytes from 128.232.39.40: icmp_seq=3 ttl=64 time=0.057 ms

Both domains have 256MB memory. MTU for both is 1500.

 -- Keir

> I have found that simply doing a ping with packetsize > 1500 (maybe
> mtu???) causes the network to hang for a short time. This is from xenU
> and xen0 on the same machine.
> 
> Can someone else _please_ test this? I am able to make the network hang
> by saying from domU:
> ping -s 1473 <dom0 ip>
> (1473 + 28 byte header = 1501 byte packet)
> 
> I'll do more testing tomorrow.
> 
> thanks
> 
> James
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
> Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
> Camcorder. More prizes in the weekly Lunch Hour Challenge.
> Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
\x1f -=- MIME -=- \x1f\f

I have found that simply doing a ping with packetsize > 1500 (maybe
mtu???) causes the network to hang for a short time. This is from xenU
and xen0 on the same machine.

Can someone else _please_ test this? I am able to make the network hang
by saying from domU:
ping -s 1473 <dom0 ip>
(1473 + 28 byte header =3D 1501 byte packet)

I'll do more testing tomorrow.

thanks

James


-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m

Re: network hang trigger

[Chris Andrews]

I'm seeing this problem:

81.187.70.17 is a secondary on DOM0's bridge. The host 'uml18' is DOM1.

uml18:~# ping -s 2473 81.187.70.17
PING 81.187.70.17 (81.187.70.17) 2473(2501) bytes of data.
2481 bytes from 81.187.70.17: icmp_seq=1 ttl=64 time=0.559 ms

.. time passes, I hit ^C ..

ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
2481 bytes from 81.187.70.17: icmp_seq=6 ttl=64 time=30297 ms
--- 81.187.70.17 ping statistics ---
12 packets transmitted, 2 received, 83% packet loss, time 34627ms
rtt min/avg/max/mdev = 0.559/15149.215/30297.872/15148.657 ms, pipe 7

DOM0 has 384M, DOM1 has 64M, MTU 1500 in both.

Different packet sizes seem to trigger the bug more or less quickly -- I 
first tried with 2000 byte packets, and didn't see the problem, then I tried 
2473, and saw the problem then. Trying 2000 bytes again did then show the 
problem.

This is running Xen-2.0 from 4th September -- I can give a more recent 
version a try, but should I be able to turn on writable pagetables with 
current code, given I had problems with that option with the code I'm 
running now?

Chris.


Keir Fraser wrote:
> I can't reproduce this. e.g.,
>  [root@druid-0 root]# ping -s 2473 128.232.39.40
>  PING 128.232.39.40 (128.232.39.40) 2473(2501) bytes of data.
>  2481 bytes from 128.232.39.40: icmp_seq=0 ttl=64 time=0.074 ms
>  2481 bytes from 128.232.39.40: icmp_seq=1 ttl=64 time=0.057 ms
>  2481 bytes from 128.232.39.40: icmp_seq=2 ttl=64 time=0.055 ms
>  2481 bytes from 128.232.39.40: icmp_seq=3 ttl=64 time=0.057 ms
> 
> Both domains have 256MB memory. MTU for both is 1500.
> 
>  -- Keir
> 
> 
>>I have found that simply doing a ping with packetsize > 1500 (maybe
>>mtu???) causes the network to hang for a short time. This is from xenU
>>and xen0 on the same machine.
>>
>>Can someone else _please_ test this? I am able to make the network hang
>>by saying from domU:
>>ping -s 1473 <dom0 ip>
>>(1473 + 28 byte header = 1501 byte packet)
>>
>>I'll do more testing tomorrow.
>>
>>thanks
>>
>>James
>>
>>
>>-------------------------------------------------------
>>This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
>>Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
>>Camcorder. More prizes in the weekly Lunch Hour Challenge.
>>Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
>>_______________________________________________
>>Xen-devel mailing list
>>Xen-devel@lists.sourceforge.net
>>https://lists.sourceforge.net/lists/listinfo/xen-devel
> 
> \x1f -=- MIME -=- \x1f\f

> I have found that simply doing a ping with packetsize > 1500 (maybe
> mtu???) causes the network to hang for a short time. This is from xenU
> and xen0 on the same machine.
> 
> Can someone else _please_ test this? I am able to make the network hang
> by saying from domU:
> ping -s 1473 <dom0 ip>
> (1473 + 28 byte header =3D 1501 byte packet)
> 
> I'll do more testing tomorrow.
> 
> thanks
> 
> James
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
> Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
> Camcorder. More prizes in the weekly Lunch Hour Challenge.
> Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
> Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
> Camcorder. More prizes in the weekly Lunch Hour Challenge.
> Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
> 



-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m

Re: network hang trigger

[Keir Fraser]

Do you see this if you ping from DOM1 to an external machine? Or from
an external machine to DOM1?

My suspicion is that this is just some weird interaction between
bridging and local packet dleivery in DOM0. I also can cause lots of
packet loss when pinging DOM1 from DOM0, if I use DOM1's hostname
rather than IP address(!!!). It's very bizarre, and tcpdump tells me
that the ICMP packets come back to DOM0 via the VIF, but it looks like
they then get swallowed somehow within the bridge.

I think that the bridging code is really intended for dedicated bridge
boxes with no local IP interface. So I'm not sure whether the support
for that might not be flaky in some circumstances.

 -- Keir

> I'm seeing this problem:
> 
> 81.187.70.17 is a secondary on DOM0's bridge. The host 'uml18' is DOM1.
> 
> uml18:~# ping -s 2473 81.187.70.17
> PING 81.187.70.17 (81.187.70.17) 2473(2501) bytes of data.
> 2481 bytes from 81.187.70.17: icmp_seq=1 ttl=64 time=0.559 ms
> 
> .. time passes, I hit ^C ..
> 
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> 2481 bytes from 81.187.70.17: icmp_seq=6 ttl=64 time=30297 ms
> --- 81.187.70.17 ping statistics ---
> 12 packets transmitted, 2 received, 83% packet loss, time 34627ms
> rtt min/avg/max/mdev = 0.559/15149.215/30297.872/15148.657 ms, pipe 7
> 
> DOM0 has 384M, DOM1 has 64M, MTU 1500 in both.
> 
> Different packet sizes seem to trigger the bug more or less quickly -- I 
> first tried with 2000 byte packets, and didn't see the problem, then I tried 
> 2473, and saw the problem then. Trying 2000 bytes again did then show the 
> problem.
> 
> This is running Xen-2.0 from 4th September -- I can give a more recent 
> version a try, but should I be able to turn on writable pagetables with 
> current code, given I had problems with that option with the code I'm 
> running now?


-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m

Re: network hang trigger

[Chris Andrews]

Keir Fraser wrote:
> Do you see this if you ping from DOM1 to an external machine? Or from
> an external machine to DOM1?

Both these cases seem fine.

I also see the 'packet loss using hostname' weirdness pinging from 0 to 1, 
yet tcpdump in dom1 tells me that icmp packets are going both ways...

I *also* see more icmp redirects than I might have expected, so maybe I'll 
try changing things around a bit -- the xenU domains are using a secondary 
on dom0's bridge device as their default as they're not on the same ip 
address range as dom0, so I could try routing instead of bridging.

(wish I could get bidirectional serial console to dom0 working ... some odd 
interaction with the Dell console-redirect-foo, I suspect.)


Chris.






-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m

Re: network hang trigger

[Keir Fraser]

> Keir Fraser wrote:
> > Do you see this if you ping from DOM1 to an external machine? Or from
> > an external machine to DOM1?
> 
> Both these cases seem fine.
> 
> I also see the 'packet loss using hostname' weirdness pinging from 0 to 1, 
> yet tcpdump in dom1 tells me that icmp packets are going both ways...

Seems this only makes it easier to cause packet loss. Sometimes I can
get the same behaviour using DOM1's IP address directly.

There is defintiely some weird bridging thing going on here...

 -- Keir

> I *also* see more icmp redirects than I might have expected, so maybe I'll 
> try changing things around a bit -- the xenU domains are using a secondary 
> on dom0's bridge device as their default as they're not on the same ip 
> address range as dom0, so I could try routing instead of bridging.
> 
> (wish I could get bidirectional serial console to dom0 working ... some odd 
> interaction with the Dell console-redirect-foo, I suspect.)
> 
> 
> Chris.
> 
> 
> 
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
> Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
> Camcorder. More prizes in the weekly Lunch Hour Challenge.
> Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m

RE: network hang trigger

[James Harper]

Hmmmm... are you running 2.6 and pinging from xenU to xen0?

I've read your other emails, and I can still get it to fail if I remove
vif1.0 from the bridge and give it its own ip address, but to properly
exclude bridging I guess I'd have to boot up the computer with bridging
never having been started...

I'm running code checked out and compiled yesterday (15th AEST), and
have writable pagetables on but only since that last compile. It fails
either way. I have no iptables/conntrack related modules loaded in
either domain.

I think we'll have to start comparing .config's, but I need to get some
sleep.

James

> -----Original Message-----
> From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
> Sent: Wednesday, 15 September 2004 19:55
> To: James Harper
> Cc: xen-devel@lists.sourceforge.net
> Subject: Re: [Xen-devel] network hang trigger
> 
> 
> I can't reproduce this. e.g.,
>  [root@druid-0 root]# ping -s 2473 128.232.39.40
>  PING 128.232.39.40 (128.232.39.40) 2473(2501) bytes of data.
>  2481 bytes from 128.232.39.40: icmp_seq=0 ttl=64 time=0.074 ms
>  2481 bytes from 128.232.39.40: icmp_seq=1 ttl=64 time=0.057 ms
>  2481 bytes from 128.232.39.40: icmp_seq=2 ttl=64 time=0.055 ms
>  2481 bytes from 128.232.39.40: icmp_seq=3 ttl=64 time=0.057 ms
> 
> Both domains have 256MB memory. MTU for both is 1500.
> 
>  -- Keir
> 
> > I have found that simply doing a ping with packetsize > 1500 (maybe
> > mtu???) causes the network to hang for a short time. This is from
xenU
> > and xen0 on the same machine.
> >
> > Can someone else _please_ test this? I am able to make the network
hang
> > by saying from domU:
> > ping -s 1473 <dom0 ip>
> > (1473 + 28 byte header = 1501 byte packet)
> >
> > I'll do more testing tomorrow.
> >
> > thanks
> >
> > James
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
> > Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
> > Camcorder. More prizes in the weekly Lunch Hour Challenge.
> > Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel
>  -=- MIME -=- 
> 
> 
> I have found that simply doing a ping with packetsize > 1500 (maybe
> mtu???) causes the network to hang for a short time. This is from xenU
> and xen0 on the same machine.
> 
> Can someone else _please_ test this? I am able to make the network
hang
> by saying from domU:
> ping -s 1473 <dom0 ip>
> (1473 + 28 byte header =3D 1501 byte packet)
> 
> I'll do more testing tomorrow.
> 
> thanks
> 
> James
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
> Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
> Camcorder. More prizes in the weekly Lunch Hour Challenge.
> Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
> 



-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m

Re: network hang trigger

[Keir Fraser]

Okay, I can reproduce this too.

DOM0:
 brctl delif xen-br0 vif1.0
 ifconfig vif1.0 10.0.0.1 up

DOM1:
 ifconfig eth0 10.0.0.2 up

Then if I ping 10.0.0.2 from DOM0 it appears to stop getting responses
after a few round trips, although tcpdump says that responses are
still getting received.

If I ping 10.0.0.1 from DOM1 then it just seems to seize up
occasionally. I can get the same buffer error messages as you if I set
a packet size > MTU.

Currently I have no idea what this could be due to. :-)

 -- Keir

> Hmmmm... are you running 2.6 and pinging from xenU to xen0?
> 
> I've read your other emails, and I can still get it to fail if I remove
> vif1.0 from the bridge and give it its own ip address, but to properly
> exclude bridging I guess I'd have to boot up the computer with bridging
> never having been started...
> 
> I'm running code checked out and compiled yesterday (15th AEST), and
> have writable pagetables on but only since that last compile. It fails
> either way. I have no iptables/conntrack related modules loaded in
> either domain.
> 
> I think we'll have to start comparing .config's, but I need to get some
> sleep.
> 
> James


-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m

RE: network hang trigger

[Charles Coffing]

I was able to reproduce the hang easily.  "ping -s 1473" worked, but
"ping -s 1499" hung.  While it was hung, I tried pinging the other
direction and that hung too.

My setup:
Pinging from DOMU to DOM0
Changeset 1.1307
DOM0: 2.6.8.1, Stock configuration
DOMU: 2.6.8.1, Stock, except writable pagetables are disabled

ccoffing2:~ # ping -s 1499 137.65.171.60
PING 137.65.171.60 (137.65.171.60) 1499(1527) bytes of data.
1507 bytes from 137.65.171.60: icmp_seq=1 ttl=64 time=0.455 ms
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
>From 137.65.171.60: icmp_seq=2 Frag reassembly time exceeded
>From 137.65.171.60 icmp_seq=2 Frag reassembly time exceeded
1507 bytes from 137.65.171.60: icmp_seq=3 ttl=64 time=28980 ms
1507 bytes from 137.65.171.60: icmp_seq=4 ttl=64 time=27980 ms
1507 bytes from 137.65.171.60: icmp_seq=5 ttl=64 time=26980 ms

--- 137.65.171.60 ping statistics ---
15 packets transmitted, 4 received, +2 errors, 73% packet loss, time
33213ms
rtt min/avg/max/mdev = 0.455/20985.849/28980.992/12136.540 ms, pipe 11





>>> "James Harper" <JamesH@bendigoit.com.au> 09/15/04 8:43 AM >>>
Hmmmm... are you running 2.6 and pinging from xenU to xen0?

I've read your other emails, and I can still get it to fail if I remove
vif1.0 from the bridge and give it its own ip address, but to properly
exclude bridging I guess I'd have to boot up the computer with bridging
never having been started...

I'm running code checked out and compiled yesterday (15th AEST), and
have writable pagetables on but only since that last compile. It fails
either way. I have no iptables/conntrack related modules loaded in
either domain.

I think we'll have to start comparing .config's, but I need to get some
sleep.

James

> -----Original Message-----
> From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
> Sent: Wednesday, 15 September 2004 19:55
> To: James Harper
> Cc: xen-devel@lists.sourceforge.net
> Subject: Re: [Xen-devel] network hang trigger
> 
> 
> I can't reproduce this. e.g.,
>  [root@druid-0 root]# ping -s 2473 128.232.39.40
>  PING 128.232.39.40 (128.232.39.40) 2473(2501) bytes of data.
>  2481 bytes from 128.232.39.40: icmp_seq=0 ttl=64 time=0.074 ms
>  2481 bytes from 128.232.39.40: icmp_seq=1 ttl=64 time=0.057 ms
>  2481 bytes from 128.232.39.40: icmp_seq=2 ttl=64 time=0.055 ms
>  2481 bytes from 128.232.39.40: icmp_seq=3 ttl=64 time=0.057 ms
> 
> Both domains have 256MB memory. MTU for both is 1500.
> 
>  -- Keir
> 
> > I have found that simply doing a ping with packetsize > 1500 (maybe
> > mtu???) causes the network to hang for a short time. This is from
xenU
> > and xen0 on the same machine.
> >
> > Can someone else _please_ test this? I am able to make the network
hang
> > by saying from domU:
> > ping -s 1473 <dom0 ip>
> > (1473 + 28 byte header = 1501 byte packet)
> >
> > I'll do more testing tomorrow.
> >
> > thanks
> >
> > James
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
> > Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
> > Camcorder. More prizes in the weekly Lunch Hour Challenge.
> > Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel
>  -=- MIME -=- 
> 
> 
> I have found that simply doing a ping with packetsize > 1500 (maybe
> mtu???) causes the network to hang for a short time. This is from xenU
> and xen0 on the same machine.
> 
> Can someone else _please_ test this? I am able to make the network
hang
> by saying from domU:
> ping -s 1473 <dom0 ip>
> (1473 + 28 byte header =3D 1501 byte packet)
> 
> I'll do more testing tomorrow.
> 
> thanks
> 
> James
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
> Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
> Camcorder. More prizes in the weekly Lunch Hour Challenge.
> Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
> 



-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m

Re: network hang trigger

[Bin Ren]

After some investigation, I seem to be pin down the cause as
skbuff_head_cache overflow. do 'cat /proc/slabinfo | grep skbuff', the
first column is the number of active skbuff_head. On my machine, it's
like this:

(1) after normal 'ping dom0', skbuff_head_cache active is 210
(2) after 'ping -s 3000 dom0' and network hang, it shoots up to 254
(3) further 'ping -s 3000 dom0' and it shoots up to 340. Get a few
replies. Entire network hang.
(4) wait for several minutes, skbuff_head_cache gets
garbage-collected, drops down to 120. Network recovers.

This problem also occurs to linux, freebsd, netbsd kernels with faulty
device drivers. It's likely that network frontend driver is not
freeing skbuff properly. Skbuff cache gets overflowed until some
threshold that it's gc'ed by force.

I'm taking a look at the netback source codes.

-- Bin Ren

On Wed, 15 Sep 2004 09:29:46 -0600, Charles Coffing <ccoffing@novell.com> wrote:
> I was able to reproduce the hang easily.  "ping -s 1473" worked, but
> "ping -s 1499" hung.  While it was hung, I tried pinging the other
> direction and that hung too.
> 
> My setup:
> Pinging from DOMU to DOM0
> Changeset 1.1307
> DOM0: 2.6.8.1, Stock configuration
> DOMU: 2.6.8.1, Stock, except writable pagetables are disabled
> 
> ccoffing2:~ # ping -s 1499 137.65.171.60
> PING 137.65.171.60 (137.65.171.60) 1499(1527) bytes of data.
> 1507 bytes from 137.65.171.60: icmp_seq=1 ttl=64 time=0.455 ms
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> From 137.65.171.60: icmp_seq=2 Frag reassembly time exceeded
> From 137.65.171.60 icmp_seq=2 Frag reassembly time exceeded
> 1507 bytes from 137.65.171.60: icmp_seq=3 ttl=64 time=28980 ms
> 1507 bytes from 137.65.171.60: icmp_seq=4 ttl=64 time=27980 ms
> 1507 bytes from 137.65.171.60: icmp_seq=5 ttl=64 time=26980 ms
> 
> --- 137.65.171.60 ping statistics ---
> 15 packets transmitted, 4 received, +2 errors, 73% packet loss, time
> 33213ms
> rtt min/avg/max/mdev = 0.455/20985.849/28980.992/12136.540 ms, pipe 11


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: network hang trigger

[Bin Ren]

After some investigation, I seem to be pin down the cause as
skbuff_head_cache overflow. do 'cat /proc/slabinfo | grep skbuff', the
first column is the number of active skbuff_head. On my machine, it's
like this:

(1) after normal 'ping dom0', skbuff_head_cache active is 210
(2) after 'ping -s 3000 dom0' and network hang, it shoots up to 254
(3) further 'ping -s 3000 dom0' and it shoots up to 340. Get a few
replies. Entire network hang.
(4) wait for several minutes, skbuff_head_cache gets
garbage-collected, drops down to 120. Network recovers.

This problem also occurs to linux, freebsd, netbsd kernels with faulty
device drivers. It's likely that network frontend driver is not
freeing skbuff properly. Skbuff cache gets overflowed until some
threshold that it's gc'ed by force.

I'm taking a look at the netfront/back source codes.

-- Bin Ren
- Hide quoted text -

On Wed, 15 Sep 2004 09:29:46 -0600, Charles Coffing <ccoffing@novell.com> 
wrote:
> I was able to reproduce the hang easily.  "ping -s 1473" worked, but
> "ping -s 1499" hung.  While it was hung, I tried pinging the other
> direction and that hung too.
>
> My setup:
> Pinging from DOMU to DOM0
> Changeset 1.1307
> DOM0: 2.6.8.1, Stock configuration
> DOMU: 2.6.8.1, Stock, except writable pagetables are disabled
>
> ccoffing2:~ # ping -s 1499 137.65.171.60
> PING 137.65.171.60 (137.65.171.60) 1499(1527) bytes of data.
> 1507 bytes from 137.65.171.60: icmp_seq=1 ttl=64 time=0.455 ms
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> From 137.65.171.60: icmp_seq=2 Frag reassembly time exceeded
> From 137.65.171.60 icmp_seq=2 Frag reassembly time exceeded
> 1507 bytes from 137.65.171.60: icmp_seq=3 ttl=64 time=28980 ms
> 1507 bytes from 137.65.171.60: icmp_seq=4 ttl=64 time=27980 ms
> 1507 bytes from 137.65.171.60: icmp_seq=5 ttl=64 time=26980 ms
>
> --- 137.65.171.60 ping statistics ---
> 15 packets transmitted, 4 received, +2 errors, 73% packet loss, time
> 33213ms
> rtt min/avg/max/mdev = 0.455/20985.849/28980.992/12136.540 ms, pipe 11


-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m

Re: network hang trigger

[Bin Ren]

Here is what I find out:

In netfront.c, the transmit function is:

network_start_xmit(struct sk_buff *skb, struct net_device *dev)

Currently, whenever there is an error, it returns 1 or -ENOBUFS 
***WITHOUT*** freeing the ***skb***. This is based on the assumption that 
the caller, seeing a non-zero return value, will free the ***skb***. Let's 
take a look at the caller:

int dev_queue_xmit(struct sk_buff *skb)

in file net/core.c
======================================
if (!dev->hard_start_xmit(skb, dev)) {
        HARD_TX_UNLOCK_BH(dev);
        goto out;
}
...

out_kfree_skb:
        kfree_skb(skb);
out:
        return rc;
======================================

Bingo, it doesn't. And take a look at other network driver source codes, 
e.g. 8139too.c, 3c501.c etc, they all ***free skb*** upon error and 
***always return 0***. This is *hidden contract* between the caller and the 
callee.

So, skbuffs don't get freed until gc'ed.

I'm going to modify the files and see the result.
Keep tuned.

-- Bin Ren


-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m

RE: network hang trigger

[James Harper]

I see this problem after 1 ping of size > MTU (which probably explains
why NFS hangs too!). If I understand correctly, the situation you
describe should only come about if an error occurs for other reasons.

So do we still have another outstanding bug?

Thanks

James

> -----Original Message-----
> From: xen-devel-admin@lists.sourceforge.net [mailto:xen-devel-
> admin@lists.sourceforge.net] On Behalf Of Bin Ren
> Sent: Thursday, 16 September 2004 08:11
> To: xen-devel@lists.sourceforge.net
> Subject: Re: [Xen-devel] network hang trigger
> 
> Here is what I find out:
> 
> In netfront.c, the transmit function is:
> 
> network_start_xmit(struct sk_buff *skb, struct net_device *dev)
> 
> Currently, whenever there is an error, it returns 1 or -ENOBUFS
> ***WITHOUT*** freeing the ***skb***. This is based on the assumption
that
> the caller, seeing a non-zero return value, will free the ***skb***.
Let's
> take a look at the caller:
> 
> int dev_queue_xmit(struct sk_buff *skb)
> 
> in file net/core.c
> ======================================
> if (!dev->hard_start_xmit(skb, dev)) {
>         HARD_TX_UNLOCK_BH(dev);
>         goto out;
> }
> ...
> 
> out_kfree_skb:
>         kfree_skb(skb);
> out:
>         return rc;
> ======================================
> 
> Bingo, it doesn't. And take a look at other network driver source
codes,
> e.g. 8139too.c, 3c501.c etc, they all ***free skb*** upon error and
> ***always return 0***. This is *hidden contract* between the caller
and
> the
> callee.
> 
> So, skbuffs don't get freed until gc'ed.
> 
> I'm going to modify the files and see the result.
> Keep tuned.
> 
> -- Bin Ren
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
> Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
> Camcorder. More prizes in the weekly Lunch Hour Challenge.
> Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m

Re: network hang trigger

[Bin Ren]

I've just modified the netfront.c (haven't touched netback.c yet). I've 
done two modifications: (1) free sk_buff properly on the transmit path (2) 
In netif_poll(...) function, packets **should not** be passed to 
netif_rx(); instead, use: int netif_receive_skb(struct sk_buff *skb).

With these two modifications, under 'ping -s 6000', network only 
occasionally loses a few packets but *very soon* recovers. It's much more 
stable than before. I'll take a closer look at netfront.c and netback.c 
tomorrow.

Here is the patch. Please try it out. With your results, the changes may 
get pushed into the repository.

-- Bin Ren

===== linux-2.6.8.1-xen-sparse/drivers/xen/netfront/netfront.c 1.49 vs 
edited ===== --- 
1.49/linux-2.6.8.1-xen-sparse/drivers/xen/netfront/netfront.c 2004-08-27 
13:28:33 +01:00 +++ 
edited/linux-2.6.8.1-xen-sparse/drivers/xen/netfront/netfront.c 2004-09-16 
00:34:33 +01:00 @@ -329,7 +329,7 @@
     {
         printk(KERN_ALERT "%s: full queue wasn't stopped!\n", dev->name);
         netif_stop_queue(dev);
-        return -ENOBUFS;
+       goto drop;
     }

     if ( unlikely((((unsigned long)skb->data & ~PAGE_MASK) + skb->len) >=
@@ -337,7 +337,7 @@
     {
         struct sk_buff *new_skb;
         if ( unlikely((new_skb = alloc_skb_page()) == NULL) )
-            return 1;
+            goto drop;
         skb_put(new_skb, skb->len);
         memcpy(new_skb->data, skb->data, skb->len);
         dev_kfree_skb(skb);
@@ -349,7 +349,7 @@
     if ( np->backend_state != BEST_CONNECTED )
     {
         spin_unlock_irq(&np->tx_lock);
-        return 1;
+        goto drop;
     }

     i = np->tx->req_prod;
@@ -385,6 +385,10 @@
         notify_via_evtchn(np->evtchn);

     return 0;
+
+ drop:
+    dev_kfree_skb(skb);
+    return 0;
 }


@@ -501,7 +505,7 @@
         skb->protocol = eth_type_trans(skb, dev);

         /* Pass it up. */
-        netif_rx(skb);
+        netif_receive_skb(skb);
         dev->last_rx = jiffies;
     }


-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m

Re: network hang trigger

[Bin Ren]

Just another note:

Before the patch, ping -s 6000 from dom0 -> domU *and* domU -> dom0 *both* 
make the network hang. Now with the patch, ping -s 6000 from dom0 -> domU 
*never* hangs (or, to be more accurate, successful for the first 127 
packets before Ctrl+C !). but from domU -> dom0 network occassionally loses 
packets.

Sth *asymmetric* is playing the trick. This may shed some light on the 
problem. I have to go to sleep before figuring it out.

-- Bin


-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m

RE: network hang trigger

[James Harper]

This patch makes no difference on my system. Looking at the line numbers
in your patch, your netfront.c appears to be a different vintage to
mine. I applied the patch manually and then rebuilt the xenU kernel and
then booted a domain with it. I haven't touched xen0. Is this the
correct thing to be doing?

Ping >mtu from xenU to xen0 causes the network to hang immediately.
Ping >mtu from xen0 to xenU sometimes causes the network to hang, but
not always. 'ping -i 0.1 -s 6000 <xenU ip>' will mostly cause the hang
in under 30 iterations.

The recovery time appears to be in the order of 60 seconds or so, with a
partial recovery and then relapse at about 30 seconds.
 
When I was thinking about this problem, I was imagining a deadlock
condition where rapid back to back packets (eg a fragmented icmp packet
from ping or a fragmented udp packet from nfs) was causing a hang until
part of the deadlock timed itself out and the packets started flowing
again. I couldn't see 1 packet causing a buffer exhaustion unless it got
itself into a loop where it kept retrying to send the second fragment
and didn't free the buffer each time, but even then the buffer bug would
be a side effect.

The deadlock would have to be caused in the transmit from xenU to xen0,
and something about the difference between sending a ping and responding
to a ping is the difference between always causing a lockup and only
sometimes causing a lockup.

Maybe we're seeing different manifestations of the same problem?

James

> -----Original Message-----
> From: xen-devel-admin@lists.sourceforge.net [mailto:xen-devel-
> admin@lists.sourceforge.net] On Behalf Of Bin Ren
> Sent: Thursday, 16 September 2004 09:54
> To: xen-devel@lists.sourceforge.net
> Subject: Re: [Xen-devel] network hang trigger
> 
> I've just modified the netfront.c (haven't touched netback.c yet).
I've
> done two modifications: (1) free sk_buff properly on the transmit path
(2)
> In netif_poll(...) function, packets **should not** be passed to
> netif_rx(); instead, use: int netif_receive_skb(struct sk_buff *skb).
> 
> With these two modifications, under 'ping -s 6000', network only
> occasionally loses a few packets but *very soon* recovers. It's much
more
> stable than before. I'll take a closer look at netfront.c and
netback.c
> tomorrow.
> 
> Here is the patch. Please try it out. With your results, the changes
may
> get pushed into the repository.
> 


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

RE: network hang trigger

[James Harper]

This lends weight to my idea that it might be a timing related deadlock,
I put some debug statements in to see just how often the drop: path was
called in network_start_xmit (never), and now I almost can't get it to
fail. I put printk's in at the start of the function, and before both
the returns.

This particular machine is not SMP btw. It is a single CPU PII running
at 350Mhz (697 bogomips)

James

> -----Original Message-----
> From: xen-devel-admin@lists.sourceforge.net [mailto:xen-devel-
> admin@lists.sourceforge.net] On Behalf Of James Harper
> Sent: Thursday, 16 September 2004 10:48
> To: Bin Ren; xen-devel@lists.sourceforge.net
> Subject: RE: [Xen-devel] network hang trigger
> 
> This patch makes no difference on my system. Looking at the line
numbers
> in your patch, your netfront.c appears to be a different vintage to
> mine. I applied the patch manually and then rebuilt the xenU kernel
and
> then booted a domain with it. I haven't touched xen0. Is this the
> correct thing to be doing?
> 
> Ping >mtu from xenU to xen0 causes the network to hang immediately.
> Ping >mtu from xen0 to xenU sometimes causes the network to hang, but
> not always. 'ping -i 0.1 -s 6000 <xenU ip>' will mostly cause the hang
> in under 30 iterations.
> 
> The recovery time appears to be in the order of 60 seconds or so, with
a
> partial recovery and then relapse at about 30 seconds.
> 
> When I was thinking about this problem, I was imagining a deadlock
> condition where rapid back to back packets (eg a fragmented icmp
packet
> from ping or a fragmented udp packet from nfs) was causing a hang
until
> part of the deadlock timed itself out and the packets started flowing
> again. I couldn't see 1 packet causing a buffer exhaustion unless it
got
> itself into a loop where it kept retrying to send the second fragment
> and didn't free the buffer each time, but even then the buffer bug
would
> be a side effect.
> 
> The deadlock would have to be caused in the transmit from xenU to
xen0,
> and something about the difference between sending a ping and
responding
> to a ping is the difference between always causing a lockup and only
> sometimes causing a lockup.
> 
> Maybe we're seeing different manifestations of the same problem?
> 
> James
> 
> > -----Original Message-----
> > From: xen-devel-admin@lists.sourceforge.net [mailto:xen-devel-
> > admin@lists.sourceforge.net] On Behalf Of Bin Ren
> > Sent: Thursday, 16 September 2004 09:54
> > To: xen-devel@lists.sourceforge.net
> > Subject: Re: [Xen-devel] network hang trigger
> >
> > I've just modified the netfront.c (haven't touched netback.c yet).
> I've
> > done two modifications: (1) free sk_buff properly on the transmit
path
> (2)
> > In netif_poll(...) function, packets **should not** be passed to
> > netif_rx(); instead, use: int netif_receive_skb(struct sk_buff
*skb).
> >
> > With these two modifications, under 'ping -s 6000', network only
> > occasionally loses a few packets but *very soon* recovers. It's much
> more
> > stable than before. I'll take a closer look at netfront.c and
> netback.c
> > tomorrow.
> >
> > Here is the patch. Please try it out. With your results, the changes
> may
> > get pushed into the repository.
> >
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
> Project Admins to receive an Apple iPod Mini FREE for your judgement
on
> who ports your project to Linux PPC the best. Sponsored by IBM.
> Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: network hang trigger

[Keir Fraser]

> When I was thinking about this problem, I was imagining a deadlock
> condition where rapid back to back packets (eg a fragmented icmp packet
> from ping or a fragmented udp packet from nfs) was causing a hang until
> part of the deadlock timed itself out and the packets started flowing
> again. I couldn't see 1 packet causing a buffer exhaustion unless it got
> itself into a loop where it kept retrying to send the second fragment
> and didn't free the buffer each time, but even then the buffer bug would
> be a side effect.
> 
> The deadlock would have to be caused in the transmit from xenU to xen0,
> and something about the difference between sending a ping and responding
> to a ping is the difference between always causing a lockup and only
> sometimes causing a lockup.

Try tcpdumping each end of teh connecttion.

I find that for ping 0->U, the 'seizure' is entirely within DOM0 --
ping responses are still received, but for some reason they don't make
it up to the ping application.

For ping U->0, it does look as though the network seizes up -- I see
no packets in either direction.

 -- Keir


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: network hang trigger

[Keir Fraser]

> in file net/core.c
> ======================================
> if (!dev->hard_start_xmit(skb, dev)) {
>         HARD_TX_UNLOCK_BH(dev);
>         goto out;
> }
> ...
> 
> out_kfree_skb:
>         kfree_skb(skb);
> out:
>         return rc;
> ======================================
> 
> Bingo, it doesn't. And take a look at other network driver source codes, 
> e.g. 8139too.c, 3c501.c etc, they all ***free skb*** upon error and 
> ***always return 0***. This is *hidden contract* between the caller and the 
> callee.

(1) See drivers like 3c59x.c. They do not free the skbuff when they
    return non-zero.

(2) Look again at the code fragment you've posted -- the body of the
    'if' statement is executed when hard_start_xmit returns zero. So
    the body corresponds to the NON-ERROR case! You'll see that the
    error case does in fact free the skbuff.

> So, skbuffs don't get freed until gc'ed.

No, there is no bug here. And if we were leaking skbuffs they would
never be freed. There is no automatic garbage collection within the
kernel.

 -- Keir


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

RE: network hang trigger

[Bin Ren]

Please use 'tcpdump -nt icmp -vv -i interface' to capture more details.

Now, with the correct NAPI rx call, i cannot produce the hang, which is 
really confusing. I ping -s 6000 from domU to dom0 and get this:

NOTE: The large request is *NOT* fragmented!! The reply is!

IP (tos 0x0, ttl 64, id 35115, offset 0, flags [none], length: 6028) 
192.168.0.101 > 192.168.0.1: icmp 6008: echo request seq 5632 IP (tos 0x0, 
ttl 64, id 24314, offset 0, flags [+], length: 1500) 192.168.0.1 > 
192.168.0.101: icmp 1480: echo reply seq 5632 IP (tos 0x0, ttl 64, id 
24314, offset 1480, flags [+], length: 1500) 192.168.0.1 > 192.168.0.101: 
icmp IP (tos 0x0, ttl 64, id 24314, offset 2960, flags [+], length: 1500) 
192.168.0.1 > 192.168.0.101: icmp IP (tos 0x0, ttl 64, id 24314, offset 
4440, flags [+], length: 1500) 192.168.0.1 > 192.168.0.101: icmp IP (tos 
0x0, ttl 64, id 24314, offset 5920, flags [none], length: 108) 192.168.0.1 
> 192.168.0.101: icmp



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

RE: network hang trigger

[Bin Ren]

Sorry, in the previous post, ping output is messed up. Here it is:

IP (tos 0x0, ttl 64, id 39991, offset 0, flags [none], length: 1528) 
192.168.0.101 > 192.168.0.1: icmp 1508: echo request seq 1024

IP (tos 0x0, ttl 64, id 29480, offset 0, flags [+], length: 1500) 
192.168.0.1 > 192.168.0.101: icmp 1480: echo reply seq 1024

IP (tos 0x0, ttl 64, id 29480, offset 1480, flags [none], length: 48) 
192.168.0.1 > 192.168.0.101: icmp

-- Bin


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

RE: network hang trigger

[Bin Ren]

I can no longer trigger a hang with the latest bkbits. I even use 'ping -f 
-s 6000 dom0' to send hundreds of icmp requests per second, no hang.

The confusing part is that on James' machine, both the big icmp request and 
reply are fragmented. On my machine, no matter how large is the packet, 
icmp request is *NOT* fragmented *without* IP Don't Fragment (DF) flag set! 
And the MTU is 1500.

To make it even more confusing, with normal size (64 bytes in ping dom0), 
DF flag is set!

Could this be a kernel problem? Both of my dom0 and domU are running 2.6.8.1

-- Bin 


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php