Network freezing for paravirt linux, 3.0.3

Network freezing for paravirt linux, 3.0.3

[Andres Lagar Cavilla]

Hi,
as part of My Xen-GL work I've been experiencing some temporary domU 
network outages.

At a very basic level I'm using the TCP/IP stack to transport OpenGL 
command from the domU to dom0, where they can be directly executed on 
some nice piece of graphics hardware. Now, beasts like Enemy Territory 
or Unreal Tournament 2004 seem to exceed a gigabit of bandwidth 
utilization. All those bots and rich textures don't help much :)

At points in the game domU networking completely freezes, becoming 
unresponsive to ssh, ping, the openGL transport, and other services I 
tried. Other simptoms:
    - Tcpdump reveals that the only traffic on the vifX.Y are inbound 
broadcasts.
    - There are no dropped packets (as shown by ifconfig and 
/proc/net/dev).
    - Happens with both transmit checksumming offload enabled and disabled.
    - Measured from both dom0 and domU.
    - Both with network bridging and routing.
    - Did not happen for an hvm doing network emulation (did happen for 
PV-on-HVM network driver).
    - For whatever it's worth, doesn't happen with vmware network 
emulation either.

It thus looks like the culprit is netfront, or possibly a 
netback/netfront interaction (Q: is netback employed in the qemu-dm 
emulation path, or packets are simply relayed from the tap to vifX.Y?).
I would love to provide more information, but I'm unsure where to get it.

Andres

RE: [Xen-devel] Network freezing for paravirt linux, 3.0.3

[Ian Pratt]

> At points in the game domU networking completely freezes, becoming
> unresponsive to ssh, ping, the openGL transport, and other services I
> tried. Other simptoms:
>     - Tcpdump reveals that the only traffic on the vifX.Y are inbound
> broadcasts.
>     - There are no dropped packets (as shown by ifconfig and
> /proc/net/dev).
>     - Happens with both transmit checksumming offload enabled and
disabled.
>     - Measured from both dom0 and domU.
>     - Both with network bridging and routing.
>     - Did not happen for an hvm doing network emulation (did happen
for
> PV-on-HVM network driver).
>     - For whatever it's worth, doesn't happen with vmware network
> emulation either.

Do you have dom0 directly on the bridge or via a netloop device as per
the normal scripts (i.e. you have both a peth0 and eth0)? Can you repro
on xen-unstable?

Ian

Re: Network freezing for paravirt linux, 3.0.3

[Keir Fraser]

On 28/11/06 15:08, "Andres Lagar Cavilla" <andreslc@cs.toronto.edu> wrote:

> It thus looks like the culprit is netfront, or possibly a
> netback/netfront interaction (Q: is netback employed in the qemu-dm
> emulation path, or packets are simply relayed from the tap to vifX.Y?).
> I would love to provide more information, but I'm unsure where to get it.

What version of netback are you running? The one from the 3.0.3 release is
known to have a bug which would cause high-bandwidth transfers to stall.

The offending function is netback.c:tx_add_credit(). You should be able to
take the version from xen-unstable and copy it into whatever version of
netback.c you are using.

 -- Keir

Re: [Xen-devel] Network freezing for paravirt linux, 3.0.3

[Andres Lagar Cavilla]

Patching the tx_add_credit function worked like a charm.
Thanks a lot!

Andres

Keir Fraser wrote:

>On 28/11/06 15:08, "Andres Lagar Cavilla" <andreslc@cs.toronto.edu> wrote:
>
>  
>
>>It thus looks like the culprit is netfront, or possibly a
>>netback/netfront interaction (Q: is netback employed in the qemu-dm
>>emulation path, or packets are simply relayed from the tap to vifX.Y?).
>>I would love to provide more information, but I'm unsure where to get it.
>>    
>>
>
>What version of netback are you running? The one from the 3.0.3 release is
>known to have a bug which would cause high-bandwidth transfers to stall.
>
>The offending function is netback.c:tx_add_credit(). You should be able to
>take the version from xen-unstable and copy it into whatever version of
>netback.c you are using.
>
> -- Keir
>
>  
>

Re: Network freezing for paravirt linux, 3.0.3

[Andres Lagar Cavilla]

As a coda to this, I found that the network routing scripts did not work 
too well, at least for me. dom0's interface would not respond to ARP 
requests from external hosts trying to find out the MAC address 
corresponding to domU's ip. Simply allowing proxy arp on dom0's 
externally visible interface solved this. Patch below.
Andres

# HG changeset patch
# User andres@plaxico.syslab.sandbox
# Date 1164747038 18000
# Node ID b2440d035bffb61004e0bab09f50d740a2347a91
# Parent  bbcaa0cad3d2b7cf90e935bf1dd61aec129b3252
Fix for network routing setup. Enables proxy arp on dom0's externally 
visble interface.
Signed-ff-by: H. Andres Lagar-Cavilla andreslc@cs.toronto.edu

diff -r bbcaa0cad3d2 -r b2440d035bff tools/examples/network-route
--- a/tools/examples/network-route      Tue Nov 28 13:34:15 2006 +0000
+++ b/tools/examples/network-route      Tue Nov 28 15:50:38 2006 -0500
@@ -16,4 +16,12 @@
 #
 #============================================================================

+dir=$(dirname "$0")
+. "$dir/xen-script-common.sh"
+
+evalVariables "$@"
+
+netdev=${netdev:-eth${vifnum}}
+
 echo 1 >/proc/sys/net/ipv4/ip_forward
+echo 1 >/proc/sys/net/ipv4/conf/${netdev}/proxy_arp

Keir Fraser wrote:

>On 28/11/06 15:08, "Andres Lagar Cavilla" <andreslc@cs.toronto.edu> wrote:
>
>  
>
>>It thus looks like the culprit is netfront, or possibly a
>>netback/netfront interaction (Q: is netback employed in the qemu-dm
>>emulation path, or packets are simply relayed from the tap to vifX.Y?).
>>I would love to provide more information, but I'm unsure where to get it.
>>    
>>
>
>What version of netback are you running? The one from the 3.0.3 release is
>known to have a bug which would cause high-bandwidth transfers to stall.
>
>The offending function is netback.c:tx_add_credit(). You should be able to
>take the version from xen-unstable and copy it into whatever version of
>netback.c you are using.
>
> -- Keir
>
>  
>