Dom0 crash at 40Mbps Iperf traffic with only 80% CPU utilization ??

[Kaushik Bhandankar <kaushikb@cc.gatech.edu>]

Hello

I have a 3 node experiment as detailed below to estimate the bridged 
networking performance in para-virtualized Xen 3.0 on Emulab 
(https://www.emulab.net/)

Here is how the 3-node topology looks (This topology is specified by 
means of an NS-2 file)

topology:

                  _______________________________
                 |                               |
Node0:eth0 ------|Node1:eth0           Node1:eth1|------ Node2:eth0
                 |_______________________________|

Node0 and Node2 run some standard 2.6.* kernel whereas Node1 runs 
para-virtualized Xen 3.0 using LVM-created root & swap partitions for 
DomU (2.6.12-xenU)

*******************************************************************
Traffic flow:

             _____________________________________
            |                Node1               |
            |   ______________________________   |
            |  |            DomU              |  |
            |  |                              |  |
            |  |    eth0 ---------> eth1      |  |
            |  |_____|_______________|________|  |
            |        |               |           |
            |      vif1.0          vif1.1        |
            |        |               |           |
            |        |               |           |
            |      xenbr0          xenbr1        |
            |        |      Dom0     |           |
            |        |               |           |
Node0:eth0--|----> eth0            eth1 ---------|-----> Node2:eth0
            |____________________________________|


Node0:eth0, Node1:DomU:eth0 are in 10.1.1/24 subnet whereas 
Node1:DomU:eth1 and Node2:eth0 are in 10.1.2/24 subnet (Since Node1:eth0 
and Node1:eth1 are attached to bridges, they do not have IP addresses on 
them and are not ping'able)

At Node0:eth0, I set the next hop of all traffic as Node1:DomU:eth0 and 
at Node2, I set the next hop of all traffic as Node1:DomU:eth1.

Now, I try to run an Iperf server at Node2 for UDP traffic (const 
64-byte packets size):
iperf -s -i 2 -l 64 -u

and an Iperf client at Node0 for UDP traffic to node2:eth0 (const 64-bit 
packet size, vary the Bandwidth from 10Mbps till possibly 80Mbps)
iperf -c <IP address of Node2:eth0> -u -l 64 -t 90 -i 2 -w 256k -b 10M

Where 10M is the bandwidth of iperf traffic between Node0 and Node2 (in 
Mbps).

I also monitor the CPU usage at Node1 Dom0 using "vmstat 1" at Dom0.

When B/W is 36.6M (Mbps), the CPU utilization at Dom0 is about 70-80% 
but when the B/W is increased to 39.4M (Mbps), the Dom0 crashes in 
typically 5-15 seconds (M SSH session to Dom0 'hangs') but before it 
crashes, the CPU utiliation is till 75-85%. At this point, I have to 
again boot Dom0 and the DomU on top of it.

Does anybody know why this happens ?? and how can I debug this situation 
(if at all) ? I am new to the Xen source code so please be as detailed 
as possible