xen-netfront does not properly transmit forwarded packets

xen-netfront does not properly transmit forwarded packets

[Rafal Wojtczuk]

Hello,

There is a very weird issue with xen-netfront (I think it is the frontend
problem, not backend). The problem manifests itself with drivers from the SUSE 
kernel-xen-2.6.34.1; I don't know whether it affects vanilla code as well.
For completeness, xen is 3.4.3, all 64bit.

The problem seems to be - xen-netfront does not properly transmit forwarded
packets (locally generated packets are txed fine).

The network looks like this (of course eth0s are xen-netfront) :

testVM             FirewallVM                NetVM
|  eth0  | <---> | vifF.0 eth0 | <---> | vifN.0 wlan0 | <---> Internet

If I do "ping someInternetIP" in FirewallVM, "tcpdump -n -i eth0" 
running in FirewallVM shows outgoing icmp packets, and "tcpdump -n -i vifN.0"
running in NetVM shows incoming packets - all fine.

If I do "ping someInternetIP" in testVM, packets arrive fine on vifF.0 and
are SNATed. Then "tcpdump -n -i eth0" running in FirewallVM shows outgoing icmp 
packets, BUT "tcpdump -n -i vifN.0" running in NetVM shows NOTHING.

The important thing is that during the latter experiment, the /proc/interrupts
line for vifN.0 shows one new interrupt per second - so vifN.0 is notified by
FirewallVM's eth0 about packet transmission, yet packets are not seen by
vifN.0. The TX bytes counter for FirewallVM's eth0 increases normally; no errors
reported by any interface; nothing in the logs.

In case it matters: there is no bridging used at all, just "bare" vifX.Y. Proxy 
arp is activated for both vifs. No IP is assigned to vifs. Turning SNAT off
in FirewallVM does not change anything. The issue has been reproduced by two
different persons on two different machines.

Does anyone have an idea why this is happening ? What is the difference in
frontend's handling of forwarded packets in comparison to locally generated
ones ? Maybe some function does not work properly in interrupt context ?
I guess not many people use netfront in a router machine, so this issue may
have survived unnoticed for a long time.

Regards,
RW

Re: xen-netfront does not properly transmit forwarded packets

[Jean Baptiste Favre]

Hello,
I got a similar issue with PCI passthrough (only with 32bits VM, 64bits
runs fine) with only 1 VM which has control on physical network card
(ie. you NetVM).

What if you try to ping from NetVM ?
I do not see echo reply coming in except for some memory amount.

I'm working on finding root cause with Konrad Rzeszutek Wilk and Ian
Campbell. First mail of the thread can be found here:
http://lists.xensource.com/archives/html/xen-devel/2011-01/msg00866.html

Maybe both issues could be related to each other so that we can join our
efforts to solve it ?

Regards,
JB


Le 28/02/2011 11:18, Rafal Wojtczuk a écrit :
> Hello,
> 
> There is a very weird issue with xen-netfront (I think it is the frontend
> problem, not backend). The problem manifests itself with drivers from the SUSE 
> kernel-xen-2.6.34.1; I don't know whether it affects vanilla code as well.
> For completeness, xen is 3.4.3, all 64bit.
> 
> The problem seems to be - xen-netfront does not properly transmit forwarded
> packets (locally generated packets are txed fine).
> 
> The network looks like this (of course eth0s are xen-netfront) :
> 
> testVM             FirewallVM                NetVM
> |  eth0  | <---> | vifF.0 eth0 | <---> | vifN.0 wlan0 | <---> Internet
> 
> If I do "ping someInternetIP" in FirewallVM, "tcpdump -n -i eth0" 
> running in FirewallVM shows outgoing icmp packets, and "tcpdump -n -i vifN.0"
> running in NetVM shows incoming packets - all fine.
> 
> If I do "ping someInternetIP" in testVM, packets arrive fine on vifF.0 and
> are SNATed. Then "tcpdump -n -i eth0" running in FirewallVM shows outgoing icmp 
> packets, BUT "tcpdump -n -i vifN.0" running in NetVM shows NOTHING.
> 
> The important thing is that during the latter experiment, the /proc/interrupts
> line for vifN.0 shows one new interrupt per second - so vifN.0 is notified by
> FirewallVM's eth0 about packet transmission, yet packets are not seen by
> vifN.0. The TX bytes counter for FirewallVM's eth0 increases normally; no errors
> reported by any interface; nothing in the logs.
> 
> In case it matters: there is no bridging used at all, just "bare" vifX.Y. Proxy 
> arp is activated for both vifs. No IP is assigned to vifs. Turning SNAT off
> in FirewallVM does not change anything. The issue has been reproduced by two
> different persons on two different machines.
> 
> Does anyone have an idea why this is happening ? What is the difference in
> frontend's handling of forwarded packets in comparison to locally generated
> ones ? Maybe some function does not work properly in interrupt context ?
> I guess not many people use netfront in a router machine, so this issue may
> have survived unnoticed for a long time.
> 
> Regards,
> RW
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

Re: xen-netfront does not properly transmit forwarded packets

[Rafal Wojtczuk]

On Mon, Feb 28, 2011 at 11:52:01AM +0100, Jean Baptiste Favre wrote:
> Hello,
> I got a similar issue with PCI passthrough (only with 32bits VM, 64bits
> runs fine) with only 1 VM which has control on physical network card
> (ie. you NetVM).
Frankly, I do not see much similarity here.

> What if you try to ping from NetVM ?
> I do not see echo reply coming in except for some memory amount.
When I ping from NetVM (or from FirewallVM as well) I can see replies from
Internet fine. In fact I am using NetVM on my workhorse laptop for quite a
few months.
In my case, there is no issue with pci passthru - just the link
between the net frontend and backend does not cope well with forwarded packets.

> I'm working on finding root cause with Konrad Rzeszutek Wilk and Ian
> Campbell. First mail of the thread can be found here:
> http://lists.xensource.com/archives/html/xen-devel/2011-01/msg00866.html
> 
> Maybe both issues could be related to each other so that we can join our
> efforts to solve it ?
Once again, this issue looks different.

Regards,
RW

Re: xen-netfront does not properly transmit forwarded packets

[Rafal Wojtczuk]

On Mon, Feb 28, 2011 at 11:18:51AM +0100, Rafal Wojtczuk wrote:
> Hello,
> 
> There is a very weird issue with xen-netfront (I think it is the frontend
> problem, not backend). The problem manifests itself with drivers from the SUSE 
> kernel-xen-2.6.34.1; I don't know whether it affects vanilla code as well.
> For completeness, xen is 3.4.3, all 64bit.
> 
> The problem seems to be - xen-netfront does not properly transmit forwarded
> packets (locally generated packets are txed fine).
> 
> The network looks like this (of course eth0s are xen-netfront) :
> 
> testVM             FirewallVM                NetVM
> |  eth0  | <---> | vifF.0 eth0 | <---> | vifN.0 wlan0 | <---> Internet
> 
> If I do "ping someInternetIP" in FirewallVM, "tcpdump -n -i eth0" 
> running in FirewallVM shows outgoing icmp packets, and "tcpdump -n -i vifN.0"
> running in NetVM shows incoming packets - all fine.
> 
> If I do "ping someInternetIP" in testVM, packets arrive fine on vifF.0 and
> are SNATed. Then "tcpdump -n -i eth0" running in FirewallVM shows outgoing icmp 
> packets, BUT "tcpdump -n -i vifN.0" running in NetVM shows NOTHING.
> 
> The important thing is that during the latter experiment, the /proc/interrupts
> line for vifN.0 shows one new interrupt per second - so vifN.0 is notified by
> FirewallVM's eth0 about packet transmission, yet packets are not seen by
> vifN.0. The TX bytes counter for FirewallVM's eth0 increases normally; no errors
> reported by any interface; nothing in the logs.
> 
> In case it matters: there is no bridging used at all, just "bare" vifX.Y. Proxy 
> arp is activated for both vifs. No IP is assigned to vifs. Turning SNAT off
> in FirewallVM does not change anything. The issue has been reproduced by two
> different persons on two different machines.
> 
> Does anyone have an idea why this is happening ? What is the difference in
> frontend's handling of forwarded packets in comparison to locally generated
> ones ? Maybe some function does not work properly in interrupt context ?
> I guess not many people use netfront in a router machine, so this issue may
> have survived unnoticed for a long time.

One more hint - after turning off scatter-gather on FirewallVM via
ethtool -K eth0 sg off
packets are forwarded fine. Which strongly suggests an issue with the
frontend driver.

RW