Jeremy's GIT-tree and network problems

Jeremy's GIT-tree and network problems

[Stephan Seitz]

[-- Attachment #1.1: Type: text/plain, Size: 3822 bytes --]

Hi!

I have bridging problems with the Dom0 kernels from Jeremy’s tree.  
I wrote a mail to xen-user (MSG-ID 
<20091220T1944.GA.ab998.stse@fsing.rootsland.net>, 20 Dec 2009), but 
without solutions. So I try xen-devel this time.


My hardware setup:
A PC with two NICs (Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI 
Express Gigabit Ethernet controller) is used as working environment 
(Dom0) and as firewall/proxy/DSL-router (DomU).
The two NICs are bridged between Dom0 and DomU.
Bridge eth0 containing peth0 and vif1.0 has an IP address in Dom0 and 
DomU. The DomU IP address is the gateway address in Dom0.
Bridge xenbr1 containing eth1 and vif1.1 has no IP address in Dom0 and 
DomU and is only used to connect the DSL modem to DomU. The IP address is 
given to the PPP interface in DomU.
Linux distribution is Debian/Testing (64bit) with XEN version 3.4.2 in 
December and 3.4.3rc3 now. The kernels are always self-compiled.


My working setup:
Dom0 with kernel 2.6.29.5 with xen-patches-2.6.29-6.tar.bz2 and DomU with 
standard kernel 2.6.32.x (and the 2.6.29.5 xen kernel before). The 
hypervisor was 3.4.2 and is now 3.4.3rc3.
Here everything works as expected. DomU acts as firewall and is using 
correct masquerading for all internet traffic.


My non-working setup:
Dom0 with the PV-Ops kernel from Jeremy’s tree (I tried the follwoing 
kernels: 2.6.31.5-00500-g34013be, 2.6.31.6-00696-g41a0695 (tested in 
December) and now from xen/stable the versions 2.6.32.10-02792-gf112549 
and 2.6.32.10-02798-gd945b01). DomU kernel and hypervisor are the same as 
in the working setup.

What is working?
IP connection between Dom0 and DomU is working and between DomU and the 
internet. Traffic from Dom0 to the internet is working if DomU is used as 
a proxy (e.g. HTTP traffice with a squid in DomU).

What is not working?
Direct IP connection between Dom0 and the internet (tested with ping and 
„telnet <host> <port>”.
If I trace in DomU I see the packets leaving the ppp0 interface 
(correctly masqueraded), but I see no answering packets.
If I trace in Dom0 using the bridge interfaces between the DSL modem and 
DomU (xenbr1, eth1, vif1.1, see hardware setup above), I don’t see the 
packets anymore. I only see packets from traffic generated directly by 
DomU.
The DomU configuration between the working and non-working setup is not 
changed, only the Dom0 kernel is changed.


So if anyone has an idea, what this could be and how to fix it, I will be 
glad.


Further information:
The NIC and the bridge driver are the same in all kernels from 2.6.29.5 
until 2.6.32.10:

osgiliath:~# ethtool -i eth1
driver: r8169
version: 2.3LK-NAPI
firmware-version:
bus-info: 0000:03:00.0
osgiliath:~# ethtool -i xenbr1
driver: bridge
version: 2.3
firmware-version: N/A
bus-info: N/A

The only difference in the output of „ethtool eth1” are additional 
information about „link partner advertised modes” in the 2.6.3x kernels.

„ethtool -k eth1” shows the error message „Cannot get device flags: 
Operation not supported” in the working setup for the working Dom0 
kernel. All other output is identical in all kernel versions:

osgiliath:~# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off

Switching rx-checksumming off does not help.

dmesg and the log files don’t show error messages in the non-working 
setup. 

Shade and sweet water!

	Stephan

-- 
| Stephan Seitz             E-Mail: stse@fsing.rootsland.net |
| PGP Public Keys: http://fsing.rootsland.net/~stse/pgp.html |

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Jeremy's GIT-tree and network problems

[S.H. Verbrugge]

On Mon, Mar 29, 2010 at 05:40:23PM +0200, Stephan Seitz wrote:
> Hi!
> 
> I have bridging problems with the Dom0 kernels from Jeremy???s tree.
> I wrote a mail to xen-user (MSG-ID
> <20091220T1944.GA.ab998.stse@fsing.rootsland.net>, 20 Dec 2009), but
> without solutions. So I try xen-devel this time.
> 
> 
> My hardware setup:
> A PC with two NICs (Realtek Semiconductor Co., Ltd. RTL8111/8168B
> PCI Express Gigabit Ethernet controller) is used as working
> environment (Dom0) and as firewall/proxy/DSL-router (DomU).
> The two NICs are bridged between Dom0 and DomU.
> Bridge eth0 containing peth0 and vif1.0 has an IP address in Dom0
> and DomU. The DomU IP address is the gateway address in Dom0.
> Bridge xenbr1 containing eth1 and vif1.1 has no IP address in Dom0
> and DomU and is only used to connect the DSL modem to DomU. The IP
> address is given to the PPP interface in DomU.
> Linux distribution is Debian/Testing (64bit) with XEN version 3.4.2
> in December and 3.4.3rc3 now. The kernels are always self-compiled.
> 
> 
> My working setup:
> Dom0 with kernel 2.6.29.5 with xen-patches-2.6.29-6.tar.bz2 and DomU
> with standard kernel 2.6.32.x (and the 2.6.29.5 xen kernel before).
> The hypervisor was 3.4.2 and is now 3.4.3rc3.
> Here everything works as expected. DomU acts as firewall and is
> using correct masquerading for all internet traffic.
> 
> 
> My non-working setup:
> Dom0 with the PV-Ops kernel from Jeremy???s tree (I tried the
> follwoing kernels: 2.6.31.5-00500-g34013be, 2.6.31.6-00696-g41a0695
> (tested in December) and now from xen/stable the versions
> 2.6.32.10-02792-gf112549 and 2.6.32.10-02798-gd945b01). DomU kernel
> and hypervisor are the same as in the working setup.
> 
> What is working?
> IP connection between Dom0 and DomU is working and between DomU and
> the internet. Traffic from Dom0 to the internet is working if DomU
> is used as a proxy (e.g. HTTP traffice with a squid in DomU).
> 
> What is not working?
> Direct IP connection between Dom0 and the internet (tested with ping
> and ???telnet <host> <port>???.
> If I trace in DomU I see the packets leaving the ppp0 interface
> (correctly masqueraded), but I see no answering packets.
> If I trace in Dom0 using the bridge interfaces between the DSL modem
> and DomU (xenbr1, eth1, vif1.1, see hardware setup above), I don???t
> see the packets anymore. I only see packets from traffic generated
> directly by DomU.
> The DomU configuration between the working and non-working setup is
> not changed, only the Dom0 kernel is changed.
> 
> 
> So if anyone has an idea, what this could be and how to fix it, I
> will be glad.
> 
> 
> Further information:
> The NIC and the bridge driver are the same in all kernels from
> 2.6.29.5 until 2.6.32.10:
> 
> osgiliath:~# ethtool -i eth1
> driver: r8169
> version: 2.3LK-NAPI
> firmware-version:
> bus-info: 0000:03:00.0
> osgiliath:~# ethtool -i xenbr1
> driver: bridge
> version: 2.3
> firmware-version: N/A
> bus-info: N/A
> 
> The only difference in the output of ???ethtool eth1??? are
> additional information about ???link partner advertised modes??? in
> the 2.6.3x kernels.
> 
> ???ethtool -k eth1??? shows the error message ???Cannot get device
> flags: Operation not supported??? in the working setup for the
> working Dom0 kernel. All other output is identical in all kernel
> versions:
> 
> osgiliath:~# ethtool -k eth1
> Offload parameters for eth1:
> rx-checksumming: on
> tx-checksumming: off
> scatter-gather: off
> tcp-segmentation-offload: off
> udp-fragmentation-offload: off
> generic-segmentation-offload: off
> generic-receive-offload: off
> large-receive-offload: off
> 
> Switching rx-checksumming off does not help.
> 
> dmesg and the log files don???t show error messages in the
> non-working setup.
> 
> Shade and sweet water!
> 
> 	Stephan
> 
> -- 
> | Stephan Seitz             E-Mail: stse@fsing.rootsland.net |
> | PGP Public Keys: http://fsing.rootsland.net/~stse/pgp.html |

Hello :-)

I seem to have the exact same problem.

Could you check your dmesg (dom0) to see if you have similar messages?

http://lists.xensource.com/archives/html/xen-devel/2010-03/msg00966.html

-- 
 /\/\  Hostingvereniging Soleus              |  Community-driven
< ** > http://soleus.nu                      |  Virtual Private Servers
 \/\/  Sen (IEF) Verbrugge (CT ProLead)      |  & more ...

Re: Jeremy's GIT-tree and network problems

[Jeremy Fitzhardinge]

On 03/29/2010 08:40 AM, Stephan Seitz wrote:
> Hi!
>
> I have bridging problems with the Dom0 kernels from Jeremy’s tree. I 
> wrote a mail to xen-user (MSG-ID 
> <20091220T1944.GA.ab998.stse@fsing.rootsland.net>, 20 Dec 2009), but 
> without solutions. So I try xen-devel this time.
>
>
> My hardware setup:
> A PC with two NICs (Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI 
> Express Gigabit Ethernet controller) is used as working environment 
> (Dom0) and as firewall/proxy/DSL-router (DomU).
> The two NICs are bridged between Dom0 and DomU.
> Bridge eth0 containing peth0 and vif1.0 has an IP address in Dom0 and 
> DomU. The DomU IP address is the gateway address in Dom0.
> Bridge xenbr1 containing eth1 and vif1.1 has no IP address in Dom0 and 
> DomU and is only used to connect the DSL modem to DomU. The IP address 
> is given to the PPP interface in DomU.
> Linux distribution is Debian/Testing (64bit) with XEN version 3.4.2 in 
> December and 3.4.3rc3 now. The kernels are always self-compiled.
>
>
> My working setup:
> Dom0 with kernel 2.6.29.5 with xen-patches-2.6.29-6.tar.bz2 and DomU 
> with standard kernel 2.6.32.x (and the 2.6.29.5 xen kernel before). 
> The hypervisor was 3.4.2 and is now 3.4.3rc3.
> Here everything works as expected. DomU acts as firewall and is using 
> correct masquerading for all internet traffic.
>
>
> My non-working setup:
> Dom0 with the PV-Ops kernel from Jeremy’s tree (I tried the follwoing 
> kernels: 2.6.31.5-00500-g34013be, 2.6.31.6-00696-g41a0695 (tested in 
> December) and now from xen/stable the versions 
> 2.6.32.10-02792-gf112549 and 2.6.32.10-02798-gd945b01). DomU kernel 
> and hypervisor are the same as in the working setup.
>
> What is working?
> IP connection between Dom0 and DomU is working and between DomU and 
> the internet. Traffic from Dom0 to the internet is working if DomU is 
> used as a proxy (e.g. HTTP traffice with a squid in DomU).
>
> What is not working?
> Direct IP connection between Dom0 and the internet (tested with ping 
> and „telnet <host> <port>”.
> If I trace in DomU I see the packets leaving the ppp0 interface 
> (correctly masqueraded), but I see no answering packets.
> If I trace in Dom0 using the bridge interfaces between the DSL modem 
> and DomU (xenbr1, eth1, vif1.1, see hardware setup above), I don’t see 
> the packets anymore. I only see packets from traffic generated 
> directly by DomU.
> The DomU configuration between the working and non-working setup is 
> not changed, only the Dom0 kernel is changed.
>
>
> So if anyone has an idea, what this could be and how to fix it, I will 
> be glad.
>
>
> Further information:
> The NIC and the bridge driver are the same in all kernels from 
> 2.6.29.5 until 2.6.32.10:
>
> osgiliath:~# ethtool -i eth1
> driver: r8169
> version: 2.3LK-NAPI
> firmware-version:
> bus-info: 0000:03:00.0
> osgiliath:~# ethtool -i xenbr1
> driver: bridge
> version: 2.3
> firmware-version: N/A
> bus-info: N/A
>
> The only difference in the output of „ethtool eth1” are additional 
> information about „link partner advertised modes” in the 2.6.3x kernels.
>
> „ethtool -k eth1” shows the error message „Cannot get device flags: 
> Operation not supported” in the working setup for the working Dom0 
> kernel. All other output is identical in all kernel versions:
>
> osgiliath:~# ethtool -k eth1
> Offload parameters for eth1:
> rx-checksumming: on
> tx-checksumming: off
> scatter-gather: off
> tcp-segmentation-offload: off
> udp-fragmentation-offload: off
> generic-segmentation-offload: off
> generic-receive-offload: off
> large-receive-offload: off
>
> Switching rx-checksumming off does not help.

Have you tried carpet-bombing the ethtools: turn off everything on all 
the dom0 interfaces (both the bridge(s) and all the component 
interfaces) and all the domU interfaces? It does look like some kind of 
checksum problem (or perhaps other offload?).

Fortunately it looks like this is going to get some systematic 
attention. I'd really like any reasonable (ie, not inherently broken for 
other reasons) network setup to just work.

Thanks,
J

Re: Jeremy's GIT-tree and network problems

[Stephan Seitz]

[-- Attachment #1.1: Type: text/plain, Size: 625 bytes --]

On Mon, Mar 29, 2010 at 08:18:15PM +0200, S.H. Verbrugge wrote:
>Could you check your dmesg (dom0) to see if you have similar messages?
>http://lists.xensource.com/archives/html/xen-devel/2010-03/msg00966.html

You mean „Attempting to checksum a non-TCP/UDP packet, dropping 
a protocol 1 packet”? Yes, I see one shortly before the NFSv4 server is 
started. But it is only one line and doesn’t show again if I start my 
connection tests.

Shade and sweet water!

	Stephan

-- 
| Stephan Seitz             E-Mail: stse@fsing.rootsland.net |
| PGP Public Keys: http://fsing.rootsland.net/~stse/pgp.html |

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Jeremy's GIT-tree and network problems

[Stephan Seitz]

[-- Attachment #1.1: Type: text/plain, Size: 1368 bytes --]

Hi Jeremy!

On Mon, Mar 29, 2010 at 11:42:40AM -0700, Jeremy Fitzhardinge wrote:
>Have you tried carpet-bombing the ethtools: turn off everything on all 
>the dom0 interfaces (both the bridge(s) and all the component 
>interfaces) and all the domU interfaces? It does look like some kind of 
>checksum problem (or perhaps other offload?).

Well, the physical interfaces only had RX on. The virtual interfaces in 
Dom0 and DomU had RX, TX, SG, TSO and GSO on. Trying to switch off RX on 
the virtual interfaces produced an error message, but after switching off 
the other features, RX was off as well.

But after everything was off (in Dom0 bridge eth0 with peth0 and vif1.0 
and bridge xenbr1 with eth1 and vif1.1 and in DomU eth0 and eth1), the 
setup didn’t start working. So if this is some kind of checksum problem, 
switching the checksum and offload features off don’t change it.

Do you have other suggestions?

>Fortunately it looks like this is going to get some systematic 
>attention. I'd really like any reasonable (ie, not inherently broken for 
>other reasons) network setup to just work.

I hope so. Anf if there are patches to test, I will test it.

Shade and sweet water!

	Stephan

-- 
| Stephan Seitz             E-Mail: stse@fsing.rootsland.net |
| PGP Public Keys: http://fsing.rootsland.net/~stse/pgp.html |

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: Jeremy's GIT-tree and network problems

[Konrad Rzeszutek Wilk]

On Tue, Mar 30, 2010 at 02:26:00PM +0200, Stephan Seitz wrote:
> On Mon, Mar 29, 2010 at 08:18:15PM +0200, S.H. Verbrugge wrote:
>> Could you check your dmesg (dom0) to see if you have similar messages?
>> http://lists.xensource.com/archives/html/xen-devel/2010-03/msg00966.html
>
> You mean „Attempting to checksum a non-TCP/UDP packet, dropping a 
> protocol 1 packet”? Yes, I see one shortly before the NFSv4 server is  
> started. But it is only one line and doesn’t show again if I start my  
> connection tests.


So here is an interesting observation (And I am late in this thread so
if I am going off on a tangent, please correct me): We compiled both in
DomU and Dom0 the kernel to not have any traces of iptables nor
igbtables. The performance of scp-ing a tarball from Dom0 to DomU went
from 0.5MB to 17MB/s (which is about the disk speed).

I don't know if you feel inclined to actually try such a drastic option
to see if that is your culprit or not, but it is worth keeping in the
back of your head.

Re: Jeremy's GIT-tree and network problems

[Stephan Seitz]

[-- Attachment #1.1: Type: text/plain, Size: 998 bytes --]

On Tue, Mar 30, 2010 at 10:30:06AM -0400, Konrad Rzeszutek Wilk wrote:
>So here is an interesting observation (And I am late in this thread so
>if I am going off on a tangent, please correct me): We compiled both in
>DomU and Dom0 the kernel to not have any traces of iptables nor
>igbtables. The performance of scp-ing a tarball from Dom0 to DomU went
>from 0.5MB to 17MB/s (which is about the disk speed).

Interesting. Copying a kernel.tar.bz2 from Dom0 to DomU with scp gives 
a speed of 30.7MB/s, even if both kernels have iptables support.

>I don't know if you feel inclined to actually try such a drastic option
>to see if that is your culprit or not, but it is worth keeping in the
>back of your head.

But I fear, that this is not my problem, and I need iptables in DomU at 
least for masquerading.

Shade and sweet water!

	Stephan

-- 
| Stephan Seitz             E-Mail: stse@fsing.rootsland.net |
| PGP Public Keys: http://fsing.rootsland.net/~stse/pgp.html |

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Jeremy's GIT-tree and network problems

[Stephan Seitz]

[-- Attachment #1.1: Type: text/plain, Size: 651 bytes --]

On Mon, Mar 29, 2010 at 11:42:40AM -0700, Jeremy Fitzhardinge wrote:
>Fortunately it looks like this is going to get some systematic 
>attention. I'd really like any reasonable (ie, not inherently broken for 
>other reasons) network setup to just work.

Do you know what is the status of this problem

At the moment I’m using 2.6.33.2 with xen-patches-2.6.33-1.tar.bz2 and 
everything is working fine. So if you think, the bug is fixed in your 
tree, I’ll try it.

Shade and sweet water!

	Stephan

-- 
| Stephan Seitz             E-Mail: stse@fsing.rootsland.net |
| PGP Public Keys: http://fsing.rootsland.net/~stse/pgp.html |

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel