pv_ops routing problem?

pv_ops routing problem?

[Michael D Labriola]

Ok, I've got a problem.  My current setup is as follows:

I have a pv_ops dom0 (xen/master from a few weeks ago) running 3 domUs. 
Each domU has an eth0 interface and one or more VLAN interfaces.

vm1, vm2, and vm3 are all domUs running with the same pv_ops kernel as 
dom0.  vm1's eth0.107 interface routes through vm3 to get to vm2's 
eth0.1020 interface.

dom0:
  eth0:     192.168.1.20/24

vm1:
  eth0:     192.168.1.10/24
  eth0.107: 10.7.1.1/16
  gateway:  10.7.0.1

vm2:
  eth0:      192.168.1.11/24
  eth0.1020: 10.20.1.1/16
  gateway:   10.20.0.1

vm3:
  eth0:      192.168.1.12/24
  eth0.107:  10.7.0.1/16
  eth0.1020: 10.20.0.1/16


>From vm1, I can ping the gateway (10.7.0.1) and vm3's 1020 interface 
(10.20.0.1), but get no response if I try to ping vm2's 10.20.1.1.

vm3 can ping both 10.7.1.1 and 10.20.1.1.

Should be a pretty simple packet forwarding example, and it used to work 
using the old xen-linux 2.6.18 kernel we were using on our deployed 
systems.  But now it's broken...

I have verified that the exact same setup running on 3 physical PCs 
bare-metal works w/out any problems.  I had the problem initially on Xen 
3.4.2, but just upgraded to 4.0.0 and have the exact same symptoms.

Anyone have any ideas?  Could this be related to the "using --physdev-out 
in the OUTPUT, FORWARD, and POSTROUTING chains for non-bridged traffic is 
not supported anymore" messages I get when I start a domU?

Thanks!

-Mike

---
Michael D Labriola
Electric Boat
mlabriol@gdeb.com
401-848-8871 (desk)
401-848-8513 (lab)
401-316-9844 (cell)

Re: pv_ops routing problem?

[Jeremy Fitzhardinge]

On 04/27/2010 02:22 PM, Michael D Labriola wrote:
> Ok, I've got a problem.  My current setup is as follows:
>
> I have a pv_ops dom0 (xen/master from a few weeks ago) running 3 domUs. 
> Each domU has an eth0 interface and one or more VLAN interfaces.
>
> vm1, vm2, and vm3 are all domUs running with the same pv_ops kernel as 
> dom0.  vm1's eth0.107 interface routes through vm3 to get to vm2's 
> eth0.1020 interface.
>
> dom0:
>   eth0:     192.168.1.20/24
>
> vm1:
>   eth0:     192.168.1.10/24
>   eth0.107: 10.7.1.1/16
>   gateway:  10.7.0.1
>
> vm2:
>   eth0:      192.168.1.11/24
>   eth0.1020: 10.20.1.1/16
>   gateway:   10.20.0.1
>
> vm3:
>   eth0:      192.168.1.12/24
>   eth0.107:  10.7.0.1/16
>   eth0.1020: 10.20.0.1/16
>
>
> >From vm1, I can ping the gateway (10.7.0.1) and vm3's 1020 interface 
> (10.20.0.1), but get no response if I try to ping vm2's 10.20.1.1.
>
> vm3 can ping both 10.7.1.1 and 10.20.1.1.
>
> Should be a pretty simple packet forwarding example, and it used to work 
> using the old xen-linux 2.6.18 kernel we were using on our deployed 
> systems.  But now it's broken...
>
> I have verified that the exact same setup running on 3 physical PCs 
> bare-metal works w/out any problems.  I had the problem initially on Xen 
> 3.4.2, but just upgraded to 4.0.0 and have the exact same symptoms.
>   

Have you tried turning off tx checksum offload?  (ethtool -K <dev> tx off)

Though for me that just fixes network access between dom0 and domU;
external traffic normally works fine.

    J

Re: pv_ops routing problem?

[Michael D Labriola]

xen-devel-bounces@lists.xensource.com wrote on 04/27/2010 06:02:12 PM:

> Jeremy Fitzhardinge <jeremy@goop.org> 
> Sent by: xen-devel-bounces@lists.xensource.com
> 
> 04/27/2010 06:02 PM
> 
> To
> 
> Michael D Labriola <mlabriol@gdeb.com>
> 
> cc
> 
> Jayson A Dyke <jdyke@gdeb.com>, xen-devel@lists.xensource.com
> 
> Subject
> 
> Re: [Xen-devel] pv_ops routing problem?
> 
> On 04/27/2010 02:22 PM, Michael D Labriola wrote:
> > Ok, I've got a problem.  My current setup is as follows:
> >
> > I have a pv_ops dom0 (xen/master from a few weeks ago) running 3 
domUs. 
> > Each domU has an eth0 interface and one or more VLAN interfaces.
> >
> > vm1, vm2, and vm3 are all domUs running with the same pv_ops kernel as 

> > dom0.  vm1's eth0.107 interface routes through vm3 to get to vm2's 
> > eth0.1020 interface.
> >
> > dom0:
> >   eth0:     192.168.1.20/24
> >
> > vm1:
> >   eth0:     192.168.1.10/24
> >   eth0.107: 10.7.1.1/16
> >   gateway:  10.7.0.1
> >
> > vm2:
> >   eth0:      192.168.1.11/24
> >   eth0.1020: 10.20.1.1/16
> >   gateway:   10.20.0.1
> >
> > vm3:
> >   eth0:      192.168.1.12/24
> >   eth0.107:  10.7.0.1/16
> >   eth0.1020: 10.20.0.1/16
> >
> >
> > >From vm1, I can ping the gateway (10.7.0.1) and vm3's 1020 interface 
> > (10.20.0.1), but get no response if I try to ping vm2's 10.20.1.1.
> >
> > vm3 can ping both 10.7.1.1 and 10.20.1.1.
> >
> > Should be a pretty simple packet forwarding example, and it used to 
work 
> > using the old xen-linux 2.6.18 kernel we were using on our deployed 
> > systems.  But now it's broken...
> >
> > I have verified that the exact same setup running on 3 physical PCs 
> > bare-metal works w/out any problems.  I had the problem initially on 
Xen 
> > 3.4.2, but just upgraded to 4.0.0 and have the exact same symptoms.
> > 
> 
> Have you tried turning off tx checksum offload?  (ethtool -K <dev> tx 
off)
> 
> Though for me that just fixes network access between dom0 and domU;
> external traffic normally works fine.

Jeremy, I haven't tried that yet.  I'll try it this morning, although I 
don't really expect that to fix the problem.  I'm also going to pull from 
the xen/master branch again to see if that makes a difference.  My last 
2.6.31.12 kernel was pulled from your tree a few weeks ago.

By the way, the packet forwarding is definitely happening on my router 
domU.  If I look at vm3's eth0 w/ wireshark, I can see the VLAN 
encapsulated ICMP echo request coming in on the 107 VLAN and I can see 
that vm3 is forwarding it out over its 1020 VLAN interface.  However, if I 
look at my dom0's vif3.0 interface with wireshark I only see the VLAN 107 
traffic...  no sign of the 1020 forwarded packet anywhere.  It's as if 
forwarded packets aren't finding their way onto dom0's vif3.0.  I've never 
seen this before, and was under the impression that dom0's vif interfaces 
_WERE_ the domU's interfaces just with a different name... 

-Mike

---
Michael D Labriola
Electric Boat
mlabriol@gdeb.com
401-848-8871 (desk)
401-848-8513 (lab)
401-316-9844 (cell)

Re: pv_ops routing problem?

[Michael D Labriola]

xen-devel-bounces@lists.xensource.com wrote on 04/28/2010 06:40:32 AM:

> Michael D Labriola <mlabriol@gdeb.com> 
> Sent by: xen-devel-bounces@lists.xensource.com
> 
> 04/28/2010 06:40 AM
> 
> To
> 
> Jeremy Fitzhardinge <jeremy@goop.org>
> 
> cc
> 
> Jayson A Dyke <jdyke@gdeb.com>, xen-devel@lists.xensource.com, xen-
> devel-bounces@lists.xensource.com
> 
> Subject
> 
> Re: [Xen-devel] pv_ops routing problem?
> 
> xen-devel-bounces@lists.xensource.com wrote on 04/27/2010 06:02:12 PM:
> 
> > Jeremy Fitzhardinge <jeremy@goop.org> 
> > Sent by: xen-devel-bounces@lists.xensource.com
> > 
> > 04/27/2010 06:02 PM
> > 
> > To
> > 
> > Michael D Labriola <mlabriol@gdeb.com>
> > 
> > cc
> > 
> > Jayson A Dyke <jdyke@gdeb.com>, xen-devel@lists.xensource.com
> > 
> > Subject
> > 
> > Re: [Xen-devel] pv_ops routing problem?
> > 
> > On 04/27/2010 02:22 PM, Michael D Labriola wrote:
> > > Ok, I've got a problem.  My current setup is as follows:
> > >
> > > I have a pv_ops dom0 (xen/master from a few weeks ago) running 3 
> domUs. 
> > > Each domU has an eth0 interface and one or more VLAN interfaces.
> > >
> > > vm1, vm2, and vm3 are all domUs running with the same pv_ops kernel 
as 
> 
> > > dom0.  vm1's eth0.107 interface routes through vm3 to get to vm2's 
> > > eth0.1020 interface.
> > >
> > > dom0:
> > >   eth0:     192.168.1.20/24
> > >
> > > vm1:
> > >   eth0:     192.168.1.10/24
> > >   eth0.107: 10.7.1.1/16
> > >   gateway:  10.7.0.1
> > >
> > > vm2:
> > >   eth0:      192.168.1.11/24
> > >   eth0.1020: 10.20.1.1/16
> > >   gateway:   10.20.0.1
> > >
> > > vm3:
> > >   eth0:      192.168.1.12/24
> > >   eth0.107:  10.7.0.1/16
> > >   eth0.1020: 10.20.0.1/16
> > >
> > >
> > > >From vm1, I can ping the gateway (10.7.0.1) and vm3's 1020 
interface 
> > > (10.20.0.1), but get no response if I try to ping vm2's 10.20.1.1.
> > >
> > > vm3 can ping both 10.7.1.1 and 10.20.1.1.
> > >
> > > Should be a pretty simple packet forwarding example, and it used to 
> work 
> > > using the old xen-linux 2.6.18 kernel we were using on our deployed 
> > > systems.  But now it's broken...
> > >
> > > I have verified that the exact same setup running on 3 physical PCs 
> > > bare-metal works w/out any problems.  I had the problem initially on 

> Xen 
> > > 3.4.2, but just upgraded to 4.0.0 and have the exact same symptoms.
> > > 
> > 
> > Have you tried turning off tx checksum offload?  (ethtool -K <dev> tx 
> off)
> > 
> > Though for me that just fixes network access between dom0 and domU;
> > external traffic normally works fine.
> 
> Jeremy, I haven't tried that yet.  I'll try it this morning, although I 
> don't really expect that to fix the problem.  I'm also going to pull 
from 
> the xen/master branch again to see if that makes a difference.  My last 
> 2.6.31.12 kernel was pulled from your tree a few weeks ago.
> 
> By the way, the packet forwarding is definitely happening on my router 
> domU.  If I look at vm3's eth0 w/ wireshark, I can see the VLAN 
> encapsulated ICMP echo request coming in on the 107 VLAN and I can see 
> that vm3 is forwarding it out over its 1020 VLAN interface.  However, if 
I 
> look at my dom0's vif3.0 interface with wireshark I only see the VLAN 
107 
> traffic...  no sign of the 1020 forwarded packet anywhere.  It's as if 
> forwarded packets aren't finding their way onto dom0's vif3.0.  I've 
never 
> seen this before, and was under the impression that dom0's vif 
interfaces 
> _WERE_ the domU's interfaces just with a different name... 

Should the tx checksum offload only be disabled on dom0?  And does it have 
to get disabled before xend starts?  I've tried disabling it on just dom0 
and also on domUs with no difference.

Also, I updated to the current xen/master and it behaves the same.

I just tried adding 107 and 1020 VLAN interfaces to my dom0 (10.7.0.2 and 
10.20.0.2) and it turns out that dom0 also cannot ping the VLAN interfaces 
on any of the domUs...  not sure if that's a separate issue or if it's all 
related...

-Mike

---
Michael D Labriola
Electric Boat
mlabriol@gdeb.com
401-848-8871 (desk)
401-848-8513 (lab)
401-316-9844 (cell)

Re: pv_ops routing problem?

[Michael D Labriola]

xen-devel-bounces@lists.xensource.com wrote on 04/28/2010 09:07:20 AM:

> Michael D Labriola <mlabriol@gdeb.com> 
> Sent by: xen-devel-bounces@lists.xensource.com
> 
> 04/28/2010 09:07 AM
> 
> To
> 
> Michael D Labriola <mlabriol@gdeb.com>
> 
> cc
> 
> Jayson A Dyke <jdyke@gdeb.com>, xen-devel@lists.xensource.com, 
> Jeremy Fitzhardinge <jeremy@goop.org>, 
xen-devel-bounces@lists.xensource.com
> 
> Subject
> 
> Re: [Xen-devel] pv_ops routing problem?
> 
> xen-devel-bounces@lists.xensource.com wrote on 04/28/2010 06:40:32 AM:
> 
> > Michael D Labriola <mlabriol@gdeb.com> 
> > Sent by: xen-devel-bounces@lists.xensource.com
> > 
> > 04/28/2010 06:40 AM
> > 
> > To
> > 
> > Jeremy Fitzhardinge <jeremy@goop.org>
> > 
> > cc
> > 
> > Jayson A Dyke <jdyke@gdeb.com>, xen-devel@lists.xensource.com, xen-
> > devel-bounces@lists.xensource.com
> > 
> > Subject
> > 
> > Re: [Xen-devel] pv_ops routing problem?
> > 
> > xen-devel-bounces@lists.xensource.com wrote on 04/27/2010 06:02:12 PM:
> > 
> > > Jeremy Fitzhardinge <jeremy@goop.org> 
> > > Sent by: xen-devel-bounces@lists.xensource.com
> > > 
> > > 04/27/2010 06:02 PM
> > > 
> > > To
> > > 
> > > Michael D Labriola <mlabriol@gdeb.com>
> > > 
> > > cc
> > > 
> > > Jayson A Dyke <jdyke@gdeb.com>, xen-devel@lists.xensource.com
> > > 
> > > Subject
> > > 
> > > Re: [Xen-devel] pv_ops routing problem?
> > > 
> > > On 04/27/2010 02:22 PM, Michael D Labriola wrote:
> > > > Ok, I've got a problem.  My current setup is as follows:
> > > >
> > > > I have a pv_ops dom0 (xen/master from a few weeks ago) running 3 
> > domUs. 
> > > > Each domU has an eth0 interface and one or more VLAN interfaces.
> > > >
> > > > vm1, vm2, and vm3 are all domUs running with the same pv_ops 
kernel 
> as 
> > 
> > > > dom0.  vm1's eth0.107 interface routes through vm3 to get to vm2's 

> > > > eth0.1020 interface.
> > > >
> > > > dom0:
> > > >   eth0:     192.168.1.20/24
> > > >
> > > > vm1:
> > > >   eth0:     192.168.1.10/24
> > > >   eth0.107: 10.7.1.1/16
> > > >   gateway:  10.7.0.1
> > > >
> > > > vm2:
> > > >   eth0:      192.168.1.11/24
> > > >   eth0.1020: 10.20.1.1/16
> > > >   gateway:   10.20.0.1
> > > >
> > > > vm3:
> > > >   eth0:      192.168.1.12/24
> > > >   eth0.107:  10.7.0.1/16
> > > >   eth0.1020: 10.20.0.1/16
> > > >
> > > >
> > > > >From vm1, I can ping the gateway (10.7.0.1) and vm3's 1020 
> interface 
> > > > (10.20.0.1), but get no response if I try to ping vm2's 10.20.1.1.
> > > >
> > > > vm3 can ping both 10.7.1.1 and 10.20.1.1.
> > > >
> > > > Should be a pretty simple packet forwarding example, and it used 
to 
> > work 
> > > > using the old xen-linux 2.6.18 kernel we were using on our 
deployed 
> > > > systems.  But now it's broken...
> > > >
> > > > I have verified that the exact same setup running on 3 physical 
PCs 
> > > > bare-metal works w/out any problems.  I had the problem initially 
on 
> 
> > Xen 
> > > > 3.4.2, but just upgraded to 4.0.0 and have the exact same 
symptoms.
> > > > 
> > > 
> > > Have you tried turning off tx checksum offload?  (ethtool -K <dev> 
tx 
> > off)
> > > 
> > > Though for me that just fixes network access between dom0 and domU;
> > > external traffic normally works fine.
> > 
> > Jeremy, I haven't tried that yet.  I'll try it this morning, although 
I 
> > don't really expect that to fix the problem.  I'm also going to pull 
> from 
> > the xen/master branch again to see if that makes a difference.  My 
last 
> > 2.6.31.12 kernel was pulled from your tree a few weeks ago.
> > 
> > By the way, the packet forwarding is definitely happening on my router 

> > domU.  If I look at vm3's eth0 w/ wireshark, I can see the VLAN 
> > encapsulated ICMP echo request coming in on the 107 VLAN and I can see 

> > that vm3 is forwarding it out over its 1020 VLAN interface.  However, 
if 
> I 
> > look at my dom0's vif3.0 interface with wireshark I only see the VLAN 
> 107 
> > traffic...  no sign of the 1020 forwarded packet anywhere.  It's as if 

> > forwarded packets aren't finding their way onto dom0's vif3.0.  I've 
> never 
> > seen this before, and was under the impression that dom0's vif 
> interfaces 
> > _WERE_ the domU's interfaces just with a different name... 
> 
> Should the tx checksum offload only be disabled on dom0?  And does it 
have 
> to get disabled before xend starts?  I've tried disabling it on just 
dom0 
> and also on domUs with no difference.
> 
> Also, I updated to the current xen/master and it behaves the same.
> 
> I just tried adding 107 and 1020 VLAN interfaces to my dom0 (10.7.0.2 
and 
> 10.20.0.2) and it turns out that dom0 also cannot ping the VLAN 
interfaces 
> on any of the domUs...  not sure if that's a separate issue or if it's 
all 
> related...

Once I lowered and re-raised dom0's VLAN interfaces dom0 could ping all 
the domU's VLAN interfaces just fine.  Not sure why I had to do that, 
though...

Still can't ping from vm1 to vm2 using v3 as the default gateway, though.

I've also tried this now on xen/stable with the exact same results.

-Mike

PS - Jeremy, when I pulled from you this morning xen/stable was actually 
one commit ahead of xen/stable-2.6.32.x... not sure if that was 
intentional.  ;-)

---
Michael D Labriola
Electric Boat
mlabriol@gdeb.com
401-848-8871 (desk)
401-848-8513 (lab)
401-316-9844 (cell)

Re: pv_ops routing problem?

[Michael D Labriola]

xen-devel-bounces@lists.xensource.com wrote on 04/28/2010 09:21:50 AM:

*snip*>
> > > > On 04/27/2010 02:22 PM, Michael D Labriola wrote:
> > > > > Ok, I've got a problem.  My current setup is as follows:
> > > > >
> > > > > I have a pv_ops dom0 (xen/master from a few weeks ago) running 3 

> > > domUs. 
> > > > > Each domU has an eth0 interface and one or more VLAN interfaces.
> > > > >
> > > > > vm1, vm2, and vm3 are all domUs running with the same pv_ops 
> kernel 
> > as 
> > > 
> > > > > dom0.  vm1's eth0.107 interface routes through vm3 to get to 
vm2's 
> 
> > > > > eth0.1020 interface.
> > > > >
> > > > > dom0:
> > > > >   eth0:     192.168.1.20/24
> > > > >
> > > > > vm1:
> > > > >   eth0:     192.168.1.10/24
> > > > >   eth0.107: 10.7.1.1/16
> > > > >   gateway:  10.7.0.1
> > > > >
> > > > > vm2:
> > > > >   eth0:      192.168.1.11/24
> > > > >   eth0.1020: 10.20.1.1/16
> > > > >   gateway:   10.20.0.1
> > > > >
> > > > > vm3:
> > > > >   eth0:      192.168.1.12/24
> > > > >   eth0.107:  10.7.0.1/16
> > > > >   eth0.1020: 10.20.0.1/16
> > > > >
> > > > >
> > > > > >From vm1, I can ping the gateway (10.7.0.1) and vm3's 1020 
> > interface 
> > > > > (10.20.0.1), but get no response if I try to ping vm2's 
10.20.1.1.
> > > > >
> > > > > vm3 can ping both 10.7.1.1 and 10.20.1.1.
> > > > >
> > > > > Should be a pretty simple packet forwarding example, and it used 

> to 
> > > work 
> > > > > using the old xen-linux 2.6.18 kernel we were using on our 
> deployed 
> > > > > systems.  But now it's broken...
> > > > >
> > > > > I have verified that the exact same setup running on 3 physical 
> PCs 
> > > > > bare-metal works w/out any problems.  I had the problem 
initially 
> on 
> > 
> > > Xen 
> > > > > 3.4.2, but just upgraded to 4.0.0 and have the exact same 
> symptoms.
> > > > > 
> > > > 
> > > > Have you tried turning off tx checksum offload?  (ethtool -K <dev> 

> tx 
> > > off)
> > > > 
> > > > Though for me that just fixes network access between dom0 and 
domU;
> > > > external traffic normally works fine.
> > > 
> > > Jeremy, I haven't tried that yet.  I'll try it this morning, 
although 
> I 
> > > don't really expect that to fix the problem.  I'm also going to pull 

> > from 
> > > the xen/master branch again to see if that makes a difference.  My 
> last 
> > > 2.6.31.12 kernel was pulled from your tree a few weeks ago.
> > > 
> > > By the way, the packet forwarding is definitely happening on my 
router 
> 
> > > domU.  If I look at vm3's eth0 w/ wireshark, I can see the VLAN 
> > > encapsulated ICMP echo request coming in on the 107 VLAN and I can 
see 
> 
> > > that vm3 is forwarding it out over its 1020 VLAN interface. However, 

> if 
> > I 
> > > look at my dom0's vif3.0 interface with wireshark I only see the 
VLAN 
> > 107 
> > > traffic...  no sign of the 1020 forwarded packet anywhere.  It's as 
if 
> 
> > > forwarded packets aren't finding their way onto dom0's vif3.0.  I've 

> > never 
> > > seen this before, and was under the impression that dom0's vif 
> > interfaces 
> > > _WERE_ the domU's interfaces just with a different name... 
> > 
> > Should the tx checksum offload only be disabled on dom0?  And does it 
> have 
> > to get disabled before xend starts?  I've tried disabling it on just 
> dom0 
> > and also on domUs with no difference.
> > 
> > Also, I updated to the current xen/master and it behaves the same.
> > 
> > I just tried adding 107 and 1020 VLAN interfaces to my dom0 (10.7.0.2 
> and 
> > 10.20.0.2) and it turns out that dom0 also cannot ping the VLAN 
> interfaces 
> > on any of the domUs...  not sure if that's a separate issue or if it's 

> all 
> > related...
> 
> Once I lowered and re-raised dom0's VLAN interfaces dom0 could ping all 
> the domU's VLAN interfaces just fine.  Not sure why I had to do that, 
> though...
> 
> Still can't ping from vm1 to vm2 using v3 as the default gateway, 
though.
> 
> I've also tried this now on xen/stable with the exact same results.
> 
> -Mike
> 
> PS - Jeremy, when I pulled from you this morning xen/stable was actually 

> one commit ahead of xen/stable-2.6.32.x... not sure if that was 
> intentional.  ;-)

Can anyone confirm or deny that routing between VLAN interfaces on domUs 
hosted on a xen/master dom0 is indeed broken?  I'm not sure what else to 
do to troubleshoot this and it's behaving like this on all my hardware.

---
Michael D Labriola
Electric Boat
mlabriol@gdeb.com
401-848-8871 (desk)
401-848-8513 (lab)
401-316-9844 (cell)

Re: pv_ops routing problem?

[Konrad Rzeszutek Wilk]

On Wed, Apr 28, 2010 at 03:06:25PM -0400, Michael D Labriola wrote:
> xen-devel-bounces@lists.xensource.com wrote on 04/28/2010 09:21:50 AM:
> 
> *snip*>
> > > > > On 04/27/2010 02:22 PM, Michael D Labriola wrote:
> > > > > > Ok, I've got a problem.  My current setup is as follows:
> > > > > >
> > > > > > I have a pv_ops dom0 (xen/master from a few weeks ago) running 3 
> 
> > > > domUs. 
> > > > > > Each domU has an eth0 interface and one or more VLAN interfaces.
> > > > > >
> > > > > > vm1, vm2, and vm3 are all domUs running with the same pv_ops 
> > kernel 
> > > as 
> > > > 
> > > > > > dom0.  vm1's eth0.107 interface routes through vm3 to get to 
> vm2's 
> > 
> > > > > > eth0.1020 interface.
> > > > > >
> > > > > > dom0:
> > > > > >   eth0:     192.168.1.20/24
> > > > > >
> > > > > > vm1:
> > > > > >   eth0:     192.168.1.10/24
> > > > > >   eth0.107: 10.7.1.1/16
> > > > > >   gateway:  10.7.0.1
> > > > > >
> > > > > > vm2:
> > > > > >   eth0:      192.168.1.11/24
> > > > > >   eth0.1020: 10.20.1.1/16
> > > > > >   gateway:   10.20.0.1
> > > > > >
> > > > > > vm3:
> > > > > >   eth0:      192.168.1.12/24
> > > > > >   eth0.107:  10.7.0.1/16
> > > > > >   eth0.1020: 10.20.0.1/16
> > > > > >
> > > > > >
> > > > > > >From vm1, I can ping the gateway (10.7.0.1) and vm3's 1020 
> > > interface 
> > > > > > (10.20.0.1), but get no response if I try to ping vm2's 
> 10.20.1.1.
> > > > > >
> > > > > > vm3 can ping both 10.7.1.1 and 10.20.1.1.
> > > > > >
> > > > > > Should be a pretty simple packet forwarding example, and it used 
> 
> > to 
> > > > work 
> > > > > > using the old xen-linux 2.6.18 kernel we were using on our 
> > deployed 
> > > > > > systems.  But now it's broken...
> > > > > >
> > > > > > I have verified that the exact same setup running on 3 physical 
> > PCs 
> > > > > > bare-metal works w/out any problems.  I had the problem 
> initially 
> > on 
> > > 
> > > > Xen 
> > > > > > 3.4.2, but just upgraded to 4.0.0 and have the exact same 
> > symptoms.
> > > > > > 
> > > > > 
> > > > > Have you tried turning off tx checksum offload?  (ethtool -K <dev> 
> 
> > tx 
> > > > off)
> > > > > 
> > > > > Though for me that just fixes network access between dom0 and 
> domU;
> > > > > external traffic normally works fine.
> > > > 
> > > > Jeremy, I haven't tried that yet.  I'll try it this morning, 
> although 
> > I 
> > > > don't really expect that to fix the problem.  I'm also going to pull 
> 
> > > from 
> > > > the xen/master branch again to see if that makes a difference.  My 
> > last 
> > > > 2.6.31.12 kernel was pulled from your tree a few weeks ago.
> > > > 
> > > > By the way, the packet forwarding is definitely happening on my 
> router 
> > 
> > > > domU.  If I look at vm3's eth0 w/ wireshark, I can see the VLAN 
> > > > encapsulated ICMP echo request coming in on the 107 VLAN and I can 
> see 
> > 
> > > > that vm3 is forwarding it out over its 1020 VLAN interface. However, 
> 
> > if 
> > > I 
> > > > look at my dom0's vif3.0 interface with wireshark I only see the 
> VLAN 
> > > 107 
> > > > traffic...  no sign of the 1020 forwarded packet anywhere.  It's as 
> if 
> > 
> > > > forwarded packets aren't finding their way onto dom0's vif3.0.  I've 
> 
> > > never 
> > > > seen this before, and was under the impression that dom0's vif 
> > > interfaces 
> > > > _WERE_ the domU's interfaces just with a different name... 
> > > 
> > > Should the tx checksum offload only be disabled on dom0?  And does it 
> > have 
> > > to get disabled before xend starts?  I've tried disabling it on just 
> > dom0 
> > > and also on domUs with no difference.
> > > 
> > > Also, I updated to the current xen/master and it behaves the same.
> > > 
> > > I just tried adding 107 and 1020 VLAN interfaces to my dom0 (10.7.0.2 
> > and 
> > > 10.20.0.2) and it turns out that dom0 also cannot ping the VLAN 
> > interfaces 
> > > on any of the domUs...  not sure if that's a separate issue or if it's 
> 
> > all 
> > > related...
> > 
> > Once I lowered and re-raised dom0's VLAN interfaces dom0 could ping all 
> > the domU's VLAN interfaces just fine.  Not sure why I had to do that, 
> > though...
> > 
> > Still can't ping from vm1 to vm2 using v3 as the default gateway, 
> though.
> > 
> > I've also tried this now on xen/stable with the exact same results.
> > 
> > -Mike
> > 
> > PS - Jeremy, when I pulled from you this morning xen/stable was actually 
> 
> > one commit ahead of xen/stable-2.6.32.x... not sure if that was 
> > intentional.  ;-)
> 
> Can anyone confirm or deny that routing between VLAN interfaces on domUs 
> hosted on a xen/master dom0 is indeed broken?  I'm not sure what else to 
> do to troubleshoot this and it's behaving like this on all my hardware.

Adnan, is this the same problem as with the TX checksum issue?