UDP checksums broken in Dom0 -> DomU vif transfer

UDP checksums broken in Dom0 -> DomU vif transfer

[Michal Suchanek]

Hello,

when I boot DomU which uses DHCP to configure IPv4 address it does
never get a lease.

The packets travel to Dom0 where the dhcp server receives them, sends
a reply, that travels to DomU where dhclient receives it, says the
checksum is invalid, and discards it.

The problem is documented here:

http://old-list-archives.xen.org/archives/html/xen-users/2006-02/msg00152.html
http://old-list-archives.xen.org/archives/html/xen-devel/2011-04/msg01235.html
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1655

The fix is to turn off UDP checksum offloading on the vif interface in
Dom0 as documented in the above mail:

I edited /etc/xen/scripts/network-bridge,
adding this command to the end of the op_start() function:

        add_to_bridge2 ${bridge} ${pdev}
        do_ifup ${netdev}
+       # disable ip checksum offloading for veth device
+       ethtool -K ${netdev} tx off
    else
        # old style without ${vdev}

Note: I am not sure which path is taken through the script, I set the
parameter manually with ethtool before I found this patch.

It some solutions suggest to turn off UDP checksum offloading in the
DomU as well but it does not seem to be necessary since the packets
would travel to the dhcp server and it would reply to them.

Some people say this is working for them.

I suspect this is because some Linux distributions already carry this patch.

Any reason why this can't be fixed in Xen upstream?

This issue is years old and has been discovered, solved, re-discovered
and re-solved numerous times already.

Thanks

Michal

Re: UDP checksums broken in Dom0 -> DomU vif transfer

[Konrad Rzeszutek Wilk]

On Mon, Dec 19, 2011 at 10:27:36AM +0100, Michal Suchanek wrote:
> Hello,
> 
> when I boot DomU which uses DHCP to configure IPv4 address it does

You didn't say what version of DomU you are running? Is it 3.1?
> never get a lease.
> 
> The packets travel to Dom0 where the dhcp server receives them, sends
> a reply, that travels to DomU where dhclient receives it, says the
> checksum is invalid, and discards it.
> 
> The problem is documented here:
> 
> http://old-list-archives.xen.org/archives/html/xen-users/2006-02/msg00152.html
> http://old-list-archives.xen.org/archives/html/xen-devel/2011-04/msg01235.html
> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1655
> 
> The fix is to turn off UDP checksum offloading on the vif interface in
> Dom0 as documented in the above mail:
> 
> I edited /etc/xen/scripts/network-bridge,
> adding this command to the end of the op_start() function:
> 
>         add_to_bridge2 ${bridge} ${pdev}
>         do_ifup ${netdev}
> +       # disable ip checksum offloading for veth device
> +       ethtool -K ${netdev} tx off
>     else
>         # old style without ${vdev}
> 
> Note: I am not sure which path is taken through the script, I set the
> parameter manually with ethtool before I found this patch.
> 
> It some solutions suggest to turn off UDP checksum offloading in the
> DomU as well but it does not seem to be necessary since the packets
> would travel to the dhcp server and it would reply to them.
> 
> Some people say this is working for them.
> 
> I suspect this is because some Linux distributions already carry this patch.
> 
> Any reason why this can't be fixed in Xen upstream?

It should be fixed in the kernel and I think it is fixed. Did you have
this problem with a 3.1 or 3.0 kernel?
> 
> This issue is years old and has been discovered, solved, re-discovered
> and re-solved numerous times already.
> 
> Thanks
> 
> Michal
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: UDP checksums broken in Dom0 -> DomU vif transfer

[Michal Suchanek]

On 19 December 2011 15:29, Konrad Rzeszutek Wilk <konrad@darnok.org> wrote:
> On Mon, Dec 19, 2011 at 10:27:36AM +0100, Michal Suchanek wrote:
>> Hello,
>>
>> when I boot DomU which uses DHCP to configure IPv4 address it does
>
> You didn't say what version of DomU you are running? Is it 3.1?

I have this problem in 2.6.32 and 3.1 kernels.

>> never get a lease.
>>
>> The packets travel to Dom0 where the dhcp server receives them, sends
>> a reply, that travels to DomU where dhclient receives it, says the
>> checksum is invalid, and discards it.
>>
>> The problem is documented here:
>>
>> http://old-list-archives.xen.org/archives/html/xen-users/2006-02/msg00152.html
>> http://old-list-archives.xen.org/archives/html/xen-devel/2011-04/msg01235.html
>> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1655
>>
>> The fix is to turn off UDP checksum offloading on the vif interface in
>> Dom0 as documented in the above mail:
>>
>> I edited /etc/xen/scripts/network-bridge,
>> adding this command to the end of the op_start() function:
>>
>>         add_to_bridge2 ${bridge} ${pdev}
>>         do_ifup ${netdev}
>> +       # disable ip checksum offloading for veth device
>> +       ethtool -K ${netdev} tx off
>>     else
>>         # old style without ${vdev}
>>
>> Note: I am not sure which path is taken through the script, I set the
>> parameter manually with ethtool before I found this patch.
>>
>> It some solutions suggest to turn off UDP checksum offloading in the
>> DomU as well but it does not seem to be necessary since the packets
>> would travel to the dhcp server and it would reply to them.
>>
>> Some people say this is working for them.
>>
>> I suspect this is because some Linux distributions already carry this patch.
>>
>> Any reason why this can't be fixed in Xen upstream?
>
> It should be fixed in the kernel and I think it is fixed. Did you have
> this problem with a 3.1 or 3.0 kernel?

Apparently it is not fixed.

Do you have hash of the Linux upstream commit that fixes it?

Thanks

Michal

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: UDP checksums broken in Dom0 -> DomU vif transfer

[Konrad Rzeszutek Wilk]

On Mon, Dec 19, 2011 at 04:20:17PM +0100, Michal Suchanek wrote:
> On 19 December 2011 15:29, Konrad Rzeszutek Wilk <konrad@darnok.org> wrote:
> > On Mon, Dec 19, 2011 at 10:27:36AM +0100, Michal Suchanek wrote:
> >> Hello,
> >>
> >> when I boot DomU which uses DHCP to configure IPv4 address it does
> >
> > You didn't say what version of DomU you are running? Is it 3.1?
> 
> I have this problem in 2.6.32 and 3.1 kernels.

Ok. Lets concentrate on 3.1 then.
> 
> >> never get a lease.
> >>
> >> The packets travel to Dom0 where the dhcp server receives them, sends
> >> a reply, that travels to DomU where dhclient receives it, says the
> >> checksum is invalid, and discards it.
> >>
> >> The problem is documented here:
> >>
> >> http://old-list-archives.xen.org/archives/html/xen-users/2006-02/msg00152.html
> >> http://old-list-archives.xen.org/archives/html/xen-devel/2011-04/msg01235.html
> >> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1655
> >>
> >> The fix is to turn off UDP checksum offloading on the vif interface in
> >> Dom0 as documented in the above mail:
> >>
> >> I edited /etc/xen/scripts/network-bridge,
> >> adding this command to the end of the op_start() function:
> >>
> >> ?? ?? ?? ?? add_to_bridge2 ${bridge} ${pdev}
> >> ?? ?? ?? ?? do_ifup ${netdev}
> >> + ?? ?? ?? # disable ip checksum offloading for veth device
> >> + ?? ?? ?? ethtool -K ${netdev} tx off
> >> ?? ?? else
> >> ?? ?? ?? ?? # old style without ${vdev}
> >>
> >> Note: I am not sure which path is taken through the script, I set the
> >> parameter manually with ethtool before I found this patch.
> >>
> >> It some solutions suggest to turn off UDP checksum offloading in the
> >> DomU as well but it does not seem to be necessary since the packets
> >> would travel to the dhcp server and it would reply to them.
> >>
> >> Some people say this is working for them.
> >>
> >> I suspect this is because some Linux distributions already carry this patch.
> >>
> >> Any reason why this can't be fixed in Xen upstream?
> >
> > It should be fixed in the kernel and I think it is fixed. Did you have
> > this problem with a 3.1 or 3.0 kernel?
> 
> Apparently it is not fixed.
> 
> Do you have hash of the Linux upstream commit that fixes it?

Ah, my appoligies - we do not have the fix, albeit I thought I saw the
fix some point. Ian might know since he is the maintainer of
xen-netback.

Re: UDP checksums broken in Dom0 -> DomU vif transfer

[Ian Campbell]

On Mon, 2011-12-19 at 19:43 +0000, Konrad Rzeszutek Wilk wrote:
> On Mon, Dec 19, 2011 at 04:20:17PM +0100, Michal Suchanek wrote:
> > On 19 December 2011 15:29, Konrad Rzeszutek Wilk <konrad@darnok.org> wrote:
> > > On Mon, Dec 19, 2011 at 10:27:36AM +0100, Michal Suchanek wrote:
> > >> Hello,
> > >>
> > >> when I boot DomU which uses DHCP to configure IPv4 address it does
> > >
> > > You didn't say what version of DomU you are running? Is it 3.1?
> > 
> > I have this problem in 2.6.32 and 3.1 kernels.
> 
> Ok. Lets concentrate on 3.1 then.
> > 
> > >> never get a lease.
> > >>
> > >> The packets travel to Dom0 where the dhcp server receives them, sends
> > >> a reply, that travels to DomU where dhclient receives it, says the
> > >> checksum is invalid, and discards it.
> > >>
> > >> The problem is documented here:
> > >>
> > >> http://old-list-archives.xen.org/archives/html/xen-users/2006-02/msg00152.html
> > >> http://old-list-archives.xen.org/archives/html/xen-devel/2011-04/msg01235.html
> > >> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1655
> > >>
> > >> The fix is to turn off UDP checksum offloading on the vif interface in
> > >> Dom0 as documented in the above mail:
> > >>
> > >> I edited /etc/xen/scripts/network-bridge,
> > >> adding this command to the end of the op_start() function:
> > >>
> > >> ?? ?? ?? ?? add_to_bridge2 ${bridge} ${pdev}
> > >> ?? ?? ?? ?? do_ifup ${netdev}
> > >> + ?? ?? ?? # disable ip checksum offloading for veth device
> > >> + ?? ?? ?? ethtool -K ${netdev} tx off
> > >> ?? ?? else
> > >> ?? ?? ?? ?? # old style without ${vdev}
> > >>
> > >> Note: I am not sure which path is taken through the script, I set the
> > >> parameter manually with ethtool before I found this patch.
> > >>
> > >> It some solutions suggest to turn off UDP checksum offloading in the
> > >> DomU as well but it does not seem to be necessary since the packets
> > >> would travel to the dhcp server and it would reply to them.
> > >>
> > >> Some people say this is working for them.
> > >>
> > >> I suspect this is because some Linux distributions already carry this patch.
> > >>
> > >> Any reason why this can't be fixed in Xen upstream?
> > >
> > > It should be fixed in the kernel and I think it is fixed. Did you have
> > > this problem with a 3.1 or 3.0 kernel?
> > 
> > Apparently it is not fixed.
> > 
> > Do you have hash of the Linux upstream commit that fixes it?
> 
> Ah, my appoligies - we do not have the fix, albeit I thought I saw the
> fix some point. Ian might know since he is the maintainer of
> xen-netback.

The issue is that dom0 does checksum offload which means the checksum is
not valid on the domU end, although we know the packet is intact because
it never hit the wire.

The network stack deals with this because skb->ip_summed is set
appropriately but when e.g. raw sockets are used it can ends up exposing
getting exposed to userspace.

We don't want to do the checksum by default since there are performance
gains from avoiding it in the general case. Note that "tx off" turns of
TCP and UDP checksum offload.

It's not clear where the bug is here, it could be a bug in dhclinet for
dropping the packet or perhaps this is something that raw socket driver
should be correcting (based on ip_summed) as the packet passes through
to userspace?

I'm not sure but this might impact native hardware too -- depends on the
H/W's handling of the checksum field on RX, you'd hope they mostly just
leave it alone, but I'm not sure how e.g. LRO/GRO effects things?

Ian.

Re: UDP checksums broken in Dom0 -> DomU vif transfer

[Jean Guyader]

On 20 December 2011 10:37, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Mon, 2011-12-19 at 19:43 +0000, Konrad Rzeszutek Wilk wrote:
>> On Mon, Dec 19, 2011 at 04:20:17PM +0100, Michal Suchanek wrote:
>> > On 19 December 2011 15:29, Konrad Rzeszutek Wilk <konrad@darnok.org> wrote:
>> > > On Mon, Dec 19, 2011 at 10:27:36AM +0100, Michal Suchanek wrote:
>> > >> Hello,
>> > >>
>> > >> when I boot DomU which uses DHCP to configure IPv4 address it does
>> > >
>> > > You didn't say what version of DomU you are running? Is it 3.1?
>> >
>> > I have this problem in 2.6.32 and 3.1 kernels.
>>
>> Ok. Lets concentrate on 3.1 then.
>> >
>> > >> never get a lease.
>> > >>
>> > >> The packets travel to Dom0 where the dhcp server receives them, sends
>> > >> a reply, that travels to DomU where dhclient receives it, says the
>> > >> checksum is invalid, and discards it.
>> > >>
>> > >> The problem is documented here:
>> > >>
>> > >> http://old-list-archives.xen.org/archives/html/xen-users/2006-02/msg00152.html
>> > >> http://old-list-archives.xen.org/archives/html/xen-devel/2011-04/msg01235.html
>> > >> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1655
>> > >>
>> > >> The fix is to turn off UDP checksum offloading on the vif interface in
>> > >> Dom0 as documented in the above mail:
>> > >>
>> > >> I edited /etc/xen/scripts/network-bridge,
>> > >> adding this command to the end of the op_start() function:
>> > >>
>> > >> ?? ?? ?? ?? add_to_bridge2 ${bridge} ${pdev}
>> > >> ?? ?? ?? ?? do_ifup ${netdev}
>> > >> + ?? ?? ?? # disable ip checksum offloading for veth device
>> > >> + ?? ?? ?? ethtool -K ${netdev} tx off
>> > >> ?? ?? else
>> > >> ?? ?? ?? ?? # old style without ${vdev}
>> > >>
>> > >> Note: I am not sure which path is taken through the script, I set the
>> > >> parameter manually with ethtool before I found this patch.
>> > >>
>> > >> It some solutions suggest to turn off UDP checksum offloading in the
>> > >> DomU as well but it does not seem to be necessary since the packets
>> > >> would travel to the dhcp server and it would reply to them.
>> > >>
>> > >> Some people say this is working for them.
>> > >>
>> > >> I suspect this is because some Linux distributions already carry this patch.
>> > >>
>> > >> Any reason why this can't be fixed in Xen upstream?
>> > >
>> > > It should be fixed in the kernel and I think it is fixed. Did you have
>> > > this problem with a 3.1 or 3.0 kernel?
>> >
>> > Apparently it is not fixed.
>> >
>> > Do you have hash of the Linux upstream commit that fixes it?
>>
>> Ah, my appoligies - we do not have the fix, albeit I thought I saw the
>> fix some point. Ian might know since he is the maintainer of
>> xen-netback.
>
> The issue is that dom0 does checksum offload which means the checksum is
> not valid on the domU end, although we know the packet is intact because
> it never hit the wire.
>
> The network stack deals with this because skb->ip_summed is set
> appropriately but when e.g. raw sockets are used it can ends up exposing
> getting exposed to userspace.
>
> We don't want to do the checksum by default since there are performance
> gains from avoiding it in the general case. Note that "tx off" turns of
> TCP and UDP checksum offload.
>
> It's not clear where the bug is here, it could be a bug in dhclinet for
> dropping the packet or perhaps this is something that raw socket driver
> should be correcting (based on ip_summed) as the packet passes through
> to userspace?
>
> I'm not sure but this might impact native hardware too -- depends on the
> H/W's handling of the checksum field on RX, you'd hope they mostly just
> leave it alone, but I'm not sure how e.g. LRO/GRO effects things?
>

I've seen this issue in the past. I belive the bug is in dhclient.
dhclient checks the checksum on the packets and drop them if it's wrong.

Jean

Re: UDP checksums broken in Dom0 -> DomU vif transfer

[Ian Campbell]

On Tue, 2011-12-20 at 09:43 +0000, Jean Guyader wrote:
> On 20 December 2011 10:37, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > The issue is that dom0 does checksum offload which means the checksum is
> > not valid on the domU end, although we know the packet is intact because
> > it never hit the wire.
> >
> > The network stack deals with this because skb->ip_summed is set
> > appropriately but when e.g. raw sockets are used it can ends up exposing
> > getting exposed to userspace.
> >
> > We don't want to do the checksum by default since there are performance
> > gains from avoiding it in the general case. Note that "tx off" turns of
> > TCP and UDP checksum offload.
> >
> > It's not clear where the bug is here, it could be a bug in dhclinet for
> > dropping the packet or perhaps this is something that raw socket driver
> > should be correcting (based on ip_summed) as the packet passes through
> > to userspace?
> >
> > I'm not sure but this might impact native hardware too -- depends on the
> > H/W's handling of the checksum field on RX, you'd hope they mostly just
> > leave it alone, but I'm not sure how e.g. LRO/GRO effects things?
> >
> 
> I've seen this issue in the past. I belive the bug is in dhclient.
> dhclient checks the checksum on the packets and drop them if it's wrong.

Indeed, googling around a bit shows that this dhclient bug affects more
than just Xen, e.g. virtio and even some native hardware appear to be
effected.

http://marc.info/?l=kvm&m=121882968407525&w=2
https://qa.mandriva.com/show_bug.cgi?id=63320
https://bugs.mageia.org/show_bug.cgi?id=1243
http://pkgs.fedoraproject.org/gitweb/?p=dhcp.git;a=blob_plain;f=dhcp-4.2.2-xen-checksum.patch;hb=HEAD

The OP didn't say what distro or version of dhclient he was using but I
think the right place to report this would be to the distro since it
appears to be using an out of date dhclient. In the meantime disabling
offload seems like a reasonable local workaround but it is not a fix we
should apply by default.

Ian.

Re: UDP checksums broken in Dom0 -> DomU vif transfer

[Michal Suchanek]

On 20 December 2011 10:55, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Tue, 2011-12-20 at 09:43 +0000, Jean Guyader wrote:
>> On 20 December 2011 10:37, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> > The issue is that dom0 does checksum offload which means the checksum is
>> > not valid on the domU end, although we know the packet is intact because
>> > it never hit the wire.
>> >
>> > The network stack deals with this because skb->ip_summed is set
>> > appropriately but when e.g. raw sockets are used it can ends up exposing
>> > getting exposed to userspace.
>> >
>> > We don't want to do the checksum by default since there are performance
>> > gains from avoiding it in the general case. Note that "tx off" turns of
>> > TCP and UDP checksum offload.
>> >
>> > It's not clear where the bug is here, it could be a bug in dhclinet for
>> > dropping the packet or perhaps this is something that raw socket driver
>> > should be correcting (based on ip_summed) as the packet passes through
>> > to userspace?
>> >
>> > I'm not sure but this might impact native hardware too -- depends on the
>> > H/W's handling of the checksum field on RX, you'd hope they mostly just
>> > leave it alone, but I'm not sure how e.g. LRO/GRO effects things?
>> >
>>
>> I've seen this issue in the past. I belive the bug is in dhclient.
>> dhclient checks the checksum on the packets and drop them if it's wrong.
>
> Indeed, googling around a bit shows that this dhclient bug affects more
> than just Xen, e.g. virtio and even some native hardware appear to be
> effected.
>
> http://marc.info/?l=kvm&m=121882968407525&w=2
> https://qa.mandriva.com/show_bug.cgi?id=63320
> https://bugs.mageia.org/show_bug.cgi?id=1243
> http://pkgs.fedoraproject.org/gitweb/?p=dhcp.git;a=blob_plain;f=dhcp-4.2.2-xen-checksum.patch;hb=HEAD
>
> The OP didn't say what distro or version of dhclient he was using but I
> think the right place to report this would be to the distro since it
> appears to be using an out of date dhclient. In the meantime disabling
> offload seems like a reasonable local workaround but it is not a fix we
> should apply by default.

Thanks for the links

I am running Ubuntu in the DomU.

Michal

Re: UDP checksums broken in Dom0 -> DomU vif transfer

[Konrad Rzeszutek Wilk]

On Tue, Dec 20, 2011 at 12:57:48PM +0100, Michal Suchanek wrote:
> On 20 December 2011 10:55, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Tue, 2011-12-20 at 09:43 +0000, Jean Guyader wrote:
> >> On 20 December 2011 10:37, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> >> > The issue is that dom0 does checksum offload which means the checksum is
> >> > not valid on the domU end, although we know the packet is intact because
> >> > it never hit the wire.
> >> >
> >> > The network stack deals with this because skb->ip_summed is set
> >> > appropriately but when e.g. raw sockets are used it can ends up exposing
> >> > getting exposed to userspace.
> >> >
> >> > We don't want to do the checksum by default since there are performance
> >> > gains from avoiding it in the general case. Note that "tx off" turns of
> >> > TCP and UDP checksum offload.
> >> >
> >> > It's not clear where the bug is here, it could be a bug in dhclinet for
> >> > dropping the packet or perhaps this is something that raw socket driver
> >> > should be correcting (based on ip_summed) as the packet passes through
> >> > to userspace?
> >> >
> >> > I'm not sure but this might impact native hardware too -- depends on the
> >> > H/W's handling of the checksum field on RX, you'd hope they mostly just
> >> > leave it alone, but I'm not sure how e.g. LRO/GRO effects things?
> >> >
> >>
> >> I've seen this issue in the past. I belive the bug is in dhclient.
> >> dhclient checks the checksum on the packets and drop them if it's wrong.
> >
> > Indeed, googling around a bit shows that this dhclient bug affects more
> > than just Xen, e.g. virtio and even some native hardware appear to be
> > effected.
> >
> > http://marc.info/?l=kvm&m=121882968407525&w=2
> > https://qa.mandriva.com/show_bug.cgi?id=63320
> > https://bugs.mageia.org/show_bug.cgi?id=1243
> > http://pkgs.fedoraproject.org/gitweb/?p=dhcp.git;a=blob_plain;f=dhcp-4.2.2-xen-checksum.patch;hb=HEAD
> >
> > The OP didn't say what distro or version of dhclient he was using but I
> > think the right place to report this would be to the distro since it
> > appears to be using an out of date dhclient. In the meantime disabling
> > offload seems like a reasonable local workaround but it is not a fix we
> > should apply by default.
> 
> Thanks for the links
> 
> I am running Ubuntu in the DomU.

Stefan, is this something you could help us with?

Ian, would it make sense to add all this awesome details in the Xen FAQ
Wiki part?

Re: UDP checksums broken in Dom0 -> DomU vif transfer

[Ian Campbell]

On Tue, 2011-12-20 at 15:35 +0000, Konrad Rzeszutek Wilk wrote:
> On Tue, Dec 20, 2011 at 12:57:48PM +0100, Michal Suchanek wrote:
> > On 20 December 2011 10:55, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > > On Tue, 2011-12-20 at 09:43 +0000, Jean Guyader wrote:
> > >> On 20 December 2011 10:37, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > >> > The issue is that dom0 does checksum offload which means the checksum is
> > >> > not valid on the domU end, although we know the packet is intact because
> > >> > it never hit the wire.
> > >> >
> > >> > The network stack deals with this because skb->ip_summed is set
> > >> > appropriately but when e.g. raw sockets are used it can ends up exposing
> > >> > getting exposed to userspace.
> > >> >
> > >> > We don't want to do the checksum by default since there are performance
> > >> > gains from avoiding it in the general case. Note that "tx off" turns of
> > >> > TCP and UDP checksum offload.
> > >> >
> > >> > It's not clear where the bug is here, it could be a bug in dhclinet for
> > >> > dropping the packet or perhaps this is something that raw socket driver
> > >> > should be correcting (based on ip_summed) as the packet passes through
> > >> > to userspace?
> > >> >
> > >> > I'm not sure but this might impact native hardware too -- depends on the
> > >> > H/W's handling of the checksum field on RX, you'd hope they mostly just
> > >> > leave it alone, but I'm not sure how e.g. LRO/GRO effects things?
> > >> >
> > >>
> > >> I've seen this issue in the past. I belive the bug is in dhclient.
> > >> dhclient checks the checksum on the packets and drop them if it's wrong.
> > >
> > > Indeed, googling around a bit shows that this dhclient bug affects more
> > > than just Xen, e.g. virtio and even some native hardware appear to be
> > > effected.
> > >
> > > http://marc.info/?l=kvm&m=121882968407525&w=2
> > > https://qa.mandriva.com/show_bug.cgi?id=63320
> > > https://bugs.mageia.org/show_bug.cgi?id=1243
> > > http://pkgs.fedoraproject.org/gitweb/?p=dhcp.git;a=blob_plain;f=dhcp-4.2.2-xen-checksum.patch;hb=HEAD
> > >
> > > The OP didn't say what distro or version of dhclient he was using but I
> > > think the right place to report this would be to the distro since it
> > > appears to be using an out of date dhclient. In the meantime disabling
> > > offload seems like a reasonable local workaround but it is not a fix we
> > > should apply by default.
> > 
> > Thanks for the links
> > 
> > I am running Ubuntu in the DomU.
> 
> Stefan, is this something you could help us with?
> 
> Ian, would it make sense to add all this awesome details in the Xen FAQ
> Wiki part?

Yes, please!

Ian.

Re: UDP checksums broken in Dom0 -> DomU vif transfer

[Konrad Rzeszutek Wilk]

> > > Thanks for the links
> > > 
> > > I am running Ubuntu in the DomU.
> > 
> > Stefan, is this something you could help us with?
> > 
> > Ian, would it make sense to add all this awesome details in the Xen FAQ
> > Wiki part?
> 
> Yes, please!

I was hoping you would do it :-)

Re: UDP checksums broken in Dom0 -> DomU vif transfer

[Ian Campbell]

On Tue, 2011-12-20 at 20:03 +0000, Konrad Rzeszutek Wilk wrote:
> > > > Thanks for the links
> > > > 
> > > > I am running Ubuntu in the DomU.
> > > 
> > > Stefan, is this something you could help us with?
> > > 
> > > Ian, would it make sense to add all this awesome details in the Xen FAQ
> > > Wiki part?
> > 
> > Yes, please!
> 
> I was hoping you would do it :-)

Guess what I was hoping ;-)

I've added it to http://wiki.xen.org/wiki/Xen_FAQ_DomU . There are loads
of ".*FAQ.*" pages so I've no idea if this is the best place for it.

Ian.

Re: UDP checksums broken in Dom0 -> DomU vif transfer

[Stefan Bader]

On 20.12.2011 16:35, Konrad Rzeszutek Wilk wrote:
> On Tue, Dec 20, 2011 at 12:57:48PM +0100, Michal Suchanek wrote:
>> On 20 December 2011 10:55, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>>> On Tue, 2011-12-20 at 09:43 +0000, Jean Guyader wrote:
>>>> On 20 December 2011 10:37, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>>>>> The issue is that dom0 does checksum offload which means the checksum is
>>>>> not valid on the domU end, although we know the packet is intact because
>>>>> it never hit the wire.
>>>>>
>>>>> The network stack deals with this because skb->ip_summed is set
>>>>> appropriately but when e.g. raw sockets are used it can ends up exposing
>>>>> getting exposed to userspace.
>>>>>
>>>>> We don't want to do the checksum by default since there are performance
>>>>> gains from avoiding it in the general case. Note that "tx off" turns of
>>>>> TCP and UDP checksum offload.
>>>>>
>>>>> It's not clear where the bug is here, it could be a bug in dhclinet for
>>>>> dropping the packet or perhaps this is something that raw socket driver
>>>>> should be correcting (based on ip_summed) as the packet passes through
>>>>> to userspace?
>>>>>
>>>>> I'm not sure but this might impact native hardware too -- depends on the
>>>>> H/W's handling of the checksum field on RX, you'd hope they mostly just
>>>>> leave it alone, but I'm not sure how e.g. LRO/GRO effects things?
>>>>>
>>>>
>>>> I've seen this issue in the past. I belive the bug is in dhclient.
>>>> dhclient checks the checksum on the packets and drop them if it's wrong.
>>>
>>> Indeed, googling around a bit shows that this dhclient bug affects more
>>> than just Xen, e.g. virtio and even some native hardware appear to be
>>> effected.
>>>
>>> http://marc.info/?l=kvm&m=121882968407525&w=2
>>> https://qa.mandriva.com/show_bug.cgi?id=63320
>>> https://bugs.mageia.org/show_bug.cgi?id=1243
>>> http://pkgs.fedoraproject.org/gitweb/?p=dhcp.git;a=blob_plain;f=dhcp-4.2.2-xen-checksum.patch;hb=HEAD
>>>
>>> The OP didn't say what distro or version of dhclient he was using but I
>>> think the right place to report this would be to the distro since it
>>> appears to be using an out of date dhclient. In the meantime disabling
>>> offload seems like a reasonable local workaround but it is not a fix we
>>> should apply by default.
>>
>> Thanks for the links
>>
>> I am running Ubuntu in the DomU.
> 
> Stefan, is this something you could help us with?

Just returning from being away over the end of year, so not yet followed any
links and such. But yes, ideally there would be a Launchpad bug reported for
this. And the bug number sent to me or here. That report should mention the
release that is used for domU, too.

-Stefan

> Ian, would it make sense to add all this awesome details in the Xen FAQ
> Wiki part?