Re: Question about TCP checksum offload in Xen

Re: Question about TCP checksum offload in Xen

[Anil Madhavapeddy]

On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote:
> Hi,
> 
> I'm working on verifying TCP checksums on incoming packets in Mirage, but
> I've run into a bit of a problem.
> 
> If TCP checksum offload is turned on on a virtual interface (this is the
> default), and if the TCP connection is local to the machine, it looks like
> Xen does not calculate the checksum at all.  This may be valid because Xen
> may be providing a stronger guarantee, but it means that incoming packets
> don't have a valid checksum in the header.  This then means that in Mirage
> we can't just have checksum verification turned on all the time.  This
> would have been the safe fall back option and detecting that checksum
> offload is on, and then not duplicating the verification in Mirage would
> have been an optimisation.  But it looks like this is not an option.  Now I
> need to know for every incoming packet whether checksum verification should
> be done or not.  It should ideally be for every packet since chksum offload
> can be turned off and on on the VIF and existing tcp connections should
> continue.  If not every packet, I need to get a notification or efficiently
> detect right away that the setting is changed on the VIF.

This is a question that seems to keep coming up even for Linux and
Windows, as the combination of local<->local VMs vs local<->off-host and
the checksum offload is quite confusing.

CCing xen-devel: is the appropriate behaviour for a guest VM that wants to
use checksum offloading in all situations documented anywhere?

-- 
Anil Madhavapeddy                                 http://anil.recoil.org

Re: Question about TCP checksum offload in Xen

[Ian Campbell]

On Thu, 2013-12-05 at 11:29 +0000, Anil Madhavapeddy wrote:
> On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote:
> > Hi,
> > 
> > I'm working on verifying TCP checksums on incoming packets in Mirage, but
> > I've run into a bit of a problem.
> > 
> > If TCP checksum offload is turned on on a virtual interface (this is the
> > default), and if the TCP connection is local to the machine, it looks like
> > Xen does not calculate the checksum at all.  This may be valid because Xen
> > may be providing a stronger guarantee, but it means that incoming packets
> > don't have a valid checksum in the header.  This then means that in Mirage
> > we can't just have checksum verification turned on all the time.  This
> > would have been the safe fall back option and detecting that checksum
> > offload is on, and then not duplicating the verification in Mirage would
> > have been an optimisation.  But it looks like this is not an option.  Now I
> > need to know for every incoming packet whether checksum verification should
> > be done or not.  It should ideally be for every packet since chksum offload
> > can be turned off and on on the VIF and existing tcp connections should
> > continue.  If not every packet, I need to get a notification or efficiently
> > detect right away that the setting is changed on the VIF.
> 
> This is a question that seems to keep coming up even for Linux and
> Windows, as the combination of local<->local VMs vs local<->off-host and
> the checksum offload is quite confusing.
> 
> CCing xen-devel: is the appropriate behaviour for a guest VM that wants to
> use checksum offloading in all situations documented anywhere?

I don't understand the question/concern. If you have enabled checksum
offload then of course you don't recalculate the checksum, that's the
whole point of offloading it.

Ian.

Re: Question about TCP checksum offload in Xen

[Richard Mortier]

On 5 Dec 2013, at 11:39, Ian Campbell wrote:

> On Thu, 2013-12-05 at 11:29 +0000, Anil Madhavapeddy wrote:
>> On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote:
>>> Hi,
>>> 
>>> I'm working on verifying TCP checksums on incoming packets in Mirage, but
>>> I've run into a bit of a problem.
>>> 
>>> If TCP checksum offload is turned on on a virtual interface (this is the
>>> default), and if the TCP connection is local to the machine, it looks like
>>> Xen does not calculate the checksum at all.  This may be valid because Xen
>>> may be providing a stronger guarantee, but it means that incoming packets
>>> don't have a valid checksum in the header.  This then means that in Mirage
>>> we can't just have checksum verification turned on all the time.  This
>>> would have been the safe fall back option and detecting that checksum
>>> offload is on, and then not duplicating the verification in Mirage would
>>> have been an optimisation.  But it looks like this is not an option.  Now I
>>> need to know for every incoming packet whether checksum verification should
>>> be done or not.  It should ideally be for every packet since chksum offload
>>> can be turned off and on on the VIF and existing tcp connections should
>>> continue.  If not every packet, I need to get a notification or efficiently
>>> detect right away that the setting is changed on the VIF.
>> 
>> This is a question that seems to keep coming up even for Linux and
>> Windows, as the combination of local<->local VMs vs local<->off-host and
>> the checksum offload is quite confusing.
>> 
>> CCing xen-devel: is the appropriate behaviour for a guest VM that wants to
>> use checksum offloading in all situations documented anywhere?
> 
> I don't understand the question/concern. If you have enabled checksum
> offload then of course you don't recalculate the checksum, that's the
> whole point of offloading it.

i think balraj's question arises because the status of checksum offload can change mid-tcp-flow. how does he know whether it's on or off for a given packet?


-- 
Cheers,

R.




This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please send it back to me, and immediately delete it.   Please do not use, copy or disclose the information contained in this message or in any attachment.  Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham.

This message has been checked for viruses but the contents of an attachment
may still contain software viruses which could damage your computer system, you are advised to perform your own checks. Email communications with the University of Nottingham may be monitored as permitted by UK legislation.

Re: Question about TCP checksum offload in Xen

[Ian Campbell]

On Thu, 2013-12-05 at 11:45 +0000, Richard Mortier wrote:
> On 5 Dec 2013, at 11:39, Ian Campbell wrote:
> 
> > On Thu, 2013-12-05 at 11:29 +0000, Anil Madhavapeddy wrote:
> >> On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote:
> >>> Hi,
> >>> 
> >>> I'm working on verifying TCP checksums on incoming packets in Mirage, but
> >>> I've run into a bit of a problem.
> >>> 
> >>> If TCP checksum offload is turned on on a virtual interface (this is the
> >>> default), and if the TCP connection is local to the machine, it looks like
> >>> Xen does not calculate the checksum at all.  This may be valid because Xen
> >>> may be providing a stronger guarantee, but it means that incoming packets
> >>> don't have a valid checksum in the header.  This then means that in Mirage
> >>> we can't just have checksum verification turned on all the time.  This
> >>> would have been the safe fall back option and detecting that checksum
> >>> offload is on, and then not duplicating the verification in Mirage would
> >>> have been an optimisation.  But it looks like this is not an option.  Now I
> >>> need to know for every incoming packet whether checksum verification should
> >>> be done or not.  It should ideally be for every packet since chksum offload
> >>> can be turned off and on on the VIF and existing tcp connections should
> >>> continue.  If not every packet, I need to get a notification or efficiently
> >>> detect right away that the setting is changed on the VIF.
> >> 
> >> This is a question that seems to keep coming up even for Linux and
> >> Windows, as the combination of local<->local VMs vs local<->off-host and
> >> the checksum offload is quite confusing.
> >> 
> >> CCing xen-devel: is the appropriate behaviour for a guest VM that wants to
> >> use checksum offloading in all situations documented anywhere?
> > 
> > I don't understand the question/concern. If you have enabled checksum
> > offload then of course you don't recalculate the checksum, that's the
> > whole point of offloading it.
> 
> i think balraj's question arises because the status of checksum
> offload can change mid-tcp-flow. how does he know whether it's on or
> off for a given packet?

It's a property of the NIC configuration.

In Linux for example a received skb gets a field set which indicates the
offload state at the time it was received.

Ian.

Re: Question about TCP checksum offload in Xen

[Paul Durrant]

> -----Original Message-----
> From: xen-devel-bounces@lists.xen.org [mailto:xen-devel-
> bounces@lists.xen.org] On Behalf Of Richard Mortier
> Sent: 05 December 2013 11:45
> To: Ian Campbell
> Cc: xen-devel@lists.xenproject.org; Balraj Singh; Mirage List; Anil
> Madhavapeddy
> Subject: Re: [Xen-devel] Question about TCP checksum offload in Xen
> 
> 
> On 5 Dec 2013, at 11:39, Ian Campbell wrote:
> 
> > On Thu, 2013-12-05 at 11:29 +0000, Anil Madhavapeddy wrote:
> >> On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote:
> >>> Hi,
> >>>
> >>> I'm working on verifying TCP checksums on incoming packets in Mirage,
> but
> >>> I've run into a bit of a problem.
> >>>
> >>> If TCP checksum offload is turned on on a virtual interface (this is the
> >>> default), and if the TCP connection is local to the machine, it looks like
> >>> Xen does not calculate the checksum at all.  This may be valid because
> Xen
> >>> may be providing a stronger guarantee, but it means that incoming
> packets
> >>> don't have a valid checksum in the header.  This then means that in
> Mirage
> >>> we can't just have checksum verification turned on all the time.  This
> >>> would have been the safe fall back option and detecting that checksum
> >>> offload is on, and then not duplicating the verification in Mirage would
> >>> have been an optimisation.  But it looks like this is not an option.  Now I
> >>> need to know for every incoming packet whether checksum verification
> should
> >>> be done or not.  It should ideally be for every packet since chksum
> offload
> >>> can be turned off and on on the VIF and existing tcp connections should
> >>> continue.  If not every packet, I need to get a notification or efficiently
> >>> detect right away that the setting is changed on the VIF.
> >>
> >> This is a question that seems to keep coming up even for Linux and
> >> Windows, as the combination of local<->local VMs vs local<->off-host and
> >> the checksum offload is quite confusing.
> >>
> >> CCing xen-devel: is the appropriate behaviour for a guest VM that wants
> to
> >> use checksum offloading in all situations documented anywhere?
> >
> > I don't understand the question/concern. If you have enabled checksum
> > offload then of course you don't recalculate the checksum, that's the
> > whole point of offloading it.
> 
> i think balraj's question arises because the status of checksum offload can
> change mid-tcp-flow. how does he know whether it's on or off for a given
> packet?
> 

Why do you assert that it can change mid-flow? The configuration is decided at ring connect time. Nothing should change after that. If, at that time, the frontend declares it can handle packets with no checksum then clearly it may get some packets that do have checksum as well as ones that don't, depending on where they originated.

  Paul

Re: Question about TCP checksum offload in Xen

[Richard Mortier]

On 5 Dec 2013, at 12:47, Paul Durrant wrote:

...
>>>> On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote:
>>>> 
...
>>>>> It should ideally be for every packet since chksum
>> offload
>>>>> can be turned off and on on the VIF and existing tcp connections should
>>>>> continue.  If not every packet, I need to get a notification or efficiently
>>>>> detect right away that the setting is changed on the VIF.

...

> earlier today, i wrote:
> 
>> i think balraj's question arises because the status of checksum offload can
>> change mid-tcp-flow. how does he know whether it's on or off for a given
>> packet?
> 
> Why do you assert that it can change mid-flow? The configuration is decided at ring connect time. Nothing should change after that. If, at that time, the frontend declares it can handle packets with no checksum then clearly it may get some packets that do have checksum as well as ones that don't, depending on where they originated.

i didn't mean to :) i was just trying to point out to ian the reason for balraj's question -- i'll leave it to balraj from here...

-- 
Cheers,

R.




This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please send it back to me, and immediately delete it.   Please do not use, copy or disclose the information contained in this message or in any attachment.  Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham.

This message has been checked for viruses but the contents of an attachment
may still contain software viruses which could damage your computer system, you are advised to perform your own checks. Email communications with the University of Nottingham may be monitored as permitted by UK legislation.

Re: Question about TCP checksum offload in Xen

[Balraj Singh]

[-- Attachment #1.1: Type: text/plain, Size: 3183 bytes --]

Thanks for clarifying Mort.  I did this test: I set up a long lived TCP
connection from a linux guest to a mirage guest each thru a VIF to the s/w
bridge.  The various offload settings including checksum offload were the
default which is ON.  All packets received by Mirage had bad checksums.
 Now from dom0 I changed the offload settings on the VIFs using ethtool, so
the checksum offload was OFF.  Now all packets had good checksums.

So to have no packets in the jeopardy of being accepted without their
checksum verified, I would like to know on each received pkt if the
checksum was already verified.  Detecting immediately that the setting on
the VIF has changed could also work, but it would involve throwing away
some packets.

Then again, I may have misunderstood something about how the stack should
interact with the backend.  We do declare (maybe just by staying mute, I
think) that we can handle packets with no checksums.  But does that mean
that we never need to do a checksum verification on RX packets?  If so
that's the best.  Though then I will need to think about what was going on
in my test.

Thanks,

Balraj


On Thu, Dec 5, 2013 at 12:55 PM, Richard Mortier <
Richard.Mortier@nottingham.ac.uk> wrote:

>
> On 5 Dec 2013, at 12:47, Paul Durrant wrote:
>
> ...
> >>>> On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote:
> >>>>
> ...
> >>>>> It should ideally be for every packet since chksum
> >> offload
> >>>>> can be turned off and on on the VIF and existing tcp connections
> should
> >>>>> continue.  If not every packet, I need to get a notification or
> efficiently
> >>>>> detect right away that the setting is changed on the VIF.
>
> ...
>
> > earlier today, i wrote:
> >
> >> i think balraj's question arises because the status of checksum offload
> can
> >> change mid-tcp-flow. how does he know whether it's on or off for a given
> >> packet?
> >
> > Why do you assert that it can change mid-flow? The configuration is
> decided at ring connect time. Nothing should change after that. If, at that
> time, the frontend declares it can handle packets with no checksum then
> clearly it may get some packets that do have checksum as well as ones that
> don't, depending on where they originated.
>
> i didn't mean to :) i was just trying to point out to ian the reason for
> balraj's question -- i'll leave it to balraj from here...
>
> --
> Cheers,
>
> R.
>
>
>
>
> This message and any attachment are intended solely for the addressee and
> may contain confidential information. If you have received this message in
> error, please send it back to me, and immediately delete it.   Please do
> not use, copy or disclose the information contained in this message or in
> any attachment.  Any views or opinions expressed by the author of this
> email do not necessarily reflect the views of the University of Nottingham.
>
> This message has been checked for viruses but the contents of an attachment
> may still contain software viruses which could damage your computer
> system, you are advised to perform your own checks. Email communications
> with the University of Nottingham may be monitored as permitted by UK
> legislation.
>
>
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 3969 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Question about TCP checksum offload in Xen

[Paul Durrant]

De-html...

------
From: balraj885@gmail.com [mailto:balraj885@gmail.com] On Behalf Of Balraj Singh
Sent: 05 December 2013 13:33
To: Richard Mortier
Cc: Paul Durrant; Ian Campbell; xen-devel@lists.xenproject.org; Mirage List; Anil Madhavapeddy
Subject: Re: [Xen-devel] Question about TCP checksum offload in Xen

Thanks for clarifying Mort.  I did this test: I set up a long lived TCP connection from a linux guest to a mirage guest each thru a VIF to the s/w bridge.  The various offload settings including checksum offload were the default which is ON.  All packets received by Mirage had bad checksums.  Now from dom0 I changed the offload settings on the VIFs using ethtool, so the checksum offload was OFF.  Now all packets had good checksums.

So to have no packets in the jeopardy of being accepted without their checksum verified, I would like to know on each received pkt if the checksum was already verified.  Detecting immediately that the setting on the VIF has changed could also work, but it would involve throwing away some packets.

Then again, I may have misunderstood something about how the stack should interact with the backend.  We do declare (maybe just by staying mute, I think) that we can handle packets with no checksums.  But does that mean that we never need to do a checksum verification on RX packets?  If so that's the best.  Though then I will need to think about what was going on in my test.
-----

And can we avoid top-posting in future, please. It makes it very hard to follow threads.

Possibly you ran into the issue because (for some reason) the default is to expect a frontend to handle IPv4 TCP/UDP checksum offload. The frontend needs to set feature-no-csum-offload to avoid this expectation. I imagine that's what you want to do.

Also, the state of the checksum field when being passed a packet for offloading will not be zero. It should contain a valid pseudo-header checksum.

  Paul

Re: Question about TCP checksum offload in Xen

[Paul Durrant]

> -----Original Message-----
> From: xen-devel-bounces@lists.xen.org [mailto:xen-devel-
> bounces@lists.xen.org] On Behalf Of Ian Campbell
> Sent: 05 December 2013 11:39
> To: Anil Madhavapeddy
> Cc: xen-devel@lists.xenproject.org; Balraj Singh; Mirage List
> Subject: Re: [Xen-devel] Question about TCP checksum offload in Xen
> 
> On Thu, 2013-12-05 at 11:29 +0000, Anil Madhavapeddy wrote:
> > On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote:
> > > Hi,
> > >
> > > I'm working on verifying TCP checksums on incoming packets in Mirage,
> but
> > > I've run into a bit of a problem.
> > >
> > > If TCP checksum offload is turned on on a virtual interface (this is the
> > > default), and if the TCP connection is local to the machine, it looks like
> > > Xen does not calculate the checksum at all.  This may be valid because
> Xen
> > > may be providing a stronger guarantee, but it means that incoming
> packets
> > > don't have a valid checksum in the header.  This then means that in
> Mirage
> > > we can't just have checksum verification turned on all the time.  This
> > > would have been the safe fall back option and detecting that checksum
> > > offload is on, and then not duplicating the verification in Mirage would
> > > have been an optimisation.  But it looks like this is not an option.  Now I
> > > need to know for every incoming packet whether checksum verification
> should
> > > be done or not.  It should ideally be for every packet since chksum
> offload
> > > can be turned off and on on the VIF and existing tcp connections should
> > > continue.  If not every packet, I need to get a notification or efficiently
> > > detect right away that the setting is changed on the VIF.
> >
> > This is a question that seems to keep coming up even for Linux and
> > Windows, as the combination of local<->local VMs vs local<->off-host and
> > the checksum offload is quite confusing.
> >
> > CCing xen-devel: is the appropriate behaviour for a guest VM that wants to
> > use checksum offloading in all situations documented anywhere?
> 
> I don't understand the question/concern. If you have enabled checksum
> offload then of course you don't recalculate the checksum, that's the
> whole point of offloading it.
> 

If your frontend doesn't advertise checksum offload at ring connect time then the backend should not advertise checksum offload to the network stack and hence all packets passing to the VM should have valid checksum. If your frontend *does* advertise offload then you need to look at the NETRXF flags in the head descriptor of any packet you handle to determine the presence/validity of any checksum.

  Paul

Re: Question about TCP checksum offload in Xen

[John Haxby]

On 05/12/13 11:39, Ian Campbell wrote:
> On Thu, 2013-12-05 at 11:29 +0000, Anil Madhavapeddy wrote:
>> > On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote:
>>> > > Hi,
>>> > > 
>>> > > I'm working on verifying TCP checksums on incoming packets in Mirage, but
>>> > > I've run into a bit of a problem.
>>> > > 
>>> > > If TCP checksum offload is turned on on a virtual interface (this is the
>>> > > default), and if the TCP connection is local to the machine, it looks like
>>> > > Xen does not calculate the checksum at all.  This may be valid because Xen
>>> > > may be providing a stronger guarantee, but it means that incoming packets
>>> > > don't have a valid checksum in the header.  This then means that in Mirage
>>> > > we can't just have checksum verification turned on all the time.  This
>>> > > would have been the safe fall back option and detecting that checksum
>>> > > offload is on, and then not duplicating the verification in Mirage would
>>> > > have been an optimisation.  But it looks like this is not an option.  Now I
>>> > > need to know for every incoming packet whether checksum verification should
>>> > > be done or not.  It should ideally be for every packet since chksum offload
>>> > > can be turned off and on on the VIF and existing tcp connections should
>>> > > continue.  If not every packet, I need to get a notification or efficiently
>>> > > detect right away that the setting is changed on the VIF.
>> > 
>> > This is a question that seems to keep coming up even for Linux and
>> > Windows, as the combination of local<->local VMs vs local<->off-host and
>> > the checksum offload is quite confusing.
>> > 
>> > CCing xen-devel: is the appropriate behaviour for a guest VM that wants to
>> > use checksum offloading in all situations documented anywhere?
> I don't understand the question/concern. If you have enabled checksum
> offload then of course you don't recalculate the checksum, that's the
> whole point of offloading it.

I get this a lot.

There are a few different cases:

  * domain to domain traffic
  * domain to external traffic with egress from a NIC that does offload
  * domain to external through a non-offloading NIC

With xen checksum offloading, domain to domain traffic appears to be
received with a bad checksum.  This is OK, there is no point in
calculating a checksum if the packets are only going through memory.  If
your memory is going to randomly corrupt packets you have more bigger
problems to worry about.   However, this does upset at least Solaris: if
you're using a Solaris guest for NAT then the NAT module on Solaris gets
all upset if the checksum is wrong and drops the packets.  (This is
Solaris's NAT module being overly picky, it may need to recalculate or
at least invalidate the existing checksum, but it doesn't need to check
it as well.)

The second two cases are of interest from the domain perspective.  A
domain has no way of knowing how any given packet is going to leave the
host (or even if it is) so it can't know ahead of time whether to
calculate any checksums: the skb's are just marked with "checksum
needed" as usual and either the egress NIC will do the job or dom0 will
do it.

There is absolutely nothing wrong in any of this (Solaris
notwithstanding).   The difficulty is getting people to realise that
checksums are only calculated when a packet hits the cat-5.  It doesn't
need documenting, it just needs a little thought.   I got tired of
hammering the point home :)

jch

Re: Question about TCP checksum offload in Xen

[Balraj Singh]

[-- Attachment #1.1: Type: text/plain, Size: 4268 bytes --]

Thanks very much.  This is fine and exactly what is observed (that the
checksum is only calculated going to/from a real wire).  The engineering
choice makes sense.  But this does mean that I need to know when the stack
receives a pkt if the checksum should be verified for that packet.  I can't
do a redundant verification because a bad chksum may be because the field
was clobbered (not set) during earlier verification.  So, what I wanted to
know was where to look for this flag - a snippet of code or a ptr to a file
:).

Thanks,

Balraj



On Thu, Dec 5, 2013 at 12:37 PM, John Haxby <john.haxby@oracle.com> wrote:

> On 05/12/13 11:39, Ian Campbell wrote:
> > On Thu, 2013-12-05 at 11:29 +0000, Anil Madhavapeddy wrote:
> >> > On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote:
> >>> > > Hi,
> >>> > >
> >>> > > I'm working on verifying TCP checksums on incoming packets in
> Mirage, but
> >>> > > I've run into a bit of a problem.
> >>> > >
> >>> > > If TCP checksum offload is turned on on a virtual interface (this
> is the
> >>> > > default), and if the TCP connection is local to the machine, it
> looks like
> >>> > > Xen does not calculate the checksum at all.  This may be valid
> because Xen
> >>> > > may be providing a stronger guarantee, but it means that incoming
> packets
> >>> > > don't have a valid checksum in the header.  This then means that
> in Mirage
> >>> > > we can't just have checksum verification turned on all the time.
>  This
> >>> > > would have been the safe fall back option and detecting that
> checksum
> >>> > > offload is on, and then not duplicating the verification in Mirage
> would
> >>> > > have been an optimisation.  But it looks like this is not an
> option.  Now I
> >>> > > need to know for every incoming packet whether checksum
> verification should
> >>> > > be done or not.  It should ideally be for every packet since
> chksum offload
> >>> > > can be turned off and on on the VIF and existing tcp connections
> should
> >>> > > continue.  If not every packet, I need to get a notification or
> efficiently
> >>> > > detect right away that the setting is changed on the VIF.
> >> >
> >> > This is a question that seems to keep coming up even for Linux and
> >> > Windows, as the combination of local<->local VMs vs local<->off-host
> and
> >> > the checksum offload is quite confusing.
> >> >
> >> > CCing xen-devel: is the appropriate behaviour for a guest VM that
> wants to
> >> > use checksum offloading in all situations documented anywhere?
> > I don't understand the question/concern. If you have enabled checksum
> > offload then of course you don't recalculate the checksum, that's the
> > whole point of offloading it.
>
> I get this a lot.
>
> There are a few different cases:
>
>   * domain to domain traffic
>   * domain to external traffic with egress from a NIC that does offload
>   * domain to external through a non-offloading NIC
>
> With xen checksum offloading, domain to domain traffic appears to be
> received with a bad checksum.  This is OK, there is no point in
> calculating a checksum if the packets are only going through memory.  If
> your memory is going to randomly corrupt packets you have more bigger
> problems to worry about.   However, this does upset at least Solaris: if
> you're using a Solaris guest for NAT then the NAT module on Solaris gets
> all upset if the checksum is wrong and drops the packets.  (This is
> Solaris's NAT module being overly picky, it may need to recalculate or
> at least invalidate the existing checksum, but it doesn't need to check
> it as well.)
>
> The second two cases are of interest from the domain perspective.  A
> domain has no way of knowing how any given packet is going to leave the
> host (or even if it is) so it can't know ahead of time whether to
> calculate any checksums: the skb's are just marked with "checksum
> needed" as usual and either the egress NIC will do the job or dom0 will
> do it.
>
> There is absolutely nothing wrong in any of this (Solaris
> notwithstanding).   The difficulty is getting people to realise that
> checksums are only calculated when a packet hits the cat-5.  It doesn't
> need documenting, it just needs a little thought.   I got tired of
> hammering the point home :)
>
> jch
>

[-- Attachment #1.2: Type: text/html, Size: 5299 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Question about TCP checksum offload in Xen

[John Haxby]

On 05/12/13 12:56, Balraj Singh wrote:
> Thanks very much.  This is fine and exactly what is observed (that the
> checksum is only calculated going to/from a real wire).  The engineering
> choice makes sense.  But this does mean that I need to know when the
> stack receives a pkt if the checksum should be verified for that packet.
>  I can't do a redundant verification because a bad chksum may be because
> the field was clobbered (not set) during earlier verification.  So, what
> I wanted to know was where to look for this flag - a snippet of code or
> a ptr to a file :).
> 

Look at the skb in the netfront: there's a flag in there to say whether
the checksum is valid.  Interestingly, if the checksum is valid then you
know that the data has come from outside the host (otherwise it would
have been dropped).

Why do you even care about the checksum?  Packets with bad checksums are
dropped by the NIC (with checksum offload) or by the NIC's driver
(without) -- the only packets you'll see with a bad checksum are those
that have come from another domain on the same host.

jch

> Thanks,
> 
> Balraj
> 
> 
> 
> On Thu, Dec 5, 2013 at 12:37 PM, John Haxby <john.haxby@oracle.com
> <mailto:john.haxby@oracle.com>> wrote:
> 
>     On 05/12/13 11:39, Ian Campbell wrote:
>     > On Thu, 2013-12-05 at 11:29 +0000, Anil Madhavapeddy wrote:
>     >> > On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote:
>     >>> > > Hi,
>     >>> > >
>     >>> > > I'm working on verifying TCP checksums on incoming packets
>     in Mirage, but
>     >>> > > I've run into a bit of a problem.
>     >>> > >
>     >>> > > If TCP checksum offload is turned on on a virtual interface
>     (this is the
>     >>> > > default), and if the TCP connection is local to the machine,
>     it looks like
>     >>> > > Xen does not calculate the checksum at all.  This may be
>     valid because Xen
>     >>> > > may be providing a stronger guarantee, but it means that
>     incoming packets
>     >>> > > don't have a valid checksum in the header.  This then means
>     that in Mirage
>     >>> > > we can't just have checksum verification turned on all the
>     time.  This
>     >>> > > would have been the safe fall back option and detecting that
>     checksum
>     >>> > > offload is on, and then not duplicating the verification in
>     Mirage would
>     >>> > > have been an optimisation.  But it looks like this is not an
>     option.  Now I
>     >>> > > need to know for every incoming packet whether checksum
>     verification should
>     >>> > > be done or not.  It should ideally be for every packet since
>     chksum offload
>     >>> > > can be turned off and on on the VIF and existing tcp
>     connections should
>     >>> > > continue.  If not every packet, I need to get a notification
>     or efficiently
>     >>> > > detect right away that the setting is changed on the VIF.
>     >> >
>     >> > This is a question that seems to keep coming up even for Linux and
>     >> > Windows, as the combination of local<->local VMs vs
>     local<->off-host and
>     >> > the checksum offload is quite confusing.
>     >> >
>     >> > CCing xen-devel: is the appropriate behaviour for a guest VM
>     that wants to
>     >> > use checksum offloading in all situations documented anywhere?
>     > I don't understand the question/concern. If you have enabled checksum
>     > offload then of course you don't recalculate the checksum, that's the
>     > whole point of offloading it.
> 
>     I get this a lot.
> 
>     There are a few different cases:
> 
>       * domain to domain traffic
>       * domain to external traffic with egress from a NIC that does offload
>       * domain to external through a non-offloading NIC
> 
>     With xen checksum offloading, domain to domain traffic appears to be
>     received with a bad checksum.  This is OK, there is no point in
>     calculating a checksum if the packets are only going through memory.  If
>     your memory is going to randomly corrupt packets you have more bigger
>     problems to worry about.   However, this does upset at least Solaris: if
>     you're using a Solaris guest for NAT then the NAT module on Solaris gets
>     all upset if the checksum is wrong and drops the packets.  (This is
>     Solaris's NAT module being overly picky, it may need to recalculate or
>     at least invalidate the existing checksum, but it doesn't need to check
>     it as well.)
> 
>     The second two cases are of interest from the domain perspective.  A
>     domain has no way of knowing how any given packet is going to leave the
>     host (or even if it is) so it can't know ahead of time whether to
>     calculate any checksums: the skb's are just marked with "checksum
>     needed" as usual and either the egress NIC will do the job or dom0 will
>     do it.
> 
>     There is absolutely nothing wrong in any of this (Solaris
>     notwithstanding).   The difficulty is getting people to realise that
>     checksums are only calculated when a packet hits the cat-5.  It doesn't
>     need documenting, it just needs a little thought.   I got tired of
>     hammering the point home :)
> 
>     jch
> 
> 

Re: Question about TCP checksum offload in Xen

[Jon Crowcroft]

[-- Attachment #1.1: Type: text/plain, Size: 3895 bytes --]

i thought a value of 0 in the tcp chcksum field indicated "no checksum" and
could be used in the cases you identify and ought not to trigger problems
in correct code?


On Thu, Dec 5, 2013 at 12:37 PM, John Haxby <john.haxby@oracle.com> wrote:

> On 05/12/13 11:39, Ian Campbell wrote:
> > On Thu, 2013-12-05 at 11:29 +0000, Anil Madhavapeddy wrote:
> >> > On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote:
> >>> > > Hi,
> >>> > >
> >>> > > I'm working on verifying TCP checksums on incoming packets in
> Mirage, but
> >>> > > I've run into a bit of a problem.
> >>> > >
> >>> > > If TCP checksum offload is turned on on a virtual interface (this
> is the
> >>> > > default), and if the TCP connection is local to the machine, it
> looks like
> >>> > > Xen does not calculate the checksum at all.  This may be valid
> because Xen
> >>> > > may be providing a stronger guarantee, but it means that incoming
> packets
> >>> > > don't have a valid checksum in the header.  This then means that
> in Mirage
> >>> > > we can't just have checksum verification turned on all the time.
>  This
> >>> > > would have been the safe fall back option and detecting that
> checksum
> >>> > > offload is on, and then not duplicating the verification in Mirage
> would
> >>> > > have been an optimisation.  But it looks like this is not an
> option.  Now I
> >>> > > need to know for every incoming packet whether checksum
> verification should
> >>> > > be done or not.  It should ideally be for every packet since
> chksum offload
> >>> > > can be turned off and on on the VIF and existing tcp connections
> should
> >>> > > continue.  If not every packet, I need to get a notification or
> efficiently
> >>> > > detect right away that the setting is changed on the VIF.
> >> >
> >> > This is a question that seems to keep coming up even for Linux and
> >> > Windows, as the combination of local<->local VMs vs local<->off-host
> and
> >> > the checksum offload is quite confusing.
> >> >
> >> > CCing xen-devel: is the appropriate behaviour for a guest VM that
> wants to
> >> > use checksum offloading in all situations documented anywhere?
> > I don't understand the question/concern. If you have enabled checksum
> > offload then of course you don't recalculate the checksum, that's the
> > whole point of offloading it.
>
> I get this a lot.
>
> There are a few different cases:
>
>   * domain to domain traffic
>   * domain to external traffic with egress from a NIC that does offload
>   * domain to external through a non-offloading NIC
>
> With xen checksum offloading, domain to domain traffic appears to be
> received with a bad checksum.  This is OK, there is no point in
> calculating a checksum if the packets are only going through memory.  If
> your memory is going to randomly corrupt packets you have more bigger
> problems to worry about.   However, this does upset at least Solaris: if
> you're using a Solaris guest for NAT then the NAT module on Solaris gets
> all upset if the checksum is wrong and drops the packets.  (This is
> Solaris's NAT module being overly picky, it may need to recalculate or
> at least invalidate the existing checksum, but it doesn't need to check
> it as well.)
>
> The second two cases are of interest from the domain perspective.  A
> domain has no way of knowing how any given packet is going to leave the
> host (or even if it is) so it can't know ahead of time whether to
> calculate any checksums: the skb's are just marked with "checksum
> needed" as usual and either the egress NIC will do the job or dom0 will
> do it.
>
> There is absolutely nothing wrong in any of this (Solaris
> notwithstanding).   The difficulty is getting people to realise that
> checksums are only calculated when a packet hits the cat-5.  It doesn't
> need documenting, it just needs a little thought.   I got tired of
> hammering the point home :)
>
> jch
>
>

[-- Attachment #1.2: Type: text/html, Size: 4874 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Question about TCP checksum offload in Xen

[Balraj Singh]

[-- Attachment #1.1: Type: text/plain, Size: 4294 bytes --]

That could work too to signal in the packet itself that the checksum is ok.
 But I don't think the received packets had 0 in the chksum field.  It was
just some random value, though I can check again.

Thanks,

Balraj


On Thu, Dec 5, 2013 at 1:15 PM, Jon Crowcroft <jon.crowcroft@cl.cam.ac.uk>wrote:

> i thought a value of 0 in the tcp chcksum field indicated "no checksum"
> and could be used in the cases you identify and ought not to trigger
> problems in correct code?
>
>
> On Thu, Dec 5, 2013 at 12:37 PM, John Haxby <john.haxby@oracle.com> wrote:
>
>> On 05/12/13 11:39, Ian Campbell wrote:
>> > On Thu, 2013-12-05 at 11:29 +0000, Anil Madhavapeddy wrote:
>> >> > On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote:
>> >>> > > Hi,
>> >>> > >
>> >>> > > I'm working on verifying TCP checksums on incoming packets in
>> Mirage, but
>> >>> > > I've run into a bit of a problem.
>> >>> > >
>> >>> > > If TCP checksum offload is turned on on a virtual interface (this
>> is the
>> >>> > > default), and if the TCP connection is local to the machine, it
>> looks like
>> >>> > > Xen does not calculate the checksum at all.  This may be valid
>> because Xen
>> >>> > > may be providing a stronger guarantee, but it means that incoming
>> packets
>> >>> > > don't have a valid checksum in the header.  This then means that
>> in Mirage
>> >>> > > we can't just have checksum verification turned on all the time.
>>  This
>> >>> > > would have been the safe fall back option and detecting that
>> checksum
>> >>> > > offload is on, and then not duplicating the verification in
>> Mirage would
>> >>> > > have been an optimisation.  But it looks like this is not an
>> option.  Now I
>> >>> > > need to know for every incoming packet whether checksum
>> verification should
>> >>> > > be done or not.  It should ideally be for every packet since
>> chksum offload
>> >>> > > can be turned off and on on the VIF and existing tcp connections
>> should
>> >>> > > continue.  If not every packet, I need to get a notification or
>> efficiently
>> >>> > > detect right away that the setting is changed on the VIF.
>> >> >
>> >> > This is a question that seems to keep coming up even for Linux and
>> >> > Windows, as the combination of local<->local VMs vs local<->off-host
>> and
>> >> > the checksum offload is quite confusing.
>> >> >
>> >> > CCing xen-devel: is the appropriate behaviour for a guest VM that
>> wants to
>> >> > use checksum offloading in all situations documented anywhere?
>> > I don't understand the question/concern. If you have enabled checksum
>> > offload then of course you don't recalculate the checksum, that's the
>> > whole point of offloading it.
>>
>> I get this a lot.
>>
>> There are a few different cases:
>>
>>   * domain to domain traffic
>>   * domain to external traffic with egress from a NIC that does offload
>>   * domain to external through a non-offloading NIC
>>
>> With xen checksum offloading, domain to domain traffic appears to be
>> received with a bad checksum.  This is OK, there is no point in
>> calculating a checksum if the packets are only going through memory.  If
>> your memory is going to randomly corrupt packets you have more bigger
>> problems to worry about.   However, this does upset at least Solaris: if
>> you're using a Solaris guest for NAT then the NAT module on Solaris gets
>> all upset if the checksum is wrong and drops the packets.  (This is
>> Solaris's NAT module being overly picky, it may need to recalculate or
>> at least invalidate the existing checksum, but it doesn't need to check
>> it as well.)
>>
>> The second two cases are of interest from the domain perspective.  A
>> domain has no way of knowing how any given packet is going to leave the
>> host (or even if it is) so it can't know ahead of time whether to
>> calculate any checksums: the skb's are just marked with "checksum
>> needed" as usual and either the egress NIC will do the job or dom0 will
>> do it.
>>
>> There is absolutely nothing wrong in any of this (Solaris
>> notwithstanding).   The difficulty is getting people to realise that
>> checksums are only calculated when a packet hits the cat-5.  It doesn't
>> need documenting, it just needs a little thought.   I got tired of
>> hammering the point home :)
>>
>> jch
>>
>>
>

[-- Attachment #1.2: Type: text/html, Size: 5581 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Question about TCP checksum offload in Xen

[James Harper]

> With xen checksum offloading, domain to domain traffic appears to be
> received with a bad checksum.  This is OK, there is no point in
> calculating a checksum if the packets are only going through memory.  If
> your memory is going to randomly corrupt packets you have more bigger
> problems to worry about.   However, this does upset at least Solaris: if
> you're using a Solaris guest for NAT then the NAT module on Solaris gets
> all upset if the checksum is wrong and drops the packets.  (This is
> Solaris's NAT module being overly picky, it may need to recalculate or
> at least invalidate the existing checksum, but it doesn't need to check
> it as well.)

Windows 2003 can't handle receiving packets that have a bad checksum either. Despite the fact that the 'checksum good' flag is set, Windows seems to go ahead and check it anyway. I wonder if it's the firewall doing that in Windows too... I always just assumed it was NDIS itself.

Windows also seems to differ from Linux in its opinion on whether the total packet length in a gso packet is included in the psuedoheader checksum. That took a bit of mucking around to find as some hardware seems to ignore the psuedoheader checksum and calculate the whole checksum again anyway, so you only see the problem on some cards.

James