ip/udp checksum offload from minios guest

ip/udp checksum offload from minios guest

[Anil Madhavapeddy]

I'm just adding checksum offload support into a custom guest, and wanted to clarify a few things.

In netfront, I can mark outgoing frames with:
- NETTXF_csum_blank
- NETTXF_data_validated

These refer to UDP or TCP checksums, and not the IP checksum, right? Linux seems to never offload IP header checksumming, so it must be offloading the UDP/TCP calculation and then adjusting the IPv4 checksum based on that calculation.

For outgoing UDP, my guest is setting the checksum to 0 (as it's optional in the protocol), and calculating the full IPv4 checksum in software, and all works (slowly).  However, setting NETTXF_csum_blank in the outgoing frame and leaving the IPv4 checksum at 0 doesn't seem to result in any adjustment by netback, and the packet gets dropped. 

How am I supposed to entirely offload the IPv4 checksum calculation for UDP?  Setting the flag unconditionally for non-TCP/UDP traffic (e.g. ARP/ICMP) results in lots of dropped frames, so I am only setting it in the outgoing UDP frames.

-anil

Re: ip/udp checksum offload from minios guest

[Ian Campbell]

On Tue, 2011-03-29 at 17:17 +0100, Anil Madhavapeddy wrote:
> I'm just adding checksum offload support into a custom guest, and wanted to clarify a few things.
> 
> In netfront, I can mark outgoing frames with:
> - NETTXF_csum_blank
> - NETTXF_data_validated
> 
> These refer to UDP or TCP checksums, and not the IP checksum, right?
> Linux seems to never offload IP header checksumming, so it must be
> offloading the UDP/TCP calculation and then adjusting the IPv4
> checksum based on that calculation.
> 
> For outgoing UDP, my guest is setting the checksum to 0 (as it's
> optional in the protocol), and calculating the full IPv4 checksum in
> software, and all works (slowly).  However, setting NETTXF_csum_blank
> in the outgoing frame and leaving the IPv4 checksum at 0 doesn't seem
> to result in any adjustment by netback, and the packet gets dropped. 

I think you need to set the checksum to the psuedo-header checksum
rather than 0 since that is what csum_blank means (such a well named
flag!)...

It's not clear why anything would drop a UDP frame with checksum==0
since, as you say, it is optional. My guess is that since the
NETTXF_csum_blank and data_validated turn into an skb->ip_summed ==
SKB_PARTIAL on the backend side this causes the Linux network stack to
drop it because that statement conflicts with the checksum being 0.

> How am I supposed to entirely offload the IPv4 checksum calculation
> for UDP?  Setting the flag unconditionally for non-TCP/UDP traffic
> (e.g. ARP/ICMP) results in lots of dropped frames, so I am only
> setting it in the outgoing UDP frames.

Which is the right thing to do.

Ian.

Re: ip/udp checksum offload from minios guest

[Anil Madhavapeddy]

On 29 Mar 2011, at 12:37, Ian Campbell wrote:

> On Tue, 2011-03-29 at 17:17 +0100, Anil Madhavapeddy wrote:
>> I'm just adding checksum offload support into a custom guest, and wanted to clarify a few things.
>> 
>> In netfront, I can mark outgoing frames with:
>> - NETTXF_csum_blank
>> - NETTXF_data_validated
>> 
>> These refer to UDP or TCP checksums, and not the IP checksum, right?
>> Linux seems to never offload IP header checksumming, so it must be
>> offloading the UDP/TCP calculation and then adjusting the IPv4
>> checksum based on that calculation.
>> 
>> For outgoing UDP, my guest is setting the checksum to 0 (as it's
>> optional in the protocol), and calculating the full IPv4 checksum in
>> software, and all works (slowly).  However, setting NETTXF_csum_blank
>> in the outgoing frame and leaving the IPv4 checksum at 0 doesn't seem
>> to result in any adjustment by netback, and the packet gets dropped. 
> 
> I think you need to set the checksum to the psuedo-header checksum
> rather than 0 since that is what csum_blank means (such a well named
> flag!)...
> 
> It's not clear why anything would drop a UDP frame with checksum==0
> since, as you say, it is optional. My guess is that since the
> NETTXF_csum_blank and data_validated turn into an skb->ip_summed ==
> SKB_PARTIAL on the backend side this causes the Linux network stack to
> drop it because that statement conflicts with the checksum being 0.

Right, but I'm setting the IPv4 checksum to 0 too, since I want to offload the whole lot and never calculate a body checksum in the guest.  But if the UDP checksum is set to 0, then the backend can't just adjust it to obtain the IPv4 checksum and has to iterate over the packet body at some stage.

I've tried setting the UDP checksum to 0, the pseudoheader, and the full checksum, and the IPv4 checksum is never set by the backend in any of these cases.  If I set the IPv4 checksum in software to the correct value, then packets go through, but the UDP checksums are never altered.

Guess it's time to start slapping printfs all over the dom0 kernel to see what's going on unless you know what I "should" be setting both the IPv4/UDP checksums to with NETTXF_csum_blank...

Anil

Re: ip/udp checksum offload from minios guest

[Ian Campbell]

On Tue, 2011-03-29 at 18:15 +0100, Anil Madhavapeddy wrote:
> On 29 Mar 2011, at 12:37, Ian Campbell wrote:
> 
> > On Tue, 2011-03-29 at 17:17 +0100, Anil Madhavapeddy wrote:
> >> I'm just adding checksum offload support into a custom guest, and wanted to clarify a few things.
> >> 
> >> In netfront, I can mark outgoing frames with:
> >> - NETTXF_csum_blank
> >> - NETTXF_data_validated
> >> 
> >> These refer to UDP or TCP checksums, and not the IP checksum, right?
> >> Linux seems to never offload IP header checksumming, so it must be
> >> offloading the UDP/TCP calculation and then adjusting the IPv4
> >> checksum based on that calculation.
> >> 
> >> For outgoing UDP, my guest is setting the checksum to 0 (as it's
> >> optional in the protocol), and calculating the full IPv4 checksum in
> >> software, and all works (slowly).  However, setting NETTXF_csum_blank
> >> in the outgoing frame and leaving the IPv4 checksum at 0 doesn't seem
> >> to result in any adjustment by netback, and the packet gets dropped. 
> > 
> > I think you need to set the checksum to the psuedo-header checksum
> > rather than 0 since that is what csum_blank means (such a well named
> > flag!)...
> > 
> > It's not clear why anything would drop a UDP frame with checksum==0
> > since, as you say, it is optional. My guess is that since the
> > NETTXF_csum_blank and data_validated turn into an skb->ip_summed ==
> > SKB_PARTIAL on the backend side this causes the Linux network stack to
> > drop it because that statement conflicts with the checksum being 0.
> 
> Right, but I'm setting the IPv4 checksum to 0 too, since I want to
> offload the whole lot and never calculate a body checksum in the
> guest.

Maybe I'm getting confused but the IPv4 checksum only covers the
headers, doesn't it? I don't think you can offload those, at least not
using the current PV protocol. So you need to include a valid IP header
checksum but that doesn't require summing over all the data, just the
headers.

> But if the UDP checksum is set to 0, then the backend can't just
> adjust it to obtain the IPv4 checksum and has to iterate over the
> packet body at some stage.

I don't think it needs to obtain the IPv4 checksum to do UDP checksum if
the UDP checksum is initially set to the partial(/pseudo header)
checksum -- it's already incorporated as necessary.

> I've tried setting the UDP checksum to 0, the pseudoheader, and the
> full checksum, and the IPv4 checksum is never set by the backend in
> any of these cases.  If I set the IPv4 checksum in software to the
> correct value, then packets go through, but the UDP checksums are
> never altered.
> 
> Guess it's time to start slapping printfs all over the dom0 kernel to
> see what's going on unless you know what I "should" be setting both
> the IPv4/UDP checksums to with NETTXF_csum_blank...

The protocol is unhelpfully "defined" to be basically "what Linux does".
i.e. whatever Linux generates in an SKB with skb->ip_summed ==
CHECKSUM_PARTIAL is what you need to generate.

Ian.

RE: ip/udp checksum offload from minios guest

[Paul Durrant]

Maybe there's some confusion of terms here... There are 2 checksums: the IPv4 header and UDP checksum.

The IPv4 header checksum must *always* be calculated by the frontend.

If NETTXF_csum_blank is set (implying NETTXF_data_validated must also be set) then the UDP checksum must be set to cover the UDP pseudo-header since the SKB will be marked CSUM_PARTIAL and thus any h/w driver it is presented to will expect the pseudo-header checksum to be valid. If the SKB is presented to another guest then CSUM_PARTIAL will translated into NETRXF_csum_blank|NETRXF_data_validated and the frontend is at liberty to present it up the stack as being checksum-valid without anything in the datapath having had to walk over the payload to actually calculate the value of that checksum.

  Paul

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of Anil Madhavapeddy
> Sent: 29 March 2011 18:16
> To: Ian Campbell
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] ip/udp checksum offload from minios guest
> 
> On 29 Mar 2011, at 12:37, Ian Campbell wrote:
> 
> > On Tue, 2011-03-29 at 17:17 +0100, Anil Madhavapeddy wrote:
> >> I'm just adding checksum offload support into a custom guest, and
> wanted to clarify a few things.
> >>
> >> In netfront, I can mark outgoing frames with:
> >> - NETTXF_csum_blank
> >> - NETTXF_data_validated
> >>
> >> These refer to UDP or TCP checksums, and not the IP checksum,
> right?
> >> Linux seems to never offload IP header checksumming, so it must
> be
> >> offloading the UDP/TCP calculation and then adjusting the IPv4
> >> checksum based on that calculation.
> >>
> >> For outgoing UDP, my guest is setting the checksum to 0 (as it's
> >> optional in the protocol), and calculating the full IPv4 checksum
> in
> >> software, and all works (slowly).  However, setting
> NETTXF_csum_blank
> >> in the outgoing frame and leaving the IPv4 checksum at 0 doesn't
> seem
> >> to result in any adjustment by netback, and the packet gets
> dropped.
> >
> > I think you need to set the checksum to the psuedo-header checksum
> > rather than 0 since that is what csum_blank means (such a well
> named
> > flag!)...
> >
> > It's not clear why anything would drop a UDP frame with
> checksum==0
> > since, as you say, it is optional. My guess is that since the
> > NETTXF_csum_blank and data_validated turn into an skb->ip_summed
> ==
> > SKB_PARTIAL on the backend side this causes the Linux network
> stack to
> > drop it because that statement conflicts with the checksum being
> 0.
> 
> Right, but I'm setting the IPv4 checksum to 0 too, since I want to
> offload the whole lot and never calculate a body checksum in the
> guest.  But if the UDP checksum is set to 0, then the backend can't
> just adjust it to obtain the IPv4 checksum and has to iterate over
> the packet body at some stage.
> 
> I've tried setting the UDP checksum to 0, the pseudoheader, and the
> full checksum, and the IPv4 checksum is never set by the backend in
> any of these cases.  If I set the IPv4 checksum in software to the
> correct value, then packets go through, but the UDP checksums are
> never altered.
> 
> Guess it's time to start slapping printfs all over the dom0 kernel
> to see what's going on unless you know what I "should" be setting
> both the IPv4/UDP checksums to with NETTXF_csum_blank...
> 
> Anil
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: ip/udp checksum offload from minios guest

[Anil Madhavapeddy]

Thanks, this is all clear now. I was not setting NETTXF_data_validated along with NETTXF_csum_blank in my transmit path, which was confusing the backend.

Ian: yes the IPv4 checksum is of course only over the header. I wrote the code right, and then re-read it wrong while hacking on a flight :) The ICMP errors confused me (since I was setting NETTXF_csum_blank for those too and the checksum functions error out).

On 29 Mar 2011, at 15:19, Paul Durrant wrote:

> Maybe there's some confusion of terms here... There are 2 checksums: the IPv4 header and UDP checksum.
> 
> The IPv4 header checksum must *always* be calculated by the frontend.
> 
> If NETTXF_csum_blank is set (implying NETTXF_data_validated must also be set) then the UDP checksum must be set to cover the UDP pseudo-header since the SKB will be marked CSUM_PARTIAL and thus any h/w driver it is presented to will expect the pseudo-header checksum to be valid. If the SKB is presented to another guest then CSUM_PARTIAL will translated into NETRXF_csum_blank|NETRXF_data_validated and the frontend is at liberty to present it up the stack as being checksum-valid without anything in the datapath having had to walk over the payload to actually calculate the value of that checksum.
> 
>  Paul
> 
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
>> bounces@lists.xensource.com] On Behalf Of Anil Madhavapeddy
>> Sent: 29 March 2011 18:16
>> To: Ian Campbell
>> Cc: xen-devel@lists.xensource.com
>> Subject: Re: [Xen-devel] ip/udp checksum offload from minios guest
>> 
>> On 29 Mar 2011, at 12:37, Ian Campbell wrote:
>> 
>>> On Tue, 2011-03-29 at 17:17 +0100, Anil Madhavapeddy wrote:
>>>> I'm just adding checksum offload support into a custom guest, and
>> wanted to clarify a few things.
>>>> 
>>>> In netfront, I can mark outgoing frames with:
>>>> - NETTXF_csum_blank
>>>> - NETTXF_data_validated
>>>> 
>>>> These refer to UDP or TCP checksums, and not the IP checksum,
>> right?
>>>> Linux seems to never offload IP header checksumming, so it must
>> be
>>>> offloading the UDP/TCP calculation and then adjusting the IPv4
>>>> checksum based on that calculation.
>>>> 
>>>> For outgoing UDP, my guest is setting the checksum to 0 (as it's
>>>> optional in the protocol), and calculating the full IPv4 checksum
>> in
>>>> software, and all works (slowly).  However, setting
>> NETTXF_csum_blank
>>>> in the outgoing frame and leaving the IPv4 checksum at 0 doesn't
>> seem
>>>> to result in any adjustment by netback, and the packet gets
>> dropped.
>>> 
>>> I think you need to set the checksum to the psuedo-header checksum
>>> rather than 0 since that is what csum_blank means (such a well
>> named
>>> flag!)...
>>> 
>>> It's not clear why anything would drop a UDP frame with
>> checksum==0
>>> since, as you say, it is optional. My guess is that since the
>>> NETTXF_csum_blank and data_validated turn into an skb->ip_summed
>> ==
>>> SKB_PARTIAL on the backend side this causes the Linux network
>> stack to
>>> drop it because that statement conflicts with the checksum being
>> 0.
>> 
>> Right, but I'm setting the IPv4 checksum to 0 too, since I want to
>> offload the whole lot and never calculate a body checksum in the
>> guest.  But if the UDP checksum is set to 0, then the backend can't
>> just adjust it to obtain the IPv4 checksum and has to iterate over
>> the packet body at some stage.
>> 
>> I've tried setting the UDP checksum to 0, the pseudoheader, and the
>> full checksum, and the IPv4 checksum is never set by the backend in
>> any of these cases.  If I set the IPv4 checksum in software to the
>> correct value, then packets go through, but the UDP checksums are
>> never altered.
>> 
>> Guess it's time to start slapping printfs all over the dom0 kernel
>> to see what's going on unless you know what I "should" be setting
>> both the IPv4/UDP checksums to with NETTXF_csum_blank...
>> 
>> Anil
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
> 

Re: ip/udp checksum offload from minios guest

[Ian Campbell]

On Tue, 2011-03-29 at 23:26 +0100, Anil Madhavapeddy wrote:
> Thanks, this is all clear now. I was not setting NETTXF_data_validated
> along with NETTXF_csum_blank in my transmit path, which was confusing
> the backend.

Right, that would have ended up confusing things, I think. c_s && !d_v
still ends up a CHECKSUM_PARTIAL in relatively recent backends. I'm not
sure that has always been implemented correctly though.

In recent netback there is fixup for the case where a GSO packet sets
csum_blank but not data_validated which recalculates the partial
checksum (since Linux requires that all GSO SKBs are CHECKSUM_PARTIAL to
simplify the software GSO checksum stuff). That's only in the GSO case
though since we can't detect it otherwise.

> Ian: yes the IPv4 checksum is of course only over the header. I wrote
> the code right, and then re-read it wrong while hacking on a flight :)

Easily done!

> The ICMP errors confused me (since I was setting NETTXF_csum_blank for
> those too and the checksum functions error out).

Yeah, don't do that ;-)

Ian.

RE: ip/udp checksum offload from minios guest

[James Harper]

[-- Attachment #1: Type: text/plain, Size: 1207 bytes --]

> 
> On Tue, 2011-03-29 at 23:26 +0100, Anil Madhavapeddy wrote:
> > Thanks, this is all clear now. I was not setting NETTXF_data_validated
> > along with NETTXF_csum_blank in my transmit path, which was confusing
> > the backend.
> 
> Right, that would have ended up confusing things, I think. c_s && !d_v
> still ends up a CHECKSUM_PARTIAL in relatively recent backends. I'm not
> sure that has always been implemented correctly though.
> 
> In recent netback there is fixup for the case where a GSO packet sets
> csum_blank but not data_validated which recalculates the partial
> checksum (since Linux requires that all GSO SKBs are CHECKSUM_PARTIAL to
> simplify the software GSO checksum stuff). That's only in the GSO case
> though since we can't detect it otherwise.
> 
> > Ian: yes the IPv4 checksum is of course only over the header. I wrote
> > the code right, and then re-read it wrong while hacking on a flight :)
> 
> Easily done!
> 
> > The ICMP errors confused me (since I was setting NETTXF_csum_blank for
> > those too and the checksum functions error out).
> 
> Yeah, don't do that ;-)
> 

Is IPv6 offload (gso and/or checksum) supported at this time?

James

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: ip/udp checksum offload from minios guest

[Ian Campbell]

On Wed, 2011-03-30 at 11:23 +0100, James Harper wrote:
> > 
> > On Tue, 2011-03-29 at 23:26 +0100, Anil Madhavapeddy wrote:
> > > Thanks, this is all clear now. I was not setting NETTXF_data_validated
> > > along with NETTXF_csum_blank in my transmit path, which was confusing
> > > the backend.
> > 
> > Right, that would have ended up confusing things, I think. c_s && !d_v
> > still ends up a CHECKSUM_PARTIAL in relatively recent backends. I'm not
> > sure that has always been implemented correctly though.
> > 
> > In recent netback there is fixup for the case where a GSO packet sets
> > csum_blank but not data_validated which recalculates the partial
> > checksum (since Linux requires that all GSO SKBs are CHECKSUM_PARTIAL to
> > simplify the software GSO checksum stuff). That's only in the GSO case
> > though since we can't detect it otherwise.
> > 
> > > Ian: yes the IPv4 checksum is of course only over the header. I wrote
> > > the code right, and then re-read it wrong while hacking on a flight :)
> > 
> > Easily done!
> > 
> > > The ICMP errors confused me (since I was setting NETTXF_csum_blank for
> > > those too and the checksum functions error out).
> > 
> > Yeah, don't do that ;-)
> > 
> 
> Is IPv6 offload (gso and/or checksum) supported at this time?

For GSO only TCPv4 is supported.

For checksum only TCPv4 and UDPv4 are supported.

Extending netback to cover more cases would be relatively trivial, I
think. e.g. extending checksum_setup and/or netbk_set_skb_gso in
netback.c, plus doing the opposite same on the guest RX paths.

I suspect most of the complexity will probably come from the correct
negotiation of the featureset via xenstore ;-)

Ian.

RE: ip/udp checksum offload from minios guest

[James Harper]

[-- Attachment #1: Type: text/plain, Size: 1063 bytes --]

> > Is IPv6 offload (gso and/or checksum) supported at this time?
> 
> For GSO only TCPv4 is supported.
> 
> For checksum only TCPv4 and UDPv4 are supported.
> 
> Extending netback to cover more cases would be relatively trivial, I
> think. e.g. extending checksum_setup and/or netbk_set_skb_gso in
> netback.c, plus doing the opposite same on the guest RX paths.
> 
> I suspect most of the complexity will probably come from the correct
> negotiation of the featureset via xenstore ;-)
> 

Yes. Given the problems I've seen under linux with offload it would be nice to have even more granularity than we do now. We'd definitely want to be able to turn offload on and off for all combinations of tcp/udp/csum/gso individually in case there is a problem somewhere.

The main problem I've seen is that Linux doesn't seem to understand that my network card supports offload functions but not in combination with VLAN, so only the untagged packets (eg vlan1) can be offloaded. That was back at 2.6.18 so hopefully the support is better now.

James

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: ip/udp checksum offload from minios guest

[Ian Campbell]

On Wed, 2011-03-30 at 11:37 +0100, James Harper wrote:
> > > Is IPv6 offload (gso and/or checksum) supported at this time?
> > 
> > For GSO only TCPv4 is supported.
> > 
> > For checksum only TCPv4 and UDPv4 are supported.
> > 
> > Extending netback to cover more cases would be relatively trivial, I
> > think. e.g. extending checksum_setup and/or netbk_set_skb_gso in
> > netback.c, plus doing the opposite same on the guest RX paths.
> > 
> > I suspect most of the complexity will probably come from the correct
> > negotiation of the featureset via xenstore ;-)
> > 
> 
> Yes. Given the problems I've seen under linux with offload it would be
> nice to have even more granularity than we do now. We'd definitely
> want to be able to turn offload on and off for all combinations of
> tcp/udp/csum/gso individually in case there is a problem somewhere.

Absolutely. Paul Durrant made some improvements in this area not so long
ago. Also from the looks of things upstream have recently separated out
the concept of what the h/w supports from what is actually currently
configured to be enabled which should be useful going forward.

> The main problem I've seen is that Linux doesn't seem to understand
> that my network card supports offload functions but not in combination
> with VLAN, so only the untagged packets (eg vlan1) can be offloaded.
> That was back at 2.6.18 so hopefully the support is better now.

I should hope so ;-)

In Linux a VLAN device is a new net device which sits on top of the
physical device and has it's own offload settings etc so at least there
is a route to fixing it there...

Ian