How exactly does CHECKSUM_COMPLETE works?

How exactly does CHECKSUM_COMPLETE works?

[Yuval Mintz]

Hi,

I was pondering what it would take for a network device support
CHECKSUM_COMPLETE on its Rx path. And I simply failed to understand the
specifics when using the napi_gro_receive() interface.

Scenario 1 - TCP/IPv4 packet. This will eventually reach tcp4_gro_receive(), and from
there into skb_gro_checksum_validate().
Since this SKB is marked as COMPLETE, the pseudo TCP checksum will be computed
and skb_gro_checksum_validate_complete() be called.
Now, it seems to me as if the only way by which the flow won't actually try to
re-calculate the checksum over TCP header and payload is if NAPI_GRO_CB(skb)->csum
is actually the one's complement to the TCP pseudo header. But that sounds.... odd,
since that's the csum that should have been listed on the SKB - and it doesn't protect
any of the data, only the "metadata".

Scenario 2 - IP-GRE packet, with CSUM option on the GRE header.
This scenario will also eventually reach skb_gro_checksum_validate_complete(),
this time via gre_gro_receive().
This one is even odder, since in order for the flow not to re-calculate the checksum
The driver should have set the SKB's csum to 0xffffffff [as it's using the
null_compute_pseudo() which will return a pseudo_csum of 0].

>From the comment in skbuff.h regarding CHECKSUM_COMPLETE it looks as if all
the HW needs to do is calculate the checksum over the entire data, starting at the
network header.

So, to summarize my questions -
1. What should a driver set as the SKBs csum value when passing CHECKSUM_COMPLETE?
2. Does the GRO flow for non-encapsulated packets work for packets marked as
CHECKSUM_COMPLETE? And by work, I obviously mean 'doesn't re-calculate checksums'.
3. Same as question 2, but for encapsulated packets.

Thanks,
Yuval



________________________________

This message and any attached documents contain information from QLogic Corporation or its wholly-owned subsidiaries that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.

Re: How exactly does CHECKSUM_COMPLETE works?

[Tom Herbert]

On Tue, Sep 30, 2014 at 5:51 AM, Yuval Mintz <Yuval.Mintz@qlogic.com> wrote:
> Hi,
>
> I was pondering what it would take for a network device support
> CHECKSUM_COMPLETE on its Rx path. And I simply failed to understand the
> specifics when using the napi_gro_receive() interface.
>
> Scenario 1 - TCP/IPv4 packet. This will eventually reach tcp4_gro_receive(), and from
> there into skb_gro_checksum_validate().
> Since this SKB is marked as COMPLETE, the pseudo TCP checksum will be computed
> and skb_gro_checksum_validate_complete() be called.
> Now, it seems to me as if the only way by which the flow won't actually try to
> re-calculate the checksum over TCP header and payload is if NAPI_GRO_CB(skb)->csum
> is actually the one's complement to the TCP pseudo header. But that sounds.... odd,
> since that's the csum that should have been listed on the SKB - and it doesn't protect
> any of the data, only the "metadata".
>
With CHECKSUM_COMPLETE, on entrance into GRO NAPI_GRO_CB(skb)->csum is
set to skb->csum. As the packet is parsed through various levels of
GRO the value is adjusted so that at any header level it is the
complete checksum starting from that header to the end of the packet.
So to validate a TCP checksum we just need to add the value of the
pseudo header checksum (a little computation) and compare to zero.

> Scenario 2 - IP-GRE packet, with CSUM option on the GRE header.
> This scenario will also eventually reach skb_gro_checksum_validate_complete(),
> this time via gre_gro_receive().
> This one is even odder, since in order for the flow not to re-calculate the checksum
> The driver should have set the SKB's csum to 0xffffffff [as it's using the
> null_compute_pseudo() which will return a pseudo_csum of 0].
>
> From the comment in skbuff.h regarding CHECKSUM_COMPLETE it looks as if all
> the HW needs to do is calculate the checksum over the entire data, starting at the
> network header.
>
GRE is actually simpler since there is no pseudo header. If a GRE
checksum bit is set then skb->csum (NAPI_GRO_CB(skb)->csum) must be
zero for a valid checksum at the time GRE header is processed and we
have CHECKSUM_COMPLETE.

> So, to summarize my questions -
> 1. What should a driver set as the SKBs csum value when passing CHECKSUM_COMPLETE?

This ones complete checksum of the Ethernet payload (start of IP
header to the end of the packet).

> 2. Does the GRO flow for non-encapsulated packets work for packets marked as
> CHECKSUM_COMPLETE? And by work, I obviously mean 'doesn't re-calculate checksums'.

Yes.

> 3. Same as question 2, but for encapsulated packets.
>
Yes.

Hope this helps,
Tom

> Thanks,
> Yuval
>
>
>
> ________________________________
>
> This message and any attached documents contain information from QLogic Corporation or its wholly-owned subsidiaries that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.

Re: How exactly does CHECKSUM_COMPLETE works?

[Alexei Starovoitov]

On Tue, Sep 30, 2014 at 7:59 AM, Tom Herbert <therbert@google.com> wrote:
>
>> So, to summarize my questions -
>> 1. What should a driver set as the SKBs csum value when passing CHECKSUM_COMPLETE?
>
> This ones complete checksum of the Ethernet payload (start of IP
> header to the end of the packet).

I think it's confusing to describe CHECKSUM_COMPLETE this way.
It is such only because driver pulls eth header before passing skb to stack.
CHECKSUM_COMPLETE should cover the whole packet. If there are multiple
vlan headers in front of IP they should be part of csum, since not all
drivers may
have hw offloading for vlan. The most simplistic HW would compute csum over
the whole packet including eth header and driver will do
eth_type_trans+skb_postpull_rcsum

Re: How exactly does CHECKSUM_COMPLETE works?

[David Miller]

From: Alexei Starovoitov <alexei.starovoitov@gmail.com>
Date: Tue, 30 Sep 2014 09:58:32 -0700

> On Tue, Sep 30, 2014 at 7:59 AM, Tom Herbert <therbert@google.com> wrote:
>>
>>> So, to summarize my questions -
>>> 1. What should a driver set as the SKBs csum value when passing CHECKSUM_COMPLETE?
>>
>> This ones complete checksum of the Ethernet payload (start of IP
>> header to the end of the packet).
> 
> I think it's confusing to describe CHECKSUM_COMPLETE this way.
> It is such only because driver pulls eth header before passing skb to stack.
> CHECKSUM_COMPLETE should cover the whole packet. If there are multiple
> vlan headers in front of IP they should be part of csum, since not all
> drivers may
> have hw offloading for vlan. The most simplistic HW would compute csum over
> the whole packet including eth header and driver will do
> eth_type_trans+skb_postpull_rcsum

The sungem computes the checksum over the entire frame.  Here are the relevant
parts from the programmer's manual:

	RX Descriptor Fields
  ...
	TCP Pseudo-Checksum - Contains the 16-bit TCP checksum calculated over
	the entire frame, starting at an offset programmable in the RX
	Configuration Register.

In the gengem driver this is the RXDMA_CFG_CSUMOFF field of the RXDMA_CFG
register.

The driver seems to set this to the size of the ethernet header, which has
the problems you mention.

RE: How exactly does CHECKSUM_COMPLETE works?

[Yuval Mintz]

> > Scenario 1 - TCP/IPv4 packet. This will eventually reach
> > tcp4_gro_receive(), and from there into skb_gro_checksum_validate().
> > Since this SKB is marked as COMPLETE, the pseudo TCP checksum will be
> > computed and skb_gro_checksum_validate_complete() be called.
> > Now, it seems to me as if the only way by which the flow won't
> > actually try to re-calculate the checksum over TCP header and payload
> > is if NAPI_GRO_CB(skb)->csum is actually the one's complement to the
> > TCP pseudo header. But that sounds.... odd, since that's the csum that
> > should have been listed on the SKB - and it doesn't protect any of the data,
> only the "metadata".
> >
> With CHECKSUM_COMPLETE, on entrance into GRO NAPI_GRO_CB(skb)->csum
> is set to skb->csum. As the packet is parsed through various levels of GRO the
> value is adjusted so that at any header level it is the complete checksum starting
> from that header to the end of the packet.
> So to validate a TCP checksum we just need to add the value of the pseudo
> header checksum (a little computation) and compare to zero.

...

>> 1. What should a driver set as the SKBs csum value when passing CHECKSUM_COMPLETE?
>This ones complete checksum of the Ethernet payload (start of IP header to the end of the packet).

Can you pinpoint me to what exactly in the code corrects NAPI_GRO_CB(skb)->csum
by negating the ipv4 part in the ones' complete checksum?

E.g., for ipv6 [in ipv6_gro_receive()] I can see that skb_gro_postpull_rcsum() is
called after pulling the ipv6 header [and before proceeding to the next protocol],
but I can't seem to find anything similar to this for ipv4 [in inet_gro_receive()].

________________________________

This message and any attached documents contain information from QLogic Corporation or its wholly-owned subsidiaries that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.

RE: How exactly does CHECKSUM_COMPLETE works?

[Eric Dumazet]

On Tue, 2014-09-30 at 19:05 +0000, Yuval Mintz wrote:

> Can you pinpoint me to what exactly in the code corrects NAPI_GRO_CB(skb)->csum
> by negating the ipv4 part in the ones' complete checksum?
> 
> E.g., for ipv6 [in ipv6_gro_receive()] I can see that skb_gro_postpull_rcsum() is
> called after pulling the ipv6 header [and before proceeding to the next protocol],
> but I can't seem to find anything similar to this for ipv4 [in inet_gro_receive()].
> 


IPv4 header is supposed to have a 0 checksum ;)

RE: How exactly does CHECKSUM_COMPLETE works?

[Eric Dumazet]

On Tue, 2014-09-30 at 12:22 -0700, Eric Dumazet wrote:
> On Tue, 2014-09-30 at 19:05 +0000, Yuval Mintz wrote:
> 
> > Can you pinpoint me to what exactly in the code corrects NAPI_GRO_CB(skb)->csum
> > by negating the ipv4 part in the ones' complete checksum?
> > 
> > E.g., for ipv6 [in ipv6_gro_receive()] I can see that skb_gro_postpull_rcsum() is
> > called after pulling the ipv6 header [and before proceeding to the next protocol],
> > but I can't seem to find anything similar to this for ipv4 [in inet_gro_receive()].
> > 
> 
> 
> IPv4 header is supposed to have a 0 checksum ;)
> 

Or if you prefer :

diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c
index 28e589c5f32d..92db7a69f2b9 100644
--- a/net/ipv4/af_inet.c
+++ b/net/ipv4/af_inet.c
@@ -1369,6 +1369,9 @@ static struct sk_buff **inet_gro_receive(struct sk_buff **head,
 	 * immediately following this IP hdr.
 	 */
 
+	/* Note : No need to call skb_gro_postpull_rcsum() here,
+	 * as we already checked checksum over ipv4 header was 0
+	 */
 	skb_gro_pull(skb, sizeof(*iph));
 	skb_set_transport_header(skb, skb_gro_offset(skb));
 

Re: How exactly does CHECKSUM_COMPLETE works?

[David Miller]

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Tue, 30 Sep 2014 12:37:12 -0700

> On Tue, 2014-09-30 at 12:22 -0700, Eric Dumazet wrote:
>> On Tue, 2014-09-30 at 19:05 +0000, Yuval Mintz wrote:
>> 
>> > Can you pinpoint me to what exactly in the code corrects NAPI_GRO_CB(skb)->csum
>> > by negating the ipv4 part in the ones' complete checksum?
>> > 
>> > E.g., for ipv6 [in ipv6_gro_receive()] I can see that skb_gro_postpull_rcsum() is
>> > called after pulling the ipv6 header [and before proceeding to the next protocol],
>> > but I can't seem to find anything similar to this for ipv4 [in inet_gro_receive()].
>> > 
>> 
>> 
>> IPv4 header is supposed to have a 0 checksum ;)
>> 
> 
> Or if you prefer :

Someone please submit this formally :)

[PATCH net-next] ipv4: mentions skb_gro_postpull_rcsum() in inet_gro_receive()

[Eric Dumazet]

From: Eric Dumazet <edumazet@google.com>

Proper CHECKSUM_COMPLETE support needs to adjust skb->csum
when we remove one header. Its done using skb_gro_postpull_rcsum()

In the case of IPv4, we know that the adjustment is not really needed,
because the checksum over IPv4 header is 0. Lets add a comment to
ease code comprehension and avoid copy/paste errors.

Signed-off-by: Eric Dumazet <edumazet@google.com>
---
 net/ipv4/af_inet.c |    3 +++
 1 file changed, 3 insertions(+)

diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c
index 28e589c5f32d..92db7a69f2b9 100644
--- a/net/ipv4/af_inet.c
+++ b/net/ipv4/af_inet.c
@@ -1369,6 +1369,9 @@ static struct sk_buff **inet_gro_receive(struct sk_buff **head,
 	 * immediately following this IP hdr.
 	 */
 
+	/* Note : No need to call skb_gro_postpull_rcsum() here,
+	 * as we already checked checksum over ipv4 header was 0
+	 */
 	skb_gro_pull(skb, sizeof(*iph));
 	skb_set_transport_header(skb, skb_gro_offset(skb));
 

Re: [PATCH net-next] ipv4: mentions skb_gro_postpull_rcsum() in inet_gro_receive()

[David Miller]

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Tue, 30 Sep 2014 22:12:05 -0700

> From: Eric Dumazet <edumazet@google.com>
> 
> Proper CHECKSUM_COMPLETE support needs to adjust skb->csum
> when we remove one header. Its done using skb_gro_postpull_rcsum()
> 
> In the case of IPv4, we know that the adjustment is not really needed,
> because the checksum over IPv4 header is 0. Lets add a comment to
> ease code comprehension and avoid copy/paste errors.
> 
> Signed-off-by: Eric Dumazet <edumazet@google.com>

Applied, thanks Eric.

RE: How exactly does CHECKSUM_COMPLETE works?

[Yuval Mintz]

> > Can you pinpoint me to what exactly in the code corrects NAPI_GRO_CB(skb)->csum
> > by negating the ipv4 part in the ones' complete checksum?
> >
> > E.g., for ipv6 [in ipv6_gro_receive()] I can see that skb_gro_postpull_rcsum() is
> > called after pulling the ipv6 header [and before proceeding to the next protocol],
> > but I can't seem to find anything similar to this for ipv4 [in inet_gro_receive()].
> >
>
>
> IPv4 header is supposed to have a 0 checksum ;)
>

Oops. Sorry, you're right.

But this is a bit strange.

I mean, if I go back to the IP-GRE case [with GRE checksum], the IP
csum negates the rest of the IP header and the GRE csum negates
everything else [GRE header, inner headers and payload].
So the mechanism might work as-is, but all the driver has to
do in order to claim that an IP-gre packet with checksum option is
CHECKSUM_COMPLETE is to write 0xffffffff on the skb's csum field.
 

RE: How exactly does CHECKSUM_COMPLETE works?

[Yuval Mintz]

>> IPv4 header is supposed to have a 0 checksum ;)
>>

>Oops. Sorry, you're right.

>But this is a bit strange.

>I mean, if I go back to the IP-GRE case [with GRE checksum], the IP
>csum negates the rest of the IP header and the GRE csum negates
>everything else [GRE header, inner headers and payload].
>So the mechanism might work as-is, but all the driver has to
>do in order to claim that an IP-gre packet with checksum option is
>CHECKSUM_COMPLETE is to write 0xffffffff on the skb's csum field.

Or looking back on the IPv4/TCP scenario, what exactly is the csum
now protecting?
Assume the HW passes the one's complement sum of the entire data,
IP header and onward to driver, which sets it in SKB->csum and marks
the SKB as CHECKSUM_COMPLETE.
In practice, SKB->csum should now equal to the one's complement of
the TCP pseudo header; But other than the length of data in the TCP
header and payload that pseudo-header doesn't include any
information about the payload itself.
If we're willing to accept the payload [and the correctness of the
header while we're at it] based solely on this, what guarantee do we
have that the TCP checksum is actually correct? 
Seems like the only thing we've guaranteed is that nothing switched in
IP header since packet passed the HW - but we've already got that
covered by the IPv4 layer.

[I assume I'm missing something elementary here. Same as before ;-)]

Re: How exactly does CHECKSUM_COMPLETE works?

[Tom Herbert]

On Tue, Sep 30, 2014 at 1:32 PM, Yuval Mintz <Yuval.Mintz@qlogic.com> wrote:
>>> IPv4 header is supposed to have a 0 checksum ;)
>>>
>
>>Oops. Sorry, you're right.
>
>>But this is a bit strange.
>
>>I mean, if I go back to the IP-GRE case [with GRE checksum], the IP
>>csum negates the rest of the IP header and the GRE csum negates
>>everything else [GRE header, inner headers and payload].
>>So the mechanism might work as-is, but all the driver has to
>>do in order to claim that an IP-gre packet with checksum option is
>>CHECKSUM_COMPLETE is to write 0xffffffff on the skb's csum field.
>
> Or looking back on the IPv4/TCP scenario, what exactly is the csum
> now protecting?
> Assume the HW passes the one's complement sum of the entire data,
> IP header and onward to driver, which sets it in SKB->csum and marks
> the SKB as CHECKSUM_COMPLETE.
> In practice, SKB->csum should now equal to the one's complement of
> the TCP pseudo header; But other than the length of data in the TCP
> header and payload that pseudo-header doesn't include any
> information about the payload itself.

But it does, as you pointed out the checksum calculation was performed
over the whole payload. The fact that the computed checksum equals the
one's complement of pseudo header is the indication that the checksum
is valid. If there were a bit corruption, the computed value would be
different (summing in pseudo header checksum would result in non-zero
value).

> If we're willing to accept the payload [and the correctness of the
> header while we're at it] based solely on this, what guarantee do we
> have that the TCP checksum is actually correct?
> Seems like the only thing we've guaranteed is that nothing switched in
> IP header since packet passed the HW - but we've already got that
> covered by the IPv4 layer.
>
> [I assume I'm missing something elementary here. Same as before ;-)]

I think your assumption is correct ;-) You might want to take a look
at some of the RFCs that describe checksum verification with pseudo
header.

RE: How exactly does CHECKSUM_COMPLETE works?

[Yuval Mintz]

> > Or looking back on the IPv4/TCP scenario, what exactly is the csum now
> > protecting?
> > Assume the HW passes the one's complement sum of the entire data, IP
> > header and onward to driver, which sets it in SKB->csum and marks the
> > SKB as CHECKSUM_COMPLETE.
> > In practice, SKB->csum should now equal to the one's complement of the
> > TCP pseudo header; But other than the length of data in the TCP header
> > and payload that pseudo-header doesn't include any information about
> > the payload itself.
> 
> But it does, as you pointed out the checksum calculation was performed over
> the whole payload. The fact that the computed checksum equals the one's
> complement of pseudo header is the indication that the checksum is valid. If
> there were a bit corruption, the computed value would be different (summing in
> pseudo header checksum would result in non-zero value).

Thanks, I've already reached that conclusion myself
[even wrote an E-mail; but it got blocked for some reason].

Re: How exactly does CHECKSUM_COMPLETE works?

[David Miller]

Yuval, the base64 encoding of your emails is not correct and many
sites are rejecting your postings that are actually making it to the
list based upon this fact.

Please post pure ASCII text to this mailing list unless that's
unavoidable (a UTF character in someone's name during a patch
submission f.e.).