Re: [net-next PATCH v2 5/9] mlx4: Add support for UDP tunnel segmentation with outer checksum offload

[Or Gerlitz <gerlitz.or@gmail.com>]

On Wed, May 4, 2016 at 7:06 PM, Alex Duyck <aduyck@mirantis.com> wrote:
> On Wed, May 4, 2016 at 8:50 AM, Or Gerlitz <ogerlitz@mellanox.com> wrote:
>> On 5/3/2016 6:29 PM, Alexander Duyck wrote:
>>>
>>> We split the one that would be a different size off via GSO.  So we
>>> end up sending up 2 frames to the device if there is going to be one
>>> piece that doesn't quite match.  We split that one piece off via GSO.
>>> That is one of the reasons why I referred to it as partial GSO as all
>>> we are using the software segmentation code for is to make sure we
>>> have the GSO block consists of segments that are all the same size.
>>
>>
>> I see, so if somehow it happens a lot that the TCP stack sends down
>> something which once segmented ends up with the last segment being of
>> different size from the other ones we would have to call the NIC xmit
>> function twice (BTW can we use xmit_more here?)  -- which could be effecting
>> performance, I guess.
>>
>> GSO_UDP_TUNNEL_CSUM (commit  0f4f4ffa7 "net: Add GSO support for UDP tunnels
>> with checksum") came to mark "that a device is capable of computing the UDP
>> checksum in the encapsulating header of a UDP tunnel" -- and the way we use
>> it here is that we do advertize that bit towards the stack for devices whose
>> HW can **not** do that, and things work b/c of LCO (this is my
>> understanding).
>>
>> I miss something in the bigger picture here, what does this buy us? e.g vs
>> just letting this (say) vxlan tunnel use zero checksum on the outer UDP
>> packet, is that has something to do with RCO?
>
> I think the piece you are missing is GSO_PARTIAL.  Basically
> GSO_PARTIAL indicates that we can perform GSO as long as all segments
> are the same size and also allows for ignoring one level of headers.
> So in the case of ixgbe for instance we can support tunnel offloads as
> long as we allow for the inner IPv4 ID to be a fixed value which is
> identified by enabling TSO_MANGLEID.  In the case of i40e, mlx4, and
> mlx5 the key bit is that we just have to have the frames the same size
> for all segments and then we can support tunnels with outer checksum
> because the checksum has been computed once and can be applied to all
> of the segmented frames.

Yep, I think to basically follow on the PARTIAL thing, which once
advertised by i40e, mlx4 and mlx5 allow them support udp (and GRE in
i40e case) tunnels with outer checksum.

My question was what this buy us for the UDP case vs. using zero
checksum for the tunnel (outer packet), I tried to figure out if it
has something to do with the remote side, e.g for RCO or alike.
Basically, under PARTIAL, on the worst case we could have ending up
with 2x packet xmitted to the NIC - e.g if each TCP message which is
to be encapsulated by the stack and later segmented by the NIC HW is
broken to two b/c otherwise the last segmented packet will not be of
equal size as of the all the preceding ones.

Or being a bit more positive... is there an expected performance  gain
when you use MANGLEID and/or PARTIAL to enable supporting UDP tunnel
segmentation checksum offload towards the stack? what is the reason
for that gain?

As for GRE tunnel segmentation checksum offload, I saw in your i40e
patch that it made your testbed to go from 12Gbs to 20Gbs, is this b/c
the stack can not actually let the HW do the segmentation w.o checksum
offload? if not, can you help understand the source of the gain?

> Hope that helps.

yes, your notes are very helpful, thanks for sparing the time..

Or.