Re: [PATCH 0/2] fix kernel crash with macvtap on top of LRO

Re: [PATCH 0/2] fix kernel crash with macvtap on top of LRO

[Michael S. Tsirkin]

On Thu, Feb 07, 2013 at 01:14:20PM -0500, David Miller wrote:
> From: Ben Hutchings <bhutchings@solarflare.com>
> Date: Thu, 7 Feb 2013 16:20:46 +0000
> 
> > If the consensus is still that we must preserve packets exactly (aside
> > from the usual modifications by IP routers) then LRO should be disabled
> > on all devices for which forwarding is enabled.
> 
> I believe this is still undoubtedly the consensus.

With virtio we are getting packets from a linux host,
so we could thinkably preserve packets exactly
even with LRO. I am guessing other hardware could be
doing this as well.

I am not sure what information would need to be preserved -
could someone help clarify please?

-- 
MST

Re: [PATCH 0/2] fix kernel crash with macvtap on top of LRO

[Ben Hutchings]

On Mon, 2013-06-10 at 10:07 +0300, Michael S. Tsirkin wrote:
> On Thu, Feb 07, 2013 at 01:14:20PM -0500, David Miller wrote:
> > From: Ben Hutchings <bhutchings@solarflare.com>
> > Date: Thu, 7 Feb 2013 16:20:46 +0000
> > 
> > > If the consensus is still that we must preserve packets exactly (aside
> > > from the usual modifications by IP routers) then LRO should be disabled
> > > on all devices for which forwarding is enabled.
> > 
> > I believe this is still undoubtedly the consensus.
> 
> With virtio we are getting packets from a linux host,
> so we could thinkably preserve packets exactly
> even with LRO. I am guessing other hardware could be
> doing this as well.
> 
> I am not sure what information would need to be preserved -
> could someone help clarify please?

Some LRO implementations may not preserve:

- Packet boundaries
  - TSO/GSO produces packets all the same size, except possibly for the
    last one.  GRO therefore flushes a flow after merging a packet with
    a different segment size.
- IPv4 TTL, IPv6 hop-limit, TCP timestamp
  - TSO/GSO will put the same values in all packets.  GRO flushes a flow
    if they change.
- IPv4 fragment ID
  - TSO/GSO produces consecutive fragment IDs.  GRO flushes a flow
    if it sees a non-consecutive fragment ID.
- MAC header, IPv4 TOS, IPv6 traffic class
  - Should be the same for all packets in a flow.  GRO actually checks
    and flushes a flow if they change.

Ben.

-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: [PATCH 0/2] fix kernel crash with macvtap on top of LRO

[Michael S. Tsirkin]

On Wed, Jun 12, 2013 at 03:56:36PM +0100, Ben Hutchings wrote:
> On Mon, 2013-06-10 at 10:07 +0300, Michael S. Tsirkin wrote:
> > On Thu, Feb 07, 2013 at 01:14:20PM -0500, David Miller wrote:
> > > From: Ben Hutchings <bhutchings@solarflare.com>
> > > Date: Thu, 7 Feb 2013 16:20:46 +0000
> > > 
> > > > If the consensus is still that we must preserve packets exactly (aside
> > > > from the usual modifications by IP routers) then LRO should be disabled
> > > > on all devices for which forwarding is enabled.
> > > 
> > > I believe this is still undoubtedly the consensus.
> > 
> > With virtio we are getting packets from a linux host,
> > so we could thinkably preserve packets exactly
> > even with LRO. I am guessing other hardware could be
> > doing this as well.
> > 
> > I am not sure what information would need to be preserved -
> > could someone help clarify please?
> 
> Some LRO implementations may not preserve:
> 
> - Packet boundaries
>   - TSO/GSO produces packets all the same size, except possibly for the
>     last one.  GRO therefore flushes a flow after merging a packet with
>     a different segment size.
> - IPv4 TTL, IPv6 hop-limit, TCP timestamp
>   - TSO/GSO will put the same values in all packets.  GRO flushes a flow
>     if they change.
> - IPv4 fragment ID
>   - TSO/GSO produces consecutive fragment IDs.  GRO flushes a flow
>     if it sees a non-consecutive fragment ID.
> - MAC header, IPv4 TOS, IPv6 traffic class
>   - Should be the same for all packets in a flow.  GRO actually checks
>     and flushes a flow if they change.
> 
> Ben.

Thanks!

Okay so since LRO in virtio is actually running on top
of GRO/TSO in linux, looks like the only things worth preserving
that we don't preserve at the moment are the packet
boundaries, for which it's enough to report
the first segment size to guest - we have the total length,
from that we can calculate the last segment size.

> -- 
> Ben Hutchings, Staff Engineer, Solarflare
> Not speaking for my employer; that's the marketing department's job.
> They asked us to note that Solarflare product names are trademarked.

Re: [PATCH 0/2] fix kernel crash with macvtap on top of LRO

[Rusty Russell]

"Michael S. Tsirkin" <mst@redhat.com> writes:
> On Wed, Jun 12, 2013 at 03:56:36PM +0100, Ben Hutchings wrote:
>> On Mon, 2013-06-10 at 10:07 +0300, Michael S. Tsirkin wrote:
>> > On Thu, Feb 07, 2013 at 01:14:20PM -0500, David Miller wrote:
>> > > From: Ben Hutchings <bhutchings@solarflare.com>
>> > > Date: Thu, 7 Feb 2013 16:20:46 +0000
>> > > 
>> > > > If the consensus is still that we must preserve packets exactly (aside
>> > > > from the usual modifications by IP routers) then LRO should be disabled
>> > > > on all devices for which forwarding is enabled.
>> > > 
>> > > I believe this is still undoubtedly the consensus.
>> > 
>> > With virtio we are getting packets from a linux host,
>> > so we could thinkably preserve packets exactly
>> > even with LRO. I am guessing other hardware could be
>> > doing this as well.
>> > 
>> > I am not sure what information would need to be preserved -
>> > could someone help clarify please?
>> 
>> Some LRO implementations may not preserve:
>> 
>> - Packet boundaries
>>   - TSO/GSO produces packets all the same size, except possibly for the
>>     last one.  GRO therefore flushes a flow after merging a packet with
>>     a different segment size.
>> - IPv4 TTL, IPv6 hop-limit, TCP timestamp
>>   - TSO/GSO will put the same values in all packets.  GRO flushes a flow
>>     if they change.
>> - IPv4 fragment ID
>>   - TSO/GSO produces consecutive fragment IDs.  GRO flushes a flow
>>     if it sees a non-consecutive fragment ID.
>> - MAC header, IPv4 TOS, IPv6 traffic class
>>   - Should be the same for all packets in a flow.  GRO actually checks
>>     and flushes a flow if they change.
>> 
>> Ben.
>
> Thanks!
>
> Okay so since LRO in virtio is actually running on top
> of GRO/TSO in linux, looks like the only things worth preserving
> that we don't preserve at the moment are the packet
> boundaries, for which it's enough to report
> the first segment size to guest - we have the total length,
> from that we can calculate the last segment size.

Isn't this reflected in virtio_net_hdr->gso_size?

But the bigger point here is that we shouldn't be feeding guests LRO
packets which don't meet the stricter GRO requirements, as we don't know
what the guest is doing with them.  It might be forwarding them itself.

I thought LRO was deprecated and GRO was the new hotness, but I haven't
been following.  Do we still care about LRO?

Cheers,
Rusty.

Re: [PATCH 0/2] fix kernel crash with macvtap on top of LRO

[Ben Hutchings]

On Mon, 2013-06-17 at 11:05 +0930, Rusty Russell wrote:
> "Michael S. Tsirkin" <mst@redhat.com> writes:
> > On Wed, Jun 12, 2013 at 03:56:36PM +0100, Ben Hutchings wrote:
> >> On Mon, 2013-06-10 at 10:07 +0300, Michael S. Tsirkin wrote:
> >> > On Thu, Feb 07, 2013 at 01:14:20PM -0500, David Miller wrote:
> >> > > From: Ben Hutchings <bhutchings@solarflare.com>
> >> > > Date: Thu, 7 Feb 2013 16:20:46 +0000
> >> > > 
> >> > > > If the consensus is still that we must preserve packets exactly (aside
> >> > > > from the usual modifications by IP routers) then LRO should be disabled
> >> > > > on all devices for which forwarding is enabled.
> >> > > 
> >> > > I believe this is still undoubtedly the consensus.
> >> > 
> >> > With virtio we are getting packets from a linux host,
> >> > so we could thinkably preserve packets exactly
> >> > even with LRO. I am guessing other hardware could be
> >> > doing this as well.
> >> > 
> >> > I am not sure what information would need to be preserved -
> >> > could someone help clarify please?
> >> 
> >> Some LRO implementations may not preserve:
> >> 
> >> - Packet boundaries
> >>   - TSO/GSO produces packets all the same size, except possibly for the
> >>     last one.  GRO therefore flushes a flow after merging a packet with
> >>     a different segment size.
> >> - IPv4 TTL, IPv6 hop-limit, TCP timestamp
> >>   - TSO/GSO will put the same values in all packets.  GRO flushes a flow
> >>     if they change.
> >> - IPv4 fragment ID
> >>   - TSO/GSO produces consecutive fragment IDs.  GRO flushes a flow
> >>     if it sees a non-consecutive fragment ID.
> >> - MAC header, IPv4 TOS, IPv6 traffic class
> >>   - Should be the same for all packets in a flow.  GRO actually checks
> >>     and flushes a flow if they change.
> >> 
> >> Ben.
> >
> > Thanks!
> >
> > Okay so since LRO in virtio is actually running on top
> > of GRO/TSO in linux, looks like the only things worth preserving
> > that we don't preserve at the moment are the packet
> > boundaries, for which it's enough to report
> > the first segment size to guest - we have the total length,
> > from that we can calculate the last segment size.
> 
> Isn't this reflected in virtio_net_hdr->gso_size?
> 
> But the bigger point here is that we shouldn't be feeding guests LRO
> packets which don't meet the stricter GRO requirements, as we don't know
> what the guest is doing with them.  It might be forwarding them itself.
> 
> I thought LRO was deprecated and GRO was the new hotness, but I haven't
> been following.  Do we still care about LRO?

The old software LRO implementation, inet_lro, is deprecated in favour
of GRO and is now only used by one or two drivers.  Hardware/firmware
implementations of LRO are still in use and not deprecated, but we try
to disable them on devices for which forwarding is enabled because of
this information loss.

The problem I was talking about is this: you can put macvlan on top of a
device that has LRO enabled, and then if the macvtap/macvlan device is
used for forwarding the output packets might not look the same as those
originally received.  So LRO should be disabled on the underlying device
whenever forwarding is enabled on the macvtap/macvlan device; however we
can't necessarily tell when that happens as the forwarding might be done
inside a VM.  Maybe this is just too obscure a use case to worry much
about getting it right automatically.

Ben.

-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: [PATCH 0/2] fix kernel crash with macvtap on top of LRO

[Rusty Russell]

Ben Hutchings <bhutchings@solarflare.com> writes:
> On Mon, 2013-06-17 at 11:05 +0930, Rusty Russell wrote:
>> I thought LRO was deprecated and GRO was the new hotness, but I haven't
>> been following.  Do we still care about LRO?
>
> The old software LRO implementation, inet_lro, is deprecated in favour
> of GRO and is now only used by one or two drivers.  Hardware/firmware
> implementations of LRO are still in use and not deprecated, but we try
> to disable them on devices for which forwarding is enabled because of
> this information loss.

Right, thanks for the clarification.

Hardware implementations of LRO which can't meet GRO rules are only
semi-useful, and that should be fed back to vendors.  Hard.

> The problem I was talking about is this: you can put macvlan on top of a
> device that has LRO enabled, and then if the macvtap/macvlan device is
> used for forwarding the output packets might not look the same as those
> originally received.  So LRO should be disabled on the underlying device
> whenever forwarding is enabled on the macvtap/macvlan device; however we
> can't necessarily tell when that happens as the forwarding might be done
> inside a VM.  Maybe this is just too obscure a use case to worry much
> about getting it right automatically.

The VM needs to tell us it's OK with such mangling, otherwise we
shouldn't do it (at least by default).  The same way we'd be annoyed if
a card rev started doing LRO without the driver explicitly enabling it.

Cheers,
Rusty.