* LRO/GRO and libpcap packet reordering
@ 2013-03-14 20:37 Andy Lutomirski
2013-03-14 21:58 ` Ben Hutchings
0 siblings, 1 reply; 9+ messages in thread
From: Andy Lutomirski @ 2013-03-14 20:37 UTC (permalink / raw)
To: Network Development
(I'm on Ubuntu's 3.5.0-23, but I haven't seen anything that would
change this behavior in newer kernels.)
I have a myri10ge device that's attached to a port mirror. It runs
tcpdump. Most of the traffic I'm capturing has another machine
attached to this switch as an endpoint. That machine is considerably
faster than the machine doing the capturing.
My captures show nasty artifacts: packets are reodered between a given
flow and the other direction of the same flow. The nasty case is when
an ACK shows up before the packet that it's acking. This thoroughly
screws up Wireshark's TCP sequencing analysis. Turning off LRO and
GRO fixes it.
Clearly, since this interface doesn't actually have an IP address,
there's no good reason to keep GRO and LRO on. Nonetheless, it would
be nice if GRO didn't coalesce packets when there's an intervening
packing in the other direction on the same flow. Can this be done
cheaply?
--Andy
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: LRO/GRO and libpcap packet reordering
2013-03-14 20:37 LRO/GRO and libpcap packet reordering Andy Lutomirski
@ 2013-03-14 21:58 ` Ben Hutchings
2013-03-14 22:45 ` Andy Lutomirski
0 siblings, 1 reply; 9+ messages in thread
From: Ben Hutchings @ 2013-03-14 21:58 UTC (permalink / raw)
To: Andy Lutomirski; +Cc: Network Development
On Thu, 2013-03-14 at 13:37 -0700, Andy Lutomirski wrote:
> (I'm on Ubuntu's 3.5.0-23, but I haven't seen anything that would
> change this behavior in newer kernels.)
>
> I have a myri10ge device that's attached to a port mirror. It runs
> tcpdump. Most of the traffic I'm capturing has another machine
> attached to this switch as an endpoint. That machine is considerably
> faster than the machine doing the capturing.
>
> My captures show nasty artifacts: packets are reodered between a given
> flow and the other direction of the same flow. The nasty case is when
> an ACK shows up before the packet that it's acking. This thoroughly
> screws up Wireshark's TCP sequencing analysis. Turning off LRO and
> GRO fixes it.
>
> Clearly, since this interface doesn't actually have an IP address,
> there's no good reason to keep GRO and LRO on. Nonetheless, it would
> be nice if GRO didn't coalesce packets when there's an intervening
> packing in the other direction on the same flow. Can this be done
> cheaply?
No, it would not be cheap.
You'll probably have to disable GRO, LRO and also RSS (unless you can
configure the RSS hash function to produce the same result for both
directions of a flow).
Ben.
--
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: LRO/GRO and libpcap packet reordering
2013-03-14 21:58 ` Ben Hutchings
@ 2013-03-14 22:45 ` Andy Lutomirski
2013-03-14 22:54 ` Eric Dumazet
2013-03-14 23:01 ` Ben Hutchings
0 siblings, 2 replies; 9+ messages in thread
From: Andy Lutomirski @ 2013-03-14 22:45 UTC (permalink / raw)
To: Ben Hutchings; +Cc: Network Development
On Thu, Mar 14, 2013 at 2:58 PM, Ben Hutchings
<bhutchings@solarflare.com> wrote:
> On Thu, 2013-03-14 at 13:37 -0700, Andy Lutomirski wrote:
>> (I'm on Ubuntu's 3.5.0-23, but I haven't seen anything that would
>> change this behavior in newer kernels.)
>>
>> I have a myri10ge device that's attached to a port mirror. It runs
>> tcpdump. Most of the traffic I'm capturing has another machine
>> attached to this switch as an endpoint. That machine is considerably
>> faster than the machine doing the capturing.
>>
>> My captures show nasty artifacts: packets are reodered between a given
>> flow and the other direction of the same flow. The nasty case is when
>> an ACK shows up before the packet that it's acking. This thoroughly
>> screws up Wireshark's TCP sequencing analysis. Turning off LRO and
>> GRO fixes it.
>>
>> Clearly, since this interface doesn't actually have an IP address,
>> there's no good reason to keep GRO and LRO on. Nonetheless, it would
>> be nice if GRO didn't coalesce packets when there's an intervening
>> packing in the other direction on the same flow. Can this be done
>> cheaply?
>
> No, it would not be cheap.
>
Hmm. What if the GRO flow hashing was something like hash(one
endpoint) ^ hash(the other endpoint)? (NB: I don't really know what
I'm talking about.)
> You'll probably have to disable GRO, LRO and also RSS (unless you can
> configure the RSS hash function to produce the same result for both
> directions of a flow).
>
RSS was already off. That was the first thing I tried.
--Andy
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: LRO/GRO and libpcap packet reordering
2013-03-14 22:45 ` Andy Lutomirski
@ 2013-03-14 22:54 ` Eric Dumazet
2013-03-14 22:58 ` Andy Lutomirski
2013-03-14 23:01 ` Ben Hutchings
1 sibling, 1 reply; 9+ messages in thread
From: Eric Dumazet @ 2013-03-14 22:54 UTC (permalink / raw)
To: Andy Lutomirski; +Cc: Ben Hutchings, Network Development
On Thu, 2013-03-14 at 15:45 -0700, Andy Lutomirski wrote:
> Hmm. What if the GRO flow hashing was something like hash(one
> endpoint) ^ hash(the other endpoint)? (NB: I don't really know what
> I'm talking about.)
There is no hashing in GRO, but a full flow match.
What you describe has to do with multi queue capability of the NIC.
Each flow is handled by a separate RX queue, therefore by different
cpus. There is no guarantee packets are delivered 'in order' in this
case, GRO being off or on.
Of course, if GRO/LRO is on, there is an additional delay to permit
aggregation, so a big GRO/LRO packet might appear _after_ an ACK.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: LRO/GRO and libpcap packet reordering
2013-03-14 22:54 ` Eric Dumazet
@ 2013-03-14 22:58 ` Andy Lutomirski
2013-03-14 23:16 ` Eric Dumazet
0 siblings, 1 reply; 9+ messages in thread
From: Andy Lutomirski @ 2013-03-14 22:58 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Ben Hutchings, Network Development
On Thu, Mar 14, 2013 at 3:54 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Thu, 2013-03-14 at 15:45 -0700, Andy Lutomirski wrote:
>
>> Hmm. What if the GRO flow hashing was something like hash(one
>> endpoint) ^ hash(the other endpoint)? (NB: I don't really know what
>> I'm talking about.)
>
> There is no hashing in GRO, but a full flow match.
>
> What you describe has to do with multi queue capability of the NIC.
>
> Each flow is handled by a separate RX queue, therefore by different
> cpus. There is no guarantee packets are delivered 'in order' in this
> case, GRO being off or on.
This happens even when there's just one rx queue. (I checked that
before scratching my head for a while.)
>
> Of course, if GRO/LRO is on, there is an additional delay to permit
> aggregation, so a big GRO/LRO packet might appear _after_ an ACK.
>
>
Right. This is the part that I'm wondering about an efficient fix to
(i.e. flush GRO state for one flow whenever a packet for the reversed
flow is seen).
--Andy
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: LRO/GRO and libpcap packet reordering
2013-03-14 22:58 ` Andy Lutomirski
@ 2013-03-14 23:16 ` Eric Dumazet
2013-03-14 23:21 ` Eric Dumazet
0 siblings, 1 reply; 9+ messages in thread
From: Eric Dumazet @ 2013-03-14 23:16 UTC (permalink / raw)
To: Andy Lutomirski; +Cc: Ben Hutchings, Network Development
On Thu, 2013-03-14 at 15:58 -0700, Andy Lutomirski wrote:
> Right. This is the part that I'm wondering about an efficient fix to
> (i.e. flush GRO state for one flow whenever a packet for the reversed
> flow is seen).
It doesnt seem doable easily, because network header and tcp headers are
handled in two separate functions in GRO stack (inet_gro_receive() &
tcp_gro_receive()
What you could do is changing MAX_GRO_SKBS from 8 to 1
(basically allowing only one flow in GRO stack)
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: LRO/GRO and libpcap packet reordering
2013-03-14 23:16 ` Eric Dumazet
@ 2013-03-14 23:21 ` Eric Dumazet
2013-03-14 23:22 ` Andy Lutomirski
0 siblings, 1 reply; 9+ messages in thread
From: Eric Dumazet @ 2013-03-14 23:21 UTC (permalink / raw)
To: Andy Lutomirski; +Cc: Ben Hutchings, Network Development
On Fri, 2013-03-15 at 00:16 +0100, Eric Dumazet wrote:
> On Thu, 2013-03-14 at 15:58 -0700, Andy Lutomirski wrote:
>
> > Right. This is the part that I'm wondering about an efficient fix to
> > (i.e. flush GRO state for one flow whenever a packet for the reversed
> > flow is seen).
>
> It doesnt seem doable easily, because network header and tcp headers are
> handled in two separate functions in GRO stack (inet_gro_receive() &
> tcp_gro_receive()
>
> What you could do is changing MAX_GRO_SKBS from 8 to 1
> (basically allowing only one flow in GRO stack)
>
>
Well, its not enough, some changes would be needed to force a flush each
time a new flow is detected.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: LRO/GRO and libpcap packet reordering
2013-03-14 23:21 ` Eric Dumazet
@ 2013-03-14 23:22 ` Andy Lutomirski
0 siblings, 0 replies; 9+ messages in thread
From: Andy Lutomirski @ 2013-03-14 23:22 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Ben Hutchings, Network Development
On Thu, Mar 14, 2013 at 4:21 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Fri, 2013-03-15 at 00:16 +0100, Eric Dumazet wrote:
>> On Thu, 2013-03-14 at 15:58 -0700, Andy Lutomirski wrote:
>>
>> > Right. This is the part that I'm wondering about an efficient fix to
>> > (i.e. flush GRO state for one flow whenever a packet for the reversed
>> > flow is seen).
>>
>> It doesnt seem doable easily, because network header and tcp headers are
>> handled in two separate functions in GRO stack (inet_gro_receive() &
>> tcp_gro_receive()
>>
>> What you could do is changing MAX_GRO_SKBS from 8 to 1
>> (basically allowing only one flow in GRO stack)
>>
>>
> Well, its not enough, some changes would be needed to force a flush each
> time a new flow is detected.
>
>
>
Hmm. Maybe a better "fix" would be to have some up-to-date
documentation or other indication of what needs to happen to get clean
packet captures. I'm not sure where it should go.
--Andy
--
Andy Lutomirski
AMA Capital Management, LLC
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: LRO/GRO and libpcap packet reordering
2013-03-14 22:45 ` Andy Lutomirski
2013-03-14 22:54 ` Eric Dumazet
@ 2013-03-14 23:01 ` Ben Hutchings
1 sibling, 0 replies; 9+ messages in thread
From: Ben Hutchings @ 2013-03-14 23:01 UTC (permalink / raw)
To: Andy Lutomirski; +Cc: Network Development
On Thu, 2013-03-14 at 15:45 -0700, Andy Lutomirski wrote:
> On Thu, Mar 14, 2013 at 2:58 PM, Ben Hutchings
> <bhutchings@solarflare.com> wrote:
> > On Thu, 2013-03-14 at 13:37 -0700, Andy Lutomirski wrote:
> >> (I'm on Ubuntu's 3.5.0-23, but I haven't seen anything that would
> >> change this behavior in newer kernels.)
> >>
> >> I have a myri10ge device that's attached to a port mirror. It runs
> >> tcpdump. Most of the traffic I'm capturing has another machine
> >> attached to this switch as an endpoint. That machine is considerably
> >> faster than the machine doing the capturing.
> >>
> >> My captures show nasty artifacts: packets are reodered between a given
> >> flow and the other direction of the same flow. The nasty case is when
> >> an ACK shows up before the packet that it's acking. This thoroughly
> >> screws up Wireshark's TCP sequencing analysis. Turning off LRO and
> >> GRO fixes it.
> >>
> >> Clearly, since this interface doesn't actually have an IP address,
> >> there's no good reason to keep GRO and LRO on. Nonetheless, it would
> >> be nice if GRO didn't coalesce packets when there's an intervening
> >> packing in the other direction on the same flow. Can this be done
> >> cheaply?
> >
> > No, it would not be cheap.
> >
>
> Hmm. What if the GRO flow hashing was something like hash(one
> endpoint) ^ hash(the other endpoint)? (NB: I don't really know what
> I'm talking about.)
GRO doesn't use a flow hash; it performs an exact comparison of the
identifying fields. (And only holds aggregated skbs for up to 8 flows
at a time.)
Ben.
> > You'll probably have to disable GRO, LRO and also RSS (unless you can
> > configure the RSS hash function to produce the same result for both
> > directions of a flow).
> >
>
> RSS was already off. That was the first thing I tried.
>
> --Andy
--
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2013-03-14 23:23 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-03-14 20:37 LRO/GRO and libpcap packet reordering Andy Lutomirski
2013-03-14 21:58 ` Ben Hutchings
2013-03-14 22:45 ` Andy Lutomirski
2013-03-14 22:54 ` Eric Dumazet
2013-03-14 22:58 ` Andy Lutomirski
2013-03-14 23:16 ` Eric Dumazet
2013-03-14 23:21 ` Eric Dumazet
2013-03-14 23:22 ` Andy Lutomirski
2013-03-14 23:01 ` Ben Hutchings
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox