sky2 DHCPOFFER packet loss under load (Was Re: [PATCH] sky2: safer transmit ring cleaning (v4))

[Michael Breuer <mbreuer@majjas.com>]

On 1/14/2010 6:51 PM, Michael Breuer wrote:
> On 1/14/2010 12:52 PM, Stephen Hemminger wrote:
>> On Thu, 14 Jan 2010 10:14:45 +0000
>> Jarek Poplawski<jarkao2@gmail.com>  wrote:
>>
>>> This makes it safe, but it still resembles the "short term fix"
>>> according do David's opinion.
>>>
>>> This change seems to affect dev->stats too. Since they are not
>>> updated in sky2_tx_clean(). Btw, I hope "&" is some optimization
>>> because it's less readable than "&&".
>> Stats don't matter for packets flushed during device reset.
>>
>> The&  is because in the most common case device is up,
>> and we don't want the additional conditional branch.
> I've been looking at what might explain the dhcp stuff - as well as 
> the dropped packets only when there's an extra hop. I came across one 
> path that seems suspect - although I'm really not familiar with the 
> network stack code... that said, I'm wondering about 
> neigh_compat_output (and eth_rebuild_header and arp_find). If I'm 
> following things correctly (or perhaps mostly correctly), the only 
> time anything goes this route (pun intentional) is when the packet was 
> routed to this box. I'm guessing that bridging makes this more likely. 
> So my dhcp stuff would all be going through here, as would the smb 
> stuff that seemed flaky. The race I'm seeing (maybe) is that when the 
> arp table is being rebuilt, there's a possibility that arp_find frees 
> the skb. There's some other locking and stuff going on that seems 
> maybe races with sky2.c in places on both the rx and tx path. I 
> *think* it's right from looking at it, but test results suggest 
> otherwise. Aside from the potential race, I think there's also a 
> corner case where neigh_compat_output can return either with or 
> without freeing the skb depending on the return from 
> dev_hard_header... this may also be part of the race.
>
> Maybe I've missed something... but as far as I can see, this is just 
> about the only difference in code path taken between stuff that is 
> working and stuff that is occasionally not.
Ok - brief update. I've confirmed that under load, outgoing DHCPOFFER 
packets are being silently dropped. I don't know yet where.

Test scenario, what I do know, etc.:

Scenario:

Two systems; one Gb switch; one wifi router; one Blackberry client.

System A: Linux host; Asus P6T Deluxe V2/Sky2. eth1-> internet eth0-> 
internal (10.0.0.1/24).
Switch: HP Procurve unmanaged - one port connected to System A; another 
to the wifi router; another to System B.
System B: Win7; Asus M2N Deluxe SLI/ Nforce 5 (10.0.0.11)
Router: Wrt54g-tm (dd-wrt) Connected to switch & various wifi clients 
including a Blackberry. WEP enabled. (10.0.0.60)
Blackberry Curve 8320 (wifi enabled). (10.0.0.56 via dhcp lease)

Test that causes packet loss:

1. Turn off BB wifi.
2. Start copy of large files (4GB) from System B to an CIFS share on 
System A.
3. Start nethogs on system A.
4. Start tcpdump on the wifi router (interface br0 - wired 10.0.0.60 
connection)
5. Start wireshark on System A
6. tail system A /var/log/messages - watching for DHCP activity
7. When smb traffic load (incoming) exceeds 40,000KBPS (nethogs) - 
enable wifi on the blackberry.
8. Stop test after multiple DHCPDISCOVER/OFFER observed without REQUEST/ACK.

Results:

1. It seems that the problem occurs intermittently below 40,000KBPS, and 
consistently over that number as reported by nethogs. Lots of 
fluctuation, so figure that the 40k is approximate.
2. wireshark (system A) shows DHCPOFFER traffic outgoing.
3. tcpdump (wifi - wired incoming interface) does NOT show DHCPOFFER 
traffic when this problem occurs.
4. Both traces show arp activity during the DISCOVER/OFFER sequence.
5. There is no evidence of tx errors or packet drops, in any statistics 
I can find.

Thoughts:

I still think there's a race happening between the arp neighbor update 
and sky2. Might be higher up, but as I'm seeing the outgoing packets 
when sniffing eth0, can't be too much higher up. This problem seems to 
be exacerbated by the more recent patches, however I believe that this 
is a result of the higher throughput achievable with these patches. With 
the older set, I saw this problem less frequently, but found it much 
harder to get over the 40K RX number.

I am also seeing (as previously reported - but haven't sniffed yet) SMB 
ACK dropped packets, but only when traversing the wifi router. Not sure 
if this is related, but hey, you never know.