Re: mlx4: dropping multicast packets at promisc leave

[Marcelo Ricardo Leitner <mleitner@redhat.com>]

On 09/20/2012 12:04 PM, Marcelo Ricardo Leitner wrote:
> On 09/20/2012 10:21 AM, Or Gerlitz wrote:
>> On 20/09/2012 03:43, Marcelo Ricardo Leitner wrote:
>>> I have a report that our mlx4 driver (RHEL 6.3) is dropping multicast
>>> packets when NIC leaves promisc mode. It seems this is being cause due
>>> to the new steering mode that took place near by commit
>>> 1679200f91da6a054b06954c9bd3eeed29b6731f. As it seems, the new
>>> steering mode needs more commands/time to leave the promisc mode,
>>> which may be leading to packet drops.
>>
>> Marcelo,
>>
>> The commit you point on below 6d19993 "net/mlx4_en: Re-design multicast
>> attachments flow" makes sure to avoid
>> doing extra firmware comments and not leave a window in time where
>> "correct" addresses are not attached. Its hard to say what's the case on
>> that RHEL 6.3 system, it would be very helpful through if you manage to
>> reproduce the problem on an upstream kernel -- BTW you didn't say on
>
> Okay, I understand that the commit prevents a window. I may be missing
> something, but isn't there another one in there? Between:
> mlx4_SET_MCAST_FLTR MLX4_MCAST_DISABLE and
> mlx4_SET_MCAST_FLTR MLX4_MCAST_ENABLE
> because mlx4_multicast_promisc_remove() was called just before those.
> Otherwise I don't how is the NIC would be receiving multicast packets in
> there.
>
....
> And then I tried 3 additional patches applied at once:
> - 60d31c1475f2 "net/mlx4_core: Looking for promiscuous entries on the
> correct port"
> - f1f75f0 - mlx4: attach multicast with correct flag
> - Yes, this one wasn't in 2.6.32-279.el6.
> - 6d19993 - net/mlx4_en: Re-design multicast attachments flow
>
> And they still reported drops.

Hi, updates on this. In summary, I couldn't reproduce it when using 
upstream kernel, even on a machine which I've seen the drops.

Checking with customer, they could reproduce the issue once when using 
commit 6d19993 but cannot reproduce it anymore.

I also could reproduce it but it gets much harder to trigger when using 
commit 6d19993.

I could run some tests with net tree kernel (up to 8ea853fd) and I 
didn't see any drops so far. More bellow.

>>> It takes 300ms to perform the change there against my 600us. Hitting
>>> something like tcpdump -c 10 in a loop helps triggering it.
>>
>> Do you have any insight for this huge difference?
>
> No idea. Couldn't track it yet.

Now I might have. ksoftirqd/0 is being triggered when running 500Mbit 
multicast traffic via iperf and is hitting 90% CPU usage. Seems it is 
blocking the execution of mlx4_en_do_set_multicast() sometimes.

This happens at my reproducer only. It does not happen on the other 
server. Customer also doesn't see it, but see the delays. They just see 
mlx4_en thread running when running tcpdump in cycles, but that is expected.

Upstream (net tree) kernel doesn't do this, even at my reproducer box.

Simple perf record/report gave me:

# Overhead      Command      Shared Object     Symbol
# ........  ...........  .................     .....................
#
      98.16%  ksoftirqd/0  [kernel.kallsyms]  [k]
  fix_small_imbalance
              |
              --- fix_small_imbalance
                  find_busiest_group
                  rebalance_domains
                  run_rebalance_domains
                  __do_softirq
                  call_softirq
                  ksoftirqd
                  kthread
                  child_rip

I could see:
[60315.574258] mlx4_core 0000:01:00.0: remove_promisc_qp 483
[60315.579652] mlx4_core 0000:01:00.0: remove_promisc_qp 485

And that was:
  482         /*remove from list of promisc qps */
  483         mlx4_warn(dev, "%s %d\n", __FUNCTION__, __LINE__);
  484         list_del(&pqp->list);
  485         mlx4_warn(dev, "%s %d\n", __FUNCTION__, __LINE__);

That really shouldn't take that long, and it doesn't when it's idle. 
Might that weird behavior open some window where NIC is configured in a 
such way that it doesn't receive the packets?

But well, this may be causing the drops somewhere else. I still have to 
check that.

Many thanks,
Marcelo.