mlx4: dropping multicast packets at promisc leave

mlx4: dropping multicast packets at promisc leave

[Marcelo Ricardo Leitner]

Hi,

I have a report that our mlx4 driver (RHEL 6.3) is dropping multicast 
packets when NIC leaves promisc mode. It seems this is being cause due 
to the new steering mode that took place near by commit 
1679200f91da6a054b06954c9bd3eeed29b6731f.

As it seems, the new steering mode needs more commands/time to leave the 
promisc mode, which may be leading to packet drops.

The details are bellow. My main question now is: wouldn't it be safer/is 
it possible to enable the MCAST_FLTR before removing the promisc entries?

I noticed that for entering promisc mode it first puts the NIC in 
promisc and only then disables MCAST_FLTR, and no packet drops are being 
observed.

Details:

At my system (line numbers won't directly match):
[633241.362775] device eth5 left promiscuous mode
[633241.362800] mlx4_en: eth5: 356 - mlx4_unicast_promisc_remove
[633241.362854] mlx4_en: eth5: 365 - mlx4_multicast_promisc_remove
[633241.363049] mlx4_en: eth5: 407 - mlx4_SET_MCAST_FLTR MLX4_MCAST_DISABLE
[633241.363098] mlx4_en: eth5: 414 - mlx4_SET_MCAST_FLTR MLX4_MCAST_CONFIG
[633241.363353] mlx4_en: eth5: 428 - mlx4_SET_MCAST_FLTR MLX4_MCAST_ENABLE

At theirs:
[  137.614253] device eth1 left promiscuous mode
[  137.667519] mlx4_en: eth1: 356 - mlx4_unicast_promisc_remove
[  137.735264] mlx4_en: eth1: 365 - mlx4_multicast_promisc_remove
[  137.805159] mlx4_en: eth1: 407 - mlx4_SET_MCAST_FLTR MLX4_MCAST_DISABLE
[  137.864581] mlx4_en: eth1: 414 - mlx4_SET_MCAST_FLTR MLX4_MCAST_CONFIG
[  137.943299] mlx4_en: eth1: 428 - mlx4_SET_MCAST_FLTR MLX4_MCAST_ENABLE

It takes 300ms to perform the change there against my 600us. Hitting 
something like tcpdump -c 10 in a loop helps triggering it.

Both NICs are:
06:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 
10GigE, PCIe 2.0 5GT/s] (rev b0)

Tested on firmwares 2.8.600 and 2.9.1200, same issue. I could not 
reproduce the drops myself so far. Their NIC is IBM branded and mine is HP.

I checked commit 
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commitdiff;h=6d19993788e080edb557178cc6aba2d963edce4e 
but even with it, it dropped packets.

Thanks,
Marcelo.

Re: mlx4: dropping multicast packets at promisc leave

[Or Gerlitz]

On 20/09/2012 03:43, Marcelo Ricardo Leitner wrote:
> I have a report that our mlx4 driver (RHEL 6.3) is dropping multicast 
> packets when NIC leaves promisc mode. It seems this is being cause due 
> to the new steering mode that took place near by commit 
> 1679200f91da6a054b06954c9bd3eeed29b6731f. As it seems, the new 
> steering mode needs more commands/time to leave the promisc mode, 
> which may be leading to packet drops.

Marcelo,

The commit you point on below 6d19993 "net/mlx4_en: Re-design multicast 
attachments flow" makes sure to avoid
doing extra firmware comments and not leave a window in time where 
"correct" addresses are not attached. Its hard to say what's the case on 
that RHEL 6.3 system, it would be very helpful through if you manage to 
reproduce the problem on an upstream kernel -- BTW you didn't say on 
which kernel you are  trying to reproduce  this, note that upstream has 
also commit 60d31c1475f2 "net/mlx4_core: Looking for promiscuous entries 
on the correct port" (reported by you...) , so if somehow this commit 
makes the diff you could use it also on their system.

> It takes 300ms to perform the change there against my 600us. Hitting 
> something like tcpdump -c 10 in a loop helps triggering it.

Do you have any insight for this huge difference?

Or.

Re: mlx4: dropping multicast packets at promisc leave

[Marcelo Ricardo Leitner]

On 09/20/2012 10:21 AM, Or Gerlitz wrote:
> On 20/09/2012 03:43, Marcelo Ricardo Leitner wrote:
>> I have a report that our mlx4 driver (RHEL 6.3) is dropping multicast
>> packets when NIC leaves promisc mode. It seems this is being cause due
>> to the new steering mode that took place near by commit
>> 1679200f91da6a054b06954c9bd3eeed29b6731f. As it seems, the new
>> steering mode needs more commands/time to leave the promisc mode,
>> which may be leading to packet drops.
>
> Marcelo,
>
> The commit you point on below 6d19993 "net/mlx4_en: Re-design multicast
> attachments flow" makes sure to avoid
> doing extra firmware comments and not leave a window in time where
> "correct" addresses are not attached. Its hard to say what's the case on
> that RHEL 6.3 system, it would be very helpful through if you manage to
> reproduce the problem on an upstream kernel -- BTW you didn't say on

Okay, I understand that the commit prevents a window. I may be missing 
something, but isn't there another one in there? Between:
mlx4_SET_MCAST_FLTR MLX4_MCAST_DISABLE and
mlx4_SET_MCAST_FLTR MLX4_MCAST_ENABLE
because mlx4_multicast_promisc_remove() was called just before those.
Otherwise I don't how is the NIC would be receiving multicast packets in 
there.

I understand the difficulty about the kernel version, I am sorry for 
that. As I'm unable to reproduce the issue by myself, I couldn't run a 
test in a plain upstream kernel so far or experiment much.

I was holding this email: I just access to a server that seems to 
reproduce the issue. It has a MT27500 ConnectX-3 NIC. Only tried our 
RHEL 6.3 stock so far. Keep you posted on further tests!

This was the result of ifconfig mlx4_2 -promisc:
[  3] 34.0-35.0 sec  61.7 MBytes   517 Mbits/sec   0.024 ms  274/43502 
(0.63%)
[  3] 34.0-35.0 sec  756 datagrams received out-of-order

> which kernel you are trying to reproduce this, note that upstream has
> also commit 60d31c1475f2 "net/mlx4_core: Looking for promiscuous entries
> on the correct port" (reported by you...) , so if somehow this commit
> makes the diff you could use it also on their system.

Sorry, that would be 2.6.32-279.el6. It has additional commits up to 
somewhere near commit
58a3de0 - mlx4_core: fix race on comm channel
but maybe not all before that one. Can't tell you for sure.

And then I tried 3 additional patches applied at once:
- 60d31c1475f2 "net/mlx4_core: Looking for promiscuous entries on the 
correct port"
- f1f75f0 - mlx4: attach multicast with correct flag
   - Yes, this one wasn't in 2.6.32-279.el6.
- 6d19993 - net/mlx4_en: Re-design multicast attachments flow

And they still reported drops.

>> It takes 300ms to perform the change there against my 600us. Hitting
>> something like tcpdump -c 10 in a loop helps triggering it.
>
> Do you have any insight for this huge difference?

No idea. Couldn't track it yet.

Thanks,
Marcelo.

Re: mlx4: dropping multicast packets at promisc leave

[Marcelo Ricardo Leitner]

On 09/20/2012 12:04 PM, Marcelo Ricardo Leitner wrote:
> On 09/20/2012 10:21 AM, Or Gerlitz wrote:
>> On 20/09/2012 03:43, Marcelo Ricardo Leitner wrote:
>>> I have a report that our mlx4 driver (RHEL 6.3) is dropping multicast
>>> packets when NIC leaves promisc mode. It seems this is being cause due
>>> to the new steering mode that took place near by commit
>>> 1679200f91da6a054b06954c9bd3eeed29b6731f. As it seems, the new
>>> steering mode needs more commands/time to leave the promisc mode,
>>> which may be leading to packet drops.
>>
>> Marcelo,
>>
>> The commit you point on below 6d19993 "net/mlx4_en: Re-design multicast
>> attachments flow" makes sure to avoid
>> doing extra firmware comments and not leave a window in time where
>> "correct" addresses are not attached. Its hard to say what's the case on
>> that RHEL 6.3 system, it would be very helpful through if you manage to
>> reproduce the problem on an upstream kernel -- BTW you didn't say on
>
> Okay, I understand that the commit prevents a window. I may be missing
> something, but isn't there another one in there? Between:
> mlx4_SET_MCAST_FLTR MLX4_MCAST_DISABLE and
> mlx4_SET_MCAST_FLTR MLX4_MCAST_ENABLE
> because mlx4_multicast_promisc_remove() was called just before those.
> Otherwise I don't how is the NIC would be receiving multicast packets in
> there.
>
....
> And then I tried 3 additional patches applied at once:
> - 60d31c1475f2 "net/mlx4_core: Looking for promiscuous entries on the
> correct port"
> - f1f75f0 - mlx4: attach multicast with correct flag
> - Yes, this one wasn't in 2.6.32-279.el6.
> - 6d19993 - net/mlx4_en: Re-design multicast attachments flow
>
> And they still reported drops.

Hi, updates on this. In summary, I couldn't reproduce it when using 
upstream kernel, even on a machine which I've seen the drops.

Checking with customer, they could reproduce the issue once when using 
commit 6d19993 but cannot reproduce it anymore.

I also could reproduce it but it gets much harder to trigger when using 
commit 6d19993.

I could run some tests with net tree kernel (up to 8ea853fd) and I 
didn't see any drops so far. More bellow.

>>> It takes 300ms to perform the change there against my 600us. Hitting
>>> something like tcpdump -c 10 in a loop helps triggering it.
>>
>> Do you have any insight for this huge difference?
>
> No idea. Couldn't track it yet.

Now I might have. ksoftirqd/0 is being triggered when running 500Mbit 
multicast traffic via iperf and is hitting 90% CPU usage. Seems it is 
blocking the execution of mlx4_en_do_set_multicast() sometimes.

This happens at my reproducer only. It does not happen on the other 
server. Customer also doesn't see it, but see the delays. They just see 
mlx4_en thread running when running tcpdump in cycles, but that is expected.

Upstream (net tree) kernel doesn't do this, even at my reproducer box.

Simple perf record/report gave me:

# Overhead      Command      Shared Object     Symbol
# ........  ...........  .................     .....................
#
      98.16%  ksoftirqd/0  [kernel.kallsyms]  [k]
  fix_small_imbalance
              |
              --- fix_small_imbalance
                  find_busiest_group
                  rebalance_domains
                  run_rebalance_domains
                  __do_softirq
                  call_softirq
                  ksoftirqd
                  kthread
                  child_rip

I could see:
[60315.574258] mlx4_core 0000:01:00.0: remove_promisc_qp 483
[60315.579652] mlx4_core 0000:01:00.0: remove_promisc_qp 485

And that was:
  482         /*remove from list of promisc qps */
  483         mlx4_warn(dev, "%s %d\n", __FUNCTION__, __LINE__);
  484         list_del(&pqp->list);
  485         mlx4_warn(dev, "%s %d\n", __FUNCTION__, __LINE__);

That really shouldn't take that long, and it doesn't when it's idle. 
Might that weird behavior open some window where NIC is configured in a 
such way that it doesn't receive the packets?

But well, this may be causing the drops somewhere else. I still have to 
check that.

Many thanks,
Marcelo.

Re: mlx4: dropping multicast packets at promisc leave

[Marcelo Ricardo Leitner]

On 09/21/2012 03:32 PM, Marcelo Ricardo Leitner wrote:
> On 09/20/2012 12:04 PM, Marcelo Ricardo Leitner wrote:
>> On 09/20/2012 10:21 AM, Or Gerlitz wrote:
>>> On 20/09/2012 03:43, Marcelo Ricardo Leitner wrote:
>>>> I have a report that our mlx4 driver (RHEL 6.3) is dropping multicast
>>>> packets when NIC leaves promisc mode. It seems this is being cause due
>>>> to the new steering mode that took place near by commit
>>>> 1679200f91da6a054b06954c9bd3eeed29b6731f. As it seems, the new
>>>> steering mode needs more commands/time to leave the promisc mode,
>>>> which may be leading to packet drops.
>>>
>>> Marcelo,
>>>
>>> The commit you point on below 6d19993 "net/mlx4_en: Re-design multicast
>>> attachments flow" makes sure to avoid
>>> doing extra firmware comments and not leave a window in time where
>>> "correct" addresses are not attached. Its hard to say what's the case on
>>> that RHEL 6.3 system, it would be very helpful through if you manage to
>>> reproduce the problem on an upstream kernel -- BTW you didn't say on
>>
>> Okay, I understand that the commit prevents a window. I may be missing
>> something, but isn't there another one in there? Between:
>> mlx4_SET_MCAST_FLTR MLX4_MCAST_DISABLE and
>> mlx4_SET_MCAST_FLTR MLX4_MCAST_ENABLE
>> because mlx4_multicast_promisc_remove() was called just before those.
>> Otherwise I don't how is the NIC would be receiving multicast packets in
>> there.
>>
> ....
>> And then I tried 3 additional patches applied at once:
>> - 60d31c1475f2 "net/mlx4_core: Looking for promiscuous entries on the
>> correct port"
>> - f1f75f0 - mlx4: attach multicast with correct flag
>> - Yes, this one wasn't in 2.6.32-279.el6.
>> - 6d19993 - net/mlx4_en: Re-design multicast attachments flow
>>
>> And they still reported drops.

Well, neither me nor they could reproduce the drops at promisc exit 
anymore. It's hard to chase a ghost, you know. I'll still track that cpu 
usage, but it seems unrelated to the driver at first glance.

Thank you for your support Or, appreciated.

Regards,
Marcelo

Re: mlx4: dropping multicast packets at promisc leave

[Or Gerlitz]

On Thu, Sep 27, 2012 at 10:37 PM, Marcelo Ricardo Leitner
<mleitner@redhat.com> wrote:
> Well, neither me nor they could reproduce the drops at promisc exit anymore.
> It's hard to chase a ghost, you know. I'll still track that cpu usage, but
> it seems unrelated to the driver at first glance.
>
> Thank you for your support Or, appreciated.

sure