* mlx4: dropping multicast packets at promisc leave @ 2012-09-20 0:43 Marcelo Ricardo Leitner 2012-09-20 13:21 ` Or Gerlitz 0 siblings, 1 reply; 6+ messages in thread From: Marcelo Ricardo Leitner @ 2012-09-20 0:43 UTC (permalink / raw) To: netdev; +Cc: Or Gerlitz Hi, I have a report that our mlx4 driver (RHEL 6.3) is dropping multicast packets when NIC leaves promisc mode. It seems this is being cause due to the new steering mode that took place near by commit 1679200f91da6a054b06954c9bd3eeed29b6731f. As it seems, the new steering mode needs more commands/time to leave the promisc mode, which may be leading to packet drops. The details are bellow. My main question now is: wouldn't it be safer/is it possible to enable the MCAST_FLTR before removing the promisc entries? I noticed that for entering promisc mode it first puts the NIC in promisc and only then disables MCAST_FLTR, and no packet drops are being observed. Details: At my system (line numbers won't directly match): [633241.362775] device eth5 left promiscuous mode [633241.362800] mlx4_en: eth5: 356 - mlx4_unicast_promisc_remove [633241.362854] mlx4_en: eth5: 365 - mlx4_multicast_promisc_remove [633241.363049] mlx4_en: eth5: 407 - mlx4_SET_MCAST_FLTR MLX4_MCAST_DISABLE [633241.363098] mlx4_en: eth5: 414 - mlx4_SET_MCAST_FLTR MLX4_MCAST_CONFIG [633241.363353] mlx4_en: eth5: 428 - mlx4_SET_MCAST_FLTR MLX4_MCAST_ENABLE At theirs: [ 137.614253] device eth1 left promiscuous mode [ 137.667519] mlx4_en: eth1: 356 - mlx4_unicast_promisc_remove [ 137.735264] mlx4_en: eth1: 365 - mlx4_multicast_promisc_remove [ 137.805159] mlx4_en: eth1: 407 - mlx4_SET_MCAST_FLTR MLX4_MCAST_DISABLE [ 137.864581] mlx4_en: eth1: 414 - mlx4_SET_MCAST_FLTR MLX4_MCAST_CONFIG [ 137.943299] mlx4_en: eth1: 428 - mlx4_SET_MCAST_FLTR MLX4_MCAST_ENABLE It takes 300ms to perform the change there against my 600us. Hitting something like tcpdump -c 10 in a loop helps triggering it. Both NICs are: 06:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0) Tested on firmwares 2.8.600 and 2.9.1200, same issue. I could not reproduce the drops myself so far. Their NIC is IBM branded and mine is HP. I checked commit http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commitdiff;h=6d19993788e080edb557178cc6aba2d963edce4e but even with it, it dropped packets. Thanks, Marcelo. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mlx4: dropping multicast packets at promisc leave 2012-09-20 0:43 mlx4: dropping multicast packets at promisc leave Marcelo Ricardo Leitner @ 2012-09-20 13:21 ` Or Gerlitz 2012-09-20 15:04 ` Marcelo Ricardo Leitner 0 siblings, 1 reply; 6+ messages in thread From: Or Gerlitz @ 2012-09-20 13:21 UTC (permalink / raw) To: Marcelo Ricardo Leitner; +Cc: netdev, Yevgeny Petrilin, Amir Vadai On 20/09/2012 03:43, Marcelo Ricardo Leitner wrote: > I have a report that our mlx4 driver (RHEL 6.3) is dropping multicast > packets when NIC leaves promisc mode. It seems this is being cause due > to the new steering mode that took place near by commit > 1679200f91da6a054b06954c9bd3eeed29b6731f. As it seems, the new > steering mode needs more commands/time to leave the promisc mode, > which may be leading to packet drops. Marcelo, The commit you point on below 6d19993 "net/mlx4_en: Re-design multicast attachments flow" makes sure to avoid doing extra firmware comments and not leave a window in time where "correct" addresses are not attached. Its hard to say what's the case on that RHEL 6.3 system, it would be very helpful through if you manage to reproduce the problem on an upstream kernel -- BTW you didn't say on which kernel you are trying to reproduce this, note that upstream has also commit 60d31c1475f2 "net/mlx4_core: Looking for promiscuous entries on the correct port" (reported by you...) , so if somehow this commit makes the diff you could use it also on their system. > It takes 300ms to perform the change there against my 600us. Hitting > something like tcpdump -c 10 in a loop helps triggering it. Do you have any insight for this huge difference? Or. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mlx4: dropping multicast packets at promisc leave 2012-09-20 13:21 ` Or Gerlitz @ 2012-09-20 15:04 ` Marcelo Ricardo Leitner 2012-09-21 18:32 ` Marcelo Ricardo Leitner 0 siblings, 1 reply; 6+ messages in thread From: Marcelo Ricardo Leitner @ 2012-09-20 15:04 UTC (permalink / raw) To: Or Gerlitz; +Cc: netdev, Yevgeny Petrilin, Amir Vadai On 09/20/2012 10:21 AM, Or Gerlitz wrote: > On 20/09/2012 03:43, Marcelo Ricardo Leitner wrote: >> I have a report that our mlx4 driver (RHEL 6.3) is dropping multicast >> packets when NIC leaves promisc mode. It seems this is being cause due >> to the new steering mode that took place near by commit >> 1679200f91da6a054b06954c9bd3eeed29b6731f. As it seems, the new >> steering mode needs more commands/time to leave the promisc mode, >> which may be leading to packet drops. > > Marcelo, > > The commit you point on below 6d19993 "net/mlx4_en: Re-design multicast > attachments flow" makes sure to avoid > doing extra firmware comments and not leave a window in time where > "correct" addresses are not attached. Its hard to say what's the case on > that RHEL 6.3 system, it would be very helpful through if you manage to > reproduce the problem on an upstream kernel -- BTW you didn't say on Okay, I understand that the commit prevents a window. I may be missing something, but isn't there another one in there? Between: mlx4_SET_MCAST_FLTR MLX4_MCAST_DISABLE and mlx4_SET_MCAST_FLTR MLX4_MCAST_ENABLE because mlx4_multicast_promisc_remove() was called just before those. Otherwise I don't how is the NIC would be receiving multicast packets in there. I understand the difficulty about the kernel version, I am sorry for that. As I'm unable to reproduce the issue by myself, I couldn't run a test in a plain upstream kernel so far or experiment much. I was holding this email: I just access to a server that seems to reproduce the issue. It has a MT27500 ConnectX-3 NIC. Only tried our RHEL 6.3 stock so far. Keep you posted on further tests! This was the result of ifconfig mlx4_2 -promisc: [ 3] 34.0-35.0 sec 61.7 MBytes 517 Mbits/sec 0.024 ms 274/43502 (0.63%) [ 3] 34.0-35.0 sec 756 datagrams received out-of-order > which kernel you are trying to reproduce this, note that upstream has > also commit 60d31c1475f2 "net/mlx4_core: Looking for promiscuous entries > on the correct port" (reported by you...) , so if somehow this commit > makes the diff you could use it also on their system. Sorry, that would be 2.6.32-279.el6. It has additional commits up to somewhere near commit 58a3de0 - mlx4_core: fix race on comm channel but maybe not all before that one. Can't tell you for sure. And then I tried 3 additional patches applied at once: - 60d31c1475f2 "net/mlx4_core: Looking for promiscuous entries on the correct port" - f1f75f0 - mlx4: attach multicast with correct flag - Yes, this one wasn't in 2.6.32-279.el6. - 6d19993 - net/mlx4_en: Re-design multicast attachments flow And they still reported drops. >> It takes 300ms to perform the change there against my 600us. Hitting >> something like tcpdump -c 10 in a loop helps triggering it. > > Do you have any insight for this huge difference? No idea. Couldn't track it yet. Thanks, Marcelo. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mlx4: dropping multicast packets at promisc leave 2012-09-20 15:04 ` Marcelo Ricardo Leitner @ 2012-09-21 18:32 ` Marcelo Ricardo Leitner 2012-09-27 20:37 ` Marcelo Ricardo Leitner 0 siblings, 1 reply; 6+ messages in thread From: Marcelo Ricardo Leitner @ 2012-09-21 18:32 UTC (permalink / raw) To: Or Gerlitz; +Cc: netdev, Yevgeny Petrilin, Amir Vadai On 09/20/2012 12:04 PM, Marcelo Ricardo Leitner wrote: > On 09/20/2012 10:21 AM, Or Gerlitz wrote: >> On 20/09/2012 03:43, Marcelo Ricardo Leitner wrote: >>> I have a report that our mlx4 driver (RHEL 6.3) is dropping multicast >>> packets when NIC leaves promisc mode. It seems this is being cause due >>> to the new steering mode that took place near by commit >>> 1679200f91da6a054b06954c9bd3eeed29b6731f. As it seems, the new >>> steering mode needs more commands/time to leave the promisc mode, >>> which may be leading to packet drops. >> >> Marcelo, >> >> The commit you point on below 6d19993 "net/mlx4_en: Re-design multicast >> attachments flow" makes sure to avoid >> doing extra firmware comments and not leave a window in time where >> "correct" addresses are not attached. Its hard to say what's the case on >> that RHEL 6.3 system, it would be very helpful through if you manage to >> reproduce the problem on an upstream kernel -- BTW you didn't say on > > Okay, I understand that the commit prevents a window. I may be missing > something, but isn't there another one in there? Between: > mlx4_SET_MCAST_FLTR MLX4_MCAST_DISABLE and > mlx4_SET_MCAST_FLTR MLX4_MCAST_ENABLE > because mlx4_multicast_promisc_remove() was called just before those. > Otherwise I don't how is the NIC would be receiving multicast packets in > there. > .... > And then I tried 3 additional patches applied at once: > - 60d31c1475f2 "net/mlx4_core: Looking for promiscuous entries on the > correct port" > - f1f75f0 - mlx4: attach multicast with correct flag > - Yes, this one wasn't in 2.6.32-279.el6. > - 6d19993 - net/mlx4_en: Re-design multicast attachments flow > > And they still reported drops. Hi, updates on this. In summary, I couldn't reproduce it when using upstream kernel, even on a machine which I've seen the drops. Checking with customer, they could reproduce the issue once when using commit 6d19993 but cannot reproduce it anymore. I also could reproduce it but it gets much harder to trigger when using commit 6d19993. I could run some tests with net tree kernel (up to 8ea853fd) and I didn't see any drops so far. More bellow. >>> It takes 300ms to perform the change there against my 600us. Hitting >>> something like tcpdump -c 10 in a loop helps triggering it. >> >> Do you have any insight for this huge difference? > > No idea. Couldn't track it yet. Now I might have. ksoftirqd/0 is being triggered when running 500Mbit multicast traffic via iperf and is hitting 90% CPU usage. Seems it is blocking the execution of mlx4_en_do_set_multicast() sometimes. This happens at my reproducer only. It does not happen on the other server. Customer also doesn't see it, but see the delays. They just see mlx4_en thread running when running tcpdump in cycles, but that is expected. Upstream (net tree) kernel doesn't do this, even at my reproducer box. Simple perf record/report gave me: # Overhead Command Shared Object Symbol # ........ ........... ................. ..................... # 98.16% ksoftirqd/0 [kernel.kallsyms] [k] fix_small_imbalance | --- fix_small_imbalance find_busiest_group rebalance_domains run_rebalance_domains __do_softirq call_softirq ksoftirqd kthread child_rip I could see: [60315.574258] mlx4_core 0000:01:00.0: remove_promisc_qp 483 [60315.579652] mlx4_core 0000:01:00.0: remove_promisc_qp 485 And that was: 482 /*remove from list of promisc qps */ 483 mlx4_warn(dev, "%s %d\n", __FUNCTION__, __LINE__); 484 list_del(&pqp->list); 485 mlx4_warn(dev, "%s %d\n", __FUNCTION__, __LINE__); That really shouldn't take that long, and it doesn't when it's idle. Might that weird behavior open some window where NIC is configured in a such way that it doesn't receive the packets? But well, this may be causing the drops somewhere else. I still have to check that. Many thanks, Marcelo. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mlx4: dropping multicast packets at promisc leave 2012-09-21 18:32 ` Marcelo Ricardo Leitner @ 2012-09-27 20:37 ` Marcelo Ricardo Leitner 2012-09-27 20:45 ` Or Gerlitz 0 siblings, 1 reply; 6+ messages in thread From: Marcelo Ricardo Leitner @ 2012-09-27 20:37 UTC (permalink / raw) To: Or Gerlitz; +Cc: netdev, Yevgeny Petrilin, Amir Vadai On 09/21/2012 03:32 PM, Marcelo Ricardo Leitner wrote: > On 09/20/2012 12:04 PM, Marcelo Ricardo Leitner wrote: >> On 09/20/2012 10:21 AM, Or Gerlitz wrote: >>> On 20/09/2012 03:43, Marcelo Ricardo Leitner wrote: >>>> I have a report that our mlx4 driver (RHEL 6.3) is dropping multicast >>>> packets when NIC leaves promisc mode. It seems this is being cause due >>>> to the new steering mode that took place near by commit >>>> 1679200f91da6a054b06954c9bd3eeed29b6731f. As it seems, the new >>>> steering mode needs more commands/time to leave the promisc mode, >>>> which may be leading to packet drops. >>> >>> Marcelo, >>> >>> The commit you point on below 6d19993 "net/mlx4_en: Re-design multicast >>> attachments flow" makes sure to avoid >>> doing extra firmware comments and not leave a window in time where >>> "correct" addresses are not attached. Its hard to say what's the case on >>> that RHEL 6.3 system, it would be very helpful through if you manage to >>> reproduce the problem on an upstream kernel -- BTW you didn't say on >> >> Okay, I understand that the commit prevents a window. I may be missing >> something, but isn't there another one in there? Between: >> mlx4_SET_MCAST_FLTR MLX4_MCAST_DISABLE and >> mlx4_SET_MCAST_FLTR MLX4_MCAST_ENABLE >> because mlx4_multicast_promisc_remove() was called just before those. >> Otherwise I don't how is the NIC would be receiving multicast packets in >> there. >> > .... >> And then I tried 3 additional patches applied at once: >> - 60d31c1475f2 "net/mlx4_core: Looking for promiscuous entries on the >> correct port" >> - f1f75f0 - mlx4: attach multicast with correct flag >> - Yes, this one wasn't in 2.6.32-279.el6. >> - 6d19993 - net/mlx4_en: Re-design multicast attachments flow >> >> And they still reported drops. Well, neither me nor they could reproduce the drops at promisc exit anymore. It's hard to chase a ghost, you know. I'll still track that cpu usage, but it seems unrelated to the driver at first glance. Thank you for your support Or, appreciated. Regards, Marcelo ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mlx4: dropping multicast packets at promisc leave 2012-09-27 20:37 ` Marcelo Ricardo Leitner @ 2012-09-27 20:45 ` Or Gerlitz 0 siblings, 0 replies; 6+ messages in thread From: Or Gerlitz @ 2012-09-27 20:45 UTC (permalink / raw) To: mleitner; +Cc: netdev, Yevgeny Petrilin, Amir Vadai On Thu, Sep 27, 2012 at 10:37 PM, Marcelo Ricardo Leitner <mleitner@redhat.com> wrote: > Well, neither me nor they could reproduce the drops at promisc exit anymore. > It's hard to chase a ghost, you know. I'll still track that cpu usage, but > it seems unrelated to the driver at first glance. > > Thank you for your support Or, appreciated. sure ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2012-09-27 20:45 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-09-20 0:43 mlx4: dropping multicast packets at promisc leave Marcelo Ricardo Leitner 2012-09-20 13:21 ` Or Gerlitz 2012-09-20 15:04 ` Marcelo Ricardo Leitner 2012-09-21 18:32 ` Marcelo Ricardo Leitner 2012-09-27 20:37 ` Marcelo Ricardo Leitner 2012-09-27 20:45 ` Or Gerlitz
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).