public inbox for netfilter@vger.kernel.org
 help / color / mirror / Atom feed
From: Odintsov Vladislav <VlOdintsov@k2.cloud>
To: "netfilter@vger.kernel.org" <netfilter@vger.kernel.org>,
	"ovs-dev@openvswitch.org" <ovs-dev@openvswitch.org>
Cc: Kovalev Evgeniy <EvgKovalev@k2.cloud>,
	Vazhnetsov Anton <AVazhnetsov@k2.cloud>,
	Rukomoinikova Aleksandra <ARukomoinikova@k2.cloud>
Subject: RE: netfilter: nf_conncount: cpu soft lockup using limiting with Open vSwitch.
Date: Mon, 8 Dec 2025 12:27:53 +0000	[thread overview]
Message-ID: <f801e56e-eba5-460e-919a-3e839b1f9905@k2.cloud> (raw)
In-Reply-To: <7a6872ce-8015-4397-bbe9-22108c65b7ec@k2.cloud>

On 08.12.2025 15:06, Rukomoinikova Aleksandra wrote:
> Hi!
> I was testing conntrack limiting using Open vSwitch and noticed the 
> following issue: under certain limits, a CPU lock occurred.
> 
> [  491.682936] watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [ovs- 
> dpctl:19437]
> 
> This occurs during a high packet frequency when trying to get the set 
> limits through ovs-dpctl ct-get-limits.
> 
> In the trace, I can see that the lock occurred on attempts to acquire a 
> spinlock.
> 
> [  491.683056]  <IRQ>
> [  491.683059]  _raw_spin_lock_bh+0x29/0x30
> [  491.683064]  count_tree+0x19b/0x1f0 [nf_conncount]
> [  491.683069]  ovs_ct_commit+0x196/0x490 [openvswitch]
> 
> Prior to this, in the trace, there was processing of a task from 
> userspace (ovs-dpctl)
> 
> [  491.683236]  </IRQ>
> [  491.683237]  <TASK>
> [  491.683238]  asm_common_interrupt+0x22/0x40
> [  491.683240] RIP: 0010:nf_conncount_gc_list+0x18a/0x200 [nf_conncount]
> 
> Inside the nf_conncount_gc_list function, a lock is taken on 
> nf_conncount.c:spin_trylock_bh(&list->list_lock):335. After this, the 
> not-so-fast __nf_conncount_gc_list function is executed. If, at this 
> moment, a packet interrupt arrives on the same сpu core (and 
> spin_trylock_bh doesn't disable interrupts on that core), then scenario 
> I encountered occurs: the first lock remains held, while the packet 
> interrupt also attempts to acquire it at 
> nf_conncount.c:spin_lock_bh(&rbconn->list.list_lock):502 while 
> committing to conntrack. This attempt fails, leading to a soft lockup.
> 
> Hence my question: shouldn't we avoid calling nf_conncount_gc_list when 
> querying limits without an skb (as OVS does in openvswitch/ 
> conntrack.c:1773)? The limit retrieval operation should be read-only 
> regarding the contract state, not involve potential modification.
> 
> Like this:
> --- a/net/netfilter/nf_conncount.c
> +++ b/net/netfilter/nf_conncount.c
> @@ -495,7 +495,6 @@ count_tree(struct net *net,
>               int ret;
> 
>               if (!skb) {
> -                nf_conncount_gc_list(net, &rbconn->list);
>                   return rbconn->list.count;
>               }
> 
> Thanks!
> 
> -- 
> regards,
> Alexandra.
> 

+ ovs-dev ML

       reply	other threads:[~2025-12-08 12:36 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <7a6872ce-8015-4397-bbe9-22108c65b7ec@k2.cloud>
2025-12-08 12:27 ` Odintsov Vladislav [this message]
2025-12-09  7:57   ` netfilter: nf_conncount: cpu soft lockup using limiting with Open vSwitch Fernando Fernandez Mancera
2025-12-09  8:42     ` Pablo Neira Ayuso
2025-12-09  8:49       ` Rukomoinikova Aleksandra
2025-12-10  8:03       ` Fernando Fernandez Mancera
2025-12-12 21:27         ` Rukomoinikova Aleksandra
2025-12-15 11:00           ` Fernando Fernandez Mancera
2025-12-15 11:07             ` Rukomoinikova Aleksandra
2025-12-15 15:38               ` Fernando Fernandez Mancera
2025-12-08 12:31 Rukomoinikova Aleksandra

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=f801e56e-eba5-460e-919a-3e839b1f9905@k2.cloud \
    --to=vlodintsov@k2.cloud \
    --cc=ARukomoinikova@k2.cloud \
    --cc=AVazhnetsov@k2.cloud \
    --cc=EvgKovalev@k2.cloud \
    --cc=netfilter@vger.kernel.org \
    --cc=ovs-dev@openvswitch.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox