Re: CVE-2024-41041: udp: Set SOCK_RCU_FREE earlier in udp_lib_get_port().

Re: CVE-2024-41041: udp: Set SOCK_RCU_FREE earlier in udp_lib_get_port().

[Siddh Raman Pant]

[-- Attachment #1: Type: text/plain, Size: 838 bytes --]

On Mon, 29 Jul 2024 16:32:36 +0200, Greg Kroah-Hartman wrote:
> In the Linux kernel, the following vulnerability has been resolved:
> 
> udp: Set SOCK_RCU_FREE earlier in udp_lib_get_port().
> 
> [...]
> 
> We had the same bug in TCP and fixed it in commit 871019b22d1b ("net:
> set SOCK_RCU_FREE before inserting socket into hashtable").
> 
> Let's apply the same fix for UDP.
> 
> [...]
> 
> The Linux kernel CVE team has assigned CVE-2024-41041 to this issue.
> 
> 
> Affected and fixed versions
> ===========================
> 
> 	Issue introduced in 4.20 with commit 6acc9b432e67 and fixed in 5.4.280 with commit 7a67c4e47626
> 	Issue introduced in 4.20 with commit 6acc9b432e67 and fixed in 5.10.222 with commit 9f965684c57c

These versions don't have the TCP fix backported. Please do so.

Thanks,
Siddh

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: CVE-2024-41041: udp: Set SOCK_RCU_FREE earlier in udp_lib_get_port().

[gregkh]

On Tue, Sep 03, 2024 at 11:56:17AM +0000, Siddh Raman Pant wrote:
> On Mon, 29 Jul 2024 16:32:36 +0200, Greg Kroah-Hartman wrote:
> > In the Linux kernel, the following vulnerability has been resolved:
> > 
> > udp: Set SOCK_RCU_FREE earlier in udp_lib_get_port().
> > 
> > [...]
> > 
> > We had the same bug in TCP and fixed it in commit 871019b22d1b ("net:
> > set SOCK_RCU_FREE before inserting socket into hashtable").
> > 
> > Let's apply the same fix for UDP.
> > 
> > [...]
> > 
> > The Linux kernel CVE team has assigned CVE-2024-41041 to this issue.
> > 
> > 
> > Affected and fixed versions
> > ===========================
> > 
> > 	Issue introduced in 4.20 with commit 6acc9b432e67 and fixed in 5.4.280 with commit 7a67c4e47626
> > 	Issue introduced in 4.20 with commit 6acc9b432e67 and fixed in 5.10.222 with commit 9f965684c57c
> 
> These versions don't have the TCP fix backported. Please do so.

What fix backported exactly to where?  Please be more specific.  Better
yet, please provide working, and tested, backports.

confused,

greg k-h

Re: CVE-2024-41041: udp: Set SOCK_RCU_FREE earlier in udp_lib_get_port().

[Eric Dumazet]

On Tue, Sep 3, 2024 at 2:07 PM gregkh@linuxfoundation.org
<gregkh@linuxfoundation.org> wrote:
>
> On Tue, Sep 03, 2024 at 11:56:17AM +0000, Siddh Raman Pant wrote:
> > On Mon, 29 Jul 2024 16:32:36 +0200, Greg Kroah-Hartman wrote:
> > > In the Linux kernel, the following vulnerability has been resolved:
> > >
> > > udp: Set SOCK_RCU_FREE earlier in udp_lib_get_port().
> > >
> > > [...]
> > >
> > > We had the same bug in TCP and fixed it in commit 871019b22d1b ("net:
> > > set SOCK_RCU_FREE before inserting socket into hashtable").
> > >
> > > Let's apply the same fix for UDP.
> > >
> > > [...]
> > >
> > > The Linux kernel CVE team has assigned CVE-2024-41041 to this issue.
> > >
> > >
> > > Affected and fixed versions
> > > ===========================
> > >
> > >     Issue introduced in 4.20 with commit 6acc9b432e67 and fixed in 5.4.280 with commit 7a67c4e47626
> > >     Issue introduced in 4.20 with commit 6acc9b432e67 and fixed in 5.10.222 with commit 9f965684c57c
> >
> > These versions don't have the TCP fix backported. Please do so.
>
> What fix backported exactly to where?  Please be more specific.  Better
> yet, please provide working, and tested, backports.


commit 871019b22d1bcc9fab2d1feba1b9a564acbb6e99
Author: Stanislav Fomichev <sdf@fomichev.me>
Date:   Wed Nov 8 13:13:25 2023 -0800

    net: set SOCK_RCU_FREE before inserting socket into hashtable
...
    Fixes: 6acc9b432e67 ("bpf: Add helper to retrieve socket in BPF")

It seems 871019b22d1bcc9fab2d1feba1b9a564acbb6e99 has not been pushed
to 5.10 or 5.4 lts

Stanislav mentioned a WARN_ONCE() being hit, I presume we could push
the patch to 5.10 and 5.4.

I guess this was skipped because of a merge conflict.

Re: CVE-2024-41041: udp: Set SOCK_RCU_FREE earlier in udp_lib_get_port().

[gregkh]

On Tue, Sep 03, 2024 at 02:53:57PM +0200, Eric Dumazet wrote:
> On Tue, Sep 3, 2024 at 2:07 PM gregkh@linuxfoundation.org
> <gregkh@linuxfoundation.org> wrote:
> >
> > On Tue, Sep 03, 2024 at 11:56:17AM +0000, Siddh Raman Pant wrote:
> > > On Mon, 29 Jul 2024 16:32:36 +0200, Greg Kroah-Hartman wrote:
> > > > In the Linux kernel, the following vulnerability has been resolved:
> > > >
> > > > udp: Set SOCK_RCU_FREE earlier in udp_lib_get_port().
> > > >
> > > > [...]
> > > >
> > > > We had the same bug in TCP and fixed it in commit 871019b22d1b ("net:
> > > > set SOCK_RCU_FREE before inserting socket into hashtable").
> > > >
> > > > Let's apply the same fix for UDP.
> > > >
> > > > [...]
> > > >
> > > > The Linux kernel CVE team has assigned CVE-2024-41041 to this issue.
> > > >
> > > >
> > > > Affected and fixed versions
> > > > ===========================
> > > >
> > > >     Issue introduced in 4.20 with commit 6acc9b432e67 and fixed in 5.4.280 with commit 7a67c4e47626
> > > >     Issue introduced in 4.20 with commit 6acc9b432e67 and fixed in 5.10.222 with commit 9f965684c57c
> > >
> > > These versions don't have the TCP fix backported. Please do so.
> >
> > What fix backported exactly to where?  Please be more specific.  Better
> > yet, please provide working, and tested, backports.
> 
> 
> commit 871019b22d1bcc9fab2d1feba1b9a564acbb6e99
> Author: Stanislav Fomichev <sdf@fomichev.me>
> Date:   Wed Nov 8 13:13:25 2023 -0800
> 
>     net: set SOCK_RCU_FREE before inserting socket into hashtable
> ...
>     Fixes: 6acc9b432e67 ("bpf: Add helper to retrieve socket in BPF")
> 
> It seems 871019b22d1bcc9fab2d1feba1b9a564acbb6e99 has not been pushed
> to 5.10 or 5.4 lts
> 
> Stanislav mentioned a WARN_ONCE() being hit, I presume we could push
> the patch to 5.10 and 5.4.
> 
> I guess this was skipped because of a merge conflict.

Yes, the commit does not apply, we need someone to send a working
backport for us to be able to take it.

Siddh, can you please do this?

thanks,

greg k-h

Re: CVE-2024-41041: udp: Set SOCK_RCU_FREE earlier in udp_lib_get_port().

[Siddh Raman Pant]

[-- Attachment #1: Type: text/plain, Size: 2428 bytes --]

On Tue, Sep 03 2024 at 18:28:14 +0530, gregkh@linuxfoundation.org
wrote:
> On Tue, Sep 03, 2024 at 02:53:57PM +0200, Eric Dumazet wrote:
> > On Tue, Sep 3, 2024 at 2:07 PM gregkh@linuxfoundation.org
> > <gregkh@linuxfoundation.org> wrote:
> > > 
> > > On Tue, Sep 03, 2024 at 11:56:17AM +0000, Siddh Raman Pant wrote:
> > > > On Mon, 29 Jul 2024 16:32:36 +0200, Greg Kroah-Hartman wrote:
> > > > > In the Linux kernel, the following vulnerability has been resolved:
> > > > > 
> > > > > udp: Set SOCK_RCU_FREE earlier in udp_lib_get_port().
> > > > > 
> > > > > [...]
> > > > > 
> > > > > We had the same bug in TCP and fixed it in commit 871019b22d1b ("net:
> > > > > set SOCK_RCU_FREE before inserting socket into hashtable").
> > > > > 
> > > > > Let's apply the same fix for UDP.
> > > > > 
> > > > > [...]
> > > > > 
> > > > > The Linux kernel CVE team has assigned CVE-2024-41041 to this issue.
> > > > > 
> > > > > 
> > > > > Affected and fixed versions
> > > > > ===========================
> > > > > 
> > > > >     Issue introduced in 4.20 with commit 6acc9b432e67 and fixed in 5.4.280 with commit 7a67c4e47626
> > > > >     Issue introduced in 4.20 with commit 6acc9b432e67 and fixed in 5.10.222 with commit 9f965684c57c
> > > > 
> > > > These versions don't have the TCP fix backported. Please do so.
> > > 
> > > What fix backported exactly to where?  Please be more specific.  Better
> > > yet, please provide working, and tested, backports.
> > 
> > 
> > commit 871019b22d1bcc9fab2d1feba1b9a564acbb6e99
> > Author: Stanislav Fomichev <sdf@fomichev.me>
> > Date:   Wed Nov 8 13:13:25 2023 -0800
> > 
> >     net: set SOCK_RCU_FREE before inserting socket into hashtable
> > ...
> >     Fixes: 6acc9b432e67 ("bpf: Add helper to retrieve socket in BPF")
> > 
> > It seems 871019b22d1bcc9fab2d1feba1b9a564acbb6e99 has not been pushed
> > to 5.10 or 5.4 lts
> > 
> > Stanislav mentioned a WARN_ONCE() being hit, I presume we could push
> > the patch to 5.10 and 5.4.
> > 
> > I guess this was skipped because of a merge conflict.
> 
> Yes, the commit does not apply, we need someone to send a working
> backport for us to be able to take it.
> 
> Siddh, can you please do this?

Sure.

I see there are Stable-dep commits too, but the seem unrelated and
require some commits from another feature patchset. Do I need to
backport them too?

Thanks,
Siddh

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: CVE-2024-41041: udp: Set SOCK_RCU_FREE earlier in udp_lib_get_port().

[gregkh]

On Wed, Sep 04, 2024 at 11:26:36AM +0000, Siddh Raman Pant wrote:
> On Tue, Sep 03 2024 at 18:28:14 +0530, gregkh@linuxfoundation.org
> wrote:
> > On Tue, Sep 03, 2024 at 02:53:57PM +0200, Eric Dumazet wrote:
> > > On Tue, Sep 3, 2024 at 2:07 PM gregkh@linuxfoundation.org
> > > <gregkh@linuxfoundation.org> wrote:
> > > > 
> > > > On Tue, Sep 03, 2024 at 11:56:17AM +0000, Siddh Raman Pant wrote:
> > > > > On Mon, 29 Jul 2024 16:32:36 +0200, Greg Kroah-Hartman wrote:
> > > > > > In the Linux kernel, the following vulnerability has been resolved:
> > > > > > 
> > > > > > udp: Set SOCK_RCU_FREE earlier in udp_lib_get_port().
> > > > > > 
> > > > > > [...]
> > > > > > 
> > > > > > We had the same bug in TCP and fixed it in commit 871019b22d1b ("net:
> > > > > > set SOCK_RCU_FREE before inserting socket into hashtable").
> > > > > > 
> > > > > > Let's apply the same fix for UDP.
> > > > > > 
> > > > > > [...]
> > > > > > 
> > > > > > The Linux kernel CVE team has assigned CVE-2024-41041 to this issue.
> > > > > > 
> > > > > > 
> > > > > > Affected and fixed versions
> > > > > > ===========================
> > > > > > 
> > > > > >     Issue introduced in 4.20 with commit 6acc9b432e67 and fixed in 5.4.280 with commit 7a67c4e47626
> > > > > >     Issue introduced in 4.20 with commit 6acc9b432e67 and fixed in 5.10.222 with commit 9f965684c57c
> > > > > 
> > > > > These versions don't have the TCP fix backported. Please do so.
> > > > 
> > > > What fix backported exactly to where?  Please be more specific.  Better
> > > > yet, please provide working, and tested, backports.
> > > 
> > > 
> > > commit 871019b22d1bcc9fab2d1feba1b9a564acbb6e99
> > > Author: Stanislav Fomichev <sdf@fomichev.me>
> > > Date:   Wed Nov 8 13:13:25 2023 -0800
> > > 
> > >     net: set SOCK_RCU_FREE before inserting socket into hashtable
> > > ...
> > >     Fixes: 6acc9b432e67 ("bpf: Add helper to retrieve socket in BPF")
> > > 
> > > It seems 871019b22d1bcc9fab2d1feba1b9a564acbb6e99 has not been pushed
> > > to 5.10 or 5.4 lts
> > > 
> > > Stanislav mentioned a WARN_ONCE() being hit, I presume we could push
> > > the patch to 5.10 and 5.4.
> > > 
> > > I guess this was skipped because of a merge conflict.
> > 
> > Yes, the commit does not apply, we need someone to send a working
> > backport for us to be able to take it.
> > 
> > Siddh, can you please do this?
> 
> Sure.
> 
> I see there are Stable-dep commits too, but the seem unrelated and
> require some commits from another feature patchset. Do I need to
> backport them too?

Do what you think you need to do :)

[PATCH 5.10, 5.4] net: set SOCK_RCU_FREE before inserting socket into hashtable

[Siddh Raman Pant]

[-- Attachment #1: Type: text/plain, Size: 2972 bytes --]

[ Upstream commit 871019b22d1bcc9fab2d1feba1b9a564acbb6e99 ]

We've started to see the following kernel traces:

 WARNING: CPU: 83 PID: 0 at net/core/filter.c:6641 sk_lookup+0x1bd/0x1d0

 Call Trace:
  <IRQ>
  __bpf_skc_lookup+0x10d/0x120
  bpf_sk_lookup+0x48/0xd0
  bpf_sk_lookup_tcp+0x19/0x20
  bpf_prog_<redacted>+0x37c/0x16a3
  cls_bpf_classify+0x205/0x2e0
  tcf_classify+0x92/0x160
  __netif_receive_skb_core+0xe52/0xf10
  __netif_receive_skb_list_core+0x96/0x2b0
  napi_complete_done+0x7b5/0xb70
  <redacted>_poll+0x94/0xb0
  net_rx_action+0x163/0x1d70
  __do_softirq+0xdc/0x32e
  asm_call_irq_on_stack+0x12/0x20
  </IRQ>
  do_softirq_own_stack+0x36/0x50
  do_softirq+0x44/0x70

__inet_hash can race with lockless (rcu) readers on the other cpus:

  __inet_hash
    __sk_nulls_add_node_rcu
    <- (bpf triggers here)
    sock_set_flag(SOCK_RCU_FREE)

Let's move the SOCK_RCU_FREE part up a bit, before we are inserting
the socket into hashtables. Note, that the race is really harmless;
the bpf callers are handling this situation (where listener socket
doesn't have SOCK_RCU_FREE set) correctly, so the only
annoyance is a WARN_ONCE.

More details from Eric regarding SOCK_RCU_FREE timeline:

Commit 3b24d854cb35 ("tcp/dccp: do not touch listener sk_refcnt under
synflood") added SOCK_RCU_FREE. At that time, the precise location of
sock_set_flag(sk, SOCK_RCU_FREE) did not matter, because the thread calling
__inet_hash() owns a reference on sk. SOCK_RCU_FREE was only tested
at dismantle time.

Commit 6acc9b432e67 ("bpf: Add helper to retrieve socket in BPF")
started checking SOCK_RCU_FREE _after_ the lookup to infer whether
the refcount has been taken care of.

Fixes: 6acc9b432e67 ("bpf: Add helper to retrieve socket in BPF")
Reviewed-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Stanislav Fomichev <sdf@google.com>
Reviewed-by: Kuniyuki Iwashima <kuniyu@amazon.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
[Resolved conflict for 5.10 and below.]
Signed-off-by: Siddh Raman Pant <siddh.raman.pant@oracle.com>
---
 net/ipv4/inet_hashtables.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/net/ipv4/inet_hashtables.c b/net/ipv4/inet_hashtables.c
index 56deddeac1b0..0fb5d758264f 100644
--- a/net/ipv4/inet_hashtables.c
+++ b/net/ipv4/inet_hashtables.c
@@ -653,6 +653,7 @@ int __inet_hash(struct sock *sk, struct sock *osk)
 		if (err)
 			goto unlock;
 	}
+	sock_set_flag(sk, SOCK_RCU_FREE);
 	if (IS_ENABLED(CONFIG_IPV6) && sk->sk_reuseport &&
 		sk->sk_family == AF_INET6)
 		__sk_nulls_add_node_tail_rcu(sk, &ilb->nulls_head);
@@ -660,7 +661,6 @@ int __inet_hash(struct sock *sk, struct sock *osk)
 		__sk_nulls_add_node_rcu(sk, &ilb->nulls_head);
 	inet_hash2(hashinfo, sk);
 	ilb->count++;
-	sock_set_flag(sk, SOCK_RCU_FREE);
 	sock_prot_inuse_add(sock_net(sk), sk->sk_prot, 1);
 unlock:
 	spin_unlock(&ilb->lock);
-- 
2.45.2


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 5.10, 5.4] net: set SOCK_RCU_FREE before inserting socket into hashtable

[gregkh]

On Wed, Sep 04, 2024 at 01:06:45PM +0000, Siddh Raman Pant wrote:
> [ Upstream commit 871019b22d1bcc9fab2d1feba1b9a564acbb6e99 ]
> 
> We've started to see the following kernel traces:
> 
>  WARNING: CPU: 83 PID: 0 at net/core/filter.c:6641 sk_lookup+0x1bd/0x1d0
> 
>  Call Trace:
>   <IRQ>
>   __bpf_skc_lookup+0x10d/0x120
>   bpf_sk_lookup+0x48/0xd0
>   bpf_sk_lookup_tcp+0x19/0x20
>   bpf_prog_<redacted>+0x37c/0x16a3
>   cls_bpf_classify+0x205/0x2e0
>   tcf_classify+0x92/0x160
>   __netif_receive_skb_core+0xe52/0xf10
>   __netif_receive_skb_list_core+0x96/0x2b0
>   napi_complete_done+0x7b5/0xb70
>   <redacted>_poll+0x94/0xb0
>   net_rx_action+0x163/0x1d70
>   __do_softirq+0xdc/0x32e
>   asm_call_irq_on_stack+0x12/0x20
>   </IRQ>
>   do_softirq_own_stack+0x36/0x50
>   do_softirq+0x44/0x70
> 
> __inet_hash can race with lockless (rcu) readers on the other cpus:
> 
>   __inet_hash
>     __sk_nulls_add_node_rcu
>     <- (bpf triggers here)
>     sock_set_flag(SOCK_RCU_FREE)
> 
> Let's move the SOCK_RCU_FREE part up a bit, before we are inserting
> the socket into hashtables. Note, that the race is really harmless;
> the bpf callers are handling this situation (where listener socket
> doesn't have SOCK_RCU_FREE set) correctly, so the only
> annoyance is a WARN_ONCE.
> 
> More details from Eric regarding SOCK_RCU_FREE timeline:
> 
> Commit 3b24d854cb35 ("tcp/dccp: do not touch listener sk_refcnt under
> synflood") added SOCK_RCU_FREE. At that time, the precise location of
> sock_set_flag(sk, SOCK_RCU_FREE) did not matter, because the thread calling
> __inet_hash() owns a reference on sk. SOCK_RCU_FREE was only tested
> at dismantle time.
> 
> Commit 6acc9b432e67 ("bpf: Add helper to retrieve socket in BPF")
> started checking SOCK_RCU_FREE _after_ the lookup to infer whether
> the refcount has been taken care of.
> 
> Fixes: 6acc9b432e67 ("bpf: Add helper to retrieve socket in BPF")
> Reviewed-by: Eric Dumazet <edumazet@google.com>
> Signed-off-by: Stanislav Fomichev <sdf@google.com>
> Reviewed-by: Kuniyuki Iwashima <kuniyu@amazon.com>
> Signed-off-by: David S. Miller <davem@davemloft.net>
> [Resolved conflict for 5.10 and below.]
> Signed-off-by: Siddh Raman Pant <siddh.raman.pant@oracle.com>
> ---
>  net/ipv4/inet_hashtables.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Now  queued up, thanks.

greg k-h