Re: [PATCH net-next v2 01/16] l2tp: update sk_user_data while holding sk_callback_lock

[Guillaume Nault <g.nault@alphalink.fr>]

On Mon, Feb 12, 2018 at 10:11:05AM +0000, James Chapman wrote:
> Since L2TP hooks on sockets opened by userspace using sk_user_data, we
> may race with other socket families that attempt to use the same
> socket.
> 
> This problem was discovered by syzbot using AF_KCM. KCM has since been
> modified to use only TCP sockets to avoid hitting this issue but we
> should prevent such races in L2TP anyway.
> 
> Fixes: c8fffcea0a079 ("l2tp: Refactor l2tp core driver to make use of the common UDP tunnel function")
> Reported-by: syzbot+8865eaff7f9acd593945@syzkaller.appspotmail.com
> 
> Kernel BUG at net/l2tp/l2tp_ppp.c:176!
> invalid opcode: 0000 [#1] SMP KASAN
> Dumping ftrace buffer:
>    (ftrace buffer empty)
> Modules linked in:
> CPU: 1 PID: 3503 Comm: syzkaller938388 Not tainted 4.15.0-rc7+ #181
> Hardware name: Google Google Compute Engine/Google Compute Engine
> RIP: 0010:pppol2tp_sock_to_session net/l2tp/l2tp_ppp.c:176 [inline]
> RIP: 0010:pppol2tp_sendmsg+0x512/0x670 net/l2tp/l2tp_ppp.c:304
> RSP: 0018:ffff8801d4887438 EFLAGS: 00010293
> RAX: ffff8801bfef2180 RBX: ffff8801bff88440 RCX: ffffffff84ffbca2
> RDX: 0000000000000000 RSI: ffff8801d4887598 RDI: ffff8801bff88820
> RBP: ffff8801d48874a8 R08: 0000000000000000 R09: 1ffff1003a910e17
> R10: 0000000000000003 R11: 0000000000000001 R12: ffff8801bfff9bc0
> R13: 0000000000000000 R14: 0000000000008000 R15: 0000000000000000
> FS:  0000000001194880(0000) GS:ffff8801db300000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000020ea0000 CR3: 00000001bfecf001 CR4: 00000000001606e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
>  sock_sendmsg_nosec net/socket.c:628 [inline]
>  sock_sendmsg+0xca/0x110 net/socket.c:638
>  kernel_sendmsg+0x47/0x60 net/socket.c:646
>  sock_no_sendpage+0x1cc/0x280 net/core/sock.c:2581
>  kernel_sendpage+0xbf/0xe0 net/socket.c:3349
>  kcm_write_msgs+0x404/0x1b80 net/kcm/kcmsock.c:646
>  kcm_sendmsg+0x148d/0x22d0 net/kcm/kcmsock.c:1035
>  sock_sendmsg_nosec net/socket.c:628 [inline]
>  sock_sendmsg+0xca/0x110 net/socket.c:638
>  ___sys_sendmsg+0x767/0x8b0 net/socket.c:2018
>  __sys_sendmsg+0xe5/0x210 net/socket.c:2052
>  SYSC_sendmsg net/socket.c:2063 [inline]
>  SyS_sendmsg+0x2d/0x50 net/socket.c:2059
>  entry_SYSCALL_64_fastpath+0x23/0x9a
> RIP: 0033:0x440159
> RSP: 002b:00007ffe74df8288 EFLAGS: 00000217 ORIG_RAX: 000000000000002e
> RAX: ffffffffffffffda RBX: ffffffffffffffff RCX: 0000000000440159
> RDX: 0000000000000000 RSI: 00000000201fcfc8 RDI: 0000000000000005
> RBP: 00000000006ca018 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000217 R12: 0000000000401ac0
> R13: 0000000000401b50 R14: 0000000000000000 R15: 0000000000000000
> Code: c5 61 70 fc 48 8b 7d d0 e8 7c c2 5b fd 84 c0 74 0d e8 b3 61 70 fc 48 89 df e8 3b 49 2f ff 41 bd f7 ff ff ff eb 86 e8 9e 61 70 fc <0f> 0b 41 bd 95 ff ff ff e9 74 ff ff ff e8 ec 32 a8 fc e9 77 fb
> RIP: pppol2tp_sock_to_session net/l2tp/l2tp_ppp.c:176 [inline] RSP: ffff8801d4887438
> RIP: pppol2tp_sendmsg+0x512/0x670 net/l2tp/l2tp_ppp.c:304 RSP: ffff8801d4887438
> ---
>  net/l2tp/l2tp_core.c | 21 ++++++++++++++++++---
>  net/l2tp/l2tp_ppp.c  |  8 ++++++--
>  2 files changed, 24 insertions(+), 5 deletions(-)
> 
> diff --git a/net/l2tp/l2tp_core.c b/net/l2tp/l2tp_core.c
> index 194a7483bb93..de7dce64173f 100644
> --- a/net/l2tp/l2tp_core.c
> +++ b/net/l2tp/l2tp_core.c
> @@ -1216,6 +1216,7 @@ static void l2tp_tunnel_destruct(struct sock *sk)
>  
>  
>  	/* Disable udp encapsulation */
> +	write_lock_bh(&sk->sk_callback_lock);
>  	switch (tunnel->encap) {
>  	case L2TP_ENCAPTYPE_UDP:
>  		/* No longer an encapsulation socket. See net/ipv4/udp.c */
> @@ -1229,7 +1230,8 @@ static void l2tp_tunnel_destruct(struct sock *sk)
>  
>  	/* Remove hooks into tunnel socket */
>  	sk->sk_destruct = tunnel->old_sk_destruct;
> -	sk->sk_user_data = NULL;
> +	rcu_assign_sk_user_data(sk, NULL);
> +	write_unlock_bh(&sk->sk_callback_lock);
>  
>  	/* Remove the tunnel struct from the tunnel list */
>  	pn = l2tp_pernet(tunnel->l2tp_net);
> @@ -1583,6 +1585,20 @@ int l2tp_tunnel_create(struct net *net, int fd, int version, u32 tunnel_id, u32
>  	}
>  #endif
>  
> +	/* Assign socket sk_user_data. Must be done with
> +	 * sk_callback_lock. Bail if sk_user_data is already assigned.
> +	 */
> +	write_lock_bh(&sk->sk_callback_lock);
> +	if (sk->sk_user_data) {
> +		err = -EALREADY;
> +		write_unlock_bh(&sk->sk_callback_lock);
> +		kfree(tunnel);
> +		tunnel = NULL;
> +		goto err;
> +	}
> +	rcu_assign_sk_user_data(sk, tunnel);
> +	write_unlock_bh(&sk->sk_callback_lock);
> +
I'd rather use this code only to replace the direct ->sk_user_data
assignment (in the 'else' branch). It looks strange to assign it here
using the new locking scheme and let setup_udp_tunnel_sock() re-assign
it later, with the same value but without locking.

Of course, if every user of ->sk_user_data was updated to the new
locking scheme, that should kill the race without having to update
setup_udp_tunnel_sock(). But only KCM locks ->sk_callback_lock for now
and, as you pointed out, it doesn't act on UDP sockets anymore.

Therefore setting ->sk_user_data here rather than at its original place
only brings confusion in my opinion.

And using rcu_assign_sk_user_data() is a bit confusing too. We never treat
->sk_user_data as RCU protected in the rest of the code.

>  	/* Mark socket as an encapsulation socket. See net/ipv4/udp.c */
>  	tunnel->encap = encap;
>  	if (encap == L2TP_ENCAPTYPE_UDP) {
> @@ -1594,8 +1610,6 @@ int l2tp_tunnel_create(struct net *net, int fd, int version, u32 tunnel_id, u32
>  		udp_cfg.encap_destroy = l2tp_udp_encap_destroy;
>  
>  		setup_udp_tunnel_sock(net, sock, &udp_cfg);
> -	} else {
> -		sk->sk_user_data = tunnel;
>  	}