[PATCH net] l2tp: remove skb_dst_set() from l2tp_xmit_skb()

[PATCH net] l2tp: remove skb_dst_set() from l2tp_xmit_skb()

[Xin Long]

In the tx path of l2tp, l2tp_xmit_skb() calls skb_dst_set() to set
skb's dst. However, it will eventually call inet6_csk_xmit() or
ip_queue_xmit() where skb's dst will be overwritten by:

   skb_dst_set_noref(skb, dst);

without releasing the old dst in skb. Then it causes dst/dev refcnt leak:

  unregister_netdevice: waiting for eth0 to become free. Usage count = 1

This can be reproduced by simply running:

  # modprobe l2tp_eth && modprobe l2tp_ip
  # sh ./tools/testing/selftests/net/l2tp.sh

So before going to inet6_csk_xmit() or ip_queue_xmit(), skb's dst
should be dropped. This patch is to fix it by removing skb_dst_set()
from l2tp_xmit_skb() and moving skb_dst_drop() into l2tp_xmit_core().

Fixes: 3557baabf280 ("[L2TP]: PPP over L2TP driver core")
Reported-by: Hangbin Liu <liuhangbin@gmail.com>
Signed-off-by: Xin Long <lucien.xin@gmail.com>
---
 net/l2tp/l2tp_core.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/net/l2tp/l2tp_core.c b/net/l2tp/l2tp_core.c
index fcb53ed..df133c24 100644
--- a/net/l2tp/l2tp_core.c
+++ b/net/l2tp/l2tp_core.c
@@ -1028,6 +1028,7 @@ static void l2tp_xmit_core(struct l2tp_session *session, struct sk_buff *skb,
 
 	/* Queue the packet to IP for output */
 	skb->ignore_df = 1;
+	skb_dst_drop(skb);
 #if IS_ENABLED(CONFIG_IPV6)
 	if (l2tp_sk_is_v6(tunnel->sock))
 		error = inet6_csk_xmit(tunnel->sock, skb, NULL);
@@ -1099,10 +1100,6 @@ int l2tp_xmit_skb(struct l2tp_session *session, struct sk_buff *skb, int hdr_len
 		goto out_unlock;
 	}
 
-	/* Get routing info from the tunnel socket */
-	skb_dst_drop(skb);
-	skb_dst_set(skb, sk_dst_check(sk, 0));
-
 	inet = inet_sk(sk);
 	fl = &inet->cork.fl;
 	switch (tunnel->encap) {
-- 
2.1.0

Re: [PATCH net] l2tp: remove skb_dst_set() from l2tp_xmit_skb()

[James Chapman]

On  Tue, Jul 07, 2020 at 02:02:32 +0800, Xin Long wrote:
> In the tx path of l2tp, l2tp_xmit_skb() calls skb_dst_set() to set
> skb's dst. However, it will eventually call inet6_csk_xmit() or
> ip_queue_xmit() where skb's dst will be overwritten by:
> 
>    skb_dst_set_noref(skb, dst);
> 
> without releasing the old dst in skb. Then it causes dst/dev refcnt leak:
> 
>   unregister_netdevice: waiting for eth0 to become free. Usage count = 1
> 
> This can be reproduced by simply running:
> 
>   # modprobe l2tp_eth && modprobe l2tp_ip
>   # sh ./tools/testing/selftests/net/l2tp.sh
> 
> So before going to inet6_csk_xmit() or ip_queue_xmit(), skb's dst
> should be dropped. This patch is to fix it by removing skb_dst_set()
> from l2tp_xmit_skb() and moving skb_dst_drop() into l2tp_xmit_core().
> 
> Fixes: 3557baabf280 ("[L2TP]: PPP over L2TP driver core")
> Reported-by: Hangbin Liu <liuhangbin@gmail.com>
> Signed-off-by: Xin Long <lucien.xin@gmail.com>
> ---
>  net/l2tp/l2tp_core.c | 5 +----
>  1 file changed, 1 insertion(+), 4 deletions(-)
> 
> diff --git a/net/l2tp/l2tp_core.c b/net/l2tp/l2tp_core.c
> index fcb53ed..df133c24 100644
> --- a/net/l2tp/l2tp_core.c
> +++ b/net/l2tp/l2tp_core.c
> @@ -1028,6 +1028,7 @@ static void l2tp_xmit_core(struct l2tp_session *session, struct sk_buff *skb,
>  
>  	/* Queue the packet to IP for output */
>  	skb->ignore_df = 1;
> +	skb_dst_drop(skb);
>  #if IS_ENABLED(CONFIG_IPV6)
>  	if (l2tp_sk_is_v6(tunnel->sock))
>  		error = inet6_csk_xmit(tunnel->sock, skb, NULL);
> @@ -1099,10 +1100,6 @@ int l2tp_xmit_skb(struct l2tp_session *session, struct sk_buff *skb, int hdr_len
>  		goto out_unlock;
>  	}
>  
> -	/* Get routing info from the tunnel socket */
> -	skb_dst_drop(skb);
> -	skb_dst_set(skb, sk_dst_check(sk, 0));
> -
>  	inet = inet_sk(sk);
>  	fl = &inet->cork.fl;
>  	switch (tunnel->encap) {
> -- 
> 2.1.0
> 

This patch doesn't seem right.

For ipv4, the skb dst is used by skb_rtable. In ip_queue_xmit, if
skb_rtable returns a route, it follows the packet_routed label and
skb_dst_set_noref isn't done. Your patch is forcing every ipv4 l2tp
packet to be routed, which isn't what we want.

I ran l2tp.sh and found that the issue happens only for l2tp tests
that use IPv6 IPSec in a routed topology.

Perhaps the real problem is that l2tp shouldn't be using
inet6_csk_xmit and should instead use ip6_xmit?

Please hold off on applying this patch while I look into it.

Re: [PATCH net] l2tp: remove skb_dst_set() from l2tp_xmit_skb()

[Xin Long]

On Wed, Jul 8, 2020 at 1:24 AM James Chapman <jchapman@katalix.com> wrote:
>
> On  Tue, Jul 07, 2020 at 02:02:32 +0800, Xin Long wrote:
> > In the tx path of l2tp, l2tp_xmit_skb() calls skb_dst_set() to set
> > skb's dst. However, it will eventually call inet6_csk_xmit() or
> > ip_queue_xmit() where skb's dst will be overwritten by:
> >
> >    skb_dst_set_noref(skb, dst);
> >
> > without releasing the old dst in skb. Then it causes dst/dev refcnt leak:
> >
> >   unregister_netdevice: waiting for eth0 to become free. Usage count = 1
> >
> > This can be reproduced by simply running:
> >
> >   # modprobe l2tp_eth && modprobe l2tp_ip
> >   # sh ./tools/testing/selftests/net/l2tp.sh
> >
> > So before going to inet6_csk_xmit() or ip_queue_xmit(), skb's dst
> > should be dropped. This patch is to fix it by removing skb_dst_set()
> > from l2tp_xmit_skb() and moving skb_dst_drop() into l2tp_xmit_core().
> >
> > Fixes: 3557baabf280 ("[L2TP]: PPP over L2TP driver core")
> > Reported-by: Hangbin Liu <liuhangbin@gmail.com>
> > Signed-off-by: Xin Long <lucien.xin@gmail.com>
> > ---
> >  net/l2tp/l2tp_core.c | 5 +----
> >  1 file changed, 1 insertion(+), 4 deletions(-)
> >
> > diff --git a/net/l2tp/l2tp_core.c b/net/l2tp/l2tp_core.c
> > index fcb53ed..df133c24 100644
> > --- a/net/l2tp/l2tp_core.c
> > +++ b/net/l2tp/l2tp_core.c
> > @@ -1028,6 +1028,7 @@ static void l2tp_xmit_core(struct l2tp_session *session, struct sk_buff *skb,
> >
> >       /* Queue the packet to IP for output */
> >       skb->ignore_df = 1;
> > +     skb_dst_drop(skb);
> >  #if IS_ENABLED(CONFIG_IPV6)
> >       if (l2tp_sk_is_v6(tunnel->sock))
> >               error = inet6_csk_xmit(tunnel->sock, skb, NULL);
> > @@ -1099,10 +1100,6 @@ int l2tp_xmit_skb(struct l2tp_session *session, struct sk_buff *skb, int hdr_len
> >               goto out_unlock;
> >       }
> >
> > -     /* Get routing info from the tunnel socket */
> > -     skb_dst_drop(skb);
> > -     skb_dst_set(skb, sk_dst_check(sk, 0));
> > -
> >       inet = inet_sk(sk);
> >       fl = &inet->cork.fl;
> >       switch (tunnel->encap) {
> > --
> > 2.1.0
> >
>
> This patch doesn't seem right.

Hi James,
>
> For ipv4, the skb dst is used by skb_rtable. In ip_queue_xmit, if
> skb_rtable returns a route, it follows the packet_routed label and
> skb_dst_set_noref isn't done. Your patch is forcing every ipv4 l2tp
> packet to be routed, which isn't what we want.
Without this patch,
it does skb_dst_drop() in l2tp_xmit_skb(),
then do:
skb_dst_set(skb, sk_dst_check(sk, 0));

With this patch:
it does skb_dst_drop() in l2tp_xmit_core()

then in ip_queue_xmit(), it will do:
rt = (struct rtable *)__sk_dst_check(sk, 0);
skb_dst_set_noref(skb, &rt->dst);

so I don't think this patch drops any useful dst for ipv4.

>
> I ran l2tp.sh and found that the issue happens only for l2tp tests
> that use IPv6 IPSec in a routed topology.
>
> Perhaps the real problem is that l2tp shouldn't be using
> inet6_csk_xmit and should instead use ip6_xmit?
>
> Please hold off on applying this patch while I look into it.

Re: [PATCH net] l2tp: remove skb_dst_set() from l2tp_xmit_skb()

[James Chapman]

On  Wed, Jul 08, 2020 at 04:08:09 +0800, Xin Long wrote:
> On Wed, Jul 8, 2020 at 1:24 AM James Chapman <jchapman@katalix.com> wrote:
> >
> > On  Tue, Jul 07, 2020 at 02:02:32 +0800, Xin Long wrote:
> > > In the tx path of l2tp, l2tp_xmit_skb() calls skb_dst_set() to set
> > > skb's dst. However, it will eventually call inet6_csk_xmit() or
> > > ip_queue_xmit() where skb's dst will be overwritten by:
> > >
> > >    skb_dst_set_noref(skb, dst);
> > >
> > > without releasing the old dst in skb. Then it causes dst/dev refcnt leak:
> > >
> > >   unregister_netdevice: waiting for eth0 to become free. Usage count = 1
> > >
> > > This can be reproduced by simply running:
> > >
> > >   # modprobe l2tp_eth && modprobe l2tp_ip
> > >   # sh ./tools/testing/selftests/net/l2tp.sh
> > >
> > > So before going to inet6_csk_xmit() or ip_queue_xmit(), skb's dst
> > > should be dropped. This patch is to fix it by removing skb_dst_set()
> > > from l2tp_xmit_skb() and moving skb_dst_drop() into l2tp_xmit_core().
> > >
> > > Fixes: 3557baabf280 ("[L2TP]: PPP over L2TP driver core")
> > > Reported-by: Hangbin Liu <liuhangbin@gmail.com>
> > > Signed-off-by: Xin Long <lucien.xin@gmail.com>
> > > ---
> > >  net/l2tp/l2tp_core.c | 5 +----
> > >  1 file changed, 1 insertion(+), 4 deletions(-)
> > >
> > > diff --git a/net/l2tp/l2tp_core.c b/net/l2tp/l2tp_core.c
> > > index fcb53ed..df133c24 100644
> > > --- a/net/l2tp/l2tp_core.c
> > > +++ b/net/l2tp/l2tp_core.c
> > > @@ -1028,6 +1028,7 @@ static void l2tp_xmit_core(struct l2tp_session *session, struct sk_buff *skb,
> > >
> > >       /* Queue the packet to IP for output */
> > >       skb->ignore_df = 1;
> > > +     skb_dst_drop(skb);
> > >  #if IS_ENABLED(CONFIG_IPV6)
> > >       if (l2tp_sk_is_v6(tunnel->sock))
> > >               error = inet6_csk_xmit(tunnel->sock, skb, NULL);
> > > @@ -1099,10 +1100,6 @@ int l2tp_xmit_skb(struct l2tp_session *session, struct sk_buff *skb, int hdr_len
> > >               goto out_unlock;
> > >       }
> > >
> > > -     /* Get routing info from the tunnel socket */
> > > -     skb_dst_drop(skb);
> > > -     skb_dst_set(skb, sk_dst_check(sk, 0));
> > > -
> > >       inet = inet_sk(sk);
> > >       fl = &inet->cork.fl;
> > >       switch (tunnel->encap) {
> > > --
> > > 2.1.0
> > >
> >
> > This patch doesn't seem right.
> 
> Hi James,
> >
> > For ipv4, the skb dst is used by skb_rtable. In ip_queue_xmit, if
> > skb_rtable returns a route, it follows the packet_routed label and
> > skb_dst_set_noref isn't done. Your patch is forcing every ipv4 l2tp
> > packet to be routed, which isn't what we want.
> Without this patch,
> it does skb_dst_drop() in l2tp_xmit_skb(),
> then do:
> skb_dst_set(skb, sk_dst_check(sk, 0));
> 
> With this patch:
> it does skb_dst_drop() in l2tp_xmit_core()
> 
> then in ip_queue_xmit(), it will do:
> rt = (struct rtable *)__sk_dst_check(sk, 0);
> skb_dst_set_noref(skb, &rt->dst);
> 
> so I don't think this patch drops any useful dst for ipv4.

You're right. My mistake.

Thanks for reporting and fixing this!

> >
> > I ran l2tp.sh and found that the issue happens only for l2tp tests
> > that use IPv6 IPSec in a routed topology.
> >
> > Perhaps the real problem is that l2tp shouldn't be using
> > inet6_csk_xmit and should instead use ip6_xmit?
> >
> > Please hold off on applying this patch while I look into it.

Acked-by: James Chapman <jchapman@katalix.com>
Tested-by: James Chapman <jchapman@katalix.com>

Re: [PATCH net] l2tp: remove skb_dst_set() from l2tp_xmit_skb()

[David Miller]

From: Xin Long <lucien.xin@gmail.com>
Date: Tue,  7 Jul 2020 02:02:32 +0800

> In the tx path of l2tp, l2tp_xmit_skb() calls skb_dst_set() to set
> skb's dst. However, it will eventually call inet6_csk_xmit() or
> ip_queue_xmit() where skb's dst will be overwritten by:
> 
>    skb_dst_set_noref(skb, dst);
> 
> without releasing the old dst in skb. Then it causes dst/dev refcnt leak:
> 
>   unregister_netdevice: waiting for eth0 to become free. Usage count = 1
> 
> This can be reproduced by simply running:
> 
>   # modprobe l2tp_eth && modprobe l2tp_ip
>   # sh ./tools/testing/selftests/net/l2tp.sh
> 
> So before going to inet6_csk_xmit() or ip_queue_xmit(), skb's dst
> should be dropped. This patch is to fix it by removing skb_dst_set()
> from l2tp_xmit_skb() and moving skb_dst_drop() into l2tp_xmit_core().
> 
> Fixes: 3557baabf280 ("[L2TP]: PPP over L2TP driver core")
> Reported-by: Hangbin Liu <liuhangbin@gmail.com>
> Signed-off-by: Xin Long <lucien.xin@gmail.com>

Applied and queued up for -stable, thank you.