Re: 2.6.8-rc4-bk1 problem: unregister_netdevice: waiting for ppp0 to become free. Usage count = 1

Re: 2.6.8-rc4-bk1 problem: unregister_netdevice: waiting for ppp0 to become free. Usage count = 1

[Vladimir B. Savkin]

On Thu, Aug 12, 2004 at 08:18:28PM +0100, Nuno Silva wrote:
> Hi!
> 
> With 2.6.8-rc4-bk1 I get "Aug 12 17:33:10 puma kernel:
> unregister_netdevice: waiting for ppp0 to become free. Usage count = 1"
> in the logs after pppd exit.
> 
> Also, the box won't reboot and print that message forever in the
> console. sysrq-U && sysrq-R did it :-)
> 
> The last version I tried was 2.6.8-rc2-bk11 and, wrt this prob, is
> running fine. So, the problem is in that window and the changelog for
> rc4 mentions something about ppp:
> http://kernel.org/pub/linux/kernel/v2.6/testing/ChangeLog-2.6.8-rc4
> 
> If someone requires more information or tests feel free to ask!

I saw this too, with 2.6.7-rc3-mm1.
I have discovered that it happens because of idle TCP socket
holds a reference to a network device.
After killing associated process, device was freed immediately.

~
:wq
                                        With best regards, 
                                           Vladimir Savkin. 

Re: 2.6.8-rc4-bk1 problem: unregister_netdevice: waiting for ppp0 to become free. Usage count = 1

[Nuno Silva]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Vladimir B. Savkin wrote:
| On Thu, Aug 12, 2004 at 08:18:28PM +0100, Nuno Silva wrote:
|
|>Hi!
|>
|>With 2.6.8-rc4-bk1 I get "Aug 12 17:33:10 puma kernel:
|>unregister_netdevice: waiting for ppp0 to become free. Usage count = 1"
|>in the logs after pppd exit.
|>
|>Also, the box won't reboot and print that message forever in the
|>console. sysrq-U && sysrq-R did it :-)
|>
|>The last version I tried was 2.6.8-rc2-bk11 and, wrt this prob, is
|>running fine. So, the problem is in that window and the changelog for
|>rc4 mentions something about ppp:
|>http://kernel.org/pub/linux/kernel/v2.6/testing/ChangeLog-2.6.8-rc4
|>
|>If someone requires more information or tests feel free to ask!
|
|
| I saw this too, with 2.6.7-rc3-mm1.
| I have discovered that it happens because of idle TCP socket
| holds a reference to a network device.
| After killing associated process, device was freed immediately.
|

I waited for 5 mins before sysrq-U && sysrq-R.
Anyway, if I 'killall pppd' and then issue 'ifconfig -a' the ifconfig
command will hang.

This didn't happen with 2.6.8-rc2-bk11 (the one I'm running now), so
something changed... I'm I the only one with ppp/pppd/pppoe who tried
2.6.8-rc4-bk1? :-) Any success reports?

Regards,
Nuno Silva

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBHSYlOPig54MP17wRAs1NAJ0dMRd7tqnpRk/tnxZj7xSeoUL3UgCgrk50
tnzeT6sy3CauOI5WK7HF16o=
=mkWf
-----END PGP SIGNATURE-----

Re: 2.6.8-rc4-bk1 problem: unregister_netdevice: waiting for ppp0 to become free. Usage count = 1

[Nuno Silva]

[-- Attachment #1: Type: text/plain, Size: 2187 bytes --]

Nuno Silva wrote:
> Vladimir B. Savkin wrote:
> | On Thu, Aug 12, 2004 at 08:18:28PM +0100, Nuno Silva wrote:
> |
> |>Hi!
> |>
> |>With 2.6.8-rc4-bk1 I get "Aug 12 17:33:10 puma kernel:
> |>unregister_netdevice: waiting for ppp0 to become free. Usage count = 1"
> |>in the logs after pppd exit.
> |>
> |>Also, the box won't reboot and print that message forever in the
> |>console. sysrq-U && sysrq-R did it :-)
> |>
> |>The last version I tried was 2.6.8-rc2-bk11 and, wrt this prob, is
> |>running fine. So, the problem is in that window and the changelog for
> |>rc4 mentions something about ppp:
> |>http://kernel.org/pub/linux/kernel/v2.6/testing/ChangeLog-2.6.8-rc4
> |>
> |>If someone requires more information or tests feel free to ask!
> |
> |
> | I saw this too, with 2.6.7-rc3-mm1.
> | I have discovered that it happens because of idle TCP socket
> | holds a reference to a network device.
> | After killing associated process, device was freed immediately.
> |
> 
> I waited for 5 mins before sysrq-U && sysrq-R.
> Anyway, if I 'killall pppd' and then issue 'ifconfig -a' the ifconfig
> command will hang.
> 
> This didn't happen with 2.6.8-rc2-bk11 (the one I'm running now), so
> something changed... I'm I the only one with ppp/pppd/pppoe who tried
> 2.6.8-rc4-bk1? :-) Any success reports?
> 

OK, I just tested again and the problem persists. However this time I 
had some spare moments so I rebooted a few times to isolate the problem.

The problem is in the QoS code. If I start ppp whithout the 
/etc/ppp/ip-up.d/wshaper script everything is fine. If I try the 
wshaper.htb it's also fine. So, I'd say that the problem is in the CBQ 
section. This time I waited for 1 hour and got hundreds of 
"unregister_netdevice: waiting for ppp0 to become free. Usage count = 1" 
in the console/syslog. pppd eats 99% CPU. ifconfig freezes. Even reboot 
isn't possible without sysrq's help.

This problem was introduced between 2.6.8-rc2-bk11 and 2.6.8-rc4-bk1 and 
always happens. Right now I'm testing with 2.6.8.1 with a patch from Mr. 
Miller -- "cacheline-align qdisc data in qdisc_create()" (attached).

If someone needs more details feel free to ask!

Regards,
Nuno Silva


[-- Attachment #2: qdisc.diff --]
[-- Type: text/x-patch, Size: 1816 bytes --]

# This is a BitKeeper generated diff -Nru style patch.
#
# ChangeSet
#   2004/08/15 19:33:16-07:00 kaber@trash.net 
#   [PKT_SCHED]: cacheline-align qdisc data in qdisc_create()
#   
#   Signed-off-by: Patrick McHardy <kaber@trash.net>
#   Signed-off-by: David S. Miller <davem@redhat.com>
# 
# net/sched/sch_api.c
#   2004/08/15 19:32:59-07:00 kaber@trash.net +13 -8
#   [PKT_SCHED]: cacheline-align qdisc data in qdisc_create()
#   
#   Signed-off-by: Patrick McHardy <kaber@trash.net>
#   Signed-off-by: David S. Miller <davem@redhat.com>
# 
diff -Nru a/net/sched/sch_api.c b/net/sched/sch_api.c
--- a/net/sched/sch_api.c	2004-08-19 07:33:10 -07:00
+++ b/net/sched/sch_api.c	2004-08-19 07:33:10 -07:00
@@ -389,7 +389,8 @@
 {
 	int err;
 	struct rtattr *kind = tca[TCA_KIND-1];
-	struct Qdisc *sch = NULL;
+	void *p = NULL;
+	struct Qdisc *sch;
 	struct Qdisc_ops *ops;
 	int size;
 
@@ -407,12 +408,18 @@
 	if (ops == NULL)
 		goto err_out;
 
-	size = sizeof(*sch) + ops->priv_size;
+	/* ensure that the Qdisc and the private data are 32-byte aligned */
+	size = ((sizeof(*sch) + QDISC_ALIGN_CONST) & ~QDISC_ALIGN_CONST);
+	size += ops->priv_size + QDISC_ALIGN_CONST;
 
-	sch = kmalloc(size, GFP_KERNEL);
+	p = kmalloc(size, GFP_KERNEL);
 	err = -ENOBUFS;
-	if (!sch)
+	if (!p)
 		goto err_out;
+	memset(p, 0, size);
+	sch = (struct Qdisc *)(((unsigned long)p + QDISC_ALIGN_CONST)
+	                       & ~QDISC_ALIGN_CONST);
+	sch->padded = (char *)sch - (char *)p;
 
 	/* Grrr... Resolve race condition with module unload */
 
@@ -420,8 +427,6 @@
 	if (ops != qdisc_lookup_ops(kind))
 		goto err_out;
 
-	memset(sch, 0, size);
-
 	INIT_LIST_HEAD(&sch->list);
 	skb_queue_head_init(&sch->q);
 
@@ -470,8 +475,8 @@
 
 err_out:
 	*errp = err;
-	if (sch)
-		kfree(sch);
+	if (p)
+		kfree(p);
 	return NULL;
 }
 

Re: 2.6.8-rc4-bk1 problem: unregister_netdevice: waiting for ppp0 to become free. Usage count = 1

[David S. Miller]

On Sun, 22 Aug 2004 03:13:55 +0100
Nuno Silva <nuno.silva@vgertech.com> wrote:

> This problem was introduced between 2.6.8-rc2-bk11 and 2.6.8-rc4-bk1 and 
> always happens. Right now I'm testing with 2.6.8.1 with a patch from Mr. 
> Miller -- "cacheline-align qdisc data in qdisc_create()" (attached).

Does that patch fix the problem?

Re: 2.6.8-rc4-bk1 problem: unregister_netdevice: waiting for ppp0 to become free. Usage count = 1

[Nuno Silva]

[-- Attachment #1: Type: text/plain, Size: 2000 bytes --]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

David S. Miller wrote:
| On Sun, 22 Aug 2004 03:13:55 +0100
| Nuno Silva <nuno.silva@vgertech.com> wrote:
|
|
|>This problem was introduced between 2.6.8-rc2-bk11 and 2.6.8-rc4-bk1 and
|>always happens. Right now I'm testing with 2.6.8.1 with a patch from Mr.
|>Miller -- "cacheline-align qdisc data in qdisc_create()" (attached).
|
|
| Does that patch fix the problem?
|

Hi!

No, but that patch is necessary for me to use QoS at all, or the kernel
oopses.

I was refering to that patch for you to know that's not the same
problem. Sorry for not being 100% clear.

If you want to freeze your networking :-) you can do this:

remove pppd (or adjust the ppp0 interface name later)
# killall pppd

# sleep 10

make a fony connection to 1.1.1.2 using /dev/null
#pppd /dev/null connect /bin/true persist demand 1.1.1.1:1.1.1.2

# sleep 1

add a CBQ root to ppp0 (or ppp1, if you didn't killall and have another
running)
# tc qdisc add dev ppp0 root handle 1: cbq avpkt 1000 bandwidth 10mbit

ping the fony remote:
# ping 1.1.1.2

kill pppd
#killall pppd

Now check dmesg, you should get "unregister_netdevice: waiting for ppp0
to become free. Usage count = 1"

Another remark: In single user mode pppd died and system load is 0. In
initlevel 3, pppd eats all my CPU... Don't know why.

Right now I'm testing 2.6.8.1-ck3 (because of that can't-burn-cdrom
thing and other stuff) so the attached /proc/config.gz is somewhat
non-vanilla, but this happens with 2.6.8-rc4-bk1[*] and above.

[*] this version is "bad" and 2.6.8-rc2-bk11 is "good". Didn't test with
rc3...

As you can see in the .config I'm using the OLD POLICER.

As always, please request more info if you need it! :-)

Regards,
Nuno Silva
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBKFL5OPig54MP17wRAsHeAJ4nZFSW7nat0/aahUokzriHIueO5wCgqPCM
EogRAFF/jofq4kN609j6dFc=
=1HPc
-----END PGP SIGNATURE-----

[-- Attachment #2: config.gz --]
[-- Type: application/x-gzip, Size: 10941 bytes --]

Re: 2.6.8-rc4-bk1 problem: unregister_netdevice: waiting for ppp0 to become free. Usage count = 1

[Herbert Xu]

Nuno Silva <nuno.silva@vgertech.com> wrote:
> 
> The problem is in the QoS code. If I start ppp whithout the 

OK, this appears to be due to the changeset titled

[PKT_SCHED]: Refcount qdisc->dev for __qdisc_destroy rcu-callback

It adds a reference to dev.

I don't see any code that cleans up that reference when the dev goes
down.  So someone needs to add that similar to the code in net/core/dst.c.

Patrick, could you please have a look at this?

Thanks,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: 2.6.8-rc4-bk1 problem: unregister_netdevice: waiting for ppp0 to become free. Usage count = 1

[Patrick McHardy]

[-- Attachment #1: Type: text/plain, Size: 712 bytes --]

Herbert Xu wrote:

>Nuno Silva <nuno.silva@vgertech.com> wrote:
>  
>
>>The problem is in the QoS code. If I start ppp whithout the 
>>    
>>
>
>OK, this appears to be due to the changeset titled
>
>[PKT_SCHED]: Refcount qdisc->dev for __qdisc_destroy rcu-callback
>
>It adds a reference to dev.
>
>I don't see any code that cleans up that reference when the dev goes
>down.  So someone needs to add that similar to the code in net/core/dst.c.
>
>Patrick, could you please have a look at this?
>  
>
The reference is dropped in __qdisc_destroy. The problem lies in the CBQ
qdisc, it doesn't destroy the root-class and leaks the inner qdisc. These
two patches for 2.4 and 2.6 fix the problem.

Regards
Patrick



[-- Attachment #2: 01-2.4-cbq-leaks.diff --]
[-- Type: text/x-patch, Size: 1438 bytes --]

# This is a BitKeeper generated diff -Nru style patch.
#
# ChangeSet
#   2004/08/22 14:37:58+02:00 kaber@coreworks.de 
#   [PKT_SCHED]: Fix class leak in CBQ scheduler
#   
#   Signed-off-by: Patrick McHardy <kaber@trash.net>
# 
# net/sched/sch_cbq.c
#   2004/08/22 14:37:27+02:00 kaber@coreworks.de +8 -6
#   [PKT_SCHED]: Fix class leak in CBQ scheduler
# 
diff -Nru a/net/sched/sch_cbq.c b/net/sched/sch_cbq.c
--- a/net/sched/sch_cbq.c	2004-08-22 14:38:18 +02:00
+++ b/net/sched/sch_cbq.c	2004-08-22 14:38:18 +02:00
@@ -1712,15 +1712,18 @@
 	}
 }
 
-static void cbq_destroy_class(struct cbq_class *cl)
+static void cbq_destroy_class(struct Qdisc *sch, struct cbq_class *cl)
 {
+	struct cbq_sched_data *q = (struct cbq_sched_data *)sch->data;
+
 	cbq_destroy_filters(cl);
 	qdisc_destroy(cl->q);
 	qdisc_put_rtab(cl->R_tab);
 #ifdef CONFIG_NET_ESTIMATOR
 	qdisc_kill_estimator(&cl->stats);
 #endif
-	kfree(cl);
+	if (cl != &q->link)
+		kfree(cl);
 }
 
 static void
@@ -1743,8 +1746,7 @@
 
 		for (cl = q->classes[h]; cl; cl = next) {
 			next = cl->next;
-			if (cl != &q->link)
-				cbq_destroy_class(cl);
+			cbq_destroy_class(sch, cl);
 		}
 	}
 
@@ -1766,7 +1768,7 @@
 		spin_unlock_bh(&sch->dev->queue_lock);
 #endif
 
-		cbq_destroy_class(cl);
+		cbq_destroy_class(sch, cl);
 	}
 }
 
@@ -2000,7 +2002,7 @@
 	sch_tree_unlock(sch);
 
 	if (--cl->refcnt == 0)
-		cbq_destroy_class(cl);
+		cbq_destroy_class(sch, cl);
 
 	return 0;
 }

[-- Attachment #3: 01-2.6-cbq-leaks.diff --]
[-- Type: text/x-patch, Size: 1419 bytes --]

# This is a BitKeeper generated diff -Nru style patch.
#
# ChangeSet
#   2004/08/22 14:30:32+02:00 kaber@coreworks.de 
#   [PKT_SCHED]: Fix class leak in CBQ scheduler
#   
#   Signed-off-by: Patrick McHardy <kaber@trash.net>
# 
# net/sched/sch_cbq.c
#   2004/08/22 14:30:13+02:00 kaber@coreworks.de +8 -6
#   [PKT_SCHED]: Fix class leak in CBQ scheduler
# 
diff -Nru a/net/sched/sch_cbq.c b/net/sched/sch_cbq.c
--- a/net/sched/sch_cbq.c	2004-08-22 14:33:59 +02:00
+++ b/net/sched/sch_cbq.c	2004-08-22 14:33:59 +02:00
@@ -1746,15 +1746,18 @@
 	}
 }
 
-static void cbq_destroy_class(struct cbq_class *cl)
+static void cbq_destroy_class(struct Qdisc *sch, struct cbq_class *cl)
 {
+	struct cbq_sched_data *q = qdisc_priv(sch);
+
 	cbq_destroy_filters(cl);
 	qdisc_destroy(cl->q);
 	qdisc_put_rtab(cl->R_tab);
 #ifdef CONFIG_NET_ESTIMATOR
 	qdisc_kill_estimator(&cl->stats);
 #endif
-	kfree(cl);
+	if (cl != &q->link)
+		kfree(cl);
 }
 
 static void
@@ -1777,8 +1780,7 @@
 
 		for (cl = q->classes[h]; cl; cl = next) {
 			next = cl->next;
-			if (cl != &q->link)
-				cbq_destroy_class(cl);
+			cbq_destroy_class(sch, cl);
 		}
 	}
 
@@ -1799,7 +1801,7 @@
 		spin_unlock_bh(&sch->dev->queue_lock);
 #endif
 
-		cbq_destroy_class(cl);
+		cbq_destroy_class(sch, cl);
 	}
 }
 
@@ -2035,7 +2037,7 @@
 	sch_tree_unlock(sch);
 
 	if (--cl->refcnt == 0)
-		cbq_destroy_class(cl);
+		cbq_destroy_class(sch, cl);
 
 	return 0;
 }

Re: 2.6.8-rc4-bk1 problem: unregister_netdevice: waiting for ppp0 to become free. Usage count = 1

[David S. Miller]

On Sun, 22 Aug 2004 14:39:57 +0200
Patrick McHardy <kaber@trash.net> wrote:

> Herbert Xu wrote:
> 
> >Nuno Silva <nuno.silva@vgertech.com> wrote:
> >  
> >
> >>The problem is in the QoS code. If I start ppp whithout the 
> >>    
> >>
> >
> >OK, this appears to be due to the changeset titled
> >
> >[PKT_SCHED]: Refcount qdisc->dev for __qdisc_destroy rcu-callback
> >
> >It adds a reference to dev.
> >
> >I don't see any code that cleans up that reference when the dev goes
> >down.  So someone needs to add that similar to the code in net/core/dst.c.
> >
> >Patrick, could you please have a look at this?
> >  
> The reference is dropped in __qdisc_destroy. The problem lies in the CBQ
> qdisc, it doesn't destroy the root-class and leaks the inner qdisc. These
> two patches for 2.4 and 2.6 fix the problem.

Awesome, good detective work guys.

Patch applied, thanks.

Re: 2.6.8-rc4-bk1 problem: unregister_netdevice: waiting for ppp0 to become free. Usage count = 1

[Nuno Silva]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Patrick McHardy wrote:
| Herbert Xu wrote:
|
|> Nuno Silva <nuno.silva@vgertech.com> wrote:
|>
|>
|>> The problem is in the QoS code. If I start ppp whithout the
|>
|>
|> OK, this appears to be due to the changeset titled
|>
|> [PKT_SCHED]: Refcount qdisc->dev for __qdisc_destroy rcu-callback
|>
|> It adds a reference to dev.
|>
|> I don't see any code that cleans up that reference when the dev goes
|> down.  So someone needs to add that similar to the code in
|> net/core/dst.c.
|>
|> Patrick, could you please have a look at this?
|>
|>
| The reference is dropped in __qdisc_destroy. The problem lies in the CBQ
| qdisc, it doesn't destroy the root-class and leaks the inner qdisc. These
| two patches for 2.4 and 2.6 fix the problem.

Hi!

Just to give some feedback: IT WORKS! Thanks!

Didn't try with 2.4, but it works very well with 2.6.8.1.
Thanks again,
Nuno Silva

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBKi/FOPig54MP17wRAiYBAJ41ZGzauhY6dDVtylWkLSD3V+vx9QCgteNF
21sEmv0wqP+9hdnXEc4DNBE=
=ByPY
-----END PGP SIGNATURE-----

Re: 2.6.8-rc4-bk1 problem: unregister_netdevice: waiting for ppp0 to become free. Usage count = 1

[Jurriaan]

From: David S. Miller <davem@redhat.com>
Date: Sun, Aug 22, 2004 at 09:47:46PM -0700
> On Sun, 22 Aug 2004 14:39:57 +0200
> Patrick McHardy <kaber@trash.net> wrote:
> 
> > Herbert Xu wrote:
> > 
> > >Nuno Silva <nuno.silva@vgertech.com> wrote:
> > >  
> > >OK, this appears to be due to the changeset titled
> > >
> > >[PKT_SCHED]: Refcount qdisc->dev for __qdisc_destroy rcu-callback
> > >
> > >It adds a reference to dev.
> > >
> > >I don't see any code that cleans up that reference when the dev goes
> > >down.  So someone needs to add that similar to the code in net/core/dst.c.
> > >
> > >Patrick, could you please have a look at this?
> > >  
> > The reference is dropped in __qdisc_destroy. The problem lies in the CBQ
> > qdisc, it doesn't destroy the root-class and leaks the inner qdisc. These
> > two patches for 2.4 and 2.6 fix the problem.
> 
> Awesome, good detective work guys.
> 
> Patch applied, thanks.

I get the same error with 2.6.9-rc1-mm1, but now for my ipv6-over-ipv4
tunnel.

unregister_netdevice: waiting for xs6all to become free

etc. when trying to reboot or shutdown. Usage count = 1

I'm pretty sure I didn't see this in 2.6.8.1 nor in 2.6.8.1-mm1.

This is a plain ipv6-over-ipv4 tunnel like this:

/etc/network/interfaces:
auto xs6all
iface xs6all inet6 v4tunnel
        endpoint xxx.xxx.xxx.xxx
        up ip route add 2000::0/3 via xxx:xxx:xxx:xxx:xxx:xxx
        address xxx:xxx:xxx:xxx:xxx:xxx
        netmask 64
        up ip tunnel change xs6all ttl 64

I'm running Debian Unstable with all the latest updates as of today.

Thanks for any hints,
Jurriaan
-- 
"At least she's on our side," said Lindholm.
Corbie looked at him. "Investigators aren't on anybody's side."
	Simon R Green - Hellworld
Debian (Unstable) GNU/Linux 2.6.9-rc1 2x6078 bogomips load 0.46