WARNING: CPU: 3 PID: 1034 at kernel/cgroup.c:492

WARNING: CPU: 3 PID: 1034 at kernel/cgroup.c:492

[Jesús Rodríguez Acosta]

Since few days ago (maybe kernel version 4.8.11, but not sure), a
warning with a call trace is appearing from time to time in the system
log, I guess when starting up a KVM machine.

The system looks fine, and if you need more information, just let me
know and I'll try to provide everything requested.

The log below is from the recently released 4.9.0:


Dec 12 10:02:58 desktop kernel: ------------[ cut here ]------------
Dec 12 10:02:58 desktop kernel: WARNING: CPU: 3 PID: 1034 at
kernel/cgroup.c:492 cgroup_get+0x55/0x70
Dec 12 10:02:58 desktop kernel: Modules linked in: vhost_net vhost
macvtap macvlan algif_skcipher af_alg xt_CHECKSUM iptable_mangle
ipt_MASQUERADE nf
Dec 12 10:02:58 desktop kernel:  ablk_helper snd_pcm mei_me e1000e
cryptd serio_raw drm lpc_ich firewire_core mei i2c_i801 snd_timer
crc_itu_t ptp pp
Dec 12 10:02:58 desktop kernel: CPU: 3 PID: 1034 Comm: vhost-1020
Tainted: P           OE   4.9.0-gentoo_kdbus-bfq-300Hz #1
Dec 12 10:02:58 desktop kernel: Hardware name: System manufacturer
System Product Name/P9X79 WS, BIOS 4601 03/05/2014
Dec 12 10:02:58 desktop kernel:  0000000000000000 ffffffff812b06cd
0000000000000000 0000000000000000
Dec 12 10:02:58 desktop kernel:  ffffffff8105134a ffff880427672400
ffff8804216a8780 ffff880428341708
Dec 12 10:02:58 desktop kernel:  ffff8803c79c6f00 ffff88043fcc3ab7
0000000000000000 ffffffff810c5ab5
Dec 12 10:02:58 desktop kernel: Call Trace:
Dec 12 10:02:58 desktop kernel:  <IRQ>
Dec 12 10:02:58 desktop kernel:  [<ffffffff812b06cd>] ? dump_stack+0x46/0x59
Dec 12 10:02:58 desktop kernel:  [<ffffffff8105134a>] ? __warn+0xba/0xe0
Dec 12 10:02:58 desktop kernel:  [<ffffffff810c5ab5>] ? cgroup_get+0x55/0x70
Dec 12 10:02:58 desktop kernel:  [<ffffffff814ff566>] ?
sk_clone_lock+0x2d6/0x380
Dec 12 10:02:58 desktop kernel:  [<ffffffff8155612c>] ?
inet_csk_clone_lock+0xc/0xb0
Dec 12 10:02:58 desktop kernel:  [<ffffffff8156fb79>] ?
tcp_create_openreq_child+0x19/0x490
Dec 12 10:02:58 desktop kernel:  [<ffffffff8156ddee>] ?
tcp_v4_syn_recv_sock+0x3e/0x2e0
Dec 12 10:02:58 desktop kernel:  [<ffffffff81570384>] ?
tcp_check_req+0x394/0x4e0
Dec 12 10:02:58 desktop kernel:  [<ffffffff8156ca2b>] ?
tcp_v4_inbound_md5_hash+0x5b/0x1a0
Dec 12 10:02:58 desktop kernel:  [<ffffffff8156f19b>] ? tcp_v4_rcv+0x49b/0xc40
Dec 12 10:02:58 desktop kernel:  [<ffffffffa016d58a>] ?
nf_nat_ipv4_fn+0x5a/0x1d0 [nf_nat_ipv4]
Dec 12 10:02:58 desktop kernel:  [<ffffffffa015e2ef>] ?
ipv4_confirm+0x6f/0xe0 [nf_conntrack_ipv4]
Dec 12 10:02:58 desktop kernel:  [<ffffffff8154ac1f>] ?
ip_local_deliver_finish+0x8f/0x1e0
Dec 12 10:02:58 desktop kernel:  [<ffffffff8154b273>] ?
ip_local_deliver+0x43/0x100
Dec 12 10:02:58 desktop kernel:  [<ffffffff8154ab90>] ?
inet_del_offload+0x30/0x30
Dec 12 10:02:58 desktop kernel:  [<ffffffff8154b5b2>] ? ip_rcv+0x282/0x3a0
Dec 12 10:02:58 desktop kernel:  [<ffffffff8154ad70>] ?
ip_local_deliver_finish+0x1e0/0x1e0
Dec 12 10:02:58 desktop kernel:  [<ffffffff81512333>] ?
__netif_receive_skb_core+0x2a3/0x9f0
Dec 12 10:02:58 desktop kernel:  [<ffffffff8151478f>] ?
netif_receive_skb_internal+0x1f/0x90
Dec 12 10:02:58 desktop kernel:  [<ffffffffa0116ec8>] ?
br_pass_frame_up+0xa8/0x150 [bridge]
Dec 12 10:02:58 desktop kernel:  [<ffffffffa0116da0>] ?
br_port_flags_change+0x10/0x10 [bridge]
Dec 12 10:02:58 desktop kernel:  [<ffffffffa0117131>] ?
br_handle_frame_finish+0x1c1/0x4f0 [bridge]
Dec 12 10:02:58 desktop kernel:  [<ffffffffa0117601>] ?
br_handle_frame+0x161/0x2f0 [bridge]
Dec 12 10:02:58 desktop kernel:  [<ffffffff815123ba>] ?
__netif_receive_skb_core+0x32a/0x9f0
Dec 12 10:02:58 desktop kernel:  [<ffffffff81078a94>] ?
cpu_load_update+0xb4/0xc0
Dec 12 10:02:58 desktop kernel:  [<ffffffff810a710f>] ?
run_timer_softirq.part.23+0x8f/0xa0
Dec 12 10:02:58 desktop kernel:  [<ffffffff81515e97>] ?
process_backlog+0x97/0x130
Dec 12 10:02:58 desktop kernel:  [<ffffffff81515cf7>] ?
net_rx_action+0x1a7/0x2b0
Dec 12 10:02:58 desktop kernel:  [<ffffffff81055963>] ? __do_softirq+0xd3/0x1c0
Dec 12 10:02:58 desktop kernel:  [<ffffffff816060bc>] ?
do_softirq_own_stack+0x1c/0x30
Dec 12 10:02:58 desktop kernel:  <EOI>
Dec 12 10:02:58 desktop kernel:  [<ffffffff8105562a>] ?
do_softirq.part.16+0x2a/0x30
Dec 12 10:02:58 desktop kernel:  [<ffffffff81511fbc>] ? netif_rx_ni+0x3c/0x50
Dec 12 10:02:58 desktop kernel:  [<ffffffff81444342>] ? tun_get_user+0x3f2/0x8e0
Dec 12 10:02:58 desktop kernel:  [<ffffffffa01b6426>] ?
translate_desc+0x76/0x150 [vhost]
Dec 12 10:02:58 desktop kernel:  [<ffffffffa01b70a7>] ?
vhost_get_vq_desc+0x2d7/0x630 [vhost]
Dec 12 10:02:58 desktop kernel:  [<ffffffff8144486c>] ? tun_sendmsg+0x3c/0x60
Dec 12 10:02:58 desktop kernel:  [<ffffffffa01c0777>] ?
handle_tx+0x207/0x490 [vhost_net]
Dec 12 10:02:58 desktop kernel:  [<ffffffffa01b44b9>] ?
vhost_worker+0x99/0xf0 [vhost]
Dec 12 10:02:58 desktop kernel:  [<ffffffffa01b4420>] ?
vhost_dev_init+0x260/0x260 [vhost]
Dec 12 10:02:58 desktop kernel:  [<ffffffff81002536>] ? do_syscall_64+0x56/0x160
Dec 12 10:02:58 desktop kernel:  [<ffffffff810550bb>] ? SyS_exit_group+0xb/0x10
Dec 12 10:02:58 desktop kernel:  [<ffffffff8106b1c9>] ? kthread+0xb9/0xd0
Dec 12 10:02:58 desktop kernel:  [<ffffffff8106b110>] ? kthread_park+0x50/0x50
Dec 12 10:02:58 desktop kernel:  [<ffffffff81604ea2>] ? ret_from_fork+0x22/0x30
Dec 12 10:02:58 desktop kernel: ---[ end trace d8a0c3bc226461ef ]---


Thanks for your time, your work and your attention.

Kind regards,
Jesús Rodríguez Acosta

--
http://es.linkedin.com/in/jesusrodriguezacosta
(+34) 653 79 09 64

IMPORTANTE: Este mensaje de correo está destinado únicamente a la
persona o entidad a la cual se dirige y puede contener información
privilegiada, confidencial y/o de acceso restringido según la
legislación vigente. Si usted no es el destinatario indicado, por la
presente queda informado de que cualquier uso, distribución o copia de
la comunicación está estrictamente prohibido. Si usted ha recibido
esta comunicación por error, por favor, avíseme de inmediato. Gracias.

IMPORTANT: This email message is intended only for the use of the
individual to whom, or entity to which, it is addressed and may
contain information that is privileged, confidential and/or exempt
from disclosure under applicable law. If you are NOT the intended
recipient, you are hereby notified that any use, dissemination,
distribution or copying of the communication is strictly prohibited.
If you have received this communication in error, please notify me
immediately. Thank you.

[PATCH cgroup/for-4.10] cgroup: cgroup_sk_alloc() is allowed to increase reference on a dead cgroup

[Tejun Heo]

Jesús,

I applied the following patch to cgroup/for-4.10 and will soon push
out to Linus.  I'm pretty sure this is what you saw but can you please
verify that the warning goes away with it applied?

Thanks a lot for the report.

------ 8< ------
From 89bb3eed858ec2ea36ceae822bbf81f1ef24e80d Mon Sep 17 00:00:00 2001
From: Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
Date: Mon, 12 Dec 2016 16:30:29 -0500
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

When a listening socket associated with a cgroup creates a new
connection, it gets cloned and the reference on the cgroup is
incremented by calling cgroup_get().  Such a socket can be left behind
in an empty cgroup after processes are migrated away from it.  As an
empty cgroup may be removed, the socket may end up trying to clone new
sockets while associated with a removed cgroup.

This doesn't break anything in terms of reference counting; however,
as a precaution, cgroup_get() triggers WARN when it's called on a
removed cgroup, which cgroup_sk_alloc() can trigger spuriously.

We know this path is legit, open code cgroup_get() in
cgroup_sk_alloc() without the sanity check to avoid spuriously
triggering the warning.

Reported-by: Jesús Rodríguez Acosta <jesus.rodriguez.acosta-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Signed-off-by: Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
Link: http://lkml.kernel.org/r/CAH=7=un56LL12Meu0j+6+ZRcdsnUHomfDLuq=raC6ZtQEwLqTA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Fixes: d979a39d7242 ("cgroup: duplicate cgroup reference when cloning sockets")
Cc: stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org # v4.5+
Cc: Johannes Weiner <jweiner-b10kYP2dOMg@public.gmane.org>
---
 kernel/cgroup.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/kernel/cgroup.c b/kernel/cgroup.c
index 85bc9be..7fe26595 100644
--- a/kernel/cgroup.c
+++ b/kernel/cgroup.c
@@ -6318,9 +6318,13 @@ void cgroup_sk_alloc(struct sock_cgroup_data *skcd)
 	if (cgroup_sk_alloc_disabled)
 		return;
 
-	/* Socket clone path */
+	/*
+	 * Socket clone path.  We open-code cgroup_get() here to avoid the
+	 * cgroup_is_dead() sanity check as a listening socket left in a
+	 * dead cgroup can still get cloned for new connections.
+	 */
 	if (skcd->val) {
-		cgroup_get(sock_cgroup_ptr(skcd));
+		css_get(&sock_cgroup_ptr(skcd)->self);
 		return;
 	}
 
-- 
2.9.3

Re: [PATCH cgroup/for-4.10] cgroup: cgroup_sk_alloc() is allowed to increase reference on a dead cgroup

[Jesús Rodríguez Acosta]

Hi Tejun, hi all:

I applied the patch this morning and I'm happy to confirm you that the
warning went away.

Thank you!
Un saludo,
Jesús Rodríguez Acosta

--
http://es.linkedin.com/in/jesusrodriguezacosta
(+34) 653 79 09 64

IMPORTANTE: Este mensaje de correo está destinado únicamente a la
persona o entidad a la cual se dirige y puede contener información
privilegiada, confidencial y/o de acceso restringido según la
legislación vigente. Si usted no es el destinatario indicado, por la
presente queda informado de que cualquier uso, distribución o copia de
la comunicación está estrictamente prohibido. Si usted ha recibido
esta comunicación por error, por favor, avíseme de inmediato. Gracias.

IMPORTANT: This email message is intended only for the use of the
individual to whom, or entity to which, it is addressed and may
contain information that is privileged, confidential and/or exempt
from disclosure under applicable law. If you are NOT the intended
recipient, you are hereby notified that any use, dissemination,
distribution or copying of the communication is strictly prohibited.
If you have received this communication in error, please notify me
immediately. Thank you.


2016-12-12 21:48 GMT+00:00 Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>:
> Jesús,
>
> I applied the following patch to cgroup/for-4.10 and will soon push
> out to Linus.  I'm pretty sure this is what you saw but can you please
> verify that the warning goes away with it applied?
>
> Thanks a lot for the report.
>
> ------ 8< ------
> From 89bb3eed858ec2ea36ceae822bbf81f1ef24e80d Mon Sep 17 00:00:00 2001
> From: Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> Date: Mon, 12 Dec 2016 16:30:29 -0500
> MIME-Version: 1.0
> Content-Type: text/plain; charset=UTF-8
> Content-Transfer-Encoding: 8bit
>
> When a listening socket associated with a cgroup creates a new
> connection, it gets cloned and the reference on the cgroup is
> incremented by calling cgroup_get().  Such a socket can be left behind
> in an empty cgroup after processes are migrated away from it.  As an
> empty cgroup may be removed, the socket may end up trying to clone new
> sockets while associated with a removed cgroup.
>
> This doesn't break anything in terms of reference counting; however,
> as a precaution, cgroup_get() triggers WARN when it's called on a
> removed cgroup, which cgroup_sk_alloc() can trigger spuriously.
>
> We know this path is legit, open code cgroup_get() in
> cgroup_sk_alloc() without the sanity check to avoid spuriously
> triggering the warning.
>
> Reported-by: Jesús Rodríguez Acosta <jesus.rodriguez.acosta@gmail.com>
> Signed-off-by: Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> Link: http://lkml.kernel.org/r/CAH=7=un56LL12Meu0j+6+ZRcdsnUHomfDLuq=raC6ZtQEwLqTA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
> Fixes: d979a39d7242 ("cgroup: duplicate cgroup reference when cloning sockets")
> Cc: stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org # v4.5+
> Cc: Johannes Weiner <jweiner-b10kYP2dOMg@public.gmane.org>
> ---
>  kernel/cgroup.c | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/kernel/cgroup.c b/kernel/cgroup.c
> index 85bc9be..7fe26595 100644
> --- a/kernel/cgroup.c
> +++ b/kernel/cgroup.c
> @@ -6318,9 +6318,13 @@ void cgroup_sk_alloc(struct sock_cgroup_data *skcd)
>         if (cgroup_sk_alloc_disabled)
>                 return;
>
> -       /* Socket clone path */
> +       /*
> +        * Socket clone path.  We open-code cgroup_get() here to avoid the
> +        * cgroup_is_dead() sanity check as a listening socket left in a
> +        * dead cgroup can still get cloned for new connections.
> +        */
>         if (skcd->val) {
> -               cgroup_get(sock_cgroup_ptr(skcd));
> +               css_get(&sock_cgroup_ptr(skcd)->self);
>                 return;
>         }
>
> --
> 2.9.3
>