From: Ido Schimmel <idosch@idosch.org>
To: Mirsad Goran Todorovac <mirsad.todorovac@alu.unizg.hr>
Cc: netdev@vger.kernel.org, "David S. Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@google.com>,
Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>,
Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Jesper Dangaard Brouer <hawk@kernel.org>,
John Fastabend <john.fastabend@gmail.com>,
Nikolay Aleksandrov <razor@blackwall.org>,
Florent Fourcot <florent.fourcot@wifirst.fr>,
Hangbin Liu <liuhangbin@gmail.com>,
Petr Machata <petrm@nvidia.com>, Jiri Pirko <jiri@resnulli.us>,
Xin Long <lucien.xin@gmail.com>,
linux-kernel@vger.kernel.org, bpf@vger.kernel.org
Subject: Re: [BUG] kmemleak in rtnetlink_rcv() triggered by selftests/drivers/net/team in build cdc9718d5e59
Date: Sun, 9 Apr 2023 20:14:13 +0300 [thread overview]
Message-ID: <ZDLyZX545Cw+aLhE@shredder> (raw)
In-Reply-To: <78a8a03b-6070-3e6b-5042-f848dab16fb8@alu.unizg.hr>
On Sun, Apr 09, 2023 at 01:49:30PM +0200, Mirsad Goran Todorovac wrote:
> Hi all,
>
> There appears to be a memleak triggered by the selftest drivers/net/team.
Thanks for the report. Not sure it's related to team, see below.
>
> # cat /sys/kernel/debug/kmemleak
> unreferenced object 0xffff8c18def8ee00 (size 256):
> comm "ip", pid 5727, jiffies 4294961159 (age 954.244s)
> hex dump (first 32 bytes):
> 00 20 09 de 18 8c ff ff 00 00 00 00 00 00 00 00 . ..............
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> backtrace:
> [<ffffffffb60fb25c>] slab_post_alloc_hook+0x8c/0x3e0
> [<ffffffffb6102b39>] __kmem_cache_alloc_node+0x1d9/0x2a0
> [<ffffffffb607684e>] kmalloc_trace+0x2e/0xc0
> [<ffffffffb6dbc00b>] vlan_vid_add+0x11b/0x290
> [<ffffffffb6dbcffc>] vlan_device_event+0x19c/0x880
> [<ffffffffb5dde4d7>] raw_notifier_call_chain+0x47/0x70
> [<ffffffffb6ab6940>] call_netdevice_notifiers_info+0x50/0xa0
> [<ffffffffb6ac7574>] dev_open+0x94/0xa0
> [<ffffffffc176515e>] 0xffffffffc176515e
Don't know what this is. Might be another issue.
> [<ffffffffb6ada6b0>] do_set_master+0x90/0xb0
> [<ffffffffb6adc5f4>] do_setlink+0x514/0x11f0
> [<ffffffffb6ae4507>] __rtnl_newlink+0x4e7/0xa10
> [<ffffffffb6ae4a8c>] rtnl_newlink+0x4c/0x70
> [<ffffffffb6adf334>] rtnetlink_rcv_msg+0x184/0x5d0
> [<ffffffffb6b6ad1e>] netlink_rcv_skb+0x5e/0x110
> [<ffffffffb6ada0e9>] rtnetlink_rcv+0x19/0x20
> unreferenced object 0xffff8c18250d3700 (size 32):
> comm "ip", pid 5727, jiffies 4294961159 (age 954.244s)
> hex dump (first 32 bytes):
> a0 ee f8 de 18 8c ff ff a0 ee f8 de 18 8c ff ff ................
> 81 00 00 00 01 00 00 00 cc cc cc cc cc cc cc cc ................
> backtrace:
> [<ffffffffb60fb25c>] slab_post_alloc_hook+0x8c/0x3e0
> [<ffffffffb6102b39>] __kmem_cache_alloc_node+0x1d9/0x2a0
> [<ffffffffb607684e>] kmalloc_trace+0x2e/0xc0
> [<ffffffffb6dbc064>] vlan_vid_add+0x174/0x290
> [<ffffffffb6dbcffc>] vlan_device_event+0x19c/0x880
> [<ffffffffb5dde4d7>] raw_notifier_call_chain+0x47/0x70
> [<ffffffffb6ab6940>] call_netdevice_notifiers_info+0x50/0xa0
> [<ffffffffb6ac7574>] dev_open+0x94/0xa0
> [<ffffffffc176515e>] 0xffffffffc176515e
> [<ffffffffb6ada6b0>] do_set_master+0x90/0xb0
> [<ffffffffb6adc5f4>] do_setlink+0x514/0x11f0
> [<ffffffffb6ae4507>] __rtnl_newlink+0x4e7/0xa10
> [<ffffffffb6ae4a8c>] rtnl_newlink+0x4c/0x70
> [<ffffffffb6adf334>] rtnetlink_rcv_msg+0x184/0x5d0
> [<ffffffffb6b6ad1e>] netlink_rcv_skb+0x5e/0x110
> [<ffffffffb6ada0e9>] rtnetlink_rcv+0x19/0x20
> unreferenced object 0xffff8c1846e16800 (size 256):
> comm "ip", pid 7837, jiffies 4295135225 (age 258.160s)
> hex dump (first 32 bytes):
> 00 20 f7 de 18 8c ff ff 00 00 00 00 00 00 00 00 . ..............
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> backtrace:
> [<ffffffffb60fb25c>] slab_post_alloc_hook+0x8c/0x3e0
> [<ffffffffb6102b39>] __kmem_cache_alloc_node+0x1d9/0x2a0
> [<ffffffffb607684e>] kmalloc_trace+0x2e/0xc0
> [<ffffffffb6dbc00b>] vlan_vid_add+0x11b/0x290
> [<ffffffffb6dbcffc>] vlan_device_event+0x19c/0x880
> [<ffffffffb5dde4d7>] raw_notifier_call_chain+0x47/0x70
> [<ffffffffb6ab6940>] call_netdevice_notifiers_info+0x50/0xa0
> [<ffffffffb6ac7574>] dev_open+0x94/0xa0
> [<ffffffffc177115e>] bond_enslave+0x34e/0x1840 [bonding]
This shows that the issue is related to the bond driver, not team.
> [<ffffffffb6ada6b0>] do_set_master+0x90/0xb0
> [<ffffffffb6adc5f4>] do_setlink+0x514/0x11f0
> [<ffffffffb6ae4507>] __rtnl_newlink+0x4e7/0xa10
> [<ffffffffb6ae4a8c>] rtnl_newlink+0x4c/0x70
> [<ffffffffb6adf334>] rtnetlink_rcv_msg+0x184/0x5d0
> [<ffffffffb6b6ad1e>] netlink_rcv_skb+0x5e/0x110
> [<ffffffffb6ada0e9>] rtnetlink_rcv+0x19/0x20
> unreferenced object 0xffff8c184c5ff2a0 (size 32):
This is 'struct vlan_vid_info'
> comm "ip", pid 7837, jiffies 4295135225 (age 258.160s)
> hex dump (first 32 bytes):
> a0 68 e1 46 18 8c ff ff a0 68 e1 46 18 8c ff ff .h.F.....h.F....
> 81 00 00 00 01 00 00 00 cc cc cc cc cc cc cc cc ................
^ VLAN ID 0
> backtrace:
> [<ffffffffb60fb25c>] slab_post_alloc_hook+0x8c/0x3e0
> [<ffffffffb6102b39>] __kmem_cache_alloc_node+0x1d9/0x2a0
> [<ffffffffb607684e>] kmalloc_trace+0x2e/0xc0
> [<ffffffffb6dbc064>] vlan_vid_add+0x174/0x290
> [<ffffffffb6dbcffc>] vlan_device_event+0x19c/0x880
> [<ffffffffb5dde4d7>] raw_notifier_call_chain+0x47/0x70
> [<ffffffffb6ab6940>] call_netdevice_notifiers_info+0x50/0xa0
> [<ffffffffb6ac7574>] dev_open+0x94/0xa0
> [<ffffffffc177115e>] bond_enslave+0x34e/0x1840 [bonding]
> [<ffffffffb6ada6b0>] do_set_master+0x90/0xb0
> [<ffffffffb6adc5f4>] do_setlink+0x514/0x11f0
> [<ffffffffb6ae4507>] __rtnl_newlink+0x4e7/0xa10
> [<ffffffffb6ae4a8c>] rtnl_newlink+0x4c/0x70
> [<ffffffffb6adf334>] rtnetlink_rcv_msg+0x184/0x5d0
> [<ffffffffb6b6ad1e>] netlink_rcv_skb+0x5e/0x110
> [<ffffffffb6ada0e9>] rtnetlink_rcv+0x19/0x20
VLAN ID 0 is automatically added by the 8021q driver when a net device
is opened. In this case it's a device being enslaved to a bond. I
believe the issue was exposed by the new bond test that was added in
commit 222c94ec0ad4 ("selftests: bonding: add tests for ether type
changes") as part of v6.3-rc3.
The VLAN is supposed to be removed by the 8021q driver when a net device
is closed and the bond driver indeed calls dev_close() when a slave is
removed. However, this function is a NOP when 'IFF_UP' is not set.
Unfortunately, when a bond changes its type to Ethernet this flag is
incorrectly cleared in bond_ether_setup(), causing this VLAN to linger.
As far as I can tell, it's not a new issue.
Temporary fix is [1]. Please test it although we might end up with a
different fix (needs more thinking and it's already late here).
Reproduced using [2]. You can see in the before/after output how the
flag is cleared/retained [3].
[1]
diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
index 236e5219c811..50dc068dc259 100644
--- a/drivers/net/bonding/bond_main.c
+++ b/drivers/net/bonding/bond_main.c
@@ -1777,14 +1777,15 @@ void bond_lower_state_changed(struct slave *slave)
/* The bonding driver uses ether_setup() to convert a master bond device
* to ARPHRD_ETHER, that resets the target netdevice's flags so we always
- * have to restore the IFF_MASTER flag, and only restore IFF_SLAVE if it was set
+ * have to restore the IFF_MASTER flag, and only restore IFF_SLAVE and IFF_UP
+ * if they were set
*/
static void bond_ether_setup(struct net_device *bond_dev)
{
- unsigned int slave_flag = bond_dev->flags & IFF_SLAVE;
+ unsigned int flags = bond_dev->flags & (IFF_SLAVE | IFF_UP);
ether_setup(bond_dev);
- bond_dev->flags |= IFF_MASTER | slave_flag;
+ bond_dev->flags |= IFF_MASTER | flags;
bond_dev->priv_flags &= ~IFF_TX_SKB_SHARING;
}
[2]
#!/bin/bash
ip link add name t-nlmon type nlmon
ip link add name t-dummy type dummy
ip link add name t-bond type bond mode active-backup
ip link set dev t-bond up
ip link set dev t-nlmon master t-bond
ip link set dev t-nlmon nomaster
ip link show dev t-bond
ip link set dev t-dummy master t-bond
ip link show dev t-bond
ip link del dev t-bond
ip link del dev t-dummy
ip link del dev t-nlmon
[3]
Before:
12: t-bond: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
link/netlink
12: t-bond: <BROADCAST,MULTICAST,MASTER,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether ce:b2:31:0a:53:83 brd ff:ff:ff:ff:ff:ff
After:
12: t-bond: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
link/netlink
12: t-bond: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 5a:18:e7:85:11:73 brd ff:ff:ff:ff:ff:ff
next prev parent reply other threads:[~2023-04-09 17:14 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-04-09 11:49 [BUG] kmemleak in rtnetlink_rcv() triggered by selftests/drivers/net/team in build cdc9718d5e59 Mirsad Goran Todorovac
2023-04-09 17:14 ` Ido Schimmel [this message]
2023-04-09 18:47 ` [BUG] [FIXED: TESTED] " Mirsad Goran Todorovac
2023-04-10 17:34 ` Mirsad Goran Todorovac
2023-04-10 17:49 ` [BUG] [TESTING: HANG] " Mirsad Goran Todorovac
2023-04-13 18:19 ` [BUG] [FIXED: TESTED] " Ido Schimmel
2023-04-21 11:35 ` Mirsad Goran Todorovac
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ZDLyZX545Cw+aLhE@shredder \
--to=idosch@idosch.org \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=florent.fourcot@wifirst.fr \
--cc=hawk@kernel.org \
--cc=jiri@resnulli.us \
--cc=john.fastabend@gmail.com \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=liuhangbin@gmail.com \
--cc=lucien.xin@gmail.com \
--cc=mirsad.todorovac@alu.unizg.hr \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=petrm@nvidia.com \
--cc=razor@blackwall.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.