RFC: crash in fib6_clean_all() while loading ipv6 module

RFC: crash in fib6_clean_all() while loading ipv6 module

[Michal Kubecek]

Hello,

Two of our customers encountered a crash in fib6_clean_all() when
booting their system:

[   12.408421] BUG: unable to handle kernel NULL pointer dereference at        (null)
[   12.408424] IP: [<ffffffffa02566b4>] fib6_clean_all+0x34/0xd0 [ipv6_lib]
[   12.408434] PGD c3590f067 PUD c29073067 PMD 0 
[   12.408436] Oops: 0000 [#1] SMP 
[   12.408439] CPU 1 
[   12.408440] Modules linked in: processor(+) ipv6(+) thermal_sys ipv6_lib
ahci(+) ixgbe(+) ehci_hcd libahci hwmon igb(+) libata usbcore i2c_i801 dca
iTCO_wdt pcspkr scsi_mod e1000e i2c_core usb_common iTCO_vendor_support
rtc_cmos container ptp pps_core button mdio
[   12.408449] Supported: Yes
[   12.408449] 
[   12.408451] Pid: 211, comm: modprobe Not tainted 3.0.76-0.11-default #1 ...
[   12.408453] RIP: 0010:[<ffffffffa02566b4>]  [<ffffffffa02566b4>] fib6_clean_all+0x34/0xd0 [ipv6_lib]
[   12.408460] RSP: 0018:ffff880c1e983b98  EFLAGS: 00010246
[   12.408462] RAX: 0000000000000000 RBX: ffffffffffffffff RCX: 0000000000000000
[   12.408463] RDX: 0000000000000000 RSI: ffffffffa0254f90 RDI: ffffffff81a72280
[   12.408464] RBP: ffffffff81a72280 R08: 0000000000000000 R09: 000000003520aec8
[   12.408466] R10: 000000005f000c28 R11: 000000003520aec8 R12: 0000000000000000
[   12.408467] R13: ffffffff81a72280 R14: 0000000000000000 R15: ffffffffa0254f90
[   12.408469] FS:  00007fb2039f1700(0000) GS:ffff880c7f040000(0000) knlGS:0000000000000000
[   12.408470] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[   12.408471] CR2: 0000000000000000 CR3: 0000000c2904e000 CR4: 00000000001407e0
[   12.408473] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   12.408474] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   12.408476] Process modprobe (pid: 211, threadinfo ffff880c1e982000, task ffff880c3242c540)
[   12.408477] Stack:
[   12.408478]  0000000000000000 0000000000000000 0000000000000000 ffffffff8146544e
[   12.408480]  000000000006a9d0 0000000000000000 000000003520aec8 000000005f000c28
[   12.408483]  000000003520aec8 0000000000000000 0000000000000000 0000000000000000
[   12.408485] Call Trace:
[   12.408507]  [<ffffffffa02567a3>] fib6_run_gc+0x53/0xf0 [ipv6_lib]
[   12.408524]  [<ffffffffa025c5c6>] ndisc_netdev_event+0x186/0x190 [ipv6_lib]
[   12.408541]  [<ffffffff81460437>] notifier_call_chain+0x37/0x70
[   12.408547]  [<ffffffff813a6e52>] dev_addr_add+0x52/0x90
[   12.408556]  [<ffffffffa032e3e9>] ixgbe_probe+0x8c9/0xd10 [ixgbe]
[   12.408565]  [<ffffffff8127d5d4>] local_pci_probe+0x54/0xe0
[   12.408570]  [<ffffffff8127d740>] __pci_device_probe+0xe0/0xf0
[   12.408573]  [<ffffffff8127e953>] pci_device_probe+0x33/0x60
[   12.408578]  [<ffffffff8132ddda>] really_probe+0x7a/0x260
[   12.408581]  [<ffffffff8132e023>] driver_probe_device+0x63/0xc0
[   12.408585]  [<ffffffff8132e113>] __driver_attach+0x93/0xa0
[   12.408588]  [<ffffffff8132d468>] bus_for_each_dev+0x88/0xb0
[   12.408591]  [<ffffffff8132cbb5>] bus_add_driver+0x155/0x2a0
[   12.408595]  [<ffffffff8132e769>] driver_register+0x79/0x170
[   12.408599]  [<ffffffff8127ebe8>] __pci_register_driver+0x58/0xe0
[   12.408604]  [<ffffffff810001cb>] do_one_initcall+0x3b/0x180
[   12.408610]  [<ffffffff810a02df>] sys_init_module+0xcf/0x240
[   12.408615]  [<ffffffff81464592>] system_call_fastpath+0x16/0x1b

The crash is caused by fib6_clean_all() dereferencing a null pointer in
init_net.ipv6.fib_table_hash because ndisc_netdev_event() handler which
calls fib6_run_gc() is registered by ndisc_init() but the relevant data
structures aren't initialized until ip6_route_init() is called. If a
device intialization falls into this window, it emits NETDEV_CHANGEADDR,
leading to a crash.

This could be prevented by setting a flag when ip6_route_init() is
complete and not calling fib6_run_gc() from ndisc_netdev_event() until
the flag is set. However, I don't like the idea of adding a test which
will be useful only in a short window while loading ipv6 module.

The only other actions ndisc_netdev_event() does are flushing the neigh
table (which should be empty at the moment) and calling
ndisc_send_unsol_na() (which should do nothing as there is no IPv6
address assigned yet). Thus I believe taking the netdev event handler
registration call out of ndisc_init() into a separate function, say
ndisc_late_init(), which would be called after ip6_route_init() should
be a safe and efficient way to avoid the race condition.

Before I submit the patch, I wanted to ask if someone can see a problem
with this solution (or a better solution).

                                                         Michal Kubecek

Re: RFC: crash in fib6_clean_all() while loading ipv6 module

[David Miller]

From: Michal Kubecek <mkubecek@suse.cz>
Date: Mon, 9 Sep 2013 12:05:15 +0200

> This could be prevented by setting a flag when ip6_route_init() is
> complete and not calling fib6_run_gc() from ndisc_netdev_event() until
> the flag is set. However, I don't like the idea of adding a test which
> will be useful only in a short window while loading ipv6 module.

Please just initialize the parts of ipv6 in the correct order necessary
to prevent this problem.

I stronly dislike using flags when it's simply an initialization
ordering problem.  It's just like registering a device interrupt
handle before the driver's data structures are properly setup.

[PATCH net] ipv6: don't call fib6_run_gc() until routing is ready

[Michal Kubecek]

When loading the ipv6 module, ndisc_init() is called before
ip6_route_init(). As the former registers a handler calling
fib6_run_gc(), this opens a window to run the garbage collector
before necessary data structures are initialized. If a network
device is initialized in this window, adding MAC address to it
triggers a NETDEV_CHANGEADDR event, leading to a crash in
fib6_clean_all().

Take the event handler registration out of ndisc_init() into a
separate function ndisc_late_init() and move it after
ip6_route_init().

Signed-off-by: Michal Kubecek <mkubecek@suse.cz>
---
 include/net/ndisc.h |  2 ++
 net/ipv6/af_inet6.c |  6 ++++++
 net/ipv6/ndisc.c    | 18 +++++++++++-------
 3 files changed, 19 insertions(+), 7 deletions(-)

diff --git a/include/net/ndisc.h b/include/net/ndisc.h
index 3c4211f..ea0cc26 100644
--- a/include/net/ndisc.h
+++ b/include/net/ndisc.h
@@ -190,7 +190,9 @@ static inline struct neighbour *__ipv6_neigh_lookup(struct net_device *dev, cons
 }
 
 extern int			ndisc_init(void);
+extern int			ndisc_late_init(void);
 
+extern void			ndisc_late_cleanup(void);
 extern void			ndisc_cleanup(void);
 
 extern int			ndisc_rcv(struct sk_buff *skb);
diff --git a/net/ipv6/af_inet6.c b/net/ipv6/af_inet6.c
index 136fe55..7c96100 100644
--- a/net/ipv6/af_inet6.c
+++ b/net/ipv6/af_inet6.c
@@ -915,6 +915,9 @@ static int __init inet6_init(void)
 	err = ip6_route_init();
 	if (err)
 		goto ip6_route_fail;
+	err = ndisc_late_init();
+	if (err)
+		goto ndisc_late_fail;
 	err = ip6_flowlabel_init();
 	if (err)
 		goto ip6_flowlabel_fail;
@@ -981,6 +984,8 @@ ipv6_exthdrs_fail:
 addrconf_fail:
 	ip6_flowlabel_cleanup();
 ip6_flowlabel_fail:
+	ndisc_late_cleanup();
+ndisc_late_fail:
 	ip6_route_cleanup();
 ip6_route_fail:
 #ifdef CONFIG_PROC_FS
@@ -1043,6 +1048,7 @@ static void __exit inet6_exit(void)
 	ipv6_exthdrs_exit();
 	addrconf_cleanup();
 	ip6_flowlabel_cleanup();
+	ndisc_late_cleanup();
 	ip6_route_cleanup();
 #ifdef CONFIG_PROC_FS
 
diff --git a/net/ipv6/ndisc.c b/net/ipv6/ndisc.c
index 1217945..f8a55ff 100644
--- a/net/ipv6/ndisc.c
+++ b/net/ipv6/ndisc.c
@@ -1727,24 +1727,28 @@ int __init ndisc_init(void)
 	if (err)
 		goto out_unregister_pernet;
 #endif
-	err = register_netdevice_notifier(&ndisc_netdev_notifier);
-	if (err)
-		goto out_unregister_sysctl;
 out:
 	return err;
 
-out_unregister_sysctl:
 #ifdef CONFIG_SYSCTL
-	neigh_sysctl_unregister(&nd_tbl.parms);
 out_unregister_pernet:
-#endif
 	unregister_pernet_subsys(&ndisc_net_ops);
 	goto out;
+#endif
 }
 
-void ndisc_cleanup(void)
+int __init ndisc_late_init(void)
+{
+	return register_netdevice_notifier(&ndisc_netdev_notifier);
+}
+
+void ndisc_late_cleanup(void)
 {
 	unregister_netdevice_notifier(&ndisc_netdev_notifier);
+}
+
+void ndisc_cleanup(void)
+{
 #ifdef CONFIG_SYSCTL
 	neigh_sysctl_unregister(&nd_tbl.parms);
 #endif
-- 
1.8.1.4

Re: [PATCH net] ipv6: don't call fib6_run_gc() until routing is ready

[David Miller]

From: Michal Kubecek <mkubecek@suse.cz>
Date: Mon,  9 Sep 2013 21:45:04 +0200 (CEST)

> When loading the ipv6 module, ndisc_init() is called before
> ip6_route_init(). As the former registers a handler calling
> fib6_run_gc(), this opens a window to run the garbage collector
> before necessary data structures are initialized. If a network
> device is initialized in this window, adding MAC address to it
> triggers a NETDEV_CHANGEADDR event, leading to a crash in
> fib6_clean_all().
> 
> Take the event handler registration out of ndisc_init() into a
> separate function ndisc_late_init() and move it after
> ip6_route_init().
> 
> Signed-off-by: Michal Kubecek <mkubecek@suse.cz>

Looks good, applied, thanks.