Re: BUG: KASAN: use-after-free in fib_table_flush

[Ido Schimmel <idosch@idosch.org>]

+Alexander

On Sun, Dec 17, 2017 at 08:55:57PM +0800, Fengguang Wu wrote:
> Hello,
> 
> FYI this happens in mainline kernel 4.15.0-rc3.
> It looks like a new regression.
> 
> It occurs in 4 out of 28 boots.
> 
> [  166.090516] ==================================================================
> [  166.092419] BUG: KASAN: use-after-free in fib_table_flush+0x76c/0x870:
> 						fib_table_flush at net/ipv4/fib_trie.c:1868
> [  166.092907] Read of size 8 at addr ffff880012fc0b18 by task kworker/u2:3/173
> [  166.093402]
> [  166.093528] CPU: 0 PID: 173 Comm: kworker/u2:3 Not tainted 4.15.0-rc3 #31
> [  166.094018] Workqueue: netns cleanup_net
> [  166.094298] Call Trace:
> [  166.094489]  print_address_description+0xa6/0x370:
> 						print_address_description at mm/kasan/report.c:253
> [  166.094867]  ? fib_table_flush+0x76c/0x870:
> 						fib_table_flush at net/ipv4/fib_trie.c:1868
> [  166.095159]  kasan_report+0x226/0x330:
> 						kasan_report_error at mm/kasan/report.c:352
> 						 (inlined by) kasan_report at mm/kasan/report.c:409
> [  166.095420]  fib_table_flush+0x76c/0x870:
> 						fib_table_flush at net/ipv4/fib_trie.c:1868
> [  166.095698]  ? fib_table_flush_external+0x5a0/0x5a0:
> 						fib_table_flush at net/ipv4/fib_trie.c:1836
> [  166.096067]  ? ip_fib_net_exit+0x94/0x360:
> 						ip_fib_net_exit at net/ipv4/fib_frontend.c:1313 (discriminator 16)
> [  166.096350]  ip_fib_net_exit+0x228/0x360:
> 						ip_fib_net_exit at net/ipv4/fib_frontend.c:1316
> [  166.096629]  ? ip_fib_net_exit+0x360/0x360:
> 						fib_net_exit at net/ipv4/fib_frontend.c:1355
> [  166.096930]  ops_exit_list+0xa8/0x160
> [  166.097233]  cleanup_net+0x414/0x860:
> 						cleanup_net at net/core/net_namespace.c:483 (discriminator 9)
> [  166.097487]  ? net_drop_ns+0x80/0x80:
> 						cleanup_net at net/core/net_namespace.c:439
> [  166.097748]  ? kvm_sched_clock_read+0x5/0x10:
> 						kvm_sched_clock_read at arch/x86/kernel/kvmclock.c:101
> [  166.098051]  ? native_sched_clock_from_tsc+0x40/0x70:
> 						__preempt_count_dec_and_test at arch/x86/include/asm/preempt.h:91
> 						 (inlined by) cyc2ns_read_end at arch/x86/kernel/tsc.c:81
> 						 (inlined by) cycles_2_ns at arch/x86/kernel/tsc.c:135
> 						 (inlined by) native_sched_clock_from_tsc at arch/x86/kernel/tsc.c:219
> [  166.098399]  ? sched_clock_cpu+0xf/0x70:
> 						sched_clock_cpu at kernel/sched/clock.c:363
> [  166.098672]  ? __lock_acquire+0x3b2/0x1fc0
> [  166.099054]  ? lock_downgrade+0x6a0/0x6a0:
> 						lock_release at kernel/locking/lockdep.c:4013
> [  166.099337]  ? lock_acquire+0x117/0x260:
> 						get_current at arch/x86/include/asm/current.h:15
> 						 (inlined by) lock_acquire at kernel/locking/lockdep.c:4006
> [  166.099609]  ? process_one_work+0x70f/0x11c0:
> 						process_one_work at kernel/workqueue.c:2087
> [  166.099938]  process_one_work+0x791/0x11c0:
> 						process_one_work at kernel/workqueue.c:2118
> [  166.100229]  ? kvm_sched_clock_read+0x5/0x10:
> 						kvm_sched_clock_read at arch/x86/kernel/kvmclock.c:101
> [  166.100532]  ? sched_clock+0x2d/0x40:
> 						paravirt_sched_clock at arch/x86/include/asm/paravirt.h:174
> 						 (inlined by) sched_clock at arch/x86/kernel/tsc.c:227
> [  166.100792]  ? cancel_delayed_work_sync+0x20/0x20:
> 						process_one_work at kernel/workqueue.c:2014
> [  166.101123]  worker_thread+0xe8/0x1070:
> 						__read_once_size at include/linux/compiler.h:183
> 						 (inlined by) list_empty at include/linux/list.h:203
> 						 (inlined by) worker_thread at kernel/workqueue.c:2247
> [  166.101392]  ? __kthread_parkme+0x164/0x230:
> 						__kthread_parkme at kernel/kthread.c:188
> [  166.101689]  ? process_one_work+0x11c0/0x11c0:
> 						worker_thread at kernel/workqueue.c:2189
> [  166.102006]  kthread+0x2fd/0x400:
> 						kthread at kernel/kthread.c:238
> [  166.102240]  ? kthread_create_on_node+0xf0/0xf0:
> 						kthread at kernel/kthread.c:198
> [  166.102561]  ret_from_fork+0x1f/0x30:
> 						ret_from_fork at arch/x86/entry/entry_64.S:447
> [  166.102855]
> [  166.102972] Allocated by task 1907:
> [  166.103235]  __kmalloc+0xf6/0x1a0:
> 						__kmalloc at mm/slub.c:3765
> [  166.103475]  fib_trie_table+0xe8/0x240:
> 						fib_trie_table at net/ipv4/fib_trie.c:2081
> [  166.103748]  fib_net_init+0x1bc/0x570:
> 						fib4_rules_init at net/ipv4/fib_frontend.c:59
> 						 (inlined by) ip_fib_net_init at net/ipv4/fib_frontend.c:1287
> 						 (inlined by) fib_net_init at net/ipv4/fib_frontend.c:1335
> [  166.104032]  ops_init+0x1c0/0x360:
> 						ops_init at net/core/net_namespace.c:119
> [  166.104269]  setup_net+0x23c/0x530:
> 						setup_net at net/core/net_namespace.c:296
> [  166.104512]  copy_net_ns+0x170/0x350:
> 						copy_net_ns at net/core/net_namespace.c:420
> [  166.104779]  create_new_namespaces+0x343/0x730:
> 						create_new_namespaces at kernel/nsproxy.c:107
> [  166.105091]  unshare_nsproxy_namespaces+0xa1/0x150:
> 						unshare_nsproxy_namespaces at kernel/nsproxy.c:206 (discriminator 4)
> [  166.105427]  SyS_unshare+0x338/0x6c0
> [  166.105682]  do_syscall_64+0x21f/0xb80:
> 						do_syscall_64 at arch/x86/entry/common.c:285
> [  166.105954]  return_from_SYSCALL_64+0x0/0x65:
> 						return_from_SYSCALL_64 at arch/x86/entry/entry_64.S:259
> [  166.106253]
> [  166.106367] Freed by task 11:
> [  166.106581]  kfree+0x102/0x1d0:
> 						slab_free at mm/slub.c:2973
> 						 (inlined by) kfree at mm/slub.c:3899
> [  166.106838]  rcu_do_batch+0x331/0x7f0:
> 						rcu_lock_release at include/linux/rcupdate.h:249
> 						 (inlined by) __rcu_reclaim at kernel/rcu/rcu.h:196
> 						 (inlined by) rcu_do_batch at kernel/rcu/tree.c:2758
> [  166.107102]  rcu_cpu_kthread+0x12a/0x160:
> 						rcu_preempt_do_callbacks at kernel/rcu/tree_plugin.h:687
> 						 (inlined by) rcu_kthread_do_work at kernel/rcu/tree_plugin.h:1142
> 						 (inlined by) rcu_cpu_kthread at kernel/rcu/tree_plugin.h:1184
> [  166.107381]  smpboot_thread_fn+0x3c1/0x820:
> 						smpboot_thread_fn at kernel/smpboot.c:164
> [  166.107669]  kthread+0x2fd/0x400:
> 						kthread at kernel/kthread.c:238
> [  166.107928]  ret_from_fork+0x1f/0x30:
> 						ret_from_fork at arch/x86/entry/entry_64.S:447
> [  166.108181]
> [  166.108295] The buggy address belongs to the object at ffff880012fc0ae0
> [  166.108295]  which belongs to the cache kmalloc-64 of size 64
> [  166.109179] The buggy address is located 56 bytes inside of
> [  166.109179]  64-byte region [ffff880012fc0ae0, ffff880012fc0b20)

Hi Alexander,

Note that CONFIG_IP_MULTIPLE_TABLES is disabled, so both the main and
local table are allocated during init and also share the same trie.

I think that what happens is that ip_fib_net_exit() frees the main table
and its trie via an RCU callback which is scheduled before the local
table is iterated over, thus resulting in a use-after-free.

I can reliably trigger the bug by adding synchronize_rcu() at the end of
each iteration of the loop.

Problem goes away if we iterate over the tables in reverse order which
is symmetric to fib4_rules_init().

What do you think?