Greetings,

I got the below dmesg and the first bad commit is

commit 975022310233fb0f0193873d79a7b8438070fa82
Author:     Eric Dumazet <edumazet@google.com>
AuthorDate: Wed Dec 11 14:46:51 2013 -0800
Commit:     David S. Miller <davem@davemloft.net>
CommitDate: Wed Dec 11 20:21:10 2013 -0500

    udp: ipv4: must add synchronization in udp_sk_rx_dst_set()
    
    Unlike TCP, UDP input path does not hold the socket lock.
    
    Before messing with sk->sk_rx_dst, we must use a spinlock, otherwise
    multiple cpus could leak a refcount.
    
    This patch also takes care of renewing a stale dst entry.
    (When the sk->sk_rx_dst would not be used by IP early demux)
    
    Fixes: 421b3885bf6d ("udp: ipv4: Add udp early demux")
    Signed-off-by: Eric Dumazet <edumazet@google.com>
    Cc: Shawn Bohrer <sbohrer@rgmadvisors.com>
    Signed-off-by: David S. Miller <davem@davemloft.net>

+----------------------------------------------+----+
|                                              |    |
+----------------------------------------------+----+
| boot_successes                               | 0  |
| boot_failures                                | 19 |
| inconsistent_SOFTIRQ-ON-W-IN-SOFTIRQ-W_usage | 19 |
+----------------------------------------------+----+

[   23.564204] [ INFO: inconsistent lock state ]
[   23.564204] 3.13.0-rc3-00689-gb1daf37 #892 Not tainted
[   23.564204] ---------------------------------
[   23.564204] inconsistent {SOFTIRQ-ON-W} -> {IN-SOFTIRQ-W} usage.
[   23.564204] swapper/0/0 [HC0[0]:SC1[1]:HE1:SE0] takes:
[   23.564204]  (&(&sk->sk_dst_lock)->rlock){+.?...}, at: [<ffffffff81a27fe2>] __udp4_lib_rcv+0x569/0x79d
[   23.564204] {SOFTIRQ-ON-W} state was registered at:
[   23.564204]   [<ffffffff810b16b7>] __lock_acquire+0x96c/0x185f
[   23.564204]   [<ffffffff810b2c97>] lock_acquire+0xab/0x12e
[   23.564204]   [<ffffffff81af12d3>] _raw_spin_lock+0x3b/0x6d
[   23.564204]   [<ffffffff81a2f13d>] inet_bind+0x18d/0x1dc
[   23.564204]   [<ffffffff8199d62c>] kernel_bind+0x10/0x12
[   23.564204]   [<ffffffff81aae764>] xs_bind+0xa0/0x11e
[   23.564204]   [<ffffffff81ab0271>] xs_create_sock.isra.14+0x20b/0x230
[   23.564204]   [<ffffffff81ab047f>] xs_tcp_setup_socket+0x5f/0x3e5
[   23.564204]   [<ffffffff8108a0d7>] process_one_work+0x247/0x443
[   23.564204]   [<ffffffff8108aeac>] worker_thread+0x1d2/0x2bc
[   23.564204]   [<ffffffff81090228>] kthread+0xf9/0x101
[   23.564204]   [<ffffffff81af92fc>] ret_from_fork+0x7c/0xb0
[   23.564204] irq event stamp: 514170
[   23.564204] hardirqs last  enabled at (514170): [<ffffffff810766af>] local_bh_enable+0xab/0xb5
[   23.564204] hardirqs last disabled at (514169): [<ffffffff8107665a>] local_bh_enable+0x56/0xb5
[   23.564204] softirqs last  enabled at (514148): [<ffffffff81076106>] _local_bh_enable+0x43/0x45
[   23.564204] softirqs last disabled at (514149): [<ffffffff81076779>] irq_exit+0x59/0xa0
[   23.564204] 
[   23.564204] other info that might help us debug this:
[   23.564204]  Possible unsafe locking scenario:
[   23.564204] 
[   23.564204]        CPU0
[   23.564204]        ----
[   23.564204]   lock(&(&sk->sk_dst_lock)->rlock);
[   23.564204]   <Interrupt>
[   23.564204]     lock(&(&sk->sk_dst_lock)->rlock);
[   23.564204] 
[   23.564204]  *** DEADLOCK ***
[   23.564204] 
[   23.564204] 2 locks held by swapper/0/0:
[   23.564204]  #0:  (rcu_read_lock){.+.+..}, at: [<ffffffff819b258a>] __netif_receive_skb_core+0x168/0x58e
[   23.564204]  #1:  (rcu_read_lock){.+.+..}, at: [<ffffffff81a0086a>] ip_local_deliver_finish+0x2f/0x2a0
[   23.564204] 
[   23.564204] stack backtrace:
[   23.564204] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.13.0-rc3-00689-gb1daf37 #892
[   23.564204] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[   23.564204]  0000000000000002 ffff88001ea039a0 ffffffff81ae56fd ffffffff82097490
[   23.564204]  ffff88001ea039f0 ffffffff81addae7 0000000000000001 ffff880000000001
[   23.564204]  ffff880000000000 ffffffff82097c08 0000000000000004 0000000000000006
[   23.564204] Call Trace:
[   23.564204]  <IRQ>  [<ffffffff81ae56fd>] dump_stack+0x4d/0x66
[   23.564204]  [<ffffffff81addae7>] print_usage_bug+0x2a8/0x2b9
[   23.564204]  [<ffffffff810af93a>] ? print_shortest_lock_dependencies+0x193/0x193
[   23.564204]  [<ffffffff810b05b5>] mark_lock+0x33c/0x57c
[   23.564204]  [<ffffffff810b1645>] __lock_acquire+0x8fa/0x185f
[   23.564204]  [<ffffffff8109fd9c>] ? sched_clock_local+0x12/0x72
[   23.564204]  [<ffffffff810addfa>] ? trace_hardirqs_off_caller+0x1f/0xb2
[   23.564204]  [<ffffffff810b2c97>] lock_acquire+0xab/0x12e
[   23.564204]  [<ffffffff81a27fe2>] ? __udp4_lib_rcv+0x569/0x79d
[   23.564204]  [<ffffffff810addfa>] ? trace_hardirqs_off_caller+0x1f/0xb2
[   23.564204]  [<ffffffff81af12d3>] _raw_spin_lock+0x3b/0x6d
[   23.564204]  [<ffffffff81a27fe2>] ? __udp4_lib_rcv+0x569/0x79d
[   23.564204]  [<ffffffff81a27fe2>] __udp4_lib_rcv+0x569/0x79d
[   23.564204]  [<ffffffff81a286ae>] udp_rcv+0x1a/0x1c
[   23.564204]  [<ffffffff81a0098c>] ip_local_deliver_finish+0x151/0x2a0
[   23.564204]  [<ffffffff81a0086a>] ? ip_local_deliver_finish+0x2f/0x2a0
[   23.564204]  [<ffffffff81a0109e>] ip_local_deliver+0x7a/0x7f
[   23.564204]  [<ffffffff81a00e75>] ip_rcv_finish+0x39a/0x461
[   23.564204]  [<ffffffff81a01326>] ip_rcv+0x283/0x2e0
[   23.564204]  [<ffffffff819b28ef>] __netif_receive_skb_core+0x4cd/0x58e
[   23.564204]  [<ffffffff819b258a>] ? __netif_receive_skb_core+0x168/0x58e
[   23.564204]  [<ffffffff819b2dd6>] __netif_receive_skb+0x1d/0x5f
[   23.564204]  [<ffffffff819b2e80>] netif_receive_skb+0x68/0xf4
[   23.564204]  [<ffffffff819b35da>] napi_gro_receive+0x3e/0x80
[   23.564204]  [<ffffffff817be870>] e1000_clean_rx_irq+0x393/0x415
[   23.564204]  [<ffffffff817bcd48>] e1000_clean+0x59d/0x71f
[   23.564204]  [<ffffffff810634d1>] ? kvm_clock_read+0x27/0x31
[   23.564204]  [<ffffffff8103edd0>] ? sched_clock+0x9/0xd
[   23.564204]  [<ffffffff819b3ed0>] net_rx_action+0xf6/0x210
[   23.564204]  [<ffffffff810762fc>] __do_softirq+0xf3/0x27a
[   23.564204]  [<ffffffff81076779>] irq_exit+0x59/0xa0
[   23.564204]  [<ffffffff81afb44d>] do_IRQ+0x9d/0xb4
[   23.564204]  [<ffffffff81af21f2>] common_interrupt+0x72/0x72
[   23.564204]  <EOI>  [<ffffffff81af5ab5>] ? __atomic_notifier_call_chain+0x5/0x9c
[   23.564204]  [<ffffffff810636c8>] ? native_safe_halt+0x6/0x8
[   23.564204]  [<ffffffff8103f9d1>] default_idle+0x38/0xc1
[   23.564204]  [<ffffffff81040154>] arch_cpu_idle+0x18/0x28
[   23.564204]  [<ffffffff810c0ac5>] cpu_startup_entry+0x15c/0x24d
[   23.564204]  [<ffffffff81ad1f25>] rest_init+0xc9/0xcd
[   23.564204]  [<ffffffff81ad1e61>] ? rest_init+0x5/0xcd
[   23.564204]  [<ffffffff823aed77>] start_kernel+0x3d4/0x3e0
[   23.564204]  [<ffffffff823ae7b1>] ? repair_env_string+0x58/0x58
[   23.564204]  [<ffffffff823ae120>] ? early_idt_handlers+0x120/0x120
[   23.564204]  [<ffffffff823ae467>] x86_64_start_reservations+0x2a/0x2c
[   23.564204]  [<ffffffff823ae5a9>] x86_64_start_kernel+0x140/0x14d
[   66.435785] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[   66.440815] NFSD: starting 90-second grace period (net ffffffff82196a80)
Kernel tests: Boot OK!

git bisect start b1daf37ff52cf2ff39b2db2399e052ad7965c6aa 374b105797c3d4f29c685f3be535c35f5689b30e --
git bisect  bad c2f84f5177293270c0203fabfd600e11e211fd73  # 12:07      0-      1  Merge 'net/master' into devel-xian-x86_64-201312131052
git bisect good 2481b0b9a5f7af0f748a18a9710371a9cb22c986  # 12:41     20+      1  Merge 'arm-soc/to-build' into devel-xian-x86_64-201312131052
git bisect good c8781cf4a309ae4d1393f5878d4e51987665898c  # 13:08     20+      0  bnx2x: avoid null pointer dereference when enabling SR-IOV
git bisect good fffc15a5012e9052d3b236efc56840841a125416  # 13:36     20+      0  vxlan: release rt when found circular route
git bisect good 4bd7b5127bd02c12c1cc837a7a0b6ce295eb2505  # 13:50     20+      0  micrel: add support for KSZ8041RNLI
git bisect good 9508fdde4d53f0d9e583e841ed08796d2f310be1  # 14:14     20+      1  Revert "8390 : Replace ei_debug with msg_enable/NETIF_MSG_* feature"
git bisect  bad 10574059ce0451c6572c85329c772aa15085f8eb  # 14:28      0-      1  xen-netback: napi: fix abuse of budget
git bisect  bad 975022310233fb0f0193873d79a7b8438070fa82  # 14:28      0-      8  udp: ipv4: must add synchronization in udp_sk_rx_dst_set()
git bisect good a1bf1750871a6f242b0fdb174cc55d2c57e7ed66  # 14:34     25+      0  net:fec: remove duplicate lines in comment about errata ERR006358
# first bad commit: [975022310233fb0f0193873d79a7b8438070fa82] udp: ipv4: must add synchronization in udp_sk_rx_dst_set()
git bisect good a1bf1750871a6f242b0fdb174cc55d2c57e7ed66  # 14:46     75+      0  net:fec: remove duplicate lines in comment about errata ERR006358
git bisect  bad b1daf37ff52cf2ff39b2db2399e052ad7965c6aa  # 14:46      0-     19  Merge 'security/for-linus' into devel-xian-x86_64-201312131052
git bisect good 2cd6a37a65f578f7e23e67986107422ad3c012ba  # 15:23     75+      0  Revert "udp: ipv4: must add synchronization in udp_sk_rx_dst_set()"
git bisect good 8ae516aa8b8161254d3e402b3348b2a9b8d1efd0  # 15:54     75+      1  Merge tag 'trace-fixes-v3.13-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace
git bisect  bad 1a63d5777d237952bb06f1a2481c3acc366aeb8d  # 15:57      0-      9  Add linux-next specific files for 20131213

Thanks,
Fengguang