[crash, bisected] Kernel BUG at ffffffff8079afb1 (__netif_schedule())

Ingo Molnar
2008-07-21T13:31:22Z

David,

-tip testing on latest -git (v2.6.26-5253-g14b395e) triggered the
following boot crash on a Core2Duo 64-bit testsystem:

ADDRCONF(NETDEV_UP): eth0: link is not ready
eth0: Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
------------[ cut here ]------------
Kernel BUG at ffffffff8079afb1 [verbose debug info unavailable]
invalid opcode: 0000 [1] SMP
CPU 0
Pid: 7, comm: events/0 Not tainted 2.6.26-rc8 #21302
RIP: 0010:[<ffffffff8079afb1>] [<ffffffff8079afb1>] __netif_schedule+0xd/0x64
RSP: 0018:ffff81003fa4be30 EFLAGS: 00010246
RAX: 00000000ffffffff RBX: ffff81003e9f49f0 RCX: ffffffff80c38fe0
RDX: ffff81003e9e7940 RSI: ffffffff80c3fdc0 RDI: ffffffff80c3fdc0
RBP: ffff81003e9f49f0 R08: 0000000000001607 R09: ffff810003b1c380
R10: 0000000000000005 R11: 00000100ffffffff R12: 0000000000000000
R13: ffff81003e9f49f0 R14: ffff81003e9f4000 R15: ffff81003e9f46c0
FS: 0000000000000000(0000) GS:ffffffff80c7c000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000de7ef0 CR3: 000000003e5f1000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process events/0 (pid: 7, threadinfo ffff81003fa4a000, task ffff81003fa20c30)
Stack: ffff81003e9f49f0 ffffffff805083a4 ffffffff80d19840 ffff81003e9f4a08
ffff81003e9e7880 0000000000000000 0000000000000000 0000000000000202
000000003f9f5f10 ffff81003fa06a40 ffffffff80507f5c ffff81003fa06a48
Call Trace:
[<ffffffff805083a4>] ? e1000_watchdog_task+0x448/0x635
[<ffffffff80507f5c>] ? e1000_watchdog_task+0x0/0x635
[<ffffffff8023e11b>] ? run_workqueue+0x80/0x112
[<ffffffff8023e9f6>] ? worker_thread+0xd9/0xe8
[<ffffffff802410af>] ? autoremove_wake_function+0x0/0x2e
[<ffffffff8023e91d>] ? worker_thread+0x0/0xe8
[<ffffffff80240f58>] ? kthread+0x47/0x73
[<ffffffff8022d557>] ? schedule_tail+0x28/0x5d
[<ffffffff8020c288>] ? child_rip+0xa/0x12
[<ffffffff80240f11>] ? kthread+0x0/0x73
[<ffffffff8020c27e>] ? child_rip+0x0/0x12

Code: c2 48 8b 42 30 48 89 06 48 89 72 30 e8 30 ab a9 ff 48 89 df 57 9d 66 0f 1f 44 00 00 5b c3 48 81 ff c0 fd c3 80 53 48 89 fe 75 04 <0f> 0b eb fe f0 0f ba 6f 30 01 19 c0 85 c0 75 45 9c 58 66 0f 1f
RIP [<ffffffff8079afb1>] __netif_schedule+0xd/0x64
RSP <ffff81003fa4be30>
Kernel panic - not syncing: Fatal exception

i've bisected it back to:

| 37437bb2e1ae8af470dfcd5b4ff454110894ccaf is first bad commit
| commit 37437bb2e1ae8af470dfcd5b4ff454110894ccaf
| Author: David S. Miller <davem@davemloft.net>
| Date: Wed Jul 16 02:15:04 2008 -0700
|
| pkt_sched: Schedule qdiscs instead of netdev_queue.

bisection log:

# bad: [14b395e1] Merge branch 'for-2.6.27' of git://linux-nfs.org/~
# good: [bce7f795] Linux 2.6.26
# good: [cadc7236] Merge branch 'bkl-removal' into next
# bad: [a0c80b8d] pkt_sched: Make default qdisc nonshared-multiqueue
# good: [30902dc4] ax25: Fix std timer socket destroy handling.
# good: [fbd8f13a] net-sched: sch_htb: move hash and sibling list rem
# good: [83aa2e9b] netlabel: return msg overflow error from netlbl_ci
# good: [0388b002] icmp: add struct net argument to icmp_out_count
# good: [ca12a1ac] inet: prepare net on the stack for NET accounting
# good: [8f0f2227] net: Implement simple sw TX hashing.
# bad: [17715e68] pkt_sched: Use per-queue locking in shutdown_sched
# good: [e2627c85] pkt_sched: Make QDISC_RUNNING a qdisc state.
# bad: [37437bbf] pkt_sched: Schedule qdiscs instead of netdev_queue
# good: [7698b4ff] pkt_sched: Add and use qdisc_root() and qdisc_root

config and crashlog:

http://redhat.com/~mingo/misc/config-Mon_Jul_21_13_59_54_CEST_2008.bad
http://redhat.com/~mingo/misc/crash-Mon_Jul_21_13_59_54_CEST_2008.log

[ Note: the bootlog says 2.6.26-rc8 - that's because bisection dived
back to when you cut that devel tree of yours. ]

As the bug is reproducible i can test patches, etc. Let me know if you
need more info than this.

Thanks,

Ingo