Re: X-freeze after clflush changes [Was: 2.6.23-rc6-mm1]

Jiri Slaby
2007-09-24T08:10:30Z

On 09/24/2007 09:37 AM, Valdis.Kletnieks@vt.edu wrote:
> (Interestingly, I can't find any of the 3 addresses listed in the 'list_add
> corruption' message anywhere *else* in the netconsole output, and the last thing
> we hear from before the kersplat is apparently an RCU callback in a softirq?)

Hmm, there must be somebody else who changes it under hands without list_add.
Any lru skin game in the mid-layer nvidia sources? Do slab and slub behave the same?

[...]
> [ 48.222000]
> [ 48.297000] POISONS (ffff810001179148): ffff810006bbc000, ffff81000341bec0
> [ 48.297000]
> [ 48.297000] Call Trace:
> [ 48.297000] <IRQ> [<ffffffff803580aa>] __list_add+0xd7/0x138
> [ 48.297000] [<ffffffff80358117>] list_add+0xc/0x11
> [ 48.297000] [<ffffffff80270865>] free_hot_cold_page+0xe8/0x16d
> [ 48.297000] [<ffffffff8027093e>] free_hot_page+0xb/0xd
> [ 48.297000] [<ffffffff80270958>] __free_pages+0x18/0x21
> [ 48.297000] [<ffffffff80270990>] free_pages+0x2f/0x34
> [ 48.297000] [<ffffffff8028922d>] kmem_freepages+0xc5/0xce
> [ 48.297000] [<ffffffff8028957f>] slab_destroy+0x3c/0x53
> [ 48.297000] [<ffffffff80289663>] free_block+0xcd/0x110
> [ 48.297000] [<ffffffff80289345>] cache_flusharray+0x71/0xa7
> [ 48.297000] [<ffffffff802894c2>] kmem_cache_free+0x99/0xb2
> [ 48.297000] [<ffffffff8029d8b2>] __d_free+0x30/0x34
> [ 48.297000] [<ffffffff8029dcb6>] d_callback+0xd/0xf
> [ 48.297000] [<ffffffff802458e0>] __rcu_process_callbacks+0x143/0x1da
> [ 48.297000] [<ffffffff8024599a>] rcu_process_callbacks+0x23/0x44
> [ 48.297000] [<ffffffff802396d2>] tasklet_action+0x54/0x9e
> [ 48.297000] [<ffffffff802395ad>] __do_softirq+0x57/0xc7
> [ 48.297000] [<ffffffff802398e3>] ksoftirqd+0x0/0x148
> [ 48.297000] [<ffffffff8020d32c>] call_softirq+0x1c/0x28
> [ 48.297000] <EOI> [<ffffffff8020e916>] do_softirq+0x34/0x87
> [ 48.297000] [<ffffffff80239956>] ksoftirqd+0x73/0x148
> [ 48.297000] [<ffffffff80247ddd>] kthread+0x49/0x78
> [ 48.297000] [<ffffffff8020cfc8>] child_rip+0xa/0x12
> [ 48.297000] [<ffffffff80247d94>] kthread+0x0/0x78
> [ 48.297000] [<ffffffff8020cfbe>] child_rip+0x0/0x12
> [ 48.297000]
> [ 48.297000] list_add corruption. next->prev should be prev (ffffffff8067e050), but was ffff8100066d59c0. (next=ffff81000119e560).
> [ 48.297000] ------------[ cut here ]------------
> [ 48.297000] kernel BUG at lib/list_debug.c:46!
> [ 48.297000] invalid opcode: 0000 [1] PREEMPT SMP
> [ 48.297000] last sysfs file: /devices/pci0000:00/0000:00:1e.0/0000:03:01.4/resource
> [ 48.297000] CPU 1
> [ 48.297000] Modules linked in: irnet(U) ppp_generic(U) slhc(U) irtty_sir(U) sir_dev(U) ircomm_tty(U) ircomm(U) irda(U) crc_ccitt(U) coretemp(U) nf_conntrack_ftp(U) xt_pkttype(U) ipt_REJECT(U) ipt_osf(U) nf_conntrack_ipv4(U) xt_ipisforif(U) ipt_recent(U) ipt_LOG(U) xt_u32(U) iptable_filter(U) ip_tables(U) xt_tcpudp(U) nf_conntrack_ipv6(U) xt_state(U) nf_conntrack(U) nfnetlink(U) ip6t_LOG(U) xt_limit(U) ip6table_filter(U) ip6_tables(U) x_tables(U) sha256(U) aes(U) fan(U) container(U) bay(U) acpi_cpufreq(U) nvram(U) pcmcia(U) firmware_class(U) yenta_socket(U) ohci1394(U) rsrc_nonstatic(U) iTCO_wdt(U) iTCO_vendor_support(U) watchdog_core(U) nvidia(P)(U) thermal(U) ieee1394(U) pcmcia_core(U) watchdog_dev(U) processor(U) snd_hda_intel(U) intel_agp(U) ac(U) button(U) video(U) battery(U) power_supply(U) output(U) rtc(U)
> [ 48.297000] Pid: 7, comm: ksoftirqd/1 Tainted: P 2.6.23-rc6-mm1 #8
> [ 48.297000] RIP: 0010:[<ffffffff803580ce>] [<ffffffff803580ce>] __list_add+0xfb/0x138
> [ 48.297000] RSP: 0000:ffff81000349fd38 EFLAGS: 00010002
> [ 48.297000] RAX: 0000000000000088 RBX: ffff810001179148 RCX: ffffffff8061dbbb
> [ 48.297000] RDX: 0000000100000000 RSI: 0000000000000006 RDI: ffffffff80672620
> [ 48.297000] RBP: ffff81000349fd58 R08: ffffffff80672628 R09: 0000000000000000
> [ 48.297000] R10: 00000000e731ffa2 R11: ffff81000349fa98 R12: ffff81000119e560
> [ 48.297000] R13: ffffffff8067e050 R14: ffffffff8067df80 R15: ffff81000346d128
> [ 48.297000] FS: 0000000000000000(0000) GS:ffff8100034689c0(0000) knlGS:0000000000000000
> [ 48.297000] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> [ 48.297000] CR2: 000000000049b9b0 CR3: 0000000004168000 CR4: 00000000000006e0
> [ 48.297000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 48.297000] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 48.297000] Process ksoftirqd/1 (pid: 7, threadinfo ffff810003494000, task ffff810003463810)
> [ 48.297000] last branch before last exception/interrupt
> [ 48.297000] from [<ffffffff80234989>] printk+0xa3/0xa4
> [ 48.297000] to [<ffffffff803580c8>] __list_add+0xf5/0x138
> [ 48.297000] Stack: ffff81000349fe60 ffff810001179120 ffffffff8067e040 0000000000000002
> [ 48.297000] ffff81000349fd68 ffffffff80358117 ffff81000349fd98 ffffffff80270865
> [ 48.297000] ffff810001000000 0000000000000000 ffff81000341bec0 ffff810006bbc000
> [ 48.297000] Call Trace:
> [ 48.297000] <IRQ> [<ffffffff80358117>] list_add+0xc/0x11
> [ 48.297000] [<ffffffff80270865>] free_hot_cold_page+0xe8/0x16d
> [ 48.297000] [<ffffffff8027093e>] free_hot_page+0xb/0xd
> [ 48.297000] [<ffffffff80270958>] __free_pages+0x18/0x21
> [ 48.297000] [<ffffffff80270990>] free_pages+0x2f/0x34
> [ 48.297000] [<ffffffff8028922d>] kmem_freepages+0xc5/0xce
> [ 48.297000] [<ffffffff8028957f>] slab_destroy+0x3c/0x53
> [ 48.297000] [<ffffffff80289663>] free_block+0xcd/0x110
> [ 48.297000] [<ffffffff80289345>] cache_flusharray+0x71/0xa7
> [ 48.297000] [<ffffffff802894c2>] kmem_cache_free+0x99/0xb2
> [ 48.297000] [<ffffffff8029d8b2>] __d_free+0x30/0x34
> [ 48.297000] [<ffffffff8029dcb6>] d_callback+0xd/0xf
> [ 48.297000] [<ffffffff802458e0>] __rcu_process_callbacks+0x143/0x1da
> [ 48.297000] [<ffffffff8024599a>] rcu_process_callbacks+0x23/0x44
> [ 48.297000] [<ffffffff802396d2>] tasklet_action+0x54/0x9e
> [ 48.297000] [<ffffffff802395ad>] __do_softirq+0x57/0xc7
> [ 48.297000] [<ffffffff802398e3>] ksoftirqd+0x0/0x148
> [ 48.297000] [<ffffffff8020d32c>] call_softirq+0x1c/0x28
> [ 48.297000] <EOI> [<ffffffff8020e916>] do_softirq+0x34/0x87
> [ 48.297000] [<ffffffff80239956>] ksoftirqd+0x73/0x148
> [ 48.297000] [<ffffffff80247ddd>] kthread+0x49/0x78
> [ 48.297000] [<ffffffff8020cfc8>] child_rip+0xa/0x12
> [ 48.297000] [<ffffffff80247d94>] kthread+0x0/0x78
> [ 48.297000] [<ffffffff8020cfbe>] child_rip+0x0/0x12
> [ 48.297000]
> [ 48.297000]
> [ 48.297000] Code: 0f 0b eb fe 49 8b 55 00 4c 39 e2 74 18 4c 89 e9 4c 89 e6 48
> [ 48.297000] RIP [<ffffffff803580ce>] __list_add+0xfb/0x138
> [ 48.297000] RSP <ffff81000349fd38>
>

--
Jiri Slaby (jirislaby@gmail.com)
Faculty of Informatics, Masaryk University