* [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b @ 2009-01-07 0:12 Justin P. Mattock 2009-01-07 6:48 ` Pekka Enberg 0 siblings, 1 reply; 15+ messages in thread From: Justin P. Mattock @ 2009-01-07 0:12 UTC (permalink / raw) To: linux-kernel With pulling git today I'm unable to shut the machine down completely. (the system just sits there with the message on the screen); * will now halt [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b [ 286.548940] IP: [<c0150ca4>] __stop_machine+0x88/0xe3 [ 286.550598] Oops: 0002 [#1] SMP [ 286.552206] last sysfs file: /sys/block/sda/removeable [ 286.553844] Modules linked in: hidp radeon drm agpgart btusb rfcomm bnep sco l2cap bluetooth fan battery container ipt_LOG xt_limit xt_tcpudp xt_state ipt_addrtype nf_nat_irc nf_conntrack_irc nf_nat_ftp nf_nat nf_conntrack_ftp ipmi_watchdog ipmi_msghandler uvcvideo isight_firmware uinput arpt_mangle arptable_filter arp_tables nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_mangle iptable_filter ip_tables x_tables coretemp eeprom acpi_cpufreq cpufreq_powersave cpufreq_performance cpufreq_ondemand cpufreq_conservative appletouch snd_had_codec_idt ohci1394 ehci_hcd snd_hda_intel snd_hda_codec thermal ath9k uhci_hcd ieee1394 joydev pata_acpi snd_hwdep snd_pcm snd_page_alloc video ac button processor applesmc evdev [ 286.560580] [ 286.560580] Pid: 3273, comm: halt Not tainted (2.6.28-06127-g238c6d5 #1) MacBookPro2,2 [ 286.560580] EIP: 0060:[<c0150ca4>] EFLAGS: 00010293 CPU: 0 [ 286.560580] EIP: is at __stop_machine+0x88/0xe3 [ 286.560580] EAX: 6b6b6b6b EBX: 00000000 ECX: 6b6b6b6b EDX: 00000000 [ 286.560580] ESI: c054abe0 EDI: c03d03a4 EBP: f1a29e54 ESP: f1a29e44 [ 286.560580] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 [ 286.560580] Process halt (pid: 3273, ti=f1a28000 task=f4530f30 task.ti=f1a28000) [ 286.560580] Stack: [ 286.560580] f1a29e60 c054abe0 00000001 00000010 f1a29e7c c03d04e4 ffffffea 00000010 [ 286.560580] 00000001 00000003 00000022 00000001 4321fedc c054abe0 f1a29e94 c012a57e [ 286.560580] 00000000 ffffffff 4321fedc 28121969 f1a29e9c c01360c0 f1a29fb0 c0136301 [ 286.560580] Call Trace: [ 286.560580] [<c03d04e4>] ? _cpu_down+0x10f/0x234 [ 286.560580] [<c012a57e>] ? disable_nonboot_cpus+0x58/0xdc [ 286.560580] [<c01360c0>] ? kernel_poweroff+0x22/0x39 [ 286.560580] [<c0136301>] ? sys_reboot+0xde/0x14c [ 286.560580] [<c01331b2>] ? complete_signal+0x179/0x191 [ 286.560580] [<c0133396>] ? send_signal+0x1cc/0x1e1 [ 286.560580] [<c03de418>] ? _spin_unlock_irqrestore+0x2d/0x3c [ 286.560580] [<c0133b65>] ? group_send_signal_info+0x58/0x61 [ 286.560580] [<c0133b9e>] ? kill_pid_info+0x30/0x3a [ 286.560580] [<c0133d49>] ? sys_kill+0x75/0x13a [ 286.560580] [<c01a06cb>] ? mntput_no_expire+ox1f/0x101 [ 286.560580] [<c019b3b3>] ? dput+0x1e/0x105 [ 286.560580] [<c018ef87>] ? __fput+0x150/0x158 [ 286.560580] [<c0157abf>] ? audit_syscall_entry+0x137/0x159 [ 286.560580] [<c010329f>] ? sysenter_do_call+0x12/0x34 [ 286.560580] Code: c7 05 10 06 62 c0 00 00 00 00 a3 f4 05 62 c0 c7 05 ec 05 62 c0 01 00 00 00 83 cb ff eb 2d a1 1c 06 62 c0 f7 d0 8b 0c 98 8d 41 04 <c7> 01 00 00 00 00 89 41 04 89 41 08 c7 41 0c ff 0c 15 c0 89 d8 [ 286.560580] EIP: [<c0150ca4>] __stop_machine+0x88/0xe3 SS:ESP 0068:f1a29e44 [ 286.639215] ---[ end trace 5b080c1ab14203ae ] --- Segmentation fault after this message appears, if I hold down the start button the system shuts off after a few seconds. (BTW hopefully the number are correct, manually writing this down, is a bit of a pain); regards; Justin P. Mattock ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b 2009-01-07 0:12 [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b Justin P. Mattock @ 2009-01-07 6:48 ` Pekka Enberg 2009-01-07 8:13 ` Justin P. Mattock 2009-01-07 8:30 ` Pekka Enberg 0 siblings, 2 replies; 15+ messages in thread From: Pekka Enberg @ 2009-01-07 6:48 UTC (permalink / raw) To: Justin P. Mattock; +Cc: linux-kernel, Rusty Russell Hi Justin, On Wed, Jan 7, 2009 at 2:12 AM, Justin P. Mattock <justinmattock@gmail.com> wrote: > With pulling git today I'm unable to shut the machine down completely. > (the system just sits there with the message on the screen); > > * will now halt > [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b That looks like use-after-free in __stop_machine() so lets cc Rusty. If you want, you can convert the oops location into human-readable form. Just search for "GDB" in Documentation/BUG-HUNTING for instructions how to do that. And don't forget to send your .config. > [ 286.548940] IP: [<c0150ca4>] __stop_machine+0x88/0xe3 > [ 286.550598] Oops: 0002 [#1] SMP > [ 286.552206] last sysfs file: /sys/block/sda/removeable > [ 286.553844] Modules linked in: hidp radeon drm agpgart btusb rfcomm bnep > sco l2cap bluetooth fan battery container ipt_LOG xt_limit xt_tcpudp > xt_state ipt_addrtype nf_nat_irc nf_conntrack_irc nf_nat_ftp nf_nat > nf_conntrack_ftp ipmi_watchdog ipmi_msghandler uvcvideo isight_firmware > uinput arpt_mangle arptable_filter arp_tables nf_conntrack_ipv4 nf_conntrack > nf_defrag_ipv4 iptable_mangle iptable_filter ip_tables x_tables coretemp > eeprom acpi_cpufreq cpufreq_powersave cpufreq_performance cpufreq_ondemand > cpufreq_conservative appletouch snd_had_codec_idt ohci1394 ehci_hcd > snd_hda_intel snd_hda_codec thermal ath9k uhci_hcd ieee1394 joydev pata_acpi > snd_hwdep snd_pcm snd_page_alloc video ac button processor applesmc evdev > [ 286.560580] > [ 286.560580] Pid: 3273, comm: halt Not tainted (2.6.28-06127-g238c6d5 #1) > MacBookPro2,2 > [ 286.560580] EIP: 0060:[<c0150ca4>] EFLAGS: 00010293 CPU: 0 > [ 286.560580] EIP: is at __stop_machine+0x88/0xe3 > [ 286.560580] EAX: 6b6b6b6b EBX: 00000000 ECX: 6b6b6b6b EDX: 00000000 > [ 286.560580] ESI: c054abe0 EDI: c03d03a4 EBP: f1a29e54 ESP: f1a29e44 > [ 286.560580] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 > [ 286.560580] Process halt (pid: 3273, ti=f1a28000 task=f4530f30 > task.ti=f1a28000) > [ 286.560580] Stack: > [ 286.560580] f1a29e60 c054abe0 00000001 00000010 f1a29e7c c03d04e4 > ffffffea 00000010 > [ 286.560580] 00000001 00000003 00000022 00000001 4321fedc c054abe0 > f1a29e94 c012a57e > [ 286.560580] 00000000 ffffffff 4321fedc 28121969 f1a29e9c c01360c0 > f1a29fb0 c0136301 > [ 286.560580] Call Trace: > [ 286.560580] [<c03d04e4>] ? _cpu_down+0x10f/0x234 > [ 286.560580] [<c012a57e>] ? disable_nonboot_cpus+0x58/0xdc > [ 286.560580] [<c01360c0>] ? kernel_poweroff+0x22/0x39 > [ 286.560580] [<c0136301>] ? sys_reboot+0xde/0x14c > [ 286.560580] [<c01331b2>] ? complete_signal+0x179/0x191 > [ 286.560580] [<c0133396>] ? send_signal+0x1cc/0x1e1 > [ 286.560580] [<c03de418>] ? _spin_unlock_irqrestore+0x2d/0x3c > [ 286.560580] [<c0133b65>] ? group_send_signal_info+0x58/0x61 > [ 286.560580] [<c0133b9e>] ? kill_pid_info+0x30/0x3a > [ 286.560580] [<c0133d49>] ? sys_kill+0x75/0x13a > [ 286.560580] [<c01a06cb>] ? mntput_no_expire+ox1f/0x101 > [ 286.560580] [<c019b3b3>] ? dput+0x1e/0x105 > [ 286.560580] [<c018ef87>] ? __fput+0x150/0x158 > [ 286.560580] [<c0157abf>] ? audit_syscall_entry+0x137/0x159 > [ 286.560580] [<c010329f>] ? sysenter_do_call+0x12/0x34 > [ 286.560580] Code: c7 05 10 06 62 c0 00 00 00 00 a3 f4 05 62 c0 c7 05 ec > 05 62 c0 01 00 00 00 83 cb ff eb 2d a1 1c 06 62 c0 f7 d0 8b 0c 98 8d 41 04 > <c7> 01 00 00 00 00 89 41 04 89 41 08 c7 41 0c ff 0c 15 c0 89 d8 > [ 286.560580] EIP: [<c0150ca4>] __stop_machine+0x88/0xe3 SS:ESP > 0068:f1a29e44 > [ 286.639215] ---[ end trace 5b080c1ab14203ae ] --- > Segmentation fault > > after this message appears, if I hold down the start button > the system shuts off after a few seconds. > (BTW hopefully the number are correct, > manually writing this down, is a bit of a pain); > > regards; > > Justin P. Mattock > > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b 2009-01-07 6:48 ` Pekka Enberg @ 2009-01-07 8:13 ` Justin P. Mattock 2009-01-07 8:30 ` Pekka Enberg 1 sibling, 0 replies; 15+ messages in thread From: Justin P. Mattock @ 2009-01-07 8:13 UTC (permalink / raw) To: Pekka Enberg; +Cc: linux-kernel, Rusty Russell Pekka Enberg wrote: > Hi Justin, > > On Wed, Jan 7, 2009 at 2:12 AM, Justin P. Mattock > <justinmattock@gmail.com> wrote: > >> With pulling git today I'm unable to shut the machine down completely. >> (the system just sits there with the message on the screen); >> >> * will now halt >> [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b >> > > That looks like use-after-free in __stop_machine() so lets cc Rusty. > If you want, you can convert the oops location into human-readable > form. Just search for "GDB" in Documentation/BUG-HUNTING for > instructions how to do that. And don't forget to send your .config. > > >> [ 286.548940] IP: [<c0150ca4>] __stop_machine+0x88/0xe3 >> [ 286.550598] Oops: 0002 [#1] SMP >> [ 286.552206] last sysfs file: /sys/block/sda/removeable >> [ 286.553844] Modules linked in: hidp radeon drm agpgart btusb rfcomm bnep >> sco l2cap bluetooth fan battery container ipt_LOG xt_limit xt_tcpudp >> xt_state ipt_addrtype nf_nat_irc nf_conntrack_irc nf_nat_ftp nf_nat >> nf_conntrack_ftp ipmi_watchdog ipmi_msghandler uvcvideo isight_firmware >> uinput arpt_mangle arptable_filter arp_tables nf_conntrack_ipv4 nf_conntrack >> nf_defrag_ipv4 iptable_mangle iptable_filter ip_tables x_tables coretemp >> eeprom acpi_cpufreq cpufreq_powersave cpufreq_performance cpufreq_ondemand >> cpufreq_conservative appletouch snd_had_codec_idt ohci1394 ehci_hcd >> snd_hda_intel snd_hda_codec thermal ath9k uhci_hcd ieee1394 joydev pata_acpi >> snd_hwdep snd_pcm snd_page_alloc video ac button processor applesmc evdev >> [ 286.560580] >> [ 286.560580] Pid: 3273, comm: halt Not tainted (2.6.28-06127-g238c6d5 #1) >> MacBookPro2,2 >> [ 286.560580] EIP: 0060:[<c0150ca4>] EFLAGS: 00010293 CPU: 0 >> [ 286.560580] EIP: is at __stop_machine+0x88/0xe3 >> [ 286.560580] EAX: 6b6b6b6b EBX: 00000000 ECX: 6b6b6b6b EDX: 00000000 >> [ 286.560580] ESI: c054abe0 EDI: c03d03a4 EBP: f1a29e54 ESP: f1a29e44 >> [ 286.560580] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 >> [ 286.560580] Process halt (pid: 3273, ti=f1a28000 task=f4530f30 >> task.ti=f1a28000) >> [ 286.560580] Stack: >> [ 286.560580] f1a29e60 c054abe0 00000001 00000010 f1a29e7c c03d04e4 >> ffffffea 00000010 >> [ 286.560580] 00000001 00000003 00000022 00000001 4321fedc c054abe0 >> f1a29e94 c012a57e >> [ 286.560580] 00000000 ffffffff 4321fedc 28121969 f1a29e9c c01360c0 >> f1a29fb0 c0136301 >> [ 286.560580] Call Trace: >> [ 286.560580] [<c03d04e4>] ? _cpu_down+0x10f/0x234 >> [ 286.560580] [<c012a57e>] ? disable_nonboot_cpus+0x58/0xdc >> [ 286.560580] [<c01360c0>] ? kernel_poweroff+0x22/0x39 >> [ 286.560580] [<c0136301>] ? sys_reboot+0xde/0x14c >> [ 286.560580] [<c01331b2>] ? complete_signal+0x179/0x191 >> [ 286.560580] [<c0133396>] ? send_signal+0x1cc/0x1e1 >> [ 286.560580] [<c03de418>] ? _spin_unlock_irqrestore+0x2d/0x3c >> [ 286.560580] [<c0133b65>] ? group_send_signal_info+0x58/0x61 >> [ 286.560580] [<c0133b9e>] ? kill_pid_info+0x30/0x3a >> [ 286.560580] [<c0133d49>] ? sys_kill+0x75/0x13a >> [ 286.560580] [<c01a06cb>] ? mntput_no_expire+ox1f/0x101 >> [ 286.560580] [<c019b3b3>] ? dput+0x1e/0x105 >> [ 286.560580] [<c018ef87>] ? __fput+0x150/0x158 >> [ 286.560580] [<c0157abf>] ? audit_syscall_entry+0x137/0x159 >> [ 286.560580] [<c010329f>] ? sysenter_do_call+0x12/0x34 >> [ 286.560580] Code: c7 05 10 06 62 c0 00 00 00 00 a3 f4 05 62 c0 c7 05 ec >> 05 62 c0 01 00 00 00 83 cb ff eb 2d a1 1c 06 62 c0 f7 d0 8b 0c 98 8d 41 04 >> <c7> 01 00 00 00 00 89 41 04 89 41 08 c7 41 0c ff 0c 15 c0 89 d8 >> [ 286.560580] EIP: [<c0150ca4>] __stop_machine+0x88/0xe3 SS:ESP >> 0068:f1a29e44 >> [ 286.639215] ---[ end trace 5b080c1ab14203ae ] --- >> Segmentation fault >> >> after this message appears, if I hold down the start button >> the system shuts off after a few seconds. >> (BTW hopefully the number are correct, >> manually writing this down, is a bit of a pain); >> >> regards; >> >> Justin P. Mattock >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> Please read the FAQ at http://www.tux.org/lkml/ >> >> > > Thats nice, thanks for the info. I like the idea of using gcc to disassemble this text. Since not knowing what I'm doing, I'll have to do my homework on this. (really curious to see what this does); regards; Justin P. Mattock ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b 2009-01-07 6:48 ` Pekka Enberg 2009-01-07 8:13 ` Justin P. Mattock @ 2009-01-07 8:30 ` Pekka Enberg 2009-01-07 9:15 ` Heiko Carstens 1 sibling, 1 reply; 15+ messages in thread From: Pekka Enberg @ 2009-01-07 8:30 UTC (permalink / raw) To: Justin P. Mattock; +Cc: linux-kernel, Rusty Russell, heiko.carstens On Wed, Jan 7, 2009 at 8:48 AM, Pekka Enberg <penberg@cs.helsinki.fi> wrote: > On Wed, Jan 7, 2009 at 2:12 AM, Justin P. Mattock > <justinmattock@gmail.com> wrote: >> With pulling git today I'm unable to shut the machine down completely. >> (the system just sits there with the message on the screen); >> >> * will now halt >> [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b > > That looks like use-after-free in __stop_machine() so lets cc Rusty. > If you want, you can convert the oops location into human-readable > form. Just search for "GDB" in Documentation/BUG-HUNTING for > instructions how to do that. And don't forget to send your .config. > >> [ 286.548940] IP: [<c0150ca4>] __stop_machine+0x88/0xe3 >> [ 286.550598] Oops: 0002 [#1] SMP >> [ 286.552206] last sysfs file: /sys/block/sda/removeable >> [ 286.553844] Modules linked in: hidp radeon drm agpgart btusb rfcomm bnep >> sco l2cap bluetooth fan battery container ipt_LOG xt_limit xt_tcpudp >> xt_state ipt_addrtype nf_nat_irc nf_conntrack_irc nf_nat_ftp nf_nat >> nf_conntrack_ftp ipmi_watchdog ipmi_msghandler uvcvideo isight_firmware >> uinput arpt_mangle arptable_filter arp_tables nf_conntrack_ipv4 nf_conntrack >> nf_defrag_ipv4 iptable_mangle iptable_filter ip_tables x_tables coretemp >> eeprom acpi_cpufreq cpufreq_powersave cpufreq_performance cpufreq_ondemand >> cpufreq_conservative appletouch snd_had_codec_idt ohci1394 ehci_hcd >> snd_hda_intel snd_hda_codec thermal ath9k uhci_hcd ieee1394 joydev pata_acpi >> snd_hwdep snd_pcm snd_page_alloc video ac button processor applesmc evdev >> [ 286.560580] >> [ 286.560580] Pid: 3273, comm: halt Not tainted (2.6.28-06127-g238c6d5 #1) >> MacBookPro2,2 >> [ 286.560580] EIP: 0060:[<c0150ca4>] EFLAGS: 00010293 CPU: 0 >> [ 286.560580] EIP: is at __stop_machine+0x88/0xe3 >> [ 286.560580] EAX: 6b6b6b6b EBX: 00000000 ECX: 6b6b6b6b EDX: 00000000 >> [ 286.560580] ESI: c054abe0 EDI: c03d03a4 EBP: f1a29e54 ESP: f1a29e44 >> [ 286.560580] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 >> [ 286.560580] Process halt (pid: 3273, ti=f1a28000 task=f4530f30 >> task.ti=f1a28000) >> [ 286.560580] Stack: >> [ 286.560580] f1a29e60 c054abe0 00000001 00000010 f1a29e7c c03d04e4 >> ffffffea 00000010 >> [ 286.560580] 00000001 00000003 00000022 00000001 4321fedc c054abe0 >> f1a29e94 c012a57e >> [ 286.560580] 00000000 ffffffff 4321fedc 28121969 f1a29e9c c01360c0 >> f1a29fb0 c0136301 >> [ 286.560580] Call Trace: >> [ 286.560580] [<c03d04e4>] ? _cpu_down+0x10f/0x234 >> [ 286.560580] [<c012a57e>] ? disable_nonboot_cpus+0x58/0xdc >> [ 286.560580] [<c01360c0>] ? kernel_poweroff+0x22/0x39 >> [ 286.560580] [<c0136301>] ? sys_reboot+0xde/0x14c >> [ 286.560580] [<c01331b2>] ? complete_signal+0x179/0x191 >> [ 286.560580] [<c0133396>] ? send_signal+0x1cc/0x1e1 >> [ 286.560580] [<c03de418>] ? _spin_unlock_irqrestore+0x2d/0x3c >> [ 286.560580] [<c0133b65>] ? group_send_signal_info+0x58/0x61 >> [ 286.560580] [<c0133b9e>] ? kill_pid_info+0x30/0x3a >> [ 286.560580] [<c0133d49>] ? sys_kill+0x75/0x13a >> [ 286.560580] [<c01a06cb>] ? mntput_no_expire+ox1f/0x101 >> [ 286.560580] [<c019b3b3>] ? dput+0x1e/0x105 >> [ 286.560580] [<c018ef87>] ? __fput+0x150/0x158 >> [ 286.560580] [<c0157abf>] ? audit_syscall_entry+0x137/0x159 >> [ 286.560580] [<c010329f>] ? sysenter_do_call+0x12/0x34 >> [ 286.560580] Code: c7 05 10 06 62 c0 00 00 00 00 a3 f4 05 62 c0 c7 05 ec >> 05 62 c0 01 00 00 00 83 cb ff eb 2d a1 1c 06 62 c0 f7 d0 8b 0c 98 8d 41 04 >> <c7> 01 00 00 00 00 89 41 04 89 41 08 c7 41 0c ff 0c 15 c0 89 d8 >> [ 286.560580] EIP: [<c0150ca4>] __stop_machine+0x88/0xe3 SS:ESP >> 0068:f1a29e44 >> [ 286.639215] ---[ end trace 5b080c1ab14203ae ] --- >> Segmentation fault >> >> after this message appears, if I hold down the start button >> the system shuts off after a few seconds. >> (BTW hopefully the number are correct, >> manually writing this down, is a bit of a pain); scripts/decodecode gives us: 0: c7 05 10 06 62 c0 00 movl $0x0,0xc0620610 7: 00 00 00 a: a3 f4 05 62 c0 mov %eax,0xc06205f4 f: c7 05 ec 05 62 c0 01 movl $0x1,0xc06205ec 16: 00 00 00 19: 83 cb ff or $0xffffffff,%ebx 1c: eb 2d jmp 0x4b 1e: a1 1c 06 62 c0 mov 0xc062061c,%eax 23: f7 d0 not %eax 25: 8b 0c 98 mov (%eax,%ebx,4),%ecx 28: 8d 41 04 lea 0x4(%ecx),%eax 2b: c7 01 00 00 00 00 movl $0x0,(%ecx) 31: 89 41 04 mov %eax,0x4(%ecx) 34: 89 41 08 mov %eax,0x8(%ecx) 37: c7 41 0c ff 0c 15 c0 movl $0xc0150cff,0xc(%ecx) 3e: 89 d8 mov %ebx,%eax 0: c7 01 00 00 00 00 movl $0x0,(%ecx) <-- oops 6: 89 41 04 mov %eax,0x4(%ecx) 9: 89 41 08 mov %eax,0x8(%ecx) c: c7 41 0c ff 0c 15 c0 movl $0xc0150cff,0xc(%ecx) 13: 89 d8 mov %ebx,%eax objdump -S -d kernel/stop_machine.o looks like this on my machine: /* Schedule the stop_cpu work on all cpus: hold this CPU so one * doesn't hit this CPU until we're ready. */ get_cpu(); for_each_online_cpu(i) { sm_work = percpu_ptr(stop_machine_work, i); INIT_WORK(sm_work, stop_cpu); 8b: c7 01 00 00 00 00 movl $0x0,(%ecx) 91: c7 41 0c e0 00 00 00 movl $0xe0,0xc(%ecx) where #define INIT_WORK(_work, _func) \ do { \ (_work)->data = (atomic_long_t) WORK_DATA_INIT(); \ and offset of ->data is zero and WORK_DATA_INIT() expands to ATOMIC_LONG_INIT(0) so looks to me like 'sm_work' is used after it has been free'd. Perhaps stop_machine_destroy() was called before or in parallel to __stop_machine()? So lets cc Heiko as well. Pekka ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b 2009-01-07 8:30 ` Pekka Enberg @ 2009-01-07 9:15 ` Heiko Carstens 2009-01-07 9:19 ` Pekka Enberg 0 siblings, 1 reply; 15+ messages in thread From: Heiko Carstens @ 2009-01-07 9:15 UTC (permalink / raw) To: Pekka Enberg; +Cc: Justin P. Mattock, linux-kernel, Rusty Russell On Wed, Jan 07, 2009 at 10:30:56AM +0200, Pekka Enberg wrote: > On Wed, Jan 7, 2009 at 8:48 AM, Pekka Enberg <penberg@cs.helsinki.fi> wrote: > > On Wed, Jan 7, 2009 at 2:12 AM, Justin P. Mattock > > <justinmattock@gmail.com> wrote: > >> With pulling git today I'm unable to shut the machine down completely. > >> (the system just sits there with the message on the screen); [...] > >> [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b > > > > That looks like use-after-free in __stop_machine() so lets cc Rusty. > > If you want, you can convert the oops location into human-readable > > form. Just search for "GDB" in Documentation/BUG-HUNTING for > > instructions how to do that. And don't forget to send your .config. [...] > >> [ 286.560580] EIP: is at __stop_machine+0x88/0xe3 > >> [ 286.560580] [<c03d04e4>] ? _cpu_down+0x10f/0x234 > >> [ 286.560580] [<c012a57e>] ? disable_nonboot_cpus+0x58/0xdc > >> [ 286.560580] [<c01360c0>] ? kernel_poweroff+0x22/0x39 > >> [ 286.560580] [<c0136301>] ? sys_reboot+0xde/0x14c > >> [ 286.560580] [<c01331b2>] ? complete_signal+0x179/0x191 > >> [ 286.560580] [<c0133396>] ? send_signal+0x1cc/0x1e1 > >> [ 286.560580] [<c03de418>] ? _spin_unlock_irqrestore+0x2d/0x3c > >> [ 286.560580] [<c0133b65>] ? group_send_signal_info+0x58/0x61 > >> [ 286.560580] [<c0133b9e>] ? kill_pid_info+0x30/0x3a > >> [ 286.560580] [<c0133d49>] ? sys_kill+0x75/0x13a > >> [ 286.560580] [<c01a06cb>] ? mntput_no_expire+ox1f/0x101 > >> [ 286.560580] [<c019b3b3>] ? dput+0x1e/0x105 > >> [ 286.560580] [<c018ef87>] ? __fput+0x150/0x158 > >> [ 286.560580] [<c0157abf>] ? audit_syscall_entry+0x137/0x159 > >> [ 286.560580] [<c010329f>] ? sysenter_do_call+0x12/0x34 [...] > and offset of ->data is zero and WORK_DATA_INIT() expands to > ATOMIC_LONG_INIT(0) so looks to me like 'sm_work' is used after it has > been free'd. Perhaps stop_machine_destroy() was called before or in > parallel to __stop_machine()? So lets cc Heiko as well. I missed to convert disable_nonboot_cpus to stop_machine_create/destroy. So it's a use before-even-allocated bug. The patch below should hopefully fix it: Subject: [PATCH] cpu hotplug: add stop_machine_create/destroy to disable_nonboot_cpus From: Heiko Carstens <heiko.carstens@de.ibm.com> disable_nonboot_cpus calls directly _cpu_down. _cpu_down however relies on the in advanced created stop_machine kernel threads which should be created by the caller (like cpu_down does). So add the missing stop_machine_create/destroy calls to disable_nonboot_cpus as well. Fixes this bug: [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b [ 286.548940] IP: [<c0150ca4>] __stop_machine+0x88/0xe3 [ 286.550598] Oops: 0002 [#1] SMP [ 286.560580] Pid: 3273, comm: halt Not tainted (2.6.28-06127-g238c6d5 [ 286.560580] EIP: is at __stop_machine+0x88/0xe3 [ 286.560580] Process halt (pid: 3273, ti=f1a28000 task=f4530f30 [ 286.560580] Call Trace: [ 286.560580] [<c03d04e4>] ? _cpu_down+0x10f/0x234 [ 286.560580] [<c012a57e>] ? disable_nonboot_cpus+0x58/0xdc [ 286.560580] [<c01360c0>] ? kernel_poweroff+0x22/0x39 [ 286.560580] [<c0136301>] ? sys_reboot+0xde/0x14c [ 286.560580] [<c01331b2>] ? complete_signal+0x179/0x191 [ 286.560580] [<c0133396>] ? send_signal+0x1cc/0x1e1 [ 286.560580] [<c03de418>] ? _spin_unlock_irqrestore+0x2d/0x3c [ 286.560580] [<c0133b65>] ? group_send_signal_info+0x58/0x61 [ 286.560580] [<c0133b9e>] ? kill_pid_info+0x30/0x3a [ 286.560580] [<c0133d49>] ? sys_kill+0x75/0x13a [ 286.560580] [<c01a06cb>] ? mntput_no_expire+ox1f/0x101 [ 286.560580] [<c019b3b3>] ? dput+0x1e/0x105 [ 286.560580] [<c018ef87>] ? __fput+0x150/0x158 [ 286.560580] [<c0157abf>] ? audit_syscall_entry+0x137/0x159 [ 286.560580] [<c010329f>] ? sysenter_do_call+0x12/0x34 Reported-by: "Justin P. Mattock" <justinmattock@gmail.com> Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> --- kernel/cpu.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) Index: linux-2.6/kernel/cpu.c =================================================================== --- linux-2.6.orig/kernel/cpu.c +++ linux-2.6/kernel/cpu.c @@ -379,8 +379,11 @@ static cpumask_var_t frozen_cpus; int disable_nonboot_cpus(void) { - int cpu, first_cpu, error = 0; + int cpu, first_cpu, error; + error = stop_machine_create(); + if (error) + return error; cpu_maps_update_begin(); first_cpu = cpumask_first(cpu_online_mask); /* We take down all of the non-boot CPUs in one shot to avoid races @@ -409,6 +412,7 @@ int disable_nonboot_cpus(void) printk(KERN_ERR "Non-boot CPUs are not disabled\n"); } cpu_maps_update_done(); + stop_machine_destroy(); return error; } ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b 2009-01-07 9:15 ` Heiko Carstens @ 2009-01-07 9:19 ` Pekka Enberg 2009-01-07 11:36 ` Jeff Chua 0 siblings, 1 reply; 15+ messages in thread From: Pekka Enberg @ 2009-01-07 9:19 UTC (permalink / raw) To: Heiko Carstens; +Cc: Justin P. Mattock, linux-kernel, Rusty Russell On Wed, 2009-01-07 at 10:15 +0100, Heiko Carstens wrote: > I missed to convert disable_nonboot_cpus to > stop_machine_create/destroy. > So it's a use before-even-allocated bug. > > The patch below should hopefully fix it: > > Subject: [PATCH] cpu hotplug: add stop_machine_create/destroy to disable_nonboot_cpus > > From: Heiko Carstens <heiko.carstens@de.ibm.com> > > disable_nonboot_cpus calls directly _cpu_down. _cpu_down however relies on > the in advanced created stop_machine kernel threads which should be created > by the caller (like cpu_down does). > > So add the missing stop_machine_create/destroy calls to disable_nonboot_cpus > as well. > > Fixes this bug: > > [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b > [ 286.548940] IP: [<c0150ca4>] __stop_machine+0x88/0xe3 > [ 286.550598] Oops: 0002 [#1] SMP > [ 286.560580] Pid: 3273, comm: halt Not tainted (2.6.28-06127-g238c6d5 > [ 286.560580] EIP: is at __stop_machine+0x88/0xe3 > [ 286.560580] Process halt (pid: 3273, ti=f1a28000 task=f4530f30 > [ 286.560580] Call Trace: > [ 286.560580] [<c03d04e4>] ? _cpu_down+0x10f/0x234 > [ 286.560580] [<c012a57e>] ? disable_nonboot_cpus+0x58/0xdc > [ 286.560580] [<c01360c0>] ? kernel_poweroff+0x22/0x39 > [ 286.560580] [<c0136301>] ? sys_reboot+0xde/0x14c > [ 286.560580] [<c01331b2>] ? complete_signal+0x179/0x191 > [ 286.560580] [<c0133396>] ? send_signal+0x1cc/0x1e1 > [ 286.560580] [<c03de418>] ? _spin_unlock_irqrestore+0x2d/0x3c > [ 286.560580] [<c0133b65>] ? group_send_signal_info+0x58/0x61 > [ 286.560580] [<c0133b9e>] ? kill_pid_info+0x30/0x3a > [ 286.560580] [<c0133d49>] ? sys_kill+0x75/0x13a > [ 286.560580] [<c01a06cb>] ? mntput_no_expire+ox1f/0x101 > [ 286.560580] [<c019b3b3>] ? dput+0x1e/0x105 > [ 286.560580] [<c018ef87>] ? __fput+0x150/0x158 > [ 286.560580] [<c0157abf>] ? audit_syscall_entry+0x137/0x159 > [ 286.560580] [<c010329f>] ? sysenter_do_call+0x12/0x34 > > Reported-by: "Justin P. Mattock" <justinmattock@gmail.com> > Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Looks good to me! Reviewed-by: Pekka Enberg <penberg@cs.helsinki.fi> > --- > kernel/cpu.c | 6 +++++- > 1 file changed, 5 insertions(+), 1 deletion(-) > > Index: linux-2.6/kernel/cpu.c > =================================================================== > --- linux-2.6.orig/kernel/cpu.c > +++ linux-2.6/kernel/cpu.c > @@ -379,8 +379,11 @@ static cpumask_var_t frozen_cpus; > > int disable_nonboot_cpus(void) > { > - int cpu, first_cpu, error = 0; > + int cpu, first_cpu, error; > > + error = stop_machine_create(); > + if (error) > + return error; > cpu_maps_update_begin(); > first_cpu = cpumask_first(cpu_online_mask); > /* We take down all of the non-boot CPUs in one shot to avoid races > @@ -409,6 +412,7 @@ int disable_nonboot_cpus(void) > printk(KERN_ERR "Non-boot CPUs are not disabled\n"); > } > cpu_maps_update_done(); > + stop_machine_destroy(); > return error; > } > ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b 2009-01-07 9:19 ` Pekka Enberg @ 2009-01-07 11:36 ` Jeff Chua 2009-01-07 12:27 ` Heiko Carstens 0 siblings, 1 reply; 15+ messages in thread From: Jeff Chua @ 2009-01-07 11:36 UTC (permalink / raw) To: Pekka Enberg Cc: Heiko Carstens, Justin P. Mattock, linux-kernel, Rusty Russell On Wed, Jan 7, 2009 at 5:19 PM, Pekka Enberg <penberg@cs.helsinki.fi> wrote: > On Wed, 2009-01-07 at 10:15 +0100, Heiko Carstens wrote: >> I missed to convert disable_nonboot_cpus to >> stop_machine_create/destroy. >> So it's a use before-even-allocated bug. >> The patch below should hopefully fix it: >> Subject: [PATCH] cpu hotplug: add stop_machine_create/destroy to disable_nonboot_cpus > Looks good to me! This also fixes the suspend-to-ram/disk problem. Without it, the system will just hang. Thanks for the fix. Thanks, Jeff. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b 2009-01-07 11:36 ` Jeff Chua @ 2009-01-07 12:27 ` Heiko Carstens 2009-01-07 13:51 ` Jeff Chua 0 siblings, 1 reply; 15+ messages in thread From: Heiko Carstens @ 2009-01-07 12:27 UTC (permalink / raw) To: Jeff Chua; +Cc: Pekka Enberg, Justin P. Mattock, linux-kernel, Rusty Russell On Wed, Jan 07, 2009 at 07:36:57PM +0800, Jeff Chua wrote: > On Wed, Jan 7, 2009 at 5:19 PM, Pekka Enberg <penberg@cs.helsinki.fi> wrote: > > On Wed, 2009-01-07 at 10:15 +0100, Heiko Carstens wrote: > >> I missed to convert disable_nonboot_cpus to > >> stop_machine_create/destroy. > >> So it's a use before-even-allocated bug. > >> The patch below should hopefully fix it: > >> Subject: [PATCH] cpu hotplug: add stop_machine_create/destroy to disable_nonboot_cpus > > Looks good to me! > > This also fixes the suspend-to-ram/disk problem. Without it, the > system will just hang. > > Thanks for the fix. Did you also see the reboot problem and does the patch fix it for you? ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b 2009-01-07 12:27 ` Heiko Carstens @ 2009-01-07 13:51 ` Jeff Chua 2009-01-07 15:19 ` [PATCH] stop_machine/cpu hotplug: fix disable_nonboot_cpus Heiko Carstens 2009-01-07 15:28 ` [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b Justin P. Mattock 0 siblings, 2 replies; 15+ messages in thread From: Jeff Chua @ 2009-01-07 13:51 UTC (permalink / raw) To: Heiko Carstens Cc: Pekka Enberg, Justin P. Mattock, linux-kernel, Rusty Russell On Wed, Jan 7, 2009 at 8:27 PM, Heiko Carstens <heiko.carstens@de.ibm.com> wrote: > On Wed, Jan 07, 2009 at 07:36:57PM +0800, Jeff Chua wrote: >> On Wed, Jan 7, 2009 at 5:19 PM, Pekka Enberg <penberg@cs.helsinki.fi> wrote: >> > On Wed, 2009-01-07 at 10:15 +0100, Heiko Carstens wrote: >> >> I missed to convert disable_nonboot_cpus to >> >> stop_machine_create/destroy. >> >> So it's a use before-even-allocated bug. >> >> The patch below should hopefully fix it: >> >> Subject: [PATCH] cpu hotplug: add stop_machine_create/destroy to disable_nonboot_cpus >> > Looks good to me! >> >> This also fixes the suspend-to-ram/disk problem. Without it, the >> system will just hang. > Did you also see the reboot problem and does the patch fix it for you? I never had problem with rebooting. Just suspend hanging which is really annoying ... walking away and come back hours later and realized that the suspend is hanging and having to do a hard boot. Thanks, Jeff. ^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH] stop_machine/cpu hotplug: fix disable_nonboot_cpus 2009-01-07 13:51 ` Jeff Chua @ 2009-01-07 15:19 ` Heiko Carstens 2009-01-07 15:23 ` Ingo Molnar 2009-01-07 15:30 ` Frédéric Weisbecker 2009-01-07 15:28 ` [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b Justin P. Mattock 1 sibling, 2 replies; 15+ messages in thread From: Heiko Carstens @ 2009-01-07 15:19 UTC (permalink / raw) To: Linus Torvalds, Andrew Morton, Rusty Russell Cc: Pekka Enberg, Justin P. Mattock, linux-kernel, Jeff Chua From: Heiko Carstens <heiko.carstens@de.ibm.com> disable_nonboot_cpus calls _cpu_down. But _cpu_down requires that the caller already created the stop_machine workqueue (like cpu_down does). Otherwise a call to stop_machine will lead to accesses to random memory regions. When introducing this new interface (9ea09af3bd3090e8349ca2899ca2011bd94cda85 "stop_machine: introduce stop_machine_create/destroy") I missed the second call site of _cpu_down. So add the missing stop_machine_create/destroy calls to disable_nonboot_cpus as well. Fixes suspend-to-ram/disk and also this bug: [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b [ 286.548940] IP: [<c0150ca4>] __stop_machine+0x88/0xe3 [ 286.550598] Oops: 0002 [#1] SMP [ 286.560580] Pid: 3273, comm: halt Not tainted (2.6.28-06127-g238c6d5 [ 286.560580] EIP: is at __stop_machine+0x88/0xe3 [ 286.560580] Process halt (pid: 3273, ti=f1a28000 task=f4530f30 [ 286.560580] Call Trace: [ 286.560580] [<c03d04e4>] ? _cpu_down+0x10f/0x234 [ 286.560580] [<c012a57e>] ? disable_nonboot_cpus+0x58/0xdc [ 286.560580] [<c01360c0>] ? kernel_poweroff+0x22/0x39 [ 286.560580] [<c0136301>] ? sys_reboot+0xde/0x14c [ 286.560580] [<c01331b2>] ? complete_signal+0x179/0x191 [ 286.560580] [<c0133396>] ? send_signal+0x1cc/0x1e1 [ 286.560580] [<c03de418>] ? _spin_unlock_irqrestore+0x2d/0x3c [ 286.560580] [<c0133b65>] ? group_send_signal_info+0x58/0x61 [ 286.560580] [<c0133b9e>] ? kill_pid_info+0x30/0x3a [ 286.560580] [<c0133d49>] ? sys_kill+0x75/0x13a [ 286.560580] [<c01a06cb>] ? mntput_no_expire+ox1f/0x101 [ 286.560580] [<c019b3b3>] ? dput+0x1e/0x105 [ 286.560580] [<c018ef87>] ? __fput+0x150/0x158 [ 286.560580] [<c0157abf>] ? audit_syscall_entry+0x137/0x159 [ 286.560580] [<c010329f>] ? sysenter_do_call+0x12/0x34 Reported-by: "Justin P. Mattock" <justinmattock@gmail.com> Reviewed-by: Pekka Enberg <penberg@cs.helsinki.fi> Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> --- kernel/cpu.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) Index: linux-2.6/kernel/cpu.c =================================================================== --- linux-2.6.orig/kernel/cpu.c +++ linux-2.6/kernel/cpu.c @@ -379,8 +379,11 @@ static cpumask_var_t frozen_cpus; int disable_nonboot_cpus(void) { - int cpu, first_cpu, error = 0; + int cpu, first_cpu, error; + error = stop_machine_create(); + if (error) + return error; cpu_maps_update_begin(); first_cpu = cpumask_first(cpu_online_mask); /* We take down all of the non-boot CPUs in one shot to avoid races @@ -409,6 +412,7 @@ int disable_nonboot_cpus(void) printk(KERN_ERR "Non-boot CPUs are not disabled\n"); } cpu_maps_update_done(); + stop_machine_destroy(); return error; } ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] stop_machine/cpu hotplug: fix disable_nonboot_cpus 2009-01-07 15:19 ` [PATCH] stop_machine/cpu hotplug: fix disable_nonboot_cpus Heiko Carstens @ 2009-01-07 15:23 ` Ingo Molnar 2009-01-07 15:30 ` Frédéric Weisbecker 1 sibling, 0 replies; 15+ messages in thread From: Ingo Molnar @ 2009-01-07 15:23 UTC (permalink / raw) To: Heiko Carstens Cc: Linus Torvalds, Andrew Morton, Rusty Russell, Pekka Enberg, Justin P. Mattock, linux-kernel, Jeff Chua * Heiko Carstens <heiko.carstens@de.ibm.com> wrote: > From: Heiko Carstens <heiko.carstens@de.ibm.com> > > disable_nonboot_cpus calls _cpu_down. But _cpu_down requires that the > caller already created the stop_machine workqueue (like cpu_down does). > Otherwise a call to stop_machine will lead to accesses to random memory > regions. btw., i got this crash earlier today: CPU0 attaching sched-domain: domain 0: span 0-1 level CPU groups: 0 1 CPU1 attaching sched-domain: domain 0: span 0-1 level CPU groups: 1 0 eth0: no IPv6 routers present BUG: Bad page state in process cc1 pfn:00879 page:c101b894 flags:00000400 count:0 mapcount:0 mapping:(null) index:0 Pid: 3060, comm: cc1 Not tainted 2.6.28-tip-07641-gb97d41d-dirty #14985 Call Trace: [<c016ce8b>] bad_page+0xcf/0xe5 [<c016d3b4>] free_pages_check+0xa7/0xc5 [<c016d400>] free_hot_cold_page+0x2e/0x138 [<c014751c>] ? __lock_acquire+0x127/0x29d [<c016d558>] free_hot_page+0xf/0x11 [<c0170963>] put_page+0x76/0x7c [<c0185071>] ? constant_test_bit+0x9/0x20 [<c0187149>] kfree+0x30/0xe5 [<c0164993>] ? trace_hardirqs_on+0x8/0x1c [<c01547dd>] free_user_ns+0x1d/0x20 [<c01547c0>] ? free_user_ns+0x0/0x20 [<c02c7a41>] kref_put+0x18/0x22 [<c0132d4c>] put_user_ns+0x16/0x18 [<c0132f52>] free_uid+0x59/0xc8 [<c0136239>] ? groups_free+0x36/0x3a [<c0140406>] put_cred_rcu+0x5f/0x70 [<c01598fb>] __rcu_process_callbacks+0x168/0x1f8 [<c03031be>] ? isicom_tx+0x0/0x31f [<c01599b1>] rcu_process_callbacks+0x26/0x46 [<c012f11d>] __do_softirq+0x9d/0x139 [<c012f080>] ? __do_softirq+0x0/0x139 <IRQ> [<c012efe2>] ? irq_exit+0x4c/0x83 [<c05cc586>] ? __irqentry_text_start+0x6e/0x7c [<c0103f61>] ? apic_timer_interrupt+0x2d/0x34 and i applied your patch (from the other thread) and never saw this bug again. So if it's the same bug (it appears to be) then you have my: Tested-by: Ingo Molnar <mingo@elte.hu> Ingo ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] stop_machine/cpu hotplug: fix disable_nonboot_cpus 2009-01-07 15:19 ` [PATCH] stop_machine/cpu hotplug: fix disable_nonboot_cpus Heiko Carstens 2009-01-07 15:23 ` Ingo Molnar @ 2009-01-07 15:30 ` Frédéric Weisbecker 2009-01-07 15:52 ` Justin P. Mattock 2009-01-08 5:13 ` Justin P. Mattock 1 sibling, 2 replies; 15+ messages in thread From: Frédéric Weisbecker @ 2009-01-07 15:30 UTC (permalink / raw) To: Heiko Carstens Cc: Linus Torvalds, Andrew Morton, Rusty Russell, Pekka Enberg, Justin P. Mattock, linux-kernel, Jeff Chua 2009/1/7 Heiko Carstens <heiko.carstens@de.ibm.com>: > From: Heiko Carstens <heiko.carstens@de.ibm.com> > > disable_nonboot_cpus calls _cpu_down. But _cpu_down requires that the > caller already created the stop_machine workqueue (like cpu_down does). > Otherwise a call to stop_machine will lead to accesses to random memory > regions. > > When introducing this new interface (9ea09af3bd3090e8349ca2899ca2011bd94cda85 > "stop_machine: introduce stop_machine_create/destroy") I missed the second > call site of _cpu_down. > So add the missing stop_machine_create/destroy calls to disable_nonboot_cpus > as well. > > Fixes suspend-to-ram/disk and also this bug: > > [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b > [ 286.548940] IP: [<c0150ca4>] __stop_machine+0x88/0xe3 > [ 286.550598] Oops: 0002 [#1] SMP > [ 286.560580] Pid: 3273, comm: halt Not tainted (2.6.28-06127-g238c6d5 > [ 286.560580] EIP: is at __stop_machine+0x88/0xe3 > [ 286.560580] Process halt (pid: 3273, ti=f1a28000 task=f4530f30 > [ 286.560580] Call Trace: > [ 286.560580] [<c03d04e4>] ? _cpu_down+0x10f/0x234 > [ 286.560580] [<c012a57e>] ? disable_nonboot_cpus+0x58/0xdc > [ 286.560580] [<c01360c0>] ? kernel_poweroff+0x22/0x39 > [ 286.560580] [<c0136301>] ? sys_reboot+0xde/0x14c > [ 286.560580] [<c01331b2>] ? complete_signal+0x179/0x191 > [ 286.560580] [<c0133396>] ? send_signal+0x1cc/0x1e1 > [ 286.560580] [<c03de418>] ? _spin_unlock_irqrestore+0x2d/0x3c > [ 286.560580] [<c0133b65>] ? group_send_signal_info+0x58/0x61 > [ 286.560580] [<c0133b9e>] ? kill_pid_info+0x30/0x3a > [ 286.560580] [<c0133d49>] ? sys_kill+0x75/0x13a > [ 286.560580] [<c01a06cb>] ? mntput_no_expire+ox1f/0x101 > [ 286.560580] [<c019b3b3>] ? dput+0x1e/0x105 > [ 286.560580] [<c018ef87>] ? __fput+0x150/0x158 > [ 286.560580] [<c0157abf>] ? audit_syscall_entry+0x137/0x159 > [ 286.560580] [<c010329f>] ? sysenter_do_call+0x12/0x34 > > Reported-by: "Justin P. Mattock" <justinmattock@gmail.com> > Reviewed-by: Pekka Enberg <penberg@cs.helsinki.fi> > Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> > --- > kernel/cpu.c | 6 +++++- > 1 file changed, 5 insertions(+), 1 deletion(-) > > Index: linux-2.6/kernel/cpu.c > =================================================================== > --- linux-2.6.orig/kernel/cpu.c > +++ linux-2.6/kernel/cpu.c > @@ -379,8 +379,11 @@ static cpumask_var_t frozen_cpus; > > int disable_nonboot_cpus(void) > { > - int cpu, first_cpu, error = 0; > + int cpu, first_cpu, error; > > + error = stop_machine_create(); > + if (error) > + return error; > cpu_maps_update_begin(); > first_cpu = cpumask_first(cpu_online_mask); > /* We take down all of the non-boot CPUs in one shot to avoid races > @@ -409,6 +412,7 @@ int disable_nonboot_cpus(void) > printk(KERN_ERR "Non-boot CPUs are not disabled\n"); > } > cpu_maps_update_done(); > + stop_machine_destroy(); > return error; > } > That should explain why suspend to disk failed on my box yesterday on the processors stage... Thanks! ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] stop_machine/cpu hotplug: fix disable_nonboot_cpus 2009-01-07 15:30 ` Frédéric Weisbecker @ 2009-01-07 15:52 ` Justin P. Mattock 2009-01-08 5:13 ` Justin P. Mattock 1 sibling, 0 replies; 15+ messages in thread From: Justin P. Mattock @ 2009-01-07 15:52 UTC (permalink / raw) To: Frédéric Weisbecker Cc: Heiko Carstens, Linus Torvalds, Andrew Morton, Rusty Russell, Pekka Enberg, linux-kernel, Jeff Chua Frédéric Weisbecker wrote: > 2009/1/7 Heiko Carstens <heiko.carstens@de.ibm.com>: > >> From: Heiko Carstens <heiko.carstens@de.ibm.com> >> >> disable_nonboot_cpus calls _cpu_down. But _cpu_down requires that the >> caller already created the stop_machine workqueue (like cpu_down does). >> Otherwise a call to stop_machine will lead to accesses to random memory >> regions. >> >> When introducing this new interface (9ea09af3bd3090e8349ca2899ca2011bd94cda85 >> "stop_machine: introduce stop_machine_create/destroy") I missed the second >> call site of _cpu_down. >> So add the missing stop_machine_create/destroy calls to disable_nonboot_cpus >> as well. >> >> Fixes suspend-to-ram/disk and also this bug: >> >> [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b >> [ 286.548940] IP: [<c0150ca4>] __stop_machine+0x88/0xe3 >> [ 286.550598] Oops: 0002 [#1] SMP >> [ 286.560580] Pid: 3273, comm: halt Not tainted (2.6.28-06127-g238c6d5 >> [ 286.560580] EIP: is at __stop_machine+0x88/0xe3 >> [ 286.560580] Process halt (pid: 3273, ti=f1a28000 task=f4530f30 >> [ 286.560580] Call Trace: >> [ 286.560580] [<c03d04e4>] ? _cpu_down+0x10f/0x234 >> [ 286.560580] [<c012a57e>] ? disable_nonboot_cpus+0x58/0xdc >> [ 286.560580] [<c01360c0>] ? kernel_poweroff+0x22/0x39 >> [ 286.560580] [<c0136301>] ? sys_reboot+0xde/0x14c >> [ 286.560580] [<c01331b2>] ? complete_signal+0x179/0x191 >> [ 286.560580] [<c0133396>] ? send_signal+0x1cc/0x1e1 >> [ 286.560580] [<c03de418>] ? _spin_unlock_irqrestore+0x2d/0x3c >> [ 286.560580] [<c0133b65>] ? group_send_signal_info+0x58/0x61 >> [ 286.560580] [<c0133b9e>] ? kill_pid_info+0x30/0x3a >> [ 286.560580] [<c0133d49>] ? sys_kill+0x75/0x13a >> [ 286.560580] [<c01a06cb>] ? mntput_no_expire+ox1f/0x101 >> [ 286.560580] [<c019b3b3>] ? dput+0x1e/0x105 >> [ 286.560580] [<c018ef87>] ? __fput+0x150/0x158 >> [ 286.560580] [<c0157abf>] ? audit_syscall_entry+0x137/0x159 >> [ 286.560580] [<c010329f>] ? sysenter_do_call+0x12/0x34 >> >> Reported-by: "Justin P. Mattock" <justinmattock@gmail.com> >> Reviewed-by: Pekka Enberg <penberg@cs.helsinki.fi> >> Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> >> --- >> kernel/cpu.c | 6 +++++- >> 1 file changed, 5 insertions(+), 1 deletion(-) >> >> Index: linux-2.6/kernel/cpu.c >> =================================================================== >> --- linux-2.6.orig/kernel/cpu.c >> +++ linux-2.6/kernel/cpu.c >> @@ -379,8 +379,11 @@ static cpumask_var_t frozen_cpus; >> >> int disable_nonboot_cpus(void) >> { >> - int cpu, first_cpu, error = 0; >> + int cpu, first_cpu, error; >> >> + error = stop_machine_create(); >> + if (error) >> + return error; >> cpu_maps_update_begin(); >> first_cpu = cpumask_first(cpu_online_mask); >> /* We take down all of the non-boot CPUs in one shot to avoid races >> @@ -409,6 +412,7 @@ int disable_nonboot_cpus(void) >> printk(KERN_ERR "Non-boot CPUs are not disabled\n"); >> } >> cpu_maps_update_done(); >> + stop_machine_destroy(); >> return error; >> } >> >> > > > That should explain why suspend to disk failed on my box yesterday on > the processors stage... > Thanks! > > O.K. applied the patch, and shutdown the machine a few times; no freeze, no bug message. sweet!!. Now I'm gonna try and dismantle a bug message for educational purposes. Thanks for the assistance. regards; Justin P. Mattock ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] stop_machine/cpu hotplug: fix disable_nonboot_cpus 2009-01-07 15:30 ` Frédéric Weisbecker 2009-01-07 15:52 ` Justin P. Mattock @ 2009-01-08 5:13 ` Justin P. Mattock 1 sibling, 0 replies; 15+ messages in thread From: Justin P. Mattock @ 2009-01-08 5:13 UTC (permalink / raw) To: Frédéric Weisbecker Cc: Heiko Carstens, Linus Torvalds, Andrew Morton, Rusty Russell, Pekka Enberg, linux-kernel, Jeff Chua Frédéric Weisbecker wrote: > 2009/1/7 Heiko Carstens <heiko.carstens@de.ibm.com>: > >> From: Heiko Carstens <heiko.carstens@de.ibm.com> >> >> disable_nonboot_cpus calls _cpu_down. But _cpu_down requires that the >> caller already created the stop_machine workqueue (like cpu_down does). >> Otherwise a call to stop_machine will lead to accesses to random memory >> regions. >> >> When introducing this new interface (9ea09af3bd3090e8349ca2899ca2011bd94cda85 >> "stop_machine: introduce stop_machine_create/destroy") I missed the second >> call site of _cpu_down. >> So add the missing stop_machine_create/destroy calls to disable_nonboot_cpus >> as well. >> >> Fixes suspend-to-ram/disk and also this bug: >> >> [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b >> [ 286.548940] IP: [<c0150ca4>] __stop_machine+0x88/0xe3 >> [ 286.550598] Oops: 0002 [#1] SMP >> [ 286.560580] Pid: 3273, comm: halt Not tainted (2.6.28-06127-g238c6d5 >> [ 286.560580] EIP: is at __stop_machine+0x88/0xe3 >> [ 286.560580] Process halt (pid: 3273, ti=f1a28000 task=f4530f30 >> [ 286.560580] Call Trace: >> [ 286.560580] [<c03d04e4>] ? _cpu_down+0x10f/0x234 >> [ 286.560580] [<c012a57e>] ? disable_nonboot_cpus+0x58/0xdc >> [ 286.560580] [<c01360c0>] ? kernel_poweroff+0x22/0x39 >> [ 286.560580] [<c0136301>] ? sys_reboot+0xde/0x14c >> [ 286.560580] [<c01331b2>] ? complete_signal+0x179/0x191 >> [ 286.560580] [<c0133396>] ? send_signal+0x1cc/0x1e1 >> [ 286.560580] [<c03de418>] ? _spin_unlock_irqrestore+0x2d/0x3c >> [ 286.560580] [<c0133b65>] ? group_send_signal_info+0x58/0x61 >> [ 286.560580] [<c0133b9e>] ? kill_pid_info+0x30/0x3a >> [ 286.560580] [<c0133d49>] ? sys_kill+0x75/0x13a >> [ 286.560580] [<c01a06cb>] ? mntput_no_expire+ox1f/0x101 >> [ 286.560580] [<c019b3b3>] ? dput+0x1e/0x105 >> [ 286.560580] [<c018ef87>] ? __fput+0x150/0x158 >> [ 286.560580] [<c0157abf>] ? audit_syscall_entry+0x137/0x159 >> [ 286.560580] [<c010329f>] ? sysenter_do_call+0x12/0x34 >> >> Reported-by: "Justin P. Mattock" <justinmattock@gmail.com> >> Reviewed-by: Pekka Enberg <penberg@cs.helsinki.fi> >> Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> >> --- >> kernel/cpu.c | 6 +++++- >> 1 file changed, 5 insertions(+), 1 deletion(-) >> >> Index: linux-2.6/kernel/cpu.c >> =================================================================== >> --- linux-2.6.orig/kernel/cpu.c >> +++ linux-2.6/kernel/cpu.c >> @@ -379,8 +379,11 @@ static cpumask_var_t frozen_cpus; >> >> int disable_nonboot_cpus(void) >> { >> - int cpu, first_cpu, error = 0; >> + int cpu, first_cpu, error; >> >> + error = stop_machine_create(); >> + if (error) >> + return error; >> cpu_maps_update_begin(); >> first_cpu = cpumask_first(cpu_online_mask); >> /* We take down all of the non-boot CPUs in one shot to avoid races >> @@ -409,6 +412,7 @@ int disable_nonboot_cpus(void) >> printk(KERN_ERR "Non-boot CPUs are not disabled\n"); >> } >> cpu_maps_update_done(); >> + stop_machine_destroy(); >> return error; >> } >> >> > > > That should explain why suspend to disk failed on my box yesterday on > the processors stage... > Thanks! > > I hate to ask this, but I'm going to anyway: when running gdb /usr/src/linux/vmlinux (hoping to see if gdb will catch the bug); I keep getting: Program terminated with signal SIGKILL, Killed. The program no longer exists. You can't do that without a process to debug. if i do a: (gdb) disassemble __stop_machine (as described in Documentation); I'll see a bit of info. How do I start/or figure out a process to debug? i.g. under the bug message that I wrote down, it says Pid: 3273 entering that in (gdb) r 3273 results in a SIGKILL. regards; Justin P. Mattock ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b 2009-01-07 13:51 ` Jeff Chua 2009-01-07 15:19 ` [PATCH] stop_machine/cpu hotplug: fix disable_nonboot_cpus Heiko Carstens @ 2009-01-07 15:28 ` Justin P. Mattock 1 sibling, 0 replies; 15+ messages in thread From: Justin P. Mattock @ 2009-01-07 15:28 UTC (permalink / raw) To: Jeff Chua; +Cc: Heiko Carstens, Pekka Enberg, linux-kernel, Rusty Russell Jeff Chua wrote: > On Wed, Jan 7, 2009 at 8:27 PM, Heiko Carstens > <heiko.carstens@de.ibm.com> wrote: > >> On Wed, Jan 07, 2009 at 07:36:57PM +0800, Jeff Chua wrote: >> >>> On Wed, Jan 7, 2009 at 5:19 PM, Pekka Enberg <penberg@cs.helsinki.fi> wrote: >>> >>>> On Wed, 2009-01-07 at 10:15 +0100, Heiko Carstens wrote: >>>> >>>>> I missed to convert disable_nonboot_cpus to >>>>> stop_machine_create/destroy. >>>>> So it's a use before-even-allocated bug. >>>>> The patch below should hopefully fix it: >>>>> Subject: [PATCH] cpu hotplug: add stop_machine_create/destroy to disable_nonboot_cpus >>>>> >>>> Looks good to me! >>>> >>> This also fixes the suspend-to-ram/disk problem. Without it, the >>> system will just hang. >>> > > >> Did you also see the reboot problem and does the patch fix it for you? >> > > I never had problem with rebooting. Just suspend hanging which is > really annoying ... walking away and come back hours later and > realized that the suspend is hanging and having to do a hard boot. > > Thanks, > Jeff. > > Man!! I missed this whole conversation (passed out, too tired); I'll go ahead and apply the patch and let you know If I get the freeze at shutdown. Then I'm still interested in knowing how to take a debug messages and dissect it to find the exact location of the problem. (but could do that later); regards; Justin P. Mattock ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2009-01-08 5:13 UTC | newest] Thread overview: 15+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2009-01-07 0:12 [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b Justin P. Mattock 2009-01-07 6:48 ` Pekka Enberg 2009-01-07 8:13 ` Justin P. Mattock 2009-01-07 8:30 ` Pekka Enberg 2009-01-07 9:15 ` Heiko Carstens 2009-01-07 9:19 ` Pekka Enberg 2009-01-07 11:36 ` Jeff Chua 2009-01-07 12:27 ` Heiko Carstens 2009-01-07 13:51 ` Jeff Chua 2009-01-07 15:19 ` [PATCH] stop_machine/cpu hotplug: fix disable_nonboot_cpus Heiko Carstens 2009-01-07 15:23 ` Ingo Molnar 2009-01-07 15:30 ` Frédéric Weisbecker 2009-01-07 15:52 ` Justin P. Mattock 2009-01-08 5:13 ` Justin P. Mattock 2009-01-07 15:28 ` [ 286.547348] BUG: unable to handle kernel paging request at 6b6b6b6b Justin P. Mattock
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.