* kexec: purgatory hang
@ 2013-06-11 22:54 Cliff Wickman
2013-06-12 1:24 ` Eric W. Biederman
2013-06-12 1:45 ` Yinghai Lu
0 siblings, 2 replies; 8+ messages in thread
From: Cliff Wickman @ 2013-06-11 22:54 UTC (permalink / raw)
To: kexec; +Cc: yinghai, ebiederm
I'm getting a hang when trying to enter a high-memory crash kernel,
and I'm at a loss as to how to debug this.
This is a 3.10.0-rc3 kernel, and set up as the crash kernel by kexec 2.0.4.
The machine is an SGI UV1000.
[ 164.027275] SysRq : Trigger a crash
[ 164.031136] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 164.031136] IP: [<ffffffff81397771>] sysrq_handle_crash+0x11/0x20
[ 164.031136] PGD 1fbe835067 PUD 1fbc2e8067 PMD 0
[ 164.031136] Oops: 0002 [#1] SMP
[ 164.031136] xpc : all partitions have deactivated
[ 164.031136] Modules linked in: autofs4 binfmt_misc af_packet rdma_ucm rdma_cm iw_cm ib_addr ib_srp scsi_transport_srp scsi_tgt ib_ipoib ib_cm ib_uverbs ib_umad iw_cxgb3 cxgb3 mdio mlx4_en mlx4_ib ib_sa mlx4_core ib_mthca ib_mad ib_core fuse nls_iso8859_1 nls_cp437 vfat fat loop uv_mmtimer dm_mod sr_mod cdrom usb_storage iTCO_wdt iTCO_vendor_support coretemp mperf kvm_intel ipv6 kvm igb sg crc32c_intel lpc_ich pcspkr mptctl i2c_algo_bit ptp i2c_i801 microcode xhci_hcd joydev ioatdma ehci_pci hid_generic pps_core i2c_core rtc_cmos mfd_core button dca usbhid hid uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif scsi_dh_hp_sw scsi_dh_alua scsi_dh_emc scsi_dh_rdac scsi_dh thermal sata_nv processor piix mptsas mptscsih scsi_transport_sas mptbase megaraid_sas fan thermal_sys hwmon ext3 jbd ata_piix ahci libahci libata scsi_mod
[ 164.031136] CPU: 10 PID: 9299 Comm: dopanic Not tainted 3.10.0-rc3-linus-cpw+ #17
[ 164.031136] Hardware name: Intel Corp. Stoutland Platform, BIOS 2.16 UEFI2.10 PI1.0 X64 2012-04-27
[ 164.031136] task: ffff88203df94440 ti: ffff88203d5c2000 task.ti: ffff88203d5c2000
[ 164.031136] RIP: 0010:[<ffffffff81397771>] [<ffffffff81397771>] sysrq_handle_crash+0x11/0x20
[ 164.031136] RSP: 0018:ffff88203d5c3e68 EFLAGS: 00010092
[ 164.031136] RAX: 000000000000000f RBX: ffffffff81a974e0 RCX: 0000000000000004
[ 164.031136] RDX: 0000000000000000 RSI: ffff881fffd0ef48 RDI: 0000000000000063
[ 164.031136] RBP: ffff88203d5c3e68 R08: ffff881fffd0d3e8 R09: 000000000004268c
[ 164.031136] R10: 0000000000000b8b R11: 0000000000000000 R12: 0000000000000063
[ 164.031136] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000296
[ 164.031136] FS: 00007ffff7fb5700(0000) GS:ffff881fffd00000(0000) knlGS:0000000000000000
[ 164.031136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 164.031136] CR2: 0000000000000000 CR3: 0000001fbea6c000 CR4: 00000000000007e0
[ 164.031136] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 164.031136] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 164.031136] Stack:
[ 164.031136] ffff88203d5c3ea8 ffffffff81398008 01ff88203d5c3e88 0000000000000002
[ 164.031136] ffff895f9d478380 ffff88203d5c3f40 00007ffff7ff8000 ffff88203d5c3f40
[ 164.031136] ffff88203d5c3ec8 ffffffff813980ad ffff88203d5c3ee8 fffffffffffffffb
[ 164.031136] Call Trace:
[ 164.031136] [<ffffffff81398008>] __handle_sysrq+0x128/0x190
[ 164.031136] [<ffffffff813980ad>] write_sysrq_trigger+0x3d/0x40
[ 164.031136] [<ffffffff811c323f>] proc_reg_write+0x4f/0x80
[ 164.031136] [<ffffffff8115f107>] vfs_write+0xe7/0x190
[ 164.031136] [<ffffffff8115f8ec>] SyS_write+0x5c/0xa0
[ 164.031136] [<ffffffff8153c092>] system_call_fastpath+0x16/0x1b
[ 164.031136] Code: 00 48 8b 75 e8 48 81 c7 08 08 00 00 e8 09 c6 19 00 31 d2 eb 95 90 90 90 90 90 55 c7 05 f5 74 96 00 01 00 00 00 48 89 e5 0f ae f8 <c6> 04 25 00 00 00 00 01 c9 c3 0f 1f 44 00 00 8d 47 d0 55 83 f8
[ 164.031136] RIP [<ffffffff81397771>] sysrq_handle_crash+0x11/0x20
[ 164.031136] RSP <ffff88203d5c3e68>
[ 164.031136] CR2: 0000000000000000
This is always the last output.
Can anyone suggest any way to debug this problem?
I suppose I can hang the processor just before it executes machine_kexec()
and look at it with crash. Any suggestions as to what to look at?
----------------------------------------------------------------------
The problem does not occur if the crash kernel is loaded below 4G, or if
the machine is an SGI UV2000.
It works as below when loaded into low memory:
[ 517.278643] SysRq : Trigger a crash
[ 517.282507] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 517.282507] IP: [<ffffffff81397771>] sysrq_handle_crash+0x11/0x20
[ 517.282507] PGD 203e209067 PUD 203e6dc067 PMD 0
[ 517.282507] Oops: 0002 [#1] SMP
[ 517.282507] xpc : all partitions have deactivated
[ 517.282507] Modules linked in: autofs4 binfmt_misc af_packet rdma_ucm rdma_cm iw_cm ib_addr ib_srp scsi_transport_srp scsi_tgt ib_ipoib ib_cm ib_uverbs ib_umad iw_cxgb3 cxgb3 mdio mlx4_en mlx4_ib ib_sa mlx4_core ib_mthca ib_mad ib_core fuse nls_iso8859_1 nls_cp437 vfat fat loop sr_mod cdrom uv_mmtimer dm_mod usb_storage coretemp mperf kvm_intel kvm crc32c_intel ipv6 microcode igb ioatdma joydev i2c_algo_bit ptp xhci_hcd hid_generic sg iTCO_wdt i2c_i801 pcspkr i2c_core iTCO_vendor_support ehci_pci lpc_ich pps_core mfd_core dca mptctl button rtc_cmos usbhid hid uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif scsi_dh_hp_sw scsi_dh_alua scsi_dh_emc scsi_dh_rdac scsi_dh thermal sata_nv processor piix mptsas mptscsih scsi_transport_sas mptbase megaraid_sas fan thermal_sys hwmon ext3 jbd ata_piix ahci libahci libata scsi_mod
[ 517.282507] CPU: 4 PID: 9989 Comm: dopanic Not tainted 3.10.0-rc3-linus-cpw+ #17
[ 517.282507] Hardware name: Intel Corp. Stoutland Platform, BIOS 2.16 UEFI2.10 PI1.0 X64 2012-04-27
[ 517.282507] task: ffff881fbe55a300 ti: ffff881fbc7fc000 task.ti: ffff881fbc7fc000
[ 517.282507] RIP: 0010:[<ffffffff81397771>] [<ffffffff81397771>] sysrq_handle_crash+0x11/0x20
[ 517.282507] RSP: 0018:ffff881fbc7fde68 EFLAGS: 00010092
[ 517.282507] RAX: 000000000000000f RBX: ffffffff81a974e0 RCX: 0000000000000004
[ 517.282507] RDX: 0000000000000000 RSI: ffff88207fd0ef48 RDI: 0000000000000063
[ 517.282507] RBP: ffff881fbc7fde68 R08: ffff88207fd0d3e8 R09: 0000000000042424
[ 517.282507] R10: 0000000000000b83 R11: 0000000000000000 R12: 0000000000000063
[ 517.282507] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000296
[ 517.282507] FS: 00007ffff7fb5700(0000) GS:ffff88207fd00000(0000) knlGS:0000000000000000
[ 517.282507] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 517.282507] CR2: 0000000000000000 CR3: 000000203dbea000 CR4: 00000000000007e0
[ 517.282507] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 517.282507] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 517.282507] Stack:
[ 517.282507] ffff881fbc7fdea8 ffffffff81398008 01ff881fbc7fde88 0000000000000002
[ 517.282507] ffff891fbdaea6c0 ffff881fbc7fdf40 00007ffff7ff8000 ffff881fbc7fdf40
[ 517.282507] ffff881fbc7fdec8 ffffffff813980ad ffff881fbc7fdee8 fffffffffffffffb
[ 517.282507] Call Trace:
[ 517.282507] [<ffffffff81398008>] __handle_sysrq+0x128/0x190
[ 517.282507] [<ffffffff813980ad>] write_sysrq_trigger+0x3d/0x40
[ 517.282507] [<ffffffff811c323f>] proc_reg_write+0x4f/0x80
[ 517.282507] [<ffffffff8115f107>] vfs_write+0xe7/0x190
[ 517.282507] [<ffffffff8115f8ec>] SyS_write+0x5c/0xa0
[ 517.282507] [<ffffffff8153c092>] system_call_fastpath+0x16/0x1b
[ 517.282507] Code: 00 48 8b 75 e8 48 81 c7 08 08 00 00 e8 09 c6 19 00 31 d2 eb 95 90 90 90 90 90 55 c7 05 f5 74 96 00 01 00 00 00 48 89 e5 0f ae f8 <c6> 04 25 00 00 00 00 01 c9 c3 0f 1f 44 00 00 8d 47 d0 55 83 f8
[ 517.282507] RIP [<ffffffff81397771>] sysrq_handle_crash+0x11/0x20
[ 517.282507] RSP <ffff881fbc7fde68>
[ 517.282507] CR2: 0000000000000000
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 3.10.0-rc3-linus-cpw+ (cpw@gulag1) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #17 SMP Fri Jun 7 10:47:45 CDT 2013
[ 0.000000] Command line: root=LABEL=uv21-sysR13 kdb=on pcie_aspm=on add_efi_memmap cgroup_disable=memory earlyprintk=ttyS0,115200n8 log_buf_len=8M processor.max_cstate=1 stop_machine.lazy=1 nobau console=ttyS0,115200n8 rcutree.rcu_cpu_stall_suppress=1 nortsched cpuidle_sysfs_switch ipmi_si.trydefaults=0 intel_idle.max_cstate=0 nmi_watchdog=0 pci=hpiosize=0,hpmemsize=0,nobar udev.children_max=128 skew_tick=1 relax_domain_level=2 nohz=off highres=off elevator=deadline sysrq=yes reset_devices irqpoll maxcpus=1 noefi acpi_rsdp=0x78d30014 memmap=exactmap memmap=568K@4K memmap=523696K@393216K acpi_rsdp=0x78d30014 elfcorehdr=916912K memmap=4K$0K memmap=4K#572K memmap=4K$1926652K memmap=4K$1926660K memmap=36K$1929652K memmap=8K$1930380K memmap=12K$1930392K memmap=48K$1937884K memmap=8K$1977476K memmap=8K$1977492K memmap=152K$1977540K memmap=64K$1977732K memmap=320K$1978116K memmap=396K#1978436K memmap=468K#1978832K memmap=160K#1979300K memmap=128K#1979460K memmap=4K$2045576K mem
map=512K$20456[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x0000000000000fff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000001000-0x000000000008efff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000008f000-0x000000000008ffff] ACPI NVS
[ 0.000000] BIOS-e820: [mem 0x0000000000090000-0x000000000009ffff] usable
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000075958fff] usable
[ 0.000000] BIOS-e820: [mem 0x000000007597d000-0x000000007597efff] usable
[ 0.000000] BIOS-e820: [mem 0x000000007597f000-0x000000007597ffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000075980000-0x0000000075980fff] usable
[ 0.000000] BIOS-e820: [mem 0x0000000075981000-0x0000000075981fff] reserved
...
Thanks for any suggestions.
-Cliff Wickman
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kexec: purgatory hang
2013-06-11 22:54 kexec: purgatory hang Cliff Wickman
@ 2013-06-12 1:24 ` Eric W. Biederman
2013-06-12 16:40 ` Cliff Wickman
2013-06-12 1:45 ` Yinghai Lu
1 sibling, 1 reply; 8+ messages in thread
From: Eric W. Biederman @ 2013-06-12 1:24 UTC (permalink / raw)
To: Cliff Wickman; +Cc: yinghai, kexec
Cliff Wickman <cpw@sgi.com> writes:
> I'm getting a hang when trying to enter a high-memory crash kernel,
> and I'm at a loss as to how to debug this.
>
> This is a 3.10.0-rc3 kernel, and set up as the crash kernel by kexec 2.0.4.
> The machine is an SGI UV1000.
>
> [ 164.027275] SysRq : Trigger a crash
> [ 164.031136] BUG: unable to handle kernel NULL pointer dereference at (null)
> [ 164.031136] IP: [<ffffffff81397771>] sysrq_handle_crash+0x11/0x20
> [ 164.031136] PGD 1fbe835067 PUD 1fbc2e8067 PMD 0
> [ 164.031136] Oops: 0002 [#1] SMP
> [ 164.031136] xpc : all partitions have deactivated
> [ 164.031136] Modules linked in: autofs4 binfmt_misc af_packet rdma_ucm rdma_cm iw_cm ib_addr ib_srp scsi_transport_srp scsi_tgt ib_ipoib ib_cm ib_uverbs ib_umad iw_cxgb3 cxgb3 mdio mlx4_en mlx4_ib ib_sa mlx4_core ib_mthca ib_mad ib_core fuse nls_iso8859_1 nls_cp437 vfat fat loop uv_mmtimer dm_mod sr_mod cdrom usb_storage iTCO_wdt iTCO_vendor_support coretemp mperf kvm_intel ipv6 kvm igb sg crc32c_intel lpc_ich pcspkr mptctl i2c_algo_bit ptp i2c_i801 microcode xhci_hcd joydev ioatdma ehci_pci hid_generic pps_core i2c_core rtc_cmos mfd_core button dca usbhid hid uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif scsi_dh_hp_sw scsi_dh_alua scsi_dh_emc scsi_dh_rdac scsi_dh thermal sata_nv processor piix mptsas mptscsih scsi_transport_sas mptbase megaraid_sas fan thermal_sys hwmon ext3 jbd ata_piix ahci libahci libata scsi_mod
> [ 164.031136] CPU: 10 PID: 9299 Comm: dopanic Not tainted 3.10.0-rc3-linus-cpw+ #17
> [ 164.031136] Hardware name: Intel Corp. Stoutland Platform, BIOS 2.16 UEFI2.10 PI1.0 X64 2012-04-27
> [ 164.031136] task: ffff88203df94440 ti: ffff88203d5c2000 task.ti: ffff88203d5c2000
> [ 164.031136] RIP: 0010:[<ffffffff81397771>] [<ffffffff81397771>] sysrq_handle_crash+0x11/0x20
> [ 164.031136] RSP: 0018:ffff88203d5c3e68 EFLAGS: 00010092
> [ 164.031136] RAX: 000000000000000f RBX: ffffffff81a974e0 RCX: 0000000000000004
> [ 164.031136] RDX: 0000000000000000 RSI: ffff881fffd0ef48 RDI: 0000000000000063
> [ 164.031136] RBP: ffff88203d5c3e68 R08: ffff881fffd0d3e8 R09: 000000000004268c
> [ 164.031136] R10: 0000000000000b8b R11: 0000000000000000 R12: 0000000000000063
> [ 164.031136] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000296
> [ 164.031136] FS: 00007ffff7fb5700(0000) GS:ffff881fffd00000(0000) knlGS:0000000000000000
> [ 164.031136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 164.031136] CR2: 0000000000000000 CR3: 0000001fbea6c000 CR4: 00000000000007e0
> [ 164.031136] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 164.031136] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 164.031136] Stack:
> [ 164.031136] ffff88203d5c3ea8 ffffffff81398008 01ff88203d5c3e88 0000000000000002
> [ 164.031136] ffff895f9d478380 ffff88203d5c3f40 00007ffff7ff8000 ffff88203d5c3f40
> [ 164.031136] ffff88203d5c3ec8 ffffffff813980ad ffff88203d5c3ee8 fffffffffffffffb
> [ 164.031136] Call Trace:
> [ 164.031136] [<ffffffff81398008>] __handle_sysrq+0x128/0x190
> [ 164.031136] [<ffffffff813980ad>] write_sysrq_trigger+0x3d/0x40
> [ 164.031136] [<ffffffff811c323f>] proc_reg_write+0x4f/0x80
> [ 164.031136] [<ffffffff8115f107>] vfs_write+0xe7/0x190
> [ 164.031136] [<ffffffff8115f8ec>] SyS_write+0x5c/0xa0
> [ 164.031136] [<ffffffff8153c092>] system_call_fastpath+0x16/0x1b
> [ 164.031136] Code: 00 48 8b 75 e8 48 81 c7 08 08 00 00 e8 09 c6 19 00 31 d2 eb 95 90 90 90 90 90 55 c7 05 f5 74 96 00 01 00 00 00 48 89 e5 0f ae f8 <c6> 04 25 00 00 00 00 01 c9 c3 0f 1f 44 00 00 8d 47 d0 55 83 f8
> [ 164.031136] RIP [<ffffffff81397771>] sysrq_handle_crash+0x11/0x20
> [ 164.031136] RSP <ffff88203d5c3e68>
> [ 164.031136] CR2: 0000000000000000
>
> This is always the last output.
>
> Can anyone suggest any way to debug this problem?
>
> I suppose I can hang the processor just before it executes machine_kexec()
> and look at it with crash. Any suggestions as to what to look at?
Hmm. You can enable print statements in purgatory.c. There is a
command line switch that allows pugatory to print to a serial console.
That should be a simple easy thing to try.
I am totally lost as to the status of the patches to make all of this
work right. But the change to let purgator work above 4G was merged
early so hopefully it is not a problem in kexec.
You might also want to enable early printk in the crash dump kernel.
Sometimes kernels get confused on the way up and we hang there.
Eric
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kexec: purgatory hang
2013-06-11 22:54 kexec: purgatory hang Cliff Wickman
2013-06-12 1:24 ` Eric W. Biederman
@ 2013-06-12 1:45 ` Yinghai Lu
2013-06-12 4:14 ` Yinghai Lu
1 sibling, 1 reply; 8+ messages in thread
From: Yinghai Lu @ 2013-06-12 1:45 UTC (permalink / raw)
To: Cliff Wickman; +Cc: kexec@lists.infradead.org, Eric W. Biederman
On Tue, Jun 11, 2013 at 3:54 PM, Cliff Wickman <cpw@sgi.com> wrote:
>
> I'm getting a hang when trying to enter a high-memory crash kernel,
> and I'm at a loss as to how to debug this.
>
> This is a 3.10.0-rc3 kernel, and set up as the crash kernel by kexec 2.0.4.
> The machine is an SGI UV1000.
what is your mem size?
Just tried on one 3T system, it works well...
in first kernel:
sca05-0a81fd78:~ # cat /proc/iomem
00000000-00000fff : reserved
00001000-0009afff : System RAM
0009b000-0009ffff : reserved
000a0000-000bffff : PCI Bus 0000:00
000c0000-000c7fff : Video ROM
000c8000-000ce7ff : Adapter ROM
000ce800-000cf7ff : Adapter ROM
000cf800-000d07ff : Adapter ROM
000e0000-000fffff : reserved
000f0000-000fffff : System ROM
00100000-68ad0fff : System RAM
01000000-020b7d40 : Kernel code
020b7d41-02bd47ff : Kernel data
02f80000-03c20fff : Kernel bss
68ad1000-69265fff : reserved
69266000-69355fff : ACPI Tables
69356000-6a0e4fff : ACPI Non-volatile Storage
6a0e5000-6bd68fff : reserved
6bd69000-6bd98fff : System RAM
6bd99000-6bd99fff : reserved
6bd9a000-7bffffff : System RAM
74000000-7bffffff : Crash kernel
...
100000000-3007fffffff : System RAM
30040000000-3007fffffff : Crash kernel
boot command line:
console=uart8250,io,0x3f8,115200n8 initrd=kernel.org/x.xz rw
root=/dev/ram0 debug ignore_loglevel unknown_nmi_panic
crashkernel=1024M,high crashkernel=128M,low pci=routeirq ip=dhcp
load_ramdisk=1 BOOT_IMAGE=kernel.org/bzImage_3.10_k8.2
kexec second kernel:
# ./kexec -p $VMLINUZ --command-line="initcall_debug nr_cpus=1
pci=routeirq ignore
_loglevel unknown_nmi_panic apic=debug ramdisk_size=$RDSZ root=/dev/ram0 rw ip=d
hcp $CONSOLE" --ramdisk=$INITRD
add_buffer: base:3007ff65000 bufsz:9a000 memsz:9a000
add_buffer: base:3007ff60000 bufsz:3800 memsz:4000
add_buffer: base:3007ff55000 bufsz:80e0 memsz:a000
add_buffer: base:3007ff4f000 bufsz:437a memsz:437a
add_buffer: base:3007d000000 bufsz:8fd240 memsz:2c1f000
add_buffer: base:30079562000 bufsz:3a9ca12 memsz:3a9ca12
...
# echo c > /proc/sysrq-trigger
[ 707.078371] SysRq : Trigger a crash
[ 707.082358] BUG: unable to handle kernel NULL pointer dereference
at (null)
[ 707.091232] IP: [<ffffffff815e4b06>] sysrq_handle_crash+0x16/0x20
[ 707.098170] PGD 0
[ 707.100533] Oops: 0002 [#1] SMP
[ 707.104262] Modules linked in:
[ 707.107753] CPU: 11 PID: 20796 Comm: bash Tainted: G I
3.10.0-rc5-yh-00891-g188560d-dirty #1736
[ 707.128620] task: ffff89de66e1a5a0 ti: ffff89de68bec000 task.ti:
ffff89de68bec000
[ 707.137014] RIP: 0010:[<ffffffff815e4b06>] [<ffffffff815e4b06>]
sysrq_handle_crash+0x16/0x20
[ 707.146651] RSP: 0018:ffff89de68bede48 EFLAGS: 00010096
[ 707.152634] RAX: 000000000000000f RBX: ffffffff82af27e0 RCX: ffff885efd9cf130
[ 707.160656] RDX: 0000000000000001 RSI: ffffffff8108edb0 RDI: 0000000000000063
[ 707.168687] RBP: ffff89de68bede48 R08: 0000000000000001 R09: 0000000000000001
[ 707.176716] R10: 0000000000000001 R11: 0000000000000002 R12: 0000000000000063
[ 707.184745] R13: 0000000000000286 R14: 0000000000000000 R15: 0000000000000001
[ 707.192774] FS: 00007f89bd578700(0000) GS:ffff885efd800000(0000)
knlGS:0000000000000000
[ 707.201863] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 707.208342] CR2: 0000000000000000 CR3: 0000023e66deb000 CR4: 00000000001407e0
[ 707.216364] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 707.224390] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 707.232418] Stack:
[ 707.234722] ffff89de68bede88 ffffffff815e52a2 ffff89de68bede88
0000000000000002
[ 707.243252] 0000000000000002 00007f89bd57d000 ffff89de68bedf50
0000000000000000
[ 707.251751] ffff89de68bedeb8 ffffffff815e53d0 00007f89bd86e290
00007f89bd57d000
[ 707.260235] Call Trace:
[ 707.262996] [<ffffffff815e52a2>] __handle_sysrq+0xc2/0x1b0
[ 707.269278] [<ffffffff815e53d0>] write_sysrq_trigger+0x40/0x50
[ 707.275948] [<ffffffff81220f42>] proc_reg_write+0x42/0x80
[ 707.282133] [<ffffffff811c03eb>] vfs_write+0xeb/0x1c0
[ 707.287911] [<ffffffff811c0865>] SyS_write+0x55/0xb0
[ 707.293610] [<ffffffff820b23da>] tracesys+0xd4/0xd9
[ 707.299166] Code: f0 4c 8b 65 f8 c9 c3 66 2e 0f 1f 84 00 00 00 00
00 0f 1f 40 00 0f 1f 44 00 00 55 c7 05 cc ff a1 01 01 00 00 00 48 89
e5 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 0f 1f 44 00 00 55 48 89 e5
53 48
[ 707.321648] RIP [<ffffffff815e4b06>] sysrq_handle_crash+0x16/0x20
[ 707.328623] RSP <ffff89de68bede48>
[ 707.332573] CR2: 0000000000000000
early console in decompress_kernel
decompress_kernel:
input: [0x3007ea682c2-0x3007f35d8f5], output: 0x3007d000000, heap:
[0x3007f365240-0x3007f36d23f]
Decompressing Linux... xz... Parsing ELF... done.
Booting the kernel.
[ 0.000000] bootconsole [uart0] enabled
[ 0.000000] real_mode_data : phys 000003007ff4f000
[ 0.000000] real_mode_data : virt ffff8b007ff4f000
[ 0.000000] boot_params : init virt ffffffff82f509e0
[ 0.000000] boot_params : phys 000003007ef509e0
[ 0.000000] boot_params : virt ffff8b007ef509e0
[ 0.000000] boot_command_line : init virt ffffffff82e24020
[ 0.000000] boot_command_line : phys 000003007ee24020
[ 0.000000] boot_command_line : virt ffff8b007ee24020
[ 0.000000] Kernel Layout:
[ 0.000000] .text: [0x3007d000000-0x3007e0bfde0]
[ 0.000000] .rodata: [0x3007e200000-0x3007e9c1fff]
[ 0.000000] .data: [0x3007ea00000-0x3007ebb9abf]
[ 0.000000] .init: [0x3007ebbb000-0x3007ef3bfff]
[ 0.000000] .bss: [0x3007ef4a000-0x3007fbf9fff]
[ 0.000000] .brk: [0x3007fbfa000-0x3007fc1efff]
[ 0.000000] memblock_reserve: [0x0009ac00-0x000fffff] * BIOS reserved
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 3.9.0-yh-02267-g2413a4c-dirty
(yhlu@linux-siqj.site) (gcc version 4.7.2 20130108 [gcc-4_7-branch
revision 195012] (SUSE Linux) ) #1507 SMP Mon Apr 29 10:52:45 PDT 2013
[ 0.000000] memblock_reserve: [0x3007d000000-0x3007fbf9fff] TEXT DATA BSS
[ 0.000000] memblock_reserve: [0x30079562000-0x3007cffefff] RAMDISK
[ 0.000000] Command line: initcall_debug nr_cpus=1 pci=routeirq
ignore_loglevel unknown_nmi_panic apic=debug ramdisk_size=262144
root=/dev/ram0 rw ip=dhcp console=uart8250,io,0x3f8,115200n8
memmap=exactmap memmap=616K@4K memmap=131072K@1900544K
memmap=1047936K@3222274048K elfcorehdr=3223321984K
memmap=960K#1722776K memmap=13884K#1723736K
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] Centaur CentaurHauls
[ 0.000000] Physical RAM map:
[ 0.000000] raw: [mem 0x0000000000000100-0x000000000009afff] usable
[ 0.000000] raw: [mem 0x000000000009b000-0x000000000009ffff] reserved
[ 0.000000] raw: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[ 0.000000] raw: [mem 0x0000000000100000-0x0000000068ad0fff] usable
[ 0.000000] raw: [mem 0x0000000068ad1000-0x0000000069265fff] reserved
[ 0.000000] raw: [mem 0x0000000069266000-0x0000000069355fff] ACPI data
[ 0.000000] raw: [mem 0x0000000069356000-0x000000006a0e4fff] ACPI NVS
[ 0.000000] raw: [mem 0x000000006a0e5000-0x000000006bd68fff] reserved
[ 0.000000] raw: [mem 0x000000006bd69000-0x000000006bd98fff] usable
[ 0.000000] raw: [mem 0x000000006bd99000-0x000000006bd99fff] reserved
[ 0.000000] raw: [mem 0x000000006bd9a000-0x000000007bffffff] usable
[ 0.000000] raw: [mem 0x0000000080000000-0x000000008fffffff] reserved
[ 0.000000] raw: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved
[ 0.000000] raw: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[ 0.000000] raw: [mem 0x0000000100000000-0x000003007fffffff] usable
[ 0.000000] e820: BIOS-provided physical RAM map (sanitized by setup):
[ 0.000000] BIOS-e820: [mem 0x0000000000000100-0x000000000009afff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009b000-0x000000000009ffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000068ad0fff] usable
[ 0.000000] BIOS-e820: [mem 0x0000000068ad1000-0x0000000069265fff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000069266000-0x0000000069355fff] ACPI data
[ 0.000000] BIOS-e820: [mem 0x0000000069356000-0x000000006a0e4fff] ACPI NVS
[ 0.000000] BIOS-e820: [mem 0x000000006a0e5000-0x000000006bd68fff] reserved
[ 0.000000] BIOS-e820: [mem 0x000000006bd69000-0x000000006bd98fff] usable
[ 0.000000] BIOS-e820: [mem 0x000000006bd99000-0x000000006bd99fff] reserved
[ 0.000000] BIOS-e820: [mem 0x000000006bd9a000-0x000000007bffffff] usable
[ 0.000000] BIOS-e820: [mem 0x0000000080000000-0x000000008fffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000003007fffffff] usable
[ 0.000000] debug: ignoring loglevel setting.
[ 0.000000] e820: last_pfn = 0x30080000 max_arch_pfn = 0x400000000
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] e820: user-defined physical RAM map:
[ 0.000000] user: [mem 0x0000000000001000-0x000000000009afff] usable
[ 0.000000] user: [mem 0x0000000069266000-0x000000006a0e4fff] ACPI data
[ 0.000000] user: [mem 0x0000000074000000-0x000000007bffffff] usable
[ 0.000000] user: [mem 0x0000030040000000-0x000003007ff5ffff] usable
...
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kexec: purgatory hang
2013-06-12 1:45 ` Yinghai Lu
@ 2013-06-12 4:14 ` Yinghai Lu
2013-06-12 16:49 ` Cliff Wickman
0 siblings, 1 reply; 8+ messages in thread
From: Yinghai Lu @ 2013-06-12 4:14 UTC (permalink / raw)
To: Cliff Wickman; +Cc: kexec@lists.infradead.org, Eric W. Biederman
On Tue, Jun 11, 2013 at 6:45 PM, Yinghai Lu <yinghai@kernel.org> wrote:
> On Tue, Jun 11, 2013 at 3:54 PM, Cliff Wickman <cpw@sgi.com> wrote:
>>
>> I'm getting a hang when trying to enter a high-memory crash kernel,
>> and I'm at a loss as to how to debug this.
>>
>> This is a 3.10.0-rc3 kernel, and set up as the crash kernel by kexec 2.0.4.
>> The machine is an SGI UV1000.
>
> what is your mem size?
also can you post your /proc/iomem in first kernel?
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kexec: purgatory hang
2013-06-12 1:24 ` Eric W. Biederman
@ 2013-06-12 16:40 ` Cliff Wickman
2013-06-12 21:52 ` Eric W. Biederman
0 siblings, 1 reply; 8+ messages in thread
From: Cliff Wickman @ 2013-06-12 16:40 UTC (permalink / raw)
To: Eric W. Biederman; +Cc: yinghai, kexec
On Tue, Jun 11, 2013 at 06:24:24PM -0700, Eric W. Biederman wrote:
> Cliff Wickman <cpw@sgi.com> writes:
>
> > I'm getting a hang when trying to enter a high-memory crash kernel,
> > and I'm at a loss as to how to debug this.
> >
> > This is a 3.10.0-rc3 kernel, and set up as the crash kernel by kexec 2.0.4.
> > The machine is an SGI UV1000.
> >
> > [ 164.027275] SysRq : Trigger a crash
> > [ 164.031136] BUG: unable to handle kernel NULL pointer dereference at (null)
> > [ 164.031136] IP: [<ffffffff81397771>] sysrq_handle_crash+0x11/0x20
> > [ 164.031136] PGD 1fbe835067 PUD 1fbc2e8067 PMD 0
> > [ 164.031136] Oops: 0002 [#1] SMP
> > [ 164.031136] xpc : all partitions have deactivated
> > [ 164.031136] Modules linked in: autofs4 binfmt_misc af_packet rdma_ucm rdma_cm iw_cm ib_addr ib_srp scsi_transport_srp scsi_tgt ib_ipoib ib_cm ib_uverbs ib_umad iw_cxgb3 cxgb3 mdio mlx4_en mlx4_ib ib_sa mlx4_core ib_mthca ib_mad ib_core fuse nls_iso8859_1 nls_cp437 vfat fat loop uv_mmtimer dm_mod sr_mod cdrom usb_storage iTCO_wdt iTCO_vendor_support coretemp mperf kvm_intel ipv6 kvm igb sg crc32c_intel lpc_ich pcspkr mptctl i2c_algo_bit ptp i2c_i801 microcode xhci_hcd joydev ioatdma ehci_pci hid_generic pps_core i2c_core rtc_cmos mfd_core button dca usbhid hid uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif scsi_dh_hp_sw scsi_dh_alua scsi_dh_emc scsi_dh_rdac scsi_dh thermal sata_nv processor piix mptsas mptscsih scsi_transport_sas mptbase megaraid_sas fan thermal_sys hwmon ext3 jbd ata_piix ahci libahci libata scsi_mod
> > [ 164.031136] CPU: 10 PID: 9299 Comm: dopanic Not tainted 3.10.0-rc3-linus-cpw+ #17
> > [ 164.031136] Hardware name: Intel Corp. Stoutland Platform, BIOS 2.16 UEFI2.10 PI1.0 X64 2012-04-27
> > [ 164.031136] task: ffff88203df94440 ti: ffff88203d5c2000 task.ti: ffff88203d5c2000
> > [ 164.031136] RIP: 0010:[<ffffffff81397771>] [<ffffffff81397771>] sysrq_handle_crash+0x11/0x20
> > [ 164.031136] RSP: 0018:ffff88203d5c3e68 EFLAGS: 00010092
> > [ 164.031136] RAX: 000000000000000f RBX: ffffffff81a974e0 RCX: 0000000000000004
> > [ 164.031136] RDX: 0000000000000000 RSI: ffff881fffd0ef48 RDI: 0000000000000063
> > [ 164.031136] RBP: ffff88203d5c3e68 R08: ffff881fffd0d3e8 R09: 000000000004268c
> > [ 164.031136] R10: 0000000000000b8b R11: 0000000000000000 R12: 0000000000000063
> > [ 164.031136] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000296
> > [ 164.031136] FS: 00007ffff7fb5700(0000) GS:ffff881fffd00000(0000) knlGS:0000000000000000
> > [ 164.031136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 164.031136] CR2: 0000000000000000 CR3: 0000001fbea6c000 CR4: 00000000000007e0
> > [ 164.031136] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > [ 164.031136] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > [ 164.031136] Stack:
> > [ 164.031136] ffff88203d5c3ea8 ffffffff81398008 01ff88203d5c3e88 0000000000000002
> > [ 164.031136] ffff895f9d478380 ffff88203d5c3f40 00007ffff7ff8000 ffff88203d5c3f40
> > [ 164.031136] ffff88203d5c3ec8 ffffffff813980ad ffff88203d5c3ee8 fffffffffffffffb
> > [ 164.031136] Call Trace:
> > [ 164.031136] [<ffffffff81398008>] __handle_sysrq+0x128/0x190
> > [ 164.031136] [<ffffffff813980ad>] write_sysrq_trigger+0x3d/0x40
> > [ 164.031136] [<ffffffff811c323f>] proc_reg_write+0x4f/0x80
> > [ 164.031136] [<ffffffff8115f107>] vfs_write+0xe7/0x190
> > [ 164.031136] [<ffffffff8115f8ec>] SyS_write+0x5c/0xa0
> > [ 164.031136] [<ffffffff8153c092>] system_call_fastpath+0x16/0x1b
> > [ 164.031136] Code: 00 48 8b 75 e8 48 81 c7 08 08 00 00 e8 09 c6 19 00 31 d2 eb 95 90 90 90 90 90 55 c7 05 f5 74 96 00 01 00 00 00 48 89 e5 0f ae f8 <c6> 04 25 00 00 00 00 01 c9 c3 0f 1f 44 00 00 8d 47 d0 55 83 f8
> > [ 164.031136] RIP [<ffffffff81397771>] sysrq_handle_crash+0x11/0x20
> > [ 164.031136] RSP <ffff88203d5c3e68>
> > [ 164.031136] CR2: 0000000000000000
> >
> > This is always the last output.
> >
> > Can anyone suggest any way to debug this problem?
> >
> > I suppose I can hang the processor just before it executes machine_kexec()
> > and look at it with crash. Any suggestions as to what to look at?
Hi Eric,
Thanks for the reply.
> Hmm. You can enable print statements in purgatory.c. There is a
> command line switch that allows pugatory to print to a serial console.
> That should be a simple easy thing to try.
Do you recall what that switch is? I don't see any condition in
purgatory.c. But the existing printf's don't give me anything.
> I am totally lost as to the status of the patches to make all of this
> work right. But the change to let purgator work above 4G was merged
> early so hopefully it is not a problem in kexec.
It works on a UV2000 or a whitebox, so it must be close.
>
> You might also want to enable early printk in the crash dump kernel.
> Sometimes kernels get confused on the way up and we hang there.
Yes, I'm using earlyprintk.
-Cliff
--
Cliff Wickman
SGI
cpw@sgi.com
(651) 683-3824
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kexec: purgatory hang
2013-06-12 4:14 ` Yinghai Lu
@ 2013-06-12 16:49 ` Cliff Wickman
2013-06-12 17:43 ` Yinghai Lu
0 siblings, 1 reply; 8+ messages in thread
From: Cliff Wickman @ 2013-06-12 16:49 UTC (permalink / raw)
To: Yinghai Lu; +Cc: kexec@lists.infradead.org, Eric W. Biederman
On Tue, Jun 11, 2013 at 09:14:19PM -0700, Yinghai Lu wrote:
> On Tue, Jun 11, 2013 at 6:45 PM, Yinghai Lu <yinghai@kernel.org> wrote:
> > On Tue, Jun 11, 2013 at 3:54 PM, Cliff Wickman <cpw@sgi.com> wrote:
> >>
> >> I'm getting a hang when trying to enter a high-memory crash kernel,
> >> and I'm at a loss as to how to debug this.
> >>
> >> This is a 3.10.0-rc3 kernel, and set up as the crash kernel by kexec 2.0.4.
> >> The machine is an SGI UV1000.
> >
> > what is your mem size?
uv21-sys:~ # head /proc/meminfo
MemTotal: 726897840 kB
> also can you post your /proc/iomem in first kernel?
This is /proc/iomem after asking for crashkernel=900M,high
But I also get the same result specifying lower addresses, such
as crashkernel=900M@256G
00000000-00000fff : reserved
00001000-0008efff : System RAM
0008f000-0008ffff : ACPI Non-volatile Storage
00090000-0009ffff : System RAM
000c0000-000c7fff : Video ROM
000cc000-000d1bff : Adapter ROM
000f0000-000fffff : System ROM
00100000-75958fff : System RAM
01000000-0153feab : Kernel code
0153feac-01ae703f : Kernel data
01c93000-01e1efff : Kernel bss
6d000000-717fffff : Crash kernel low
75959000-7597cfff : Unusable memory
7597d000-7597efff : System RAM
7597f000-7597ffff : reserved
75980000-75980fff : System RAM
75981000-75981fff : reserved
75982000-75c5efff : System RAM
75c5f000-75c63fff : Unusable memory
75c64000-75c6cfff : System RAM
75c6d000-75c75fff : reserved
75c76000-75ca3fff : System RAM
75ca4000-75cabfff : Unusable memory
75cac000-75cb3fff : System RAM
75cb4000-75cb5fff : Unusable memory
75cb6000-75cfefff : System RAM
75cff000-75d18fff : Unusable memory
75d19000-75d22fff : System RAM
75d23000-75d24fff : reserved
75d25000-75d25fff : System RAM
75d26000-75d28fff : reserved
75d29000-7617bfff : System RAM
7617c000-761abfff : Unusable memory
761ac000-76261fff : System RAM
76262000-76263fff : Unusable memory
76264000-76278fff : System RAM
76279000-7627bfff : Unusable memory
7627c000-76289fff : System RAM
7628a000-7628ffff : Unusable memory
76290000-76296fff : System RAM
76297000-7629efff : Unusable memory
7629f000-76476fff : System RAM
76477000-76482fff : reserved
76483000-78b20fff : System RAM
78b21000-78b22fff : reserved
78b23000-78b24fff : Unusable memory
78b25000-78b26fff : reserved
78b27000-78b30fff : Unusable memory
78b31000-78b56fff : reserved
78b57000-78b60fff : Unusable memory
78b61000-78b70fff : reserved
78b71000-78bc0fff : Unusable memory
78bc1000-78c10fff : reserved
78bc6018-78bc6027 : APEI ERST
78bc6030-78bc606f : APEI ERST
78bc6070-78bc8017 : APEI ERST
78c11000-78c73fff : ACPI Non-volatile Storage
78c74000-78ce8fff : ACPI Tables
78ce9000-78d10fff : ACPI Non-volatile Storage
78d11000-78d30fff : ACPI Tables
78d31000-7cda1fff : System RAM
7cda2000-7cda2fff : reserved
7cda3000-7cdb4fff : System RAM
7cdb5000-7ce34fff : reserved
7ce35000-7effffff : System RAM
7f000000-7fffffff : RAM buffer
80000000-8fffffff : reserved
80000000-806fffff : PCI MMCONFIG 0000 [bus 00-06]
90000000-970fffff : PCI Bus 0003:01
90000000-970fffff : PCI Bus 0003:02
90000000-970fffff : PCI Bus 0003:05
90000000-970fffff : PCI Bus 0003:06
90000000-930fffff : PCI Bus 0003:08
90000000-91ffffff : 0003:08:00.0
92000000-92ffffff : 0003:08:00.0
93000000-9307ffff : 0003:08:00.0
94000000-970fffff : PCI Bus 0003:07
94000000-95ffffff : 0003:07:00.0
96000000-96ffffff : 0003:07:00.0
97000000-9707ffff : 0003:07:00.0
97100000-97103fff : 0003:00:16.0
97100000-97103fff : ioatdma
97104000-97107fff : 0003:00:16.1
97104000-97107fff : ioatdma
97108000-9710bfff : 0003:00:16.2
97108000-9710bfff : ioatdma
9710c000-9710ffff : 0003:00:16.3
9710c000-9710ffff : ioatdma
97110000-97113fff : 0003:00:16.4
97110000-97113fff : ioatdma
97114000-97117fff : 0003:00:16.5
97114000-97117fff : ioatdma
97118000-9711bfff : 0003:00:16.6
97118000-9711bfff : ioatdma
9711c000-9711ffff : 0003:00:16.7
9711c000-9711ffff : ioatdma
97120000-97120fff : 0003:00:13.0
b0000000-b0ffffff : PCI Bus 0000:05
b0000000-b0ffffff : 0000:05:00.0
b0000000-b03a9fff : efifb
b1000000-b18fffff : PCI Bus 0000:05
b1000000-b17fffff : 0000:05:00.0
b1800000-b180ffff : 0000:05:00.0
b1810000-b1813fff : 0000:05:00.0
b1c00000-b20fffff : PCI Bus 0000:04
b1c00000-b1ffffff : 0000:04:00.0
b2000000-b200ffff : 0000:04:00.0
b2000000-b200ffff : mpt
b2010000-b2013fff : 0000:04:00.0
b2010000-b2013fff : mpt
b2100000-b22fffff : PCI Bus 0000:01
b2100000-b211ffff : 0000:01:00.1
b2120000-b213ffff : 0000:01:00.1
b2120000-b213ffff : igb
b2140000-b215ffff : 0000:01:00.1
b2140000-b215ffff : igb
b2160000-b217ffff : 0000:01:00.0
b2180000-b219ffff : 0000:01:00.0
b2180000-b219ffff : igb
b21a0000-b21bffff : 0000:01:00.0
b21a0000-b21bffff : igb
b21c0000-b21dffff : 0000:01:00.0
b21e0000-b21fffff : 0000:01:00.0
b2200000-b221ffff : 0000:01:00.1
b2220000-b223ffff : 0000:01:00.1
b2240000-b2243fff : 0000:01:00.1
b2240000-b2243fff : igb
b2244000-b2247fff : 0000:01:00.0
b2244000-b2247fff : igb
b2300000-b2303fff : 0000:00:16.0
b2300000-b2303fff : ioatdma
b2304000-b2307fff : 0000:00:16.1
b2304000-b2307fff : ioatdma
b2308000-b230bfff : 0000:00:16.2
b2308000-b230bfff : ioatdma
b230c000-b230ffff : 0000:00:16.3
b230c000-b230ffff : ioatdma
b2310000-b2313fff : 0000:00:16.4
b2310000-b2313fff : ioatdma
b2314000-b2317fff : 0000:00:16.5
b2314000-b2317fff : ioatdma
b2318000-b231bfff : 0000:00:16.6
b2318000-b231bfff : ioatdma
b231c000-b231ffff : 0000:00:16.7
b231c000-b231ffff : ioatdma
b2320000-b23203ff : 0000:00:1d.7
b2321000-b23213ff : 0000:00:1a.7
b2321000-b23213ff : ehci_hcd
b2322000-b23220ff : 0000:00:1f.3
b2323000-b2323fff : 0000:00:13.0
b3000000-b3003fff : 0001:00:16.0
b3000000-b3003fff : ioatdma
b3004000-b3007fff : 0001:00:16.1
b3004000-b3007fff : ioatdma
b3008000-b300bfff : 0001:00:16.2
b3008000-b300bfff : ioatdma
b300c000-b300ffff : 0001:00:16.3
b300c000-b300ffff : ioatdma
b3010000-b3013fff : 0001:00:16.4
b3010000-b3013fff : ioatdma
b3014000-b3017fff : 0001:00:16.5
b3014000-b3017fff : ioatdma
b3018000-b301bfff : 0001:00:16.6
b3018000-b301bfff : ioatdma
b301c000-b301ffff : 0001:00:16.7
b301c000-b301ffff : ioatdma
b3020000-b3020fff : 0001:00:13.0
b4000000-b4003fff : 0002:00:16.0
b4000000-b4003fff : ioatdma
b4004000-b4007fff : 0002:00:16.1
b4004000-b4007fff : ioatdma
b4008000-b400bfff : 0002:00:16.2
b4008000-b400bfff : ioatdma
b400c000-b400ffff : 0002:00:16.3
b400c000-b400ffff : ioatdma
b4010000-b4013fff : 0002:00:16.4
b4010000-b4013fff : ioatdma
b4014000-b4017fff : 0002:00:16.5
b4014000-b4017fff : ioatdma
b4018000-b401bfff : 0002:00:16.6
b4018000-b401bfff : ioatdma
b401c000-b401ffff : 0002:00:16.7
b401c000-b401ffff : ioatdma
b4020000-b4020fff : 0002:00:13.0
b5000000-b50fffff : PCI Bus 0004:02
b5000000-b5001fff : 0004:02:00.0
b5000000-b5001fff : xhci_hcd
b5100000-b5103fff : 0004:00:16.0
b5100000-b5103fff : ioatdma
b5104000-b5107fff : 0004:00:16.1
b5104000-b5107fff : ioatdma
b5108000-b510bfff : 0004:00:16.2
b5108000-b510bfff : ioatdma
b510c000-b510ffff : 0004:00:16.3
b510c000-b510ffff : ioatdma
b5110000-b5113fff : 0004:00:16.4
b5110000-b5113fff : ioatdma
b5114000-b5117fff : 0004:00:16.5
b5114000-b5117fff : ioatdma
b5118000-b511bfff : 0004:00:16.6
b5118000-b511bfff : ioatdma
b511c000-b511ffff : 0004:00:16.7
b511c000-b511ffff : ioatdma
b5120000-b5120fff : 0004:00:13.0
f0000000-fbffffff : reserved
fd020000-fd020fff : dmar0
fd040000-fd040fff : dmar1
fd060000-fd060fff : dmar2
fd080000-fd080fff : dmar3
fd0a0000-fd0a0fff : dmar4
feb00000-febfffff : reserved
fec00000-fec003ff : IOAPIC 0
fec02000-fec023ff : IOAPIC 1
fec04000-fec043ff : IOAPIC 2
fec06000-fec063ff : IOAPIC 3
fec08000-fec083ff : IOAPIC 4
fec0a000-fec0a3ff : IOAPIC 5
fed00000-fed003ff : HPET 0
fed12000-fed1200f : pnp 00:06
fed12010-fed1201f : pnp 00:06
fed1b000-fed1bfff : pnp 00:06
fed1c000-fed3ffff : pnp 00:06
fed1f410-fed1f414 : iTCO_wdt
fed45000-fed8bfff : pnp 00:06
fee00000-feefffff : pnp 00:06
ff000000-ffffffff : pnp 00:06
100000000-207fffffff : System RAM
4000000000-5ffeffffff : System RAM
5fff000000-5fffffffff : RAM buffer
8000000000-9ffeffffff : System RAM
9fff000000-9fffffffff : RAM buffer
c000000000-cffeffffff : System RAM
cfff000000-cfffffffff : RAM buffer
10000000000-11ffeffffff : System RAM
11fff000000-11fffffffff : RAM buffer
14000000000-15ffeffffff : System RAM
15fc6000000-15ffe3fffff : Crash kernel
15fff000000-15fffffffff : RAM buffer
1fd80000000-1fd83cfffff : reserved
1fd83e00000-1fd83ffffff : PCI MMCONFIG 1000 [bus 3e-3f]
1fd83e00000-1fd83ffffff : reserved
1fd87e00000-1fd8bcfffff : reserved
1fd87e00000-1fd87ffffff : PCI MMCONFIG 1001 [bus 3e-3f]
1fd88000000-1fd882fffff : PCI MMCONFIG 0001 [bus 00-02]
1fd8be00000-1fd8fcfffff : reserved
1fd8be00000-1fd8bffffff : PCI MMCONFIG 1002 [bus 3e-3f]
1fd8c000000-1fd8c2fffff : PCI MMCONFIG 0002 [bus 00-02]
1fd8fe00000-1fd93cfffff : reserved
1fd8fe00000-1fd8fffffff : PCI MMCONFIG 1003 [bus 3e-3f]
1fd90000000-1fd90afffff : PCI MMCONFIG 0003 [bus 00-0a]
1fd93e00000-1fd97cfffff : reserved
1fd93e00000-1fd93ffffff : PCI MMCONFIG 1004 [bus 3e-3f]
1fd94000000-1fd942fffff : PCI MMCONFIG 0004 [bus 00-02]
1fd97e00000-1fd97ffffff : PCI MMCONFIG 1005 [bus 3e-3f]
1fd97e00000-1fd97ffffff : reserved
1fdc0000000-1fdc2ffffff : reserved
1fdc4000000-1fdc6ffffff : reserved
1fdc8000000-1fdcaffffff : reserved
1fdcc000000-1fdceffffff : reserved
1fdd0000000-1fdd2ffffff : reserved
1fdd4000000-1fdd6ffffff : reserved
1ff00000000-1ff1fffffff : PCI Bus 0003:01
1ff00000000-1ff1fffffff : PCI Bus 0003:02
1ff00000000-1ff1fffffff : PCI Bus 0003:05
1ff00000000-1ff1fffffff : PCI Bus 0003:06
1ff00000000-1ff0fffffff : PCI Bus 0003:08
1ff00000000-1ff0fffffff : 0003:08:00.0
1ff10000000-1ff1fffffff : PCI Bus 0003:07
1ff10000000-1ff1fffffff : 0003:07:00.0
--
Cliff Wickman
SGI
cpw@sgi.com
(651) 683-3824
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kexec: purgatory hang
2013-06-12 16:49 ` Cliff Wickman
@ 2013-06-12 17:43 ` Yinghai Lu
0 siblings, 0 replies; 8+ messages in thread
From: Yinghai Lu @ 2013-06-12 17:43 UTC (permalink / raw)
To: Cliff Wickman; +Cc: kexec@lists.infradead.org, Eric W. Biederman
On Wed, Jun 12, 2013 at 9:49 AM, Cliff Wickman <cpw@sgi.com> wrote:
> On Tue, Jun 11, 2013 at 09:14:19PM -0700, Yinghai Lu wrote:
>> On Tue, Jun 11, 2013 at 6:45 PM, Yinghai Lu <yinghai@kernel.org> wrote:
>> > On Tue, Jun 11, 2013 at 3:54 PM, Cliff Wickman <cpw@sgi.com> wrote:
>> >>
>> >> I'm getting a hang when trying to enter a high-memory crash kernel,
>> >> and I'm at a loss as to how to debug this.
>> >>
>> >> This is a 3.10.0-rc3 kernel, and set up as the crash kernel by kexec 2.0.4.
>> >> The machine is an SGI UV1000.
>> >
>> > what is your mem size?
>
> uv21-sys:~ # head /proc/meminfo
> MemTotal: 726897840 kB
only 700G?
>
>> also can you post your /proc/iomem in first kernel?
>
> This is /proc/iomem after asking for crashkernel=900M,high
>
> But I also get the same result specifying lower addresses, such
> as crashkernel=900M@256G
>
> 00000000-00000fff : reserved
> 00001000-0008efff : System RAM
> 0008f000-0008ffff : ACPI Non-volatile Storage
> 00090000-0009ffff : System RAM
> 000c0000-000c7fff : Video ROM
> 000cc000-000d1bff : Adapter ROM
> 000f0000-000fffff : System ROM
> 00100000-75958fff : System RAM
> 01000000-0153feab : Kernel code
> 0153feac-01ae703f : Kernel data
> 01c93000-01e1efff : Kernel bss
> 6d000000-717fffff : Crash kernel low
Can you use latest linus tree and lastest kexec-tools tree?
We removed the low according to Vivek.
> 75959000-7597cfff : Unusable memory
> 7597d000-7597efff : System RAM
> 7597f000-7597ffff : reserved
> 75980000-75980fff : System RAM
> 75981000-75981fff : reserved
> 75982000-75c5efff : System RAM
> 75c5f000-75c63fff : Unusable memory
> 75c64000-75c6cfff : System RAM
> 75c6d000-75c75fff : reserved
> 75c76000-75ca3fff : System RAM
> 75ca4000-75cabfff : Unusable memory
> 75cac000-75cb3fff : System RAM
> 75cb4000-75cb5fff : Unusable memory
> 75cb6000-75cfefff : System RAM
> 75cff000-75d18fff : Unusable memory
> 75d19000-75d22fff : System RAM
> 75d23000-75d24fff : reserved
> 75d25000-75d25fff : System RAM
> 75d26000-75d28fff : reserved
> 75d29000-7617bfff : System RAM
> 7617c000-761abfff : Unusable memory
> 761ac000-76261fff : System RAM
> 76262000-76263fff : Unusable memory
> 76264000-76278fff : System RAM
> 76279000-7627bfff : Unusable memory
> 7627c000-76289fff : System RAM
> 7628a000-7628ffff : Unusable memory
> 76290000-76296fff : System RAM
> 76297000-7629efff : Unusable memory
> 7629f000-76476fff : System RAM
> 76477000-76482fff : reserved
> 76483000-78b20fff : System RAM
> 78b21000-78b22fff : reserved
> 78b23000-78b24fff : Unusable memory
> 78b25000-78b26fff : reserved
> 78b27000-78b30fff : Unusable memory
> 78b31000-78b56fff : reserved
> 78b57000-78b60fff : Unusable memory
> 78b61000-78b70fff : reserved
> 78b71000-78bc0fff : Unusable memory
> 78bc1000-78c10fff : reserved
> 78bc6018-78bc6027 : APEI ERST
> 78bc6030-78bc606f : APEI ERST
> 78bc6070-78bc8017 : APEI ERST
> 78c11000-78c73fff : ACPI Non-volatile Storage
> 78c74000-78ce8fff : ACPI Tables
> 78ce9000-78d10fff : ACPI Non-volatile Storage
> 78d11000-78d30fff : ACPI Tables
> 78d31000-7cda1fff : System RAM
> 7cda2000-7cda2fff : reserved
> 7cda3000-7cdb4fff : System RAM
> 7cdb5000-7ce34fff : reserved
> 7ce35000-7effffff : System RAM
> 7f000000-7fffffff : RAM buffer
> 80000000-8fffffff : reserved
> 80000000-806fffff : PCI MMCONFIG 0000 [bus 00-06]
> 90000000-970fffff : PCI Bus 0003:01
> 90000000-970fffff : PCI Bus 0003:02
> 90000000-970fffff : PCI Bus 0003:05
> 90000000-970fffff : PCI Bus 0003:06
> 90000000-930fffff : PCI Bus 0003:08
> 90000000-91ffffff : 0003:08:00.0
> 92000000-92ffffff : 0003:08:00.0
> 93000000-9307ffff : 0003:08:00.0
> 94000000-970fffff : PCI Bus 0003:07
> 94000000-95ffffff : 0003:07:00.0
> 96000000-96ffffff : 0003:07:00.0
> 97000000-9707ffff : 0003:07:00.0
> 97100000-97103fff : 0003:00:16.0
> 97100000-97103fff : ioatdma
> 97104000-97107fff : 0003:00:16.1
> 97104000-97107fff : ioatdma
> 97108000-9710bfff : 0003:00:16.2
> 97108000-9710bfff : ioatdma
> 9710c000-9710ffff : 0003:00:16.3
> 9710c000-9710ffff : ioatdma
> 97110000-97113fff : 0003:00:16.4
> 97110000-97113fff : ioatdma
> 97114000-97117fff : 0003:00:16.5
> 97114000-97117fff : ioatdma
> 97118000-9711bfff : 0003:00:16.6
> 97118000-9711bfff : ioatdma
> 9711c000-9711ffff : 0003:00:16.7
> 9711c000-9711ffff : ioatdma
> 97120000-97120fff : 0003:00:13.0
> b0000000-b0ffffff : PCI Bus 0000:05
> b0000000-b0ffffff : 0000:05:00.0
> b0000000-b03a9fff : efifb
> b1000000-b18fffff : PCI Bus 0000:05
> b1000000-b17fffff : 0000:05:00.0
> b1800000-b180ffff : 0000:05:00.0
> b1810000-b1813fff : 0000:05:00.0
> b1c00000-b20fffff : PCI Bus 0000:04
> b1c00000-b1ffffff : 0000:04:00.0
> b2000000-b200ffff : 0000:04:00.0
> b2000000-b200ffff : mpt
> b2010000-b2013fff : 0000:04:00.0
> b2010000-b2013fff : mpt
> b2100000-b22fffff : PCI Bus 0000:01
> b2100000-b211ffff : 0000:01:00.1
> b2120000-b213ffff : 0000:01:00.1
> b2120000-b213ffff : igb
> b2140000-b215ffff : 0000:01:00.1
> b2140000-b215ffff : igb
> b2160000-b217ffff : 0000:01:00.0
> b2180000-b219ffff : 0000:01:00.0
> b2180000-b219ffff : igb
> b21a0000-b21bffff : 0000:01:00.0
> b21a0000-b21bffff : igb
> b21c0000-b21dffff : 0000:01:00.0
> b21e0000-b21fffff : 0000:01:00.0
> b2200000-b221ffff : 0000:01:00.1
> b2220000-b223ffff : 0000:01:00.1
> b2240000-b2243fff : 0000:01:00.1
> b2240000-b2243fff : igb
> b2244000-b2247fff : 0000:01:00.0
> b2244000-b2247fff : igb
> b2300000-b2303fff : 0000:00:16.0
> b2300000-b2303fff : ioatdma
> b2304000-b2307fff : 0000:00:16.1
> b2304000-b2307fff : ioatdma
> b2308000-b230bfff : 0000:00:16.2
> b2308000-b230bfff : ioatdma
> b230c000-b230ffff : 0000:00:16.3
> b230c000-b230ffff : ioatdma
> b2310000-b2313fff : 0000:00:16.4
> b2310000-b2313fff : ioatdma
> b2314000-b2317fff : 0000:00:16.5
> b2314000-b2317fff : ioatdma
> b2318000-b231bfff : 0000:00:16.6
> b2318000-b231bfff : ioatdma
> b231c000-b231ffff : 0000:00:16.7
> b231c000-b231ffff : ioatdma
> b2320000-b23203ff : 0000:00:1d.7
> b2321000-b23213ff : 0000:00:1a.7
> b2321000-b23213ff : ehci_hcd
> b2322000-b23220ff : 0000:00:1f.3
> b2323000-b2323fff : 0000:00:13.0
> b3000000-b3003fff : 0001:00:16.0
> b3000000-b3003fff : ioatdma
> b3004000-b3007fff : 0001:00:16.1
> b3004000-b3007fff : ioatdma
> b3008000-b300bfff : 0001:00:16.2
> b3008000-b300bfff : ioatdma
> b300c000-b300ffff : 0001:00:16.3
> b300c000-b300ffff : ioatdma
> b3010000-b3013fff : 0001:00:16.4
> b3010000-b3013fff : ioatdma
> b3014000-b3017fff : 0001:00:16.5
> b3014000-b3017fff : ioatdma
> b3018000-b301bfff : 0001:00:16.6
> b3018000-b301bfff : ioatdma
> b301c000-b301ffff : 0001:00:16.7
> b301c000-b301ffff : ioatdma
> b3020000-b3020fff : 0001:00:13.0
> b4000000-b4003fff : 0002:00:16.0
> b4000000-b4003fff : ioatdma
> b4004000-b4007fff : 0002:00:16.1
> b4004000-b4007fff : ioatdma
> b4008000-b400bfff : 0002:00:16.2
> b4008000-b400bfff : ioatdma
> b400c000-b400ffff : 0002:00:16.3
> b400c000-b400ffff : ioatdma
> b4010000-b4013fff : 0002:00:16.4
> b4010000-b4013fff : ioatdma
> b4014000-b4017fff : 0002:00:16.5
> b4014000-b4017fff : ioatdma
> b4018000-b401bfff : 0002:00:16.6
> b4018000-b401bfff : ioatdma
> b401c000-b401ffff : 0002:00:16.7
> b401c000-b401ffff : ioatdma
> b4020000-b4020fff : 0002:00:13.0
> b5000000-b50fffff : PCI Bus 0004:02
> b5000000-b5001fff : 0004:02:00.0
> b5000000-b5001fff : xhci_hcd
> b5100000-b5103fff : 0004:00:16.0
> b5100000-b5103fff : ioatdma
> b5104000-b5107fff : 0004:00:16.1
> b5104000-b5107fff : ioatdma
> b5108000-b510bfff : 0004:00:16.2
> b5108000-b510bfff : ioatdma
> b510c000-b510ffff : 0004:00:16.3
> b510c000-b510ffff : ioatdma
> b5110000-b5113fff : 0004:00:16.4
> b5110000-b5113fff : ioatdma
> b5114000-b5117fff : 0004:00:16.5
> b5114000-b5117fff : ioatdma
> b5118000-b511bfff : 0004:00:16.6
> b5118000-b511bfff : ioatdma
> b511c000-b511ffff : 0004:00:16.7
> b511c000-b511ffff : ioatdma
> b5120000-b5120fff : 0004:00:13.0
> f0000000-fbffffff : reserved
> fd020000-fd020fff : dmar0
> fd040000-fd040fff : dmar1
> fd060000-fd060fff : dmar2
> fd080000-fd080fff : dmar3
> fd0a0000-fd0a0fff : dmar4
> feb00000-febfffff : reserved
> fec00000-fec003ff : IOAPIC 0
> fec02000-fec023ff : IOAPIC 1
> fec04000-fec043ff : IOAPIC 2
> fec06000-fec063ff : IOAPIC 3
> fec08000-fec083ff : IOAPIC 4
> fec0a000-fec0a3ff : IOAPIC 5
> fed00000-fed003ff : HPET 0
> fed12000-fed1200f : pnp 00:06
> fed12010-fed1201f : pnp 00:06
> fed1b000-fed1bfff : pnp 00:06
> fed1c000-fed3ffff : pnp 00:06
> fed1f410-fed1f414 : iTCO_wdt
> fed45000-fed8bfff : pnp 00:06
> fee00000-feefffff : pnp 00:06
> ff000000-ffffffff : pnp 00:06
> 100000000-207fffffff : System RAM
> 4000000000-5ffeffffff : System RAM
> 5fff000000-5fffffffff : RAM buffer
> 8000000000-9ffeffffff : System RAM
> 9fff000000-9fffffffff : RAM buffer
> c000000000-cffeffffff : System RAM
> cfff000000-cfffffffff : RAM buffer
> 10000000000-11ffeffffff : System RAM
> 11fff000000-11fffffffff : RAM buffer
> 14000000000-15ffeffffff : System RAM
> 15fc6000000-15ffe3fffff : Crash kernel
so it is about 1.4T
> 15fff000000-15fffffffff : RAM buffer
> 1fd80000000-1fd83cfffff : reserved
> 1fd83e00000-1fd83ffffff : PCI MMCONFIG 1000 [bus 3e-3f]
> 1fd83e00000-1fd83ffffff : reserved
> 1fd87e00000-1fd8bcfffff : reserved
> 1fd87e00000-1fd87ffffff : PCI MMCONFIG 1001 [bus 3e-3f]
> 1fd88000000-1fd882fffff : PCI MMCONFIG 0001 [bus 00-02]
> 1fd8be00000-1fd8fcfffff : reserved
> 1fd8be00000-1fd8bffffff : PCI MMCONFIG 1002 [bus 3e-3f]
> 1fd8c000000-1fd8c2fffff : PCI MMCONFIG 0002 [bus 00-02]
> 1fd8fe00000-1fd93cfffff : reserved
> 1fd8fe00000-1fd8fffffff : PCI MMCONFIG 1003 [bus 3e-3f]
> 1fd90000000-1fd90afffff : PCI MMCONFIG 0003 [bus 00-0a]
> 1fd93e00000-1fd97cfffff : reserved
> 1fd93e00000-1fd93ffffff : PCI MMCONFIG 1004 [bus 3e-3f]
> 1fd94000000-1fd942fffff : PCI MMCONFIG 0004 [bus 00-02]
> 1fd97e00000-1fd97ffffff : PCI MMCONFIG 1005 [bus 3e-3f]
> 1fd97e00000-1fd97ffffff : reserved
> 1fdc0000000-1fdc2ffffff : reserved
> 1fdc4000000-1fdc6ffffff : reserved
> 1fdc8000000-1fdcaffffff : reserved
> 1fdcc000000-1fdceffffff : reserved
> 1fdd0000000-1fdd2ffffff : reserved
> 1fdd4000000-1fdd6ffffff : reserved
> 1ff00000000-1ff1fffffff : PCI Bus 0003:01
> 1ff00000000-1ff1fffffff : PCI Bus 0003:02
> 1ff00000000-1ff1fffffff : PCI Bus 0003:05
> 1ff00000000-1ff1fffffff : PCI Bus 0003:06
> 1ff00000000-1ff0fffffff : PCI Bus 0003:08
> 1ff00000000-1ff0fffffff : 0003:08:00.0
> 1ff10000000-1ff1fffffff : PCI Bus 0003:07
> 1ff10000000-1ff1fffffff : 0003:07:00.0
>
> --
> Cliff Wickman
> SGI
> cpw@sgi.com
> (651) 683-3824
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kexec: purgatory hang
2013-06-12 16:40 ` Cliff Wickman
@ 2013-06-12 21:52 ` Eric W. Biederman
0 siblings, 0 replies; 8+ messages in thread
From: Eric W. Biederman @ 2013-06-12 21:52 UTC (permalink / raw)
To: Cliff Wickman; +Cc: yinghai, kexec
Cliff Wickman <cpw@sgi.com> writes:
> Do you recall what that switch is? I don't see any condition in
> purgatory.c. But the existing printf's don't give me anything.
kexec --load --console-serial --serial=<port> --serial-baud=<baud_rate> ...
Possibly
kexec --load --console-vga ...
Eric
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2013-06-12 21:53 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-06-11 22:54 kexec: purgatory hang Cliff Wickman
2013-06-12 1:24 ` Eric W. Biederman
2013-06-12 16:40 ` Cliff Wickman
2013-06-12 21:52 ` Eric W. Biederman
2013-06-12 1:45 ` Yinghai Lu
2013-06-12 4:14 ` Yinghai Lu
2013-06-12 16:49 ` Cliff Wickman
2013-06-12 17:43 ` Yinghai Lu
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.