* Linux 4.4 boot crash on xen 4.5.3 with dom0_mem == max
@ 2016-05-17 4:28 Ed Swierk
2016-05-17 7:43 ` Juergen Gross
0 siblings, 1 reply; 8+ messages in thread
From: Ed Swierk @ 2016-05-17 4:28 UTC (permalink / raw)
To: xen-devel, jgross; +Cc: eswierk
I'm trying to figure out a crash when booting a Linux 4.4 dom0 on
a recent stable xen 4.5. I'm capping the dom0 memory by setting
dom0_mem=18432M,max:18432M on the xen command line, and the kernel
config has CONFIG_XEN_BALLOON unset.
The crash seems dependent on the contents of the e820 table; identical
hardware with a different BIOS version does not crash.
I instrumented various functions arch/x86/xen/setup.c, and isolated
the problem to the final call to xen_del_extra_mem() in
xen_remap_memory(). The BUG_ON(n_pfns > size_r) assertion is triggered
in the "Start of region" block.
...
[ 0.000000] Released 0 page(s)
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] Xen: [mem 0x0000000000000000-0x000000000005ffff] usable
[ 0.000000] Xen: [mem 0x0000000000060000-0x0000000000067fff] reserved
[ 0.000000] Xen: [mem 0x0000000000068000-0x000000000009afff] usable
[ 0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[ 0.000000] Xen: [mem 0x0000000000100000-0x0000000075afdfff] usable
[ 0.000000] Xen: [mem 0x0000000075bab000-0x000000007a1f5fff] usable
[ 0.000000] Xen: [mem 0x000000007b7d7000-0x000000007b7fffff] usable
[ 0.000000] Xen: [mem 0x000000007bf00000-0x000000007bffffff] usable
[ 0.000000] Xen: [mem 0x00000000c7ffc000-0x00000000c7ffcfff] reserved
[ 0.000000] Xen: [mem 0x00000000fbffc000-0x00000000fbffcfff] reserved
[ 0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec01fff] reserved
[ 0.000000] Xen: [mem 0x00000000fec40000-0x00000000fec40fff] reserved
[ 0.000000] Xen: [mem 0x00000000fed20000-0x00000000fed2ffff] usable
[ 0.000000] Xen: [mem 0x00000000fee00000-0x00000000feefffff] reserved
[ 0.000000] Xen: [mem 0x0000000100000000-0x0000000505de9fff] usable
[ 0.000000] bootconsole [xenboot0] enabled
[ 0.000000] debug: ignoring loglevel setting.
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] e820: user-defined physical RAM map:
[ 0.000000] user: [mem 0x0000000000000000-0x000000000005ffff] usable
[ 0.000000] user: [mem 0x0000000000060000-0x0000000000067fff] reserved
[ 0.000000] user: [mem 0x0000000000068000-0x000000000009afff] usable
[ 0.000000] user: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[ 0.000000] user: [mem 0x0000000000100000-0x0000000075afdfff] usable
[ 0.000000] user: [mem 0x0000000075bab000-0x000000007a1f5fff] usable
[ 0.000000] user: [mem 0x000000007b7d7000-0x000000007b7fffff] usable
[ 0.000000] user: [mem 0x000000007bf00000-0x000000007bffffff] usable
[ 0.000000] user: [mem 0x0000000080000000-0x000000008fffffff] reserved
[ 0.000000] user: [mem 0x00000000c7ffc000-0x00000000c7ffcfff] reserved
[ 0.000000] user: [mem 0x00000000fbffc000-0x00000000fbffcfff] reserved
[ 0.000000] user: [mem 0x00000000fec00000-0x00000000fec01fff] reserved
[ 0.000000] user: [mem 0x00000000fec40000-0x00000000fec40fff] reserved
[ 0.000000] user: [mem 0x00000000fed20000-0x00000000fed2ffff] usable
[ 0.000000] user: [mem 0x00000000fee00000-0x00000000feefffff] reserved
[ 0.000000] user: [mem 0x0000000100000000-0x0000000505de9fff] usable
[ 0.000000] SMBIOS 2.7 present.
[ 0.000000] DMI: ABCD, BIOS SE5C610.86B.01.01.0009.C1.060120151350 06/01/2015
[ 0.000000] Hypervisor detected: Xen
[ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[ 0.000000] e820: last_pfn = 0x505dea max_arch_pfn = 0x400000000
[ 0.000000] MTRR: Disabled
[ 0.000000] e820: last_pfn = 0xfed30 max_arch_pfn = 0x400000000
[ 0.000000] Scanning 1 areas for low memory corruption
[ 0.000000] Base memory trampoline at [ffff880000095000] 95000 size 24576
[ 0.000000] BRK [0x02f67000, 0x02f67fff] PGTABLE
[ 0.000000] BRK [0x02f68000, 0x02f68fff] PGTABLE
[ 0.000000] BRK [0x02f69000, 0x02f69fff] PGTABLE
[ 0.000000] BRK [0x02f6a000, 0x02f6afff] PGTABLE
[ 0.000000] BRK [0x02f6b000, 0x02f6bfff] PGTABLE
[ 0.000000] BRK [0x02f6c000, 0x02f6cfff] PGTABLE
[ 0.000000] RAMDISK: [mem 0x04000000-0x14afefff]
[ 0.000000] ACPI: Early table checksum verification disabled
[ 0.000000] ACPI: RSDP 0x00000000000F0460 000024 (v02 INTEL )
[ 0.000000] ACPI: XSDT 0x000000007B78D188 0000CC (v01 INTEL S2600WT 00000000 INTL 01000013)
[ 0.000000] ACPI: FACP 0x000000007B7D2000 0000F4 (v04 INTEL S2600WT 00000000 INTL 20091013)
[ 0.000000] ACPI: DSDT 0x000000007B798000 0313B2 (v02 INTEL S2600WT 00000003 INTL 20091013)
[ 0.000000] ACPI: FACS 0x000000007B6FC000 000040
[ 0.000000] ACPI: HPET 0x000000007B7D1000 000038 (v01 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: APIC 0x000000007B7D0000 00085C (v03 INTEL S2600WT 00000000 INTL 20091013)
[ 0.000000] ACPI: MCFG 0x000000007B7CF000 00003C (v01 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: MSCT 0x000000007B7CE000 000090 (v01 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: SLIT 0x000000007B7CD000 00006C (v01 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: SRAT 0x000000007B7CC000 000730 (v03 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: SPMI 0x000000007B7CB000 000041 (v05 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: WDDT 0x000000007B7CA000 000040 (v01 INTEL S2600WT 00000000 INTL 20091013)
[ 0.000000] ACPI: PRAD 0x000000007B793000 000102 (v02 INTEL SpsPrAgg 00000002 INTL 20130328)
[ 0.000000] ACPI: XMAR 0x000000007B7D5000 000118 (v01 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: TCPA 0x000000007B7D4000 000064 (v02 INTEL S2600WT 00000002 INTL 01000013)
[ 0.000000] ACPI: SSDT 0x000000007B7D3000 0003A2 (v02 INTEL S2600WT 00001000 INTL 20130328)
[ 0.000000] ACPI: SSDT 0x0000000075AFE000 0ACA42 (v02 INTEL S2600WT 00004000 INTL 20130328)
[ 0.000000] ACPI: SSDT 0x000000007B795000 00216B (v02 INTEL S2600WT 00000002 INTL 20130328)
[ 0.000000] ACPI: SSDT 0x000000007B794000 000064 (v02 INTEL S2600WT 00000002 INTL 20130328)
[ 0.000000] ACPI: HEST 0x000000007B792000 0000A8 (v01 INTEL S2600WT 00000001 INTL 00000001)
[ 0.000000] ACPI: BERT 0x000000007B791000 000030 (v01 INTEL S2600WT 00000001 INTL 00000001)
[ 0.000000] ACPI: ERST 0x000000007B790000 000230 (v01 INTEL S2600WT 00000001 INTL 00000001)
[ 0.000000] ACPI: EINJ 0x000000007B78F000 000130 (v01 INTEL S2600WT 00000001 INTL 00000001)
[ 0.000000] ACPI: SPCR 0x000000007B78E000 000050 (v01 INTEL S2600WT 00000000 INTL 00000000)
[ 0.000000] ACPI: Local APIC address 0xfee00000
[ 0.000000] Setting APIC routing to Xen PV.
[ 0.000000] set_pte_vaddr ffffffffff5f9000 to 800000201e103063
[ 0.000000] NUMA turned off
[ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000505de9fff]
[ 0.000000] NODE_DATA(0) allocated [mem 0x47dc05000-0x47dc09fff]
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff]
[ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff]
[ 0.000000] Normal [mem 0x0000000100000000-0x0000000505de9fff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000001000-0x000000000005ffff]
[ 0.000000] node 0: [mem 0x0000000000068000-0x000000000009afff]
[ 0.000000] node 0: [mem 0x0000000000100000-0x0000000075afdfff]
[ 0.000000] node 0: [mem 0x0000000075bab000-0x000000007a1f5fff]
[ 0.000000] node 0: [mem 0x000000007b7d7000-0x000000007b7fffff]
[ 0.000000] node 0: [mem 0x000000007bf00000-0x000000007bffffff]
[ 0.000000] node 0: [mem 0x00000000fed20000-0x00000000fed2ffff]
[ 0.000000] node 0: [mem 0x0000000100000000-0x0000000505de9fff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000505de9fff]
[ 0.000000] On node 0 totalpages: 4718590
[ 0.000000] DMA zone: 64 pages used for memmap
[ 0.000000] DMA zone: 21 pages reserved
[ 0.000000] DMA zone: 3986 pages, LIFO batch:0
[ 0.000000] DMA32 zone: 7755 pages used for memmap
[ 0.000000] DMA32 zone: 496258 pages, LIFO batch:31
[ 0.000000] Normal zone: 65912 pages used for memmap
[ 0.000000] Normal zone: 4218346 pages, LIFO batch:31
[ 0.000000] p2m virtual area at ffffc90000000000, size is 2a00000
...
[ 0.000000] set_pte_mfn buf=0xffffffff8208b000 xen_remap_mfn=0x201c09b PAGE_KERNEL
[ 0.000000] set_pte_vaddr ffffffff8208b000 to 800000201c09b063
[ 0.000000] remapped 548323 so far
[ 0.000000] set_pte_mfn buf=0xffffffff8208b000 xen_remap_mfn=0x201c060 PAGE_KERNEL
[ 0.000000] set_pte_vaddr ffffffff8208b000 to 800000201c060063
[ 0.000000] remapped 548331 so far
[ 0.000000] final xen_del_extra_mem pfn_s=0x480000 len=0x85deb
[ 0.000000] xen_del_extra_mem i=0
[ 0.000000] xen_del_extra_mem i=0 start_r=0x480000 size_r=0x85dea n_pfns=0x85deb
(XEN) traps.c:459:d0v0 Unhandled invalid opcode fault/trap [#6] on VCPU 0 [ec=0000]
(XEN) domain_crash_sync called from entry.S: fault at ffff82d0802286c3 create_bounce_frame+0x12b/0x13a
(XEN) Domain 0 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-4.5.4-pre x86_64 debug=n Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e033:[<ffffffff820002c4>]
(XEN) RFLAGS: 0000000000000297 EM: 1 CONTEXT: pv guest (d0v0)
(XEN) rax: 0000000000000044 rbx: 0000000000085deb rcx: 0000000000000000
(XEN) rdx: 0000000000000044 rsi: 0000000000000001 rdi: ffffffff816cbc7a
(XEN) rbp: ffffffff81b57d00 rsp: ffffffff81b57c58 r8: 0000000000000001
(XEN) r9: 0000000000000001 r10: 0000000000480000 r11: 00000000000009cb
(XEN) r12: 0000000000480000 r13: 0000000000085deb r14: ffffffff8208ca20
(XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 00000000001526f0
(XEN) cr3: 000000201d6e1000 cr2: ffffc9000282e000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033
(XEN) Guest stack trace from rsp=ffffffff81b57c58:
(XEN) 0000000000000000 00000000000009cb ffffffff820002c4 000000010000e030
(XEN) 0000000000010097 ffffffff81b57c90 000000000000e02b 0000000000085deb
(XEN) 0000000000505deb ffffffff81b57cb0 ffffffff810ca31a ffffffff81b57d10
(XEN) ffffffff811427fb 0000000000085dea ffffffff81b57d20 0000000000480000
(XEN) ffffffff81b57d20 0000000000085deb 0000000000085deb 0000000000085deb
(XEN) 0000000000480000 ffffffff81b57d90 ffffffff820009c5 ffffffff81b57d90
(XEN) ffffffff820009c5 00003ffffffff000 0000000000480000 000000000201c067
(XEN) ffffffffffc05de3 0000000000085deb 000000000201e08b 000000201c067001
(XEN) 0000000000480007 ffff880014affff8 ffffffff81b57db0 ffff880014b00ff8
(XEN) 0000010000000000 00003ffffffff000 00003ffffffff000 ffffffff81b57e20
(XEN) ffffffff82003322 ffffffff81b57e20 ffffffff82003322 0000000000000f7d
(XEN) 0000000000000000 ffffffff00000000 ffff880000000000 0000000000001000
(XEN) 000000047dc0a000 0000000400000000 ffff880014aff000 ffff880014afe6dc
(XEN) ffffffff81b57e40 0000000000000000 0000000004000000 0000000000000000
(XEN) 0000000014aff000 ffffffff81b57ea0 ffffffff8200601a ffffffff81b57ea0
(XEN) ffffffff8200601a ffffffff81b57e50 ffffffff810ca31a 0000000004000000
(XEN) ffffffff811427fb 0000000010afe6dc ffffffff81b57ec0 0000000000000000
(XEN) ffffffff81b57ec0 ffffffff81b57f38 0000000000000000 0000000010afe6dc
(XEN) 0000000000000000 ffffffff81b57ef0 ffffffff81ffccd6 ffffffff81b57ef0
(XEN) ffffffff81ffccd6 0000000000000000 ffffffff820930a0 0000000000000000
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
I can work around the crash by passing dom0_mem=18432M,max:18433M
instead.
[ 0.000000] Released 0 page(s)
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] Xen: [mem 0x0000000000000000-0x000000000005ffff] usable
[ 0.000000] Xen: [mem 0x0000000000060000-0x0000000000067fff] reserved
[ 0.000000] Xen: [mem 0x0000000000068000-0x000000000009afff] usable
[ 0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[ 0.000000] Xen: [mem 0x0000000000100000-0x0000000075afdfff] usable
[ 0.000000] Xen: [mem 0x0000000075bab000-0x000000007a1f5fff] usable
[ 0.000000] Xen: [mem 0x000000007b7d7000-0x000000007b7fffff] usable
[ 0.000000] Xen: [mem 0x000000007bf00000-0x000000007bffffff] usable
[ 0.000000] Xen: [mem 0x00000000c7ffc000-0x00000000c7ffcfff] reserved
[ 0.000000] Xen: [mem 0x00000000fbffc000-0x00000000fbffcfff] reserved
[ 0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec01fff] reserved
[ 0.000000] Xen: [mem 0x00000000fec40000-0x00000000fec40fff] reserved
[ 0.000000] Xen: [mem 0x00000000fed20000-0x00000000fed2ffff] usable
[ 0.000000] Xen: [mem 0x00000000fee00000-0x00000000feefffff] reserved
[ 0.000000] Xen: [mem 0x0000000100000000-0x0000000505ee9fff] usable
[ 0.000000] bootconsole [xenboot0] enabled
[ 0.000000] debug: ignoring loglevel setting.
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] e820: user-defined physical RAM map:
[ 0.000000] user: [mem 0x0000000000000000-0x000000000005ffff] usable
[ 0.000000] user: [mem 0x0000000000060000-0x0000000000067fff] reserved
[ 0.000000] user: [mem 0x0000000000068000-0x000000000009afff] usable
[ 0.000000] user: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[ 0.000000] user: [mem 0x0000000000100000-0x0000000075afdfff] usable
[ 0.000000] user: [mem 0x0000000075bab000-0x000000007a1f5fff] usable
[ 0.000000] user: [mem 0x000000007b7d7000-0x000000007b7fffff] usable
[ 0.000000] user: [mem 0x000000007bf00000-0x000000007bffffff] usable
[ 0.000000] user: [mem 0x0000000080000000-0x000000008fffffff] reserved
[ 0.000000] user: [mem 0x00000000c7ffc000-0x00000000c7ffcfff] reserved
[ 0.000000] user: [mem 0x00000000fbffc000-0x00000000fbffcfff] reserved
[ 0.000000] user: [mem 0x00000000fec00000-0x00000000fec01fff] reserved
[ 0.000000] user: [mem 0x00000000fec40000-0x00000000fec40fff] reserved
[ 0.000000] user: [mem 0x00000000fed20000-0x00000000fed2ffff] usable
[ 0.000000] user: [mem 0x00000000fee00000-0x00000000feefffff] reserved
[ 0.000000] user: [mem 0x0000000100000000-0x0000000505ee9fff] usable
[ 0.000000] SMBIOS 2.7 present.
[ 0.000000] DMI: ABCD, BIOS SE5C610.86B.01.01.0009.C1.060120151350 06/01/2015
[ 0.000000] Hypervisor detected: Xen
[ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[ 0.000000] e820: last_pfn = 0x505eea max_arch_pfn = 0x400000000
[ 0.000000] MTRR: Disabled
[ 0.000000] e820: last_pfn = 0xfed30 max_arch_pfn = 0x400000000
[ 0.000000] Scanning 1 areas for low memory corruption
[ 0.000000] Base memory trampoline at [ffff880000095000] 95000 size 24576
[ 0.000000] BRK [0x02f67000, 0x02f67fff] PGTABLE
[ 0.000000] BRK [0x02f68000, 0x02f68fff] PGTABLE
[ 0.000000] BRK [0x02f69000, 0x02f69fff] PGTABLE
[ 0.000000] BRK [0x02f6a000, 0x02f6afff] PGTABLE
[ 0.000000] BRK [0x02f6b000, 0x02f6bfff] PGTABLE
[ 0.000000] BRK [0x02f6c000, 0x02f6cfff] PGTABLE
[ 0.000000] RAMDISK: [mem 0x04000000-0x14afefff]
[ 0.000000] ACPI: Early table checksum verification disabled
[ 0.000000] ACPI: RSDP 0x00000000000F0460 000024 (v02 INTEL )
[ 0.000000] ACPI: XSDT 0x000000007B78D188 0000CC (v01 INTEL S2600WT 00000000 INTL 01000013)
[ 0.000000] ACPI: FACP 0x000000007B7D2000 0000F4 (v04 INTEL S2600WT 00000000 INTL 20091013)
[ 0.000000] ACPI: DSDT 0x000000007B798000 0313B2 (v02 INTEL S2600WT 00000003 INTL 20091013)
[ 0.000000] ACPI: FACS 0x000000007B6FC000 000040
[ 0.000000] ACPI: HPET 0x000000007B7D1000 000038 (v01 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: APIC 0x000000007B7D0000 00085C (v03 INTEL S2600WT 00000000 INTL 20091013)
[ 0.000000] ACPI: MCFG 0x000000007B7CF000 00003C (v01 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: MSCT 0x000000007B7CE000 000090 (v01 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: SLIT 0x000000007B7CD000 00006C (v01 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: SRAT 0x000000007B7CC000 000730 (v03 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: SPMI 0x000000007B7CB000 000041 (v05 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: WDDT 0x000000007B7CA000 000040 (v01 INTEL S2600WT 00000000 INTL 20091013)
[ 0.000000] ACPI: PRAD 0x000000007B793000 000102 (v02 INTEL SpsPrAgg 00000002 INTL 20130328)
[ 0.000000] ACPI: XMAR 0x000000007B7D5000 000118 (v01 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: TCPA 0x000000007B7D4000 000064 (v02 INTEL S2600WT 00000002 INTL 01000013)
[ 0.000000] ACPI: SSDT 0x000000007B7D3000 0003A2 (v02 INTEL S2600WT 00001000 INTL 20130328)
[ 0.000000] ACPI: SSDT 0x0000000075AFE000 0ACA42 (v02 INTEL S2600WT 00004000 INTL 20130328)
[ 0.000000] ACPI: SSDT 0x000000007B795000 00216B (v02 INTEL S2600WT 00000002 INTL 20130328)
[ 0.000000] ACPI: SSDT 0x000000007B794000 000064 (v02 INTEL S2600WT 00000002 INTL 20130328)
[ 0.000000] ACPI: HEST 0x000000007B792000 0000A8 (v01 INTEL S2600WT 00000001 INTL 00000001)
[ 0.000000] ACPI: BERT 0x000000007B791000 000030 (v01 INTEL S2600WT 00000001 INTL 00000001)
[ 0.000000] ACPI: ERST 0x000000007B790000 000230 (v01 INTEL S2600WT 00000001 INTL 00000001)
[ 0.000000] ACPI: EINJ 0x000000007B78F000 000130 (v01 INTEL S2600WT 00000001 INTL 00000001)
[ 0.000000] ACPI: SPCR 0x000000007B78E000 000050 (v01 INTEL S2600WT 00000000 INTL 00000000)
[ 0.000000] ACPI: Local APIC address 0xfee00000
[ 0.000000] Setting APIC routing to Xen PV.
[ 0.000000] set_pte_vaddr ffffffffff5f9000 to 800000201e103063
[ 0.000000] NUMA turned off
[ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000505ee9fff]
[ 0.000000] NODE_DATA(0) allocated [mem 0x47dc04000-0x47dc08fff]
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff]
[ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff]
[ 0.000000] Normal [mem 0x0000000100000000-0x0000000505ee9fff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000001000-0x000000000005ffff]
[ 0.000000] node 0: [mem 0x0000000000068000-0x000000000009afff]
[ 0.000000] node 0: [mem 0x0000000000100000-0x0000000075afdfff]
[ 0.000000] node 0: [mem 0x0000000075bab000-0x000000007a1f5fff]
[ 0.000000] node 0: [mem 0x000000007b7d7000-0x000000007b7fffff]
[ 0.000000] node 0: [mem 0x000000007bf00000-0x000000007bffffff]
[ 0.000000] node 0: [mem 0x00000000fed20000-0x00000000fed2ffff]
[ 0.000000] node 0: [mem 0x0000000100000000-0x0000000505ee9fff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000505ee9fff]
[ 0.000000] On node 0 totalpages: 4718846
[ 0.000000] DMA zone: 64 pages used for memmap
[ 0.000000] DMA zone: 21 pages reserved
[ 0.000000] DMA zone: 3986 pages, LIFO batch:0
[ 0.000000] DMA32 zone: 7755 pages used for memmap
[ 0.000000] DMA32 zone: 496258 pages, LIFO batch:31
[ 0.000000] Normal zone: 65916 pages used for memmap
[ 0.000000] Normal zone: 4218602 pages, LIFO batch:31
[ 0.000000] p2m virtual area at ffffc90000000000, size is 2a00000
...
[ 0.000000] set_pte_mfn buf=0xffffffff8208b000 xen_remap_mfn=0x201c09b PAGE_KERNEL
[ 0.000000] set_pte_vaddr ffffffff8208b000 to 800000201c09b063
[ 0.000000] remapped 548323 so far
[ 0.000000] set_pte_mfn buf=0xffffffff8208b000 xen_remap_mfn=0x201c060 PAGE_KERNEL
[ 0.000000] set_pte_vaddr ffffffff8208b000 to 800000201c060063
[ 0.000000] remapped 548331 so far
[ 0.000000] final xen_del_extra_mem pfn_s=0x480000 len=0x85deb
[ 0.000000] xen_del_extra_mem i=0
[ 0.000000] xen_del_extra_mem i=0 start_r=0x480000 size_r=0x85eea n_pfns=0x85deb
[ 0.000000] memblock_free(0x480000000, 0x85deb000)
[ 0.000000] final set_pte_mfn buf=0xffffffff8208b000 mfn_save=0x201e08b PAGE_KERNEL
[ 0.000000] set_pte_vaddr ffffffff8208b000 to 800000201e08b063
[ 0.000000] Remapped 548331 page(s)
...
Any idea what is causing the BUG_ON()?
(Incidentally, such assertions would be far easier to debug if
BUG_ON() printed clues like the condition, line number, and filename,
before crashing.)
Thanks,
--Ed
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Linux 4.4 boot crash on xen 4.5.3 with dom0_mem == max
2016-05-17 4:28 Linux 4.4 boot crash on xen 4.5.3 with dom0_mem == max Ed Swierk
@ 2016-05-17 7:43 ` Juergen Gross
2016-05-17 13:37 ` Ed Swierk
0 siblings, 1 reply; 8+ messages in thread
From: Juergen Gross @ 2016-05-17 7:43 UTC (permalink / raw)
To: Ed Swierk, xen-devel
On 17/05/16 06:28, Ed Swierk wrote:
> I'm trying to figure out a crash when booting a Linux 4.4 dom0 on
> a recent stable xen 4.5. I'm capping the dom0 memory by setting
> dom0_mem=18432M,max:18432M on the xen command line, and the kernel
> config has CONFIG_XEN_BALLOON unset.
>
> The crash seems dependent on the contents of the e820 table; identical
> hardware with a different BIOS version does not crash.
>
> I instrumented various functions arch/x86/xen/setup.c, and isolated
> the problem to the final call to xen_del_extra_mem() in
> xen_remap_memory(). The BUG_ON(n_pfns > size_r) assertion is triggered
> in the "Start of region" block.
Hmm, weird.
Could you please add some more instrumentation? I'd be interested in all
calls to xen_add_extra_mem() and xen_del_extra_mem() with the parameters
Something like:
printk("%s(%lx, %lx)\n", __func__, start_pfn, n_pfns);
at the beginning of both functions.
This should help in finding the bug.
...
> [ 0.000000] Xen: [mem 0x0000000100000000-0x0000000505de9fff] usable
...
> [ 0.000000] final xen_del_extra_mem pfn_s=0x480000 len=0x85deb
> [ 0.000000] xen_del_extra_mem i=0
> [ 0.000000] xen_del_extra_mem i=0 start_r=0x480000 size_r=0x85dea n_pfns=0x85deb
...
> Any idea what is causing the BUG_ON()?
There seems to be an off by one error. n_pfns on the last call is
wrong, IMO.
> (Incidentally, such assertions would be far easier to debug if
> BUG_ON() printed clues like the condition, line number, and filename,
> before crashing.)
Indeed.
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Linux 4.4 boot crash on xen 4.5.3 with dom0_mem == max
2016-05-17 7:43 ` Juergen Gross
@ 2016-05-17 13:37 ` Ed Swierk
2016-05-17 20:50 ` Ed Swierk
0 siblings, 1 reply; 8+ messages in thread
From: Ed Swierk @ 2016-05-17 13:37 UTC (permalink / raw)
To: xen-devel, jgross; +Cc: eswierk
Here is the instrumented output with dom0_mem=18432M,max:18432M.
...
[ 0.000000] xen_count_remap_pages(max_pfn=0x480000) == 0x85dea
[ 0.000000] max_pages 0x505dea
[ 0.000000] xen_add_extra_mem(480000, 85dea)
[ 0.000000] memblock_reserve(0x480000000, 0x85dea000)
[ 0.000000] Released 0 page(s)
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] Xen: [mem 0x0000000000000000-0x000000000005ffff] usable
[ 0.000000] Xen: [mem 0x0000000000060000-0x0000000000067fff] reserved
[ 0.000000] Xen: [mem 0x0000000000068000-0x000000000009afff] usable
[ 0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[ 0.000000] Xen: [mem 0x0000000000100000-0x0000000075afdfff] usable
[ 0.000000] Xen: [mem 0x0000000075bab000-0x000000007a1f5fff] usable
[ 0.000000] Xen: [mem 0x000000007b7d7000-0x000000007b7fffff] usable
[ 0.000000] Xen: [mem 0x000000007bf00000-0x000000007bffffff] usable
[ 0.000000] Xen: [mem 0x00000000c7ffc000-0x00000000c7ffcfff] reserved
[ 0.000000] Xen: [mem 0x00000000fbffc000-0x00000000fbffcfff] reserved
[ 0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec01fff] reserved
[ 0.000000] Xen: [mem 0x00000000fec40000-0x00000000fec40fff] reserved
[ 0.000000] Xen: [mem 0x00000000fed20000-0x00000000fed2ffff] usable
[ 0.000000] Xen: [mem 0x00000000fee00000-0x00000000feefffff] reserved
[ 0.000000] Xen: [mem 0x0000000100000000-0x0000000505de9fff] usable
[ 0.000000] bootconsole [xenboot0] enabled
[ 0.000000] debug: ignoring loglevel setting.
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] e820: user-defined physical RAM map:
[ 0.000000] user: [mem 0x0000000000000000-0x000000000005ffff] usable
[ 0.000000] user: [mem 0x0000000000060000-0x0000000000067fff] reserved
[ 0.000000] user: [mem 0x0000000000068000-0x000000000009afff] usable
[ 0.000000] user: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[ 0.000000] user: [mem 0x0000000000100000-0x0000000075afdfff] usable
[ 0.000000] user: [mem 0x0000000075bab000-0x000000007a1f5fff] usable
[ 0.000000] user: [mem 0x000000007b7d7000-0x000000007b7fffff] usable
[ 0.000000] user: [mem 0x000000007bf00000-0x000000007bffffff] usable
[ 0.000000] user: [mem 0x0000000080000000-0x000000008fffffff] reserved
[ 0.000000] user: [mem 0x00000000c7ffc000-0x00000000c7ffcfff] reserved
[ 0.000000] user: [mem 0x00000000fbffc000-0x00000000fbffcfff] reserved
[ 0.000000] user: [mem 0x00000000fec00000-0x00000000fec01fff] reserved
[ 0.000000] user: [mem 0x00000000fec40000-0x00000000fec40fff] reserved
[ 0.000000] user: [mem 0x00000000fed20000-0x00000000fed2ffff] usable
[ 0.000000] user: [mem 0x00000000fee00000-0x00000000feefffff] reserved
[ 0.000000] user: [mem 0x0000000100000000-0x0000000505de9fff] usable
[ 0.000000] SMBIOS 2.7 present.
[ 0.000000] DMI: Skyport Systems CloudDock/S2600WTT, BIOS SE5C610.86B.01.01.0009.C1.060120151350 06/01/2015
[ 0.000000] Hypervisor detected: Xen
[ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[ 0.000000] e820: last_pfn = 0x505dea max_arch_pfn = 0x400000000
[ 0.000000] MTRR: Disabled
[ 0.000000] e820: last_pfn = 0xfed30 max_arch_pfn = 0x400000000
[ 0.000000] Scanning 1 areas for low memory corruption
[ 0.000000] Base memory trampoline at [ffff880000095000] 95000 size 24576
[ 0.000000] BRK [0x02f67000, 0x02f67fff] PGTABLE
[ 0.000000] BRK [0x02f68000, 0x02f68fff] PGTABLE
[ 0.000000] BRK [0x02f69000, 0x02f69fff] PGTABLE
[ 0.000000] BRK [0x02f6a000, 0x02f6afff] PGTABLE
[ 0.000000] BRK [0x02f6b000, 0x02f6bfff] PGTABLE
[ 0.000000] BRK [0x02f6c000, 0x02f6cfff] PGTABLE
[ 0.000000] RAMDISK: [mem 0x04000000-0x14afefff]
[ 0.000000] ACPI: Early table checksum verification disabled
[ 0.000000] ACPI: RSDP 0x00000000000F0460 000024 (v02 INTEL )
[ 0.000000] ACPI: XSDT 0x000000007B78D188 0000CC (v01 INTEL S2600WT 00000000 INTL 01000013)
[ 0.000000] ACPI: FACP 0x000000007B7D2000 0000F4 (v04 INTEL S2600WT 00000000 INTL 20091013)
[ 0.000000] ACPI: DSDT 0x000000007B798000 0313B2 (v02 INTEL S2600WT 00000003 INTL 20091013)
[ 0.000000] ACPI: FACS 0x000000007B6FC000 000040
[ 0.000000] ACPI: HPET 0x000000007B7D1000 000038 (v01 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: APIC 0x000000007B7D0000 00085C (v03 INTEL S2600WT 00000000 INTL 20091013)
[ 0.000000] ACPI: MCFG 0x000000007B7CF000 00003C (v01 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: MSCT 0x000000007B7CE000 000090 (v01 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: SLIT 0x000000007B7CD000 00006C (v01 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: SRAT 0x000000007B7CC000 000730 (v03 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: SPMI 0x000000007B7CB000 000041 (v05 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: WDDT 0x000000007B7CA000 000040 (v01 INTEL S2600WT 00000000 INTL 20091013)
[ 0.000000] ACPI: PRAD 0x000000007B793000 000102 (v02 INTEL SpsPrAgg 00000002 INTL 20130328)
[ 0.000000] ACPI: XMAR 0x000000007B7D5000 000118 (v01 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: TCPA 0x000000007B7D4000 000064 (v02 INTEL S2600WT 00000002 INTL 01000013)
[ 0.000000] ACPI: SSDT 0x000000007B7D3000 0003A2 (v02 INTEL S2600WT 00001000 INTL 20130328)
[ 0.000000] ACPI: SSDT 0x0000000075AFE000 0ACA42 (v02 INTEL S2600WT 00004000 INTL 20130328)
[ 0.000000] ACPI: SSDT 0x000000007B795000 00216B (v02 INTEL S2600WT 00000002 INTL 20130328)
[ 0.000000] ACPI: SSDT 0x000000007B794000 000064 (v02 INTEL S2600WT 00000002 INTL 20130328)
[ 0.000000] ACPI: HEST 0x000000007B792000 0000A8 (v01 INTEL S2600WT 00000001 INTL 00000001)
[ 0.000000] ACPI: BERT 0x000000007B791000 000030 (v01 INTEL S2600WT 00000001 INTL 00000001)
[ 0.000000] ACPI: ERST 0x000000007B790000 000230 (v01 INTEL S2600WT 00000001 INTL 00000001)
[ 0.000000] ACPI: EINJ 0x000000007B78F000 000130 (v01 INTEL S2600WT 00000001 INTL 00000001)
[ 0.000000] ACPI: SPCR 0x000000007B78E000 000050 (v01 INTEL S2600WT 00000000 INTL 00000000)
[ 0.000000] ACPI: Local APIC address 0xfee00000
[ 0.000000] Setting APIC routing to Xen PV.
[ 0.000000] set_pte_vaddr ffffffffff5f9000 to 800000201e103063
[ 0.000000] NUMA turned off
[ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000505de9fff]
[ 0.000000] NODE_DATA(0) allocated [mem 0x47dc05000-0x47dc09fff]
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff]
[ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff]
[ 0.000000] Normal [mem 0x0000000100000000-0x0000000505de9fff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000001000-0x000000000005ffff]
[ 0.000000] node 0: [mem 0x0000000000068000-0x000000000009afff]
[ 0.000000] node 0: [mem 0x0000000000100000-0x0000000075afdfff]
[ 0.000000] node 0: [mem 0x0000000075bab000-0x000000007a1f5fff]
[ 0.000000] node 0: [mem 0x000000007b7d7000-0x000000007b7fffff]
[ 0.000000] node 0: [mem 0x000000007bf00000-0x000000007bffffff]
[ 0.000000] node 0: [mem 0x00000000fed20000-0x00000000fed2ffff]
[ 0.000000] node 0: [mem 0x0000000100000000-0x0000000505de9fff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000505de9fff]
[ 0.000000] On node 0 totalpages: 4718590
[ 0.000000] DMA zone: 64 pages used for memmap
[ 0.000000] DMA zone: 21 pages reserved
[ 0.000000] DMA zone: 3986 pages, LIFO batch:0
[ 0.000000] DMA32 zone: 7755 pages used for memmap
[ 0.000000] DMA32 zone: 496258 pages, LIFO batch:31
[ 0.000000] Normal zone: 65912 pages used for memmap
[ 0.000000] Normal zone: 4218346 pages, LIFO batch:31
[ 0.000000] p2m virtual area at ffffc90000000000, size is 2a00000
...
[ 0.000000] set_pte_mfn buf=0xffffffff8208b000 xen_remap_mfn=0x201c09b PAGE_KERNEL
[ 0.000000] set_pte_vaddr ffffffff8208b000 to 800000201c09b063
[ 0.000000] remapped 548323 so far
[ 0.000000] set_pte_mfn buf=0xffffffff8208b000 xen_remap_mfn=0x201c060 PAGE_KERNEL
[ 0.000000] set_pte_vaddr ffffffff8208b000 to 800000201c060063
[ 0.000000] remapped 548331 so far
[ 0.000000] final xen_del_extra_mem pfn_s=0x480000 len=0x85deb
[ 0.000000] xen_del_extra_mem(480000, 85deb)
[ 0.000000] xen_del_extra_mem i=0
[ 0.000000] xen_del_extra_mem i=0 start_r=0x480000 size_r=0x85dea n_pfns=0x85deb
(XEN) traps.c:459:d0v0 Unhandled invalid opcode fault/trap [#6] on VCPU 0 [ec=0000]
(XEN) domain_crash_sync called from entry.S: fault at ffff82d0802286c3 create_bounce_frame+0x12b/0x13a
(XEN) Domain 0 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-4.5.4-pre x86_64 debug=n Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e033:[<ffffffff820002f7>]
(XEN) RFLAGS: 0000000000000202 EM: 1 CONTEXT: pv guest (d0v0)
(XEN) rax: 0000000000000044 rbx: 0000000000085deb rcx: 0000000000000000
(XEN) rdx: 0000000000000044 rsi: 0000000000000001 rdi: ffffffff816cbc7a
(XEN) rbp: ffffffff81b57d00 rsp: ffffffff81b57c58 r8: 0000000000000001
(XEN) r9: 0000000000000001 r10: 0000000000480000 r11: 00000000000009cd
(XEN) r12: 0000000000480000 r13: 0000000000085deb r14: ffffffff8208ca20
(XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 00000000001526f0
(XEN) cr3: 000000201d6e1000 cr2: ffffc9000282e000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033
(XEN) Guest stack trace from rsp=ffffffff81b57c58:
(XEN) 0000000000000000 00000000000009cd ffffffff820002f7 000000010000e030
(XEN) 0000000000010002 ffffffff81b57c90 000000000000e02b 0000000000085deb
(XEN) 0000000000505deb ffffffff81b57cb0 ffffffff810ca31a ffffffff81b57d10
(XEN) ffffffff811427fb 0000000000085dea ffffffff81b57d20 0000000000480000
(XEN) ffffffff81b57d20 0000000000085deb 0000000000085deb 0000000000085deb
(XEN) 0000000000480000 ffffffff81b57d90 ffffffff820009f8 ffffffff81b57d90
(XEN) ffffffff820009f8 00003ffffffff000 0000000000480000 000000000201c067
(XEN) ffffffffffc05de3 0000000000085deb 000000000201e08b 000000201c067001
(XEN) 0000000000480007 ffff880014affff8 ffffffff81b57db0 ffff880014b00ff8
(XEN) 0000010000000000 00003ffffffff000 00003ffffffff000 ffffffff81b57e20
(XEN) ffffffff82003355 ffffffff81b57e20 ffffffff82003355 0000000000000f7d
(XEN) 0000000000000000 ffffffff00000000 ffff880000000000 0000000000001000
(XEN) 000000047dc0a000 0000000400000000 ffff880014aff000 ffff880014afe6dc
(XEN) ffffffff81b57e40 0000000000000000 0000000004000000 0000000000000000
(XEN) 0000000014aff000 ffffffff81b57ea0 ffffffff8200604d ffffffff81b57ea0
(XEN) ffffffff8200604d ffffffff81b57e50 ffffffff810ca31a 0000000004000000
(XEN) ffffffff811427fb 0000000010afe6dc ffffffff81b57ec0 0000000000000000
(XEN) ffffffff81b57ec0 ffffffff81b57f38 0000000000000000 0000000010afe6dc
(XEN) 0000000000000000 ffffffff81b57ef0 ffffffff81ffccd6 ffffffff81b57ef0
(XEN) ffffffff81ffccd6 0000000000000000 ffffffff820930a0 0000000000000000
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
And again with dom0_mem=18432M,max:18433M.
...
[ 0.000000] xen_count_remap_pages(max_pfn=0x480000) == 0x85dea
[ 0.000000] max_pages 0x505eea
[ 0.000000] xen_add_extra_mem(480000, 85eea)
[ 0.000000] memblock_reserve(0x480000000, 0x85eea000)
[ 0.000000] Released 0 page(s)
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] Xen: [mem 0x0000000000000000-0x000000000005ffff] usable
[ 0.000000] Xen: [mem 0x0000000000060000-0x0000000000067fff] reserved
[ 0.000000] Xen: [mem 0x0000000000068000-0x000000000009afff] usable
[ 0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[ 0.000000] Xen: [mem 0x0000000000100000-0x0000000075afdfff] usable
[ 0.000000] Xen: [mem 0x0000000075bab000-0x000000007a1f5fff] usable
[ 0.000000] Xen: [mem 0x000000007b7d7000-0x000000007b7fffff] usable
[ 0.000000] Xen: [mem 0x000000007bf00000-0x000000007bffffff] usable
[ 0.000000] Xen: [mem 0x00000000c7ffc000-0x00000000c7ffcfff] reserved
[ 0.000000] Xen: [mem 0x00000000fbffc000-0x00000000fbffcfff] reserved
[ 0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec01fff] reserved
[ 0.000000] Xen: [mem 0x00000000fec40000-0x00000000fec40fff] reserved
[ 0.000000] Xen: [mem 0x00000000fed20000-0x00000000fed2ffff] usable
[ 0.000000] Xen: [mem 0x00000000fee00000-0x00000000feefffff] reserved
[ 0.000000] Xen: [mem 0x0000000100000000-0x0000000505ee9fff] usable
[ 0.000000] bootconsole [xenboot0] enabled
[ 0.000000] debug: ignoring loglevel setting.
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] e820: user-defined physical RAM map:
[ 0.000000] user: [mem 0x0000000000000000-0x000000000005ffff] usable
[ 0.000000] user: [mem 0x0000000000060000-0x0000000000067fff] reserved
[ 0.000000] user: [mem 0x0000000000068000-0x000000000009afff] usable
[ 0.000000] user: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[ 0.000000] user: [mem 0x0000000000100000-0x0000000075afdfff] usable
[ 0.000000] user: [mem 0x0000000075bab000-0x000000007a1f5fff] usable
[ 0.000000] user: [mem 0x000000007b7d7000-0x000000007b7fffff] usable
[ 0.000000] user: [mem 0x000000007bf00000-0x000000007bffffff] usable
[ 0.000000] user: [mem 0x0000000080000000-0x000000008fffffff] reserved
[ 0.000000] user: [mem 0x00000000c7ffc000-0x00000000c7ffcfff] reserved
[ 0.000000] user: [mem 0x00000000fbffc000-0x00000000fbffcfff] reserved
[ 0.000000] user: [mem 0x00000000fec00000-0x00000000fec01fff] reserved
[ 0.000000] user: [mem 0x00000000fec40000-0x00000000fec40fff] reserved
[ 0.000000] user: [mem 0x00000000fed20000-0x00000000fed2ffff] usable
[ 0.000000] user: [mem 0x00000000fee00000-0x00000000feefffff] reserved
[ 0.000000] user: [mem 0x0000000100000000-0x0000000505ee9fff] usable
[ 0.000000] SMBIOS 2.7 present.
[ 0.000000] DMI: Skyport Systems CloudDock/S2600WTT, BIOS SE5C610.86B.01.01.0009.C1.060120151350 06/01/2015
[ 0.000000] Hypervisor detected: Xen
[ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[ 0.000000] e820: last_pfn = 0x505eea max_arch_pfn = 0x400000000
[ 0.000000] MTRR: Disabled
[ 0.000000] e820: last_pfn = 0xfed30 max_arch_pfn = 0x400000000
[ 0.000000] Scanning 1 areas for low memory corruption
[ 0.000000] Base memory trampoline at [ffff880000095000] 95000 size 24576
[ 0.000000] BRK [0x02f67000, 0x02f67fff] PGTABLE
[ 0.000000] BRK [0x02f68000, 0x02f68fff] PGTABLE
[ 0.000000] BRK [0x02f69000, 0x02f69fff] PGTABLE
[ 0.000000] BRK [0x02f6a000, 0x02f6afff] PGTABLE
[ 0.000000] BRK [0x02f6b000, 0x02f6bfff] PGTABLE
[ 0.000000] BRK [0x02f6c000, 0x02f6cfff] PGTABLE
[ 0.000000] RAMDISK: [mem 0x04000000-0x14afefff]
[ 0.000000] ACPI: Early table checksum verification disabled
[ 0.000000] ACPI: RSDP 0x00000000000F0460 000024 (v02 INTEL )
[ 0.000000] ACPI: XSDT 0x000000007B78D188 0000CC (v01 INTEL S2600WT 00000000 INTL 01000013)
[ 0.000000] ACPI: FACP 0x000000007B7D2000 0000F4 (v04 INTEL S2600WT 00000000 INTL 20091013)
[ 0.000000] ACPI: DSDT 0x000000007B798000 0313B2 (v02 INTEL S2600WT 00000003 INTL 20091013)
[ 0.000000] ACPI: FACS 0x000000007B6FC000 000040
[ 0.000000] ACPI: HPET 0x000000007B7D1000 000038 (v01 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: APIC 0x000000007B7D0000 00085C (v03 INTEL S2600WT 00000000 INTL 20091013)
[ 0.000000] ACPI: MCFG 0x000000007B7CF000 00003C (v01 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: MSCT 0x000000007B7CE000 000090 (v01 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: SLIT 0x000000007B7CD000 00006C (v01 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: SRAT 0x000000007B7CC000 000730 (v03 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: SPMI 0x000000007B7CB000 000041 (v05 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: WDDT 0x000000007B7CA000 000040 (v01 INTEL S2600WT 00000000 INTL 20091013)
[ 0.000000] ACPI: PRAD 0x000000007B793000 000102 (v02 INTEL SpsPrAgg 00000002 INTL 20130328)
[ 0.000000] ACPI: XMAR 0x000000007B7D5000 000118 (v01 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] ACPI: TCPA 0x000000007B7D4000 000064 (v02 INTEL S2600WT 00000002 INTL 01000013)
[ 0.000000] ACPI: SSDT 0x000000007B7D3000 0003A2 (v02 INTEL S2600WT 00001000 INTL 20130328)
[ 0.000000] ACPI: SSDT 0x0000000075AFE000 0ACA42 (v02 INTEL S2600WT 00004000 INTL 20130328)
[ 0.000000] ACPI: SSDT 0x000000007B795000 00216B (v02 INTEL S2600WT 00000002 INTL 20130328)
[ 0.000000] ACPI: SSDT 0x000000007B794000 000064 (v02 INTEL S2600WT 00000002 INTL 20130328)
[ 0.000000] ACPI: HEST 0x000000007B792000 0000A8 (v01 INTEL S2600WT 00000001 INTL 00000001)
[ 0.000000] ACPI: BERT 0x000000007B791000 000030 (v01 INTEL S2600WT 00000001 INTL 00000001)
[ 0.000000] ACPI: ERST 0x000000007B790000 000230 (v01 INTEL S2600WT 00000001 INTL 00000001)
[ 0.000000] ACPI: EINJ 0x000000007B78F000 000130 (v01 INTEL S2600WT 00000001 INTL 00000001)
[ 0.000000] ACPI: SPCR 0x000000007B78E000 000050 (v01 INTEL S2600WT 00000000 INTL 00000000)
[ 0.000000] ACPI: Local APIC address 0xfee00000
[ 0.000000] Setting APIC routing to Xen PV.
[ 0.000000] set_pte_vaddr ffffffffff5f9000 to 800000201e103063
[ 0.000000] NUMA turned off
[ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000505ee9fff]
[ 0.000000] NODE_DATA(0) allocated [mem 0x47dc04000-0x47dc08fff]
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff]
[ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff]
[ 0.000000] Normal [mem 0x0000000100000000-0x0000000505ee9fff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000001000-0x000000000005ffff]
[ 0.000000] node 0: [mem 0x0000000000068000-0x000000000009afff]
[ 0.000000] node 0: [mem 0x0000000000100000-0x0000000075afdfff]
[ 0.000000] node 0: [mem 0x0000000075bab000-0x000000007a1f5fff]
[ 0.000000] node 0: [mem 0x000000007b7d7000-0x000000007b7fffff]
[ 0.000000] node 0: [mem 0x000000007bf00000-0x000000007bffffff]
[ 0.000000] node 0: [mem 0x00000000fed20000-0x00000000fed2ffff]
[ 0.000000] node 0: [mem 0x0000000100000000-0x0000000505ee9fff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000505ee9fff]
[ 0.000000] On node 0 totalpages: 4718846
[ 0.000000] DMA zone: 64 pages used for memmap
[ 0.000000] DMA zone: 21 pages reserved
[ 0.000000] DMA zone: 3986 pages, LIFO batch:0
[ 0.000000] DMA32 zone: 7755 pages used for memmap
[ 0.000000] DMA32 zone: 496258 pages, LIFO batch:31
[ 0.000000] Normal zone: 65916 pages used for memmap
[ 0.000000] Normal zone: 4218602 pages, LIFO batch:31
[ 0.000000] p2m virtual area at ffffc90000000000, size is 2a00000
...
[ 0.000000] set_pte_mfn buf=0xffffffff8208b000 xen_remap_mfn=0x201c09b PAGE_KERNEL
[ 0.000000] set_pte_vaddr ffffffff8208b000 to 800000201c09b063
[ 0.000000] remapped 548323 so far
[ 0.000000] set_pte_mfn buf=0xffffffff8208b000 xen_remap_mfn=0x201c060 PAGE_KERNEL
[ 0.000000] set_pte_vaddr ffffffff8208b000 to 800000201c060063
[ 0.000000] remapped 548331 so far
[ 0.000000] final xen_del_extra_mem pfn_s=0x480000 len=0x85deb
[ 0.000000] xen_del_extra_mem(480000, 85deb)
[ 0.000000] xen_del_extra_mem i=0
[ 0.000000] xen_del_extra_mem i=0 start_r=0x480000 size_r=0x85eea n_pfns=0x85deb
[ 0.000000] memblock_free(0x480000000, 0x85deb000)
[ 0.000000] final set_pte_mfn buf=0xffffffff8208b000 mfn_save=0x201e08b PAGE_KERNEL
[ 0.000000] set_pte_vaddr ffffffff8208b000 to 800000201e08b063
[ 0.000000] Remapped 548331 page(s)
...
--Ed
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Linux 4.4 boot crash on xen 4.5.3 with dom0_mem == max
2016-05-17 13:37 ` Ed Swierk
@ 2016-05-17 20:50 ` Ed Swierk
2016-05-18 8:30 ` Juergen Gross
2016-05-18 11:44 ` Juergen Gross
0 siblings, 2 replies; 8+ messages in thread
From: Ed Swierk @ 2016-05-17 20:50 UTC (permalink / raw)
To: xen-devel, jgross; +Cc: eswierk
I added some more instrumentation and discovered that the result of
xen_count_remap_pages() (0x85dea) is one less than the actual number
of pages remapped by xen_set_identity_and_remap() (0x85deb).
The two functions differ in their handling of a xen_e820_map entry
whose size is not a multiple of the page size. The entry starting at
0x68000 has size 0x33400. xen_count_remap_pages() rounds up when
computing the end_pfn (to 0x9c), while xen_set_identity_and_remap()
rounds down (to 0x9b). Thus xen_count_remap_pages() counts the
remapped space following the entry as one page smaller than the other
function does.
(Confusingly, the "BIOS-provided physical RAM map" shows both the
start and end addresses rounded down to page boundaries, rather than
the addresses actually provided by the BIOS.)
[ 0.000000] xen_count_remap_pages i=0 addr=0x0 size=0x60000 start_pfn=0x0 end_pfn=0x0
[ 0.000000] xen_count_remap_pages i=0 plus end_pfn=0x60 type=0x1
[ 0.000000] xen_count_remap_pages i=1 addr=0x60000 size=0x8000 start_pfn=0x60 end_pfn=0x60
[ 0.000000] xen_count_remap_pages i=1 plus end_pfn=0x68 type=0x2
[ 0.000000] xen_count_remap_pages i=2 addr=0x68000 size=0x33400 start_pfn=0x68 end_pfn=0x68
[ 0.000000] xen_count_remap_pages i=2 plus end_pfn=0x9c type=0x1
[ 0.000000] xen_count_remap_pages i=3 addr=0x100000 size=0x759fe000 start_pfn=0x100 end_pfn=0x9c
[ 0.000000] xen_count_remap_pages i=3 plus end_pfn=0x75afe type=0x1
[ 0.000000] xen_count_remap_pages i=4 addr=0x75bab000 size=0x464b000 start_pfn=0x75bab end_pfn=0x75afe
[ 0.000000] xen_count_remap_pages i=4 plus end_pfn=0x7a1f6 type=0x1
[ 0.000000] xen_count_remap_pages i=5 addr=0x7b7d7000 size=0x29000 start_pfn=0x7b7d7 end_pfn=0x7a1f6
[ 0.000000] xen_count_remap_pages i=5 plus end_pfn=0x7b800 type=0x1
[ 0.000000] xen_count_remap_pages i=6 addr=0x7bf00000 size=0x100000 start_pfn=0x7bf00 end_pfn=0x7b800
[ 0.000000] xen_count_remap_pages i=6 plus end_pfn=0x7c000 type=0x1
[ 0.000000] xen_count_remap_pages i=7 addr=0xc7ffc000 size=0x1000 start_pfn=0xc7ffc end_pfn=0x7c000
[ 0.000000] xen_count_remap_pages i=7 plus end_pfn=0xc7ffd type=0x2
[ 0.000000] xen_count_remap_pages i=8 addr=0xfbffc000 size=0x1000 start_pfn=0xfbffc end_pfn=0xc7ffd
[ 0.000000] xen_count_remap_pages i=8 plus end_pfn=0xfbffd type=0x2
[ 0.000000] xen_count_remap_pages i=9 addr=0xfec00000 size=0x2000 start_pfn=0xfec00 end_pfn=0xfbffd
[ 0.000000] xen_count_remap_pages i=9 plus end_pfn=0xfec02 type=0x2
[ 0.000000] xen_count_remap_pages i=10 addr=0xfec40000 size=0x1000 start_pfn=0xfec40 end_pfn=0xfec02
[ 0.000000] xen_count_remap_pages i=10 plus end_pfn=0xfec41 type=0x2
[ 0.000000] xen_count_remap_pages i=11 addr=0xfed20000 size=0x10000 start_pfn=0xfed20 end_pfn=0xfec41
[ 0.000000] xen_count_remap_pages i=11 plus end_pfn=0xfed30 type=0x1
[ 0.000000] xen_count_remap_pages i=12 addr=0xfee00000 size=0x100000 start_pfn=0xfee00 end_pfn=0xfed30
[ 0.000000] xen_count_remap_pages i=12 plus end_pfn=0xfef00 type=0x2
[ 0.000000] xen_count_remap_pages i=13 addr=0x100000000 size=0x1f80000000 start_pfn=0x100000 end_pfn=0xfef00
[ 0.000000] xen_count_remap_pages i=13 plus end_pfn=0x480000 type=0x1
[ 0.000000] xen_count_remap_pages(max_pfn=0x480000) == 0x85dea
[ 0.000000] max_pages 0x505dea
[ 0.000000] xen_add_extra_mem(480000, 85dea)
[ 0.000000] memblock_reserve(0x480000000, 0x85dea000)
[ 0.000000] xen_set_identity_and_remap i=0 addr=0x0 size=0x60000 type=0x1
[ 0.000000] xen_set_identity_and_remap i=1 addr=0x60000 size=0x8000 type=0x2
[ 0.000000] xen_set_identity_and_remap i=2 addr=0x68000 size=0x33400 type=0x1
[ 0.000000] xen_set_identity_and_remap_chunk start_pfn=0x60 end_pfn=0x68 nr_pages=0x480000 remap_pfn=0x480000
[ 0.000000] xen_set_identity_and_remap_chunk i=0x0 cur_pfn=0x60 left=0x8 size=0x8 remap_range_size=0x1c00000
[ 0.000000] xen_set_identity_and_remap i=3 addr=0x100000 size=0x759fe000 type=0x1
[ 0.000000] xen_set_identity_and_remap_chunk start_pfn=0x9b end_pfn=0x100 nr_pages=0x480000 remap_pfn=0x480008
[ 0.000000] xen_set_identity_and_remap_chunk i=0x0 cur_pfn=0x9b left=0x65 size=0x65 remap_range_size=0x1bffff8
[ 0.000000] xen_set_identity_and_remap i=4 addr=0x75bab000 size=0x464b000 type=0x1
[ 0.000000] xen_set_identity_and_remap_chunk start_pfn=0x75afe end_pfn=0x75bab nr_pages=0x480000 remap_pfn=0x48006d
[ 0.000000] xen_set_identity_and_remap_chunk i=0x0 cur_pfn=0x75afe left=0xad size=0xad remap_range_size=0x1bfff93
[ 0.000000] xen_set_identity_and_remap i=5 addr=0x7b7d7000 size=0x29000 type=0x1
[ 0.000000] xen_set_identity_and_remap_chunk start_pfn=0x7a1f6 end_pfn=0x7b7d7 nr_pages=0x480000 remap_pfn=0x48011a
[ 0.000000] xen_set_identity_and_remap_chunk i=0x0 cur_pfn=0x7a1f6 left=0x15e1 size=0x15e1 remap_range_size=0x1bffee6
[ 0.000000] xen_set_identity_and_remap i=6 addr=0x7bf00000 size=0x100000 type=0x1
[ 0.000000] xen_set_identity_and_remap_chunk start_pfn=0x7b800 end_pfn=0x7bf00 nr_pages=0x480000 remap_pfn=0x4816fb
[ 0.000000] xen_set_identity_and_remap_chunk i=0x0 cur_pfn=0x7b800 left=0x700 size=0x700 remap_range_size=0x1bfe905
[ 0.000000] xen_set_identity_and_remap i=7 addr=0xc7ffc000 size=0x1000 type=0x2
[ 0.000000] xen_set_identity_and_remap i=8 addr=0xfbffc000 size=0x1000 type=0x2
[ 0.000000] xen_set_identity_and_remap i=9 addr=0xfec00000 size=0x2000 type=0x2
[ 0.000000] xen_set_identity_and_remap i=10 addr=0xfec40000 size=0x1000 type=0x2
[ 0.000000] xen_set_identity_and_remap i=11 addr=0xfed20000 size=0x10000 type=0x1
[ 0.000000] xen_set_identity_and_remap_chunk start_pfn=0x7c000 end_pfn=0xfed20 nr_pages=0x480000 remap_pfn=0x481dfb
[ 0.000000] xen_set_identity_and_remap_chunk i=0x0 cur_pfn=0x7c000 left=0x82d20 size=0x82d20 remap_range_size=0x1bfe205
[ 0.000000] xen_set_identity_and_remap i=12 addr=0xfee00000 size=0x100000 type=0x2
[ 0.000000] xen_set_identity_and_remap i=13 addr=0x100000000 size=0x1f80000000 type=0x1
[ 0.000000] xen_set_identity_and_remap_chunk start_pfn=0xfed30 end_pfn=0x100000 nr_pages=0x480000 remap_pfn=0x504b1b
[ 0.000000] xen_set_identity_and_remap_chunk i=0x0 cur_pfn=0xfed30 left=0x12d0 size=0x12d0 remap_range_size=0x1b7b4e5
[ 0.000000] Released 0 page(s)
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] Xen: [mem 0x0000000000000000-0x000000000005ffff] usable
[ 0.000000] Xen: [mem 0x0000000000060000-0x0000000000067fff] reserved
[ 0.000000] Xen: [mem 0x0000000000068000-0x000000000009afff] usable
[ 0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[ 0.000000] Xen: [mem 0x0000000000100000-0x0000000075afdfff] usable
[ 0.000000] Xen: [mem 0x0000000075bab000-0x000000007a1f5fff] usable
[ 0.000000] Xen: [mem 0x000000007b7d7000-0x000000007b7fffff] usable
[ 0.000000] Xen: [mem 0x000000007bf00000-0x000000007bffffff] usable
[ 0.000000] Xen: [mem 0x00000000c7ffc000-0x00000000c7ffcfff] reserved
[ 0.000000] Xen: [mem 0x00000000fbffc000-0x00000000fbffcfff] reserved
[ 0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec01fff] reserved
[ 0.000000] Xen: [mem 0x00000000fec40000-0x00000000fec40fff] reserved
[ 0.000000] Xen: [mem 0x00000000fed20000-0x00000000fed2ffff] usable
[ 0.000000] Xen: [mem 0x00000000fee00000-0x00000000feefffff] reserved
[ 0.000000] Xen: [mem 0x0000000100000000-0x0000000505de9fff] usable
[ 0.000000] bootconsole [xenboot0] enabled
[ 0.000000] debug: ignoring loglevel setting.
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] e820: user-defined physical RAM map:
[ 0.000000] user: [mem 0x0000000000000000-0x000000000005ffff] usable
[ 0.000000] user: [mem 0x0000000000060000-0x0000000000067fff] reserved
[ 0.000000] user: [mem 0x0000000000068000-0x000000000009afff] usable
[ 0.000000] user: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[ 0.000000] user: [mem 0x0000000000100000-0x0000000075afdfff] usable
[ 0.000000] user: [mem 0x0000000075bab000-0x000000007a1f5fff] usable
[ 0.000000] user: [mem 0x000000007b7d7000-0x000000007b7fffff] usable
[ 0.000000] user: [mem 0x000000007bf00000-0x000000007bffffff] usable
[ 0.000000] user: [mem 0x0000000080000000-0x000000008fffffff] reserved
[ 0.000000] user: [mem 0x00000000c7ffc000-0x00000000c7ffcfff] reserved
[ 0.000000] user: [mem 0x00000000fbffc000-0x00000000fbffcfff] reserved
[ 0.000000] user: [mem 0x00000000fec00000-0x00000000fec01fff] reserved
[ 0.000000] user: [mem 0x00000000fec40000-0x00000000fec40fff] reserved
[ 0.000000] user: [mem 0x00000000fed20000-0x00000000fed2ffff] usable
[ 0.000000] user: [mem 0x00000000fee00000-0x00000000feefffff] reserved
[ 0.000000] user: [mem 0x0000000100000000-0x0000000505de9fff] usable
It seems to me we'd be better off with just one function counting
remapped pages rather than two, so as an experiment I tried combining
them. The patch below adds a do_remap argument to
xen_set_identity_and_remap() that tells it whether to do the actual
remapping or just return the number pages that would be remapped.
This change fixes the crash on my system.
diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
index 7ab2951..2746b7b 100644
--- a/arch/x86/xen/setup.c
+++ b/arch/x86/xen/setup.c
@@ -438,11 +438,13 @@ static unsigned long __init xen_set_identity_and_remap_chunk(
return remap_pfn;
}
-static void __init xen_set_identity_and_remap(unsigned long nr_pages)
+static unsigned long __init xen_set_identity_and_remap(unsigned long nr_pages,
+ int do_remap)
{
phys_addr_t start = 0;
unsigned long last_pfn = nr_pages;
const struct e820entry *entry = xen_e820_map;
+ unsigned long remap_pages = 0;
int i;
/*
@@ -465,15 +467,21 @@ static void __init xen_set_identity_and_remap(unsigned long nr_pages)
if (entry->type == E820_RAM)
end_pfn = PFN_UP(entry->addr);
- if (start_pfn < end_pfn)
- last_pfn = xen_set_identity_and_remap_chunk(
- start_pfn, end_pfn, nr_pages,
- last_pfn);
+ if (start_pfn < end_pfn) {
+ if (do_remap)
+ last_pfn = xen_set_identity_and_remap_chunk(
+ start_pfn, end_pfn, nr_pages,
+ last_pfn);
+ remap_pages += end_pfn - start_pfn;
+ }
start = end;
}
}
- pr_info("Released %ld page(s)\n", xen_released_pages);
+ if (do_remap)
+ pr_info("Released %ld page(s)\n", xen_released_pages);
+
+ return remap_pages;
}
/*
@@ -596,35 +604,6 @@ static void __init xen_ignore_unusable(void)
}
}
-static unsigned long __init xen_count_remap_pages(unsigned long max_pfn)
-{
- unsigned long extra = 0;
- unsigned long start_pfn, end_pfn;
- const struct e820entry *entry = xen_e820_map;
- int i;
-
- end_pfn = 0;
- for (i = 0; i < xen_e820_map_entries; i++, entry++) {
- start_pfn = PFN_DOWN(entry->addr);
- /* Adjacent regions on non-page boundaries handling! */
- end_pfn = min(end_pfn, start_pfn);
-
- if (start_pfn >= max_pfn)
- return extra + max_pfn - end_pfn;
-
- /* Add any holes in map to result. */
- extra += start_pfn - end_pfn;
-
- end_pfn = PFN_UP(entry->addr + entry->size);
- end_pfn = min(end_pfn, max_pfn);
-
- if (entry->type != E820_RAM)
- extra += end_pfn - start_pfn;
- }
-
- return extra;
-}
-
bool __init xen_is_e820_reserved(phys_addr_t start, phys_addr_t size)
{
struct e820entry *entry;
@@ -804,7 +783,7 @@ char * __init xen_memory_setup(void)
max_pages = xen_get_max_pages();
/* How many extra pages do we need due to remapping? */
- max_pages += xen_count_remap_pages(max_pfn);
+ max_pages += xen_set_identity_and_remap(max_pfn, 0);
if (max_pages > max_pfn)
extra_pages += max_pages - max_pfn;
@@ -922,7 +901,7 @@ char * __init xen_memory_setup(void)
* Set identity map on non-RAM pages and prepare remapping the
* underlying RAM.
*/
- xen_set_identity_and_remap(max_pfn);
+ xen_set_identity_and_remap(max_pfn, 1);
return "Xen";
}
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: Linux 4.4 boot crash on xen 4.5.3 with dom0_mem == max
2016-05-17 20:50 ` Ed Swierk
@ 2016-05-18 8:30 ` Juergen Gross
2016-05-18 11:44 ` Juergen Gross
1 sibling, 0 replies; 8+ messages in thread
From: Juergen Gross @ 2016-05-18 8:30 UTC (permalink / raw)
To: Ed Swierk, xen-devel
On 17/05/16 22:50, Ed Swierk wrote:
> I added some more instrumentation and discovered that the result of
> xen_count_remap_pages() (0x85dea) is one less than the actual number
> of pages remapped by xen_set_identity_and_remap() (0x85deb).
>
> The two functions differ in their handling of a xen_e820_map entry
> whose size is not a multiple of the page size. The entry starting at
> 0x68000 has size 0x33400. xen_count_remap_pages() rounds up when
> computing the end_pfn (to 0x9c), while xen_set_identity_and_remap()
> rounds down (to 0x9b). Thus xen_count_remap_pages() counts the
> remapped space following the entry as one page smaller than the other
> function does.
>
> (Confusingly, the "BIOS-provided physical RAM map" shows both the
> start and end addresses rounded down to page boundaries, rather than
> the addresses actually provided by the BIOS.)
>
> [ 0.000000] xen_count_remap_pages i=0 addr=0x0 size=0x60000 start_pfn=0x0 end_pfn=0x0
> [ 0.000000] xen_count_remap_pages i=0 plus end_pfn=0x60 type=0x1
> [ 0.000000] xen_count_remap_pages i=1 addr=0x60000 size=0x8000 start_pfn=0x60 end_pfn=0x60
> [ 0.000000] xen_count_remap_pages i=1 plus end_pfn=0x68 type=0x2
> [ 0.000000] xen_count_remap_pages i=2 addr=0x68000 size=0x33400 start_pfn=0x68 end_pfn=0x68
> [ 0.000000] xen_count_remap_pages i=2 plus end_pfn=0x9c type=0x1
> [ 0.000000] xen_count_remap_pages i=3 addr=0x100000 size=0x759fe000 start_pfn=0x100 end_pfn=0x9c
> [ 0.000000] xen_count_remap_pages i=3 plus end_pfn=0x75afe type=0x1
> [ 0.000000] xen_count_remap_pages i=4 addr=0x75bab000 size=0x464b000 start_pfn=0x75bab end_pfn=0x75afe
> [ 0.000000] xen_count_remap_pages i=4 plus end_pfn=0x7a1f6 type=0x1
> [ 0.000000] xen_count_remap_pages i=5 addr=0x7b7d7000 size=0x29000 start_pfn=0x7b7d7 end_pfn=0x7a1f6
> [ 0.000000] xen_count_remap_pages i=5 plus end_pfn=0x7b800 type=0x1
> [ 0.000000] xen_count_remap_pages i=6 addr=0x7bf00000 size=0x100000 start_pfn=0x7bf00 end_pfn=0x7b800
> [ 0.000000] xen_count_remap_pages i=6 plus end_pfn=0x7c000 type=0x1
> [ 0.000000] xen_count_remap_pages i=7 addr=0xc7ffc000 size=0x1000 start_pfn=0xc7ffc end_pfn=0x7c000
> [ 0.000000] xen_count_remap_pages i=7 plus end_pfn=0xc7ffd type=0x2
> [ 0.000000] xen_count_remap_pages i=8 addr=0xfbffc000 size=0x1000 start_pfn=0xfbffc end_pfn=0xc7ffd
> [ 0.000000] xen_count_remap_pages i=8 plus end_pfn=0xfbffd type=0x2
> [ 0.000000] xen_count_remap_pages i=9 addr=0xfec00000 size=0x2000 start_pfn=0xfec00 end_pfn=0xfbffd
> [ 0.000000] xen_count_remap_pages i=9 plus end_pfn=0xfec02 type=0x2
> [ 0.000000] xen_count_remap_pages i=10 addr=0xfec40000 size=0x1000 start_pfn=0xfec40 end_pfn=0xfec02
> [ 0.000000] xen_count_remap_pages i=10 plus end_pfn=0xfec41 type=0x2
> [ 0.000000] xen_count_remap_pages i=11 addr=0xfed20000 size=0x10000 start_pfn=0xfed20 end_pfn=0xfec41
> [ 0.000000] xen_count_remap_pages i=11 plus end_pfn=0xfed30 type=0x1
> [ 0.000000] xen_count_remap_pages i=12 addr=0xfee00000 size=0x100000 start_pfn=0xfee00 end_pfn=0xfed30
> [ 0.000000] xen_count_remap_pages i=12 plus end_pfn=0xfef00 type=0x2
> [ 0.000000] xen_count_remap_pages i=13 addr=0x100000000 size=0x1f80000000 start_pfn=0x100000 end_pfn=0xfef00
> [ 0.000000] xen_count_remap_pages i=13 plus end_pfn=0x480000 type=0x1
> [ 0.000000] xen_count_remap_pages(max_pfn=0x480000) == 0x85dea
> [ 0.000000] max_pages 0x505dea
> [ 0.000000] xen_add_extra_mem(480000, 85dea)
> [ 0.000000] memblock_reserve(0x480000000, 0x85dea000)
> [ 0.000000] xen_set_identity_and_remap i=0 addr=0x0 size=0x60000 type=0x1
> [ 0.000000] xen_set_identity_and_remap i=1 addr=0x60000 size=0x8000 type=0x2
> [ 0.000000] xen_set_identity_and_remap i=2 addr=0x68000 size=0x33400 type=0x1
> [ 0.000000] xen_set_identity_and_remap_chunk start_pfn=0x60 end_pfn=0x68 nr_pages=0x480000 remap_pfn=0x480000
> [ 0.000000] xen_set_identity_and_remap_chunk i=0x0 cur_pfn=0x60 left=0x8 size=0x8 remap_range_size=0x1c00000
> [ 0.000000] xen_set_identity_and_remap i=3 addr=0x100000 size=0x759fe000 type=0x1
> [ 0.000000] xen_set_identity_and_remap_chunk start_pfn=0x9b end_pfn=0x100 nr_pages=0x480000 remap_pfn=0x480008
> [ 0.000000] xen_set_identity_and_remap_chunk i=0x0 cur_pfn=0x9b left=0x65 size=0x65 remap_range_size=0x1bffff8
> [ 0.000000] xen_set_identity_and_remap i=4 addr=0x75bab000 size=0x464b000 type=0x1
> [ 0.000000] xen_set_identity_and_remap_chunk start_pfn=0x75afe end_pfn=0x75bab nr_pages=0x480000 remap_pfn=0x48006d
> [ 0.000000] xen_set_identity_and_remap_chunk i=0x0 cur_pfn=0x75afe left=0xad size=0xad remap_range_size=0x1bfff93
> [ 0.000000] xen_set_identity_and_remap i=5 addr=0x7b7d7000 size=0x29000 type=0x1
> [ 0.000000] xen_set_identity_and_remap_chunk start_pfn=0x7a1f6 end_pfn=0x7b7d7 nr_pages=0x480000 remap_pfn=0x48011a
> [ 0.000000] xen_set_identity_and_remap_chunk i=0x0 cur_pfn=0x7a1f6 left=0x15e1 size=0x15e1 remap_range_size=0x1bffee6
> [ 0.000000] xen_set_identity_and_remap i=6 addr=0x7bf00000 size=0x100000 type=0x1
> [ 0.000000] xen_set_identity_and_remap_chunk start_pfn=0x7b800 end_pfn=0x7bf00 nr_pages=0x480000 remap_pfn=0x4816fb
> [ 0.000000] xen_set_identity_and_remap_chunk i=0x0 cur_pfn=0x7b800 left=0x700 size=0x700 remap_range_size=0x1bfe905
> [ 0.000000] xen_set_identity_and_remap i=7 addr=0xc7ffc000 size=0x1000 type=0x2
> [ 0.000000] xen_set_identity_and_remap i=8 addr=0xfbffc000 size=0x1000 type=0x2
> [ 0.000000] xen_set_identity_and_remap i=9 addr=0xfec00000 size=0x2000 type=0x2
> [ 0.000000] xen_set_identity_and_remap i=10 addr=0xfec40000 size=0x1000 type=0x2
> [ 0.000000] xen_set_identity_and_remap i=11 addr=0xfed20000 size=0x10000 type=0x1
> [ 0.000000] xen_set_identity_and_remap_chunk start_pfn=0x7c000 end_pfn=0xfed20 nr_pages=0x480000 remap_pfn=0x481dfb
> [ 0.000000] xen_set_identity_and_remap_chunk i=0x0 cur_pfn=0x7c000 left=0x82d20 size=0x82d20 remap_range_size=0x1bfe205
> [ 0.000000] xen_set_identity_and_remap i=12 addr=0xfee00000 size=0x100000 type=0x2
> [ 0.000000] xen_set_identity_and_remap i=13 addr=0x100000000 size=0x1f80000000 type=0x1
> [ 0.000000] xen_set_identity_and_remap_chunk start_pfn=0xfed30 end_pfn=0x100000 nr_pages=0x480000 remap_pfn=0x504b1b
> [ 0.000000] xen_set_identity_and_remap_chunk i=0x0 cur_pfn=0xfed30 left=0x12d0 size=0x12d0 remap_range_size=0x1b7b4e5
> [ 0.000000] Released 0 page(s)
> [ 0.000000] e820: BIOS-provided physical RAM map:
> [ 0.000000] Xen: [mem 0x0000000000000000-0x000000000005ffff] usable
> [ 0.000000] Xen: [mem 0x0000000000060000-0x0000000000067fff] reserved
> [ 0.000000] Xen: [mem 0x0000000000068000-0x000000000009afff] usable
> [ 0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
> [ 0.000000] Xen: [mem 0x0000000000100000-0x0000000075afdfff] usable
> [ 0.000000] Xen: [mem 0x0000000075bab000-0x000000007a1f5fff] usable
> [ 0.000000] Xen: [mem 0x000000007b7d7000-0x000000007b7fffff] usable
> [ 0.000000] Xen: [mem 0x000000007bf00000-0x000000007bffffff] usable
> [ 0.000000] Xen: [mem 0x00000000c7ffc000-0x00000000c7ffcfff] reserved
> [ 0.000000] Xen: [mem 0x00000000fbffc000-0x00000000fbffcfff] reserved
> [ 0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec01fff] reserved
> [ 0.000000] Xen: [mem 0x00000000fec40000-0x00000000fec40fff] reserved
> [ 0.000000] Xen: [mem 0x00000000fed20000-0x00000000fed2ffff] usable
> [ 0.000000] Xen: [mem 0x00000000fee00000-0x00000000feefffff] reserved
> [ 0.000000] Xen: [mem 0x0000000100000000-0x0000000505de9fff] usable
> [ 0.000000] bootconsole [xenboot0] enabled
> [ 0.000000] debug: ignoring loglevel setting.
> [ 0.000000] NX (Execute Disable) protection: active
> [ 0.000000] e820: user-defined physical RAM map:
> [ 0.000000] user: [mem 0x0000000000000000-0x000000000005ffff] usable
> [ 0.000000] user: [mem 0x0000000000060000-0x0000000000067fff] reserved
> [ 0.000000] user: [mem 0x0000000000068000-0x000000000009afff] usable
> [ 0.000000] user: [mem 0x00000000000a0000-0x00000000000fffff] reserved
> [ 0.000000] user: [mem 0x0000000000100000-0x0000000075afdfff] usable
> [ 0.000000] user: [mem 0x0000000075bab000-0x000000007a1f5fff] usable
> [ 0.000000] user: [mem 0x000000007b7d7000-0x000000007b7fffff] usable
> [ 0.000000] user: [mem 0x000000007bf00000-0x000000007bffffff] usable
> [ 0.000000] user: [mem 0x0000000080000000-0x000000008fffffff] reserved
> [ 0.000000] user: [mem 0x00000000c7ffc000-0x00000000c7ffcfff] reserved
> [ 0.000000] user: [mem 0x00000000fbffc000-0x00000000fbffcfff] reserved
> [ 0.000000] user: [mem 0x00000000fec00000-0x00000000fec01fff] reserved
> [ 0.000000] user: [mem 0x00000000fec40000-0x00000000fec40fff] reserved
> [ 0.000000] user: [mem 0x00000000fed20000-0x00000000fed2ffff] usable
> [ 0.000000] user: [mem 0x00000000fee00000-0x00000000feefffff] reserved
> [ 0.000000] user: [mem 0x0000000100000000-0x0000000505de9fff] usable
>
> It seems to me we'd be better off with just one function counting
> remapped pages rather than two, so as an experiment I tried combining
> them. The patch below adds a do_remap argument to
> xen_set_identity_and_remap() that tells it whether to do the actual
> remapping or just return the number pages that would be remapped.
> This change fixes the crash on my system.
But it is wrong.
> diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
> index 7ab2951..2746b7b 100644
> --- a/arch/x86/xen/setup.c
> +++ b/arch/x86/xen/setup.c
> @@ -438,11 +438,13 @@ static unsigned long __init xen_set_identity_and_remap_chunk(
> return remap_pfn;
> }
>
> -static void __init xen_set_identity_and_remap(unsigned long nr_pages)
> +static unsigned long __init xen_set_identity_and_remap(unsigned long nr_pages,
> + int do_remap)
> {
> phys_addr_t start = 0;
> unsigned long last_pfn = nr_pages;
> const struct e820entry *entry = xen_e820_map;
> + unsigned long remap_pages = 0;
> int i;
>
> /*
> @@ -465,15 +467,21 @@ static void __init xen_set_identity_and_remap(unsigned long nr_pages)
> if (entry->type == E820_RAM)
> end_pfn = PFN_UP(entry->addr);
>
> - if (start_pfn < end_pfn)
> - last_pfn = xen_set_identity_and_remap_chunk(
> - start_pfn, end_pfn, nr_pages,
> - last_pfn);
> + if (start_pfn < end_pfn) {
> + if (do_remap)
> + last_pfn = xen_set_identity_and_remap_chunk(
> + start_pfn, end_pfn, nr_pages,
> + last_pfn);
> + remap_pages += end_pfn - start_pfn;
> + }
> start = end;
You fail to skip counting remap_pages above nr_pages.
> }
> }
>
> - pr_info("Released %ld page(s)\n", xen_released_pages);
> + if (do_remap)
> + pr_info("Released %ld page(s)\n", xen_released_pages);
> +
> + return remap_pages;
> }
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Linux 4.4 boot crash on xen 4.5.3 with dom0_mem == max
2016-05-17 20:50 ` Ed Swierk
2016-05-18 8:30 ` Juergen Gross
@ 2016-05-18 11:44 ` Juergen Gross
2016-05-18 14:39 ` Ed Swierk
1 sibling, 1 reply; 8+ messages in thread
From: Juergen Gross @ 2016-05-18 11:44 UTC (permalink / raw)
To: Ed Swierk, xen-devel
[-- Attachment #1: Type: text/plain, Size: 898 bytes --]
On 17/05/16 22:50, Ed Swierk wrote:
> I added some more instrumentation and discovered that the result of
> xen_count_remap_pages() (0x85dea) is one less than the actual number
> of pages remapped by xen_set_identity_and_remap() (0x85deb).
>
> The two functions differ in their handling of a xen_e820_map entry
> whose size is not a multiple of the page size. The entry starting at
> 0x68000 has size 0x33400. xen_count_remap_pages() rounds up when
> computing the end_pfn (to 0x9c), while xen_set_identity_and_remap()
> rounds down (to 0x9b). Thus xen_count_remap_pages() counts the
> remapped space following the entry as one page smaller than the other
> function does.
Could you please test the attached patch? It follows your idea of the
combined function, but is using a function pointer instead of a flag.
The patch is based on 4.6, but I believe it should just work on 4.4.
Juergen
[-- Attachment #2: 0001-xen-use-same-main-loop-for-counting-and-remapping-pa.patch --]
[-- Type: text/x-patch, Size: 4500 bytes --]
>From 0d665e37ef79c35cc71830a46d3cf1746b5d01ee Mon Sep 17 00:00:00 2001
From: Juergen Gross <jgross@suse.com>
Date: Wed, 18 May 2016 12:54:42 +0200
Subject: [PATCH] xen: use same main loop for counting and remapping pages
Instead of having two functions for cycling through the E820 map in
order to count to be remapped pages and remap them later, just use one
function with a caller supplied sub-function called for each region to
be processed. This eliminates the possibility of a mismatch between
both loops which showed up in certain configurations.
Suggested-by: Ed Swierk <eswierk@skyportsystems.com>
Signed-off-by: Juergen Gross <jgross@suse.com>
---
arch/x86/xen/setup.c | 65 +++++++++++++++++++++-------------------------------
1 file changed, 26 insertions(+), 39 deletions(-)
diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
index 7ab2951..42b8fda 100644
--- a/arch/x86/xen/setup.c
+++ b/arch/x86/xen/setup.c
@@ -393,6 +393,9 @@ static unsigned long __init xen_set_identity_and_remap_chunk(
unsigned long i = 0;
unsigned long n = end_pfn - start_pfn;
+ if (remap_pfn == 0)
+ remap_pfn = nr_pages;
+
while (i < n) {
unsigned long cur_pfn = start_pfn + i;
unsigned long left = n - i;
@@ -438,17 +441,29 @@ static unsigned long __init xen_set_identity_and_remap_chunk(
return remap_pfn;
}
-static void __init xen_set_identity_and_remap(unsigned long nr_pages)
+static unsigned long __init xen_count_remap_pages(
+ unsigned long start_pfn, unsigned long end_pfn, unsigned long nr_pages,
+ unsigned long remap_pages)
+{
+ if (start_pfn >= nr_pages)
+ return remap_pages;
+
+ return remap_pages + min(end_pfn, nr_pages) - start_pfn;
+}
+
+static unsigned long __init xen_foreach_remap_area(unsigned long nr_pages,
+ unsigned long (*func)(unsigned long start_pfn, unsigned long end_pfn,
+ unsigned long nr_pages, unsigned long last_val))
{
phys_addr_t start = 0;
- unsigned long last_pfn = nr_pages;
+ unsigned long ret_val = 0;
const struct e820entry *entry = xen_e820_map;
int i;
/*
* Combine non-RAM regions and gaps until a RAM region (or the
- * end of the map) is reached, then set the 1:1 map and
- * remap the memory in those non-RAM regions.
+ * end of the map) is reached, then call the provided function
+ * to perform it's duty on the non-RAM region.
*
* The combined non-RAM regions are rounded to a whole number
* of pages so any partial pages are accessible via the 1:1
@@ -466,14 +481,13 @@ static void __init xen_set_identity_and_remap(unsigned long nr_pages)
end_pfn = PFN_UP(entry->addr);
if (start_pfn < end_pfn)
- last_pfn = xen_set_identity_and_remap_chunk(
- start_pfn, end_pfn, nr_pages,
- last_pfn);
+ ret_val = func(start_pfn, end_pfn, nr_pages,
+ ret_val);
start = end;
}
}
- pr_info("Released %ld page(s)\n", xen_released_pages);
+ return ret_val;
}
/*
@@ -596,35 +610,6 @@ static void __init xen_ignore_unusable(void)
}
}
-static unsigned long __init xen_count_remap_pages(unsigned long max_pfn)
-{
- unsigned long extra = 0;
- unsigned long start_pfn, end_pfn;
- const struct e820entry *entry = xen_e820_map;
- int i;
-
- end_pfn = 0;
- for (i = 0; i < xen_e820_map_entries; i++, entry++) {
- start_pfn = PFN_DOWN(entry->addr);
- /* Adjacent regions on non-page boundaries handling! */
- end_pfn = min(end_pfn, start_pfn);
-
- if (start_pfn >= max_pfn)
- return extra + max_pfn - end_pfn;
-
- /* Add any holes in map to result. */
- extra += start_pfn - end_pfn;
-
- end_pfn = PFN_UP(entry->addr + entry->size);
- end_pfn = min(end_pfn, max_pfn);
-
- if (entry->type != E820_RAM)
- extra += end_pfn - start_pfn;
- }
-
- return extra;
-}
-
bool __init xen_is_e820_reserved(phys_addr_t start, phys_addr_t size)
{
struct e820entry *entry;
@@ -804,7 +789,7 @@ char * __init xen_memory_setup(void)
max_pages = xen_get_max_pages();
/* How many extra pages do we need due to remapping? */
- max_pages += xen_count_remap_pages(max_pfn);
+ max_pages += xen_foreach_remap_area(max_pfn, xen_count_remap_pages);
if (max_pages > max_pfn)
extra_pages += max_pages - max_pfn;
@@ -922,7 +907,9 @@ char * __init xen_memory_setup(void)
* Set identity map on non-RAM pages and prepare remapping the
* underlying RAM.
*/
- xen_set_identity_and_remap(max_pfn);
+ xen_foreach_remap_area(max_pfn, xen_set_identity_and_remap_chunk);
+
+ pr_info("Released %ld page(s)\n", xen_released_pages);
return "Xen";
}
--
2.6.6
[-- Attachment #3: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: Linux 4.4 boot crash on xen 4.5.3 with dom0_mem == max
2016-05-18 11:44 ` Juergen Gross
@ 2016-05-18 14:39 ` Ed Swierk
2016-05-18 14:45 ` Juergen Gross
0 siblings, 1 reply; 8+ messages in thread
From: Ed Swierk @ 2016-05-18 14:39 UTC (permalink / raw)
To: Juergen Gross; +Cc: xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 1165 bytes --]
Nice implementation. I tested it and it fixes the problem on the affected
system.
Just a minor typo in a comment: "it's duty" should be "its duty".
--Ed
On Wed, May 18, 2016 at 4:44 AM, Juergen Gross <jgross@suse.com> wrote:
> On 17/05/16 22:50, Ed Swierk wrote:
> > I added some more instrumentation and discovered that the result of
> > xen_count_remap_pages() (0x85dea) is one less than the actual number
> > of pages remapped by xen_set_identity_and_remap() (0x85deb).
> >
> > The two functions differ in their handling of a xen_e820_map entry
> > whose size is not a multiple of the page size. The entry starting at
> > 0x68000 has size 0x33400. xen_count_remap_pages() rounds up when
> > computing the end_pfn (to 0x9c), while xen_set_identity_and_remap()
> > rounds down (to 0x9b). Thus xen_count_remap_pages() counts the
> > remapped space following the entry as one page smaller than the other
> > function does.
>
> Could you please test the attached patch? It follows your idea of the
> combined function, but is using a function pointer instead of a flag.
> The patch is based on 4.6, but I believe it should just work on 4.4.
>
>
> Juergen
>
>
[-- Attachment #1.2: Type: text/html, Size: 1740 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Linux 4.4 boot crash on xen 4.5.3 with dom0_mem == max
2016-05-18 14:39 ` Ed Swierk
@ 2016-05-18 14:45 ` Juergen Gross
0 siblings, 0 replies; 8+ messages in thread
From: Juergen Gross @ 2016-05-18 14:45 UTC (permalink / raw)
To: Ed Swierk; +Cc: xen-devel
On 18/05/16 16:39, Ed Swierk wrote:
> Nice implementation. I tested it and it fixes the problem on the
> affected system.
>
> Just a minor typo in a comment: "it's duty" should be "its duty".
Thanks. Corrected and sent to lkml.
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2016-05-18 14:45 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-05-17 4:28 Linux 4.4 boot crash on xen 4.5.3 with dom0_mem == max Ed Swierk
2016-05-17 7:43 ` Juergen Gross
2016-05-17 13:37 ` Ed Swierk
2016-05-17 20:50 ` Ed Swierk
2016-05-18 8:30 ` Juergen Gross
2016-05-18 11:44 ` Juergen Gross
2016-05-18 14:39 ` Ed Swierk
2016-05-18 14:45 ` Juergen Gross
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.