Re: mmotm boot panic bootmem-avoid-dma32-zone-by-default.patch

[Yinghai Lu <yinghai@kernel.org>]

On 03/04/2010 09:00 PM, Yinghai Lu wrote:
> On 03/04/2010 07:21 PM, Johannes Weiner wrote:
>> Hello Greg,
>>
>> On Thu, Mar 04, 2010 at 01:21:41PM -0800, Greg Thelen wrote:
>>> On several systems I am seeing a boot panic if I use mmotm
>>> (stamp-2010-03-02-18-38).  If I remove
>>> bootmem-avoid-dma32-zone-by-default.patch then no panic is seen.  I
>>> find that:
>>> * 2.6.33 boots fine.
>>> * 2.6.33 + mmotm w/o bootmem-avoid-dma32-zone-by-default.patch: boots fine.
>>> * 2.6.33 + mmotm (including
>>> bootmem-avoid-dma32-zone-by-default.patch): panics.
>>> Note: I had to enable earlyprintk to see the panic.  Without
>>> earlyprintk no console output was seen.  The system appeared to hang
>>> after the loader.
>>
>> Thanks for your report.  A few notes below.
>>
>>> Here's the panic seen with earlyprintk using 2.6.33 + mmotm:
>>>
>>> Starting up ...
>>> [    0.000000] Initializing cgroup subsys cpuset
>>> [    0.000000] Initializing cgroup subsys cpu
>>> [    0.000000] Linux version 2.6.33-mm1+
>>> (gthelen@ninji.mtv.corp.google.com) (gcc version 4.2.4 (Ubuntu
>>> 4.2.4-1ubuntu4)) #1 SMP Thu Mar 4 12:03:29 PST 2010
>>> [    0.000000] Command line:
>>> root=UUID=a77f406a-7cc7-4f49-9cc2-818b2b4159ae ro console=tty0
>>> console=ttyS0,115200n8 earlyprintk=serial,ttyS0,9600
>>> [    0.000000] BIOS-provided physical RAM map:
>>> [    0.000000]  BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
>>> [    0.000000]  BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
>>> [    0.000000]  BIOS-e820: 00000000000e8000 - 0000000000100000 (reserved)
>>> [    0.000000]  BIOS-e820: 0000000000100000 - 000000000fff0000 (usable)
>>> [    0.000000]  BIOS-e820: 000000000fff0000 - 0000000010000000 (ACPI data)
>>> [    0.000000]  BIOS-e820: 00000000fffbd000 - 0000000100000000 (reserved)
>>> [    0.000000] bootconsole [earlyser0] enabled
>>> [    0.000000] NX (Execute Disable) protection: active
>>> [    0.000000] DMI 2.4 present.
>>> [    0.000000] No AGP bridge found
>>> [    0.000000] last_pfn = 0xfff0 max_arch_pfn = 0x400000000
>>> [    0.000000] PAT not supported by CPU.
>>> [    0.000000] CPU MTRRs all blank - virtualized system.
>>> [    0.000000] Scanning 1 areas for low memory corruption
>>> [    0.000000] modified physical RAM map:
>>> [    0.000000]  modified: 0000000000000000 - 0000000000010000 (reserved)
>>> [    0.000000]  modified: 0000000000010000 - 000000000009fc00 (usable)
>>> [    0.000000]  modified: 000000000009fc00 - 00000000000a0000 (reserved)
>>> [    0.000000]  modified: 00000000000e8000 - 0000000000100000 (reserved)
>>> [    0.000000]  modified: 0000000000100000 - 000000000fff0000 (usable)
>>> [    0.000000]  modified: 000000000fff0000 - 0000000010000000 (ACPI data)
>>> [    0.000000]  modified: 00000000fffbd000 - 0000000100000000 (reserved)
>>> [    0.000000] init_memory_mapping: 0000000000000000-000000000fff0000
>>
>> 256MB of memory, right?
>>
>>> [    0.000000] RAMDISK: 0fd9d000 - 0ffdf539
>>> [    0.000000] ACPI: RSDP 00000000000fb450 00014 (v00 QEMU  )
>>> [    0.000000] ACPI: RSDT 000000000fff0000 00030 (v01 QEMU   QEMURSDT
>>> 00000001 QEMU 00000001)
>>> [    0.000000] ACPI: FACP 000000000fff0030 00074 (v01 QEMU   QEMUFACP
>>> 00000001 QEMU 00000001)
>>> [    0.000000] ACPI: DSDT 000000000fff0100 0089D (v01   BXPC   BXDSDT
>>> 00000001 INTL 20061109)
>>> [    0.000000] ACPI: FACS 000000000fff00c0 00040
>>> [    0.000000] ACPI: APIC 000000000fff09d8 00068 (v01 QEMU   QEMUAPIC
>>> 00000001 QEMU 00000001)
>>> [    0.000000] ACPI: SSDT 000000000fff099d 00037 (v01 QEMU   QEMUSSDT
>>> 00000001 QEMU 00000001)
>>> [    0.000000] No NUMA configuration found
>>> [    0.000000] Faking a node at 0000000000000000-000000000fff0000
>>> [    0.000000] Initmem setup node 0 0000000000000000-000000000fff0000
>>> [    0.000000]   NODE_DATA [0000000001c4e040 - 0000000001c5303f]
>>> [    0.000000] BUG: unable to handle kernel NULL pointer dereference at (null)
>>> [    0.000000] IP: [<ffffffff81b0f5f7>] memory_present+0x9a/0xbf
>>> [    0.000000] PGD 0
>>> [    0.000000] Oops: 0000 [#1] SMP
>>> [    0.000000] last sysfs file:
>>> [    0.000000] CPU 0
>>> [    0.000000] Modules linked in:
>>> [    0.000000]
>>> [    0.000000] Pid: 0, comm: swapper Not tainted 2.6.33-mm1+ #1 /
>>> [    0.000000] RIP: 0010:[<ffffffff81b0f5f7>]  [<ffffffff81b0f5f7>]
>>> memory_present+0x9a/0xbf
>>> [    0.000000] RSP: 0000:ffffffff81a01e18  EFLAGS: 00010046
>>> [    0.000000] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000002
>>> [    0.000000] RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000000
>>> [    0.000000] RBP: ffffffff81a01e58 R08: ffffffffffffffff R09: 0000000000000040
>>> [    0.000000] R10: ffff880001c4e040 R11: 0000000000004100 R12: 0000000000000000
>>> [    0.000000] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
>>> [    0.000000] FS:  0000000000000000(0000) GS:ffffffff81adf000(0000)
>>> knlGS:0000000000000000
>>> [    0.000000] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [    0.000000] CR2: 0000000000000000 CR3: 0000000001a08000 CR4: 00000000000000b0
>>> [    0.000000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>> [    0.000000] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>> [    0.000000] Process swapper (pid: 0, threadinfo ffffffff81a00000,
>>> task ffffffff81a10020)
>>> [    0.000000] Stack:
>>> [    0.000000]  000000000fff0000 000000000000009f 0000000000000000
>>> 0000000000000000
>>> [    0.000000] <0> 0000000000000040 ffffffff81a01ef8 0000000000000000
>>> 0000000000000000
>>> [    0.000000] <0> ffffffff81a01e78 ffffffff81b0dd0e ffffffff81a01e88
>>> 000000000fff0000
>>> [    0.000000] Call Trace:
>>> [    0.000000]  [<ffffffff81b0dd0e>]
>>> sparse_memory_present_with_active_regions+0x31/0x47
>>> [    0.000000]  [<ffffffff81b0688a>] paging_init+0x3f/0x5b
>>> [    0.000000]  [<ffffffff81af81a7>] setup_arch+0x964/0xa03
>>> [    0.000000]  [<ffffffff8103014a>] ? need_resched+0x1e/0x28
>>> [    0.000000]  [<ffffffff8103015d>] ? should_resched+0x9/0x2a
>>> [    0.000000]  [<ffffffff8152de24>] ? _cond_resched+0x9/0x1d
>>> [    0.000000]  [<ffffffff81af4a34>] start_kernel+0x9f/0x382
>>> [    0.000000]  [<ffffffff81af4299>] x86_64_start_reservations+0xa9/0xad
>>> [    0.000000]  [<ffffffff81af4383>] x86_64_start_kernel+0xe6/0xed
>>> [    0.000000] Code: c7 00 56 c2 81 e8 a0 f9 a1 ff 48 83 3c dd 00 16
>>> c2 81 00 75 08 4c 89 2c dd 00 16 c2 81 fe 05 11 60 11 00 4c 89 ff e8
>>> 85 3b 5c ff <48> 83 38 00 75 03 4c 89 30 49 81 c4 00 80 00 00 4c 3b 65
>>> c8 72
>>> [    0.000000] RIP  [<ffffffff81b0f5f7>] memory_present+0x9a/0xbf
>>> [    0.000000]  RSP <ffffffff81a01e18>
>>> [    0.000000] CR2: 0000000000000000
>>> [    0.000000] ---[ end trace 4eaa2a86a8e2da22 ]---
>>> [    0.000000] Kernel panic - not syncing: Attempted to kill the idle task!
>>> [    0.000000] Pid: 0, comm: swapper Tainted: G      D    2.6.33-mm1+ #1
>>> [    0.000000] Call Trace:
>>> [    0.000000]  [<ffffffff8103c78c>] panic+0x9e/0x113
>>> [    0.000000]  [<ffffffff8103d3d6>] ? printk+0x67/0x69
>>> [    0.000000]  [<ffffffff8105914e>] ? blocking_notifier_call_chain+0xf/0x11
>>> [    0.000000]  [<ffffffff8103f8b4>] do_exit+0x78/0x70f
>>> [    0.000000]  [<ffffffff8103ca2f>] ? spin_unlock_irqrestore+0x9/0xb
>>> [    0.000000]  [<ffffffff8103dcde>] ? kmsg_dump+0x112/0x138
>>> [    0.000000]  [<ffffffff81530061>] oops_end+0xb2/0xba
>>> [    0.000000]  [<ffffffff810258d3>] no_context+0x1f5/0x204
>>> [    0.000000]  [<ffffffff81025b1b>] __bad_area_nosemaphore+0x17f/0x1a2
>>> [    0.000000]  [<ffffffff81025bb4>] bad_area_nosemaphore+0xe/0x10
>>> [    0.000000]  [<ffffffff81531e36>] do_page_fault+0x122/0x24c
>>> [    0.000000]  [<ffffffff8152f59f>] page_fault+0x1f/0x30
>>> [    0.000000]  [<ffffffff81b0f5f7>] ? memory_present+0x9a/0xbf
>>> [    0.000000]  [<ffffffff81b0f5f7>] ? memory_present+0x9a/0xbf
>>> [    0.000000]  [<ffffffff81b0dd0e>]
>>> sparse_memory_present_with_active_regions+0x31/0x47
>>> [    0.000000]  [<ffffffff81b0688a>] paging_init+0x3f/0x5b
>>> [    0.000000]  [<ffffffff81af81a7>] setup_arch+0x964/0xa03
>>> [    0.000000]  [<ffffffff8103014a>] ? need_resched+0x1e/0x28
>>> [    0.000000]  [<ffffffff8103015d>] ? should_resched+0x9/0x2a
>>> [    0.000000]  [<ffffffff8152de24>] ? _cond_resched+0x9/0x1d
>>> [    0.000000]  [<ffffffff81af4a34>] start_kernel+0x9f/0x382
>>> [    0.000000]  [<ffffffff81af4299>] x86_64_start_reservations+0xa9/0xad
>>> [    0.000000]  [<ffffffff81af4383>] x86_64_start_kernel+0xe6/0xed
>>>
>>> The kernel was built with 'make mrproper && make defconfig && make
>>> ARCH=x86_64 CONFIG=smp -j 6'.  This panic is seen on every attempt, so
>>> I can provide more diagnostics.
>>
>> Okay, if you did defconfig and just hit enter to all questions, you
>> should have SPARSEMEM_EXTREME and NO_BOOTMEM enabled.  This means that
>> the 'mem_section' is an array of pointers and the following happens in
>> memory_present():
>>
>> 	for_one_pfn_in_each_section() {
>> 		sparse_index_init(); /* no return value check */
>> 		ms = __nr_to_section();
>> 		if (!ms->section_mem_map) /* bang */
>> 			...;
>> 	}
>>
>> where sparse_index_init(), in the SPARSEMEM_EXTREME case, will allocate
>> the mem_section descriptor with bootmem.  If this would fail, the box
>> would panic immediately earlier, but NO_BOOTMEM does not seem to get it
>> right.
>>
>> Greg, could you retry _with_ my bootmem patch applied, but with setting
>> CONFIG_NO_BOOTMEM=n up front?
>>
>> I think NO_BOOTMEM has several problems.  Yinghai, can you verify them?
>>
>> 1. It does not seem to handle goal appropriately: bootmem would try
>> without the goal if it does not make sense.  And in this case, the
>> goal is 4G (above DMA32) and the amount of memory is 256M.
>>
>> And if I did not miss something, this is the difference with my patch:
>> without it, the default goal is 16M, which is no problem as it is well
>> within your available memory.  But the change of the default goal moved
>> it outside it which the bootmem replacement can not handle.
>>
>> 2. The early reservation stuff seems to return NULL but callsites assume
>> that the bootmem interface never does that.  Okay, the result is the same,
>> we crash.  But it still moves error reporting to a possibly much later
>> point where somebody actually dereferences the returned pointer.
> 
> related change could be: __alloc_bootmem_node_high...

no should be here...

static struct mem_section noinline __init_refok *sparse_index_alloc(int nid)
{                                     
        struct mem_section *section = NULL;
        unsigned long array_size = SECTIONS_PER_ROOT *
                                   sizeof(struct mem_section);

        if (slab_is_available()) {
                if (node_state(nid, N_HIGH_MEMORY))
                        section = kmalloc_node(array_size, GFP_KERNEL, nid);
                else
                        section = kmalloc(array_size, GFP_KERNEL);
        } else
                section = alloc_bootmem_node(NODE_DATA(nid), array_size);

and

#define alloc_bootmem_node(pgdat, x) \
        __alloc_bootmem_node(pgdat, x, SMP_CACHE_BYTES, __pa(MAX_DMA_ADDRESS))


then you change that goal MAX_DMA_ADDRESS to 4g..., but the system only have 256M 

YH

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>