kdump fails to load and crashes [was: mmotm 2009-11-01-10-01 uploaded]

kdump fails to load and crashes [was: mmotm 2009-11-01-10-01 uploaded]

[Jiri Slaby]

On 11/01/2009 07:07 PM, akpm@linux-foundation.org wrote:
> The mm-of-the-moment snapshot 2009-11-01-10-01 has been uploaded to

Hi, kdump loading crashes:
BUG: unable to handle kernel paging request at ffff8800010a7000
IP: [<ffffffff8101cd8b>] machine_kexec_prepare+0x16b/0x13e0
PGD 1806063 PUD 180a063 PMD 10001e1
Oops: 0003 [#1] SMP
last sysfs file: /sys/firmware/memmap/10/type
CPU 1
Modules linked in: fuse ath5k ath
Pid: 3250, comm: kexec Tainted: G        W  2.6.32-rc5-mm1_64 #869 To Be
Filled By O.E.M.
RIP: 0010:[<ffffffff8101cd8b>]  [<ffffffff8101cd8b>]
machine_kexec_prepare+0x16b/0x13e0
RSP: 0018:ffff8801c95d1e88  EFLAGS: 00010206
RAX: 00000000000001e3 RBX: ffff8801ca2aac00 RCX: 0000000040000000
RDX: 0000000000200000 RSI: ffff8800010a7000 RDI: ffff8800010a7000
RBP: ffff8801c95d1ee8 R08: 00000000000010a7 R09: 00000000010a7fff
R10: 0000000000000fff R11: 0000000004ffffff R12: 0000000000000000
R13: ffff8800010a6000 R14: 0000008000000000 R15: ffff880000000000
FS:  00007f4f97b4c6f0(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffff8800010a7000 CR3: 00000001c8cf2000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kexec (pid: 3250, threadinfo ffff8801c95d0000, task
ffff8801ca116700)
Stack:
ffff8801c95d1ee8 ffff8800010a6000 ffff8800010a4000 ffff8800010a4000
<0> 0000000000000000 00000001cc000000 ffff8801c95d1ee8 0000000000000006
<0> ffff8801ca2aac00 0000000000000000 0000000000000000 000000000146eae0
Call Trace:
[<ffffffff8106d32c>] sys_kexec_load+0xec/0x5e0
[<ffffffff814365b5>] ? do_page_fault+0x185/0x350
[<ffffffff81002e6b>] system_call_fastpath+0x16/0x1b
Code: 00 40 48 c1 e6 0c 4c 01 fe 48 39 ca 73 21 48 89 f7 0f 1f 80 00 00
00 00 48 89 d0 48 81 c2 00 00 20 00 48 0d e3 01 00 00 48 39 d1 <48> ab
77 e9 48 89 f7 49 81 c4 00 00 00 40 e8 e2 93 00 00 4c 89
RIP  [<ffffffff8101cd8b>] machine_kexec_prepare+0x16b/0x13e0
RSP <ffff8801c95d1e88>
CR2: ffff8800010a7000
---[ end trace 4eaa2a86a8e2da24 ]---

ffff8800010a7000 should be OK, as crashkernel=64M@16M was passed to the
kernel.

It's here:
static void init_level2_page(pmd_t *level2p, unsigned long addr)
{
        unsigned long end_addr;

        addr &= PAGE_MASK;
        end_addr = addr + PUD_SIZE;
        while (addr < end_addr) {
here ->         set_pmd(level2p++, __pmd(addr | __PAGE_KERNEL_LARGE_EXEC));
                addr += PMD_SIZE;
        }
}

Corresponding -fverbose-asm with -g:
        .loc 1 65 0
        addq    $2097152, %rdx  #, addr
.LVL31:
.LBB416:
        .loc 5 77 0
        orq     $483, %rax      #, tmp140
.LBE416:
        .loc 1 63 0
        cmpq    %rdx, %rcx      # addr, end_addr
.LBB417:
        .loc 5 77 0
        stosq
.LVL32:
.LBE417:
        .loc 1 63 0
        ja      .L26    #,




_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kdump fails to load and crashes [was: mmotm 2009-11-01-10-01 uploaded]

[Vivek Goyal]

On Tue, Nov 03, 2009 at 12:11:58AM +0100, Jiri Slaby wrote:
> On 11/01/2009 07:07 PM, akpm@linux-foundation.org wrote:
> > The mm-of-the-moment snapshot 2009-11-01-10-01 has been uploaded to
> 
> Hi, kdump loading crashes:
> BUG: unable to handle kernel paging request at ffff8800010a7000
> IP: [<ffffffff8101cd8b>] machine_kexec_prepare+0x16b/0x13e0
> PGD 1806063 PUD 180a063 PMD 10001e1
> Oops: 0003 [#1] SMP
> last sysfs file: /sys/firmware/memmap/10/type
> CPU 1
> Modules linked in: fuse ath5k ath
> Pid: 3250, comm: kexec Tainted: G        W  2.6.32-rc5-mm1_64 #869 To Be
> Filled By O.E.M.
> RIP: 0010:[<ffffffff8101cd8b>]  [<ffffffff8101cd8b>]
> machine_kexec_prepare+0x16b/0x13e0
> RSP: 0018:ffff8801c95d1e88  EFLAGS: 00010206
> RAX: 00000000000001e3 RBX: ffff8801ca2aac00 RCX: 0000000040000000
> RDX: 0000000000200000 RSI: ffff8800010a7000 RDI: ffff8800010a7000
> RBP: ffff8801c95d1ee8 R08: 00000000000010a7 R09: 00000000010a7fff
> R10: 0000000000000fff R11: 0000000004ffffff R12: 0000000000000000
> R13: ffff8800010a6000 R14: 0000008000000000 R15: ffff880000000000
> FS:  00007f4f97b4c6f0(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: ffff8800010a7000 CR3: 00000001c8cf2000 CR4: 00000000000006e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process kexec (pid: 3250, threadinfo ffff8801c95d0000, task
> ffff8801ca116700)
> Stack:
> ffff8801c95d1ee8 ffff8800010a6000 ffff8800010a4000 ffff8800010a4000
> <0> 0000000000000000 00000001cc000000 ffff8801c95d1ee8 0000000000000006
> <0> ffff8801ca2aac00 0000000000000000 0000000000000000 000000000146eae0
> Call Trace:
> [<ffffffff8106d32c>] sys_kexec_load+0xec/0x5e0
> [<ffffffff814365b5>] ? do_page_fault+0x185/0x350
> [<ffffffff81002e6b>] system_call_fastpath+0x16/0x1b
> Code: 00 40 48 c1 e6 0c 4c 01 fe 48 39 ca 73 21 48 89 f7 0f 1f 80 00 00
> 00 00 48 89 d0 48 81 c2 00 00 20 00 48 0d e3 01 00 00 48 39 d1 <48> ab
> 77 e9 48 89 f7 49 81 c4 00 00 00 40 e8 e2 93 00 00 4c 89
> RIP  [<ffffffff8101cd8b>] machine_kexec_prepare+0x16b/0x13e0
> RSP <ffff8801c95d1e88>
> CR2: ffff8800010a7000
> ---[ end trace 4eaa2a86a8e2da24 ]---
> 
> ffff8800010a7000 should be OK, as crashkernel=64M@16M was passed to the
> kernel.
> 

This is strange. As you said, ffff8800010a7000, should be a valid address
as you reserved memory from 16M.

I took above -mm and for me kernel loads fine. I reserved memory 64M@32M
as by default kernel is loading at address 16M.

How does your /proc/iomem look like in first kernel?

Thanks
Vivek

> It's here:
> static void init_level2_page(pmd_t *level2p, unsigned long addr)
> {
>         unsigned long end_addr;
> 
>         addr &= PAGE_MASK;
>         end_addr = addr + PUD_SIZE;
>         while (addr < end_addr) {
> here ->         set_pmd(level2p++, __pmd(addr | __PAGE_KERNEL_LARGE_EXEC));
>                 addr += PMD_SIZE;
>         }
> }
> 
> Corresponding -fverbose-asm with -g:
>         .loc 1 65 0
>         addq    $2097152, %rdx  #, addr
> .LVL31:
> .LBB416:
>         .loc 5 77 0
>         orq     $483, %rax      #, tmp140
> .LBE416:
>         .loc 1 63 0
>         cmpq    %rdx, %rcx      # addr, end_addr
> .LBB417:
>         .loc 5 77 0
>         stosq
> .LVL32:
> .LBE417:
>         .loc 1 63 0
>         ja      .L26    #,
> 
> 

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kdump fails to load and crashes [was: mmotm 2009-11-01-10-01 uploaded]

[Jiri Slaby]

On 11/03/2009 03:44 PM, Vivek Goyal wrote:
> On Tue, Nov 03, 2009 at 12:11:58AM +0100, Jiri Slaby wrote:
>> On 11/01/2009 07:07 PM, akpm@linux-foundation.org wrote:
>>> The mm-of-the-moment snapshot 2009-11-01-10-01 has been uploaded to
>>
>> Hi, kdump loading crashes:
>> BUG: unable to handle kernel paging request at ffff8800010a7000
>> IP: [<ffffffff8101cd8b>] machine_kexec_prepare+0x16b/0x13e0
>> PGD 1806063 PUD 180a063 PMD 10001e1
...
>> ffff8800010a7000 should be OK, as crashkernel=64M@16M was passed to the
>> kernel.
>>
> 
> This is strange. As you said, ffff8800010a7000, should be a valid address
> as you reserved memory from 16M.

Hmm, PMD has not RW bit set and we do store (see below).

> I took above -mm and for me kernel loads fine. I reserved memory 64M@32M
> as by default kernel is loading at address 16M.
> 
> How does your /proc/iomem look like in first kernel?

I'm not any longer on that machine. init_pgtable() overwrites random
memory, so possibly even host's page tables, hence the oops above, I
suppose.

I'm currently chasing it down in the qemu virtual machine, since I
already got a garbage into my ~/.viminfo (some overwritten page got
flushed, I guess).

There /proc/iomem looks like:
00100000-12beffff : System RAM
  01000000-04ffffff : Crash kernel
    01000000-014680c2 : Kernel code
    014680c3-016f39f7 : Kernel data
    0175b000-017d97ab : Kernel bss

I'll keep you informed.

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kdump fails to load and crashes [was: mmotm 2009-11-01-10-01 uploaded]

[Vivek Goyal]

On Tue, Nov 03, 2009 at 04:14:25PM +0100, Jiri Slaby wrote:
> On 11/03/2009 03:44 PM, Vivek Goyal wrote:
> > On Tue, Nov 03, 2009 at 12:11:58AM +0100, Jiri Slaby wrote:
> >> On 11/01/2009 07:07 PM, akpm@linux-foundation.org wrote:
> >>> The mm-of-the-moment snapshot 2009-11-01-10-01 has been uploaded to
> >>
> >> Hi, kdump loading crashes:
> >> BUG: unable to handle kernel paging request at ffff8800010a7000
> >> IP: [<ffffffff8101cd8b>] machine_kexec_prepare+0x16b/0x13e0
> >> PGD 1806063 PUD 180a063 PMD 10001e1
> ...
> >> ffff8800010a7000 should be OK, as crashkernel=64M@16M was passed to the
> >> kernel.
> >>
> > 
> > This is strange. As you said, ffff8800010a7000, should be a valid address
> > as you reserved memory from 16M.
> 
> Hmm, PMD has not RW bit set and we do store (see below).
> 
> > I took above -mm and for me kernel loads fine. I reserved memory 64M@32M
> > as by default kernel is loading at address 16M.
> > 
> > How does your /proc/iomem look like in first kernel?
> 
> I'm not any longer on that machine. init_pgtable() overwrites random
> memory, so possibly even host's page tables, hence the oops above, I
> suppose.
> 
> I'm currently chasing it down in the qemu virtual machine, since I
> already got a garbage into my ~/.viminfo (some overwritten page got
> flushed, I guess).
> 
> There /proc/iomem looks like:
> 00100000-12beffff : System RAM
>   01000000-04ffffff : Crash kernel
>     01000000-014680c2 : Kernel code
>     014680c3-016f39f7 : Kernel data
>     0175b000-017d97ab : Kernel bss

Hmm.., this looks fishy. Reserved memory is starting at 16MB at the same
time first kernel is loaded at 16MB. So when we try to load a crash kernel
it can try to overwrite existing kernel.

In the past if crash kernel memory area used to overlap with kernel code,
data or bss, we would not reserve the memory for "Crash Kernel" and this
entry would not appear in /proc/iomem. Not sure what has changed.

Anyway, can you just try reserving memory at some other address, say
64M@32M and see if it works.

Thanks
Vivek

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kdump fails to load and crashes [was: mmotm 2009-11-01-10-01 uploaded]

[Jiri Slaby]

On 11/03/2009 04:24 PM, Vivek Goyal wrote:
>> There /proc/iomem looks like:
>> 00100000-12beffff : System RAM
>>   01000000-04ffffff : Crash kernel
>>     01000000-014680c2 : Kernel code
>>     014680c3-016f39f7 : Kernel data
>>     0175b000-017d97ab : Kernel bss
...
> Anyway, can you just try reserving memory at some other address, say
> 64M@32M and see if it works.

Yup, that's it of course. I missed it overlaps with current kernel. I
though they are areas for the crash kernels. Thanks for pointing out. So
the error is we reserve the space for kdump even we can't in fact?

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec