pvgrub crashing after selecting kernel to boot

pvgrub crashing after selecting kernel to boot

[Jeremy Fitzhardinge]

I've also started seeing crashes on 64-bit pvgrub after selecting the 
kernel to boot:

Thread "kbdfront close": pointer: 0x2020004580, stack: 0xcf0000
close fb: backend at /local/domain/0/backend/vfb/1/0
close kbd: backend at /local/domain/0/backend/vkbd/1/0
shutdown_kbdfront: error changing state to 5: ENOENT
Thread "kbdfront close" exited.
Page fault at linear address 0x100953340, rip 0x4b768, regs 0xcdfa68, sp 0xcdfb18, our_sp 0xcdfa20, code 2
Thread: main
RIP: e030:[<000000000004b768>]
RSP: e02b:0000000000cdfb18  EFLAGS: 00010293
RAX: 0000000000e2e000 RBX: 0000000000cdfb98 RCX: 0000000100953338
RDX: 0000000000000001 RSI: 0000000000953188 RDI: 0000000000000000
RBP: 0000000000cdfb28 R08: 0000000100953338 R09: 0000000000e2c000
R10: 00000000deadbeef R11: 0000000000200030 R12: 0000000000000003
R13: 0000002020305018 R14: 0000000000000000 R15: 0000000000002000
base is 0xcdfb28 caller is 0x57cbd
base is 0xcdfb88 caller is 0x3067
base is 0xcdfc88 caller is 0x5c1b3
base is 0xcdfcd8 caller is 0x5be0a
base is 0xcdfcf8 caller is 0x3db6
base is 0xcdfd38 caller is 0x402e
base is 0xcdfd58 caller is 0x8341
base is 0xcdfd98 caller is 0xaa2f
base is 0xcdfdd8 caller is 0x108ca
base is 0xcdfe88 caller is 0x10f62
base is 0xcdff48 caller is 0x4343
base is 0xcdff58 caller is 0x4b48a
base is 0xcdffe8 caller is 0x33da

cdfb00: 18 fb cd 00 00 00 00 00 2b e0 00 00 00 00 00 00
cdfb10: 20 18 5a bb 00 00 00 00 25 40 98 33 01 00 00 00
cdfb20: 98 fb cd 00 00 00 00 00 88 fb cd 00 00 00 00 00
cdfb30: bd 7c 05 00 00 00 00 00 25 40 98 33 01 00 00 00

cdfb10: 20 18 5a bb 00 00 00 00 25 40 98 33 01 00 00 00
cdfb20: 98 fb cd 00 00 00 00 00 88 fb cd 00 00 00 00 00
cdfb30: bd 7c 05 00 00 00 00 00 25 40 98 33 01 00 00 00
cdfb40: 38 18 5a bb 00 00 00 00 03 00 00 00 01 00 00 00

4b750: f2 b9 80 31 95 00 48 8b 04 f1 4c 8b 00 4c 89 04
4b760: f1 48 8b 08 48 8b 70 08 48 89 71 08 39 d7 74 49
4b770: be 01 00 00 00 41 b8 80 31 95 00 83 ea 01 8d 4a
4b780: 0c 48 89 f3 48 d3 e3 4c 8d 0c 18 41 89 51 10 4c
Pagetable walk from virt 100953340, base a9c000:
  L4 = 00000000bf570067 (0xa9d000)  [offset = 0]
   L3 = 0000000000000000 (0xfffffffffffff000)  [offset = 4]
Page fault in pagetable walk (access to invalid memory?).



It was working until a week or two ago, but I'm not sure what has caused 
the regression.

     J

Re: pvgrub crashing after selecting kernel to boot

[Samuel Thibault]

Jeremy Fitzhardinge, le Wed 03 Mar 2010 16:32:54 -0800, a écrit :
> base is 0xcdfb28 caller is 0x57cbd
> base is 0xcdfb88 caller is 0x3067
> base is 0xcdfc88 caller is 0x5c1b3
> base is 0xcdfcd8 caller is 0x5be0a
> base is 0xcdfcf8 caller is 0x3db6
> base is 0xcdfd38 caller is 0x402e
> base is 0xcdfd58 caller is 0x8341
> base is 0xcdfd98 caller is 0xaa2f
> base is 0xcdfdd8 caller is 0x108ca
> base is 0xcdfe88 caller is 0x10f62
> base is 0xcdff48 caller is 0x4343
> base is 0xcdff58 caller is 0x4b48a
> base is 0xcdffe8 caller is 0x33da

Please addr2line these callers and the crash IP :)

Samuel

Re: pvgrub crashing after selecting kernel to boot

[Jeremy Fitzhardinge]

On 03/03/2010 04:32 PM, Jeremy Fitzhardinge wrote:
> I've also started seeing crashes on 64-bit pvgrub after selecting the 
> kernel to boot:
>
> Thread "kbdfront close": pointer: 0x2020004580, stack: 0xcf0000
> close fb: backend at /local/domain/0/backend/vfb/1/0
> close kbd: backend at /local/domain/0/backend/vkbd/1/0
> shutdown_kbdfront: error changing state to 5: ENOENT
> Thread "kbdfront close" exited.
> Page fault at linear address 0x100953340, rip 0x4b768, regs 0xcdfa68, 
> sp 0xcdfb18, our_sp 0xcdfa20, code 2
> Thread: main
> RIP: e030:[<000000000004b768>]
extras/mini-os/mm.c:276

> RSP: e02b:0000000000cdfb18  EFLAGS: 00010293
> RAX: 0000000000e2e000 RBX: 0000000000cdfb98 RCX: 0000000100953338
> RDX: 0000000000000001 RSI: 0000000000953188 RDI: 0000000000000000
> RBP: 0000000000cdfb28 R08: 0000000100953338 R09: 0000000000e2c000
> R10: 00000000deadbeef R11: 0000000000200030 R12: 0000000000000003
> R13: 0000002020305018 R14: 0000000000000000 R15: 0000000000002000
> base is 0xcdfb28 caller is 0x57cbd
extras/mini-os/arch/x86/traps.c:147

> base is 0xcdfb88 caller is 0x3067
??:0
> base is 0xcdfc88 caller is 0x5c1b3
stubdom/newlib-x86_64/x86_64-xen-elf/newlib/libc/stdlib/../../../../../newlib-1.16.0/newlib/libc/stdlib/mallocr.c:2947

> base is 0xcdfcd8 caller is 0x5be0a
stubdom/newlib-x86_64/x86_64-xen-elf/newlib/libc/stdlib/../../../../../newlib-1.16.0/newlib/libc/stdlib/realloc.c:19

> base is 0xcdfcf8 caller is 0x3db6
stubdom/grub/mini-os.c:165

> base is 0xcdfd38 caller is 0x402e
stubdom/grub/mini-os.c:187

> base is 0xcdfd58 caller is 0x8341
stubdom/grub/../grub-upstream/stage2/builtins.c:2713

> base is 0xcdfd98 caller is 0xaa2f
stubdom/grub/../grub-upstream/stage2/cmdline.c:256


> base is 0xcdfdd8 caller is 0x108ca
stubdom/grub/../grub-upstream/stage2/stage2.c:769


> base is 0xcdfe88 caller is 0x10f62
stubdom/grub/../grub-upstream/stage2/stage2.c:1121


> base is 0xcdff48 caller is 0x4343
stubdom/grub/mini-os.c:763

> base is 0xcdff58 caller is 0x4b48a
extras/mini-os/main.c:162

> base is 0xcdffe8 caller is 0x33da
gdtoa-hexnan.c:0
>
> cdfb00: 18 fb cd 00 00 00 00 00 2b e0 00 00 00 00 00 00
> cdfb10: 20 18 5a bb 00 00 00 00 25 40 98 33 01 00 00 00
> cdfb20: 98 fb cd 00 00 00 00 00 88 fb cd 00 00 00 00 00
> cdfb30: bd 7c 05 00 00 00 00 00 25 40 98 33 01 00 00 00
>
> cdfb10: 20 18 5a bb 00 00 00 00 25 40 98 33 01 00 00 00
> cdfb20: 98 fb cd 00 00 00 00 00 88 fb cd 00 00 00 00 00
> cdfb30: bd 7c 05 00 00 00 00 00 25 40 98 33 01 00 00 00
> cdfb40: 38 18 5a bb 00 00 00 00 03 00 00 00 01 00 00 00
>
> 4b750: f2 b9 80 31 95 00 48 8b 04 f1 4c 8b 00 4c 89 04
> 4b760: f1 48 8b 08 48 8b 70 08 48 89 71 08 39 d7 74 49
> 4b770: be 01 00 00 00 41 b8 80 31 95 00 83 ea 01 8d 4a
> 4b780: 0c 48 89 f3 48 d3 e3 4c 8d 0c 18 41 89 51 10 4c
> Pagetable walk from virt 100953340, base a9c000:
>  L4 = 00000000bf570067 (0xa9d000)  [offset = 0]
>   L3 = 0000000000000000 (0xfffffffffffff000)  [offset = 4]
> Page fault in pagetable walk (access to invalid memory?).
>

> Please addr2line these callers and the crash IP

I'm disappointed you can't just intuit it.

     J

Re: pvgrub crashing after selecting kernel to boot

[Samuel Thibault]

Jeremy Fitzhardinge, le Wed 03 Mar 2010 16:54:04 -0800, a écrit :
> On 03/03/2010 04:32 PM, Jeremy Fitzhardinge wrote:
> >I've also started seeing crashes on 64-bit pvgrub after selecting the 
> >kernel to boot:

When using the same configuration and kernel, is it systematic or random?

Samuel

Re: pvgrub crashing after selecting kernel to boot

[Jeremy Fitzhardinge]

On 03/03/2010 05:16 PM, Samuel Thibault wrote:
> When using the same configuration and kernel, is it systematic or random?
>    

Happens consistently.  Seems independent of dom0 kernel or guest kernel 
(though they're all various forms of pvops).  I see it on two separate 
machines.

     J

Re: pvgrub crashing after selecting kernel to boot

[Samuel Thibault]

Jeremy Fitzhardinge, le Wed 03 Mar 2010 17:43:54 -0800, a écrit :
> On 03/03/2010 05:16 PM, Samuel Thibault wrote:
> >When using the same configuration and kernel, is it systematic or random?
> >   
> 
> Happens consistently.  Seems independent of dom0 kernel or guest kernel 
> (though they're all various forms of pvops).  I see it on two separate 
> machines.

Does it depend on the amount of domU RAM? If so, how much exactly poses
problem?  Please also paste the MiniOS start log talking about memory
addresses & such.

Samuel

Re: pvgrub crashing after selecting kernel to boot

[Jeremy Fitzhardinge]

On 03/03/2010 05:51 PM, Samuel Thibault wrote:
> Jeremy Fitzhardinge, le Wed 03 Mar 2010 17:43:54 -0800, a écrit :
>    
>> On 03/03/2010 05:16 PM, Samuel Thibault wrote:
>>      
>>> When using the same configuration and kernel, is it systematic or random?
>>>
>>>        
>> Happens consistently.  Seems independent of dom0 kernel or guest kernel
>> (though they're all various forms of pvops).  I see it on two separate
>> machines.
>>      
> Does it depend on the amount of domU RAM? If so, how much exactly poses
> problem?  Please also paste the MiniOS start log talking about memory
> addresses&  such.
>
>    

Below is the full crash with 512MB.   I tried 32, 64, 128, 256 and 
1024MB with much the same results, though sometimes it just hung in a 
spin loop rather than crashing (inconsistent from boot to boot, not a 
function of memory size).  I also noticed when booting the Fedora 12 
distro kernel it did actually start the kernel, but it failed to make 
much progress (or at least, no console output).  The same kernel boots 
fine with pygrub.

     J

blktap_sysfs_create: adding attributes for dev ffff8800b1945000
Bootstrapping...
Xen Minimal OS!
   start_info: 0xa99000(VA)
     nr_pages: 0x20000
   shared_inf: 0x91c57000(MA)
      pt_base: 0xa9c000(VA)
nr_pt_frames: 0x9
     mfn_list: 0x999000(VA)
    mod_start: 0x0(VA)
      mod_len: 0
        flags: 0x0
     cmd_line: (hd0,0)/grub/grub.conf
   stack:      0x958980-0x978980
MM: Init
       _text: 0x0(VA)
      _etext: 0x69194(VA)
    _erodata: 0x82000(VA)
      _edata: 0x8aae0(VA)
stack start: 0x958980(VA)
        _end: 0x998f88(VA)
   start_pfn: aa8
     max_pfn: 20000
Mapping memory range 0xc00000 - 0x20000000
setting 0x0-0x82000 readonly
skipped 0x1000
MM: Initialise page allocator for ba2000(ba2000)-20000000(20000000)
MM: done
Demand map pfns at 20001000-2020001000.
Heap resides at 2020002000-4020002000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0x20001000.
Initialising scheduler
Thread "Idle": pointer: 0x2020002050, stack: 0xcb0000
Initialising xenbus
Thread "xenstore": pointer: 0x2020002800, stack: 0xcc0000
Dummy main: start_info=0x978a80
Thread "main": pointer: 0x2020002fb0, stack: 0xcd0000
Thread "pcifront": pointer: 0x2020003760, stack: 0xce0000
pcifront_watches: waiting for backend path to happear device/pci/0/backend
vbd 768 is hd0
******************* BLKFRONT for device/vbd/768 **********


backend at /local/domain/0/backend/vbd/19/768
Failed to read /local/domain/0/backend/vbd/19/768/feature-flush-cache.
20971520 sectors of 512 bytes
**************************
Thread "kbdfront": pointer: 0x2020004590, stack: 0xcf0000
******************* FBFRONT for device/vfb/0 **********


******************* KBDFRONT for device/vkbd/0 **********


backend at /local/domain/0/backend/vkbd/19/0
backend at /local/domain/0/backend/vfb/19/0
/local/domain/0/backend/vkbd/19/0 connected
************************** KBDFRONT
Thread "kbdfront" exited.
/local/domain/0/backend/vfb/19/0 connected
************************** FBFRONT
Thread "kbdfront close": pointer: 0x2020004590, stack: 0xcf0000
close fb: backend at /local/domain/0/backend/vfb/19/0
close kbd: backend at /local/domain/0/backend/vkbd/19/0
shutdown_kbdfront: error changing state to 5: ENOENT
Thread "kbdfront close" exited.
Page fault at linear address 0x100953340, rip 0x4b768, regs 0xcdfa68, sp 0xcdfb18, our_sp 0xcdfa20, code 2
Thread: main
RIP: e030:[<000000000004b768>]
RSP: e02b:0000000000cdfb18  EFLAGS: 00010293
RAX: 0000000000e2e000 RBX: 0000000000cdfb98 RCX: 0000000100953338
RDX: 0000000000000001 RSI: 0000000000953188 RDI: 0000000000000000
RBP: 0000000000cdfb28 R08: 0000000100953338 R09: 0000000000e2c000
R10: 00000000deadbeef R11: 0000000000200030 R12: 0000000000000003
R13: 0000002020304e18 R14: 0000000000000000 R15: 0000000000001200
base is 0xcdfb28 caller is 0x57cbd
base is 0xcdfb88 caller is 0x3067
base is 0xcdfc88 caller is 0x5c1b3
base is 0xcdfcd8 caller is 0x5be0a
base is 0xcdfcf8 caller is 0x3db6
base is 0xcdfd38 caller is 0x402e
base is 0xcdfd58 caller is 0x8341
base is 0xcdfd98 caller is 0xaa2f
base is 0xcdfdd8 caller is 0x108ca
base is 0xcdfe88 caller is 0x10f62
base is 0xcdff48 caller is 0x4343
base is 0xcdff58 caller is 0x4b48a
base is 0xcdffe8 caller is 0x33da

cdfb00: 18 fb cd 00 00 00 00 00 2b e0 00 00 00 00 00 00
cdfb10: 18 58 e2 90 00 00 00 00 25 10 c5 91 00 00 00 00
cdfb20: 98 fb cd 00 00 00 00 00 88 fb cd 00 00 00 00 00
cdfb30: bd 7c 05 00 00 00 00 00 25 10 c5 91 00 00 00 00

cdfb10: 18 58 e2 90 00 00 00 00 25 10 c5 91 00 00 00 00
cdfb20: 98 fb cd 00 00 00 00 00 88 fb cd 00 00 00 00 00
cdfb30: bd 7c 05 00 00 00 00 00 25 10 c5 91 00 00 00 00
cdfb40: 30 58 e2 90 00 00 00 00 03 00 00 00 00 00 00 00

4b750: f2 b9 80 31 95 00 48 8b 04 f1 4c 8b 00 4c 89 04
4b760: f1 48 8b 08 48 8b 70 08 48 89 71 08 39 d7 74 49
4b770: be 01 00 00 00 41 b8 80 31 95 00 83 ea 01 8d 4a
4b780: 0c 48 89 f3 48 d3 e3 4c 8d 0c 18 41 89 51 10 4c
Pagetable walk from virt 100953340, base a9c000:
  L4 = 00000000911b4067 (0xa9d000)  [offset = 0]
   L3 = 0000000000000000 (0xfffffffffffff000)  [offset = 4]
Page fault in pagetable walk (access to invalid memory?).
blktap_sysfs_destroy

Re: pvgrub crashing after selecting kernel to boot

[Samuel Thibault]

Can you try the PV-GRUB patch I have just uploaded?  The gunzip bug may
very well have made minios gone wild.  If it doesn't solve the issue,
could you post a minimal boot image that crashes for you? (i.e. probably
just containing the kernel and initrd), as well as the pv-grub kernel
image, to make sure we have the same things?

Samuel

Re: pvgrub crashing after selecting kernel to boot

[Jeremy Fitzhardinge]

On 03/07/2010 02:47 PM, Samuel Thibault wrote:
> Can you try the PV-GRUB patch I have just uploaded?  The gunzip bug may
> very well have made minios gone wild.

No, it didn't help unfortunately.

>    If it doesn't solve the issue,
> could you post a minimal boot image that crashes for you? (i.e. probably
> just containing the kernel and initrd), as well as the pv-grub kernel
> image, to make sure we have the same things?
>    

OK:

http://www.goop.org/~jeremy/lib-xen-boot.tar.gz - the pvgrub images
http://www.goop.org/~jeremy/boot.tar.gz - /boot from my domain

     J

Re: pvgrub crashing after selecting kernel to boot

[Samuel Thibault]

Hello,

Jeremy Fitzhardinge, le Mon 08 Mar 2010 13:35:28 -0800, a écrit :
> >could you post a minimal boot image that crashes for you? (i.e. probably
> >just containing the kernel and initrd), as well as the pv-grub kernel
> >image, to make sure we have the same things?
> >   
> 
> OK:
> 
> http://www.goop.org/~jeremy/lib-xen-boot.tar.gz - the pvgrub images
> http://www.goop.org/~jeremy/boot.tar.gz - /boot from my domain

Unfortunately that works for me, all these kernels...  Which FS is your
/boot on?  Maybe you could put it on a separate FS so we can have the
same for tests?

Samuel

Re: pvgrub crashing after selecting kernel to boot

[Jeremy Fitzhardinge]

On 03/08/2010 05:24 PM, Samuel Thibault wrote:
> Unfortunately that works for me, all these kernels...  Which FS is your
> /boot on?  Maybe you could put it on a separate FS so we can have the
> same for tests?
>    

/boot is already a separate filesystem, so I'll just send you an image.

     J