Fwd: domU lenny not working: BUG: unable to handle kernel paging request at e7437cc0

Fwd: domU lenny not working: BUG: unable to handle kernel paging request at e7437cc0

[Sally-Anne Edwards]

Re-sending, because Gmail sent the original as HTML and the list rejected it
xm-debian.cfg from http://people.debian.org/~joeyh/d-i/images/daily(24/08/08), options selected (ignoring the comments and the code atthe bottom):
memory = 256name = "debtest1"vif = ['bridge=eth0']disk = ['phy:/dev/mapper/VolGroup00-debtest1,xvda,w']
domU started like this:
xm create -c xm-debtest1.cfg install=trueinstall-installer=http://people.debian.org/~joeyh/d-i/images/daily
The error:
  ┌───────────────┤ Detecting disks and all other hardware ├────────────────┐  │                                                                         │  │                                    0%                                   │  │                                                                         │  │ [   97.520018] BUG: unable to handle kernel paging request at e7437cc0  │[   97.520018] IP: [<d0837767>] :xen_blkfront:blkif_interrupt+0x5d/0x18a    │[   97.520018] Oops: 0000 [#1] SMP ─────────────────────────────────────────┘[   97.520018] Modules linked in: xen_blkfront(+) rsrc_nonstaticpcmcia_core thermal_sys xen_netfront[   97.520018][   97.520018] Pid: 475, comm: klogd Not tainted (2.6.26-1-686-bigmem #1)[   97.520018] EIP: 0061:[<d0837767>] EFLAGS: 00010807 CPU: 0[   97.520018] EIP is at blkif_interrupt+0x5d/0x18a [xen_blkfront][   97.520018] EAX: 18009c00 EBX: cf42e000 ECX: 00000200 EDX: 00000200[   97.520018] ESI: 00000000 EDI: cd4fe0ac EBP: ca000100 ESP: ce411f2c[   97.520018]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069[   97.520018] Process klogd (pid: 475, ti=ce410000 task=cfc60140task.ti=ce410000)[   97.520018] Stack: c01043ca 00000001 00000002 00000200 c14efd0000000001 cfdfaaa0 00000000[   97.520018]        00000000 00000007 c0158c36 c0386ec0 0000000700000000 c0386ef0 c0159ed5[   97.520018]        000002a0 c0386ec0 00000000 00000007 c010afd200000000 00000001 fffffffe[   97.520018] Call Trace:[   97.520018]  [<c01043ca>] xen_mc_flush+0x118/0x13b[   97.520018]  [<c0158c36>] handle_IRQ_event+0x23/0x51[   97.520018]  [<c0159ed5>] handle_level_irq+0x73/0xb6[   97.520018]  [<c010afd2>] do_IRQ+0x4d/0x63[   97.520018]  [<c022d350>] xen_evtchn_do_upcall+0x8b/0xf1[   97.520018]  [<c0109543>] xen_do_upcall+0x7/0xc[   97.520018]  =======================[   97.520018] Code: 8b 43 1c 89 44 24 04 89 44 24 14 e9 e2 00 00 008b 43 20 31 f6 48 23 44 24 04 6b c0 6c 8d 78 40 03 7b 24 8b 2f 69 c59c 00 00 00 <8b> 94 18 c0 00 00 00 8d 44 18 54 89 44 24 10 89 14 24 eb12 8b[   97.520018] EIP: [<d0837767>] blkif_interrupt+0x5d/0x18a[xen_blkfront] SS:ESP 0069:ce411f2c[   97.520018] Kernel panic - not syncing: Fatal exception in interruptЪТХ╨{.nг+┴╥÷╝┴╜├+%┼кЪ╠Ищ╤\x17╔┼wЪ╨{.nг+┴╥╔┼{╠ЧG╚²ИЪ┼{ay╨\x1dй┤з≥К,j\a╜╒fё╒╥h ▐О│ЙЪ▒ЙГz_Х╝\x03(╜И ▌┼щ╒j"²З\x1a╤^[m╖ЪЪ╬\a╚ЧG╚²ИЪ╒╦?≥╗Х╜з&ёЬ╖~▐А╤iO∙Ф╛z╥ vь^\x14\x04\x1a╤^[m╖ЪЪц\fЪ╤ЛЪ╒╦?√I╔

Re: Fwd: domU lenny not working: BUG: unable to handle kernel paging request at e7437cc0

[Jeremy Fitzhardinge]

Sally-Anne Edwards wrote:
> Re-sending, because Gmail sent the original as HTML and the list rejected it
>
> xm-debian.cfg from http://people.debian.org/~joeyh/d-i/images/daily
> (24/08/08), options selected (ignoring the comments and the code at
> the bottom):
>
> memory = 256
> name = "debtest1"
> vif = ['bridge=eth0']
> disk = ['phy:/dev/mapper/VolGroup00-debtest1,xvda,w']
>
> domU started like this:
>
> xm create -c xm-debtest1.cfg install=true
> install-installer=http://people.debian.org/~joeyh/d-i/images/daily
>
> The error:
>   

Does this happen consistently, or is it a one-off?

>   ┌───────────────┤ Detecting disks and all other hardware ├────────────────┐
>   │                                                                         │
>   │                                    0%                                   │
>   │                                                                         │
>   │ [   97.520018] BUG: unable to handle kernel paging request at e7437cc0  │
> [   97.520018] IP: [<d0837767>] :xen_blkfront:blkif_interrupt+0x5d/0x18a    │
> [   97.520018] Oops: 0000 [#1] SMP ─────────────────────────────────────────┘
> [   97.520018] Modules linked in: xen_blkfront(+) rsrc_nonstatic
> pcmcia_core thermal_sys xen_netfront
> [   97.520018]
> [   97.520018] Pid: 475, comm: klogd Not tainted (2.6.26-1-686-bigmem #1)
> [   97.520018] EIP: 0061:[<d0837767>] EFLAGS: 00010807 CPU: 0
> [   97.520018] EIP is at blkif_interrupt+0x5d/0x18a [xen_blkfront]
> [   97.520018] EAX: 18009c00 EBX: cf42e000 ECX: 00000200 EDX: 00000200
> [   97.520018] ESI: 00000000 EDI: cd4fe0ac EBP: ca000100 ESP: ce411f2c
> [   97.520018]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
> [   97.520018] Process klogd (pid: 475, ti=ce410000 task=cfc60140
> task.ti=ce410000)
> [   97.520018] Stack: c01043ca 00000001 00000002 00000200 c14efd00
> 00000001 cfdfaaa0 00000000
> [   97.520018]        00000000 00000007 c0158c36 c0386ec0 00000007
> 00000000 c0386ef0 c0159ed5
> [   97.520018]        000002a0 c0386ec0 00000000 00000007 c010afd2
> 00000000 00000001 fffffffe
> [   97.520018] Call Trace:
> [   97.520018]  [<c01043ca>] xen_mc_flush+0x118/0x13b
> [   97.520018]  [<c0158c36>] handle_IRQ_event+0x23/0x51
> [   97.520018]  [<c0159ed5>] handle_level_irq+0x73/0xb6
> [   97.520018]  [<c010afd2>] do_IRQ+0x4d/0x63
> [   97.520018]  [<c022d350>] xen_evtchn_do_upcall+0x8b/0xf1
> [   97.520018]  [<c0109543>] xen_do_upcall+0x7/0xc
> [   97.520018]  =======================
> [   97.520018] Code: 8b 43 1c 89 44 24 04 89 44 24 14 e9 e2 00 00 00
> 8b 43 20 31 f6 48 23 44 24 04 6b c0 6c 8d 78 40 03 7b 24 8b 2f 69 c5
> 9c 00 00 00 <8b> 94 18 c0 00 00 00 8d 44 18 54 89 44 24 10 89 14 24 eb
> 12 8b
> [   97.520018] EIP: [<d0837767>] blkif_interrupt+0x5d/0x18a
>   

I haven't see this oops before, but I think the code in question
correlates to:

	bret = RING_GET_RESPONSE(&info->ring, i);
	id   = bret->id;
	req  = (struct request *)info->shadow[id].request;

which suggests the backend driver is passing back a bogus request ID.

What's your host environment?

    J

Re: Fwd: domU lenny not working: BUG: unable to handle kernel paging request at e7437cc0

[Ian Campbell]

[-- Attachment #1: Type: text/plain, Size: 1263 bytes --]

On Mon, 2008-08-25 at 12:35 -0700, Jeremy Fitzhardinge wrote:
> I haven't see this oops before, but I think the code in question
> correlates to:
> 
> 	bret = RING_GET_RESPONSE(&info->ring, i);
> 	id   = bret->id;
> 	req  = (struct request *)info->shadow[id].request;
> 
> which suggests the backend driver is passing back a bogus request ID.
> 
> What's your host environment?

I believe it's a 64 bit Etch kernel running on a Lenny domain 0. I
wonder if that kernel doesn't write the protocol field in xenstore so
the 64 bit backend Sally-Anne is using cannot talk to the 32 bit
frontend correctly.

Sally, can you run xenstore-ls on the domain's path
(/local/domain/<domid>). Is there a protocol field in the vbd entry?
If not can you try starting the domain in the paused state (xm create
-p) and use xenstore-write to fill in the protocol node in the frontend
to "x86_32-abi". e.g.
        xenstore write /local/domain/<domid>/device/vbd/<nr>/protocol x86_32-abi
before unpausing.

I thought the tools did this (so expected the Lenny tools to be correct)
but looking at the code it seems it is the domain 0 kernel which does
it.

Ian.

-- 
Ian Campbell

Success is getting what you want; happiness is wanting what you get.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Fwd: domU lenny not working: BUG: unable to handle kernel paging request at e7437cc0

[Ian Campbell]

[-- Attachment #1: Type: text/plain, Size: 755 bytes --]

On Mon, 2008-08-25 at 20:46 +0100, Ian Campbell wrote:
> 
> I thought the tools did this (so expected the Lenny tools to be
> correct) but looking at the code it seems it is the domain 0 kernel
> which does it.

Uh, I'm thinking backwards here. It's the frontend which writes it but
the backend needs to understand it which the Etch kernel does not.

Unfortunately this means that writing the protocol node as I suggest
won't work.

Only thing I can recommend at the moment is to use a kernel which does
have this (i.e. one from
http://xenbits.xensource.com/linux-2.6.18-xen.hg) or to use a 32 bit
Etch kernel if you want to run 32 bit guests.

Ian.
-- 
Ian Campbell

If life isn't what you wanted, have you asked for anything else?

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Fwd: domU lenny not working: BUG: unable to handle kernel paging request at e7437cc0

[Jeremy Fitzhardinge]

Ian Campbell wrote:
> On Mon, 2008-08-25 at 20:46 +0100, Ian Campbell wrote:
>   
>> I thought the tools did this (so expected the Lenny tools to be
>> correct) but looking at the code it seems it is the domain 0 kernel
>> which does it.
>>     
>
> Uh, I'm thinking backwards here. It's the frontend which writes it but
> the backend needs to understand it which the Etch kernel does not.
>   

Should we push that patch back into stable kernels?

    J

Re: Fwd: domU lenny not working: BUG: unable to handle kernel paging request at e7437cc0

[Ian Campbell]

[-- Attachment #1: Type: text/plain, Size: 767 bytes --]

On Mon, 2008-08-25 at 13:25 -0700, Jeremy Fitzhardinge wrote:
> Ian Campbell wrote:
> > On Mon, 2008-08-25 at 20:46 +0100, Ian Campbell wrote:
> >   
> >> I thought the tools did this (so expected the Lenny tools to be
> >> correct) but looking at the code it seems it is the domain 0 kernel
> >> which does it.
> >>     
> >
> > Uh, I'm thinking backwards here. It's the frontend which writes it but
> > the backend needs to understand it which the Etch kernel does not.
> >   
> 
> Should we push that patch back into stable kernels?

The pvops kernel frontend already does the right thing. I'll look into
what it would take to get the support added to the backends in Etch.

Ian.

-- 
Ian Campbell

Matrimony isn't a word, it's a sentence.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Fwd: domU lenny not working: BUG: unable to handle kernel paging request at e7437cc0

[Jeremy Fitzhardinge]

Ian Campbell wrote:
> On Mon, 2008-08-25 at 20:46 +0100, Ian Campbell wrote:
>   
>> I thought the tools did this (so expected the Lenny tools to be
>> correct) but looking at the code it seems it is the domain 0 kernel
>> which does it.
>>     
>
> Uh, I'm thinking backwards here. It's the frontend which writes it but
> the backend needs to understand it which the Etch kernel does not.
>
> Unfortunately this means that writing the protocol node as I suggest
> won't work.
>
> Only thing I can recommend at the moment is to use a kernel which does
> have this (i.e. one from
> http://xenbits.xensource.com/linux-2.6.18-xen.hg) or to use a 32 bit
> Etch kernel if you want to run 32 bit guests.

Or run a 64-bit domU kernel?  Should work fine with a 32-bit userspace.

    J

Re: Fwd: domU lenny not working: BUG: unable to handle kernel paging request at e7437cc0

[Ian Campbell]

[-- Attachment #1: Type: text/plain, Size: 1151 bytes --]

On Mon, 2008-08-25 at 13:59 -0700, Jeremy Fitzhardinge wrote:
> Ian Campbell wrote:
> > On Mon, 2008-08-25 at 20:46 +0100, Ian Campbell wrote:
> >   
> >> I thought the tools did this (so expected the Lenny tools to be
> >> correct) but looking at the code it seems it is the domain 0 kernel
> >> which does it.
> >>     
> >
> > Uh, I'm thinking backwards here. It's the frontend which writes it but
> > the backend needs to understand it which the Etch kernel does not.
> >
> > Unfortunately this means that writing the protocol node as I suggest
> > won't work.
> >
> > Only thing I can recommend at the moment is to use a kernel which does
> > have this (i.e. one from
> > http://xenbits.xensource.com/linux-2.6.18-xen.hg) or to use a 32 bit
> > Etch kernel if you want to run 32 bit guests.
> 
> Or run a 64-bit domU kernel?  Should work fine with a 32-bit userspace.

Unfortunately Debian has settled on 2.6.26 for the Lenny release so no
64 bit support.

Ian.

-- 
Ian Campbell

Look, we play the Star Spangled Banner before every game.  You want us
to pay income taxes, too?
		-- Bill Veeck, Chicago White Sox

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Fwd: domU lenny not working: BUG: unable to handle kernel paging request at e7437cc0

[Sally-Anne Edwards]

2008/8/25 Jeremy Fitzhardinge <jeremy@goop.org>:> Sally-Anne Edwards wrote:>> Re-sending, because Gmail sent the original as HTML and the list rejected it>>>> xm-debian.cfg from http://people.debian.org/~joeyh/d-i/images/daily>> (24/08/08), options selected (ignoring the comments and the code at>> the bottom):>>>> memory = 256>> name = "debtest1">> vif = ['bridge=eth0']>> disk = ['phy:/dev/mapper/VolGroup00-debtest1,xvda,w']>>>> domU started like this:>>>> xm create -c xm-debtest1.cfg install=true>> install-installer=http://people.debian.org/~joeyh/d-i/images/daily>>>> The error:>>>> Does this happen consistently, or is it a one-off?
It is consistent - I believe Ian has seen it too
>>>   ┌───────────────┤ Detecting disks and all other hardware ├────────────────┐>>   │                                                                         │>>   │                                    0%                                   │>>   │                                                                         │>>   │ [   97.520018] BUG: unable to handle kernel paging request at e7437cc0  │>> [   97.520018] IP: [<d0837767>] :xen_blkfront:blkif_interrupt+0x5d/0x18a    │>> [   97.520018] Oops: 0000 [#1] SMP ─────────────────────────────────────────┘>> [   97.520018] Modules linked in: xen_blkfront(+) rsrc_nonstatic>> pcmcia_core thermal_sys xen_netfront>> [   97.520018]>> [   97.520018] Pid: 475, comm: klogd Not tainted (2.6.26-1-686-bigmem #1)>> [   97.520018] EIP: 0061:[<d0837767>] EFLAGS: 00010807 CPU: 0>> [   97.520018] EIP is at blkif_interrupt+0x5d/0x18a [xen_blkfront]>> [   97.520018] EAX: 18009c00 EBX: cf42e000 ECX: 00000200 EDX: 00000200>> [   97.520018] ESI: 00000000 EDI: cd4fe0ac EBP: ca000100 ESP: ce411f2c>> [   97.520018]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069>> [   97.520018] Process klogd (pid: 475, ti=ce410000 task=cfc60140>> task.ti=ce410000)>> [   97.520018] Stack: c01043ca 00000001 00000002 00000200 c14efd00>> 00000001 cfdfaaa0 00000000>> [   97.520018]        00000000 00000007 c0158c36 c0386ec0 00000007>> 00000000 c0386ef0 c0159ed5>> [   97.520018]        000002a0 c0386ec0 00000000 00000007 c010afd2>> 00000000 00000001 fffffffe>> [   97.520018] Call Trace:>> [   97.520018]  [<c01043ca>] xen_mc_flush+0x118/0x13b>> [   97.520018]  [<c0158c36>] handle_IRQ_event+0x23/0x51>> [   97.520018]  [<c0159ed5>] handle_level_irq+0x73/0xb6>> [   97.520018]  [<c010afd2>] do_IRQ+0x4d/0x63>> [   97.520018]  [<c022d350>] xen_evtchn_do_upcall+0x8b/0xf1>> [   97.520018]  [<c0109543>] xen_do_upcall+0x7/0xc>> [   97.520018]  =======================>> [   97.520018] Code: 8b 43 1c 89 44 24 04 89 44 24 14 e9 e2 00 00 00>> 8b 43 20 31 f6 48 23 44 24 04 6b c0 6c 8d 78 40 03 7b 24 8b 2f 69 c5>> 9c 00 00 00 <8b> 94 18 c0 00 00 00 8d 44 18 54 89 44 24 10 89 14 24 eb>> 12 8b>> [   97.520018] EIP: [<d0837767>] blkif_interrupt+0x5d/0x18a>>>> I haven't see this oops before, but I think the code in question> correlates to:>>        bret = RING_GET_RESPONSE(&info->ring, i);>        id   = bret->id;>        req  = (struct request *)info->shadow[id].request;>> which suggests the backend driver is passing back a bogus request ID.>> What's your host environment?>
Debian, freshly installed from a recent lenny test CD, amd64
Kernel is 2.6.18 amd64 from Debian etch
2.6.18-6-xen-amd64 #1 SMP Tue Aug 19 06:15:09 UTC 2008 x86_64 GNU/LinuxЪТХ╨{.nг+┴╥÷╝┴╜├+%┼кЪ╠Ищ╤\x17╔┼wЪ╨{.nг+┴╥╔┼{╠ЧG╚²ИЪ┼{ay╨\x1dй┤з≥К,j\a╜╒fё╒╥h ▐О│ЙЪ▒ЙГz_Х╝\x03(╜И ▌┼щ╒j"²З\x1a╤^[m╖ЪЪ╬\a╚ЧG╚²ИЪ╒╦?≥╗Х╜з&ёЬ╖~▐А╤iO∙Ф╛z╥ vь^\x14\x04\x1a╤^[m╖ЪЪц\fЪ╤ЛЪ╒╦?√I╔