kernel Bug after upgrading to 3.0.2-2

kernel Bug after upgrading to 3.0.2-2

[Brian Hays]

[-- Attachment #1.1: Type: text/plain, Size: 1590 bytes --]

Here's the error seen on many of the domU console after upgrading to 3.0.2-2.
Anyone seen this or know of a fix?

 ------------[ cut here ]------------
kernel BUG at arch/xen/i386/mm/pgtable.c:285!
invalid operand: 0000 [#3]
SMP
Modules linked in: sd_mod scsi_mod
CPU:    0
EIP:    0061:[<c0114c06>]    Not tainted VLI
EFLAGS: 00010282   (2.6.12.6-vpsX)
EIP is at pgd_ctor+0x26/0x30
eax: fffffff4   ebx: 00000001   ecx: f578c000   edx: 00000000
esi: c112ed80   edi: c2fbf9f8   ebp: c2fbf9e0   esp: c2b83dc4
ds: 007b   es: 007b   ss: 0069
Process cron (pid: 13727, threadinfo=c2b82000 task=c4344020)
Stack: c3a37000 00000000 00000020 c01493e1 c3a37000 c112ed80 00000001
c2fbf9e0
       c3a37000 c112ed80 00000000 c0149583 c112ed80 c2fbf9e0 00000001
000000d0
       c112ede0 00000001 000000d0 c113c980 0000000c 000000d0 00000000
c014979b
Call Trace:
 [<c01493e1>] cache_init_objs+0x71/0x80
 [<c0149583>] cache_grow+0x113/0x1b0
 [<c014979b>] cache_alloc_refill+0x17b/0x230
 [<c02522a1>] _atomic_dec_and_lock+0x31/0x50
 [<c0149a6f>] kmem_cache_alloc+0x7f/0x90
 [<c0114c8d>] pgd_alloc+0x1d/0x310
 [<c017b943>] dput+0x33/0x1d0
 [<c011dd4e>] mm_init+0xce/0x100
 [<c011ddb6>] mm_alloc+0x36/0x50
 [<c016f4fc>] do_execve+0x7c/0x220
 [<c01090e6>] sys_execve+0x46/0xa0
 [<c010a6dd>] syscall_call+0x7/0xb
Code: 00 f3 ab 5f c3 83 ec 0c b8 20 00 00 00 89 44 24 08 31 c0 89 44 24 04
8b 44 24 10 89 04 24 e8 d2 28 00 00 85 c0 75 04 83 c4 0c c3 <0f> 0b 1d 01 aa
4b 31 c0 eb f2 83 ec 08 b8 90 a9 34 c0 89 5c 24
 ------------[ cut here ]------------

Thank you,
Brian

[-- Attachment #1.2: Type: text/html, Size: 2021 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: kernel Bug after upgrading to 3.0.2-2

[Ian Pratt]

> Here's the error seen on many of the domU console after 
> upgrading to 3.0.2-2. Anyone seen this or know of a fix?

Please can you try booting setting dom0_mem explicitly on the xen
command line rather than relying on the the 'auto ballooning'
dom0_min_mem setting in the xend-config.sxp file.

Thanks,
Ian
 
>  ------------[ cut here ]------------
> kernel BUG at arch/xen/i386/mm/pgtable.c:285!
> invalid operand: 0000 [#3] 
> SMP
> Modules linked in: sd_mod scsi_mod
> CPU:    0
> EIP:    0061:[<c0114c06>]    Not tainted VLI
> EFLAGS: 00010282   (2.6.12.6-vpsX)
> EIP is at pgd_ctor+0x26/0x30
> eax: fffffff4   ebx: 00000001   ecx: f578c000   edx: 00000000 
> esi: c112ed80   edi: c2fbf9f8   ebp: c2fbf9e0   esp: c2b83dc4
> ds: 007b   es: 007b   ss: 0069
> Process cron (pid: 13727, threadinfo=c2b82000 task=c4344020)
> Stack: c3a37000 00000000 00000020 c01493e1 c3a37000 c112ed80 
> 00000001 c2fbf9e0 
>        c3a37000 c112ed80 00000000 c0149583 c112ed80 c2fbf9e0 
> 00000001 000000d0
>        c112ede0 00000001 000000d0 c113c980 0000000c 000000d0 
> 00000000 c014979b
> Call Trace:
>  [<c01493e1>] cache_init_objs+0x71/0x80 
>  [<c0149583>] cache_grow+0x113/0x1b0
>  [<c014979b>] cache_alloc_refill+0x17b/0x230
>  [<c02522a1>] _atomic_dec_and_lock+0x31/0x50
>  [<c0149a6f>] kmem_cache_alloc+0x7f/0x90
>  [<c0114c8d>] pgd_alloc+0x1d/0x310 
>  [<c017b943>] dput+0x33/0x1d0
>  [<c011dd4e>] mm_init+0xce/0x100
>  [<c011ddb6>] mm_alloc+0x36/0x50
>  [<c016f4fc>] do_execve+0x7c/0x220
>  [<c01090e6>] sys_execve+0x46/0xa0
>  [<c010a6dd>] syscall_call+0x7/0xb 
> Code: 00 f3 ab 5f c3 83 ec 0c b8 20 00 00 00 89 44 24 08 31 
> c0 89 44 24 04 8b 44 24 10 89 04 24 e8 d2 28 00 00 85 c0 75 
> 04 83 c4 0c c3 <0f> 0b 1d 01 aa 4b 31 c0 eb f2 83 ec 08 b8 90 
> a9 34 c0 89 5c 24
>  ------------[ cut here ]------------ 
> 
> Thank you,
> Brian
> 
> 

Re: kernel Bug after upgrading to 3.0.2-2

[Brian Hays]

[-- Attachment #1.1: Type: text/plain, Size: 2242 bytes --]

Already have it set. Here's my grub.conf settings ...

title Xen 3.0 / 2.6.16
        kernel /xen.gz dom0_mem=200000
        module /vmlinuz-2.6.16-xen0 root=/dev/sda2 ro

Thank you,
Brian

On 4/28/06, Ian Pratt <m+Ian.Pratt@cl.cam.ac.uk> wrote:
>
> > Here's the error seen on many of the domU console after
> > upgrading to 3.0.2-2. Anyone seen this or know of a fix?
>
> Please can you try booting setting dom0_mem explicitly on the xen
> command line rather than relying on the the 'auto ballooning'
> dom0_min_mem setting in the xend-config.sxp file.
>
> Thanks,
> Ian
>
> >  ------------[ cut here ]------------
> > kernel BUG at arch/xen/i386/mm/pgtable.c:285!
> > invalid operand: 0000 [#3]
> > SMP
> > Modules linked in: sd_mod scsi_mod
> > CPU:    0
> > EIP:    0061:[<c0114c06>]    Not tainted VLI
> > EFLAGS: 00010282   (2.6.12.6-vpsX)
> > EIP is at pgd_ctor+0x26/0x30
> > eax: fffffff4   ebx: 00000001   ecx: f578c000   edx: 00000000
> > esi: c112ed80   edi: c2fbf9f8   ebp: c2fbf9e0   esp: c2b83dc4
> > ds: 007b   es: 007b   ss: 0069
> > Process cron (pid: 13727, threadinfo=c2b82000 task=c4344020)
> > Stack: c3a37000 00000000 00000020 c01493e1 c3a37000 c112ed80
> > 00000001 c2fbf9e0
> >        c3a37000 c112ed80 00000000 c0149583 c112ed80 c2fbf9e0
> > 00000001 000000d0
> >        c112ede0 00000001 000000d0 c113c980 0000000c 000000d0
> > 00000000 c014979b
> > Call Trace:
> >  [<c01493e1>] cache_init_objs+0x71/0x80
> >  [<c0149583>] cache_grow+0x113/0x1b0
> >  [<c014979b>] cache_alloc_refill+0x17b/0x230
> >  [<c02522a1>] _atomic_dec_and_lock+0x31/0x50
> >  [<c0149a6f>] kmem_cache_alloc+0x7f/0x90
> >  [<c0114c8d>] pgd_alloc+0x1d/0x310
> >  [<c017b943>] dput+0x33/0x1d0
> >  [<c011dd4e>] mm_init+0xce/0x100
> >  [<c011ddb6>] mm_alloc+0x36/0x50
> >  [<c016f4fc>] do_execve+0x7c/0x220
> >  [<c01090e6>] sys_execve+0x46/0xa0
> >  [<c010a6dd>] syscall_call+0x7/0xb
> > Code: 00 f3 ab 5f c3 83 ec 0c b8 20 00 00 00 89 44 24 08 31
> > c0 89 44 24 04 8b 44 24 10 89 04 24 e8 d2 28 00 00 85 c0 75
> > 04 83 c4 0c c3 <0f> 0b 1d 01 aa 4b 31 c0 eb f2 83 ec 08 b8 90
> > a9 34 c0 89 5c 24
> >  ------------[ cut here ]------------
> >
> > Thank you,
> > Brian
> >
> >
>

[-- Attachment #1.2: Type: text/html, Size: 3209 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel Bug after upgrading to 3.0.2-2

[Keir Fraser]

On 28 Apr 2006, at 17:06, Brian Hays wrote:

> Already have it set. Here's my grub.conf settings ...
>
> title Xen 3.0 / 2.6.16
>         kernel /xen.gz dom0_mem=200000
>         module /vmlinuz-2.6.16-xen0 root=/dev/sda2 ro

A little-known fact about PAE Xen is that you really want to use the 
lowmem_emergency_pool boot parameter.

e.g., lowmem_emegency_pool=16M

Each megabyte of emergency pool will allow you to guarantee to be able 
to support 256 processes (counted across all VMs that you are running). 
So the above value would be good for 4096 processes. You'll have to 
decide what you need to set that parameter to.

  -- Keir

Re: kernel Bug after upgrading to 3.0.2-2

[Brian Hays]

[-- Attachment #1.1: Type: text/plain, Size: 1030 bytes --]

Thank you. This would be specified on the "kernel" line, correct? Also, do
you have any suggestions as to what a good value may be for a machine
running 15 domU's? Is there a general "rule-of-thumb" that one could use for
this setting based on a 'per domU' basis?

Thanks,
Brian

On 4/28/06, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
>
>
> On 28 Apr 2006, at 17:06, Brian Hays wrote:
>
> > Already have it set. Here's my grub.conf settings ...
> >
> > title Xen 3.0 / 2.6.16
> > kernel /xen.gz dom0_mem=200000
> > module /vmlinuz-2.6.16-xen0 root=/dev/sda2 ro
>
> A little-known fact about PAE Xen is that you really want to use the
> lowmem_emergency_pool boot parameter.
>
> e.g., lowmem_emegency_pool=16M
>
> Each megabyte of emergency pool will allow you to guarantee to be able
> to support 256 processes (counted across all VMs that you are running).
> So the above value would be good for 4096 processes. You'll have to
> decide what you need to set that parameter to.
>
>   -- Keir
>
>

[-- Attachment #1.2: Type: text/html, Size: 1367 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel Bug after upgrading to 3.0.2-2

[Keir Fraser]

On 28 Apr 2006, at 17:27, Brian Hays wrote:

> hank you. This would be specified on the "kernel" line, correct? Also, 
> do you have any suggestions as to what a good value may be for a 
> machine running 15 domU's? Is there a general "rule-of-thumb" that one 
> could use for this setting based on a 'per domU' basis?

Yes, it goes on the kernel line.

Setting it to 128M would be reasonably conservative (allowing 2000 
processes per VM). The downside is that you may 'waste' up to 128MB RAM 
if you aren't actually running very many processes.

We're working on making the failure mode less terminal. :-)

  -- Keir

Re: kernel Bug after upgrading to 3.0.2-2

[Christopher G. Stach II]

Keir Fraser wrote:
> Each megabyte of emergency pool will allow you to guarantee to be able
> to support 256 processes (counted across all VMs that you are running).
> So the above value would be good for 4096 processes. You'll have to
> decide what you need to set that parameter to.

Is this including LWPs?

-- 
Christopher G. Stach II

Re: kernel Bug after upgrading to 3.0.2-2

[Keir Fraser]

On 28 Apr 2006, at 20:24, Christopher G. Stach II wrote:

>> Each megabyte of emergency pool will allow you to guarantee to be able
>> to support 256 processes (counted across all VMs that you are 
>> running).
>> So the above value would be good for 4096 processes. You'll have to
>> decide what you need to set that parameter to.
>
> Is this including LWPs?

No, threads sharing an address space do not count.

  -- Keir

Re: kernel Bug after upgrading to 3.0.2-2

[Jan Beulich]

>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 28.04.06 18:22 >>>
>
>On 28 Apr 2006, at 17:06, Brian Hays wrote:
>
>> Already have it set. Here's my grub.conf settings ...
>>
>> title Xen 3.0 / 2.6.16
>>         kernel /xen.gz dom0_mem=200000
>>         module /vmlinuz-2.6.16-xen0 root=/dev/sda2 ro
>
>A little-known fact about PAE Xen is that you really want to use the 
>lowmem_emergency_pool boot parameter.
>
>e.g., lowmem_emegency_pool=16M
>
>Each megabyte of emergency pool will allow you to guarantee to be able 
>to support 256 processes (counted across all VMs that you are running). 
>So the above value would be good for 4096 processes. You'll have to 
>decide what you need to set that parameter to.

As I understand it, this is only needed to be this large because xenlinux allocates an entire page for the pgd where 32
bytes would be sufficient (as is done in native Linux). Wouldn't it thus make more sense to work towards a solution
where, like on native, the other possible pgdirs on the same page can actually get used. As allocation already goes
through a constructor this wouldn't seem this hard. On the other hand, if it's not hard, you would probably have done it
already, so I guess I'm missing something here...

Jan

Re: kernel Bug after upgrading to 3.0.2-2

[Keir Fraser]

On 2 May 2006, at 16:21, Jan Beulich wrote:

> As I understand it, this is only needed to be this large because 
> xenlinux allocates an entire page for the pgd where 32
> bytes would be sufficient (as is done in native Linux). Wouldn't it 
> thus make more sense to work towards a solution
> where, like on native, the other possible pgdirs on the same page can 
> actually get used. As allocation already goes
> through a constructor this wouldn't seem this hard. On the other hand, 
> if it's not hard, you would probably have done it
> already, so I guess I'm missing something here...

At the moment PAE page directories always need to be page-aligned. 
That's probably not a difficult restriction to lift though.

  -- Keir