kvm-84 and guests with more than 3536 MB Ram?

kvm-84 and guests with more than 3536 MB Ram?

[Lukas Kolbe]

Hi!

This is my first post here so please bear with me; we have a Debian
Lenny system with kernel 2.6.28 and kvm-84, and can't start a guest with
more than 3536 MB Ram. With kvm-72 (the version lenny released with) we
can use all 7GB that is intended for that guest.

Is this a known bug? Someone on freenode #kvm said he'd seen something
like this but couldn't remember where or when.

-- 
Lukas

Re: kvm-84 and guests with more than 3536 MB Ram?

[Avi Kivity]

Lukas Kolbe wrote:
> Hi!
>
> This is my first post here so please bear with me; we have a Debian
> Lenny system with kernel 2.6.28 and kvm-84, and can't start a guest with
> more than 3536 MB Ram. With kvm-72 (the version lenny released with) we
> can use all 7GB that is intended for that guest.
>
> Is this a known bug? Someone on freenode #kvm said he'd seen something
> like this but couldn't remember where or when.
>
>   

Can you describe the failure?  I just booted a guest (FC6) with 6GB, 
without issue (admittedly the guest is only using 300MB or so).


-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: kvm-84 and guests with more than 3536 MB Ram?

[Lukas Kolbe]

On Di, 2009-03-24 at 14:59 +0200, Avi Kivity wrote:
> Lukas Kolbe wrote:
> > Hi!
> >
> > This is my first post here so please bear with me; we have a Debian
> > Lenny system with kernel 2.6.28 and kvm-84, and can't start a guest with
> > more than 3536 MB Ram. With kvm-72 (the version lenny released with) we
> > can use all 7GB that is intended for that guest.
> >
> > Is this a known bug? Someone on freenode #kvm said he'd seen something
> > like this but couldn't remember where or when.
> >
> >   
> 
> Can you describe the failure?  I just booted a guest (FC6) with 6GB, 
> without issue (admittedly the guest is only using 300MB or so).

Well, when I start the guest like this:

kvm -smp 8 -drive if=virtio,file=/dev/system/test_root,boot=on -m 3585 \
-nographic -name test -kernel /boot/kvm/test/vmlinuz-2.6.28-1-amd64 \
-initrd /boot/kvm/test/initrd.img-2.6.28-1-amd64 \
-append 'root=/dev/vda ro console=ttyS0,115200' \
-serial mon:unix:/etc/kvm/consoles/test.sock,server,nowait \

I get the following output:

qemu: loading initrd (0x781b93 bytes) at 0x000000007f87e000
create_userspace_phys_mem: Invalid argument
kvm_cpu_register_physical_memory: failed

And back to the console. When I try the same with 3584MB, I can boot
into the machine flawlessly.

Sorry for getting the numbers wrong in the first mail - the actual
problem starts at 3585MB Ram for the guest.

If you can't reproduce it with yout 2.6.28 and kvm-84, I should possibly
take this to the debian bugtracker ...

-- 
Lukas

Re: kvm-84 and guests with more than 3536 MB Ram?

[Anthony Liguori]

Lukas Kolbe wrote:
> On Di, 2009-03-24 at 14:59 +0200, Avi Kivity wrote:
>   
>> Lukas Kolbe wrote:
>>     
>>> Hi!
>>>
>>> This is my first post here so please bear with me; we have a Debian
>>> Lenny system with kernel 2.6.28 and kvm-84, and can't start a guest with
>>> more than 3536 MB Ram. With kvm-72 (the version lenny released with) we
>>> can use all 7GB that is intended for that guest.
>>>
>>> Is this a known bug? Someone on freenode #kvm said he'd seen something
>>> like this but couldn't remember where or when.
>>>
>>>   
>>>       
>> Can you describe the failure?  I just booted a guest (FC6) with 6GB, 
>> without issue (admittedly the guest is only using 300MB or so).
>>     
>
> Well, when I start the guest like this:
>
> kvm -smp 8 -drive if=virtio,file=/dev/system/test_root,boot=on -m 3585 \
> -nographic -name test -kernel /boot/kvm/test/vmlinuz-2.6.28-1-amd64 \
> -initrd /boot/kvm/test/initrd.img-2.6.28-1-amd64 \
> -append 'root=/dev/vda ro console=ttyS0,115200' \
> -serial mon:unix:/etc/kvm/consoles/test.sock,server,nowait \
>
> I get the following output:
>
> qemu: loading initrd (0x781b93 bytes) at 0x000000007f87e000
> create_userspace_phys_mem: Invalid argument
> kvm_cpu_register_physical_memory: failed
>
> And back to the console. When I try the same with 3584MB, I can boot
> into the machine flawlessly.
>
> Sorry for getting the numbers wrong in the first mail - the actual
> problem starts at 3585MB Ram for the guest.
>
> If you can't reproduce it with yout 2.6.28 and kvm-84, I should possibly
> take this to the debian bugtracker ...
>   

kvm-72 is pretty old.  It used to be that we used phys_ram_base for 
loading kernel/initrds which would break when using > 3.5GB of memory.  
I wouldn't be surprised if that fix happened post kvm-72.

Regards,

Anthony Liguori

Re: kvm-84 and guests with more than 3536 MB Ram?

[Tomasz Chmielewski]

Anthony Liguori schrieb:
> Lukas Kolbe wrote:
>> On Di, 2009-03-24 at 14:59 +0200, Avi Kivity wrote:
>>  
>>> Lukas Kolbe wrote:
>>>    
>>>> Hi!
>>>>
>>>> This is my first post here so please bear with me; we have a Debian
>>>> Lenny system with kernel 2.6.28 and kvm-84, and can't start a guest 
>>>> with
>>>> more than 3536 MB Ram. With kvm-72 (the version lenny released with) we
>>>> can use all 7GB that is intended for that guest.

(...)

>> qemu: loading initrd (0x781b93 bytes) at 0x000000007f87e000
>> create_userspace_phys_mem: Invalid argument
>> kvm_cpu_register_physical_memory: failed
>>
>> And back to the console. When I try the same with 3584MB, I can boot
>> into the machine flawlessly.
>>
>> Sorry for getting the numbers wrong in the first mail - the actual
>> problem starts at 3585MB Ram for the guest.
>>
>> If you can't reproduce it with yout 2.6.28 and kvm-84, I should possibly
>> take this to the debian bugtracker ...
>>   
> 
> kvm-72 is pretty old.  It used to be that we used phys_ram_base for 
> loading kernel/initrds which would break when using > 3.5GB of memory.  
> I wouldn't be surprised if that fix happened post kvm-72.

Doesn't he say that it did work for him with kvm-72, but does not with 
kvm-84?


-- 
Tomasz Chmielewski
http://wpkg.org

Re: kvm-84 and guests with more than 3536 MB Ram?

[Lukas Kolbe]

Am Dienstag, den 24.03.2009, 17:04 +0100 schrieb Tomasz Chmielewski:
> Anthony Liguori schrieb:
> > Lukas Kolbe wrote:
> >> On Di, 2009-03-24 at 14:59 +0200, Avi Kivity wrote:
> >>  
> >>> Lukas Kolbe wrote:
> >>>    
> >>>> Hi!
> >>>>
> >>>> This is my first post here so please bear with me; we have a Debian
> >>>> Lenny system with kernel 2.6.28 and kvm-84, and can't start a guest 
> >>>> with
> >>>> more than 3536 MB Ram. With kvm-72 (the version lenny released with) we
> >>>> can use all 7GB that is intended for that guest.
> 
> (...)
> 
> >> qemu: loading initrd (0x781b93 bytes) at 0x000000007f87e000
> >> create_userspace_phys_mem: Invalid argument
> >> kvm_cpu_register_physical_memory: failed
> >>
> >> And back to the console. When I try the same with 3584MB, I can boot
> >> into the machine flawlessly.
> >>
> >> Sorry for getting the numbers wrong in the first mail - the actual
> >> problem starts at 3585MB Ram for the guest.
> >>
> >> If you can't reproduce it with yout 2.6.28 and kvm-84, I should possibly
> >> take this to the debian bugtracker ...
> >>   
> > 
> > kvm-72 is pretty old.  It used to be that we used phys_ram_base for 
> > loading kernel/initrds which would break when using > 3.5GB of memory.  
> > I wouldn't be surprised if that fix happened post kvm-72.
> 
> Doesn't he say that it did work for him with kvm-72, but does not with 
> kvm-84?

Exactly that :)
kvm-72 works like a charm, with the noted exception that it crashes
every now and than on our system (during heavy load), which was the
reason we tried 084 in the first place.

-- 
Lukas

Re: kvm-84 and guests with more than 3536 MB Ram?

[Ryan Harper]

* Lukas Kolbe <l-lists@einfachkaffee.de> [2009-03-24 14:32]:
> Am Dienstag, den 24.03.2009, 17:04 +0100 schrieb Tomasz Chmielewski:
> > Anthony Liguori schrieb:
> > > Lukas Kolbe wrote:
> > >> On Di, 2009-03-24 at 14:59 +0200, Avi Kivity wrote:
> > >>  
> > >>> Lukas Kolbe wrote:
> > >>>    
> > >>>> Hi!
> > >>>>
> > >>>> This is my first post here so please bear with me; we have a Debian
> > >>>> Lenny system with kernel 2.6.28 and kvm-84, and can't start a guest 
> > >>>> with
> > >>>> more than 3536 MB Ram. With kvm-72 (the version lenny released with) we
> > >>>> can use all 7GB that is intended for that guest.
> > 
> > (...)
> > 
> > >> qemu: loading initrd (0x781b93 bytes) at 0x000000007f87e000
> > >> create_userspace_phys_mem: Invalid argument
> > >> kvm_cpu_register_physical_memory: failed
> > >>
> > >> And back to the console. When I try the same with 3584MB, I can boot
> > >> into the machine flawlessly.
> > >>
> > >> Sorry for getting the numbers wrong in the first mail - the actual
> > >> problem starts at 3585MB Ram for the guest.
> > >>
> > >> If you can't reproduce it with yout 2.6.28 and kvm-84, I should possibly
> > >> take this to the debian bugtracker ...
> > >>   
> > > 
> > > kvm-72 is pretty old.  It used to be that we used phys_ram_base for 
> > > loading kernel/initrds which would break when using > 3.5GB of memory.  
> > > I wouldn't be surprised if that fix happened post kvm-72.
> > 
> > Doesn't he say that it did work for him with kvm-72, but does not with 
> > kvm-84?
> 
> Exactly that :)
> kvm-72 works like a charm, with the noted exception that it crashes
> every now and than on our system (during heavy load), which was the
> reason we tried 084 in the first place.

running kvm-84 with -m 8192 -- passes memtest on ubuntu 8.10 x86_64
server cd.  Maybe wrong bios?  From kvm-84 make install, I have:

% md5sum /usr/local/share/qemu/bios.bin 
470d43e4838be608a2447b23a9f83d90  /usr/local/share/qemu/bios.bin


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com

Re: kvm-84 and guests with more than 3536 MB Ram?

[Lukas Kolbe]

Am Dienstag, den 24.03.2009, 15:40 -0500 schrieb Ryan Harper:

Hi Ryan,

> > > Doesn't he say that it did work for him with kvm-72, but does not with 
> > > kvm-84?
> > 
> > Exactly that :)
> > kvm-72 works like a charm, with the noted exception that it crashes
> > every now and than on our system (during heavy load), which was the
> > reason we tried 084 in the first place.
> 
> running kvm-84 with -m 8192 -- passes memtest on ubuntu 8.10 x86_64
> server cd.  Maybe wrong bios?  From kvm-84 make install, I have:
> 
> % md5sum /usr/local/share/qemu/bios.bin 
> 470d43e4838be608a2447b23a9f83d90  /usr/local/share/qemu/bios.bin

Hm, ours is

% md5sum /usr/share/kvm/bios.bin
a1dbfc18f5f5656da939cc8c243741da  /usr/share/kvm/bios.bin

I'll try to circumvent debian and install kvm-84 from the vanilla
sources to see wether it makes a difference, in the meantime I'll open a
bug report with debian.

-- 
Lukas

Re: kvm-84 and guests with more than 3536 MB Ram?

[Ryan Harper]

* Lukas Kolbe <l-lists@einfachkaffee.de> [2009-03-24 16:04]:
> Am Dienstag, den 24.03.2009, 15:40 -0500 schrieb Ryan Harper:
> 
> Hi Ryan,
> 
> > > > Doesn't he say that it did work for him with kvm-72, but does not with 
> > > > kvm-84?
> > > 
> > > Exactly that :)
> > > kvm-72 works like a charm, with the noted exception that it crashes
> > > every now and than on our system (during heavy load), which was the
> > > reason we tried 084 in the first place.
> > 
> > running kvm-84 with -m 8192 -- passes memtest on ubuntu 8.10 x86_64
> > server cd.  Maybe wrong bios?  From kvm-84 make install, I have:
> > 
> > % md5sum /usr/local/share/qemu/bios.bin 
> > 470d43e4838be608a2447b23a9f83d90  /usr/local/share/qemu/bios.bin
> 
> Hm, ours is
> 
> % md5sum /usr/share/kvm/bios.bin
> a1dbfc18f5f5656da939cc8c243741da  /usr/share/kvm/bios.bin
> 
> I'll try to circumvent debian and install kvm-84 from the vanilla
> sources to see wether it makes a difference, in the meantime I'll open a
> bug report with debian.

you can test kvm-84 bios with:

% cd kvm-84/qemu
% ./x86_64-softmmu/qemu-system-x86_64 -L pc-bios <all your other options>

That will force qemu to look in the kvm-84 pc-bios dir for bios instead
of /usr/local/share/qemu

if *that* works, then you didn't make install kvm-84, and running with
mismatched bios bins is sure way screw things up.


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com

Re: kvm-84 and guests with more than 3536 MB Ram?

[Lukas Kolbe]

On Di, 2009-03-24 at 16:27 -0500, Ryan Harper wrote:

Hi Ryan (and all), the issue seems to be solved, see below.
 
> you can test kvm-84 bios with:
> 
> % cd kvm-84/qemu
> % ./x86_64-softmmu/qemu-system-x86_64 -L pc-bios <all your other options>
> 
> That will force qemu to look in the kvm-84 pc-bios dir for bios instead
> of /usr/local/share/qemu
> 
> if *that* works, then you didn't make install kvm-84, and running with
> mismatched bios bins is sure way screw things up.

This didn't work (neither the self-compiled kvm-84 nor the
debian-provided one), but using a newer 2.6.28 snapshot from debian
(based on 2.6.28.8, where the previously used one was based on .4 IIRC)
we are able to boot into the 7G Ram guest again. Thanks for your help
(and patience)!


-- 
Lukas