Large sized guest taking for ever to boot...

Large sized guest taking for ever to boot...

[Chegu Vinod]

Hello,

I picked up a recent version of the qemu (1.0.92 with some fixes) and tried it 
on x86_64 server (with host and the guest running 3.4.1 kernel). 

While trying to boot a large guest (80 vcpus + 512GB) I observed that the guest 
took for ever to boot up...  ~1 hr or even more. [This wasn't the case when I 
was using RHEL 6.x related bits]

Lots of the following type of errors show up on the console...

...

udevd [886] : worker: [7321] unexpectedly returned with status  0x0100
udevd [886] : worker: [7321] failed while handling 
'/devices/system/memory/memory1351'

udevd [886] : worker: [7371] unexpectedly returned with status  0x0100
udevd [886] : worker: [7371] failed while handling 
'/devices/system/memory/memory1352'

....


Wondering if this is due to some very basic configuration issue on my side? or 
if I am using an in-comptible version of the qemu with host and guest kernel..

Vinod

Re: Large sized guest taking for ever to boot...

[Alex Williamson]

On Fri, 2012-06-08 at 16:29 +0000, Chegu Vinod wrote:
> Hello,
> 
> I picked up a recent version of the qemu (1.0.92 with some fixes) and tried it 
> on x86_64 server (with host and the guest running 3.4.1 kernel). 
> 
> While trying to boot a large guest (80 vcpus + 512GB) I observed that the guest 
> took for ever to boot up...  ~1 hr or even more. [This wasn't the case when I 
> was using RHEL 6.x related bits]

Was either case using device assignment?  Device assignment will map and
pin each page of guest memory before startup, which can be a noticeable
pause on smallish (<16GB) guests.  That should be linear scaling though
and if you're using qemu and not qemu-kvm, not related.  Thanks,

Alex

Re: Large sized guest taking for ever to boot...

[Chegu Vinod]

On 6/8/2012 9:46 AM, Alex Williamson wrote:
> On Fri, 2012-06-08 at 16:29 +0000, Chegu Vinod wrote:
>> Hello,
>>
>> I picked up a recent version of the qemu (1.0.92 with some fixes) and tried it
>> on x86_64 server (with host and the guest running 3.4.1 kernel).

BTW, I observe the same thing if i were to use 1.1.50 version of the 
qemu... not sure if this is really
related to qemu...

>>
>> While trying to boot a large guest (80 vcpus + 512GB) I observed that the guest
>> took for ever to boot up...  ~1 hr or even more. [This wasn't the case when I
>> was using RHEL 6.x related bits]
> Was either case using device assignment?  Device assignment will map and
> pin each page of guest memory before startup, which can be a noticeable
> pause on smallish (<16GB) guests.  That should be linear scaling though
> and if you're using qemu and not qemu-kvm, not related.  Thanks,

I don't have any device assignment at this point . Yes I am using qemu  
(not qemu-kvm)...

The issue seems very basic... 'was earlier running RHEL6.3 RC1 on the 
host and the guest and the host and the guest seemed to boot fine..

Then I switched both the host and the guest to use 3.4.1 kernel (and the 
recent qemu). udevd is unhappy...

Perhaps the existing udevd is incompatible with 3.4.1 kernel or doesn't 
like something in the pre-existing "database" of devices....


Vinod

>
> Alex
>

Re: Large sized guest taking for ever to boot...

[Alex Williamson]

On Fri, 2012-06-08 at 10:10 -0700, Chegu Vinod wrote:
> On 6/8/2012 9:46 AM, Alex Williamson wrote:
> > On Fri, 2012-06-08 at 16:29 +0000, Chegu Vinod wrote:
> >> Hello,
> >>
> >> I picked up a recent version of the qemu (1.0.92 with some fixes) and tried it
> >> on x86_64 server (with host and the guest running 3.4.1 kernel).
> 
> BTW, I observe the same thing if i were to use 1.1.50 version of the 
> qemu... not sure if this is really
> related to qemu...
> 
> >>
> >> While trying to boot a large guest (80 vcpus + 512GB) I observed that the guest
> >> took for ever to boot up...  ~1 hr or even more. [This wasn't the case when I
> >> was using RHEL 6.x related bits]
> > Was either case using device assignment?  Device assignment will map and
> > pin each page of guest memory before startup, which can be a noticeable
> > pause on smallish (<16GB) guests.  That should be linear scaling though
> > and if you're using qemu and not qemu-kvm, not related.  Thanks,
> 
> I don't have any device assignment at this point . Yes I am using qemu  
> (not qemu-kvm)...

Just to be safe, are you using --enable-kvm with qemu?

> The issue seems very basic... 'was earlier running RHEL6.3 RC1 on the 
> host and the guest and the host and the guest seemed to boot fine..

Note that RHEL is based on qemu-kvm.  Thanks,

Alex

Re: Large sized guest taking for ever to boot...

[Chegu Vinod]

On 6/8/2012 10:42 AM, Alex Williamson wrote:
> On Fri, 2012-06-08 at 10:10 -0700, Chegu Vinod wrote:
>> On 6/8/2012 9:46 AM, Alex Williamson wrote:
>>> On Fri, 2012-06-08 at 16:29 +0000, Chegu Vinod wrote:
>>>> Hello,
>>>>
>>>> I picked up a recent version of the qemu (1.0.92 with some fixes) and tried it
>>>> on x86_64 server (with host and the guest running 3.4.1 kernel).
>> BTW, I observe the same thing if i were to use 1.1.50 version of the
>> qemu... not sure if this is really
>> related to qemu...
>>
>>>> While trying to boot a large guest (80 vcpus + 512GB) I observed that the guest
>>>> took for ever to boot up...  ~1 hr or even more. [This wasn't the case when I
>>>> was using RHEL 6.x related bits]
>>> Was either case using device assignment?  Device assignment will map and
>>> pin each page of guest memory before startup, which can be a noticeable
>>> pause on smallish (<16GB) guests.  That should be linear scaling though
>>> and if you're using qemu and not qemu-kvm, not related.  Thanks,
>> I don't have any device assignment at this point . Yes I am using qemu
>> (not qemu-kvm)...
> Just to be safe, are you using --enable-kvm with qemu?

Yes...

-----

/etc/qemu-ifup tap0

/usr/local/bin/qemu-system-x86_64 -enable-kvm \

-cpu 
Westmere,+rdtscp,+pdpe1gb,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pclmuldq,+pbe,+tm,+ht,+s
s,+acpi,+ds,+vme \
-m 524288 -smp 80,sockets=80,cores=1,threads=1 \
-name vm1 \
-chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/vm1.monitor,server,nowait \
-drive 
file=/dev/libvirt_lvm/vm.img,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native 
\
-device 
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 
\
-monitor stdio \
-net nic,macaddr=52:54:00:71:01:01 \
-net tap,ifname=tap0,script=no,downscript=no \
-vnc :4

/etc/qemu-ifdown tap0

----

>> The issue seems very basic... 'was earlier running RHEL6.3 RC1 on the
>> host and the guest and the host and the guest seemed to boot fine..
> Note that RHEL is based on qemu-kvm.  Thanks,

Yep..knew that :)

I was using upstream qemu-kvm and was encouraged to move away from 
it...to qemu.

Vinod

>
> Alex
>

Re: Large sized guest taking for ever to boot...

[Jan Kiszka]

[CC'ing qemu as this discusses its code base]

On 2012-06-08 19:57, Chegu Vinod wrote:
> On 6/8/2012 10:42 AM, Alex Williamson wrote:
>> On Fri, 2012-06-08 at 10:10 -0700, Chegu Vinod wrote:
>>> On 6/8/2012 9:46 AM, Alex Williamson wrote:
>>>> On Fri, 2012-06-08 at 16:29 +0000, Chegu Vinod wrote:
>>>>> Hello,
>>>>>
>>>>> I picked up a recent version of the qemu (1.0.92 with some fixes)
>>>>> and tried it
>>>>> on x86_64 server (with host and the guest running 3.4.1 kernel).
>>> BTW, I observe the same thing if i were to use 1.1.50 version of the
>>> qemu... not sure if this is really
>>> related to qemu...
>>>
>>>>> While trying to boot a large guest (80 vcpus + 512GB) I observed
>>>>> that the guest
>>>>> took for ever to boot up...  ~1 hr or even more. [This wasn't the
>>>>> case when I
>>>>> was using RHEL 6.x related bits]
>>>> Was either case using device assignment?  Device assignment will map
>>>> and
>>>> pin each page of guest memory before startup, which can be a noticeable
>>>> pause on smallish (<16GB) guests.  That should be linear scaling though
>>>> and if you're using qemu and not qemu-kvm, not related.  Thanks,
>>> I don't have any device assignment at this point . Yes I am using qemu
>>> (not qemu-kvm)...
>> Just to be safe, are you using --enable-kvm with qemu?
> 
> Yes...

Unless you are using current qemu.git master (where it is enabled by
default), --enable-kvm does not activate the in-kernel irqchip support
for you. Not sure if that can make such a huge difference, but it is a
variation from qemu-kvm. You can enable it in qemu-1.1 with -machine
kernel_irqchip=on.

> 
> -----
> 
> /etc/qemu-ifup tap0
> 
> /usr/local/bin/qemu-system-x86_64 -enable-kvm \
> 
> -cpu
> Westmere,+rdtscp,+pdpe1gb,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pclmuldq,+pbe,+tm,+ht,+s
> 
> s,+acpi,+ds,+vme \
> -m 524288 -smp 80,sockets=80,cores=1,threads=1 \
> -name vm1 \
> -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/vm1.monitor,server,nowait
> \
> -drive
> file=/dev/libvirt_lvm/vm.img,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native
> \
> -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
> \
> -monitor stdio \
> -net nic,macaddr=52:54:00:71:01:01 \
> -net tap,ifname=tap0,script=no,downscript=no \
> -vnc :4
> 
> /etc/qemu-ifdown tap0
> 
> ----
> 
>>> The issue seems very basic... 'was earlier running RHEL6.3 RC1 on the
>>> host and the guest and the host and the guest seemed to boot fine..
>> Note that RHEL is based on qemu-kvm.  Thanks,
> 
> Yep..knew that :)
> 
> I was using upstream qemu-kvm and was encouraged to move away from
> it...to qemu.

And that is good. :)

Is the problem present in current qemu-kvm.git? If yes, can you bisect
when it was introduced?

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: Large sized guest taking for ever to boot...

[Chegu Vinod]

On 6/8/2012 11:08 AM, Jan Kiszka wrote:
> [CC'ing qemu as this discusses its code base]
>
> On 2012-06-08 19:57, Chegu Vinod wrote:
>> On 6/8/2012 10:42 AM, Alex Williamson wrote:
>>> On Fri, 2012-06-08 at 10:10 -0700, Chegu Vinod wrote:
>>>> On 6/8/2012 9:46 AM, Alex Williamson wrote:
>>>>> On Fri, 2012-06-08 at 16:29 +0000, Chegu Vinod wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I picked up a recent version of the qemu (1.0.92 with some fixes)
>>>>>> and tried it
>>>>>> on x86_64 server (with host and the guest running 3.4.1 kernel).
>>>> BTW, I observe the same thing if i were to use 1.1.50 version of the
>>>> qemu... not sure if this is really
>>>> related to qemu...
>>>>
>>>>>> While trying to boot a large guest (80 vcpus + 512GB) I observed
>>>>>> that the guest
>>>>>> took for ever to boot up...  ~1 hr or even more. [This wasn't the
>>>>>> case when I
>>>>>> was using RHEL 6.x related bits]
>>>>> Was either case using device assignment?  Device assignment will map
>>>>> and
>>>>> pin each page of guest memory before startup, which can be a noticeable
>>>>> pause on smallish (<16GB) guests.  That should be linear scaling though
>>>>> and if you're using qemu and not qemu-kvm, not related.  Thanks,
>>>> I don't have any device assignment at this point . Yes I am using qemu
>>>> (not qemu-kvm)...
>>> Just to be safe, are you using --enable-kvm with qemu?
>> Yes...
> Unless you are using current qemu.git master (where it is enabled by
> default), --enable-kvm does not activate the in-kernel irqchip support
> for you. Not sure if that can make such a huge difference, but it is a
> variation from qemu-kvm. You can enable it in qemu-1.1 with -machine
> kernel_irqchip=on.

Thanks for pointing this out...i will add that.

I was using qemu.git    not the master


>> -----
>>
>> /etc/qemu-ifup tap0
>>
>> /usr/local/bin/qemu-system-x86_64 -enable-kvm \
>>
>> -cpu
>> Westmere,+rdtscp,+pdpe1gb,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pclmuldq,+pbe,+tm,+ht,+s
>>
>> s,+acpi,+ds,+vme \
>> -m 524288 -smp 80,sockets=80,cores=1,threads=1 \
>> -name vm1 \
>> -chardev
>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/vm1.monitor,server,nowait
>> \
>> -drive
>> file=/dev/libvirt_lvm/vm.img,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native
>> \
>> -device
>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
>> \
>> -monitor stdio \
>> -net nic,macaddr=52:54:00:71:01:01 \
>> -net tap,ifname=tap0,script=no,downscript=no \
>> -vnc :4
>>
>> /etc/qemu-ifdown tap0
>>
>> ----
>>
>>>> The issue seems very basic... 'was earlier running RHEL6.3 RC1 on the
>>>> host and the guest and the host and the guest seemed to boot fine..
>>> Note that RHEL is based on qemu-kvm.  Thanks,
>> Yep..knew that :)
>>
>> I was using upstream qemu-kvm and was encouraged to move away from
>> it...to qemu.
> And that is good. :)
>
> Is the problem present in current qemu-kvm.git? If yes, can you bisect
> when it was introduced?
Shall try out the qemu-kvm.git  (after finishing some experiments..).

BTW,  another data point ...if I try to boot a the RHEL6.3 kernel in the 
guest (with the latest qemu.git and the 3.4.1 on the host) it boots just 
fine....

So something to do with the 3.4.1 kernel in the guest and the existing 
udev... Need to dig
deeper.

Vinod
>
> Jan
>

Re: Large sized guest taking for ever to boot...

[Jan Kiszka]

On 2012-06-08 20:20, Chegu Vinod wrote:
> On 6/8/2012 11:08 AM, Jan Kiszka wrote:
>> [CC'ing qemu as this discusses its code base]
>>
>> On 2012-06-08 19:57, Chegu Vinod wrote:
>>> On 6/8/2012 10:42 AM, Alex Williamson wrote:
>>>> On Fri, 2012-06-08 at 10:10 -0700, Chegu Vinod wrote:
>>>>> On 6/8/2012 9:46 AM, Alex Williamson wrote:
>>>>>> On Fri, 2012-06-08 at 16:29 +0000, Chegu Vinod wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> I picked up a recent version of the qemu (1.0.92 with some fixes)
>>>>>>> and tried it
>>>>>>> on x86_64 server (with host and the guest running 3.4.1 kernel).
>>>>> BTW, I observe the same thing if i were to use 1.1.50 version of the
>>>>> qemu... not sure if this is really
>>>>> related to qemu...
>>>>>
>>>>>>> While trying to boot a large guest (80 vcpus + 512GB) I observed
>>>>>>> that the guest
>>>>>>> took for ever to boot up...  ~1 hr or even more. [This wasn't the
>>>>>>> case when I
>>>>>>> was using RHEL 6.x related bits]
>>>>>> Was either case using device assignment?  Device assignment will map
>>>>>> and
>>>>>> pin each page of guest memory before startup, which can be a noticeable
>>>>>> pause on smallish (<16GB) guests.  That should be linear scaling though
>>>>>> and if you're using qemu and not qemu-kvm, not related.  Thanks,
>>>>> I don't have any device assignment at this point . Yes I am using qemu
>>>>> (not qemu-kvm)...
>>>> Just to be safe, are you using --enable-kvm with qemu?
>>> Yes...
>> Unless you are using current qemu.git master (where it is enabled by
>> default), --enable-kvm does not activate the in-kernel irqchip support
>> for you. Not sure if that can make such a huge difference, but it is a
>> variation from qemu-kvm. You can enable it in qemu-1.1 with -machine
>> kernel_irqchip=on.
> 
> Thanks for pointing this out...i will add that.
> 
> I was using qemu.git    not the master

With "qemu.git master" I meant the latest version you can pull from the
master branch or qemu.git. If you are using an older version, please
specify the hash.

BTW, you can check if irqchip is on by looking at the out of "info
qtree" in the qemu monitor. One of the last devices listed must be
called "kvm-apic".

> 
> 
>>> -----
>>>
>>> /etc/qemu-ifup tap0
>>>
>>> /usr/local/bin/qemu-system-x86_64 -enable-kvm \
>>>
>>> -cpu
>>> Westmere,+rdtscp,+pdpe1gb,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pclmuldq,+pbe,+tm,+ht,+s
>>>
>>> s,+acpi,+ds,+vme \
>>> -m 524288 -smp 80,sockets=80,cores=1,threads=1 \
>>> -name vm1 \
>>> -chardev
>>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/vm1.monitor,server,nowait
>>> \
>>> -drive
>>> file=/dev/libvirt_lvm/vm.img,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native
>>> \
>>> -device
>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
>>> \
>>> -monitor stdio \
>>> -net nic,macaddr=52:54:00:71:01:01 \
>>> -net tap,ifname=tap0,script=no,downscript=no \
>>> -vnc :4
>>>
>>> /etc/qemu-ifdown tap0
>>>
>>> ----
>>>
>>>>> The issue seems very basic... 'was earlier running RHEL6.3 RC1 on the
>>>>> host and the guest and the host and the guest seemed to boot fine..
>>>> Note that RHEL is based on qemu-kvm.  Thanks,
>>> Yep..knew that :)
>>>
>>> I was using upstream qemu-kvm and was encouraged to move away from
>>> it...to qemu.
>> And that is good. :)
>>
>> Is the problem present in current qemu-kvm.git? If yes, can you bisect
>> when it was introduced?
> Shall try out the qemu-kvm.git  (after finishing some experiments..).

Yes, please.

> 
> BTW,  another data point ...if I try to boot a the RHEL6.3 kernel in the 
> guest (with the latest qemu.git and the 3.4.1 on the host) it boots just 
> fine....
> 
> So something to do with the 3.4.1 kernel in the guest and the existing 
> udev... Need to dig
> deeper.

Maybe. But lets focus on getting the problematic guest running fine in
some configuration. If that turns out to be impossible, we may see a
guest issue.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: Large sized guest taking for ever to boot...

[Chegu Vinod]

On 6/8/2012 11:37 AM, Jan Kiszka wrote:
> On 2012-06-08 20:20, Chegu Vinod wrote:
>> On 6/8/2012 11:08 AM, Jan Kiszka wrote:
>>> [CC'ing qemu as this discusses its code base]
>>>
>>> On 2012-06-08 19:57, Chegu Vinod wrote:
>>>> On 6/8/2012 10:42 AM, Alex Williamson wrote:
>>>>> On Fri, 2012-06-08 at 10:10 -0700, Chegu Vinod wrote:
>>>>>> On 6/8/2012 9:46 AM, Alex Williamson wrote:
>>>>>>> On Fri, 2012-06-08 at 16:29 +0000, Chegu Vinod wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I picked up a recent version of the qemu (1.0.92 with some fixes)
>>>>>>>> and tried it
>>>>>>>> on x86_64 server (with host and the guest running 3.4.1 kernel).
>>>>>> BTW, I observe the same thing if i were to use 1.1.50 version of the
>>>>>> qemu... not sure if this is really
>>>>>> related to qemu...
>>>>>>
>>>>>>>> While trying to boot a large guest (80 vcpus + 512GB) I observed
>>>>>>>> that the guest
>>>>>>>> took for ever to boot up...  ~1 hr or even more. [This wasn't the
>>>>>>>> case when I
>>>>>>>> was using RHEL 6.x related bits]
>>>>>>> Was either case using device assignment?  Device assignment will map
>>>>>>> and
>>>>>>> pin each page of guest memory before startup, which can be a noticeable
>>>>>>> pause on smallish (<16GB) guests.  That should be linear scaling though
>>>>>>> and if you're using qemu and not qemu-kvm, not related.  Thanks,
>>>>>> I don't have any device assignment at this point . Yes I am using qemu
>>>>>> (not qemu-kvm)...
>>>>> Just to be safe, are you using --enable-kvm with qemu?
>>>> Yes...
>>> Unless you are using current qemu.git master (where it is enabled by
>>> default), --enable-kvm does not activate the in-kernel irqchip support
>>> for you. Not sure if that can make such a huge difference, but it is a
>>> variation from qemu-kvm. You can enable it in qemu-1.1 with -machine
>>> kernel_irqchip=on.
>> Thanks for pointing this out...i will add that.
>>
>> I was using qemu.git    not the master
> With "qemu.git master" I meant the latest version you can pull from the
> master branch or qemu.git. If you are using an older version, please
> specify the hash.
>
> BTW, you can check if irqchip is on by looking at the out of "info
> qtree" in the qemu monitor. One of the last devices listed must be
> called "kvm-apic".


Sorry for the confusion.  I was using the qemu.git  and I do see the 
"kvm-apic" stuff  (in the info qtree output) without specifying the 
-machine kernel_irqchip=on option..


>
>>
>>>> -----
>>>>
>>>> /etc/qemu-ifup tap0
>>>>
>>>> /usr/local/bin/qemu-system-x86_64 -enable-kvm \
>>>>
>>>> -cpu
>>>> Westmere,+rdtscp,+pdpe1gb,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pclmuldq,+pbe,+tm,+ht,+s
>>>>
>>>> s,+acpi,+ds,+vme \
>>>> -m 524288 -smp 80,sockets=80,cores=1,threads=1 \
>>>> -name vm1 \
>>>> -chardev
>>>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/vm1.monitor,server,nowait
>>>> \
>>>> -drive
>>>> file=/dev/libvirt_lvm/vm.img,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native
>>>> \
>>>> -device
>>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
>>>> \
>>>> -monitor stdio \
>>>> -net nic,macaddr=52:54:00:71:01:01 \
>>>> -net tap,ifname=tap0,script=no,downscript=no \
>>>> -vnc :4
>>>>
>>>> /etc/qemu-ifdown tap0
>>>>
>>>> ----
>>>>
>>>>>> The issue seems very basic... 'was earlier running RHEL6.3 RC1 on the
>>>>>> host and the guest and the host and the guest seemed to boot fine..
>>>>> Note that RHEL is based on qemu-kvm.  Thanks,
>>>> Yep..knew that :)
>>>>
>>>> I was using upstream qemu-kvm and was encouraged to move away from
>>>> it...to qemu.
>>> And that is good. :)
>>>
>>> Is the problem present in current qemu-kvm.git? If yes, can you bisect
>>> when it was introduced?
>> Shall try out the qemu-kvm.git  (after finishing some experiments..).
> Yes, please.

Just did that and below are the results...

>
>> BTW,  another data point ...if I try to boot a the RHEL6.3 kernel in the
>> guest (with the latest qemu.git and the 3.4.1 on the host) it boots just
>> fine....
>>
>> So something to do with the 3.4.1 kernel in the guest and the existing
>> udev... Need to dig
>> deeper.
> Maybe. But lets focus on getting the problematic guest running fine in
> some configuration. If that turns out to be impossible, we may see a
> guest issue.
Host config. : 80 Cores + 1TB.    Guest config : 40VCPUs + 512GB.

I rebuilt the 3.4.1 kernel in the guest from scratch and retried my 
experiments and measured
the boot times...

a) Host :  RHEL6.3 RC1 + qemu-kvm (that came with it) &  Guest : RHEL6.3 
RC1:  ~1 min

b) Host :3.4.1 + qemu-kvm.git & Guest : RHEL6.3RC1      -   ~1 min

c) Host :3.4.1 + qemu-kvm.git & Guest : 3.4.1      -   ~10 mins

d) Host :3.4.1 + qemu.git & Guest : RHEL6.3 RC1  -    ~1 min

e) Host :3.4.1 + qemu.git & Guest : 3.4.1                   -    ~14 mins


In both the case (c) & (e) had quite a few warning/error messages from 
udevd.

FYI..
Vinod
PS:  Haven't had the time to do any further analysis...as the machine is 
being used for
other experiments...


>
> Jan
>

Re: Large sized guest taking for ever to boot...

[Gleb Natapov]

On Tue, Jun 12, 2012 at 08:33:59AM -0700, Chegu Vinod wrote:
> I rebuilt the 3.4.1 kernel in the guest from scratch and retried my
> experiments and measured
> the boot times...
> 
> a) Host :  RHEL6.3 RC1 + qemu-kvm (that came with it) &  Guest :
> RHEL6.3 RC1:  ~1 min
> 
> b) Host :3.4.1 + qemu-kvm.git & Guest : RHEL6.3RC1      -   ~1 min
> 
> c) Host :3.4.1 + qemu-kvm.git & Guest : 3.4.1      -   ~10 mins
> 
> d) Host :3.4.1 + qemu.git & Guest : RHEL6.3 RC1  -    ~1 min
> 
> e) Host :3.4.1 + qemu.git & Guest : 3.4.1                   -    ~14 mins
> 
> 
> In both the case (c) & (e) had quite a few warning/error messages
> from udevd.
> 
> FYI..
> Vinod
> PS:  Haven't had the time to do any further analysis...as the
> machine is being used for
> other experiments...
> 
> 
Next time you get the machine can you try running (b) but adding
-hypervisor to your -cpu config.

--
			Gleb.

Re: Large sized guest taking for ever to boot...

[Chegu Vinod]

On 6/12/2012 8:39 AM, Gleb Natapov wrote:
> On Tue, Jun 12, 2012 at 08:33:59AM -0700, Chegu Vinod wrote:
>> I rebuilt the 3.4.1 kernel in the guest from scratch and retried my
>> experiments and measured
>> the boot times...
>>
>> a) Host :  RHEL6.3 RC1 + qemu-kvm (that came with it)&   Guest :
>> RHEL6.3 RC1:  ~1 min
>>
>> b) Host :3.4.1 + qemu-kvm.git&  Guest : RHEL6.3RC1      -   ~1 min
>>
>> c) Host :3.4.1 + qemu-kvm.git&  Guest : 3.4.1      -   ~10 mins
>>
>> d) Host :3.4.1 + qemu.git&  Guest : RHEL6.3 RC1  -    ~1 min
>>
>> e) Host :3.4.1 + qemu.git&  Guest : 3.4.1                   -    ~14 mins
>>
>>
>> In both the case (c)&  (e) had quite a few warning/error messages
>> from udevd.
>>
>> FYI..
>> Vinod
>> PS:  Haven't had the time to do any further analysis...as the
>> machine is being used for
>> other experiments...
>>
>>
> Next time you get the machine can you try running (b) but adding
> -hypervisor to your -cpu config.

I tried that and didn't make any difference...i.e. the RHEL6.3 RC1 guest 
booted in about ~1 min.

Vinod
>
> --
> 			Gleb.
>

Re: Large sized guest taking for ever to boot...

[Gleb Natapov]

On Tue, Jun 12, 2012 at 11:44:07AM -0700, Chegu Vinod wrote:
> On 6/12/2012 8:39 AM, Gleb Natapov wrote:
> >On Tue, Jun 12, 2012 at 08:33:59AM -0700, Chegu Vinod wrote:
> >>I rebuilt the 3.4.1 kernel in the guest from scratch and retried my
> >>experiments and measured
> >>the boot times...
> >>
> >>a) Host :  RHEL6.3 RC1 + qemu-kvm (that came with it)&   Guest :
> >>RHEL6.3 RC1:  ~1 min
> >>
> >>b) Host :3.4.1 + qemu-kvm.git&  Guest : RHEL6.3RC1      -   ~1 min
> >>
> >>c) Host :3.4.1 + qemu-kvm.git&  Guest : 3.4.1      -   ~10 mins
> >>
> >>d) Host :3.4.1 + qemu.git&  Guest : RHEL6.3 RC1  -    ~1 min
> >>
> >>e) Host :3.4.1 + qemu.git&  Guest : 3.4.1                   -    ~14 mins
> >>
> >>
> >>In both the case (c)&  (e) had quite a few warning/error messages
> >>from udevd.
> >>
> >>FYI..
> >>Vinod
> >>PS:  Haven't had the time to do any further analysis...as the
> >>machine is being used for
> >>other experiments...
> >>
> >>
> >Next time you get the machine can you try running (b) but adding
> >-hypervisor to your -cpu config.
> 
> I tried that and didn't make any difference...i.e. the RHEL6.3 RC1
> guest booted in about ~1 min.
> 
Thank you. So this means that ticket spinlock is not to blame for the
slow down compared ti RHEL kernel.

Where 3.4.1 spends most of the time while booting? Is slow down begins
before or after the kernel starts userspace?

--
			Gleb.

Re: Large sized guest taking for ever to boot...

[Avi Kivity]

On 06/13/2012 10:12 AM, Gleb Natapov wrote:
>> 
>> I tried that and didn't make any difference...i.e. the RHEL6.3 RC1
>> guest booted in about ~1 min.
>> 
> Thank you. So this means that ticket spinlock is not to blame for the
> slow down compared ti RHEL kernel.
> 
> Where 3.4.1 spends most of the time while booting? Is slow down begins
> before or after the kernel starts userspace?

In addition:
   kvm_stat
   perf top
   perf kvm top

while the slowdown occurs.



-- 
error compiling committee.c: too many arguments to function

Re: Large sized guest taking for ever to boot...

[Gleb Natapov]

On Fri, Jun 08, 2012 at 11:20:53AM -0700, Chegu Vinod wrote:
> On 6/8/2012 11:08 AM, Jan Kiszka wrote:
> BTW,  another data point ...if I try to boot a the RHEL6.3 kernel in
> the guest (with the latest qemu.git and the 3.4.1 on the host) it
> boots just fine....
> 
> So something to do with the 3.4.1 kernel in the guest and the
> existing udev... Need to dig
> deeper.
> 
How many CPUs do you have on the host? RHEL6.3 uses unfair spinlock
when it runs as a guest.

--
			Gleb.

Re: Large sized guest taking for ever to boot...

[Chegu Vinod]

On 6/10/2012 2:30 AM, Gleb Natapov wrote:
> On Fri, Jun 08, 2012 at 11:20:53AM -0700, Chegu Vinod wrote:
>> On 6/8/2012 11:08 AM, Jan Kiszka wrote:
>> BTW,  another data point ...if I try to boot a the RHEL6.3 kernel in
>> the guest (with the latest qemu.git and the 3.4.1 on the host) it
>> boots just fine....
>>
>> So something to do with the 3.4.1 kernel in the guest and the
>> existing udev... Need to dig
>> deeper.
>>
> How many CPUs do you have on the host? RHEL6.3 uses unfair spinlock
> when it runs as a guest.

80 active cores on the host.

Vinod
>
> --
> 			Gleb.
>

Re: Large sized guest taking for ever to boot...

[Chegu Vinod]

On 6/8/2012 10:10 AM, Chegu Vinod wrote:
> On 6/8/2012 9:46 AM, Alex Williamson wrote:
>> On Fri, 2012-06-08 at 16:29 +0000, Chegu Vinod wrote:
>>> Hello,
>>>
>>> I picked up a recent version of the qemu (1.0.92 with some fixes) 
>>> and tried it
>>> on x86_64 server (with host and the guest running 3.4.1 kernel).
>
> BTW, I observe the same thing if i were to use 1.1.50 version of the 
> qemu... not sure if this is really
> related to qemu...
>
>>>
>>> While trying to boot a large guest (80 vcpus + 512GB) I observed 
>>> that the guest
>>> took for ever to boot up...  ~1 hr or even more. [This wasn't the 
>>> case when I
>>> was using RHEL 6.x related bits]
>> Was either case using device assignment?  Device assignment will map and
>> pin each page of guest memory before startup, which can be a noticeable
>> pause on smallish (<16GB) guests.  That should be linear scaling though
>> and if you're using qemu and not qemu-kvm, not related.  Thanks,
>
> I don't have any device assignment at this point . Yes I am using 
> qemu  (not qemu-kvm)...
>
> The issue seems very basic... 'was earlier running RHEL6.3 RC1 on the 
> host and the guest and the host and the guest seemed to boot fine..
>
> Then I switched both the host and the guest to use 3.4.1 kernel (and 
> the recent qemu). udevd is unhappy...
>
> Perhaps the existing udevd is incompatible with 3.4.1 kernel or 
> doesn't like something in the pre-existing "database" of devices....

Currently the host and the guest are using udev version 147.  Host  
seemed to boot ok with 3.4.1 kernel and this version of udev... but the 
guest is the one that has these problems...

Wondering if any others have observed similar problems  ? If yes...Did 
updating the udev to a more recent version solve it ?

Any clues/pointers are appreciated...

Thanks
Vinod
>
>
> Vinod
>
>>
>> Alex
>>
>