Testing nested virtualization

Testing nested virtualization

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 1516 bytes --]


Hi, all!

I'm testing nested virtualization with Linux 2.6.32.2 and
qemu-kvm-0.12.1.1 in the host.

I was looking for if in the KVM official site there is something
documented, but I didn't find information so any reference will be
welcome.

What I got to do until the moment is to load the module in host of the
following way:

# modprobe kvm_amd nested=1

Then, to boot guest with the parameter -enable-nesting:

# /usr/local/qemu-kvm/bin/qemu-system-x86_64 -hda /dev/vm/test-disk -m \
  1024 -boot c -net nic,vlan=0,macaddr=00:16:3e:00:00:34 -net tap \
  -daemonize -vnc :3 -k es -localtime -serial \
  telnet:localhost:5000,server,nowait -monitor \
  telnet:localhost:5001,server,nowait -enable-nesting

In this guest I'm using Linux 2.6.32, with userspace kvm-85 of
backports. For the VM booted within of 'test', I am using a qcow2 file.
But when connecting via VNC, the boot process is hung in "Press for F12
boot menu". Sometimes it gets to pass this instance and I see the Debian
install menu, but while I cross the options, it is hung.

With these options I boot the guest within test:

# kvm -hda disk01.img -m 512 -boot d -cdrom debian-500-amd64-netinst.iso \
  -net nic,vlan=0,macaddr=00:16:3e:00:00:33 -net tap -daemonize -vnc :2 \
  -k es -localtime


Which can be the problem?

Thanks in advance for your replies.

Regards,
Daniel
-- 
Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Testing nested virtualization

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 816 bytes --]

On Friday, 01 January 2010 17:20:00 -0300,
Daniel Bareiro wrote:

> In this guest I'm using Linux 2.6.32, with userspace kvm-85 of
> backports. For the VM booted within of 'test', I am using a qcow2
> file. But when connecting via VNC, the boot process is hung in "Press
> for F12 boot menu". Sometimes it gets to pass this instance and I see
> the Debian install menu, but while I cross the options, it is hung.

Also I was trying to use qemu-kvm-0.12.1.1 with to Linux 2.6.32 in
guest within 'test'. And here it happens something similar. Sometimes
I get to select the option of the menu of the installer, but after
boot, the installation is hung again.

Regards,
Daniel
-- 
Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Testing nested virtualization

[Alexander Graf]

On 01.01.2010, at 21:56, Daniel Bareiro wrote:

> On Friday, 01 January 2010 17:20:00 -0300,
> Daniel Bareiro wrote:
> 
>> In this guest I'm using Linux 2.6.32, with userspace kvm-85 of
>> backports. For the VM booted within of 'test', I am using a qcow2
>> file. But when connecting via VNC, the boot process is hung in "Press
>> for F12 boot menu". Sometimes it gets to pass this instance and I see
>> the Debian install menu, but while I cross the options, it is hung.
> 
> Also I was trying to use qemu-kvm-0.12.1.1 with to Linux 2.6.32 in
> guest within 'test'. And here it happens something similar. Sometimes
> I get to select the option of the menu of the installer, but after
> boot, the installation is hung again.

Can your host do nested paging? Shadow shadow paging can be a bit flaky at times :-(.

Alex

Re: Testing nested virtualization

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 1789 bytes --]

Hi, Alexander.

On Saturday, 02 January 2010 14:11:04 +0100,
Alexander Graf wrote:

> >> In this guest I'm using Linux 2.6.32, with userspace kvm-85 of
> >> backports. For the VM booted within of 'test', I am using a qcow2
> >> file. But when connecting via VNC, the boot process is hung in
> >> "Press for F12 boot menu". Sometimes it gets to pass this instance
> >> and I see the Debian install menu, but while I cross the options,
> >> it is hung.

> > Also I was trying to use qemu-kvm-0.12.1.1 with to Linux 2.6.32 in
> > guest within 'test'. And here it happens something similar.
> > Sometimes I get to select the option of the menu of the installer,
> > but after boot, the installation is hung again.
 
> Can your host do nested paging? Shadow shadow paging can be a bit
> flaky at times :-(.

Hmmm... is there some way to verify that?

The CPUs model is AMD Athlon 64 X2 Dual Core 3800+. Perhaps some flag
related? In such case, these are the available ones in the host:

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext
fxsr_opt rdtscp lm 3dnowext 3dnow rep_good pni cx16 lahf_lm cmp_legacy
svm extapic cr8_legacy


When loading the module with "kvm_amd nested=1", /var/log/messages
registers the following thing:

kvm: Nested Virtualization enabled


And these are flags in test, booting with -enable-nesting:

flags           : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm up pni cx16
hypervisor lahf_lm svm


Thanks for your reply.

Regards,
Daniel
-- 
Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Testing nested virtualization

[Alexander Graf]

On 02.01.2010, at 14:45, Daniel Bareiro wrote:

> Hi, Alexander.
> 
> On Saturday, 02 January 2010 14:11:04 +0100,
> Alexander Graf wrote:
> 
>>>> In this guest I'm using Linux 2.6.32, with userspace kvm-85 of
>>>> backports. For the VM booted within of 'test', I am using a qcow2
>>>> file. But when connecting via VNC, the boot process is hung in
>>>> "Press for F12 boot menu". Sometimes it gets to pass this instance
>>>> and I see the Debian install menu, but while I cross the options,
>>>> it is hung.
> 
>>> Also I was trying to use qemu-kvm-0.12.1.1 with to Linux 2.6.32 in
>>> guest within 'test'. And here it happens something similar.
>>> Sometimes I get to select the option of the menu of the installer,
>>> but after boot, the installation is hung again.
> 
>> Can your host do nested paging? Shadow shadow paging can be a bit
>> flaky at times :-(.
> 
> Hmmm... is there some way to verify that?
> 
> The CPUs model is AMD Athlon 64 X2 Dual Core 3800+. Perhaps some flag
> related? In such case, these are the available ones in the host:
> 
> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext
> fxsr_opt rdtscp lm 3dnowext 3dnow rep_good pni cx16 lahf_lm cmp_legacy
> svm extapic cr8_legacy
> 
> 
> When loading the module with "kvm_amd nested=1", /var/log/messages
> registers the following thing:
> 
> kvm: Nested Virtualization enabled

With nested paging it would show something like this in dmesg:

kvm: Nested Paging enabled

All family 10 AMD CPUs have nested paging (read: quad-core and above).


So I guess you found something going wrong with shadow-shadow paging. CC'ing Jörg. He's more into memory management than me :-).


Alex

Re: Testing nested virtualization

[Alexander Graf]

On 02.01.2010, at 14:48, Alexander Graf wrote:

> 
> On 02.01.2010, at 14:45, Daniel Bareiro wrote:
> 
>> Hi, Alexander.
>> 
>> On Saturday, 02 January 2010 14:11:04 +0100,
>> Alexander Graf wrote:
>> 
>>>>> In this guest I'm using Linux 2.6.32, with userspace kvm-85 of
>>>>> backports. For the VM booted within of 'test', I am using a qcow2
>>>>> file. But when connecting via VNC, the boot process is hung in
>>>>> "Press for F12 boot menu". Sometimes it gets to pass this instance
>>>>> and I see the Debian install menu, but while I cross the options,
>>>>> it is hung.
>> 
>>>> Also I was trying to use qemu-kvm-0.12.1.1 with to Linux 2.6.32 in
>>>> guest within 'test'. And here it happens something similar.
>>>> Sometimes I get to select the option of the menu of the installer,
>>>> but after boot, the installation is hung again.
>> 
>>> Can your host do nested paging? Shadow shadow paging can be a bit
>>> flaky at times :-(.
>> 
>> Hmmm... is there some way to verify that?
>> 
>> The CPUs model is AMD Athlon 64 X2 Dual Core 3800+. Perhaps some flag
>> related? In such case, these are the available ones in the host:
>> 
>> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
>> mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext
>> fxsr_opt rdtscp lm 3dnowext 3dnow rep_good pni cx16 lahf_lm cmp_legacy
>> svm extapic cr8_legacy
>> 
>> 
>> When loading the module with "kvm_amd nested=1", /var/log/messages
>> registers the following thing:
>> 
>> kvm: Nested Virtualization enabled
> 
> With nested paging it would show something like this in dmesg:
> 
> kvm: Nested Paging enabled
> 
> All family 10 AMD CPUs have nested paging (read: quad-core and above).
> 
> 
> So I guess you found something going wrong with shadow-shadow paging. CC'ing Jörg. He's more into memory management than me :-).

Eh - CC'ing :-).

Alex

Re: Testing nested virtualization

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 2270 bytes --]

Hi, Alex.

On Saturday, 02 January 2010 14:48:26 +0100,
Alexander Graf wrote:

> >>>> In this guest I'm using Linux 2.6.32, with userspace kvm-85 of
> >>>> backports. For the VM booted within of 'test', I am using a qcow2
> >>>> file. But when connecting via VNC, the boot process is hung in
> >>>> "Press for F12 boot menu". Sometimes it gets to pass this
> >>>> instance and I see the Debian install menu, but while I cross the
> >>>> options, it is hung.

> >>> Also I was trying to use qemu-kvm-0.12.1.1 with to Linux 2.6.32 in
> >>> guest within 'test'. And here it happens something similar.
> >>> Sometimes I get to select the option of the menu of the installer,
> >>> but after boot, the installation is hung again.

> >> Can your host do nested paging? Shadow shadow paging can be a bit
> >> flaky at times :-(.
 
> > Hmmm... is there some way to verify that?
> > 
> > The CPUs model is AMD Athlon 64 X2 Dual Core 3800+. Perhaps some flag
> > related? In such case, these are the available ones in the host:
> > 
> > flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> > mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext
> > fxsr_opt rdtscp lm 3dnowext 3dnow rep_good pni cx16 lahf_lm cmp_legacy
> > svm extapic cr8_legacy
> > 
> > 
> > When loading the module with "kvm_amd nested=1", /var/log/messages
> > registers the following thing:
> > 
> > kvm: Nested Virtualization enabled

> With nested paging it would show something like this in dmesg:
> 
> kvm: Nested Paging enabled

Hmmm... dmesg shows just like /var/log/messages:


kvm: Nested Virtualization enabled


"Nested Paging enabled" would have to be a complementary message?
 
> All family 10 AMD CPUs have nested paging (read: quad-core and above).

But this model is dual-Core :-(. That would mean that it cannot do
nested paging?
 
> So I guess you found something going wrong with shadow-shadow paging.
> CC'ing Jörg. He's more into memory management than me :-).

We hope then that Jörg can illuminate a little more to us :-).

Thanks for your reply.

Regards,
Daniel
-- 
Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Testing nested virtualization

[Joerg Roedel]

> >>>> Also I was trying to use qemu-kvm-0.12.1.1 with to Linux 2.6.32 in
> >>>> guest within 'test'. And here it happens something similar.
> >>>> Sometimes I get to select the option of the menu of the installer,
> >>>> but after boot, the installation is hung again.

The problem has probably to do with the TSC bugs I fixed lately for
nested SVM. You can try to disable KVM-Clock for the L1 and the L2 guest
or use the latest 2.6.32.x kernel on the Host. Does one of this fix the
issues for you?

	Joerg

Re: Testing nested virtualization

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 1127 bytes --]

Hi, Joerg.

On Monday, 04 January 2010 12:11:46 +0100,
Joerg Roedel wrote:

> > >>>> Also I was trying to use qemu-kvm-0.12.1.1 with to Linux 2.6.32
> > >>>> in guest within 'test'. And here it happens something similar.
> > >>>> Sometimes I get to select the option of the menu of the
> > >>>> installer, but after boot, the installation is hung again.
 
> The problem has probably to do with the TSC bugs I fixed lately for
> nested SVM. You can try to disable KVM-Clock for the L1 and the L2
> guest or use the latest 2.6.32.x kernel on the Host. Does one of this
> fix the issues for you?

I'm using Linux 2.6.32.2 with qemu-kvm-0.12.1.1 in the host, so I will
try to disable KVM-Clock for L1 and L2 guest. How I can get it?

Alexander said in another mail that AMD family 10 have nested paging
(read quad-core and above) but my processor is dual-core. The problem
can be related to that?

Thanks for your reply.

Regards,
Daniel
-- 
Daniel Bareiro - GNU/Linux registered user #188.598
Proudly running Debian GNU/Linux with uptime:
10:52:11 up 19:37, 11 users,  load average: 0.12, 0.17, 0.11

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Testing nested virtualization

[Alexander Graf]

On 04.01.2010, at 15:20, Daniel Bareiro wrote:

> Hi, Joerg.
> 
> On Monday, 04 January 2010 12:11:46 +0100,
> Joerg Roedel wrote:
> 
>>>>>>> Also I was trying to use qemu-kvm-0.12.1.1 with to Linux 2.6.32
>>>>>>> in guest within 'test'. And here it happens something similar.
>>>>>>> Sometimes I get to select the option of the menu of the
>>>>>>> installer, but after boot, the installation is hung again.
> 
>> The problem has probably to do with the TSC bugs I fixed lately for
>> nested SVM. You can try to disable KVM-Clock for the L1 and the L2
>> guest or use the latest 2.6.32.x kernel on the Host. Does one of this
>> fix the issues for you?
> 
> I'm using Linux 2.6.32.2 with qemu-kvm-0.12.1.1 in the host, so I will
> try to disable KVM-Clock for L1 and L2 guest. How I can get it?
> 
> Alexander said in another mail that AMD family 10 have nested paging
> (read quad-core and above) but my processor is dual-core. The problem
> can be related to that?

It just means you're using code paths that I usually don't use, since my main development machine can do nested paging. It should work nevertheless, just be horribly slow.

Alex

Re: Testing nested virtualization

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 1891 bytes --]

Hi, Alex.

On Monday, 04 January 2010 16:07:52 +0100,
Alexander Graf wrote:

> >>>>>>> Also I was trying to use qemu-kvm-0.12.1.1 with to Linux
> >>>>>>> 2.6.32 in guest within 'test'. And here it happens something
> >>>>>>> similar.  Sometimes I get to select the option of the menu of
> >>>>>>> the installer, but after boot, the installation is hung again.

> >> The problem has probably to do with the TSC bugs I fixed lately for
> >> nested SVM. You can try to disable KVM-Clock for the L1 and the L2
> >> guest or use the latest 2.6.32.x kernel on the Host. Does one of
> >> this fix the issues for you?

> > I'm using Linux 2.6.32.2 with qemu-kvm-0.12.1.1 in the host, so I
> > will try to disable KVM-Clock for L1 and L2 guest. How I can get it?
> > 
> > Alexander said in another mail that AMD family 10 have nested paging
> > (read quad-core and above) but my processor is dual-core. The
> > problem can be related to that?

> It just means you're using code paths that I usually don't use, since
> my main development machine can do nested paging. It should work
> nevertheless, just be horribly slow.

Then what it would seem it hangs can in fact be a horribly slow
operation?

What it draws attention to me is that, as I said in another mail, this
happens at different moments. Can this be related to TSC bug mentioned
by Joerg? The 'Clocksource tsc unstable' in dmesg is due to this bug? 

In 'test' host I've the following thing:

test:~# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
kvm-clock

test:~# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
kvm-clock hpet acpi_pm


Thanks for your reply.

Regards,
Daniel
-- 
Daniel Bareiro - GNU/Linux registered user #188.598
Proudly running Debian GNU/Linux with uptime:
12:21:25 up 21:06, 11 users,  load average: 0.08, 0.11, 0.09

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Testing nested virtualization

[Joerg Roedel]

Hi Daniel,

On Mon, Jan 04, 2010 at 12:52:47PM -0300, Daniel Bareiro wrote:
> test:~# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
> kvm-clock
> 
> test:~# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
> kvm-clock hpet acpi_pm

Can you try to boot L1 and L2 guest with the 'no-kvmclock' kernel
parameter? This disables the kvm-clock.

	Joerg

Re: Testing nested virtualization

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 1147 bytes --]

On Monday, 04 January 2010 17:04:16 +0100,
Joerg Roedel wrote:

> Hi Daniel,

Hi, Joerg.

> > test:~# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
> > kvm-clock
> > 
> > test:~# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
> > kvm-clock hpet acpi_pm
 
> Can you try to boot L1 and L2 guest with the 'no-kvmclock' kernel
> parameter? This disables the kvm-clock.

Question: what is L1 and L2? Because it sounds to cache levels, but I
suppose that against this background one will talk about another thing.

Beyond that, using no-kvmclock kernel parameter en 'test', now I have
the following thing:

test:~# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
hpet

test:~# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
hpet acpi_pm


But the same happens. The Debian installer hangs in the initial screen.

Thanks for your reply.

Regards,
Daniel
-- 
Daniel Bareiro - GNU/Linux registered user #188.598
Proudly running Debian GNU/Linux with uptime:
20:43:21 up 1 day,  5:28, 11 users,  load average: 0.21, 0.20, 0.12

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]