Booting Nested KVM

Booting Nested KVM

[Jacob Abraham Graff]

Hello,

I am a student at Columbia University. My research partner Andy (CC'd)
and I are working on booting a nested KVM virtual machine on x86.
However, we have run into a problem that we have not been able to fix.

The host machine, L0, has the 'kvm-intel.nested=1'  flag set in it's
grub config file, and the guest operating system, L1, does not
complain when attempting to start a VM using virt-install.

When we attempt to actually install, we run into a couple of issues.
When we run virt-install, using the command:

virt-install -r 1024 --accelerate -n Ubu \
-f /var/lib/libvirt/images/guest.img \
--cdrom /root/ubuntu-14.04.2-server-amd64.iso

We receive the error message

Could not start virtual network default

In order to get around this error message, we run pass the nonetworks flag:

virt-install -r 1024 --accelerate --nonetworks -n Ubu \
-f /var/lib/libvirt/images/guest.img \
--cdrom /root/ubuntu-14.04.2-server-amd64.iso

At this point, the install seems to succeed, and the nested VM starts
to boot. However, we cannot see any output at all - the console hangs,
and we cannot type anything into the console at this point. If we open
a new console, and run virt-viewer, we get what appears to be an
ubuntu login screen, but the text is unreadable (blurry and pixelated)
and again, we cannot type anything into this window. The virt-manager
gui seems to indicate that this guest does have activity, which makes
it seem strange that we are unable to communicate with it in any way.

Is there something we have missed? Do we have to specify IO in some special way?

Thanks,
Jacob Graff

Re: Booting Nested KVM

[Kashyap Chamarthy]

On Thu, Mar 24, 2016 at 11:21:32AM -0400, Jacob Abraham Graff wrote:
> Hello,
> 
> I am a student at Columbia University. My research partner Andy (CC'd)
> and I are working on booting a nested KVM virtual machine on x86.
> However, we have run into a problem that we have not been able to fix.
> 
> The host machine, L0, has the 'kvm-intel.nested=1'  flag set in it's
> grub config file, and the guest operating system, L1, does not
> complain when attempting to start a VM using virt-install.
> 
> When we attempt to actually install, we run into a couple of issues.
> When we run virt-install, using the command:
> 
> virt-install -r 1024 --accelerate -n Ubu \
> -f /var/lib/libvirt/images/guest.img \
> --cdrom /root/ubuntu-14.04.2-server-amd64.iso

> 
> We receive the error message
> 
> Could not start virtual network default

It seems like you don't have the default libvirt network active.  If it
were active, you'd see something like:

    $ virsh net-list
     Name                 State      Autostart     Persistent
    ----------------------------------------------------------
     default              active     yes           yes

If you don't, you can start the 'default' libvirt network manually:

    $ virsh net-start default
    $ virsh net-autostart default
    $ virsh net-list # Here the 'default' network should be active

Then, re-run your `virt-install` invocation.

Once you are inside L1, check if you have KVM character device exposed
in it, by doing `file /dev/kvm`.  If it is not present, then you should
expose the virtualization extensions to your L1 guest, by doing:

    $ virt-xml L1VM --edit --cpu host-passthrough,clearxml=yes

Reboot the guest, and check again if the KVM char device is present.

* * *

Speaking of installing VMs, for what it's worth, I prefer this  approach
for installing VMs quickly (I'm using Fedora-based distribution as an
example) using the 'virt-builder'[1] utility.

  (1) Build the VM template.
  
    # Install the tools
    $ sudo dnf install libguestfs-tools-c

    # Build the template
    $ virt-builder fedora-23 --size 40G

  (2) Import the image into libvirt

    $ virt-install --name f21vm --ram 2048 \
        --disk path=./fedora-23.img,format=raw \
        --nographics --import --os-variant fedora22

Then you can manipulate it using `virt-manager` (or from command-line,
if you prefer that way).


[1] http://libguestfs.org/virt-builder.1.html

Some related notes:

https://kashyapc.fedorapeople.org/virt/procedure-to-enable-nested-virt-on-intel-machines.txt

> In order to get around this error message, we run pass the nonetworks flag:
> 
> virt-install -r 1024 --accelerate --nonetworks -n Ubu \
> -f /var/lib/libvirt/images/guest.img \
> --cdrom /root/ubuntu-14.04.2-server-amd64.iso
> 
> At this point, the install seems to succeed, and the nested VM starts
> to boot. However, we cannot see any output at all - the console hangs,
> and we cannot type anything into the console at this point. If we open
> a new console, and run virt-viewer, we get what appears to be an
> ubuntu login screen, but the text is unreadable (blurry and pixelated)
> and again, we cannot type anything into this window. The virt-manager
> gui seems to indicate that this guest does have activity, which makes
> it seem strange that we are unable to communicate with it in any way.
> 
> Is there something we have missed? Do we have to specify IO in some
> special way?
> 

-- 
/kashyap

Re: Booting Nested KVM

[Paolo Bonzini]

On 25/03/2016 08:33, Kashyap Chamarthy wrote:
>> > We receive the error message
>> > 
>> > Could not start virtual network default
> It seems like you don't have the default libvirt network active.  If it
> were active, you'd see something like:
> 
>     $ virsh net-list
>      Name                 State      Autostart     Persistent
>     ----------------------------------------------------------
>      default              active     yes           yes

I think his problem is that he has 192.168.122.0/24 configured on both
host and guest for the default libvirt network.  Jacob, to fix this you
need to do "virsh net-edit default" (as root) and change the occurrences
of 122 to another number such as 123.

Paolo

Re: Booting Nested KVM

[Jacob Abraham Graff]

Thanks for the responses! Unfortunately, I'm still getting problems.
After using virsh net-edit, I attempted to virt-install, and still got
the same error. When I run 'virsh net-start default', I get this
error:

error: Failed to start network default
error: failed to add iptables rule to allow DHCP requests from 'virbr0'

None of the solutions I have found on Google have a fix for this. Any ideas?

On Fri, Mar 25, 2016 at 6:05 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>
> On 25/03/2016 08:33, Kashyap Chamarthy wrote:
>>> > We receive the error message
>>> >
>>> > Could not start virtual network default
>> It seems like you don't have the default libvirt network active.  If it
>> were active, you'd see something like:
>>
>>     $ virsh net-list
>>      Name                 State      Autostart     Persistent
>>     ----------------------------------------------------------
>>      default              active     yes           yes
>
> I think his problem is that he has 192.168.122.0/24 configured on both
> host and guest for the default libvirt network.  Jacob, to fix this you
> need to do "virsh net-edit default" (as root) and change the occurrences
> of 122 to another number such as 123.
>
> Paolo

Re: Booting Nested KVM

[Kashyap Chamarthy]

[Sorre for the late response -- I was away for the last week or so.]

On Sat, Mar 26, 2016 at 01:26:20PM -0400, Jacob Abraham Graff wrote:
> Thanks for the responses! Unfortunately, I'm still getting problems.
> After using virsh net-edit, I attempted to virt-install, and still got
> the same error. When I run 'virsh net-start default', I get this
> error:
> 
> error: Failed to start network default
> error: failed to add iptables rule to allow DHCP requests from 'virbr0'

I don't see any such errors as test on latest stable release
libvirt-1.3.2-3.fc23.x86_64.  Given that you say you've looked up few
things, I'd assume you've tried (a) ensuring `firewalld` is running; (b)
have 'iptables' package installed.
 
> None of the solutions I have found on Google have a fix for this. Any
> ideas?

[...]

> > I think his problem is that he has 192.168.122.0/24 configured on
> > both host and guest for the default libvirt network.  Jacob, to fix
> > this you need to do "virsh net-edit default" (as root) and change
> > the occurrences of 122 to another number such as 123.

If that's really the case, then upstream libvirt has fixed[1] the
default network conflicts.  I remember testing the very same case[2] of
avoiding default network conflicts for nested virtualization -- using
the said fix[1] avoids it.  It is available in version libvirt-1.2.8 or
above.


[1] http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=5f719596 --
    network: try to eliminate default network conflict during package
    install
[2] http://kashyapc.com/2014/09/16/libvirt-default-network-conflicts-not-anymore/

-- 
/kashyap