Re: [Xen-users] Nested Virtualisation question

Re: [Xen-users] Nested Virtualisation question

[Jeroen Groenewegen van der Weyden]

Hello,

I have A question about nested virtualisation, the xen user list 
suggested to post the question here.
I need nested virtualisation to run an application on out en 
environment, which uses kvm it-self.

I suppose I have to apply these patches:
- 
http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid:z33lihxnbdjghua5+state:results
-  or http://xenbits.xen.org/ext/cegger/nestedhvm/

Is it correct these patches are now in the xen-unstable branch?
Anything else to get me on my way.


My goal is

DOM0
  openSUSE 11.4 with xen 4.X

DOMU1
  SLES 11 SP1
   kvm guest HVM domu1a (propetary OS)
   kvm guest HVM domu1b (sles 10 sp3)
   kvm guest HVM domu1c (sles 10 sp3)
   kvm guest HVM domu1b (sles 10 sp3)

Op 3-4-2011 23:29, Todd Deshane schreef:
> On Sun, Apr 3, 2011 at 8:32 PM, Jeroen Groenewegen van der Weyden
> <groen692@grosc.com>  wrote:
>> I now am in the position to start testing your suggestion.
>>
>> An additional question, is it correct the patches you mention here are part
>> of the xen4.2-unstable branch?
>> are these patches for vt-x or for svm only? the developer seems to be from
>> amd. My machines are running on intel.
>>
> The latest patch series is here:
> http://xen.1045712.n5.nabble.com/PATCH-00-12-Nested-Virtualization-Overview-td3415672.html
>
> Not sure if it has been accepted yet or not. You can email the
> developer and/or devel list for confirmation.
>
> Thanks,
> Todd
>
>> mfg,
>> Jeroen
>>
>> Op 15-2-2011 16:21, Todd Deshane schreef:
>>> On Tue, Feb 15, 2011 at 6:47 AM, Jeroen Groenewegen van der Weyden
>>> <groen692@grosc.com>    wrote:
>>>> The why is a good question,
>>>> We use xen as out main virtualisation platform. But now there are
>>>> applications that use virtualisation it self. In my case I have an
>>>> application that uses kvm. So to put it on our xen server it needs to be
>>>> nested. I guess in the future more of these situation will appear.
>>>>
>>>> I read somethings about nested virtualisation and I wonder If somebody
>>>> has
>>>> any experience with it.
>>>>
>>> To run KVM inside of a guest you need virtualization support within
>>> the guest (running nested xen doesn't require the double nested
>>> (unless you want to do HVM within the nested of course))
>>>
>>> Patches can be found here:
>>>
>>> http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid:z33lihxnbdjghua5+state:results
>>>
>>> I believe KVM should also have support for nested vmx and nested svm.
>>>
>>> The use cases are fairly slim, so finding users/testers is often
>>> difficult.
>>>
>>> Please report any bugs you find to the appropriate devel lists and cc
>>> the appropriate nested virtualization maintainers .
>>>
>>> Hope that helps.
>>>
>>> Thanks,
>>> Todd
>>>
>>> _______________________________________________
>>> Xen-users mailing list
>>> Xen-users@lists.xensource.com
>>> http://lists.xensource.com/xen-users
>>>
>>
>
>

Nested Virtualisation question

[Jeroen Groenewegen van der Weyden]

Hello,

I have A question about nested virtualisation, the xen user list 
suggested to post the question here.
I need nested virtualisation to run an application on out en 
environment, which uses kvm it-self.

I suppose I have to apply these patches:
- 
http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid:z33lihxnbdjghua5+state:results
-  or http://xenbits.xen.org/ext/cegger/nestedhvm/

Is it correct these patches are now in the xen-unstable branch?
Anything else to get me on my way.


My goal is

DOM0
  openSUSE 11.4 with xen 4.X

DOMU1
  SLES 11 SP1
   kvm guest HVM domu1a (propetary OS)
   kvm guest HVM domu1b (sles 10 sp3)
   kvm guest HVM domu1c (sles 10 sp3)
   kvm guest HVM domu1b (sles 10 sp3)

> Op 3-4-2011 23:29, Todd Deshane schreef:
>> On Sun, Apr 3, 2011 at 8:32 PM, Jeroen Groenewegen van der Weyden
>> <groen692@grosc.com>  wrote:
>>> I now am in the position to start testing your suggestion.
>>>
>>> An additional question, is it correct the patches you mention here 
>>> are part
>>> of the xen4.2-unstable branch?
>>> are these patches for vt-x or for svm only? the developer seems to 
>>> be from
>>> amd. My machines are running on intel.
>>>
>> The latest patch series is here:
>> http://xen.1045712.n5.nabble.com/PATCH-00-12-Nested-Virtualization-Overview-td3415672.html 
>>
>>
>> Not sure if it has been accepted yet or not. You can email the
>> developer and/or devel list for confirmation.
>>
>> Thanks,
>> Todd
>>
>>> mfg,
>>> Jeroen
>>>
>>> Op 15-2-2011 16:21, Todd Deshane schreef:
>>>> On Tue, Feb 15, 2011 at 6:47 AM, Jeroen Groenewegen van der Weyden
>>>> <groen692@grosc.com>    wrote:
>>>>> The why is a good question,
>>>>> We use xen as out main virtualisation platform. But now there are
>>>>> applications that use virtualisation it self. In my case I have an
>>>>> application that uses kvm. So to put it on our xen server it needs 
>>>>> to be
>>>>> nested. I guess in the future more of these situation will appear.
>>>>>
>>>>> I read somethings about nested virtualisation and I wonder If 
>>>>> somebody
>>>>> has
>>>>> any experience with it.
>>>>>
>>>> To run KVM inside of a guest you need virtualization support within
>>>> the guest (running nested xen doesn't require the double nested
>>>> (unless you want to do HVM within the nested of course))
>>>>
>>>> Patches can be found here:
>>>>
>>>> http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid:z33lihxnbdjghua5+state:results 
>>>>
>>>>
>>>> I believe KVM should also have support for nested vmx and nested svm.
>>>>
>>>> The use cases are fairly slim, so finding users/testers is often
>>>> difficult.
>>>>
>>>> Please report any bugs you find to the appropriate devel lists and cc
>>>> the appropriate nested virtualization maintainers .
>>>>
>>>> Hope that helps.
>>>>
>>>> Thanks,
>>>> Todd
>>>>
>>>> _______________________________________________
>>>> Xen-users mailing list
>>>> Xen-users@lists.xensource.com
>>>> http://lists.xensource.com/xen-users
>>>>
>>>
>>
>>
>

Re: Re: [Xen-users] Nested Virtualisation question

[Tim Deegan]

At 07:47 +0100 on 04 Apr (1301903251), Jeroen Groenewegen van der Weyden wrote:
> Hello,
> 
> I have A question about nested virtualisation, the xen user list 
> suggested to post the question here.
> I need nested virtualisation to run an application on out en 
> environment, which uses kvm it-self.
> 
> I suppose I have to apply these patches:
> - 
> http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid:z33lihxnbdjghua5+state:results
> -  or http://xenbits.xen.org/ext/cegger/nestedhvm/
> 
> Is it correct these patches are now in the xen-unstable branch?

Not just yet.  There were two problems outstanding with patch #12 of the
latest version (a TLB flush race and a subsequent deadlock in the fix),
which are both fixed.  As soon as Christoph posts his fixes, the patch
series will be applied. 

Cheers,

Tim.

> Anything else to get me on my way.
> 
> 
> My goal is
> 
> DOM0
>   openSUSE 11.4 with xen 4.X
> 
> DOMU1
>   SLES 11 SP1
>    kvm guest HVM domu1a (propetary OS)
>    kvm guest HVM domu1b (sles 10 sp3)
>    kvm guest HVM domu1c (sles 10 sp3)
>    kvm guest HVM domu1b (sles 10 sp3)
> 
> Op 3-4-2011 23:29, Todd Deshane schreef:
> > On Sun, Apr 3, 2011 at 8:32 PM, Jeroen Groenewegen van der Weyden
> > <groen692@grosc.com>  wrote:
> >> I now am in the position to start testing your suggestion.
> >>
> >> An additional question, is it correct the patches you mention here are part
> >> of the xen4.2-unstable branch?
> >> are these patches for vt-x or for svm only? the developer seems to be from
> >> amd. My machines are running on intel.
> >>
> > The latest patch series is here:
> > http://xen.1045712.n5.nabble.com/PATCH-00-12-Nested-Virtualization-Overview-td3415672.html
> >
> > Not sure if it has been accepted yet or not. You can email the
> > developer and/or devel list for confirmation.
> >
> > Thanks,
> > Todd
> >
> >> mfg,
> >> Jeroen
> >>
> >> Op 15-2-2011 16:21, Todd Deshane schreef:
> >>> On Tue, Feb 15, 2011 at 6:47 AM, Jeroen Groenewegen van der Weyden
> >>> <groen692@grosc.com>    wrote:
> >>>> The why is a good question,
> >>>> We use xen as out main virtualisation platform. But now there are
> >>>> applications that use virtualisation it self. In my case I have an
> >>>> application that uses kvm. So to put it on our xen server it needs to be
> >>>> nested. I guess in the future more of these situation will appear.
> >>>>
> >>>> I read somethings about nested virtualisation and I wonder If somebody
> >>>> has
> >>>> any experience with it.
> >>>>
> >>> To run KVM inside of a guest you need virtualization support within
> >>> the guest (running nested xen doesn't require the double nested
> >>> (unless you want to do HVM within the nested of course))
> >>>
> >>> Patches can be found here:
> >>>
> >>> http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid:z33lihxnbdjghua5+state:results
> >>>
> >>> I believe KVM should also have support for nested vmx and nested svm.
> >>>
> >>> The use cases are fairly slim, so finding users/testers is often
> >>> difficult.
> >>>
> >>> Please report any bugs you find to the appropriate devel lists and cc
> >>> the appropriate nested virtualization maintainers .
> >>>
> >>> Hope that helps.
> >>>
> >>> Thanks,
> >>> Todd
> >>>
> >>> _______________________________________________
> >>> Xen-users mailing list
> >>> Xen-users@lists.xensource.com
> >>> http://lists.xensource.com/xen-users
> >>>
> >>
> >
> >
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Re: Nested Virtualisation question

[Jeroen Groenewegen van der Weyden]

Ok thank you, Anything I can do, to test or something (I do not want to 
sound like I'm in a hurry)

mfg,
jeroen

Op 4-4-2011 11:51, Tim Deegan schreef:
> At 07:47 +0100 on 04 Apr (1301903251), Jeroen Groenewegen van der Weyden wrote:
>> Hello,
>>
>> I have A question about nested virtualisation, the xen user list
>> suggested to post the question here.
>> I need nested virtualisation to run an application on out en
>> environment, which uses kvm it-self.
>>
>> I suppose I have to apply these patches:
>> -
>> http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid:z33lihxnbdjghua5+state:results
>> -  or http://xenbits.xen.org/ext/cegger/nestedhvm/
>>
>> Is it correct these patches are now in the xen-unstable branch?
> Not just yet.  There were two problems outstanding with patch #12 of the
> latest version (a TLB flush race and a subsequent deadlock in the fix),
> which are both fixed.  As soon as Christoph posts his fixes, the patch
> series will be applied.
>
> Cheers,
>
> Tim.
>
>> Anything else to get me on my way.
>>
>>
>> My goal is
>>
>> DOM0
>>    openSUSE 11.4 with xen 4.X
>>
>> DOMU1
>>    SLES 11 SP1
>>     kvm guest HVM domu1a (propetary OS)
>>     kvm guest HVM domu1b (sles 10 sp3)
>>     kvm guest HVM domu1c (sles 10 sp3)
>>     kvm guest HVM domu1b (sles 10 sp3)
>>
>> Op 3-4-2011 23:29, Todd Deshane schreef:
>>> On Sun, Apr 3, 2011 at 8:32 PM, Jeroen Groenewegen van der Weyden
>>> <groen692@grosc.com>   wrote:
>>>> I now am in the position to start testing your suggestion.
>>>>
>>>> An additional question, is it correct the patches you mention here are part
>>>> of the xen4.2-unstable branch?
>>>> are these patches for vt-x or for svm only? the developer seems to be from
>>>> amd. My machines are running on intel.
>>>>
>>> The latest patch series is here:
>>> http://xen.1045712.n5.nabble.com/PATCH-00-12-Nested-Virtualization-Overview-td3415672.html
>>>
>>> Not sure if it has been accepted yet or not. You can email the
>>> developer and/or devel list for confirmation.
>>>
>>> Thanks,
>>> Todd
>>>
>>>> mfg,
>>>> Jeroen
>>>>
>>>> Op 15-2-2011 16:21, Todd Deshane schreef:
>>>>> On Tue, Feb 15, 2011 at 6:47 AM, Jeroen Groenewegen van der Weyden
>>>>> <groen692@grosc.com>     wrote:
>>>>>> The why is a good question,
>>>>>> We use xen as out main virtualisation platform. But now there are
>>>>>> applications that use virtualisation it self. In my case I have an
>>>>>> application that uses kvm. So to put it on our xen server it needs to be
>>>>>> nested. I guess in the future more of these situation will appear.
>>>>>>
>>>>>> I read somethings about nested virtualisation and I wonder If somebody
>>>>>> has
>>>>>> any experience with it.
>>>>>>
>>>>> To run KVM inside of a guest you need virtualization support within
>>>>> the guest (running nested xen doesn't require the double nested
>>>>> (unless you want to do HVM within the nested of course))
>>>>>
>>>>> Patches can be found here:
>>>>>
>>>>> http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid:z33lihxnbdjghua5+state:results
>>>>>
>>>>> I believe KVM should also have support for nested vmx and nested svm.
>>>>>
>>>>> The use cases are fairly slim, so finding users/testers is often
>>>>> difficult.
>>>>>
>>>>> Please report any bugs you find to the appropriate devel lists and cc
>>>>> the appropriate nested virtualization maintainers .
>>>>>
>>>>> Hope that helps.
>>>>>
>>>>> Thanks,
>>>>> Todd
>>>>>
>>>>> _______________________________________________
>>>>> Xen-users mailing list
>>>>> Xen-users@lists.xensource.com
>>>>> http://lists.xensource.com/xen-users
>>>>>
>>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel

Re: Re: [Xen-users] Nested Virtualisation question

[Jeroen Groenewegen van der Weyden]

I couldn't help noticing some patches were added in the 
/staging/xen-unstable branch. So I gave it a go.

I know it is not called the unstable branch for nothing but, but I want 
to let you know that it the entire system crashes when any domain is 
created:
xl create someDomain -> results in immediate system crash

System:
- domO system openSUSE 11.4 x86_64. kernel 2.6.37.1-1.2-xen
- xen build from staging/xen-unstable (c/s 23172)
- proc intel q9450


+++++++++++++console output
hydra:/etc/xen/vm # xl create groscWindowsXpWorkstation
Parsing config file groscWindowsXpWorkstation
xc: info: VIRTUAL MEMORY ARRANGEMENT:
   Loader:        0000000000100000->00000000001795b0
   TOTAL:         0000000000000000->0000000017800000
   ENTRY ADDRESS: 0000000000100000
xc: info: PHYSICAL MEMORY ALLOCATION:
   4KB PAGES: 0x0000000000000200
   2MB PAGES: 0x00000000000000bb
   1GB PAGES: 0x0000000000000000
libxl: error: libxl_device.c:475:libxl__wait_for_device_model Device 
Model not ready
xl: fatal error: libxl_create.c:532, rc=-1: 
libxl__confirm_device_model_startup

+++++++++++++

Op 4-4-2011 11:51, Tim Deegan schreef:
> At 07:47 +0100 on 04 Apr (1301903251), Jeroen Groenewegen van der Weyden wrote:
>> Hello,
>>
>> I have A question about nested virtualisation, the xen user list
>> suggested to post the question here.
>> I need nested virtualisation to run an application on out en
>> environment, which uses kvm it-self.
>>
>> I suppose I have to apply these patches:
>> -
>> http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid:z33lihxnbdjghua5+state:results
>> -  or http://xenbits.xen.org/ext/cegger/nestedhvm/
>>
>> Is it correct these patches are now in the xen-unstable branch?
> Not just yet.  There were two problems outstanding with patch #12 of the
> latest version (a TLB flush race and a subsequent deadlock in the fix),
> which are both fixed.  As soon as Christoph posts his fixes, the patch
> series will be applied.
>
> Cheers,
>
> Tim.
>
>> Anything else to get me on my way.
>>
>>
>> My goal is
>>
>> DOM0
>>    openSUSE 11.4 with xen 4.X
>>
>> DOMU1
>>    SLES 11 SP1
>>     kvm guest HVM domu1a (propetary OS)
>>     kvm guest HVM domu1b (sles 10 sp3)
>>     kvm guest HVM domu1c (sles 10 sp3)
>>     kvm guest HVM domu1b (sles 10 sp3)
>>
>> Op 3-4-2011 23:29, Todd Deshane schreef:
>>> On Sun, Apr 3, 2011 at 8:32 PM, Jeroen Groenewegen van der Weyden
>>> <groen692@grosc.com>   wrote:
>>>> I now am in the position to start testing your suggestion.
>>>>
>>>> An additional question, is it correct the patches you mention here are part
>>>> of the xen4.2-unstable branch?
>>>> are these patches for vt-x or for svm only? the developer seems to be from
>>>> amd. My machines are running on intel.
>>>>
>>> The latest patch series is here:
>>> http://xen.1045712.n5.nabble.com/PATCH-00-12-Nested-Virtualization-Overview-td3415672.html
>>>
>>> Not sure if it has been accepted yet or not. You can email the
>>> developer and/or devel list for confirmation.
>>>
>>> Thanks,
>>> Todd
>>>
>>>> mfg,
>>>> Jeroen
>>>>
>>>> Op 15-2-2011 16:21, Todd Deshane schreef:
>>>>> On Tue, Feb 15, 2011 at 6:47 AM, Jeroen Groenewegen van der Weyden
>>>>> <groen692@grosc.com>     wrote:
>>>>>> The why is a good question,
>>>>>> We use xen as out main virtualisation platform. But now there are
>>>>>> applications that use virtualisation it self. In my case I have an
>>>>>> application that uses kvm. So to put it on our xen server it needs to be
>>>>>> nested. I guess in the future more of these situation will appear.
>>>>>>
>>>>>> I read somethings about nested virtualisation and I wonder If somebody
>>>>>> has
>>>>>> any experience with it.
>>>>>>
>>>>> To run KVM inside of a guest you need virtualization support within
>>>>> the guest (running nested xen doesn't require the double nested
>>>>> (unless you want to do HVM within the nested of course))
>>>>>
>>>>> Patches can be found here:
>>>>>
>>>>> http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid:z33lihxnbdjghua5+state:results
>>>>>
>>>>> I believe KVM should also have support for nested vmx and nested svm.
>>>>>
>>>>> The use cases are fairly slim, so finding users/testers is often
>>>>> difficult.
>>>>>
>>>>> Please report any bugs you find to the appropriate devel lists and cc
>>>>> the appropriate nested virtualization maintainers .
>>>>>
>>>>> Hope that helps.
>>>>>
>>>>> Thanks,
>>>>> Todd
>>>>>
>>>>> _______________________________________________
>>>>> Xen-users mailing list
>>>>> Xen-users@lists.xensource.com
>>>>> http://lists.xensource.com/xen-users
>>>>>
>>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel

Re: Re: [Xen-users] Nested Virtualisation question

[Keir Fraser]

On 07/04/2011 06:36, "Jeroen Groenewegen van der Weyden"
<groen692@grosc.com> wrote:

> I couldn't help noticing some patches were added in the
> /staging/xen-unstable branch. So I gave it a go.
> 
> I know it is not called the unstable branch for nothing but, but I want
> to let you know that it the entire system crashes when any domain is
> created:
> xl create someDomain -> results in immediate system crash

Probably the same as this, happening often in the automated tests:

Apr  7 03:09:18.695893 (XEN) hvm_vcpu_initialise: nestedhvm_vcpu_initialise
returned -95
Apr  7 03:09:18.703835 (XEN) Assertion 'd->arch.paging.hap.total_pages == 0'
failed at hap.c:699
[...]
Apr  7 03:09:18.915908 (XEN) Xen call trace:
Apr  7 03:09:18.915950 (XEN)    [<ffff82c48020429f>]
hap_teardown+0x1a8/0x229
Apr  7 03:09:18.923861 (XEN)    [<ffff82c4801ccb7c>]
paging_teardown+0x25/0x135
Apr  7 03:09:18.923922 (XEN)    [<ffff82c4801584d1>]
domain_relinquish_resources+0x6b/0x23c
Apr  7 03:09:18.935903 (XEN)    [<ffff82c480105287>] domain_kill+0x95/0xe9
Apr  7 03:09:18.943841 (XEN)    [<ffff82c48010324c>] do_domctl+0xd8e/0x1a02
Apr  7 03:09:18.943900 (XEN)    [<ffff82c480212f28>]
syscall_enter+0xc8/0x122
Apr  7 03:09:18.955837 (XEN)
Apr  7 03:09:18.955875 (XEN)
Apr  7 03:09:18.955909 (XEN) ****************************************
Apr  7 03:09:18.955961 (XEN) Panic on CPU 7:
Apr  7 03:09:18.956003 (XEN) Assertion 'd->arch.paging.hap.total_pages == 0'
failed at hap.c:699
Apr  7 03:09:18.963863 (XEN) ****************************************


> System:
> - domO system openSUSE 11.4 x86_64. kernel 2.6.37.1-1.2-xen
> - xen build from staging/xen-unstable (c/s 23172)
> - proc intel q9450
> 
> 
> +++++++++++++console output
> hydra:/etc/xen/vm # xl create groscWindowsXpWorkstation
> Parsing config file groscWindowsXpWorkstation
> xc: info: VIRTUAL MEMORY ARRANGEMENT:
>    Loader:        0000000000100000->00000000001795b0
>    TOTAL:         0000000000000000->0000000017800000
>    ENTRY ADDRESS: 0000000000100000
> xc: info: PHYSICAL MEMORY ALLOCATION:
>    4KB PAGES: 0x0000000000000200
>    2MB PAGES: 0x00000000000000bb
>    1GB PAGES: 0x0000000000000000
> libxl: error: libxl_device.c:475:libxl__wait_for_device_model Device
> Model not ready
> xl: fatal error: libxl_create.c:532, rc=-1:
> libxl__confirm_device_model_startup
> 
> +++++++++++++
> 
> Op 4-4-2011 11:51, Tim Deegan schreef:
>> At 07:47 +0100 on 04 Apr (1301903251), Jeroen Groenewegen van der Weyden
>> wrote:
>>> Hello,
>>> 
>>> I have A question about nested virtualisation, the xen user list
>>> suggested to post the question here.
>>> I need nested virtualisation to run an application on out en
>>> environment, which uses kvm it-self.
>>> 
>>> I suppose I have to apply these patches:
>>> -
>>> http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid
>>> :z33lihxnbdjghua5+state:results
>>> -  or http://xenbits.xen.org/ext/cegger/nestedhvm/
>>> 
>>> Is it correct these patches are now in the xen-unstable branch?
>> Not just yet.  There were two problems outstanding with patch #12 of the
>> latest version (a TLB flush race and a subsequent deadlock in the fix),
>> which are both fixed.  As soon as Christoph posts his fixes, the patch
>> series will be applied.
>> 
>> Cheers,
>> 
>> Tim.
>> 
>>> Anything else to get me on my way.
>>> 
>>> 
>>> My goal is
>>> 
>>> DOM0
>>>    openSUSE 11.4 with xen 4.X
>>> 
>>> DOMU1
>>>    SLES 11 SP1
>>>     kvm guest HVM domu1a (propetary OS)
>>>     kvm guest HVM domu1b (sles 10 sp3)
>>>     kvm guest HVM domu1c (sles 10 sp3)
>>>     kvm guest HVM domu1b (sles 10 sp3)
>>> 
>>> Op 3-4-2011 23:29, Todd Deshane schreef:
>>>> On Sun, Apr 3, 2011 at 8:32 PM, Jeroen Groenewegen van der Weyden
>>>> <groen692@grosc.com>   wrote:
>>>>> I now am in the position to start testing your suggestion.
>>>>> 
>>>>> An additional question, is it correct the patches you mention here are
>>>>> part
>>>>> of the xen4.2-unstable branch?
>>>>> are these patches for vt-x or for svm only? the developer seems to be from
>>>>> amd. My machines are running on intel.
>>>>> 
>>>> The latest patch series is here:
>>>> http://xen.1045712.n5.nabble.com/PATCH-00-12-Nested-Virtualization-Overview
>>>> -td3415672.html
>>>> 
>>>> Not sure if it has been accepted yet or not. You can email the
>>>> developer and/or devel list for confirmation.
>>>> 
>>>> Thanks,
>>>> Todd
>>>> 
>>>>> mfg,
>>>>> Jeroen
>>>>> 
>>>>> Op 15-2-2011 16:21, Todd Deshane schreef:
>>>>>> On Tue, Feb 15, 2011 at 6:47 AM, Jeroen Groenewegen van der Weyden
>>>>>> <groen692@grosc.com>     wrote:
>>>>>>> The why is a good question,
>>>>>>> We use xen as out main virtualisation platform. But now there are
>>>>>>> applications that use virtualisation it self. In my case I have an
>>>>>>> application that uses kvm. So to put it on our xen server it needs to be
>>>>>>> nested. I guess in the future more of these situation will appear.
>>>>>>> 
>>>>>>> I read somethings about nested virtualisation and I wonder If somebody
>>>>>>> has
>>>>>>> any experience with it.
>>>>>>> 
>>>>>> To run KVM inside of a guest you need virtualization support within
>>>>>> the guest (running nested xen doesn't require the double nested
>>>>>> (unless you want to do HVM within the nested of course))
>>>>>> 
>>>>>> Patches can be found here:
>>>>>> 
>>>>>> http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+
>>>>>> mid:z33lihxnbdjghua5+state:results
>>>>>> 
>>>>>> I believe KVM should also have support for nested vmx and nested svm.
>>>>>> 
>>>>>> The use cases are fairly slim, so finding users/testers is often
>>>>>> difficult.
>>>>>> 
>>>>>> Please report any bugs you find to the appropriate devel lists and cc
>>>>>> the appropriate nested virtualization maintainers .
>>>>>> 
>>>>>> Hope that helps.
>>>>>> 
>>>>>> Thanks,
>>>>>> Todd
>>>>>> 
>>>>>> _______________________________________________
>>>>>> Xen-users mailing list
>>>>>> Xen-users@lists.xensource.com
>>>>>> http://lists.xensource.com/xen-users
>>>>>> 
>>>> 
>>> 
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Re: [Xen-users] Nested Virtualisation question

[Keir Fraser]

On 07/04/2011 08:09, "Keir Fraser" <keir.xen@gmail.com> wrote:

> On 07/04/2011 06:36, "Jeroen Groenewegen van der Weyden"
> <groen692@grosc.com> wrote:
> 
>> I couldn't help noticing some patches were added in the
>> /staging/xen-unstable branch. So I gave it a go.
>> 
>> I know it is not called the unstable branch for nothing but, but I want
>> to let you know that it the entire system crashes when any domain is
>> created:
>> xl create someDomain -> results in immediate system crash
> 
> Probably the same as this, happening often in the automated tests:

Likely reason: nestedhvm_vcpu_initialise() being called unconditionally in
hvm_vcpu_initialise() even though nestedhvm has not been configured for this
domain. This then fails because there is no nestedhvm handler for Intel yet.
And beyond that it's most likely an issue of a broken failure path, or a
problem in the HAP code with cleaning up a half-created guest. So it's
probably two bugs. :-)

 -- Keir

> Apr  7 03:09:18.695893 (XEN) hvm_vcpu_initialise: nestedhvm_vcpu_initialise
> returned -95
> Apr  7 03:09:18.703835 (XEN) Assertion 'd->arch.paging.hap.total_pages == 0'
> failed at hap.c:699
> [...]
> Apr  7 03:09:18.915908 (XEN) Xen call trace:
> Apr  7 03:09:18.915950 (XEN)    [<ffff82c48020429f>]
> hap_teardown+0x1a8/0x229
> Apr  7 03:09:18.923861 (XEN)    [<ffff82c4801ccb7c>]
> paging_teardown+0x25/0x135
> Apr  7 03:09:18.923922 (XEN)    [<ffff82c4801584d1>]
> domain_relinquish_resources+0x6b/0x23c
> Apr  7 03:09:18.935903 (XEN)    [<ffff82c480105287>] domain_kill+0x95/0xe9
> Apr  7 03:09:18.943841 (XEN)    [<ffff82c48010324c>] do_domctl+0xd8e/0x1a02
> Apr  7 03:09:18.943900 (XEN)    [<ffff82c480212f28>]
> syscall_enter+0xc8/0x122
> Apr  7 03:09:18.955837 (XEN)
> Apr  7 03:09:18.955875 (XEN)
> Apr  7 03:09:18.955909 (XEN) ****************************************
> Apr  7 03:09:18.955961 (XEN) Panic on CPU 7:
> Apr  7 03:09:18.956003 (XEN) Assertion 'd->arch.paging.hap.total_pages == 0'
> failed at hap.c:699
> Apr  7 03:09:18.963863 (XEN) ****************************************
> 
> 
>> System:
>> - domO system openSUSE 11.4 x86_64. kernel 2.6.37.1-1.2-xen
>> - xen build from staging/xen-unstable (c/s 23172)
>> - proc intel q9450
>> 
>> 
>> +++++++++++++console output
>> hydra:/etc/xen/vm # xl create groscWindowsXpWorkstation
>> Parsing config file groscWindowsXpWorkstation
>> xc: info: VIRTUAL MEMORY ARRANGEMENT:
>>    Loader:        0000000000100000->00000000001795b0
>>    TOTAL:         0000000000000000->0000000017800000
>>    ENTRY ADDRESS: 0000000000100000
>> xc: info: PHYSICAL MEMORY ALLOCATION:
>>    4KB PAGES: 0x0000000000000200
>>    2MB PAGES: 0x00000000000000bb
>>    1GB PAGES: 0x0000000000000000
>> libxl: error: libxl_device.c:475:libxl__wait_for_device_model Device
>> Model not ready
>> xl: fatal error: libxl_create.c:532, rc=-1:
>> libxl__confirm_device_model_startup
>> 
>> +++++++++++++
>> 
>> Op 4-4-2011 11:51, Tim Deegan schreef:
>>> At 07:47 +0100 on 04 Apr (1301903251), Jeroen Groenewegen van der Weyden
>>> wrote:
>>>> Hello,
>>>> 
>>>> I have A question about nested virtualisation, the xen user list
>>>> suggested to post the question here.
>>>> I need nested virtualisation to run an application on out en
>>>> environment, which uses kvm it-self.
>>>> 
>>>> I suppose I have to apply these patches:
>>>> -
>>>> 
http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mi>>>>
d
>>>> :z33lihxnbdjghua5+state:results
>>>> -  or http://xenbits.xen.org/ext/cegger/nestedhvm/
>>>> 
>>>> Is it correct these patches are now in the xen-unstable branch?
>>> Not just yet.  There were two problems outstanding with patch #12 of the
>>> latest version (a TLB flush race and a subsequent deadlock in the fix),
>>> which are both fixed.  As soon as Christoph posts his fixes, the patch
>>> series will be applied.
>>> 
>>> Cheers,
>>> 
>>> Tim.
>>> 
>>>> Anything else to get me on my way.
>>>> 
>>>> 
>>>> My goal is
>>>> 
>>>> DOM0
>>>>    openSUSE 11.4 with xen 4.X
>>>> 
>>>> DOMU1
>>>>    SLES 11 SP1
>>>>     kvm guest HVM domu1a (propetary OS)
>>>>     kvm guest HVM domu1b (sles 10 sp3)
>>>>     kvm guest HVM domu1c (sles 10 sp3)
>>>>     kvm guest HVM domu1b (sles 10 sp3)
>>>> 
>>>> Op 3-4-2011 23:29, Todd Deshane schreef:
>>>>> On Sun, Apr 3, 2011 at 8:32 PM, Jeroen Groenewegen van der Weyden
>>>>> <groen692@grosc.com>   wrote:
>>>>>> I now am in the position to start testing your suggestion.
>>>>>> 
>>>>>> An additional question, is it correct the patches you mention here are
>>>>>> part
>>>>>> of the xen4.2-unstable branch?
>>>>>> are these patches for vt-x or for svm only? the developer seems to be
>>>>>> from
>>>>>> amd. My machines are running on intel.
>>>>>> 
>>>>> The latest patch series is here:
>>>>> 
http://xen.1045712.n5.nabble.com/PATCH-00-12-Nested-Virtualization-Overvie>>>>>
w
>>>>> -td3415672.html
>>>>> 
>>>>> Not sure if it has been accepted yet or not. You can email the
>>>>> developer and/or devel list for confirmation.
>>>>> 
>>>>> Thanks,
>>>>> Todd
>>>>> 
>>>>>> mfg,
>>>>>> Jeroen
>>>>>> 
>>>>>> Op 15-2-2011 16:21, Todd Deshane schreef:
>>>>>>> On Tue, Feb 15, 2011 at 6:47 AM, Jeroen Groenewegen van der Weyden
>>>>>>> <groen692@grosc.com>     wrote:
>>>>>>>> The why is a good question,
>>>>>>>> We use xen as out main virtualisation platform. But now there are
>>>>>>>> applications that use virtualisation it self. In my case I have an
>>>>>>>> application that uses kvm. So to put it on our xen server it needs to
>>>>>>>> be
>>>>>>>> nested. I guess in the future more of these situation will appear.
>>>>>>>> 
>>>>>>>> I read somethings about nested virtualisation and I wonder If somebody
>>>>>>>> has
>>>>>>>> any experience with it.
>>>>>>>> 
>>>>>>> To run KVM inside of a guest you need virtualization support within
>>>>>>> the guest (running nested xen doesn't require the double nested
>>>>>>> (unless you want to do HVM within the nested of course))
>>>>>>> 
>>>>>>> Patches can be found here:
>>>>>>> 
>>>>>>> 
http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1>>>>>>>
+
>>>>>>> mid:z33lihxnbdjghua5+state:results
>>>>>>> 
>>>>>>> I believe KVM should also have support for nested vmx and nested svm.
>>>>>>> 
>>>>>>> The use cases are fairly slim, so finding users/testers is often
>>>>>>> difficult.
>>>>>>> 
>>>>>>> Please report any bugs you find to the appropriate devel lists and cc
>>>>>>> the appropriate nested virtualization maintainers .
>>>>>>> 
>>>>>>> Hope that helps.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Todd
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> Xen-users mailing list
>>>>>>> Xen-users@lists.xensource.com
>>>>>>> http://lists.xensource.com/xen-users
>>>>>>> 
>>>>> 
>>>> 
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xensource.com
>>>> http://lists.xensource.com/xen-devel
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
> 
> 

Re: Re: [Xen-users] Nested Virtualisation question

[Tim Deegan]

At 08:14 +0100 on 07 Apr (1302164093), Keir Fraser wrote:
> On 07/04/2011 08:09, "Keir Fraser" <keir.xen@gmail.com> wrote:
> > Probably the same as this, happening often in the automated tests:
> 
> Likely reason: nestedhvm_vcpu_initialise() being called unconditionally in
> hvm_vcpu_initialise() even though nestedhvm has not been configured for this
> domain. This then fails because there is no nestedhvm handler for Intel yet.

This bug should be fixed by 23175:e5a750d1bf9b, arriving at a staging
tree near you at any moment. 

> And beyond that it's most likely an issue of a broken failure path, or a
> problem in the HAP code with cleaning up a half-created guest.

This one I'll look into now.

Tim.


-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Re: Re: [Xen-users] Nested Virtualisation question

[Jeroen Groenewegen van der Weyden]

The result with c/s23197

Regular domain now start, en stop normaly, (xl seems to have a problem 
though, after domain closes a domain null remains in de xl list/ xmdoe 
not have this.)
for nestedVM I added the "nestedhvm=1" in the config file of the domain. 
but the domain does not start because of "invalid argument"

+++++++++ beging console
hydra:/etc/xen/vm # xm create groscNestedVM
Using config file "./groscNestedVM".
Error: (22, 'Invalid argument')
+++++++++ end console

+++++++++ begin xend.log
hydra:/etc/xen/vm # tail /var/log/xen/xend.log
     dominfo = XendDomainInfo.create(config)
   File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py", 
line 106, in create
     vm.start()
   File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py", 
line 473, in start
     XendTask.log_progress(0, 30, self._constructDomain)
   File "/usr/lib64/python2.7/site-packages/xen/xend/XendTask.py", line 
209, in log_progress
     retval = func(*args, **kwds)
   File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py", 
line 2598, in _constructDomain
     xc.hvm_set_param(self.domid, HVM_PARAM_NESTEDHVM, long(nestedhvm))
Error: (22, 'Invalid argument')
+++++++++++ end xend.log


Op 7-4-2011 12:29, Tim Deegan schreef:
> At 08:14 +0100 on 07 Apr (1302164093), Keir Fraser wrote:
>> On 07/04/2011 08:09, "Keir Fraser"<keir.xen@gmail.com>  wrote:
>>> Probably the same as this, happening often in the automated tests:
>> Likely reason: nestedhvm_vcpu_initialise() being called unconditionally in
>> hvm_vcpu_initialise() even though nestedhvm has not been configured for this
>> domain. This then fails because there is no nestedhvm handler for Intel yet.
> This bug should be fixed by 23175:e5a750d1bf9b, arriving at a staging
> tree near you at any moment.
>
>> And beyond that it's most likely an issue of a broken failure path, or a
>> problem in the HAP code with cleaning up a half-created guest.
> This one I'll look into now.
>
> Tim.
>
>

Re: Re: [Xen-users] Nested Virtualisation question

[Keir Fraser]

On 09/04/2011 06:07, "Jeroen Groenewegen van der Weyden"
<groen692@grosc.com> wrote:

> The result with c/s23197
> 
> Regular domain now start, en stop normaly, (xl seems to have a problem
> though, after domain closes a domain null remains in de xl list/ xmdoe
> not have this.)
> for nestedVM I added the "nestedhvm=1" in the config file of the domain.
> but the domain does not start because of "invalid argument"

NestedHVM currently requires you to be using an AMD processor with hardware
nested paging (known as HAP by Xen). Most likely you need to add hap=1 to
your config file (and hope your processor supports it).

 -- Keir

> +++++++++ beging console
> hydra:/etc/xen/vm # xm create groscNestedVM
> Using config file "./groscNestedVM".
> Error: (22, 'Invalid argument')
> +++++++++ end console
> 
> +++++++++ begin xend.log
> hydra:/etc/xen/vm # tail /var/log/xen/xend.log
>      dominfo = XendDomainInfo.create(config)
>    File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py",
> line 106, in create
>      vm.start()
>    File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py",
> line 473, in start
>      XendTask.log_progress(0, 30, self._constructDomain)
>    File "/usr/lib64/python2.7/site-packages/xen/xend/XendTask.py", line
> 209, in log_progress
>      retval = func(*args, **kwds)
>    File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py",
> line 2598, in _constructDomain
>      xc.hvm_set_param(self.domid, HVM_PARAM_NESTEDHVM, long(nestedhvm))
> Error: (22, 'Invalid argument')
> +++++++++++ end xend.log
> 
> 
> Op 7-4-2011 12:29, Tim Deegan schreef:
>> At 08:14 +0100 on 07 Apr (1302164093), Keir Fraser wrote:
>>> On 07/04/2011 08:09, "Keir Fraser"<keir.xen@gmail.com>  wrote:
>>>> Probably the same as this, happening often in the automated tests:
>>> Likely reason: nestedhvm_vcpu_initialise() being called unconditionally in
>>> hvm_vcpu_initialise() even though nestedhvm has not been configured for this
>>> domain. This then fails because there is no nestedhvm handler for Intel yet.
>> This bug should be fixed by 23175:e5a750d1bf9b, arriving at a staging
>> tree near you at any moment.
>> 
>>> And beyond that it's most likely an issue of a broken failure path, or a
>>> problem in the HAP code with cleaning up a half-created guest.
>> This one I'll look into now.
>> 
>> Tim.
>> 
>> 
> 

Re: Re: [Xen-users] Nested Virtualisation question

[Jeroen Groenewegen van der Weyden]

The HAP is default set to one, I check this also with a dry run.
Please confirm all this support just amd, because I only have Intel 
overhere.

mfg,
Jeroen

Op 9-4-2011 8:22, Keir Fraser schreef:
> On 09/04/2011 06:07, "Jeroen Groenewegen van der Weyden"
> <groen692@grosc.com>  wrote:
>
>> The result with c/s23197
>>
>> Regular domain now start, en stop normaly, (xl seems to have a problem
>> though, after domain closes a domain null remains in de xl list/ xmdoe
>> not have this.)
>> for nestedVM I added the "nestedhvm=1" in the config file of the domain.
>> but the domain does not start because of "invalid argument"
> NestedHVM currently requires you to be using an AMD processor with hardware
> nested paging (known as HAP by Xen). Most likely you need to add hap=1 to
> your config file (and hope your processor supports it).
>
>   -- Keir
>
>> +++++++++ beging console
>> hydra:/etc/xen/vm # xm create groscNestedVM
>> Using config file "./groscNestedVM".
>> Error: (22, 'Invalid argument')
>> +++++++++ end console
>>
>> +++++++++ begin xend.log
>> hydra:/etc/xen/vm # tail /var/log/xen/xend.log
>>       dominfo = XendDomainInfo.create(config)
>>     File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py",
>> line 106, in create
>>       vm.start()
>>     File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py",
>> line 473, in start
>>       XendTask.log_progress(0, 30, self._constructDomain)
>>     File "/usr/lib64/python2.7/site-packages/xen/xend/XendTask.py", line
>> 209, in log_progress
>>       retval = func(*args, **kwds)
>>     File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py",
>> line 2598, in _constructDomain
>>       xc.hvm_set_param(self.domid, HVM_PARAM_NESTEDHVM, long(nestedhvm))
>> Error: (22, 'Invalid argument')
>> +++++++++++ end xend.log
>>
>>
>> Op 7-4-2011 12:29, Tim Deegan schreef:
>>> At 08:14 +0100 on 07 Apr (1302164093), Keir Fraser wrote:
>>>> On 07/04/2011 08:09, "Keir Fraser"<keir.xen@gmail.com>   wrote:
>>>>> Probably the same as this, happening often in the automated tests:
>>>> Likely reason: nestedhvm_vcpu_initialise() being called unconditionally in
>>>> hvm_vcpu_initialise() even though nestedhvm has not been configured for this
>>>> domain. This then fails because there is no nestedhvm handler for Intel yet.
>>> This bug should be fixed by 23175:e5a750d1bf9b, arriving at a staging
>>> tree near you at any moment.
>>>
>>>> And beyond that it's most likely an issue of a broken failure path, or a
>>>> problem in the HAP code with cleaning up a half-created guest.
>>> This one I'll look into now.
>>>
>>> Tim.
>>>
>>>
>
>

Re: Re: [Xen-users] Nested Virtualisation question

[Keir Fraser]

On 09/04/2011 11:18, "Jeroen Groenewegen van der Weyden"
<groen692@grosc.com> wrote:

> The HAP is default set to one, I check this also with a dry run.
> Please confirm all this support just amd, because I only have Intel
> overhere.

Well, quite apart from being AMD only right now (Intel patches have been
posted in the past, but they're not in tree yet, and I very much doubt old
patches would directly apply to our current development tree), I'm also
pretty sure that your machine doesn't support Intel EPT -- if the processor
can't support HAP then we silently discard that configuration request when
creating a domain. That's why you get error 22 (EINVAL). But even if you did
have Intel EPT, because Intel's nested virt patches aren't in tree yet you'd
just get error 95 (EOPNOTSUPP) instead!

By the way you can tell if your system supports EPT by grepping in 'xm
dmesg' output (i.e., Xen's boot log) for a line stating "Extended Page
Tables (EPT)".

 -- Keir

> mfg,
> Jeroen
> 
> Op 9-4-2011 8:22, Keir Fraser schreef:
>> On 09/04/2011 06:07, "Jeroen Groenewegen van der Weyden"
>> <groen692@grosc.com>  wrote:
>> 
>>> The result with c/s23197
>>> 
>>> Regular domain now start, en stop normaly, (xl seems to have a problem
>>> though, after domain closes a domain null remains in de xl list/ xmdoe
>>> not have this.)
>>> for nestedVM I added the "nestedhvm=1" in the config file of the domain.
>>> but the domain does not start because of "invalid argument"
>> NestedHVM currently requires you to be using an AMD processor with hardware
>> nested paging (known as HAP by Xen). Most likely you need to add hap=1 to
>> your config file (and hope your processor supports it).
>> 
>>   -- Keir
>> 
>>> +++++++++ beging console
>>> hydra:/etc/xen/vm # xm create groscNestedVM
>>> Using config file "./groscNestedVM".
>>> Error: (22, 'Invalid argument')
>>> +++++++++ end console
>>> 
>>> +++++++++ begin xend.log
>>> hydra:/etc/xen/vm # tail /var/log/xen/xend.log
>>>       dominfo = XendDomainInfo.create(config)
>>>     File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py",
>>> line 106, in create
>>>       vm.start()
>>>     File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py",
>>> line 473, in start
>>>       XendTask.log_progress(0, 30, self._constructDomain)
>>>     File "/usr/lib64/python2.7/site-packages/xen/xend/XendTask.py", line
>>> 209, in log_progress
>>>       retval = func(*args, **kwds)
>>>     File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py",
>>> line 2598, in _constructDomain
>>>       xc.hvm_set_param(self.domid, HVM_PARAM_NESTEDHVM, long(nestedhvm))
>>> Error: (22, 'Invalid argument')
>>> +++++++++++ end xend.log
>>> 
>>> 
>>> Op 7-4-2011 12:29, Tim Deegan schreef:
>>>> At 08:14 +0100 on 07 Apr (1302164093), Keir Fraser wrote:
>>>>> On 07/04/2011 08:09, "Keir Fraser"<keir.xen@gmail.com>   wrote:
>>>>>> Probably the same as this, happening often in the automated tests:
>>>>> Likely reason: nestedhvm_vcpu_initialise() being called unconditionally in
>>>>> hvm_vcpu_initialise() even though nestedhvm has not been configured for
>>>>> this
>>>>> domain. This then fails because there is no nestedhvm handler for Intel
>>>>> yet.
>>>> This bug should be fixed by 23175:e5a750d1bf9b, arriving at a staging
>>>> tree near you at any moment.
>>>> 
>>>>> And beyond that it's most likely an issue of a broken failure path, or a
>>>>> problem in the HAP code with cleaning up a half-created guest.
>>>> This one I'll look into now.
>>>> 
>>>> Tim.
>>>> 
>>>> 
>> 
>> 
> 

Re: Re: [Xen-users] Nested Virtualisation question

[Jeroen Groenewegen van der Weyden]

I'm now testing on a intel q9450, which does not seem to have this. I 
will move my testing to a intel e5645 which seems to have this capability.

Question then is, when will the intel support be implemented? is there 
commitment from some developer? is there a roadmap planned. in short 
what is the story here.

mfg,
Jeroen

Op 9-4-2011 13:15, Keir Fraser schreef:
> On 09/04/2011 11:18, "Jeroen Groenewegen van der Weyden"
> <groen692@grosc.com>  wrote:
>
>> The HAP is default set to one, I check this also with a dry run.
>> Please confirm all this support just amd, because I only have Intel
>> overhere.
> Well, quite apart from being AMD only right now (Intel patches have been
> posted in the past, but they're not in tree yet, and I very much doubt old
> patches would directly apply to our current development tree), I'm also
> pretty sure that your machine doesn't support Intel EPT -- if the processor
> can't support HAP then we silently discard that configuration request when
> creating a domain. That's why you get error 22 (EINVAL). But even if you did
> have Intel EPT, because Intel's nested virt patches aren't in tree yet you'd
> just get error 95 (EOPNOTSUPP) instead!
>
> By the way you can tell if your system supports EPT by grepping in 'xm
> dmesg' output (i.e., Xen's boot log) for a line stating "Extended Page
> Tables (EPT)".
>
>   -- Keir
>
>> mfg,
>> Jeroen
>>
>> Op 9-4-2011 8:22, Keir Fraser schreef:
>>> On 09/04/2011 06:07, "Jeroen Groenewegen van der Weyden"
>>> <groen692@grosc.com>   wrote:
>>>
>>>> The result with c/s23197
>>>>
>>>> Regular domain now start, en stop normaly, (xl seems to have a problem
>>>> though, after domain closes a domain null remains in de xl list/ xmdoe
>>>> not have this.)
>>>> for nestedVM I added the "nestedhvm=1" in the config file of the domain.
>>>> but the domain does not start because of "invalid argument"
>>> NestedHVM currently requires you to be using an AMD processor with hardware
>>> nested paging (known as HAP by Xen). Most likely you need to add hap=1 to
>>> your config file (and hope your processor supports it).
>>>
>>>    -- Keir
>>>
>>>> +++++++++ beging console
>>>> hydra:/etc/xen/vm # xm create groscNestedVM
>>>> Using config file "./groscNestedVM".
>>>> Error: (22, 'Invalid argument')
>>>> +++++++++ end console
>>>>
>>>> +++++++++ begin xend.log
>>>> hydra:/etc/xen/vm # tail /var/log/xen/xend.log
>>>>        dominfo = XendDomainInfo.create(config)
>>>>      File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py",
>>>> line 106, in create
>>>>        vm.start()
>>>>      File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py",
>>>> line 473, in start
>>>>        XendTask.log_progress(0, 30, self._constructDomain)
>>>>      File "/usr/lib64/python2.7/site-packages/xen/xend/XendTask.py", line
>>>> 209, in log_progress
>>>>        retval = func(*args, **kwds)
>>>>      File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py",
>>>> line 2598, in _constructDomain
>>>>        xc.hvm_set_param(self.domid, HVM_PARAM_NESTEDHVM, long(nestedhvm))
>>>> Error: (22, 'Invalid argument')
>>>> +++++++++++ end xend.log
>>>>
>>>>
>>>> Op 7-4-2011 12:29, Tim Deegan schreef:
>>>>> At 08:14 +0100 on 07 Apr (1302164093), Keir Fraser wrote:
>>>>>> On 07/04/2011 08:09, "Keir Fraser"<keir.xen@gmail.com>    wrote:
>>>>>>> Probably the same as this, happening often in the automated tests:
>>>>>> Likely reason: nestedhvm_vcpu_initialise() being called unconditionally in
>>>>>> hvm_vcpu_initialise() even though nestedhvm has not been configured for
>>>>>> this
>>>>>> domain. This then fails because there is no nestedhvm handler for Intel
>>>>>> yet.
>>>>> This bug should be fixed by 23175:e5a750d1bf9b, arriving at a staging
>>>>> tree near you at any moment.
>>>>>
>>>>>> And beyond that it's most likely an issue of a broken failure path, or a
>>>>>> problem in the HAP code with cleaning up a half-created guest.
>>>>> This one I'll look into now.
>>>>>
>>>>> Tim.
>>>>>
>>>>>
>>>
>
>

Re: Re: [Xen-users] Nested Virtualisation question

[Keir Fraser]

On 09/04/2011 14:32, "Jeroen Groenewegen van der Weyden"
<groen692@grosc.com> wrote:

> I'm now testing on a intel q9450, which does not seem to have this. I
> will move my testing to a intel e5645 which seems to have this capability.
> 
> Question then is, when will the intel support be implemented? is there
> commitment from some developer? is there a roadmap planned. in short
> what is the story here.

It has been implemented already, for example see a previous patchset
submission: 
http://xen.1045712.n5.nabble.com/PATCH-00-16-Nested-virtualization-for-VMX-t
d2808075.html

It just needs updating to development tip, and in particular sync'ing with
the nestedhvm common bits that got applied last week. Someone from Intel
would have to speak for how long it will take to schedule and do that work,
but I would guess we're talking weeks, rather than days or months.

 -- Keir

> mfg,
> Jeroen
> 
> Op 9-4-2011 13:15, Keir Fraser schreef:
>> On 09/04/2011 11:18, "Jeroen Groenewegen van der Weyden"
>> <groen692@grosc.com>  wrote:
>> 
>>> The HAP is default set to one, I check this also with a dry run.
>>> Please confirm all this support just amd, because I only have Intel
>>> overhere.
>> Well, quite apart from being AMD only right now (Intel patches have been
>> posted in the past, but they're not in tree yet, and I very much doubt old
>> patches would directly apply to our current development tree), I'm also
>> pretty sure that your machine doesn't support Intel EPT -- if the processor
>> can't support HAP then we silently discard that configuration request when
>> creating a domain. That's why you get error 22 (EINVAL). But even if you did
>> have Intel EPT, because Intel's nested virt patches aren't in tree yet you'd
>> just get error 95 (EOPNOTSUPP) instead!
>> 
>> By the way you can tell if your system supports EPT by grepping in 'xm
>> dmesg' output (i.e., Xen's boot log) for a line stating "Extended Page
>> Tables (EPT)".
>> 
>>   -- Keir
>> 
>>> mfg,
>>> Jeroen
>>> 
>>> Op 9-4-2011 8:22, Keir Fraser schreef:
>>>> On 09/04/2011 06:07, "Jeroen Groenewegen van der Weyden"
>>>> <groen692@grosc.com>   wrote:
>>>> 
>>>>> The result with c/s23197
>>>>> 
>>>>> Regular domain now start, en stop normaly, (xl seems to have a problem
>>>>> though, after domain closes a domain null remains in de xl list/ xmdoe
>>>>> not have this.)
>>>>> for nestedVM I added the "nestedhvm=1" in the config file of the domain.
>>>>> but the domain does not start because of "invalid argument"
>>>> NestedHVM currently requires you to be using an AMD processor with hardware
>>>> nested paging (known as HAP by Xen). Most likely you need to add hap=1 to
>>>> your config file (and hope your processor supports it).
>>>> 
>>>>    -- Keir
>>>> 
>>>>> +++++++++ beging console
>>>>> hydra:/etc/xen/vm # xm create groscNestedVM
>>>>> Using config file "./groscNestedVM".
>>>>> Error: (22, 'Invalid argument')
>>>>> +++++++++ end console
>>>>> 
>>>>> +++++++++ begin xend.log
>>>>> hydra:/etc/xen/vm # tail /var/log/xen/xend.log
>>>>>        dominfo = XendDomainInfo.create(config)
>>>>>      File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py",
>>>>> line 106, in create
>>>>>        vm.start()
>>>>>      File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py",
>>>>> line 473, in start
>>>>>        XendTask.log_progress(0, 30, self._constructDomain)
>>>>>      File "/usr/lib64/python2.7/site-packages/xen/xend/XendTask.py", line
>>>>> 209, in log_progress
>>>>>        retval = func(*args, **kwds)
>>>>>      File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py",
>>>>> line 2598, in _constructDomain
>>>>>        xc.hvm_set_param(self.domid, HVM_PARAM_NESTEDHVM, long(nestedhvm))
>>>>> Error: (22, 'Invalid argument')
>>>>> +++++++++++ end xend.log
>>>>> 
>>>>> 
>>>>> Op 7-4-2011 12:29, Tim Deegan schreef:
>>>>>> At 08:14 +0100 on 07 Apr (1302164093), Keir Fraser wrote:
>>>>>>> On 07/04/2011 08:09, "Keir Fraser"<keir.xen@gmail.com>    wrote:
>>>>>>>> Probably the same as this, happening often in the automated tests:
>>>>>>> Likely reason: nestedhvm_vcpu_initialise() being called unconditionally
>>>>>>> in
>>>>>>> hvm_vcpu_initialise() even though nestedhvm has not been configured for
>>>>>>> this
>>>>>>> domain. This then fails because there is no nestedhvm handler for Intel
>>>>>>> yet.
>>>>>> This bug should be fixed by 23175:e5a750d1bf9b, arriving at a staging
>>>>>> tree near you at any moment.
>>>>>> 
>>>>>>> And beyond that it's most likely an issue of a broken failure path, or a
>>>>>>> problem in the HAP code with cleaning up a half-created guest.
>>>>>> This one I'll look into now.
>>>>>> 
>>>>>> Tim.
>>>>>> 
>>>>>> 
>>>> 
>> 
>> 
> 

Re: Re: [Xen-users] Nested Virtualisation question

[Christoph Egger]

On 04/07/11 09:09, Keir Fraser wrote:
> On 07/04/2011 06:36, "Jeroen Groenewegen van der Weyden"
> <groen692@grosc.com>  wrote:
>
>> I couldn't help noticing some patches were added in the
>> /staging/xen-unstable branch. So I gave it a go.
>>
>> I know it is not called the unstable branch for nothing but, but I want
>> to let you know that it the entire system crashes when any domain is
>> created:
>> xl create someDomain ->  results in immediate system crash
>
> Probably the same as this, happening often in the automated tests:
>
> Apr  7 03:09:18.695893 (XEN) hvm_vcpu_initialise: nestedhvm_vcpu_initialise
> returned -95
> Apr  7 03:09:18.703835 (XEN) Assertion 'd->arch.paging.hap.total_pages == 0'
> failed at hap.c:699
> [...]
> Apr  7 03:09:18.915908 (XEN) Xen call trace:
> Apr  7 03:09:18.915950 (XEN)    [<ffff82c48020429f>]
> hap_teardown+0x1a8/0x229
> Apr  7 03:09:18.923861 (XEN)    [<ffff82c4801ccb7c>]
> paging_teardown+0x25/0x135
> Apr  7 03:09:18.923922 (XEN)    [<ffff82c4801584d1>]
> domain_relinquish_resources+0x6b/0x23c
> Apr  7 03:09:18.935903 (XEN)    [<ffff82c480105287>] domain_kill+0x95/0xe9
> Apr  7 03:09:18.943841 (XEN)    [<ffff82c48010324c>] do_domctl+0xd8e/0x1a02
> Apr  7 03:09:18.943900 (XEN)    [<ffff82c480212f28>]
> syscall_enter+0xc8/0x122
> Apr  7 03:09:18.955837 (XEN)
> Apr  7 03:09:18.955875 (XEN)
> Apr  7 03:09:18.955909 (XEN) ****************************************
> Apr  7 03:09:18.955961 (XEN) Panic on CPU 7:
> Apr  7 03:09:18.956003 (XEN) Assertion 'd->arch.paging.hap.total_pages == 0'
> failed at hap.c:699
> Apr  7 03:09:18.963863 (XEN) ****************************************
>

This seems to happen on guest destroy.
Which value has d->arch.paging.hap.total_pages ?

Christoph


-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85689 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

Re: Re: [Xen-users] Nested Virtualisation question

[Keir Fraser]

On 07/04/2011 10:09, "Christoph Egger" <Christoph.Egger@amd.com> wrote:

> This seems to happen on guest destroy.

The original cause is nestedhvm_vcpu_initialise() being unconditionally
called on the create path, and always failing on Intel.

The eventual crash is a secondary, latent bug I suspect.

 -- Keir

> Which value has d->arch.paging.hap.total_pages ?

Re: Re: [Xen-users] Nested Virtualisation question

[Christoph Egger]

On 04/07/11 07:36, Jeroen Groenewegen van der Weyden wrote:
> I couldn't help noticing some patches were added in the
> /staging/xen-unstable branch. So I gave it a go.
>
> I know it is not called the unstable branch for nothing but, but I want
> to let you know that it the entire system crashes when any domain is
> created:
> xl create someDomain ->  results in immediate system crash

Can you give the backtrace output, please ?

Christoph


> System:
> - domO system openSUSE 11.4 x86_64. kernel 2.6.37.1-1.2-xen
> - xen build from staging/xen-unstable (c/s 23172)
> - proc intel q9450
>
>
> +++++++++++++console output
> hydra:/etc/xen/vm # xl create groscWindowsXpWorkstation
> Parsing config file groscWindowsXpWorkstation
> xc: info: VIRTUAL MEMORY ARRANGEMENT:
>     Loader:        0000000000100000->00000000001795b0
>     TOTAL:         0000000000000000->0000000017800000
>     ENTRY ADDRESS: 0000000000100000
> xc: info: PHYSICAL MEMORY ALLOCATION:
>     4KB PAGES: 0x0000000000000200
>     2MB PAGES: 0x00000000000000bb
>     1GB PAGES: 0x0000000000000000
> libxl: error: libxl_device.c:475:libxl__wait_for_device_model Device
> Model not ready
> xl: fatal error: libxl_create.c:532, rc=-1:
> libxl__confirm_device_model_startup
>
> +++++++++++++
>



-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85689 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

Re: Re: [Xen-users] Nested Virtualisation question

[Jeroen Groenewegen van der Weyden]

I'll be glad to do that, Only I'm really new to this, please give me 
information on how to gather this information.

mfg,
Jeroen


Op 7-4-2011 11:04, Christoph Egger schreef:
> On 04/07/11 07:36, Jeroen Groenewegen van der Weyden wrote:
>> I couldn't help noticing some patches were added in the
>> /staging/xen-unstable branch. So I gave it a go.
>>
>> I know it is not called the unstable branch for nothing but, but I want
>> to let you know that it the entire system crashes when any domain is
>> created:
>> xl create someDomain ->  results in immediate system crash
>
> Can you give the backtrace output, please ?
>
> Christoph
>
>
>> System:
>> - domO system openSUSE 11.4 x86_64. kernel 2.6.37.1-1.2-xen
>> - xen build from staging/xen-unstable (c/s 23172)
>> - proc intel q9450
>>
>>
>> +++++++++++++console output
>> hydra:/etc/xen/vm # xl create groscWindowsXpWorkstation
>> Parsing config file groscWindowsXpWorkstation
>> xc: info: VIRTUAL MEMORY ARRANGEMENT:
>>     Loader:        0000000000100000->00000000001795b0
>>     TOTAL:         0000000000000000->0000000017800000
>>     ENTRY ADDRESS: 0000000000100000
>> xc: info: PHYSICAL MEMORY ALLOCATION:
>>     4KB PAGES: 0x0000000000000200
>>     2MB PAGES: 0x00000000000000bb
>>     1GB PAGES: 0x0000000000000000
>> libxl: error: libxl_device.c:475:libxl__wait_for_device_model Device
>> Model not ready
>> xl: fatal error: libxl_create.c:532, rc=-1:
>> libxl__confirm_device_model_startup
>>
>> +++++++++++++
>>
>
>
>