From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78B4CC4338F for ; Sun, 25 Jul 2021 02:47:37 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1E75260725 for ; Sun, 25 Jul 2021 02:47:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 1E75260725 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=comcast.net Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.160544.295167 (Exim 4.92) (envelope-from ) id 1m7UAc-0001Vl-RD; Sun, 25 Jul 2021 02:47:14 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 160544.295167; Sun, 25 Jul 2021 02:47:14 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1m7UAc-0001Ve-Nn; Sun, 25 Jul 2021 02:47:14 +0000 Received: by outflank-mailman (input) for mailman id 160544; Sun, 25 Jul 2021 02:47:12 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1m7UAa-0001VY-Ld for xen-devel@lists.xenproject.org; Sun, 25 Jul 2021 02:47:12 +0000 Received: from resqmta-po-04v.sys.comcast.net (unknown [2001:558:fe16:19:96:114:154:163]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 1f3cd1f1-5528-463a-a757-9f820b0f5d31; Sun, 25 Jul 2021 02:47:10 +0000 (UTC) Received: from resomta-po-16v.sys.comcast.net ([96.114.154.240]) by resqmta-po-04v.sys.comcast.net with ESMTP id 7U6Dma5m7bVMM7UAXm6m2b; Sun, 25 Jul 2021 02:47:09 +0000 Received: from ASUSN550JK ([IPv6:2601:80:4203:a8a1:8422:6d3f:11dc:fd93]) by resomta-po-16v.sys.comcast.net with ESMTPSA id 7UAVm9pyeQlkw7UAWmtAcS; Sun, 25 Jul 2021 02:47:09 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 1f3cd1f1-5528-463a-a757-9f820b0f5d31 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=20190202a; t=1627181229; bh=vIEhIMU8BzmITAhbbJRTKgqnf0JMgEzwjNAy96B/4FQ=; h=Received:Received:From:To:Subject:Date:Message-ID:MIME-Version: Content-Type; b=kTsPhlsOAjAXhtPd9pHEWqSu/a/LQ9Le3OsfdwCLQjQLW/kSS4SIlkBsVCqRsdv5J 3yOvB6p/SGMBuCOotyuHLk06VeRj2w8qkT5piNWkzSEf0CrAja/qkeh8IfCWDHFDsD G2u9OgIqLYOGJ7TO2GCX5xN47Sou3FF6mkKGr6F5/IbCSJSgXo81GKijLGYh4VS+++ spCm4/xFLX8MTaLQrx2cKYDwfZP802FGjfkEo+p7t4yI3dDodh7pAmTjZHL2t3YrB8 sOEcgbOmmcO4c9bEBIrBpI0zouNYn1U6IlDNkJFTQVYNanzW/hqraT1cxjkhdseJZ2 kZMyypUTPJJVQ== X-Xfinity-VMeta: sc=0.00;st=legit From: "Xentrigued" To: Cc: References: <001401d77de6$34ff5de0$9efe19a0$@comcast.net> In-Reply-To: Subject: RE: Nested Virtualization of Hyper-V on Xen Not Working Date: Sat, 24 Jul 2021 22:47:06 -0400 Message-ID: <007a01d780ff$5caff450$160fdcf0$@comcast.net> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_007B_01D780DD.D5A33650" X-Mailer: Microsoft Outlook 16.0 Thread-Index: AQL3Fd3+iiQxDo+LxWnbDNTlihC0wQH+TtHMqQPSvJA= Content-Language: en-us This is a multipart message in MIME format. ------=_NextPart_000_007B_01D780DD.D5A33650 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable First and foremost, many thanks for your thoughtful and thorough = response and also for providing a multitude of genuinely helpful = information! =20 Secondly: Wow, that=E2=80=99s quite a homework assignment!! =20 I will absolutely begin to work my way through the resources you cited = and will report back once some of those tests have been completed. = You=E2=80=99ve given me an excellent starting point for further inquiry. =20 To be very honest, I wasn=E2=80=99t sure where to turn next in the event = that no member of this august body had anything to say about this. = (It=E2=80=99s kind of intimidating and not unlike going before the = Wizard of Oz.) =20 So again, thank you so much for all of the good information and also for = your kindness in reaching out. =20 From: Rob Townley =20 Sent: Saturday, July 24, 2021 9:33 PM To: Xentrigued Cc: xen-devel@lists.xenproject.org Subject: Re: Nested Virtualization of Hyper-V on Xen Not Working =20 I encourage you to run the Windows Hardware Lab Kit 11/02/2018 = or HLK or = maybe try the VHLK. The VHLK = is a free VHD file download of win2016 that has all the = tests necessary built-in. So you could manually download the test kit = on your existing Windows VM or attempt the VHD. "Default login = credentials are HLKAdminUser with password Testpassword,1" =20 =20 Please post the results. Citrix 8.1 and 8.2 are listed as validated = and so would be = very interesting to see any differences in test results running XCP-ng = 8.2 and Citrix 8.2. =20 =20 Why run the hardware lab kit in a virtualized environment and directly = on the underlying hardware? Because those tests are used to validate = for the SVVP = . = Microsoft has something similar to their Hardware Compatibility List, = aka HCL. SVVP = is = Microsoft's Server Virtualization Validation Program. SVVP validates = that Windows Operating Systems and APPS run on top of other hypervisors = and once validated will receive technical support. SVVP has been around = for over a decade but has of course changed over the years. Recently, = it has been making news because Win11 / Win2022 requires a TPM 2.0 chip, = but XCP-NG XEN does not yet support that = . If the hypervisor is = SVVP certified, then running MS Hyper-V Windows on top of any validated = hypervisor would be much more likely to work and possibly supported = directly by MS and tsanet.org . Canonical and = RedHat are in tsanet, but would like to see the Linux Foundation or = Vates itself. =20 =20 = Microsoft server software = and supported virtualization environments 09/08/2020 6 minutes to read = Support partners for = non-Microsoft hardware virtualization software WindowsServerCatalog.com and then click on SVVP in the upper right and = then Products = =20 = =20 Design Session - Alternative vTPM 2.0 Backend to Comply with Upcoming = SVVP Changes =20 https://www.youtube.com/watch?v=3DabkRRcoYWCQ =20 Enabling UEFI Secure Boot on Xen - Robert Eshleman, Vates SAS = =20 https://www.youtube.com/watch?v=3DA_IhKjK7EgA = &t=3D388s Support vTPM for guests #471 https://github.com/xcp-ng/xcp/issues/471 = =20 https://github.com/xcp-ng/xcp/issues/471 https://docs.microsoft.com/en-us/windows-hardware/test/hlk/ =20 On Tue, Jul 20, 2021 at 11:12 PM Xentrigued > wrote: RATIONALE: Features in recent versions of Windows now REQUIRE Hyper-V support to work. In particular, Windows Containers, Sandbox, Docker = Desktop and the Windows Subsystem for Linux version 2 (WSL2). Running Windows = in a VM as a development and test platform is currently a common requirement = for various user segments and will likely become necessary for production in = the future. Nested virtualization of Hyper-V currently works on VMware = ESXi, Microsoft Hyper-V and KVM-based hypervisors. This puts Xen and its derivatives at a disadvantage when choosing a hypervisor. WHAT IS NOT WORKING? Provided the requirements set forth in: https://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen have been = met, an hvm guest running Windows 10 PRO Version 21H1 x64 shows that all four requirements for running Hyper-V are available using the msinfo32.exe or systeminfo.exe commands. More granular knowledge of the CPU = capabilities exposed to the guest can be observed using the Sysinternals = Coreinfo64.exe command. CPUID flags present appear to mirror those on other working = nested hypervisor configurations. Enabling Windows Features for Hyper-V, Virtual Machine Platform, etc. = all appear to work without error. However, after the finishing reboot, = Hyper-V is simply not active. This--despite the fact that vmcompute.exe = (Hyper-V host compute service) is running and there are no errors in the logs. = In addition, all four Hyper-V prerequisites continue to show as available. By contrast, after the finishing reboot of an analogous Windows VM = running on ESXi, the four prerequisites are reversed: hypervisor is now active; vmx, ept and urg (unrestricted guest) are all off as viewed with the Coreinfo64.exe -v command. Furthermore, all functions requiring Hyper-V = are now active and working as expected. This deficiency has been observed in two test setups running Xen 4.15 = from source and XCP-ng 8.2, both running on Intel with all of the latest, generally available patches. We presume that the same behavior is = present on Citrix Hypervisor 8.2 as well. SUMMATION: Clearly, much effort has already been expended to support the Viridian enlightenments that optimize running Windows on Xen. It also looks like = a significant amount of effort has been put forth to advance nested virtualization in general. Therefore, if it would be helpful, I am willing to perform testing and provide feedback and logs as appropriate in order to help get this = working. While my day job is managing a heterogeneous collection of systems = running on various hypervisors, I have learned the rudiments of integrating = patches and rebuilding Xen from source so could no doubt be useful in assisting = you with this worthwhile endeavor. ------=_NextPart_000_007B_01D780DD.D5A33650 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

First and foremost, = many thanks for your thoughtful and thorough response and also for = providing a multitude of genuinely helpful = information!

 

Secondly:=C2=A0 = Wow, that=E2=80=99s quite a homework = assignment!!

 

I will absolutely = begin to work my way through the resources you cited and will report = back once some of those tests have been completed.=C2=A0 You=E2=80=99ve = given me an excellent starting point for further = inquiry.

 

To be very honest, = I wasn=E2=80=99t sure where to turn next in the event that no member of = this august body had anything to say about this.=C2=A0 (It=E2=80=99s = kind of intimidating and not unlike going before the Wizard of = Oz.)

 

So again, thank you = so much for all of the good information and also for your kindness in = reaching out.

 

From: Rob Townley = <rob.townley@gmail.com>
Sent: Saturday, July 24, 2021 = 9:33 PM
To: Xentrigued = <xentrigued@comcast.net>
Cc: = xen-devel@lists.xenproject.org
Subject: Re: Nested = Virtualization of Hyper-V on Xen Not Working

 

I = encourage you to run the Wind= ows Hardware Lab Kit 11/02/2018 or HLK or maybe try the VHLK.  = The VHLK is a free VHD file download of win2016 = that has all the tests necessary built-in.  So you could manually = download the test kit on your existing Windows VM or attempt the = VHD.   "Default login credentials are HLKAdminUser = with password Testpassword,1"    =

 

Please post the results.   Citrix 8.1 and 8.2 are listed as validated and = so would be very interesting to see any differences in test results = running XCP-ng 8.2 and Citrix 8.2. =  

 

Why run the hardware lab kit in a virtualized environment = and directly on the underlying hardware?  Because those tests are = used to validate for the SVVP.   Microsoft has something similar to their = Hardware Compatibility List, aka HCL.   SVVP is Microsoft's Server Virtualization Validation Program. =   SVVP validates that Windows Operating Systems and APPS run on top = of other hypervisors and once validated will receive technical = support.  SVVP has been around for over a decade but has of course = changed over the years.   Recently,  it has been making news = because Win11 / = Win2022 requires a TPM 2.0 chip, but XCP-NG XEN does not yet support = that.    If the hypervisor is SVVP certified, then running = MS Hyper-V Windows on top of any validated hypervisor would be = much more likely to work and possibly supported directly by MS and tsanet.org.  = Canonical and RedHat are in tsanet, but would like to see the Linux = Foundation or Vates itself. 

 


=

 

 

On = Tue, Jul 20, 2021 at 11:12 PM Xentrigued <xentrigued@comcast.net> = wrote:

RATIONALE: Features in recent versions of = Windows now REQUIRE Hyper-V
support to work.  In particular, = Windows Containers, Sandbox, Docker Desktop
and the Windows Subsystem = for Linux version 2 (WSL2).  Running Windows in a
VM as a = development and test platform is currently a common requirement = for
various user segments and will likely become necessary for = production in the
future.  Nested virtualization of Hyper-V = currently works on VMware ESXi,
Microsoft Hyper-V and KVM-based = hypervisors.  This puts Xen and its
derivatives at a = disadvantage when choosing a hypervisor.

WHAT IS NOT = WORKING?  Provided the requirements set forth in:
https://wiki.xenproject.org/wiki/Nested_Virtualization_= in_Xen have been met,
an hvm guest running Windows 10 PRO Version = 21H1 x64 shows that all four
requirements for running Hyper-V are = available using the msinfo32.exe or
systeminfo.exe commands.  = More granular knowledge of the CPU capabilities
exposed to the guest = can be observed using the Sysinternals Coreinfo64.exe
command.  = CPUID flags present appear to mirror those on other working = nested
hypervisor configurations.

Enabling Windows Features = for Hyper-V, Virtual Machine Platform, etc. all
appear to work = without error.  However, after the finishing reboot, Hyper-V
is = simply not active.  This--despite the fact that vmcompute.exe = (Hyper-V
host compute service) is running and there are no errors in = the logs.  In
addition, all four Hyper-V prerequisites continue = to show as available.

By contrast, after the finishing reboot of = an analogous Windows VM running
on ESXi, the four prerequisites are = reversed:  hypervisor is now active;
vmx, ept and urg = (unrestricted guest) are all off as viewed with the
Coreinfo64.exe -v = command.  Furthermore, all functions requiring Hyper-V are
now = active and working as expected.

This deficiency has been observed = in two test setups running Xen 4.15 from
source and XCP-ng 8.2, both = running on Intel with all of the latest,
generally available = patches.  We presume that the same behavior is present
on Citrix = Hypervisor 8.2 as well.

SUMMATION:
Clearly, much effort has = already been expended to support the Viridian
enlightenments that = optimize running Windows on Xen.  It also looks like = a
significant amount of effort has been put forth to advance = nested
virtualization in general.

Therefore, if it would be = helpful, I am willing to perform testing and
provide feedback and = logs as appropriate in order to help get this working.

While my = day job is managing a heterogeneous collection of systems running
on = various hypervisors, I have learned the rudiments of integrating = patches
and rebuilding Xen from source so could no doubt be useful in = assisting you
with this worthwhile = endeavor.


------=_NextPart_000_007B_01D780DD.D5A33650--