New "IndustryStandard" fw_cfg?

New "IndustryStandard" fw_cfg?

[Dionna Amalie Glaze]

Hi y'all, I'm Dionna. I work on Confidential VMs at Google Cloud. I've
been keeping up with the TDX and SEV-SNP developments in OVMF and
Linux, and some in Qemu.

There's a new UEFI feature in v2.9 of the specification (March 2021)
that allows for memory ranges to be classified as "unaccepted", since
both TDX and SEV-SNP require that the guest VM accept any host-made
changes to page state. We should expect newer technologies on non-x86
architectures to require memory acceptance as well. Operating systems
are not necessarily going to support this memory type, however.

This leads to a problem: how does the UEFI know that the OS it's going
to boot will support unaccepted memory? Right now we (Google Compute
Engine) have a system of "tagging" for guest image providers to state
that their OS supports some new feature so that we can enable
appropriate configurations for certain images.

I could go about adding a Google-specific fw_cfg file path and
definition to tell our custom OVMF build to use unaccepted memory or
not, but I personally prefer open source. I don't know y'all's process
though, so I'm asking before making a patch set.

There are two approaches I've considered.

1. An arch-specific config key for a u64 value:

The idea would be that I would add QemuFwCfgItemUnacceptedMinimum = 0x8005 here
https://github.com/tianocore/edk2/blob/master/OvmfPkg/Include/IndustryStandard/QemuFwCfg.h#L50

For Qemu, the main code I see for adding config is here, but I'm not
sure what y'all's preferred external configuration method is to get a
value from an invocation (flag, config file, etc) to fw_cfg.c:
https://github.com/qemu/qemu/blob/58b53669e87fed0d70903e05cd42079fbbdbc195/hw/i386/fw_cfg.c#L95

We'd add something like

fw_cfg_add_u64(fw_cfg, FW_CFG_MINIMUM_ACCEPTED_MEMORY_SIZE,
ms->minimum_accepted_memory_size);

where FW_CFG_MINIMUM_ACCEPTED_MEMORY_SIZE is #defined as
FW_CFG_ARCH_LOCAL + 5 in
https://github.com/qemu/qemu/blob/266469947161aa10b1d36843580d369d5aa38589/hw/i386/fw_cfg.h

The name has "minimum" in it since the firmware can choose to accept
more than the minimum, and specifically interpret 0 as UINT64_MAX.

2. A "well-known" file path to be included in the file slots starting
at 0x0020, such as "etc/min_accepted_mem_size", still plumbed through
like in 1.

Thanks!

-- 
-Dionna Glaze, PhD (she/her)

Re: New "IndustryStandard" fw_cfg?

[Gerd Hoffmann]

  Hi,

> > There's a new UEFI feature in v2.9 of the specification (March 2021) that
> > allows for memory ranges to be classified as "unaccepted", since both TDX
> > and SEV-SNP require that the guest VM accept any host-made changes to
> > page state. We should expect newer technologies on non-x86 architectures
> > to require memory acceptance as well. Operating systems are not
> > necessarily going to support this memory type, however.

> > For Qemu, the main code I see for adding config is here, but I'm not sure
> > what y'all's preferred external configuration method is to get a value from an

Ideally no external configuration, although I suspect we need something
at least temporarily.

IMHO the long-term goal should be to make this fully automatic, by
having efi apps (which includes the linux kernel's efi stub) and
firmware negotiate this.  Problem is this most likely requires changing
the uefi specs, which will take a while.

One possible way I see is extending efi boot services with a
GetMemoryMapEx() call, with an additional flags parameter where the
caller can specify that it can handle unaccepted memory with a flag
bit.  When the guest does not set the flag (or uses the old GetMemoryMap
call) the firmware must accept all memory and return a memory map
without unaccepted memory.

> > 2. A "well-known" file path to be included in the file slots starting at 0x0020,
> > such as "etc/min_accepted_mem_size", still plumbed through like in 1.

New options should use a file path.

See also docs/specs/fw_cfg.txt in qemu source tree.

take care,
  Gerd

Re: New "IndustryStandard" fw_cfg?

[Xiaoyao Li]

On 6/15/2022 8:46 AM, Xu, Min M wrote:
> I would like to add more engineers (Confidential Computing Reviewers in EDK2 community and Intel's QEMU engineers) in this mail thread.
> 
>> -----Original Message-----
>> From: Dionna Amalie Glaze <dionnaglaze@google.com>
>> Sent: Wednesday, June 15, 2022 2:09 AM
>> To: qemu-devel@nongnu.org
>> Cc: Xu, Min M <min.m.xu@intel.com>; Lendacky, Thomas
>> <Thomas.Lendacky@amd.com>
>> Subject: New "IndustryStandard" fw_cfg?
>>
>> Hi y'all, I'm Dionna. I work on Confidential VMs at Google Cloud. I've been
>> keeping up with the TDX and SEV-SNP developments in OVMF and Linux,
>> and some in Qemu.
>>
>> There's a new UEFI feature in v2.9 of the specification (March 2021) that
>> allows for memory ranges to be classified as "unaccepted", since both TDX
>> and SEV-SNP require that the guest VM accept any host-made changes to
>> page state. We should expect newer technologies on non-x86 architectures
>> to require memory acceptance as well. Operating systems are not
>> necessarily going to support this memory type, however.
>>
>> This leads to a problem: how does the UEFI know that the OS it's going to
>> boot will support unaccepted memory? 

Why does UEFI need to know it?

Per my understanding, Unaccepted Memory in UEFI is introduced for 
confidential VMs, i.e., for Intel TDX and AMD SEV-SNP. The only reason 
UEFI/OVMF reports "Unaccepted Memory" to OS, is a confidential VM is 
desired. Thus, the (guset) OS has to be enlightened to know how to 
handle unaccepted memory. And of course, the non-confidential 
enlightened OS, e.g., old linux kernel, fails boot/hits issue if it 
doesn't support unaccepted memory.

Re: New "IndustryStandard" fw_cfg?

[Tom Lendacky]

On 6/15/22 10:19, Xiaoyao Li wrote:
> On 6/15/2022 8:46 AM, Xu, Min M wrote:
>> I would like to add more engineers (Confidential Computing Reviewers in 
>> EDK2 community and Intel's QEMU engineers) in this mail thread.
>>
>>> -----Original Message-----
>>> From: Dionna Amalie Glaze <dionnaglaze@google.com>
>>> Sent: Wednesday, June 15, 2022 2:09 AM
>>> To: qemu-devel@nongnu.org
>>> Cc: Xu, Min M <min.m.xu@intel.com>; Lendacky, Thomas
>>> <Thomas.Lendacky@amd.com>
>>> Subject: New "IndustryStandard" fw_cfg?
>>>
>>> Hi y'all, I'm Dionna. I work on Confidential VMs at Google Cloud. I've 
>>> been
>>> keeping up with the TDX and SEV-SNP developments in OVMF and Linux,
>>> and some in Qemu.
>>>
>>> There's a new UEFI feature in v2.9 of the specification (March 2021) that
>>> allows for memory ranges to be classified as "unaccepted", since both TDX
>>> and SEV-SNP require that the guest VM accept any host-made changes to
>>> page state. We should expect newer technologies on non-x86 architectures
>>> to require memory acceptance as well. Operating systems are not
>>> necessarily going to support this memory type, however.
>>>
>>> This leads to a problem: how does the UEFI know that the OS it's going to
>>> boot will support unaccepted memory? 
> 
> Why does UEFI need to know it?
> 
> Per my understanding, Unaccepted Memory in UEFI is introduced for 
> confidential VMs, i.e., for Intel TDX and AMD SEV-SNP. The only reason 
> UEFI/OVMF reports "Unaccepted Memory" to OS, is a confidential VM is 
> desired. Thus, the (guset) OS has to be enlightened to know how to handle 
> unaccepted memory. And of course, the non-confidential enlightened OS, 
> e.g., old linux kernel, fails boot/hits issue if it doesn't support 
> unaccepted memory.

As of today, SNP guest support is part of current OVMF and Linux 5.19-rcX, 
but support for unaccepted memory is not. The current OVMF SNP guest 
support will accept all the guest memory and the Linux SNP guest support 
will terminate the SNP guest if a page is accessed that has not been accepted.

Thanks,
Tom

> 

Re: New "IndustryStandard" fw_cfg?

[Dionna Amalie Glaze]

> > > For Qemu, the main code I see for adding config is here, but I'm not sure
> > > what y'all's preferred external configuration method is to get a value from an
>
> Ideally no external configuration, although I suspect we need something
> at least temporarily.

Yes, whereas TDX can assume unaccepted memory is supported as part of
its "TDX support" set of capabilities an OS has, SEV-SNP has already
been released and is supported. We therefore need to not break
existing images that "support SEV-SNP".

>
> IMHO the long-term goal should be to make this fully automatic, by
> having efi apps (which includes the linux kernel's efi stub) and
> firmware negotiate this.  Problem is this most likely requires changing
> the uefi specs, which will take a while.
>
> One possible way I see is extending efi boot services with a
> GetMemoryMapEx() call, with an additional flags parameter where the
> caller can specify that it can handle unaccepted memory with a flag
> bit.  When the guest does not set the flag (or uses the old GetMemoryMap
> call) the firmware must accept all memory and return a memory map
> without unaccepted memory.

To allow for future weird memory extensions, I'd recommend this being
a struct with initial size field, but yes.
Sounds like a new UEFI spec would be needed for this negotiation.

>
> > > 2. A "well-known" file path to be included in the file slots starting at 0x0020,
> > > such as "etc/min_accepted_mem_size", still plumbed through like in 1.
>
> New options should use a file path.
>
> See also docs/specs/fw_cfg.txt in qemu source tree.
>

Thanks for this.

> take care,
>   Gerd
>


-- 
-Dionna Glaze, PhD (she/her)

Re: New "IndustryStandard" fw_cfg?

[Gerd Hoffmann]

  Hi,

> Per my understanding, Unaccepted Memory in UEFI is introduced for
> confidential VMs, i.e., for Intel TDX and AMD SEV-SNP. The only reason
> UEFI/OVMF reports "Unaccepted Memory" to OS, is a confidential VM is
> desired.

No.  Reporting "Unaccepted Memory" to the OS is not a hard requirement
for confidential VMs, it only optimizes boot times.  Instead of doing
that time-consuming process in the firmware for all memory we tell the
guest OS which memory is accepted already and which is not.  So the
guest OS can go accept the remaining memory in a background process.

take care,
  Gerd

Re: New "IndustryStandard" fw_cfg?

[Xiaoyao Li]

On 6/16/2022 1:37 PM, Gerd Hoffmann wrote:
>    Hi,
> 
>> Per my understanding, Unaccepted Memory in UEFI is introduced for
>> confidential VMs, i.e., for Intel TDX and AMD SEV-SNP. The only reason
>> UEFI/OVMF reports "Unaccepted Memory" to OS, is a confidential VM is
>> desired.
> 
> No.  Reporting "Unaccepted Memory" to the OS is not a hard requirement
> for confidential VMs, it only optimizes boot times.  Instead of doing
> that time-consuming process in the firmware for all memory we tell the
> guest OS which memory is accepted already and which is not.  So the
> guest OS can go accept the remaining memory in a background process.

But for non-confidential VMs, even a range of memory is reported as 
unaccepted nothing prevents it from being accessed without accepting it, 
and it's not time-consuming. Did I miss anything?

> take care,
>    Gerd
> 

Re: New "IndustryStandard" fw_cfg?

[Gerd Hoffmann]

On Thu, Jun 16, 2022 at 01:49:11PM +0800, Xiaoyao Li wrote:
> On 6/16/2022 1:37 PM, Gerd Hoffmann wrote:
> >    Hi,
> > 
> > > Per my understanding, Unaccepted Memory in UEFI is introduced for
> > > confidential VMs, i.e., for Intel TDX and AMD SEV-SNP. The only reason
> > > UEFI/OVMF reports "Unaccepted Memory" to OS, is a confidential VM is
> > > desired.
> > 
> > No.  Reporting "Unaccepted Memory" to the OS is not a hard requirement
> > for confidential VMs, it only optimizes boot times.  Instead of doing
> > that time-consuming process in the firmware for all memory we tell the
> > guest OS which memory is accepted already and which is not.  So the
> > guest OS can go accept the remaining memory in a background process.
> 
> But for non-confidential VMs, even a range of memory is reported as
> unaccepted nothing prevents it from being accessed without accepting it, and
> it's not time-consuming. Did I miss anything?

The concept of "Unaccepted Memory" only exists for confidential VM.

That doesn't imply the guest OS must be able to handle unaccepted
memory though because it is possible to simply accept all memory
in the firmware.  Which in fact is the common case today.

take care,
  Gerd

Re: New "IndustryStandard" fw_cfg?

[Xiaoyao Li]

On 6/15/2022 8:46 AM, Xu, Min M wrote:
> I would like to add more engineers (Confidential Computing Reviewers in EDK2 community and Intel's QEMU engineers) in this mail thread.
> 
>> -----Original Message-----
>> From: Dionna Amalie Glaze <dionnaglaze@google.com>
>> Sent: Wednesday, June 15, 2022 2:09 AM
>> To: qemu-devel@nongnu.org
>> Cc: Xu, Min M <min.m.xu@intel.com>; Lendacky, Thomas
>> <Thomas.Lendacky@amd.com>
>> Subject: New "IndustryStandard" fw_cfg?
>>
>> Hi y'all, I'm Dionna. I work on Confidential VMs at Google Cloud. I've been
>> keeping up with the TDX and SEV-SNP developments in OVMF and Linux,
>> and some in Qemu.
>>
>> There's a new UEFI feature in v2.9 of the specification (March 2021) that
>> allows for memory ranges to be classified as "unaccepted", since both TDX
>> and SEV-SNP require that the guest VM accept any host-made changes to
>> page state. We should expect newer technologies on non-x86 architectures
>> to require memory acceptance as well. Operating systems are not
>> necessarily going to support this memory type, however.
>>
>> This leads to a problem: how does the UEFI know that the OS it's going to
>> boot will support unaccepted memory? 

After re-read and re-think, I think the problem is better to state as: 
we need an interface for QEMU to tell OVMF how much memory it needs to 
accept, from [Minimum to All]. So for the case that user wants to boot 
an partial-enabled confidential VMs (like current Linux TDX and SNP 
guest), user needs to specify from QEMU to tell OVMF to accept all the 
memory.

> Right now we (Google Compute
>> Engine) have a system of "tagging" for guest image providers to state that
>> their OS supports some new feature so that we can enable appropriate
>> configurations for certain images.
>>
>> I could go about adding a Google-specific fw_cfg file path and definition to
>> tell our custom OVMF build to use unaccepted memory or not, but I
>> personally prefer open source. I don't know y'all's process though, so I'm
>> asking before making a patch set.
>>
>> There are two approaches I've considered.
>>
>> 1. An arch-specific config key for a u64 value:
>>
>> The idea would be that I would add QemuFwCfgItemUnacceptedMinimum =
>> 0x8005 here
>> https://github.com/tianocore/edk2/blob/master/OvmfPkg/Include/Industry
>> Standard/QemuFwCfg.h#L50
>>
>> For Qemu, the main code I see for adding config is here, but I'm not sure
>> what y'all's preferred external configuration method is to get a value from an
>> invocation (flag, config file, etc) to fw_cfg.c:
>> https://github.com/qemu/qemu/blob/58b53669e87fed0d70903e05cd42079
>> fbbdbc195/hw/i386/fw_cfg.c#L95
>>
>> We'd add something like
>>
>> fw_cfg_add_u64(fw_cfg, FW_CFG_MINIMUM_ACCEPTED_MEMORY_SIZE,
>> ms->minimum_accepted_memory_size);
>>
>> where FW_CFG_MINIMUM_ACCEPTED_MEMORY_SIZE is #defined as
>> FW_CFG_ARCH_LOCAL + 5 in
>> https://github.com/qemu/qemu/blob/266469947161aa10b1d36843580d36
>> 9d5aa38589/hw/i386/fw_cfg.h
>>
>> The name has "minimum" in it since the firmware can choose to accept
>> more than the minimum, and specifically interpret 0 as UINT64_MAX.
>>
>> 2. A "well-known" file path to be included in the file slots starting at 0x0020,
>> such as "etc/min_accepted_mem_size", still plumbed through like in 1.
>>
>> Thanks!
>>
>> --
>> -Dionna Glaze, PhD (she/her)

Re: New "IndustryStandard" fw_cfg?

[Gerd Hoffmann]

  Hi,

> After re-read and re-think, I think the problem is better to state as: we
> need an interface for QEMU to tell OVMF how much memory it needs to accept,
> from [Minimum to All]. So for the case that user wants to boot an
> partial-enabled confidential VMs (like current Linux TDX and SNP guest),
> user needs to specify from QEMU to tell OVMF to accept all the memory.

Asking the user to manually configure stuff sucks, that's why I think
it makes sense to let firmware and guest negotiate this automatically.

That doesn't work today though, so we will need some config option
indeed.

The proposal in the parallel thread is to just accept all low memory
(below 4G) unconditionally.  So maybe it is enough to have:

  * accept all memory below 4G
  * accept all memory

Possibly we need:

  * accept all memory below 4G
  * accept all memory below 4G, plus x GB of high memory.
  * accept all memory

In any case the config option should be designed in a way that we can
add a 'automatic' choice later, i.e. we can have ...

  * automatic (default)
  * accept all memory below 4G
  * accept all memory

... once the automatic negotiation is available.

take care,
  Gerd

Re: New "IndustryStandard" fw_cfg?

[Xiaoyao Li]

On 6/16/2022 4:28 PM, Gerd Hoffmann wrote:
>    Hi,
> 
>> After re-read and re-think, I think the problem is better to state as: we
>> need an interface for QEMU to tell OVMF how much memory it needs to accept,
>> from [Minimum to All]. So for the case that user wants to boot an
>> partial-enabled confidential VMs (like current Linux TDX and SNP guest),
>> user needs to specify from QEMU to tell OVMF to accept all the memory.
> 
> Asking the user to manually configure stuff sucks, that's why I think
> it makes sense to let firmware and guest negotiate this automatically.
> 
> That doesn't work today though, so we will need some config option
> indeed.
> 
> The proposal in the parallel thread is to just accept all low memory
> (below 4G) unconditionally.  So maybe it is enough to have:
> 
>    * accept all memory below 4G
>    * accept all memory
> 
> Possibly we need:
> 
>    * accept all memory below 4G
>    * accept all memory below 4G, plus x GB of high memory.
>    * accept all memory
> 
> In any case the config option should be designed in a way that we can
> add a 'automatic' choice later, i.e. we can have ...
> 
>    * automatic (default)
>    * accept all memory below 4G
>    * accept all memory
> 
> ... once the automatic negotiation is available.

IMO, basically, we need two options:

1. accept all memory, for the case that guest OS cannot handle 
unaccepted memory;

2. accept minimal memory, for the case that guest OS is capable of
accepting memory.

    Accepting memory is time consuming. To save as much time as possible,
    OVMF only needs to accept:

    a. the enough memory for its own code running;

    b. the enough memory for guest OS to setup everything ready to accept
       memory on guest own. e.g.,

       i.  the memory where guest kernel locates;
       ii. the stack/heap/memory buffer, from which guest kernel needs
           to allocate memory

    As long as both a) and b) are meet, we can accept as less as possible
    to save the time OVMF is running.

    Min did some POC on this for TDX, that accepts 256M/128M (I don't
    remember the exact number) in OVMF for guest kernel to operate
    normally and guest can accept remaining unaccepted memory on its own.

More flexible, we can provide a 3) option, that guest tells OVMF the GPA 
ranges that need to be accepted by OVMF.

> take care,
>    Gerd
> 

Re: New "IndustryStandard" fw_cfg?

[Dionna Amalie Glaze]

I think the option should be boolean since it doesn't look like we're
going to need to tune the number very much.
It all boils down to "does the OS affirmatively support unaccepted
memory?" as in, we have no way to negotiate it, but force unaccepted
memory on.
Ovmf can interpret the existence of an opt/ovmf/unaccepted_memory file
to mean that it's allowed to create unaccepted memory entries in the
memory map.
It's then up to the firmware if it will minimize its use of unaccepted
memory or not. It's not Qemu's place to say.


> >    * accept all memory below 4G
> >    * accept all memory
> >
> > Possibly we need:
> >
> >    * accept all memory below 4G
> >    * accept all memory below 4G, plus x GB of high memory.
> >    * accept all memory
> >
> > In any case the config option should be designed in a way that we can
> > add a 'automatic' choice later, i.e. we can have ...
> >
> >    * automatic (default)
> >    * accept all memory below 4G
> >    * accept all memory
> >

I think "false" can mean either accept all memory or "do what you need
to" and negotiate if the memory map boot service can create unaccepted
memory entries. Whichever appears supported.
Then "true" can be "do whatever, including creating unaccepted memory
entries in the memory map".

That seems the simplest way to allow a configuration of this feature.

-- 
-Dionna Glaze, PhD (she/her)

Re: New "IndustryStandard" fw_cfg?

[Gerd Hoffmann]

  Hi,

> Ovmf can interpret the existence of an opt/ovmf/unaccepted_memory file

> I think "false" can mean either accept all memory or "do what you need
> to" and negotiate if the memory map boot service can create unaccepted
> memory entries. Whichever appears supported.
> Then "true" can be "do whatever, including creating unaccepted memory
> entries in the memory map".
> 
> That seems the simplest way to allow a configuration of this feature.

Well, we actually have three states: file is not present, file is
present and true, file is present and false.  The existing options
use CamelCase, so we should stick to that.  A name which makes clear
what the option does would be good I think.  So, how about this:

    opt/ovmf/AcceptAllMemory=true
	ovmf must accept all memory.

    opt/ovmf/AcceptAllMemory=false
	ovmf should pass memory above 4G as unaccepted
	memory to the guest os.

    opt/ovmf/AcceptAllMemory not present
	ovmf can choose what to do (including negotiate).

take care,
  Gerd