[PATCH 0/3] Documentation: KVM: add a place to document API quirks and (x86) CPU errata

[PATCH 0/3] Documentation: KVM: add a place to document API quirks and (x86) CPU errata

[Paolo Bonzini]

Reorganize Documentation/virt/kvm to be less x86-centric, and add a new
file to document the ways in which x86 emulation is "wrong".  There are
surely many more, for now just add the skeleton.  Please reply to this
message with things that you'd like to see documented.

While at it add also a section for places where the KVM API is bad or
returns wrong values.  Please also include them in reply to this message
if you are interested in seeing them documented.

Paolo

Paolo Bonzini (3):
  Documentation: KVM: add separate directories for architecture-specific
    documentation
  Documentation: KVM: add virtual CPU errata documentation
  Documentation: KVM: add API issues section

 Documentation/virt/kvm/api.rst                | 46 +++++++++++++++++++
 Documentation/virt/kvm/index.rst              | 28 ++++-------
 Documentation/virt/kvm/s390/index.rst         | 12 +++++
 .../virt/kvm/{ => s390}/s390-diag.rst         |  0
 .../virt/kvm/{ => s390}/s390-pv-boot.rst      |  0
 Documentation/virt/kvm/{ => s390}/s390-pv.rst |  0
 .../kvm/{ => x86}/amd-memory-encryption.rst   |  0
 Documentation/virt/kvm/{ => x86}/cpuid.rst    |  0
 Documentation/virt/kvm/x86/errata.rst         | 39 ++++++++++++++++
 .../virt/kvm/{ => x86}/halt-polling.rst       |  0
 .../virt/kvm/{ => x86}/hypercalls.rst         |  0
 Documentation/virt/kvm/x86/index.rst          | 19 ++++++++
 Documentation/virt/kvm/{ => x86}/mmu.rst      |  0
 Documentation/virt/kvm/{ => x86}/msr.rst      |  0
 .../virt/kvm/{ => x86}/nested-vmx.rst         |  0
 .../kvm/{ => x86}/running-nested-guests.rst   |  0
 .../virt/kvm/{ => x86}/timekeeping.rst        |  0
 17 files changed, 124 insertions(+), 20 deletions(-)
 create mode 100644 Documentation/virt/kvm/s390/index.rst
 rename Documentation/virt/kvm/{ => s390}/s390-diag.rst (100%)
 rename Documentation/virt/kvm/{ => s390}/s390-pv-boot.rst (100%)
 rename Documentation/virt/kvm/{ => s390}/s390-pv.rst (100%)
 rename Documentation/virt/kvm/{ => x86}/amd-memory-encryption.rst (100%)
 rename Documentation/virt/kvm/{ => x86}/cpuid.rst (100%)
 create mode 100644 Documentation/virt/kvm/x86/errata.rst
 rename Documentation/virt/kvm/{ => x86}/halt-polling.rst (100%)
 rename Documentation/virt/kvm/{ => x86}/hypercalls.rst (100%)
 create mode 100644 Documentation/virt/kvm/x86/index.rst
 rename Documentation/virt/kvm/{ => x86}/mmu.rst (100%)
 rename Documentation/virt/kvm/{ => x86}/msr.rst (100%)
 rename Documentation/virt/kvm/{ => x86}/nested-vmx.rst (100%)
 rename Documentation/virt/kvm/{ => x86}/running-nested-guests.rst (100%)
 rename Documentation/virt/kvm/{ => x86}/timekeeping.rst (100%)

-- 
2.35.1

[PATCH 1/3] Documentation: KVM: add separate directories for architecture-specific documentation

[Paolo Bonzini]

ARM already has an arm/ subdirectory, but s390 and x86 do not even though
they have a relatively large number of files specific to them.  Create
new directories in Documentation/virt/kvm for these two architectures
as well.

While at it, group the API documentation and the developer documentation
in the table of contents.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 Documentation/virt/kvm/index.rst              | 28 ++++++-------------
 Documentation/virt/kvm/s390/index.rst         | 12 ++++++++
 .../virt/kvm/{ => s390}/s390-diag.rst         |  0
 .../virt/kvm/{ => s390}/s390-pv-boot.rst      |  0
 Documentation/virt/kvm/{ => s390}/s390-pv.rst |  0
 .../kvm/{ => x86}/amd-memory-encryption.rst   |  0
 Documentation/virt/kvm/{ => x86}/cpuid.rst    |  0
 .../virt/kvm/{ => x86}/halt-polling.rst       |  0
 .../virt/kvm/{ => x86}/hypercalls.rst         |  0
 Documentation/virt/kvm/x86/index.rst          | 18 ++++++++++++
 Documentation/virt/kvm/{ => x86}/mmu.rst      |  0
 Documentation/virt/kvm/{ => x86}/msr.rst      |  0
 .../virt/kvm/{ => x86}/nested-vmx.rst         |  0
 .../kvm/{ => x86}/running-nested-guests.rst   |  0
 .../virt/kvm/{ => x86}/timekeeping.rst        |  0
 15 files changed, 38 insertions(+), 20 deletions(-)
 create mode 100644 Documentation/virt/kvm/s390/index.rst
 rename Documentation/virt/kvm/{ => s390}/s390-diag.rst (100%)
 rename Documentation/virt/kvm/{ => s390}/s390-pv-boot.rst (100%)
 rename Documentation/virt/kvm/{ => s390}/s390-pv.rst (100%)
 rename Documentation/virt/kvm/{ => x86}/amd-memory-encryption.rst (100%)
 rename Documentation/virt/kvm/{ => x86}/cpuid.rst (100%)
 rename Documentation/virt/kvm/{ => x86}/halt-polling.rst (100%)
 rename Documentation/virt/kvm/{ => x86}/hypercalls.rst (100%)
 create mode 100644 Documentation/virt/kvm/x86/index.rst
 rename Documentation/virt/kvm/{ => x86}/mmu.rst (100%)
 rename Documentation/virt/kvm/{ => x86}/msr.rst (100%)
 rename Documentation/virt/kvm/{ => x86}/nested-vmx.rst (100%)
 rename Documentation/virt/kvm/{ => x86}/running-nested-guests.rst (100%)
 rename Documentation/virt/kvm/{ => x86}/timekeeping.rst (100%)

diff --git a/Documentation/virt/kvm/index.rst b/Documentation/virt/kvm/index.rst
index b6833c7bb474..e0a2c74e1043 100644
--- a/Documentation/virt/kvm/index.rst
+++ b/Documentation/virt/kvm/index.rst
@@ -8,25 +8,13 @@ KVM
    :maxdepth: 2
 
    api
-   amd-memory-encryption
-   cpuid
-   halt-polling
-   hypercalls
-   locking
-   mmu
-   msr
-   nested-vmx
-   ppc-pv
-   s390-diag
-   s390-pv
-   s390-pv-boot
-   timekeeping
-   vcpu-requests
-
-   review-checklist
-
-   arm/index
-
    devices/index
 
-   running-nested-guests
+   arm/index
+   s390/index
+   ppc-pv
+   x86/index
+
+   locking
+   vcpu-requests
+   review-checklist
diff --git a/Documentation/virt/kvm/s390/index.rst b/Documentation/virt/kvm/s390/index.rst
new file mode 100644
index 000000000000..605f488f0cc5
--- /dev/null
+++ b/Documentation/virt/kvm/s390/index.rst
@@ -0,0 +1,12 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+====================
+KVM for s390 systems
+====================
+
+.. toctree::
+   :maxdepth: 2
+
+   s390-diag
+   s390-pv
+   s390-pv-boot
diff --git a/Documentation/virt/kvm/s390-diag.rst b/Documentation/virt/kvm/s390/s390-diag.rst
similarity index 100%
rename from Documentation/virt/kvm/s390-diag.rst
rename to Documentation/virt/kvm/s390/s390-diag.rst
diff --git a/Documentation/virt/kvm/s390-pv-boot.rst b/Documentation/virt/kvm/s390/s390-pv-boot.rst
similarity index 100%
rename from Documentation/virt/kvm/s390-pv-boot.rst
rename to Documentation/virt/kvm/s390/s390-pv-boot.rst
diff --git a/Documentation/virt/kvm/s390-pv.rst b/Documentation/virt/kvm/s390/s390-pv.rst
similarity index 100%
rename from Documentation/virt/kvm/s390-pv.rst
rename to Documentation/virt/kvm/s390/s390-pv.rst
diff --git a/Documentation/virt/kvm/amd-memory-encryption.rst b/Documentation/virt/kvm/x86/amd-memory-encryption.rst
similarity index 100%
rename from Documentation/virt/kvm/amd-memory-encryption.rst
rename to Documentation/virt/kvm/x86/amd-memory-encryption.rst
diff --git a/Documentation/virt/kvm/cpuid.rst b/Documentation/virt/kvm/x86/cpuid.rst
similarity index 100%
rename from Documentation/virt/kvm/cpuid.rst
rename to Documentation/virt/kvm/x86/cpuid.rst
diff --git a/Documentation/virt/kvm/halt-polling.rst b/Documentation/virt/kvm/x86/halt-polling.rst
similarity index 100%
rename from Documentation/virt/kvm/halt-polling.rst
rename to Documentation/virt/kvm/x86/halt-polling.rst
diff --git a/Documentation/virt/kvm/hypercalls.rst b/Documentation/virt/kvm/x86/hypercalls.rst
similarity index 100%
rename from Documentation/virt/kvm/hypercalls.rst
rename to Documentation/virt/kvm/x86/hypercalls.rst
diff --git a/Documentation/virt/kvm/x86/index.rst b/Documentation/virt/kvm/x86/index.rst
new file mode 100644
index 000000000000..55ede8e070b6
--- /dev/null
+++ b/Documentation/virt/kvm/x86/index.rst
@@ -0,0 +1,18 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+===================
+KVM for x86 systems
+===================
+
+.. toctree::
+   :maxdepth: 2
+
+   amd-memory-encryption
+   cpuid
+   halt-polling
+   hypercalls
+   mmu
+   msr
+   nested-vmx
+   running-nested-guests
+   timekeeping
diff --git a/Documentation/virt/kvm/mmu.rst b/Documentation/virt/kvm/x86/mmu.rst
similarity index 100%
rename from Documentation/virt/kvm/mmu.rst
rename to Documentation/virt/kvm/x86/mmu.rst
diff --git a/Documentation/virt/kvm/msr.rst b/Documentation/virt/kvm/x86/msr.rst
similarity index 100%
rename from Documentation/virt/kvm/msr.rst
rename to Documentation/virt/kvm/x86/msr.rst
diff --git a/Documentation/virt/kvm/nested-vmx.rst b/Documentation/virt/kvm/x86/nested-vmx.rst
similarity index 100%
rename from Documentation/virt/kvm/nested-vmx.rst
rename to Documentation/virt/kvm/x86/nested-vmx.rst
diff --git a/Documentation/virt/kvm/running-nested-guests.rst b/Documentation/virt/kvm/x86/running-nested-guests.rst
similarity index 100%
rename from Documentation/virt/kvm/running-nested-guests.rst
rename to Documentation/virt/kvm/x86/running-nested-guests.rst
diff --git a/Documentation/virt/kvm/timekeeping.rst b/Documentation/virt/kvm/x86/timekeeping.rst
similarity index 100%
rename from Documentation/virt/kvm/timekeeping.rst
rename to Documentation/virt/kvm/x86/timekeeping.rst
-- 
2.35.1

[PATCH 2/3] Documentation: KVM: add virtual CPU errata documentation

[Paolo Bonzini]

Add a file to document all the different ways in which the virtual CPU
emulation is imperfect.  Include an example to show how to document
such errata.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 Documentation/virt/kvm/x86/errata.rst | 39 +++++++++++++++++++++++++++
 Documentation/virt/kvm/x86/index.rst  |  1 +
 2 files changed, 40 insertions(+)
 create mode 100644 Documentation/virt/kvm/x86/errata.rst

diff --git a/Documentation/virt/kvm/x86/errata.rst b/Documentation/virt/kvm/x86/errata.rst
new file mode 100644
index 000000000000..df394d34a836
--- /dev/null
+++ b/Documentation/virt/kvm/x86/errata.rst
@@ -0,0 +1,39 @@
+
+=======================================
+Known limitations of CPU virtualization
+=======================================
+
+Whenever perfect emulation of a CPU feature is impossible or too hard, KVM
+has to choose between not implementing the feature at all or introducing
+behavioral differences between virtual machines and bare metal systems.
+
+This file documents some of the known limitations that KVM has in
+virtualizing CPU features.
+
+x86
+===
+
+``KVM_GET_SUPPORTED_CPUID`` issues
+----------------------------------
+
+x87 features
+~~~~~~~~~~~~
+
+Unlike most other CPUID feature bits, CPUID[EAX=7,ECX=0]:EBX[6]
+(FDP_EXCPTN_ONLY) and CPUID[EAX=7,ECX=0]:EBX]13] (ZERO_FCS_FDS) are
+clear if the features are present and set if the features are not present.
+
+Clearing these bits in CPUID has no effect on the operation of the guest;
+if these bits are set on hardware, the features will not be present on
+any virtual machine that runs on that hardware.
+
+**Workaround:** It is recommended to always set these bits in guest CPUID.
+Note however that any software (e.g ``WIN87EM.DLL``) expecting these features
+to be present likely predates these CPUID feature bits, and therefore
+doesn't know to check for them anyway.
+
+Nested virtualization features
+------------------------------
+
+TBD
+
diff --git a/Documentation/virt/kvm/x86/index.rst b/Documentation/virt/kvm/x86/index.rst
index 55ede8e070b6..7ff588826b9f 100644
--- a/Documentation/virt/kvm/x86/index.rst
+++ b/Documentation/virt/kvm/x86/index.rst
@@ -9,6 +9,7 @@ KVM for x86 systems
 
    amd-memory-encryption
    cpuid
+   errata
    halt-polling
    hypercalls
    mmu
-- 
2.35.1

Re: [PATCH 2/3] Documentation: KVM: add virtual CPU errata documentation

[Jim Mattson]

On Tue, Mar 22, 2022 at 4:07 AM Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> Add a file to document all the different ways in which the virtual CPU
> emulation is imperfect.  Include an example to show how to document
> such errata.
This is fantastic. Thank you. Along these lines, I wonder if we should
come up with a mechanism a la IA32_ARCH_CAPABILITIES (or AMD's OSVW
MSRs) for declaring fixed errata.

For example, today, KVM ignores the guest-only and host-only bits in
PerfEvtSel MSRs, but I can't imagine we're going to do that forever.

Reviewed-by: Jim Mattson <jmattson@google.com>

Re: [PATCH 2/3] Documentation: KVM: add virtual CPU errata documentation

[Oliver Upton]

Hi Paolo,

On Tue, Mar 22, 2022 at 12:07:11PM +0100, Paolo Bonzini wrote:
> Add a file to document all the different ways in which the virtual CPU
> emulation is imperfect.  Include an example to show how to document
> such errata.
>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

+100, this is excellent!

Reviewed-by: Oliver Upton <oupton@google.com>

> ---
>  Documentation/virt/kvm/x86/errata.rst | 39 +++++++++++++++++++++++++++
>  Documentation/virt/kvm/x86/index.rst  |  1 +
>  2 files changed, 40 insertions(+)
>  create mode 100644 Documentation/virt/kvm/x86/errata.rst
> 
> diff --git a/Documentation/virt/kvm/x86/errata.rst b/Documentation/virt/kvm/x86/errata.rst
> new file mode 100644
> index 000000000000..df394d34a836
> --- /dev/null
> +++ b/Documentation/virt/kvm/x86/errata.rst
> @@ -0,0 +1,39 @@
> +
> +=======================================
> +Known limitations of CPU virtualization
> +=======================================

In an attempt for us KVM developers to save face, would it make
sense to rip the table for KVM_CAP_DISABLE_QUIRKS2 out of api.rst and
drop it here?

That way, we can discuss all of the known errata in KVM *and* mention
the chicken bits we provide to mitigate :)

--
Thanks,
Oliver

[PATCH 3/3] Documentation: KVM: add API issues section

[Paolo Bonzini]

Add a section to document all the different ways in which the KVM API sucks.

I am sure there are way more, give people a place to vent so that userspace
authors are aware.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 Documentation/virt/kvm/api.rst | 46 ++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 9f3172376ec3..8787fcd3b23f 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -7575,3 +7575,49 @@ The argument to KVM_ENABLE_CAP is also a bitmask, and must be a subset
 of the result of KVM_CHECK_EXTENSION.  KVM will forward to userspace
 the hypercalls whose corresponding bit is in the argument, and return
 ENOSYS for the others.
+
+9. Known KVM API problems
+=========================
+
+In some cases, KVM's API has some inconsistencies or common pitfalls
+that userspace need to be aware of.  This section details some of
+these issues.
+
+Most of them are architecture specific, so the section is split by
+architecture.
+
+9.1. x86
+--------
+
+``KVM_GET_SUPPORTED_CPUID`` issues
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In general, ``KVM_GET_SUPPORTED_CPUID`` is designed so that it is possible
+to take its result and pass it directly to ``KVM_SET_CPUID2``.  This section
+documents some cases in which that requires some care.
+
+Local APIC features
+~~~~~~~~~~~~~~~~~~~
+
+CPU[EAX=1]:ECX[21] (X2APIC) is reported by ``KVM_GET_SUPPORTED_CPUID``,
+but it can only be enabled if ``KVM_CREATE_IRQCHIP`` or
+``KVM_ENABLE_CAP(KVM_CAP_IRQCHIP_SPLIT)`` are used to enable in-kernel emulation of
+the local APIC.
+
+The same is true for the ``KVM_FEATURE_PV_UNHALT`` paravirtualized feature.
+
+CPU[EAX=1]:ECX[24] (TSC_DEADLINE) is not reported by ``KVM_GET_SUPPORTED_CPUID``.
+It can be enabled if ``KVM_CAP_TSC_DEADLINE_TIMER`` is present and the kernel
+has enabled in-kernel emulation of the local APIC.
+
+Obsolete ioctls and capabilities
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+KVM_CAP_DISABLE_QUIRKS does not let userspace know which quirks are actually
+available.  Use ``KVM_CHECK_EXTENSION(KVM_CAP_DISABLE_QUIRKS2)`` instead if
+available.
+
+Ordering of KVM_GET_*/KVM_SET_* ioctls
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+TBD
-- 
2.35.1

Re: [PATCH 3/3] Documentation: KVM: add API issues section

[Oliver Upton]

Hi Paolo,

On Tue, Mar 22, 2022 at 12:07:12PM +0100, Paolo Bonzini wrote:
> Add a section to document all the different ways in which the KVM API sucks.
> 
> I am sure there are way more, give people a place to vent so that userspace
> authors are aware.
> 
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Do you think we should vent about our mistakes inline with the
descriptions of the corresponding UAPI? One example that comes to mind
is ARM's CNTV_CVAL_EL0/CNTVCT_EL0 mixup, which is mentioned in 4.68
'KVM_SET_ONE_REG'. That, of course, doesn't cover the
previously-undocumented bits of UAPI that are problematic :)

If we go that route we likely should have a good format for documenting
the ugliness.

--
Thanks,
Oliver

Re: [PATCH 3/3] Documentation: KVM: add API issues section

[Paolo Bonzini]

On 3/23/22 19:16, Oliver Upton wrote:
> Do you think we should vent about our mistakes inline with the
> descriptions of the corresponding UAPI? One example that comes to mind
> is ARM's CNTV_CVAL_EL0/CNTVCT_EL0 mixup, which is mentioned in 4.68
> 'KVM_SET_ONE_REG'. That, of course, doesn't cover the
> previously-undocumented bits of UAPI that are problematic:)

It depends.  My intention was to use this document more for hidden 
interdependencies, in this case between KVM_GET_SUPPORTED_CPUID and 
KVM_CREATE_IRQCHIP, KVM_ENABLE_CAP(KVM_CAP_IRQCHIP_SPLIT), 
KVM_CAP_TSC_DEADLINE_TIMER.

Paolo

Re: [PATCH 3/3] Documentation: KVM: add API issues section

[Oliver Upton]

On Wed, Mar 23, 2022 at 10:32:50PM +0100, Paolo Bonzini wrote:
> On 3/23/22 19:16, Oliver Upton wrote:
> > Do you think we should vent about our mistakes inline with the
> > descriptions of the corresponding UAPI? One example that comes to mind
> > is ARM's CNTV_CVAL_EL0/CNTVCT_EL0 mixup, which is mentioned in 4.68
> > 'KVM_SET_ONE_REG'. That, of course, doesn't cover the
> > previously-undocumented bits of UAPI that are problematic:)
> 
> It depends.  My intention was to use this document more for hidden
> interdependencies, in this case between KVM_GET_SUPPORTED_CPUID and
> KVM_CREATE_IRQCHIP, KVM_ENABLE_CAP(KVM_CAP_IRQCHIP_SPLIT),
> KVM_CAP_TSC_DEADLINE_TIMER.

Ah good point. I agree bad cross interactions need to be called out
separately.

Thanks!

--
Oliver