Re: [PATCH v3] docs/virt/kvm: Document configuring and running nested guests

[Kashyap Chamarthy <kchamart@redhat.com>]

On Tue, May 05, 2020 at 02:02:58PM +0200, Paolo Bonzini wrote:
> On 05/05/20 13:28, Kashyap Chamarthy wrote:

[...]

> > +Limitations on Linux kernel versions older than 5.3 (x86)
> > +---------------------------------------------------------
> > +
> > +On Linux kernel versions older than 5.3, once an L1 guest has started an
> > +L2 guest, the L1 guest would no longer capable of being migrated, saved,
> > +or loaded (refer to QEMU documentation on "save"/"load") until the L2
> > +guest shuts down.
> > +
> > +Attempting to migrate or save-and-load an L1 guest while an L2 guest is
> > +running will result in undefined behavior.  You might see a ``kernel
> > +BUG!`` entry in ``dmesg``, a kernel 'oops', or an outright kernel panic.
> > +Such a migrated or loaded L1 guest can no longer be considered stable or
> > +secure, and must be restarted.
> > +
> > +Migrating an L1 guest merely configured to support nesting, while not
> > +actually running L2 guests, is expected to function normally.
> > +Live-migrating an L2 guest from one L1 guest to another is also expected
> > +to succeed.
> > +
> 
> This is a bit optimistic, as AMD is not supported yet.  Please review
> the following incremental patch:
> 
> diff --git a/Documentation/virt/kvm/running-nested-guests.rst b/Documentation/virt/kvm/running-nested-guests.rst
> --- a/Documentation/virt/kvm/running-nested-guests.rst
> +++ b/Documentation/virt/kvm/running-nested-guests.rst
> @@ -182,11 +182,23 @@ Enabling "nested" (s390x)
>  Live migration with nested KVM
>  ------------------------------
>  
> -The below live migration scenarios should work as of Linux kernel 5.3
> -and QEMU 4.2.0 for x86; for s390x, even older versions might work.
> -In all the below cases, L1 exposes ``/dev/kvm`` in it, i.e. the L2 guest
> -is a "KVM-accelerated guest", not a "plain emulated guest" (as done by
> -QEMU's TCG).
> +Migrating an L1 guest, with a  *live* nested guest in it, to another
> +bare metal host, works as of Linux kernel 5.3 and QEMU 4.2.0 for
> +Intel x86 systems, and even on older versions for s390x.
> +
> +On AMD systems, once an L1 guest has started an L2 guest, the L1 guest
> +should no longer be migrated or saved (refer to QEMU documentation on
> +"savevm"/"loadvm") until the L2 guest shuts down.  Attempting to migrate
> +or save-and-load an L1 guest while an L2 guest is running will result in
> +undefined behavior.  You might see a ``kernel BUG!`` entry in ``dmesg``, a
> +kernel 'oops', or an outright kernel panic.  Such a migrated or loaded L1
> +guest can no longer be considered stable or secure, and must be restarted.
> +Migrating an L1 guest merely configured to support nesting, while not
> +actually running L2 guests, is expected to function normally even on AMD
> +systems but may fail once guests are started.
> +
> +Migrating an L2 guest is expected to succeed, so all the following
> +scenarios should work even on AMD systems:
>  
>  - Migrating a nested guest (L2) to another L1 guest on the *same* bare
>    metal host.
> @@ -194,30 +206,7 @@ QEMU's TCG).
>  - Migrating a nested guest (L2) to another L1 guest on a *different*
>    bare metal host.
>  
> -- Migrating an L1 guest, with an *offline* nested guest in it, to
> -  another bare metal host.
> -
> -- Migrating an L1 guest, with a  *live* nested guest in it, to another
> -  bare metal host.
> -
> -Limitations on Linux kernel versions older than 5.3 (x86)
> ----------------------------------------------------------
> -
> -On Linux kernel versions older than 5.3, once an L1 guest has started an
> -L2 guest, the L1 guest would no longer capable of being migrated, saved,
> -or loaded (refer to QEMU documentation on "save"/"load") until the L2
> -guest shuts down.
> -
> -Attempting to migrate or save-and-load an L1 guest while an L2 guest is
> -running will result in undefined behavior.  You might see a ``kernel
> -BUG!`` entry in ``dmesg``, a kernel 'oops', or an outright kernel panic.
> -Such a migrated or loaded L1 guest can no longer be considered stable or
> -secure, and must be restarted.
> -
> -Migrating an L1 guest merely configured to support nesting, while not
> -actually running L2 guests, is expected to function normally.
> -Live-migrating an L2 guest from one L1 guest to another is also expected
> -to succeed.
> +- Migrating a nested guest (L2) to a bare metal host.
>  
>  Reporting bugs from nested setups
>  -----------------------------------

Thanks for the important corrections, Paolo.  Your `diff` reads well to
me.  FWIW: 

    Reviewed-by: Kashyap Chamarthy <kchamart@redhat.com> 

-- 
/kashyap