Re: [PATCH] KVM: Document mmu

[Marcelo Tosatti <mtosatti@redhat.com>]

On Wed, Apr 21, 2010 at 04:09:21PM +0300, Avi Kivity wrote:
> Signed-off-by: Avi Kivity <avi@redhat.com>
> ---
>  Documentation/kvm/mmu.txt |  296 +++++++++++++++++++++++++++++++++++++++++++++
>  1 files changed, 296 insertions(+), 0 deletions(-)
>  create mode 100644 Documentation/kvm/mmu.txt
> 
> diff --git a/Documentation/kvm/mmu.txt b/Documentation/kvm/mmu.txt
> new file mode 100644
> index 0000000..176f834
> --- /dev/null
> +++ b/Documentation/kvm/mmu.txt
> @@ -0,0 +1,296 @@
> +The x86 kvm shadow mmu
> +======================
> +
> +The mmu (in arch/x86/kvm, files mmu.[ch] and paging_tmpl.h) is responsible
> +for presenting a standard x86 mmu to the guest, while translating guest
> +physical addresses to host physical addresses.
> +
> +The mmu code attempts to satisfy the following requirements:
> +
> +- correctness: the guest should not be able to determine that it is running
> +               on an emulated mmu except for timing (we attempt to comply
> +               with the specification, not emulate the characteristics of
> +               a particular implementation such as tlb size)
> +- security:    the guest must not be able to touch host memory not assigned
> +               to it
> +- performance: minimize the performance penalty imposed by the mmu
> +- scaling:     need to scale to large memory and large vcpu guests
> +- hardware:    support the full range of x86 virtualization hardware
> +- integration: Linux memory management code must be in control of guest memory
> +               so that swapping, page migration, page merging, transparent
> +               hugepages, and similar features work without change
> +- dirty tracking: report writes to guest memory to enable live migration
> +               and framebuffer-based displays
> +- footprint:   keep the amount of pinned kernel memory low (most memory
> +               should be shrinkable)
> +- reliablity:  avoid multipage or GFP_ATOMIC allocations
> +
> +Acronyms
> +========
> +
> +pfn   host page frame number
> +hpa   host physical address
> +hva   host virtual address
> +gfn   guest frame number
> +gpa   guest page address

guest physical address

> +gva   guest virtual address
> +ngpa  nested guest physical address
> +ngva  nested guest virtual address
> +pte   page table entry (used also to refer generically to paging structure
> +      entries)
> +gpte  guest pte (referring to gfns)
> +spte  shadow pte (referring to pfns)
> +tdp   two dimensional paging (vendor neutral term for NPT and EPT)
> +
> +Virtual and real hardware supported
> +===================================
> +
> +The mmu supports first-generation mmu hardware, which allows an atomic switch
> +of the current paging mode and cr3 during guest entry, as well as
> +two-dimensional paging (AMD's NPT and Intel's EPT).  The emulated hardware
> +it exposes is the traditional 2/3/4 level x86 mmu, with support for global
> +pages, pae, pse, pse36, cr0.wp, and 1GB pages.  Work is in progress to support
> +exposing NPT capable hardware on NPT capable hosts.
> +
> +Translation
> +===========
> +
> +The primary job of the mmu is to program the processor's mmu to translate
> +addresses for the guest.  Different translations are required at different
> +times:
> +
> +- when guest paging is disabled, we translate guest physical addresses to
> +  host physical addresses (gpa->hpa)
> +- when guest paging is enabled, we translate guest virtual addresses, to
> +  guest virtual addresses, to host physical addresses (gva->gpa->hpa)
> +- when the guest launches a guest of its own, we translate nested guest
> +  virtual addresses, to nested guest physical addresses, to guest physical
> +  addresses, to host physical addresses (ngva->ngpa->gpa->hpa)
> +
> +The primary challenge is to encode between 1 and 3 translations into hardware
> +that support only 1 (traditional) and 2 (tdp) translations.  When the
> +number of required translations matches the hardware, the mmu operates in
> +direct mode; otherwise it operates in shadow mode (see below).
> +
> +Memory
> +======
> +
> +Guest memory (gpa) is part of user address space of the process that is using
> +kvm.  Userspace defines the translation between guest addresses and user
> +addresses (gpa->gva); 

gpa->hva

> note that two gpas may alias to the same gva, but not
> +vice versa.
> +
> +These gvas may be backed using any method available to the host: anonymous
> +memory, file backed memory, and device memory.  Memory might be paged by the
> +host at any time.
> +
> +Events
> +======
> +
> +The mmu is driven by events, some from the guest, some from the host.
> +
> +Guest generated events:
> +- writes to control registers (especially cr3)
> +- invlpg/invlpga instruction execution
> +- access to missing or protected translations
> +
> +Host generated events:
> +- changes in the gpa->hpa translation (either through gpa->hva changes or
> +  through hva->hpa changes)
> +- memory pressure (the shrinker)
> +
> +Shadow pages
> +============
> +
> +The principal data structure is the shadow page, 'struct kvm_mmu_page'.  A
> +shadow page contains 512 sptes, which can be either leaf or nonleaf sptes.  A
> +shadow page may contain a mix of leaf and nonleaf sptes.
> +
> +A nonleaf spte allows the hardware mmu to reach the leaf pages and
> +is not related to a translation directly.  It points to other shadow pages.
> +
> +A leaf spte corresponds to either one or two translations encoded into
> +one paging structure entry.  These are always the lowest level of the
> +translation stack, with an optional higher level translations left to NPT/EPT.
> +Leaf ptes point at guest pages.
> +
> +The following table shows translations encoded by leaf ptes, with higher-level
> +translations in parentheses:
> +
> + Non-nested guests:
> +  nonpaging:     gpa->hpa
> +  paging:        gva->gpa->hpa
> +  paging, tdp:   (gva->)gpa->hpa
> + Nested guests:
> +  non-tdp:       ngva->gpa->hpa  (*)
> +  tdp:           (ngva->)ngpa->gpa->hpa
> +
> +(*) the guest hypervisor will encode the ngva->gpa translation into its page
> +    tables if npt is not present
> +
> +Shadow pages contain the following information:
> +  role.level:
> +    The level in the shadow paging hierarchy that this shadow page belongs to.
> +    1=4k sptes, 2=2M sptes, 3=1G sptes, etc.
> +  role.direct:
> +    If set, leaf sptes reachable from this page are for a linear range.
> +    Examples include real mode translation, large guest pages backed by small
> +    host pages, and gpa->hpa translations when NPT or EPT is active.
> +    The linear range starts at (gfn << PAGE_SHIFT) and its size is determined
> +    by role.level (2MB for first level, 1GB for second level, 0.5TB for third
> +    level, 256TB for fourth level)
> +    If clear, this page corresponds to a guest page table denoted by the gfn
> +    field.
> +  role.quadrant:
> +    When role.cr4_pae=0, the guest uses 32-bit gptes while the host uses 64-bit
> +    sptes.  That means a guest page table contains more ptes than the host,
> +    so multiple shadow pages are needed to shadow one guest page.
> +    For first-level shadow pages, role.quadrant can be 0 or 1 and denotes the
> +    first or second 512-gpte block in the guest page table.  For second-level
> +    page tables, each 32-bit gpte is converted to two 64-bit sptes
> +    (since each first-level guest page is shadowed by two first-level
> +    shadow pages) so role.quadrant takes values in the range 0..3.  Each
> +    quadrant maps 1GB virtual address space.
> +  role.access:
> +    Inherited guest access permissions in the form uwx.  Note execute
> +    permission is positive, not negative.
> +  role.invalid:
> +    The page is invalid and should not be used.  It is a root page that is
> +    currently pinned (by a cpu hardware register pointing to it); one it is

once

Looks good otherwise. Perhaps add a pointer to Joerg's NPT slides,
although they're AMD specific.