Shadow Code Reorganization

Shadow Code Reorganization

[Michael Vrable]

I'm trying to understand some of the changes that were made to the
shadow page table code a couple weeks ago to add support for 64-bit
modes: changesets f261f14b9781f949f95a328433eff187c677a70b and
0bcfd66a431ebfc70fc068a134e684568ac02966, committed on July 11.

I'm trying to make sense of what seems to be some code duplication.  In
xen/arch/x86, there are three files related to shadow mode: shadow.c,
shadow32.c, and shadow_public.c.  In my x86 build, only shadow32.c is
built.  shadow.c seems to contain a lot of similar code.  The comments
at the top of the files are also confusing; shadow32.c claims that it is
shadow.c, and shadow.c claims that it is shadow_64.c.

The Makefile is not very illuminating; for an x86_64 build, shadow32.o
is not built (not surprising), otherwise, shadow.o, shadow_public.o, and
shadow_xxx.o are skipped (the first two names sound rather generic to
me, and shadow_xxx.o looks like part of a not-quite-finished change).

Can anyone explain how this code is supposed to be structured, what is
supposed to end up in each of the files, and whether the files are still
in need of some renaming (say, shadow.c -> shadow64.c)?

--Michael Vrable

RE: Shadow Code Reorganization

[Ian Pratt]

> I'm trying to make sense of what seems to be some code 
> duplication.  In xen/arch/x86, there are three files related 
> to shadow mode: shadow.c, shadow32.c, and shadow_public.c.  
> In my x86 build, only shadow32.c is built.  shadow.c seems to 
> contain a lot of similar code.  The comments at the top of 
> the files are also confusing; shadow32.c claims that it is 
> shadow.c, and shadow.c claims that it is shadow_64.c.

The new file called shadow.c can be compiled for both 32 and 64 bit, but
is currently only used for 64bit by default.

The shadow32.c file (formerly shadow.c) still exists because it's had
more testing than the new file, and we want both in the tree until we're
convinced there's been no regression between the two. We'll then delete
shadow32 and compile shadow.c in 32 bit mode. Base any new development
on the new shadow.c

Make sense?

Best,
Ian

Re: Shadow Code Reorganization

[Gerd Knorr]

"Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk> writes:

> The shadow32.c file (formerly shadow.c) still exists because it's had
> more testing than the new file, and we want both in the tree until we're
> convinced there's been no regression between the two. We'll then delete
> shadow32 and compile shadow.c in 32 bit mode. Base any new development
> on the new shadow.c

While we are at it and I'll go work on shadow mode as well soon some
followup questions ;)

 * "normal" shadow mode is almost identical to shadow mode off,
   i.e. the guest does machine<=>phys translations.  The guest hasn't
   to mark pages containing page tables r/o because xen can do that in
   the shadow tables.

 * In "translated" shadow mode the guest kernel handles a linear
   physical address space starting at addr zero and xen does the
   machine<=>phys translations when copying guest tables into the
   shadow page tables.

 * "external" is translated shadow mode + separated address space,
   i.e. no hypervisor window in the address space, used for vt.

Correct so far?

Enabling/switching shadow modes works by dropping all shadow tables,
start with a zero-ed toplevel page directory (almost, except
hypervisor window) and update the shadow pagetable tree as faults are
coming in.  Correct?

But it's not clear to me how the transition between "normal" and
"translated" shadow mode works.  Does that need support by the guest
os?

  Gerd

-- 
panic("it works"); /* avoid being flooded with debug messages */

RE: Shadow Code Reorganization

[Ian Pratt]

> While we are at it and I'll go work on shadow mode as well 
> soon some followup questions ;)
> 
>  * "normal" shadow mode is almost identical to shadow mode off,
>    i.e. the guest does machine<=>phys translations.  The guest hasn't
>    to mark pages containing page tables r/o because xen can do that in
>    the shadow tables.

Depends what you mean by "normal". The mode that we need for migration
of a paravirtualized guest is log_dity but with refcounts in the *guest*
pages i.e. SHM_refcounts is not set. The guest still has to mark
pagetable pages RO and use the normal paravirtualized interface.
 
>  * In "translated" shadow mode the guest kernel handles a linear
>    physical address space starting at addr zero and xen does the
>    machine<=>phys translations when copying guest tables into the
>    shadow page tables.

Yes. SHM_refcounts is typically set.

> 
>  * "external" is translated shadow mode + separated address space,
>    i.e. no hypervisor window in the address space, used for vt.

Yes, this is used by VT-x.
 
> Enabling/switching shadow modes works by dropping all shadow 
> tables, start with a zero-ed toplevel page directory (almost, 
> except hypervisor window) and update the shadow pagetable 
> tree as faults are coming in.  Correct?

Only in modes where SHM_refcounts is set. For so-called "lightweight
shadow mode" (as used by migration) we continue using the guest
refcounts.

> But it's not clear to me how the transition between "normal" 
> and "translated" shadow mode works.  Does that need support 
> by the guest os?

Guests typically don't make this transition -- it's currently a guest
compile time option whether to use the paravirtualized or translated
shadow mode interface. This isn't strictly necessary.

Ian

Re: Shadow Code Reorganization

[Gerd Knorr]

  Hi,

> Depends what you mean by "normal". The mode that we need for migration
> of a paravirtualized guest is log_dity but with refcounts in the *guest*
> pages i.e. SHM_refcounts is not set.

Ah, *that* the log_dirty option is good for.  Looked like some
performance measurement counter on a first quick look ...

> > But it's not clear to me how the transition between "normal" 
> > and "translated" shadow mode works.  Does that need support 
> > by the guest os?
> 
> Guests typically don't make this transition -- it's currently a guest
> compile time option whether to use the paravirtualized or translated
> shadow mode interface. This isn't strictly necessary.

Ok, more general:  How does xen and/or the guest os handle the
changes in machine<=>phys mapping as they happen on
suspend/resume and migration because the guest gets a different
set of machine pages?  Especially the guest page table updates?

  Gerd

-- 
panic("it works"); /* avoid being flooded with debug messages */

RE: Shadow Code Reorganization

[Ian Pratt]

> Ok, more general:  How does xen and/or the guest os handle 
> the changes in machine<=>phys mapping as they happen on 
> suspend/resume and migration because the guest gets a 
> different set of machine pages?  Especially the guest page 
> table updates?

The guest is notified that it should suspend itself. It should then get
its self into a 'safe' state i.e. ensure that no VCPUs are in the midst
of pagetable updates [*], create a suspend record, and then call the
suspend sched op hypercall. [upon resume, the sched op hypercall will
appear to return]

xc_linux_save is then responsible for reading the state out of the
domain, converting any pages containing pagetables (as indiciated by the
page type field) into 'canonical form' i.e. changing mfns into pfns.  At
restore time, xc_linux_restore will rewrite them to the new mfns.

The suspend record contains a page containing the list of mfns that make
up the p2m table for the guest. For PAE mode we'll need to extend this
table to cover larger phys memory sizes as it's only 1024 entries at
present. 

Ian

[*] We're currently mising code to get other VCPUs to safety. We could
hotunplug them, but its probably easiest just to have the VCPU that
executes the suspend thread count the others into a tasklet. The
save/restore code can trivially be extended to save multiple contexts.