Setting up page directories and tables

Setting up page directories and tables

[Dave Pacheco]

[-- Attachment #1.1: Type: text/plain, Size: 1669 bytes --]

I'm working on a small OS which is similar in many ways to the Mini-OS (and
I've borrowed much code for the time being). One important difference is
that it supports multiple processes, each with its own address space. I'm
having problems setting this up under Xen.

This OS is designed for 32-bit x86 (no PAE), so I intend to use a two-level
paging system. Each process has a page directory - one page with entries
which point to page tables, whose entries describe actual pages. As far as I
can tell, this is what's done in the Mini-OS source, with page directories
referred to as L1 page tables, and what I'm calling page tables referred to
as L2 page tables.

So when creating a process, I do the following:
    allocate a page P for the process's page directory
    fill P with zeros
    update the existing page table mapping for P (previously set up by what
is essentially the build_pagetable() code from Mini-OS) to give it L1_PROT &
~_PAGE_RW
        (using HYPERVISOR_mmu_update(...))
    pin the page (using HYPERVISOR_mmuext_op({MMUEXT_PIN_L1_TABLE, ...}...))

All of these steps succeed (the hypercall returns 0, and the argument
indicating how many updates succeeded is 1 in both cases). But later, when
switching into the context of this process, I try to update the user base
pointer with

 HYPERVISOR_mmuext_op({MMUEXT_NEW_USER_BASEPTR, mfn of P}...).

This call fails (returns -1, and indicates that 0 operations succeeded). Why
might that be? As I've established, the page is zero'd out (so it's not
invalid), and it seems to be mapped properly (not writable, pinned L1 page
table). So why would this operation fail?

Thanks in advance,
Dave Pacheco

[-- Attachment #1.2: Type: text/html, Size: 1872 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Setting up page directories and tables

[Keir Fraser]

[-- Attachment #1.1: Type: text/plain, Size: 466 bytes --]




On 27/4/07 21:11, "Dave Pacheco" <dap@cs.brown.edu> wrote:

> All of these steps succeed (the hypercall returns 0, and the argument
> indicating how many updates succeeded is 1 in both cases). But later, when
> switching into the context of this process, I try to update the user base
> pointer with 
> 
>  HYPERVISOR_mmuext_op({MMUEXT_NEW_USER_BASEPTR, mfn of P}...).

Only x86/64 has a separate USER_BASEPTR. You want to use just
MMUEXT_NEW_BASEPTR.

 -- Keir


[-- Attachment #1.2: Type: text/html, Size: 942 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Setting up page directories and tables

[Dave Pacheco]

[-- Attachment #1.1: Type: text/plain, Size: 982 bytes --]

On 4/27/07, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
>
>  On 27/4/07 21:11, "Dave Pacheco" <dap@cs.brown.edu> wrote:
>
> All of these steps succeed (the hypercall returns 0, and the argument
> indicating how many updates succeeded is 1 in both cases). But later, when
> switching into the context of this process, I try to update the user base
> pointer with
>
>  HYPERVISOR_mmuext_op({MMUEXT_NEW_USER_BASEPTR, mfn of P}...).
>
>
> Only x86/64 has a separate USER_BASEPTR. You want to use just
> MMUEXT_NEW_BASEPTR.
>

When I use that, the domain immediately crashes. I would have expected a
page fault, since the page tables aren't actually set up yet, but I don't
even get that. (It's possible that it faulted again trying to load the fault
handler.)

/var/log/xen/xend.log doesn't say much useful, other than that the domain
crashed. Is there a way to tell why a domain crashed (i.e. executing an
illegal instruction, or triple fault, or something like that)?

Thanks,
Dave

[-- Attachment #1.2: Type: text/html, Size: 1745 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Setting up page directories and tables

[Jeremy Fitzhardinge]

Dave Pacheco wrote:
>
> /var/log/xen/xend.log doesn't say much useful, other than that the
> domain crashed. Is there a way to tell why a domain crashed (i.e.
> executing an illegal instruction, or triple fault, or something like
> that)?

If you compile Xen with debug mode turned on, and then you get useful
stuff on the console.  It's generally fairly cryptic, but enough to give
you some hints.  Also, you can use gdbserver-xen to poke around in your
domain so long as it has valid pagetables.  It's useful for stepping up
to your cr3 load and looking at the state of things.

I see from your description that you're pinning L1 pages.  That's
generally not how Xen linux kernels do it these days; they just pin the
whole pagetable from L2 down (in your non-PAE case).  Since pinning from
top-down is the same as loading cr3 from a validity-checking
perspective, it's useful to see if Xen likes your pagetables without
actually trashing them (if you load a bad cr3, you end up in limbo with
no useful pagetables, and Xen can't even print a useful stack backtrace
as it destroys your domain).

Also, remember that all pages in your new pagetable have to be mapped RO
in *all* other pagetables.  The easiest way to achieve that is to make
all pagetables share the same L1 pages in your "kernel" address space,
assuming you have such a thing.  Otherwise when you're allocating a new
page for a pagetable, you have to walk through all other pagetables and
be sure to update any mappings for that page to RO.

    J

Re: Setting up page directories and tables

[Dave Pacheco]

[-- Attachment #1.1: Type: text/plain, Size: 1060 bytes --]

On 4/27/07, Jeremy Fitzhardinge <jeremy@xensource.com> wrote:

> I see from your description that you're pinning L1 pages.  That's
> generally not how Xen linux kernels do it these days; they just pin the
> whole pagetable from L2 down (in your non-PAE case).  Since pinning from
> top-down is the same as loading cr3 from a validity-checking
> perspective, it's useful to see if Xen likes your pagetables without
> actually trashing them (if you load a bad cr3, you end up in limbo with
> no useful pagetables, and Xen can't even print a useful stack backtrace
> as it destroys your domain).


I don't really understand what you mean here. To "pin the whole pagetable
from L2 down" - does that mean pinning the L2 pagetable page itself and then
all of the L1 pages, or just the L2 pagetable page?

What exactly does pinning do? Does it simply ensure that the page won't be
paged out and that Xen knows it's a page table? If that's the case, it seems
like you still want to pin L1 pages so that you know that Xen is okay with
your L1 page table.

Thanks,
Dave

[-- Attachment #1.2: Type: text/html, Size: 1431 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Setting up page directories and tables

[Jeremy Fitzhardinge]

Dave Pacheco wrote:
> I don't really understand what you mean here. To "pin the whole
> pagetable from L2 down" - does that mean pinning the L2 pagetable page
> itself and then all of the L1 pages, or just the L2 pagetable page?
>
> What exactly does pinning do? Does it simply ensure that the page
> won't be paged out and that Xen knows it's a page table? If that's the
> case, it seems like you still want to pin L1 pages so that you know
> that Xen is okay with your L1 page table.

"Pinning" is a bit of a misnomer.  What it means is that the pinned page
is always treated as part of an active pagetable (ie, pointed to by
%cr3), so all the normal rules apply: it must be RO, and all updates
must go through Xen and be validated.  This means that Xen doesn't need
to revalidate it when it does become an active pagetable.

By definition, if you pin a pagetable page, you also pin all the other
pagetable pages it refers to.  So if you pin the L2 (L3 for PAE) root
page, then you're pinning the entire pagetable.  You could pin the leaf
L1 pages (ie the ones actually containing pte entries), but there's not
much point.

    J

Re: Setting up page directories and tables

[Mark Williamson]

On Tuesday 01 May 2007, Dave Pacheco wrote:
> On 4/27/07, Jeremy Fitzhardinge <jeremy@xensource.com> wrote:
> > I see from your description that you're pinning L1 pages.  That's
> > generally not how Xen linux kernels do it these days; they just pin the
> > whole pagetable from L2 down (in your non-PAE case).  Since pinning from
> > top-down is the same as loading cr3 from a validity-checking
> > perspective, it's useful to see if Xen likes your pagetables without
> > actually trashing them (if you load a bad cr3, you end up in limbo with
> > no useful pagetables, and Xen can't even print a useful stack backtrace
> > as it destroys your domain).
>
> I don't really understand what you mean here. To "pin the whole pagetable
> from L2 down" - does that mean pinning the L2 pagetable page itself and
> then all of the L1 pages, or just the L2 pagetable page?

L2 is the top level of the hierarchy, so if you pin it all the L1 (leaf) 
pagetables will get validated in the process.

> What exactly does pinning do? Does it simply ensure that the page won't be
> paged out and that Xen knows it's a page table? If that's the case, it
> seems like you still want to pin L1 pages so that you know that Xen is okay
> with your L1 page table.

Pinning gets Xen to validate the pagetable.  It can subsequently be reloaded 
without validation until it's unpinned (otherwise Xen would need to validate 
it each time it was loaded into CR3).  It basically does this by validating 
the pagetable, then bumping the refcounts so that the pages can't be used in 
inappropriate ways (e.g. won't let them be mapped RW because that would allow 
malicious guests to break out of their memory).

Pinning an L2 table which references L1 tables will cause the L1 tables to be 
validated as part of this process (ensuring that they don't contain any 
inappropriate mappings).

Cheers,
Mark


-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

Re: Setting up page directories and tables

[Mark Williamson]

> This OS is designed for 32-bit x86 (no PAE), so I intend to use a two-level
> paging system. Each process has a page directory - one page with entries
> which point to page tables, whose entries describe actual pages. As far as
> I can tell, this is what's done in the Mini-OS source, with page
> directories referred to as L1 page tables, and what I'm calling page tables
> referred to as L2 page tables.

I think you've got this backwards: in Xen code, L1 page tables are the leaf 
nodes, L2 are their parents, L3 are their parents, etc etc.

Cheers,
Mark

> So when creating a process, I do the following:
>     allocate a page P for the process's page directory
>     fill P with zeros
>     update the existing page table mapping for P (previously set up by what
> is essentially the build_pagetable() code from Mini-OS) to give it L1_PROT
> & ~_PAGE_RW
>         (using HYPERVISOR_mmu_update(...))
>     pin the page (using HYPERVISOR_mmuext_op({MMUEXT_PIN_L1_TABLE,
> ...}...))
>
> All of these steps succeed (the hypercall returns 0, and the argument
> indicating how many updates succeeded is 1 in both cases). But later, when
> switching into the context of this process, I try to update the user base
> pointer with
>
>  HYPERVISOR_mmuext_op({MMUEXT_NEW_USER_BASEPTR, mfn of P}...).
>
> This call fails (returns -1, and indicates that 0 operations succeeded).
> Why might that be? As I've established, the page is zero'd out (so it's not
> invalid), and it seems to be mapped properly (not writable, pinned L1 page
> table). So why would this operation fail?
>
> Thanks in advance,
> Dave Pacheco



-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!