address space reorganization

[Gerd Knorr <kraxel@bytesex.org>]

  Hi,

On my devel machine xen comes up fine with PAE paging
enabled.  Well, it boots not that far yet, it stops at the
end of paging_init() right now.  The address space issues
need fixing now before I can attempt to boot domain 0 with
PAE ...

At the moment the xen virtual address space (top 64 MB)
looks like this:

  0xffc0  |  4 MB  |  ioremap area
  0xff80  |  4 MB  |  mapping cache
  0xff40  |  4 MB  |  per domain mapping (gdt, ...)
  0xff00  |  4 MB  |  shadow linear page tables
  0xfec0  |  4 MB  |  linear page tables
  0xfd40  | 24 MB  |  frame table
  0xfd00  |  4 MB  |  MPT (rw)
  0xfc40  | 12 MB  |  low mem, xen code, xen heap
  0xfc00  |  4 MB  |  MPT (ro)

For PAE we'll have to change:

  * linear page tables and linear shadow tables need 8 MB
    each (because pte size is doubled with PAE).
  * frame table needs to grow, the size depends on the
    amount of memory we are willing to support (total),
    for 16 GB it would be 96 MB.
  * MPT might need more space, depending on how much memory
    we are willing to support (per domain).  With a 4GB
    per-domain limit the current 4 MB size would be fine.
    [ side note: the shadow code seems to reuse the MPT
      address space for something else in some cases, not
      sure which implications this has ]
  * not sure about xen's heap.  What this is used for?
    Might we need more space here as well to support large
    amounts of memory?

If we touch the address space anyway we might fix some other
issues along the way.  Ian mentioned he wants to move the
ioremap area to the bottom.  I guess next to the ro MPT
table, so it's easy to grant domains read-only access to
ACPI tables?

Is it possible (and/or useful) to make the address layout
dynamic?  So the size of the frametable can be adjusted at
boot time depending on the amount of memory installed in the
machine?  That would imply the ro MPT doesn't have a fixed
address any more, not sure this is possible ...

In any case I'd try to make the memory layout as fixed as
possible, i.e. move the fixed size stuff to the top, below
the data structures which are not fixed-size, at the bottom
the ro MPT + ioremap area for r/o domain access, i.e.
something like this:

[ fixed size ]
    0xff00  | 16 MB  |  low mem, xen code, xen heap
    0xfec0  |  4 MB  |  mapping cache
    0xfe80  |  4 MB  |  per domain mapping (gdt, ...)

[ Hmm, debatable whenever make that fixed-size or not.
  It would waste some address space in the non-pae case,
  on the other hand the memory layout would be identical
  for both pae and non-pae. ]
    0xfe00  |  8 MB  |  shadow linear page tables
    0xfd80  |  8 MB  |  linear page tables
    0xfbc0  |  4 MB  |  MPT (rw)

[ not fixed size ]
    0xfc00  | 24 MB  |  frame table (larger for PAE ...)

[ r/o access for domains ]
    0xfb80  |  4 MB  |  MPT (ro)
    0xfb40  |  4 MB  |  ioremap area

Comments?  Anything else to consider when touching the
address layout anyway?

  Gerd

PS: my current patches are @ http://dl.bytesex.org/patches/xen/