address space reorganization

address space reorganization

[Gerd Knorr]

  Hi,

On my devel machine xen comes up fine with PAE paging
enabled.  Well, it boots not that far yet, it stops at the
end of paging_init() right now.  The address space issues
need fixing now before I can attempt to boot domain 0 with
PAE ...

At the moment the xen virtual address space (top 64 MB)
looks like this:

  0xffc0  |  4 MB  |  ioremap area
  0xff80  |  4 MB  |  mapping cache
  0xff40  |  4 MB  |  per domain mapping (gdt, ...)
  0xff00  |  4 MB  |  shadow linear page tables
  0xfec0  |  4 MB  |  linear page tables
  0xfd40  | 24 MB  |  frame table
  0xfd00  |  4 MB  |  MPT (rw)
  0xfc40  | 12 MB  |  low mem, xen code, xen heap
  0xfc00  |  4 MB  |  MPT (ro)

For PAE we'll have to change:

  * linear page tables and linear shadow tables need 8 MB
    each (because pte size is doubled with PAE).
  * frame table needs to grow, the size depends on the
    amount of memory we are willing to support (total),
    for 16 GB it would be 96 MB.
  * MPT might need more space, depending on how much memory
    we are willing to support (per domain).  With a 4GB
    per-domain limit the current 4 MB size would be fine.
    [ side note: the shadow code seems to reuse the MPT
      address space for something else in some cases, not
      sure which implications this has ]
  * not sure about xen's heap.  What this is used for?
    Might we need more space here as well to support large
    amounts of memory?

If we touch the address space anyway we might fix some other
issues along the way.  Ian mentioned he wants to move the
ioremap area to the bottom.  I guess next to the ro MPT
table, so it's easy to grant domains read-only access to
ACPI tables?

Is it possible (and/or useful) to make the address layout
dynamic?  So the size of the frametable can be adjusted at
boot time depending on the amount of memory installed in the
machine?  That would imply the ro MPT doesn't have a fixed
address any more, not sure this is possible ...

In any case I'd try to make the memory layout as fixed as
possible, i.e. move the fixed size stuff to the top, below
the data structures which are not fixed-size, at the bottom
the ro MPT + ioremap area for r/o domain access, i.e.
something like this:

[ fixed size ]
    0xff00  | 16 MB  |  low mem, xen code, xen heap
    0xfec0  |  4 MB  |  mapping cache
    0xfe80  |  4 MB  |  per domain mapping (gdt, ...)

[ Hmm, debatable whenever make that fixed-size or not.
  It would waste some address space in the non-pae case,
  on the other hand the memory layout would be identical
  for both pae and non-pae. ]
    0xfe00  |  8 MB  |  shadow linear page tables
    0xfd80  |  8 MB  |  linear page tables
    0xfbc0  |  4 MB  |  MPT (rw)

[ not fixed size ]
    0xfc00  | 24 MB  |  frame table (larger for PAE ...)

[ r/o access for domains ]
    0xfb80  |  4 MB  |  MPT (ro)
    0xfb40  |  4 MB  |  ioremap area

Comments?  Anything else to consider when touching the
address layout anyway?

  Gerd

PS: my current patches are @ http://dl.bytesex.org/patches/xen/

Re: address space reorganization

[Keir Fraser]

On 13 Apr 2005, at 18:59, Gerd Knorr wrote:

> If we touch the address space anyway we might fix some other
> issues along the way.  Ian mentioned he wants to move the
> ioremap area to the bottom.  I guess next to the ro MPT
> table, so it's easy to grant domains read-only access to
> ACPI tables?

Ian was talking about the ioremap area in XenLinux, not in Xen. Doing 
this for XenLinux would be useful if we were to make HYPERVISOR_START a 
run-time, rather than compile-time, constant. However, I am not 
convinced that this is a very good idea (see my reasoning below).

That given, it is also not clear that reorg'ing the Xen address space 
is a worthwhile effort. Yes, things like the location of the Xen heap 
will be different on PAE vs. non-PAE builds, but it will still be a 
constant decided at compile time.  Reorg'ing is hopefully a pain we can 
do without for the time being. We should revisit these considerations 
if/when we want a single Xen binary that can do both PAE and non-PAE. 
But let's just get the basic support working first. :-)

> Is it possible (and/or useful) to make the address layout
> dynamic?  So the size of the frametable can be adjusted at
> boot time depending on the amount of memory installed in the
> machine?  That would imply the ro MPT doesn't have a fixed
> address any more, not sure this is possible ...

I do not think that we should have run-time selected 
HYPERVISOR_VIRT_START. This is because it will make it impossible to 
migrate a guest to a machine with more memory than the one on which it 
is currently executing.

The reasoning is: the new target machine will have a lower 
HYPERVISOR_VIRT_START, but the guest will have sized its lowmem area 
according to the available space on the original machine. It therefore 
cannot run  on the new target because its lowmem area simply will not 
fit.

I think we should just fix on two VIRT_START values: -64MB for non-PAE 
and something like -192MB for PAE (or whatever allows us to map up to 
16GB -- I think we will treat bigger memory configs than that as rare 
enough to ignore).

  -- Keir

Re: address space reorganization

[Keir Fraser]

On 13 Apr 2005, at 20:29, Keir Fraser wrote:

> I do not think that we should have run-time selected 
> HYPERVISOR_VIRT_START. This is because it will make it impossible to 
> migrate a guest to a machine with more memory than the one on which it 
> is currently executing.
>
> The reasoning is: the new target machine will have a lower 
> HYPERVISOR_VIRT_START, but the guest will have sized its lowmem area 
> according to the available space on the original machine. It therefore 
> cannot run  on the new target because its lowmem area simply will not 
> fit.
>
> I think we should just fix on two VIRT_START values: -64MB for non-PAE 
> and something like -192MB for PAE (or whatever allows us to map up to 
> 16GB -- I think we will treat bigger memory configs than that as rare 
> enough to ignore).

I chatted to Christian a bit about this and he changed my mind. There 
probably are some situations where a variable virt_start would be 
useful for us, although we still may not want to do it for an initial 
pae patch.

We need generally to think about how flexible we want to be in allowing 
migration between different machine configurations. Shoudl we require 
identical h/w specs, or allow differences in I/O devices, CPU and/or 
memory? We will already have to be careful about downgrading cpu specs 
when we migrate (e.g., Linux locks onto using multimedia instructions 
for software raid that are unavailable post-migrate). A pragmatic 
middleground may be that, if people want to migrate in a heterogeneous 
cluster, we require them to configure 'worst-case specs' up front when 
building a domain (e.g., lowest-spec cpu the domain should run on; 
biggest hypervisor address-space hole the domain should work with).

  -- Keir

Re: address space reorganization

[Jacob Gorm Hansen]

Keir Fraser wrote:

> We need generally to think about how flexible we want to be in allowing 
> migration between different machine configurations. Shoudl we require 
> identical h/w specs, or allow differences in I/O devices, CPU and/or 
> memory? We will already have to be careful about downgrading cpu specs 
> when we migrate (e.g., Linux locks onto using multimedia instructions 
> for software raid that are unavailable post-migrate).

Why not treat the functions that use special mm-instructions (like the 
software RAID code) as critical sections that cannot overlap with 
migration, and then have the guestOS re-calibrate its use of these 
features upon arrival?

[ insert standard plug of self-migration here :-) ]

Jacob

RE: address space reorganization

[Ian Pratt]

> > We need generally to think about how flexible we want to be in 
> > allowing migration between different machine 
> configurations. Shoudl we 
> > require identical h/w specs, or allow differences in I/O 
> devices, CPU 
> > and/or memory? We will already have to be careful about downgrading 
> > cpu specs when we migrate (e.g., Linux locks onto using multimedia 
> > instructions for software raid that are unavailable post-migrate).
> 
> Why not treat the functions that use special mm-instructions 
> (like the software RAID code) as critical sections that 
> cannot overlap with migration, and then have the guestOS 
> re-calibrate its use of these features upon arrival?

That works OK for the kernel, but you might have user space apps that
have adapted their behviour based on what the've found in /proc/cpuinfo.

A particularly nasty case is apps or libraries that go at 'cpuid'
directly, as we can't trap that instruction. I guess VMware have the
same problem, as I don't believe they translate ring 3 code.

As regards your proposed critical region, we already effectively do
this.  We don't recalibrate stuff after a migration yet though (some of
the tests are quite slow, so I'm not sure you'd want to do them all
anyhow).

Ian

Re: address space reorganization

[Jacob Gorm Hansen]

Ian Pratt wrote:

> That works OK for the kernel, but you might have user space apps that
> have adapted their behviour based on what the've found in /proc/cpuinfo.

A compromise then would be to lie to userspace and still recalibrate the 
kernel.

> A particularly nasty case is apps or libraries that go at 'cpuid'
> directly, as we can't trap that instruction. I guess VMware have the
> same problem, as I don't believe they translate ring 3 code.

Yeah, nothing we can do there really, except tell people not to :-(

> As regards your proposed critical region, we already effectively do
> this.  We don't recalibrate stuff after a migration yet though (some of
> the tests are quite slow, so I'm not sure you'd want to do them all
> anyhow).

Could perhaps do them in advance, or during the pre-copy phase as they 
are likely to stay constant for the lifetime of the machine, but that 
demands that the receiving side knows what you are looking for.
My system uploads a bootstrapper to the target VM in advance, so I could 
have this info ready when the rest of the OS arrives.

Jacob

Re: address space reorganization

[Kip Macy]

> > That works OK for the kernel, but you might have user space apps that
> > have adapted their behviour based on what the've found in /proc/cpuinfo.
> 
> A compromise then would be to lie to userspace and still recalibrate the
> kernel.

If you're saying what I think you are - I don't think people who have
customized their apps for SSE2 would like that.

You'll either have to have xend propagate a bitmask of the cpuid
capabilities so that xfrd won't migrate it or you tell users that "the
behaviour is undefined" if they use migration in a heterogeneous
cluster.

      -Kip

Re: address space reorganization

[Jacob Gorm Hansen]

Kip Macy wrote:
>>>That works OK for the kernel, but you might have user space apps that
>>>have adapted their behviour based on what the've found in /proc/cpuinfo.
>>
>>A compromise then would be to lie to userspace and still recalibrate the
>>kernel.
> 
> 
> If you're saying what I think you are - I don't think people who have
> customized their apps for SSE2 would like that.

This is what Keir was suggesting, but still giving the kernel a chance
to use SSE[12] for its own purposes. It is not perfect, the only perfect
solution would alert user-space apps about the impending change, so that
they can react to or veto the migration.

Jacob

Re: address space reorganization

[Adam Heath]

On Wed, 13 Apr 2005, Jacob Gorm Hansen wrote:

> Kip Macy wrote:
> >>>That works OK for the kernel, but you might have user space apps that
> >>>have adapted their behviour based on what the've found in /proc/cpuinfo.
> >>
> >>A compromise then would be to lie to userspace and still recalibrate the
> >>kernel.
> >
> >
> > If you're saying what I think you are - I don't think people who have
> > customized their apps for SSE2 would like that.
>
> This is what Keir was suggesting, but still giving the kernel a chance
> to use SSE[12] for its own purposes. It is not perfect, the only perfect
> solution would alert user-space apps about the impending change, so that
> they can react to or veto the migration.

How does beowulf handle this situation?

Re: address space reorganization

[Rik van Riel]

On Wed, 13 Apr 2005, Kip Macy wrote:

> You'll either have to have xend propagate a bitmask of the cpuid
> capabilities so that xfrd won't migrate it or you tell users that "the
> behaviour is undefined" if they use migration in a heterogeneous
> cluster.

I'd prefer the latter.   It really is a configuration problem,
and not something I'd want the software to solve for me.

People whose applications need SSE2 will install CPUs that
have those instructions.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: address space reorganization

[Kip Macy]

> People whose applications need SSE2 will install CPUs that
> have those instructions.
> 
One would hope, but just because a customer has lots of money to spend
on hardware doesn't mean that he is rowing with both oars. This is a
supportability issue. The xen{source} folks would do themselves a
favor by trapping a guest's use of unsupported instructions and
logging it. That would make it easy enough to track down if a
customer's apps stop working when using migration.


           -Kip

Re: address space reorganization

[Rik van Riel]

On Thu, 14 Apr 2005, Kip Macy wrote:

> > People whose applications need SSE2 will install CPUs that
> > have those instructions.
> > 
> One would hope, but just because a customer has lots of money to spend
> on hardware doesn't mean that he is rowing with both oars. This is a
> supportability issue. The xen{source} folks would do themselves a
> favor by trapping a guest's use of unsupported instructions and
> logging it. That would make it easy enough to track down if a
> customer's apps stop working when using migration.

Good idea, trapping unsupported instructions and printing out
the category the instruction belongs to (eg. SSE2) will make
things a lot easier to track.  I like this idea a lot...

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

RE: address space reorganization

[Ian Pratt]

 
> > One would hope, but just because a customer has lots of 
> money to spend 
> > on hardware doesn't mean that he is rowing with both oars. 
> This is a 
> > supportability issue. The xen{source} folks would do themselves a 
> > favor by trapping a guest's use of unsupported instructions and 
> > logging it. That would make it easy enough to track down if a 
> > customer's apps stop working when using migration.
> 
> Good idea, trapping unsupported instructions and printing out 
> the category the instruction belongs to (eg. SSE2) will make 
> things a lot easier to track.  I like this idea a lot...

Yep, although we can't trap the cpuid, we can trap the use of e.g. SSE2.

We have to be a bit careful though, to prevent DoS of the Xen console.
We'd need to rate limit such messages. Patches welcome :-)

Ian

Re: address space reorganization

[Chris Wedgwood]

On Thu, Apr 14, 2005 at 08:03:18PM +0100, Ian Pratt wrote:

> Yep, although we can't trap the cpuid, we can trap the use of
> e.g. SSE2.

and emulate them or just warn?  for the former it's going to be slow
as hell, and i would almost argue you could stop the domain doing this
with a warning...