portability issues

portability issues

[Jan Beulich]

Being in the process of trying to enable sysenter/syscall use from compat mode
guests and compat mode apps in native guests (for performance to a certain
degree, but more importantly - at least for the compat mode app case - to get
closer to native behavior, i.e. mid/long term requiring less kernel modifications)
I'm facing a few backwards compatibility issues that I'm not really sure how to
deal with:

1) Obviously I need to extend the guest_context structure (to store the
additional callback addresses), but there do not seem to be provisions to do
so without breaking the dom0 interface. I'm currently considering adding a
flag indicating use of the larger structure, but this certainly doesn't scale well
considering future additions. An alternative might be to add a single flag
covering all future additions, and using the first field past the current size to
store the overall or add-on size, so that the hypervisor has a way to know
how much of the structure to copy.

2) While the x86-32 hv can't support syscall and is unlikely to support sysenter,
save/restore/migration (which hopefully will work at least from 32-bit hv to
64-bit hv in the future) imposes an issue here in that native wouldn't need
these extra fields, but a compat mode guest would have to have a way to
store them (in compat mode format), implying that guest_context would also
need to be extended for 32-bits.

3) Currently, the 32-bit kernel check X86_FEATURE_SEP and the return
status of setting the sysenter hypercall, to detect its availability when run
in supervisor mode. This, however, is being done only on the boot CPU,
which works thanks to a quirk in how Xen handles the hypercall - MSRs for
all CPUs get set by this single call, which clearly doesn't match native
behavior (where in theory all CPUs could have distinct settings and have
to establish them as they come up). The problem with this is that if things
turn out to work as intended, X86_FEATURE_SEP will be seen enabled by
the guest when run on a 64-bit hv, and setting the sysenter callback will
also succeed, but existing guests will fail to set the callback on all CPUs.
I have to admit that I'm rather reluctant to add the same kind of quirk for
!supervisor_mode_kernel in the hypervisor's handling of CALLBACK_sysenter.

Thanks for suggestions/opinions,
Jan

Re: portability issues

[Keir Fraser]

(1) and (2): I want to kill off use of vcpu_guest_context in dom0 tools,
make hvm save/restore a generic state load/save interface, and define
extensible structures at that interface (pass a stream of state chunks back
and forth at the interface, each chunk having a size in its header, and so
increasing the size of a chunk allows it to be naturally appended to and
hence extended).

(3) Bump the XEN_INTERFACE_VERSION, rename current CALLBACKTYPE_sysenter to
a compat name, and introduce new define for CALLBACKTYPE_sysenter with
proper native semantics.

 -- Keir

On 21/6/07 09:10, "Jan Beulich" <jbeulich@novell.com> wrote:

> Being in the process of trying to enable sysenter/syscall use from compat mode
> guests and compat mode apps in native guests (for performance to a certain
> degree, but more importantly - at least for the compat mode app case - to get
> closer to native behavior, i.e. mid/long term requiring less kernel
> modifications)
> I'm facing a few backwards compatibility issues that I'm not really sure how
> to
> deal with:
> 
> 1) Obviously I need to extend the guest_context structure (to store the
> additional callback addresses), but there do not seem to be provisions to do
> so without breaking the dom0 interface. I'm currently considering adding a
> flag indicating use of the larger structure, but this certainly doesn't scale
> well
> considering future additions. An alternative might be to add a single flag
> covering all future additions, and using the first field past the current size
> to
> store the overall or add-on size, so that the hypervisor has a way to know
> how much of the structure to copy.
> 
> 2) While the x86-32 hv can't support syscall and is unlikely to support
> sysenter,
> save/restore/migration (which hopefully will work at least from 32-bit hv to
> 64-bit hv in the future) imposes an issue here in that native wouldn't need
> these extra fields, but a compat mode guest would have to have a way to
> store them (in compat mode format), implying that guest_context would also
> need to be extended for 32-bits.
> 
> 3) Currently, the 32-bit kernel check X86_FEATURE_SEP and the return
> status of setting the sysenter hypercall, to detect its availability when run
> in supervisor mode. This, however, is being done only on the boot CPU,
> which works thanks to a quirk in how Xen handles the hypercall - MSRs for
> all CPUs get set by this single call, which clearly doesn't match native
> behavior (where in theory all CPUs could have distinct settings and have
> to establish them as they come up). The problem with this is that if things
> turn out to work as intended, X86_FEATURE_SEP will be seen enabled by
> the guest when run on a 64-bit hv, and setting the sysenter callback will
> also succeed, but existing guests will fail to set the callback on all CPUs.
> I have to admit that I'm rather reluctant to add the same kind of quirk for
> !supervisor_mode_kernel in the hypervisor's handling of CALLBACK_sysenter.
> 
> Thanks for suggestions/opinions,
> Jan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: portability issues

[Jan Beulich]

>>> Keir Fraser <keir@xensource.com> 21.06.07 10:37 >>>
>(1) and (2): I want to kill off use of vcpu_guest_context in dom0 tools,
>make hvm save/restore a generic state load/save interface, and define
>extensible structures at that interface (pass a stream of state chunks back
>and forth at the interface, each chunk having a size in its header, and so
>increasing the size of a chunk allows it to be naturally appended to and
>hence extended).

I wasn't concerned about the tools interface. The real compatibility problem
is VCPUOP_initialize.

Jan

Re: portability issues

[Keir Fraser]

On 21/6/07 10:53, "Jan Beulich" <jbeulich@novell.com> wrote:

>> (1) and (2): I want to kill off use of vcpu_guest_context in dom0 tools,
>> make hvm save/restore a generic state load/save interface, and define
>> extensible structures at that interface (pass a stream of state chunks back
>> and forth at the interface, each chunk having a size in its header, and so
>> increasing the size of a chunk allows it to be naturally appended to and
>> hence extended).
> 
> I wasn't concerned about the tools interface. The real compatibility problem
> is VCPUOP_initialize.

I don't see the problem. The guest will not be able to initialise secondary
VCPU's sysenter/syscall state via this interface. So what?

 -- Keir

Re: portability issues

[Jan Beulich]

>>> Keir Fraser <keir@xensource.com> 21.06.07 11:58 >>>
>On 21/6/07 10:53, "Jan Beulich" <jbeulich@novell.com> wrote:
>
>>> (1) and (2): I want to kill off use of vcpu_guest_context in dom0 tools,
>>> make hvm save/restore a generic state load/save interface, and define
>>> extensible structures at that interface (pass a stream of state chunks back
>>> and forth at the interface, each chunk having a size in its header, and so
>>> increasing the size of a chunk allows it to be naturally appended to and
>>> hence extended).
>> 
>> I wasn't concerned about the tools interface. The real compatibility problem
>> is VCPUOP_initialize.
>
>I don't see the problem. The guest will not be able to initialise secondary
>VCPU's sysenter/syscall state via this interface. So what?

For (1), the guest will supply a too short guest_context structure, and
currently Xen has no way of detecting this. I was proposing two possible
solutions, neither of which seemed ideal to me.

For (2), I am just not certain whether there isn't an alternative not breaking
the interface for pure 32-bits.

Jan

Re: portability issues

[Keir Fraser]

On 21/6/07 11:16, "Jan Beulich" <jbeulich@novell.com> wrote:

>> I don't see the problem. The guest will not be able to initialise secondary
>> VCPU's sysenter/syscall state via this interface. So what?
> 
> For (1), the guest will supply a too short guest_context structure, and
> currently Xen has no way of detecting this. I was proposing two possible
> solutions, neither of which seemed ideal to me.
> 
> For (2), I am just not certain whether there isn't an alternative not breaking
> the interface for pure 32-bits.

I'm proposing we do not change the guest_context structure at all. Instead
the tools will get at this extra state via a generalised form of the current
hvm save/restore interface. The guest itself will always have to set the
callback via the callback_op hypercall.

 -- Keir