Getting the XSAVE size from userspace

Getting the XSAVE size from userspace

[Razvan Cojocaru]

Hello,

I need to get the XSAVE size from userspace. The easiest way seems to be
to use the XEN_DOMCTL_getvcpuextstate hypercall, but that hypercall is
not public / there's no xenctrl.h wrapper for it.

There's also struct hvm_hw_cpu_xsave, which I can get to, but it doesn't
have a size member:

542 /*
543  * The save area of XSAVE/XRSTOR.
544  */
545
546 struct hvm_hw_cpu_xsave {
547     uint64_t xfeature_mask;        /* Ignored */
548     uint64_t xcr0;                 /* Updated by XSETBV */
549     uint64_t xcr0_accum;           /* Updated by XSETBV */
550     struct {
551         struct { char x[512]; } fpu_sse;
552
553         struct {
554             uint64_t xstate_bv;         /* Updated by XRSTOR */
555             uint64_t reserved[7];
556         } xsave_hdr;                    /* The 64-byte header */
557
558         struct { char x[0]; } ymm;    /* YMM */
559     } save_area;
560 };

I see that in the hypervisor code the length is computed by using the
HVM_CPU_XSAVE_SIZE() macro:

2126 #define HVM_CPU_XSAVE_SIZE(xcr0) (offsetof(struct hvm_hw_cpu_xsave, \
2127                                            save_area) + \
2128                                   xstate_ctxt_size(xcr0))

where:

256 static unsigned int _xstate_ctxt_size(u64 xcr0)
257 {
258     u64 act_xcr0 = get_xcr0();
259     u32 eax, ebx = 0, ecx, edx;
260     bool_t ok = set_xcr0(xcr0);
261
262     ASSERT(ok);
263     cpuid_count(XSTATE_CPUID, 0, &eax, &ebx, &ecx, &edx);
264     ASSERT(ebx <= ecx);
265     ok = set_xcr0(act_xcr0);
266     ASSERT(ok);
267
268     return ebx;
269 }
270
271 /* Fastpath for common xstate size requests, avoiding reloads of
xcr0. */
272 unsigned int xstate_ctxt_size(u64 xcr0)
273 {
274     if ( xcr0 == xfeature_mask )
275         return xsave_cntxt_size;
276
277     if ( xcr0 == 0 )
278         return 0;
279
280     return _xstate_ctxt_size(xcr0);
281 }

But that doesn't seem to translate cleanly to userspace code.

I had hoped that I would be able to get this with no custom Xen patches,
is there a simpler way I'm not aware of to get to this information? And
if there isn't, would you prefer a libxc patch that exposes
XEN_DOMCTL_getvcpuextstate, or one that adds a size member to struct
hvm_hw_cpu_xsave (I'd guess the latter)?


Thanks,
Razvan

Re: Getting the XSAVE size from userspace

[Jan Beulich]

>>> On 05.11.15 at 10:52, <rcojocaru@bitdefender.com> wrote:
> I need to get the XSAVE size from userspace. The easiest way seems to be
> to use the XEN_DOMCTL_getvcpuextstate hypercall, but that hypercall is
> not public / there's no xenctrl.h wrapper for it.

Before going into any detail of the rest of your mail - any reason you
can't just consult CPUID output?

Jan

Re: Getting the XSAVE size from userspace

[Razvan Cojocaru]

On 11/05/2015 12:42 PM, Jan Beulich wrote:
>>>> On 05.11.15 at 10:52, <rcojocaru@bitdefender.com> wrote:
>> I need to get the XSAVE size from userspace. The easiest way seems to be
>> to use the XEN_DOMCTL_getvcpuextstate hypercall, but that hypercall is
>> not public / there's no xenctrl.h wrapper for it.
> 
> Before going into any detail of the rest of your mail - any reason you
> can't just consult CPUID output?

That's because the userspace application doesn't live in dom0, but in a
dedicated privileged domain, and I'm unsure if a CPUID issued there
yields the same results as a CPUID issued in dom0. So I thought the
safest way is to get the information directly from the hypervisor. Is
this assumption incorrect?


Thanks,
Razvan

Re: Getting the XSAVE size from userspace

[Jan Beulich]

>>> On 05.11.15 at 11:47, <rcojocaru@bitdefender.com> wrote:
> On 11/05/2015 12:42 PM, Jan Beulich wrote:
>>>>> On 05.11.15 at 10:52, <rcojocaru@bitdefender.com> wrote:
>>> I need to get the XSAVE size from userspace. The easiest way seems to be
>>> to use the XEN_DOMCTL_getvcpuextstate hypercall, but that hypercall is
>>> not public / there's no xenctrl.h wrapper for it.
>> 
>> Before going into any detail of the rest of your mail - any reason you
>> can't just consult CPUID output?
> 
> That's because the userspace application doesn't live in dom0, but in a
> dedicated privileged domain, and I'm unsure if a CPUID issued there
> yields the same results as a CPUID issued in dom0. So I thought the
> safest way is to get the information directly from the hypervisor. Is
> this assumption incorrect?

See my other reply (to Andrew) - as long as there's no problem with
using the maximum possible size, I don't see why you couldn't use
just CPUID.

Jan

Re: Getting the XSAVE size from userspace

[Andrew Cooper]

On 05/11/15 10:47, Razvan Cojocaru wrote:
> On 11/05/2015 12:42 PM, Jan Beulich wrote:
>>>>> On 05.11.15 at 10:52, <rcojocaru@bitdefender.com> wrote:
>>> I need to get the XSAVE size from userspace. The easiest way seems to be
>>> to use the XEN_DOMCTL_getvcpuextstate hypercall, but that hypercall is
>>> not public / there's no xenctrl.h wrapper for it.
>> Before going into any detail of the rest of your mail - any reason you
>> can't just consult CPUID output?
> That's because the userspace application doesn't live in dom0, but in a
> dedicated privileged domain, and I'm unsure if a CPUID issued there
> yields the same results as a CPUID issued in dom0. So I thought the
> safest way is to get the information directly from the hypervisor. Is
> this assumption incorrect?

What purpose are you wanting the information for?

Using cpuid (should) get you the information concerning your domain,
which is liable to be different to what another domain might see.

Currently, the information available through the domain cpuid policy is
inaccurate, and *not* migration safe.  I am working on fixing this as
part 2 of my cpuid levelling fixes.

~Andrew

Re: Getting the XSAVE size from userspace

[Andrew Cooper]

On 05/11/15 10:42, Jan Beulich wrote:
>>>> On 05.11.15 at 10:52, <rcojocaru@bitdefender.com> wrote:
>> I need to get the XSAVE size from userspace. The easiest way seems to be
>> to use the XEN_DOMCTL_getvcpuextstate hypercall, but that hypercall is
>> not public / there's no xenctrl.h wrapper for it.
> Before going into any detail of the rest of your mail - any reason you
> can't just consult CPUID output?

It depends on precisely what you want.

CPUID.0xD[0].ecx gives you the maximum xsave area on this processor
CPUID.0xD[0].ebx gives you the current size for the value in xcr0, but
that is not very useful from userspace.

~Andrew

Re: Getting the XSAVE size from userspace

[Jan Beulich]

>>> On 05.11.15 at 11:49, <andrew.cooper3@citrix.com> wrote:
> On 05/11/15 10:42, Jan Beulich wrote:
>>>>> On 05.11.15 at 10:52, <rcojocaru@bitdefender.com> wrote:
>>> I need to get the XSAVE size from userspace. The easiest way seems to be
>>> to use the XEN_DOMCTL_getvcpuextstate hypercall, but that hypercall is
>>> not public / there's no xenctrl.h wrapper for it.
>> Before going into any detail of the rest of your mail - any reason you
>> can't just consult CPUID output?
> 
> It depends on precisely what you want.
> 
> CPUID.0xD[0].ecx gives you the maximum xsave area on this processor
> CPUID.0xD[0].ebx gives you the current size for the value in xcr0, but
> that is not very useful from userspace.

Why would the maximum size not be sufficient for most (all?) user
mode purposes?

Jan

Re: Getting the XSAVE size from userspace

[Andrei LUTAS]

On 11/5/2015 12:51 PM, Jan Beulich wrote:
>>>> On 05.11.15 at 11:49, <andrew.cooper3@citrix.com> wrote:
>> On 05/11/15 10:42, Jan Beulich wrote:
>>>>>> On 05.11.15 at 10:52, <rcojocaru@bitdefender.com> wrote:
>>>> I need to get the XSAVE size from userspace. The easiest way seems to be
>>>> to use the XEN_DOMCTL_getvcpuextstate hypercall, but that hypercall is
>>>> not public / there's no xenctrl.h wrapper for it.
>>> Before going into any detail of the rest of your mail - any reason you
>>> can't just consult CPUID output?
>> It depends on precisely what you want.
>>
>> CPUID.0xD[0].ecx gives you the maximum xsave area on this processor
>> CPUID.0xD[0].ebx gives you the current size for the value in xcr0, but
>> that is not very useful from userspace.
> Why would the maximum size not be sufficient for most (all?) user
> mode purposes?
>
> Jan
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>
Hello,

The use-case is the following: whenever an EPT violation is triggered 
inside a monitored VM, the introspection logic needs to know how many 
bytes were accessed (read/written). This is done by inspecting the 
faulting instruction and directly inferring the size, which is not 
straight-forward for XSAVE/XRSTOR family. Using the maximum possible 
size is wrong, as in any given moment the OS may or may not desire to 
XSAVE/XRSTOR the entire state (and thinking that the instruction tries 
to access more than it actually does may yield undesired effects). 
Therefore, the size needed for the currently enabled features of the 
monitored guest is required instead. Normally, it could be done by 
running CPUID with eax = 0xD and ecx = i, where i >= 2 and XCR0[i] is 1 
(XCR0 belongs to the monitored guest), but I am unsure if using CPUID 
this way would be safe/desired: will Xen expose the same CPUID features, 
for XSAVE related functionality, on all VMs? (using XCPUID with eax = 
0xD and ecx = 0 would give us the needed size for the SVA, and like I 
said, using the maximum size would not be safe, even if it's the same 
across all VMs on a given host). Also, I'm unsure how this would get 
along with migration...

Thanks,
Andrei.

Re: Getting the XSAVE size from userspace

[Andrew Cooper]

On 05/11/15 11:35, Andrei LUTAS wrote:
> On 11/5/2015 12:51 PM, Jan Beulich wrote:
>>>>> On 05.11.15 at 11:49, <andrew.cooper3@citrix.com> wrote:
>>> On 05/11/15 10:42, Jan Beulich wrote:
>>>>>>> On 05.11.15 at 10:52, <rcojocaru@bitdefender.com> wrote:
>>>>> I need to get the XSAVE size from userspace. The easiest way seems
>>>>> to be
>>>>> to use the XEN_DOMCTL_getvcpuextstate hypercall, but that
>>>>> hypercall is
>>>>> not public / there's no xenctrl.h wrapper for it.
>>>> Before going into any detail of the rest of your mail - any reason you
>>>> can't just consult CPUID output?
>>> It depends on precisely what you want.
>>>
>>> CPUID.0xD[0].ecx gives you the maximum xsave area on this processor
>>> CPUID.0xD[0].ebx gives you the current size for the value in xcr0, but
>>> that is not very useful from userspace.
>> Why would the maximum size not be sufficient for most (all?) user
>> mode purposes?
>>
>> Jan
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>>
> Hello,
>
> The use-case is the following: whenever an EPT violation is triggered
> inside a monitored VM, the introspection logic needs to know how many
> bytes were accessed (read/written). This is done by inspecting the
> faulting instruction and directly inferring the size, which is not
> straight-forward for XSAVE/XRSTOR family. Using the maximum possible
> size is wrong, as in any given moment the OS may or may not desire to
> XSAVE/XRSTOR the entire state (and thinking that the instruction tries
> to access more than it actually does may yield undesired effects).
> Therefore, the size needed for the currently enabled features of the
> monitored guest is required instead. Normally, it could be done by
> running CPUID with eax = 0xD and ecx = i, where i >= 2 and XCR0[i] is
> 1 (XCR0 belongs to the monitored guest), but I am unsure if using
> CPUID this way would be safe/desired: will Xen expose the same CPUID
> features, for XSAVE related functionality, on all VMs? (using XCPUID
> with eax = 0xD and ecx = 0 would give us the needed size for the SVA,
> and like I said, using the maximum size would not be safe, even if
> it's the same across all VMs on a given host). Also, I'm unsure how
> this would get along with migration...

Hmm yes - there is no way to do this currently.

Xen's CPUID handling for xsave related things is broken in levelling and
migration scenarios, which is why it is *still* disabled by default in
XenServer.

I am working on fixing it, and will take this usecase into account
(although I think I had already included enough for this usecase to work).

At the point of the xsave/xrestor trap, you need to know xcr0 and be
able to perfom a cpuid instruction in the context of a target domain, to
make use of 0xD[0].ebx to get the "current size based on xcr0".

~Andrew

Re: Getting the XSAVE size from userspace

[Razvan Cojocaru]

On 11/05/2015 01:44 PM, Andrew Cooper wrote:
> On 05/11/15 11:35, Andrei LUTAS wrote:
>> The use-case is the following: whenever an EPT violation is triggered
>> inside a monitored VM, the introspection logic needs to know how many
>> bytes were accessed (read/written). This is done by inspecting the
>> faulting instruction and directly inferring the size, which is not
>> straight-forward for XSAVE/XRSTOR family. Using the maximum possible
>> size is wrong, as in any given moment the OS may or may not desire to
>> XSAVE/XRSTOR the entire state (and thinking that the instruction tries
>> to access more than it actually does may yield undesired effects).
>> Therefore, the size needed for the currently enabled features of the
>> monitored guest is required instead. Normally, it could be done by
>> running CPUID with eax = 0xD and ecx = i, where i >= 2 and XCR0[i] is
>> 1 (XCR0 belongs to the monitored guest), but I am unsure if using
>> CPUID this way would be safe/desired: will Xen expose the same CPUID
>> features, for XSAVE related functionality, on all VMs? (using XCPUID
>> with eax = 0xD and ecx = 0 would give us the needed size for the SVA,
>> and like I said, using the maximum size would not be safe, even if
>> it's the same across all VMs on a given host). Also, I'm unsure how
>> this would get along with migration...
> 
> Hmm yes - there is no way to do this currently.
> 
> Xen's CPUID handling for xsave related things is broken in levelling and
> migration scenarios, which is why it is *still* disabled by default in
> XenServer.
> 
> I am working on fixing it, and will take this usecase into account
> (although I think I had already included enough for this usecase to work).
> 
> At the point of the xsave/xrestor trap, you need to know xcr0 and be
> able to perfom a cpuid instruction in the context of a target domain, to
> make use of 0xD[0].ebx to get the "current size based on xcr0".

So then the closest thing to what we need would be to add a size field
to struct hvm_hw_cpu_xsave, and just assign the size variable to it in
hvm_save_cpu_xsave_states (migration aside)?

2130 static int hvm_save_cpu_xsave_states(struct domain *d,
hvm_domain_context_t *h)
2131 {
2132     struct vcpu *v;
2133     struct hvm_hw_cpu_xsave *ctxt;
2134
2135     if ( !cpu_has_xsave )
2136         return 0;   /* do nothing */
2137
2138     for_each_vcpu ( d, v )
2139     {
2140         unsigned int size = HVM_CPU_XSAVE_SIZE(v->arch.xcr0_accum);
2141
2142         if ( !xsave_enabled(v) )
2143             continue;
2144         if ( _hvm_init_entry(h, CPU_XSAVE_CODE, v->vcpu_id, size) )
2145             return 1;
2146         ctxt = (struct hvm_hw_cpu_xsave *)&h->data[h->cur];
2147         h->cur += size;
2148
2149         ctxt->xfeature_mask = xfeature_mask;
2150         ctxt->xcr0 = v->arch.xcr0;
2151         ctxt->xcr0_accum = v->arch.xcr0_accum;
2152         memcpy(&ctxt->save_area, v->arch.xsave_area,
2153                size - offsetof(struct hvm_hw_cpu_xsave, save_area));
2154     }
2155
2156     return 0;
2157 }


Thanks,
Razvan

Re: Getting the XSAVE size from userspace

[Andrew Cooper]

On 05/11/15 12:26, Razvan Cojocaru wrote:
> On 11/05/2015 01:44 PM, Andrew Cooper wrote:
>> On 05/11/15 11:35, Andrei LUTAS wrote:
>>> The use-case is the following: whenever an EPT violation is triggered
>>> inside a monitored VM, the introspection logic needs to know how many
>>> bytes were accessed (read/written). This is done by inspecting the
>>> faulting instruction and directly inferring the size, which is not
>>> straight-forward for XSAVE/XRSTOR family. Using the maximum possible
>>> size is wrong, as in any given moment the OS may or may not desire to
>>> XSAVE/XRSTOR the entire state (and thinking that the instruction tries
>>> to access more than it actually does may yield undesired effects).
>>> Therefore, the size needed for the currently enabled features of the
>>> monitored guest is required instead. Normally, it could be done by
>>> running CPUID with eax = 0xD and ecx = i, where i >= 2 and XCR0[i] is
>>> 1 (XCR0 belongs to the monitored guest), but I am unsure if using
>>> CPUID this way would be safe/desired: will Xen expose the same CPUID
>>> features, for XSAVE related functionality, on all VMs? (using XCPUID
>>> with eax = 0xD and ecx = 0 would give us the needed size for the SVA,
>>> and like I said, using the maximum size would not be safe, even if
>>> it's the same across all VMs on a given host). Also, I'm unsure how
>>> this would get along with migration...
>> Hmm yes - there is no way to do this currently.
>>
>> Xen's CPUID handling for xsave related things is broken in levelling and
>> migration scenarios, which is why it is *still* disabled by default in
>> XenServer.
>>
>> I am working on fixing it, and will take this usecase into account
>> (although I think I had already included enough for this usecase to work).
>>
>> At the point of the xsave/xrestor trap, you need to know xcr0 and be
>> able to perfom a cpuid instruction in the context of a target domain, to
>> make use of 0xD[0].ebx to get the "current size based on xcr0".
> So then the closest thing to what we need would be to add a size field
> to struct hvm_hw_cpu_xsave, and just assign the size variable to it in
> hvm_save_cpu_xsave_states (migration aside)?
>
> 2130 static int hvm_save_cpu_xsave_states(struct domain *d,
> hvm_domain_context_t *h)
> 2131 {
> 2132     struct vcpu *v;
> 2133     struct hvm_hw_cpu_xsave *ctxt;
> 2134
> 2135     if ( !cpu_has_xsave )
> 2136         return 0;   /* do nothing */
> 2137
> 2138     for_each_vcpu ( d, v )
> 2139     {
> 2140         unsigned int size = HVM_CPU_XSAVE_SIZE(v->arch.xcr0_accum);
> 2141
> 2142         if ( !xsave_enabled(v) )
> 2143             continue;
> 2144         if ( _hvm_init_entry(h, CPU_XSAVE_CODE, v->vcpu_id, size) )
> 2145             return 1;
> 2146         ctxt = (struct hvm_hw_cpu_xsave *)&h->data[h->cur];
> 2147         h->cur += size;
> 2148
> 2149         ctxt->xfeature_mask = xfeature_mask;
> 2150         ctxt->xcr0 = v->arch.xcr0;
> 2151         ctxt->xcr0_accum = v->arch.xcr0_accum;
> 2152         memcpy(&ctxt->save_area, v->arch.xsave_area,
> 2153                size - offsetof(struct hvm_hw_cpu_xsave, save_area));
> 2154     }
> 2155
> 2156     return 0;
> 2157 }

I don't see any difference between this pasted code and the current
hvm_save_cpu_xsave_states().  What have you changed?

You can't use this size value, and it is the accumulated xcr0 over the
life of the VM, not the xcr0 in use at the time of the intercepted
instruction.

You also can't blindly modify the ctxt structure, or you will break
migration.

The xcr0 -> size mapping is static, and won't change going forwards. 
Your best bet is just to query each one and stash all the results.

~Andrew

Re: Getting the XSAVE size from userspace

[Razvan Cojocaru]

On 11/05/2015 04:05 PM, Andrew Cooper wrote:
> On 05/11/15 12:26, Razvan Cojocaru wrote:
>> On 11/05/2015 01:44 PM, Andrew Cooper wrote:
>>> On 05/11/15 11:35, Andrei LUTAS wrote:
>>>> The use-case is the following: whenever an EPT violation is triggered
>>>> inside a monitored VM, the introspection logic needs to know how many
>>>> bytes were accessed (read/written). This is done by inspecting the
>>>> faulting instruction and directly inferring the size, which is not
>>>> straight-forward for XSAVE/XRSTOR family. Using the maximum possible
>>>> size is wrong, as in any given moment the OS may or may not desire to
>>>> XSAVE/XRSTOR the entire state (and thinking that the instruction tries
>>>> to access more than it actually does may yield undesired effects).
>>>> Therefore, the size needed for the currently enabled features of the
>>>> monitored guest is required instead. Normally, it could be done by
>>>> running CPUID with eax = 0xD and ecx = i, where i >= 2 and XCR0[i] is
>>>> 1 (XCR0 belongs to the monitored guest), but I am unsure if using
>>>> CPUID this way would be safe/desired: will Xen expose the same CPUID
>>>> features, for XSAVE related functionality, on all VMs? (using XCPUID
>>>> with eax = 0xD and ecx = 0 would give us the needed size for the SVA,
>>>> and like I said, using the maximum size would not be safe, even if
>>>> it's the same across all VMs on a given host). Also, I'm unsure how
>>>> this would get along with migration...
>>> Hmm yes - there is no way to do this currently.
>>>
>>> Xen's CPUID handling for xsave related things is broken in levelling and
>>> migration scenarios, which is why it is *still* disabled by default in
>>> XenServer.
>>>
>>> I am working on fixing it, and will take this usecase into account
>>> (although I think I had already included enough for this usecase to work).
>>>
>>> At the point of the xsave/xrestor trap, you need to know xcr0 and be
>>> able to perfom a cpuid instruction in the context of a target domain, to
>>> make use of 0xD[0].ebx to get the "current size based on xcr0".
>> So then the closest thing to what we need would be to add a size field
>> to struct hvm_hw_cpu_xsave, and just assign the size variable to it in
>> hvm_save_cpu_xsave_states (migration aside)?
>>
>> 2130 static int hvm_save_cpu_xsave_states(struct domain *d,
>> hvm_domain_context_t *h)
>> 2131 {
>> 2132     struct vcpu *v;
>> 2133     struct hvm_hw_cpu_xsave *ctxt;
>> 2134
>> 2135     if ( !cpu_has_xsave )
>> 2136         return 0;   /* do nothing */
>> 2137
>> 2138     for_each_vcpu ( d, v )
>> 2139     {
>> 2140         unsigned int size = HVM_CPU_XSAVE_SIZE(v->arch.xcr0_accum);
>> 2141
>> 2142         if ( !xsave_enabled(v) )
>> 2143             continue;
>> 2144         if ( _hvm_init_entry(h, CPU_XSAVE_CODE, v->vcpu_id, size) )
>> 2145             return 1;
>> 2146         ctxt = (struct hvm_hw_cpu_xsave *)&h->data[h->cur];
>> 2147         h->cur += size;
>> 2148
>> 2149         ctxt->xfeature_mask = xfeature_mask;
>> 2150         ctxt->xcr0 = v->arch.xcr0;
>> 2151         ctxt->xcr0_accum = v->arch.xcr0_accum;
>> 2152         memcpy(&ctxt->save_area, v->arch.xsave_area,
>> 2153                size - offsetof(struct hvm_hw_cpu_xsave, save_area));
>> 2154     }
>> 2155
>> 2156     return 0;
>> 2157 }
> 
> I don't see any difference between this pasted code and the current
> hvm_save_cpu_xsave_states().  What have you changed?

I haven't changed anything, I was just pointing out what code I'm
referring to (which size variable I'm talking about), sorry for not
being as clear as possible.

> You can't use this size value, and it is the accumulated xcr0 over the
> life of the VM, not the xcr0 in use at the time of the intercepted
> instruction.

OK.

> You also can't blindly modify the ctxt structure, or you will break
> migration.

Well, yes, not blindly, that assumes that something like a patch for
mainline is agreed upon, or that migration is disabled for guests that
need this, and so on.

> The xcr0 -> size mapping is static, and won't change going forwards. 
> Your best bet is just to query each one and stash all the results.

OK.


Thanks,
Razvan