[HYBRID]: XEN_EMULATE_PREFIX in user process

[HYBRID]: XEN_EMULATE_PREFIX in user process

[Mukesh Rathor]

Hi,

Booting latest upstream pv linux as dom0, I noticed dm_mapper, user
level process, is running with XEN_EMULATE_PREFIX cpuid. I am
wondering if the prefix is statically put there, or how its put there?

For hybrid, the hvm container traps cpuid, so we don't need to prefix
cpuid. Less code if I don't have to worry about trapping the invalid
op. There seems places where the cpuid is run without the xen prefix in
user mode.

BTW, for cpuid vmexit, if kernel mode I call pv_cpuid, if user mode, I
just return cpuid values, as that's what would happen without the HVM
container.

thanks,
Mukesh

Re: [HYBRID]: XEN_EMULATE_PREFIX in user process

[Konrad Rzeszutek Wilk]

On Thu, Jun 28, 2012 at 06:00:07PM -0700, Mukesh Rathor wrote:
> Hi,
> 
> Booting latest upstream pv linux as dom0, I noticed dm_mapper, user
> level process, is running with XEN_EMULATE_PREFIX cpuid. I am
> wondering if the prefix is statically put there, or how its put there?

That is impressive. It should be only be in the kernel via the pvops
cpuid call which does this:

 252         asm(XEN_EMULATE_PREFIX "cpuid"
 253                 : "=a" (*ax),
 254                   "=b" (*bx),
 255                   "=c" (*cx),
 256                   "=d" (*dx)
 257                 : "0" (*ax), "2" (*cx));
 258 


> 
> For hybrid, the hvm container traps cpuid, so we don't need to prefix
> cpuid. Less code if I don't have to worry about trapping the invalid
> op. There seems places where the cpuid is run without the xen prefix in
> user mode.

Right, you can make the .cpuid pvops call point to the native.

> 
> BTW, for cpuid vmexit, if kernel mode I call pv_cpuid, if user mode, I
> just return cpuid values, as that's what would happen without the HVM
> container.

You mean in PV mode in user-space you would get the unfiltered cpuid value?
Yes, that is true. Which is why I am surprised that dm_mapper (are you sure
it is not a kernel thread?) is doing that.

> 
> thanks,
> Mukesh
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [HYBRID]: XEN_EMULATE_PREFIX in user process

[Mukesh Rathor]

On Thu, 28 Jun 2012 21:09:03 -0400
Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

> On Thu, Jun 28, 2012 at 06:00:07PM -0700, Mukesh Rathor wrote:
> > Hi,
> > 
> > Booting latest upstream pv linux as dom0, I noticed dm_mapper, user
> > level process, is running with XEN_EMULATE_PREFIX cpuid. I am
> > wondering if the prefix is statically put there, or how its put
> > there?
> 
> That is impressive. It should be only be in the kernel via the pvops
> cpuid call which does this:
> 
>  252         asm(XEN_EMULATE_PREFIX "cpuid"
>  253                 : "=a" (*ax),
>  254                   "=b" (*bx),
>  255                   "=c" (*cx),
>  256                   "=d" (*dx)
>  257                 : "0" (*ax), "2" (*cx));
>  258 
> 
> 
> > 
> > For hybrid, the hvm container traps cpuid, so we don't need to
> > prefix cpuid. Less code if I don't have to worry about trapping the
> > invalid op. There seems places where the cpuid is run without the
> > xen prefix in user mode.
> 
> Right, you can make the .cpuid pvops call point to the native.

Yup. In the kernel I do that already.

> > 
> > BTW, for cpuid vmexit, if kernel mode I call pv_cpuid, if user
> > mode, I just return cpuid values, as that's what would happen
> > without the HVM container.
> 
> You mean in PV mode in user-space you would get the unfiltered cpuid
> value? Yes, that is true. Which is why I am surprised that dm_mapper
> (are you sure it is not a kernel thread?) is doing that.

User mode:

RIP: 0x0000000000400692 CS: 0x0033

[0]xkdb> dd 0x0000000000400680 32 0
0000000000400680:  89f88953e5894855 db8500000000b9d3
0000000000400690:  0f6e65780b0f0574 4e89045e890689a2

thanks,
Mukesh

Re: [HYBRID]: XEN_EMULATE_PREFIX in user process

[Ian Campbell]

On Fri, 2012-06-29 at 02:26 +0100, Mukesh Rathor wrote:
> On Thu, 28 Jun 2012 21:09:03 -0400
> Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> 
> > On Thu, Jun 28, 2012 at 06:00:07PM -0700, Mukesh Rathor wrote:
> > > Hi,
> > > 
> > > Booting latest upstream pv linux as dom0, I noticed dm_mapper, user
> > > level process, is running with XEN_EMULATE_PREFIX cpuid. I am
> > > wondering if the prefix is statically put there, or how its put
> > > there?
> > 
> > That is impressive. It should be only be in the kernel via the pvops
> > cpuid call which does this:
> > 
> >  252         asm(XEN_EMULATE_PREFIX "cpuid"
> >  253                 : "=a" (*ax),
> >  254                   "=b" (*bx),
> >  255                   "=c" (*cx),
> >  256                   "=d" (*dx)
> >  257                 : "0" (*ax), "2" (*cx));
> >  258 
> > 
> > 
> > > 
> > > For hybrid, the hvm container traps cpuid, so we don't need to
> > > prefix cpuid. Less code if I don't have to worry about trapping the
> > > invalid op. There seems places where the cpuid is run without the
> > > xen prefix in user mode.
> > 
> > Right, you can make the .cpuid pvops call point to the native.
> 
> Yup. In the kernel I do that already.
> 
> > > 
> > > BTW, for cpuid vmexit, if kernel mode I call pv_cpuid, if user
> > > mode, I just return cpuid values, as that's what would happen
> > > without the HVM container.
> > 
> > You mean in PV mode in user-space you would get the unfiltered cpuid
> > value? Yes, that is true. Which is why I am surprised that dm_mapper
> > (are you sure it is not a kernel thread?) is doing that.
> 
> User mode:
> 
> RIP: 0x0000000000400692 CS: 0x0033

CS 0x033 could still be kernel mode on regular PV 64 bit, but I suppose
not on hybrid?

It's unlikely but not impossible for a userspace developer to have done
some "Xen magic" and used the prefix in userspace.

What version of dm_mapper do you have? I checked the version in Debian
Sid and it doesn't do this (at least not directly).

Are you able to run the dm_mapper process under gdb and inspect it to
find the prefix?

Ian.

> 
> [0]xkdb> dd 0x0000000000400680 32 0
> 0000000000400680:  89f88953e5894855 db8500000000b9d3
> 0000000000400690:  0f6e65780b0f0574 4e89045e890689a2
> 
> thanks,
> Mukesh
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [HYBRID]: XEN_EMULATE_PREFIX in user process

[Mukesh Rathor]

On Fri, 29 Jun 2012 09:09:29 +0100
Ian Campbell <Ian.Campbell@citrix.com> wrote:

> On Fri, 2012-06-29 at 02:26 +0100, Mukesh Rathor wrote:
> > On Thu, 28 Jun 2012 21:09:03 -0400
> 
> CS 0x033 could still be kernel mode on regular PV 64 bit, but I
> suppose not on hybrid?

correct, not on hybrid, unless some bug! But then the RIP also
appears user mode.


> It's unlikely but not impossible for a userspace developer to have
> done some "Xen magic" and used the prefix in userspace.
> 
> What version of dm_mapper do you have? I checked the version in Debian
> Sid and it doesn't do this (at least not directly).
> 
> Are you able to run the dm_mapper process under gdb and inspect it to
> find the prefix?

Well, let me take that back. dm_mapper is the last printk, but it could
be anything after that or that. 

Unfortunately, it's dom0 and during boot, so can't run gdb on it. I am
hacking the kernel now to print every user process name in schedule.

thanks,
Mukesh

Re: [HYBRID]: XEN_EMULATE_PREFIX in user process

[Mukesh Rathor]

On Fri, 29 Jun 2012 11:15:46 -0700
Mukesh Rathor <mukesh.rathor@oracle.com> wrote:
> > It's unlikely but not impossible for a userspace developer to have
> > done some "Xen magic" and used the prefix in userspace.
> > 
> > What version of dm_mapper do you have? I checked the version in
> > Debian Sid and it doesn't do this (at least not directly).
> > 
> > Are you able to run the dm_mapper process under gdb and inspect it
> > to find the prefix?
> 
> Well, let me take that back. dm_mapper is the last printk, but it
> could be anything after that or that. 
> 
> Unfortunately, it's dom0 and during boot, so can't run gdb on it. I am
> hacking the kernel now to print every user process name in schedule.

Ah, it's "xen-detect" coming in right after dm_mapper. I see the
xen prefix in it. Hmm... let me see if I can add some run time check
for hybrid dom0, then won't have to worry about invalid_op trap. Less
code that way. That's the end goal anyways...

Thanks,
Mukesh

Re: [HYBRID]: XEN_EMULATE_PREFIX in user process

[Ian Campbell]

On Fri, 2012-06-29 at 20:07 +0100, Mukesh Rathor wrote:
> On Fri, 29 Jun 2012 11:15:46 -0700
> Mukesh Rathor <mukesh.rathor@oracle.com> wrote:
> > > It's unlikely but not impossible for a userspace developer to have
> > > done some "Xen magic" and used the prefix in userspace.
> > > 
> > > What version of dm_mapper do you have? I checked the version in
> > > Debian Sid and it doesn't do this (at least not directly).
> > > 
> > > Are you able to run the dm_mapper process under gdb and inspect it
> > > to find the prefix?
> > 
> > Well, let me take that back. dm_mapper is the last printk, but it
> > could be anything after that or that. 
> > 
> > Unfortunately, it's dom0 and during boot, so can't run gdb on it. I am
> > hacking the kernel now to print every user process name in schedule.
> 
> Ah, it's "xen-detect" coming in right after dm_mapper. I see the
> xen prefix in it. Hmm... let me see if I can add some run time check
> for hybrid dom0, then won't have to worry about invalid_op trap. Less
> code that way. That's the end goal anyways...

I don't think reducing code should come at the expense of adding special
cases for hybrid to userspace programs...

I think you'll have to handle the invalid op, it can't be that much
code, can it?

Ian

Re: [HYBRID]: XEN_EMULATE_PREFIX in user process

[Mukesh Rathor]

On Fri, 29 Jun 2012 15:04:14 -0700 (PDT)
Ian Campbell <Ian.Campbell@citrix.com> wrote:

> On Fri, 2012-06-29 at 20:07 +0100, Mukesh Rathor wrote:
> 
> I don't think reducing code should come at the expense of adding
> special cases for hybrid to userspace programs...
> 
> I think you'll have to handle the invalid op, it can't be that much
> code, can it?

Not a whole lot of code, but other than xen-detect, just curious,
what is the possibility of any user level using XEN_EMULATE_PREFIX cpuid?
The kernel won't, since it's only the hybrid running in hvm container,
and it's modified to use native cpuid. The user level can just run
cpuid and will be trapped in vmexit handler to call pv_cpuid().

thanks,
Mukesh

Re: [HYBRID]: XEN_EMULATE_PREFIX in user process

[Jan Beulich]

>>> On 30.06.12 at 00:50, Mukesh Rathor <mukesh.rathor@oracle.com> wrote:
> On Fri, 29 Jun 2012 15:04:14 -0700 (PDT)
> Ian Campbell <Ian.Campbell@citrix.com> wrote:
> 
>> On Fri, 2012-06-29 at 20:07 +0100, Mukesh Rathor wrote:
>> 
>> I don't think reducing code should come at the expense of adding
>> special cases for hybrid to userspace programs...
>> 
>> I think you'll have to handle the invalid op, it can't be that much
>> code, can it?
> 
> Not a whole lot of code, but other than xen-detect, just curious,
> what is the possibility of any user level using XEN_EMULATE_PREFIX cpuid?

Obviously any code cloned from xen-detect could, i.e. anything
that tries to be Xen-PV-aware.

Jan

Re: [HYBRID]: XEN_EMULATE_PREFIX in user process

[Ian Campbell]

On Mon, 2012-07-02 at 09:27 +0100, Jan Beulich wrote:
> >>> On 30.06.12 at 00:50, Mukesh Rathor <mukesh.rathor@oracle.com> wrote:
> > On Fri, 29 Jun 2012 15:04:14 -0700 (PDT)
> > Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > 
> >> On Fri, 2012-06-29 at 20:07 +0100, Mukesh Rathor wrote:
> >> 
> >> I don't think reducing code should come at the expense of adding
> >> special cases for hybrid to userspace programs...
> >> 
> >> I think you'll have to handle the invalid op, it can't be that much
> >> code, can it?
> > 
> > Not a whole lot of code, but other than xen-detect, just curious,
> > what is the possibility of any user level using XEN_EMULATE_PREFIX cpuid?
> 
> Obviously any code cloned from xen-detect could, i.e. anything
> that tries to be Xen-PV-aware.

Which shouldn't be all that much code but we can't really just declare
it to be wrong...

Ian.

Re: [HYBRID]: XEN_EMULATE_PREFIX in user process

[Mukesh Rathor]

On Mon, 02 Jul 2012 09:27:47 +0100
"Jan Beulich" <JBeulich@suse.com> wrote:

> >>> On 30.06.12 at 00:50, Mukesh Rathor <mukesh.rathor@oracle.com>
> >>> wrote:
> > On Fri, 29 Jun 2012 15:04:14 -0700 (PDT)
> > Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > 
> >> On Fri, 2012-06-29 at 20:07 +0100, Mukesh Rathor wrote:
> >> 
> >> I don't think reducing code should come at the expense of adding
> >> special cases for hybrid to userspace programs...
> >> 
> >> I think you'll have to handle the invalid op, it can't be that much
> >> code, can it?
> > 
> > Not a whole lot of code, but other than xen-detect, just curious,
> > what is the possibility of any user level using XEN_EMULATE_PREFIX
> > cpuid?
> 
> Obviously any code cloned from xen-detect could, i.e. anything
> that tries to be Xen-PV-aware.
> 
> Jan
> 

Ok, so it is official to do that from user level. I will add support
for it.

thanks,
Mukesh

Re: [HYBRID]: XEN_EMULATE_PREFIX in user process

[Jan Beulich]

>>> On 29.06.12 at 03:00, Mukesh Rathor <mukesh.rathor@oracle.com> wrote:
> BTW, for cpuid vmexit, if kernel mode I call pv_cpuid, if user mode, I
> just return cpuid values, as that's what would happen without the HVM
> container.

Which means you're not leveraging one of the things you could
gain from hybrid (after all, returning the native value to user
mode is one of the weaknesses of PV).

Jan

Re: [HYBRID]: XEN_EMULATE_PREFIX in user process

[Ian Campbell]

On Fri, 2012-06-29 at 08:56 +0100, Jan Beulich wrote:
> >>> On 29.06.12 at 03:00, Mukesh Rathor <mukesh.rathor@oracle.com> wrote:
> > BTW, for cpuid vmexit, if kernel mode I call pv_cpuid, if user mode, I
> > just return cpuid values, as that's what would happen without the HVM
> > container.
> 
> Which means you're not leveraging one of the things you could
> gain from hybrid (after all, returning the native value to user
> mode is one of the weaknesses of PV).

cpuid masking could happen in the HVM layer too, couldn't it?

Probably "XEN_EMULATE_PREFIX cpuid" and raw "cpuid" ought to return the
same set of PV values in hybrid?

Ian.

Re: [HYBRID]: XEN_EMULATE_PREFIX in user process

[Jan Beulich]

>>> On 29.06.12 at 10:07, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Fri, 2012-06-29 at 08:56 +0100, Jan Beulich wrote:
>> >>> On 29.06.12 at 03:00, Mukesh Rathor <mukesh.rathor@oracle.com> wrote:
>> > BTW, for cpuid vmexit, if kernel mode I call pv_cpuid, if user mode, I
>> > just return cpuid values, as that's what would happen without the HVM
>> > container.
>> 
>> Which means you're not leveraging one of the things you could
>> gain from hybrid (after all, returning the native value to user
>> mode is one of the weaknesses of PV).
> 
> cpuid masking could happen in the HVM layer too, couldn't it?

But CPUID masking is rather limited (after all that's why CPUID
faulting got added later).

> Probably "XEN_EMULATE_PREFIX cpuid" and raw "cpuid" ought to return the
> same set of PV values in hybrid?

Yes, that's what I would think.

Jan

Re: [HYBRID]: XEN_EMULATE_PREFIX in user process

[Mukesh Rathor]

On Fri, 29 Jun 2012 08:56:39 +0100
"Jan Beulich" <JBeulich@suse.com> wrote:

> >>> On 29.06.12 at 03:00, Mukesh Rathor <mukesh.rathor@oracle.com>
> >>> wrote:
> > BTW, for cpuid vmexit, if kernel mode I call pv_cpuid, if user
> > mode, I just return cpuid values, as that's what would happen
> > without the HVM container.
> 
> Which means you're not leveraging one of the things you could
> gain from hybrid (after all, returning the native value to user
> mode is one of the weaknesses of PV).
> 
> Jan
> 

Easy, fixed. So, now, for both kernel/user I return pv_cpuid.