Stack switching in a PV guest

Stack switching in a PV guest

[Simon Martin]

[-- Attachment #1.1: Type: text/plain, Size: 635 bytes --]

Hi all,

I am currently trying to implement multitasking inside my little PV. 
This is done by a a simple stack switch.

Whenever I touch the stack in the PV it crashes. I assume that this is 
due to the iret at the end of the event handler returning control to the 
Hypervisor. Can anyone point me in the right direction to sort this out 
please.

Regards.


--
  _     _  Debian GNU User   Simon Martin
| |   (_)_ __  _   ___  __  Project Manager
| |   | | '_ \| | | \ \/ /  Milliways
| |___| | | | | |_| |>  <   mailto: smartin@milliways.cl
|_____|_|_| |_|\__,_/_/\_\
Si Hoc Legere Scis Nimium Eruditionis Habes

[-- Attachment #1.2: Type: text/html, Size: 1772 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Stack switching in a PV guest

[Ian Campbell]

On Mon, 2013-12-16 at 13:31 +0000, Simon Martin wrote:
> Hi all,
>  
> I am currently trying to implement multitasking inside my little PV.
> This is done by a a simple stack switch.
>  
> Whenever I touch the stack in the PV it crashes. I assume that this is
> due to the iret at the end of the event handler returning control to
> the Hypervisor. Can anyone point me in the right direction to sort
> this out please. 

Need more details of what you have done to advise properly I think.

Perhaps you need to be using HYPERVISOR_iret, or perhaps you need to
make sure that your stack frame actually returns you to the correct
context (which I expect hypervisor context is not).

Ian.

Re: Stack switching in a PV guest

[Simon Martin]

Thanks Ian,

>> I am currently trying to implement multitasking inside my little PV.
>> This is done by a a simple stack switch.
>>  
> Need more details of what you have done to advise properly I think.

For the moment I have just intercepted the main hypervisor callback.

I have simplified to the bare minimum what I am doing here (removed
the  nested  hypercall  handling  and  storing  the register file in a
static  memory  area). As long as I don't touch the stack pointer then
it  works quite happily. As soon as point it to a different stack then
everything falls over.

> Perhaps you need to be using HYPERVISOR_iret, or perhaps you need to
> make sure that your stack frame actually returns you to the correct
> context (which I expect hypervisor context is not).

I  have  checked  using gdb and when it gets to the iretq statement in
HYPERVISOR_iret the stack is as follows

%rsp    -> valid instruction pointer
%rsp+8  -> same CS as the original stack frame.
%rsp+16 -> 0 (initial rFLAGS)

Regards.

Re: Stack switching in a PV guest

[Jan Beulich]

>>> On 16.12.13 at 14:50, Simon Martin <furryfuttock@gmail.com> wrote:
> I  have  checked  using gdb and when it gets to the iretq statement in
> HYPERVISOR_iret the stack is as follows
> 
> %rsp    -> valid instruction pointer
> %rsp+8  -> same CS as the original stack frame.
> %rsp+16 -> 0 (initial rFLAGS)

%rsp+24 -> ???
%rsp+32 -> ???

Those two are the crucial fields when you're having stack issues.

Furthermore, the guest's kernel stack wants to be switched via
__HYPERVISOR_stack_switch, or else the next interrupt/exception
would end up still using the old stack.

Jan

Re: Stack switching in a PV guest

[Simon Martin]

Thanks Jan,

> %rsp+24 ->> ???
> %rsp+32 ->> ???

> Those two are the crucial fields when you're having stack issues.

>From  the pt_regs description those looked like unused. I'll make sure
I fill them in now.

> Furthermore, the guest's kernel stack wants to be switched via
> __HYPERVISOR_stack_switch, or else the next interrupt/exception
> would end up still using the old stack.

Thank you very much. I'll investigate that.


-- 
Best regards,
 Simon                            mailto:furryfuttock@gmail.com

Re: Stack switching in a PV guest

[Simon Martin]

[-- Attachment #1.1: Type: text/plain, Size: 983 bytes --]

Hi Jan


>>  Furthermore, the guest's kernel stack wants to be switched via
>>  __HYPERVISOR_stack_switch, or else the next interrupt/exception
>>  would end up still using the old stack.
I think I've finally got my head round using HYPERVISOR_stack_switch. It 
seems to replace all my stack magic, and at least it's not crashing now.

However, I now have a different problem. I am running my context switch 
off of a single shot timer. When the timer fires Xen transfers control 
to my PV guest via the hypercall_callback.

The hypercall_callback handler stores the current CPU register file, 
however the stored rsp always points to the hypercall stack, not the 
stack that was active in my PV at the moment the timer event fired.

I need to store the PV stack pointer so I can recover it later when my 
scheduler activates it. I have looked back up the stack and I don't 
really see where my active stack pointer is. Any idea where I can get 
it?

Regards.

[-- Attachment #1.2: Type: text/html, Size: 2357 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Stack switching in a PV guest

[Jan Beulich]

>>> On 20.12.13 at 15:12, "Simon Martin" <smartin@milliways.cl> wrote:
> However, I now have a different problem. I am running my context switch 
> off of a single shot timer. When the timer fires Xen transfers control 
> to my PV guest via the hypercall_callback.
> 
> The hypercall_callback handler stores the current CPU register file, 
> however the stored rsp always points to the hypercall stack, not the 
> stack that was active in my PV at the moment the timer event fired.

I'm afraid I'm missing some context here. What is a "hypercall stack"?

> I need to store the PV stack pointer so I can recover it later when my 
> scheduler activates it. I have looked back up the stack and I don't 
> really see where my active stack pointer is. Any idea where I can get 
> it?

The callback gets invoked much like other exceptions would be, with
a normal exception entry stack frame. The stack pointer would be
there (RCX, R11, RIP, CS, EFLAGS, RSP, SS) just as normal.

Jan

Re: Stack switching in a PV guest

[Simon Martin]

[-- Attachment #1.1: Type: text/plain, Size: 783 bytes --]


>I'm afraid I'm missing some context here. What is a "hypercall stack"?
Hmm. When I look at the stack frame it's not what I expected, so I 
assumed that the hypervisor was assigning a stack. Means I've got 
another bug.

>
>The callback gets invoked much like other exceptions would be, with
>a normal exception entry stack frame. The stack pointer would be
>there (RCX, R11, RIP, CS, EFLAGS, RSP, SS) just as normal.
Probably a knock on from the problem I am seeing above then. I've 
searched around for examples of HYPERVISOR_stack_switch usage, but I 
can't find anything. I can find the declaration and handler in the Xen 
source and however no example of making it work. Is there anything you 
can point me to to see how it was intended to be used?

Thanks.


[-- Attachment #1.2: Type: text/html, Size: 1869 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Stack switching in a PV guest

[Jan Beulich]

>>> On 20.12.13 at 16:10, "Simon Martin" <smartin@milliways.cl> wrote:

>>I'm afraid I'm missing some context here. What is a "hypercall stack"?
> Hmm. When I look at the stack frame it's not what I expected, so I 
> assumed that the hypervisor was assigning a stack. Means I've got 
> another bug.
> 
>>The callback gets invoked much like other exceptions would be, with
>>a normal exception entry stack frame. The stack pointer would be
>>there (RCX, R11, RIP, CS, EFLAGS, RSP, SS) just as normal.
> Probably a knock on from the problem I am seeing above then. I've 
> searched around for examples of HYPERVISOR_stack_switch usage, but I 
> can't find anything. I can find the declaration and handler in the Xen 
> source and however no example of making it work. Is there anything you 
> can point me to to see how it was intended to be used?

As to the implementation - look for do_stack_switch() in
xen/arch/x86/x86_64/mm.c.

As for usage examples, see current Linux'es
{HYPERVISOR,MULTI}_stack_switch() in
arch/x86/include/asm/xen/hypercall.h or __switch_to() in
linux-2.6.18-xen.hg's arch/x86_64/kernel/process-xen.c.

Jan

Re: Stack switching in a PV guest

[Mukesh Rathor]

On Fri, 20 Dec 2013 14:12:55 +0000
"Simon Martin" <smartin@milliways.cl> wrote:

> Hi Jan
> 
> 
> >>  Furthermore, the guest's kernel stack wants to be switched via
> >>  __HYPERVISOR_stack_switch, or else the next interrupt/exception
> >>  would end up still using the old stack.
> I think I've finally got my head round using HYPERVISOR_stack_switch.
> It seems to replace all my stack magic, and at least it's not
> crashing now.
> 
> However, I now have a different problem. I am running my context
> switch off of a single shot timer. When the timer fires Xen transfers
> control to my PV guest via the hypercall_callback.
> 
> The hypercall_callback handler stores the current CPU register file, 
> however the stored rsp always points to the hypercall stack, not the 
> stack that was active in my PV at the moment the timer event fired.
> 
> I need to store the PV stack pointer so I can recover it later when
> my scheduler activates it. I have looked back up the stack and I
> don't really see where my active stack pointer is. Any idea where I
> can get it?
> 
> Regards.

May I recommend taking a look at PVH, it would simplify things a
lot for you. Take a look at what linux PVH does, you could start with
linux PVH patches:

"[PATCH v11] PVH support for Linux kernel." email by konrad on
xen-devel.

If you have any questions, I can help, next year tho.

Mukesh