Question regarding Xen scheduler

Question regarding Xen scheduler

[Zytaruk, Kelly]

I would like to ask a question as to how the Xen scheduler handles certain situations.  
>From everything that I have read I understand that the scheduler treats Dom0 just like any other DomU in regards to scheduling.

My question is in regards to interrupt handlers, critical sections and kernel code.  Is the Xen scheduler aware of whether the Dom0 is executing in kernel code versus user mode code?  Can Dom0 kernel mode code be interrupted by the Xen scheduler and then a DomU run for a while before returning back to Dom0 kernel mode code?

What about interrupt handling routines either in Dom0 or DomU.  Can a Task switch to another DomU / Dom0 occur if in the middle of an interrupt handler?

What if I am doing something time critical or sequence critical that can't be interrupted?  Is there a way to tell the Xen scheduler that "I am in a critical section and can't be interrupted by another Dom" so that Xen won't task switch until I am done?

Other than reading the source code is there a good reference that I can read for Xen internal details and architecture?  

Thanks,
Kelly

Re: Question regarding Xen scheduler

[Andrew Cooper]

On 08/10/14 14:05, Zytaruk, Kelly wrote:
> I would like to ask a question as to how the Xen scheduler handles certain situations.  
> From everything that I have read I understand that the scheduler treats Dom0 just like any other DomU in regards to scheduling.

Correct.  (XenServer has a tweak which allows dom0's credit to scale
relative to current system load, but nothing more complicated.)

>
> My question is in regards to interrupt handlers, critical sections and kernel code.  Is the Xen scheduler aware of whether the Dom0 is executing in kernel code versus user mode code?  Can Dom0 kernel mode code be interrupted by the Xen scheduler and then a DomU run for a while before returning back to Dom0 kernel mode code?

All domains may be interrupted by Xen.  Xen itself knows whether the
domain is in kernel or user mode, but the scheduler does not. 

>
> What about interrupt handling routines either in Dom0 or DomU.  Can a Task switch to another DomU / Dom0 occur if in the middle of an interrupt handler?

Yes

>
> What if I am doing something time critical or sequence critical that can't be interrupted?  Is there a way to tell the Xen scheduler that "I am in a critical section and can't be interrupted by another Dom" so that Xen won't task switch until I am done?

No.

>
> Other than reading the source code is there a good reference that I can read for Xen internal details and architecture?  

Not which comes to mind.

~Andrew

Re: Question regarding Xen scheduler

[Razvan Cojocaru]

On 10/08/2014 04:57 PM, Andrew Cooper wrote:
> On 08/10/14 14:05, Zytaruk, Kelly wrote:
>> Other than reading the source code is there a good reference that I can read for Xen internal details and architecture?  
> 
> Not which comes to mind.

I'm not sure how useful it is as scheduler documentation, but I've found
this book somewhat useful:

http://www.amazon.com/Definitive-Hypervisor-Prentice-Software-Development/dp/0133582493/

It's quite outdated, but there's some insight to be gained from reading
it. It might at least make going through the source code easier.


Hope this helps,
Razvan

Re: Question regarding Xen scheduler

[Zytaruk, Kelly]

> -----Original Message-----
> From: Razvan Cojocaru [mailto:rcojocaru@bitdefender.com]
> Sent: Wednesday, October 08, 2014 10:03 AM
> To: Andrew Cooper; Zytaruk, Kelly; xen-devel@lists.xen.org
> Subject: Re: [Xen-devel] Question regarding Xen scheduler
> 
> On 10/08/2014 04:57 PM, Andrew Cooper wrote:
> > On 08/10/14 14:05, Zytaruk, Kelly wrote:
> >> Other than reading the source code is there a good reference that I can read
> for Xen internal details and architecture?
> >
> > Not which comes to mind.
> 
> I'm not sure how useful it is as scheduler documentation, but I've found this
> book somewhat useful:
> 
> http://www.amazon.com/Definitive-Hypervisor-Prentice-Software-
> Development/dp/0133582493/
> 
> It's quite outdated, but there's some insight to be gained from reading it. It
> might at least make going through the source code easier.
> 
> 
> Hope this helps,
> Razvan

Thanks Razvan, I'll check it out.

Kelly

Re: Question regarding Xen scheduler

[Zytaruk, Kelly]

> -----Original Message-----
> From: Andrew Cooper [mailto:andrew.cooper3@citrix.com]
> Sent: Wednesday, October 08, 2014 9:57 AM
> To: Zytaruk, Kelly; xen-devel@lists.xen.org
> Subject: Re: [Xen-devel] Question regarding Xen scheduler
> 
> On 08/10/14 14:05, Zytaruk, Kelly wrote:
> > I would like to ask a question as to how the Xen scheduler handles certain
> situations.
> > From everything that I have read I understand that the scheduler treats Dom0
> just like any other DomU in regards to scheduling.
> 
> Correct.  (XenServer has a tweak which allows dom0's credit to scale relative to
> current system load, but nothing more complicated.)
> 
> >
> > My question is in regards to interrupt handlers, critical sections and kernel
> code.  Is the Xen scheduler aware of whether the Dom0 is executing in kernel
> code versus user mode code?  Can Dom0 kernel mode code be interrupted by
> the Xen scheduler and then a DomU run for a while before returning back to
> Dom0 kernel mode code?
> 
> All domains may be interrupted by Xen.  Xen itself knows whether the domain is
> in kernel or user mode, but the scheduler does not.
> 
> >
> > What about interrupt handling routines either in Dom0 or DomU.  Can a Task
> switch to another DomU / Dom0 occur if in the middle of an interrupt handler?
> 
> Yes
> 
> >
> > What if I am doing something time critical or sequence critical that can't be
> interrupted?  Is there a way to tell the Xen scheduler that "I am in a critical
> section and can't be interrupted by another Dom" so that Xen won't task switch
> until I am done?
> 
> 
> No.
> 
> >
> > Other than reading the source code is there a good reference that I can read
> for Xen internal details and architecture?
> 
> Not which comes to mind.
> 
> ~Andrew

Thanks Andrew, with the exception of the last answer("> Not which comes to mind")  :-) that's exactly what I needed to know.  It helps a lot.

Kelly

Re: Question regarding Xen scheduler

[Jan Beulich]

>>> On 08.10.14 at 15:57, <andrew.cooper3@citrix.com> wrote:
> On 08/10/14 14:05, Zytaruk, Kelly wrote:
>> What if I am doing something time critical or sequence critical that can't 
> be interrupted?  Is there a way to tell the Xen scheduler that "I am in a 
> critical section and can't be interrupted by another Dom" so that Xen won't 
> task switch until I am done?
> 
> No.

But I'd like to note that suggestions to add some mechanism to
allow this have been made more than once in the past.

Jan

Re: Question regarding Xen scheduler

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1392 bytes --]

On Fri, 2014-10-10 at 09:12 +0100, Jan Beulich wrote:
> >>> On 08.10.14 at 15:57, <andrew.cooper3@citrix.com> wrote:
> > On 08/10/14 14:05, Zytaruk, Kelly wrote:
> >> What if I am doing something time critical or sequence critical that can't 
> > be interrupted?  Is there a way to tell the Xen scheduler that "I am in a 
> > critical section and can't be interrupted by another Dom" so that Xen won't 
> > task switch until I am done?
> > 
> > No.
> 
> But I'd like to note that suggestions to add some mechanism to
> allow this have been made more than once in the past.
> 
And it does seems sensible, as a mechanism, at least IMO. However, the
problem, as often in scheduling, especially when dealing with a complex
architecture, such as Xen's one, would be how to use it "right"...
Assuming we even know what "right" means in this case.

In fact, this is the typical thing that can bring a lot of benefits in
one situation/workload, and at the same time disrupt substantially
another one! :-/

For instance, Kelly, if I can ask, how where you thinking to use it, if
it were there?

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Question regarding Xen scheduler

[Zytaruk, Kelly]

> -----Original Message-----
> From: Dario Faggioli [mailto:dario.faggioli@citrix.com]
> Sent: Friday, October 24, 2014 10:56 AM
> To: Jan Beulich
> Cc: Zytaruk, Kelly; Andrew Cooper; xen-devel@lists.xen.org
> Subject: Re: [Xen-devel] Question regarding Xen scheduler
> 
> On Fri, 2014-10-10 at 09:12 +0100, Jan Beulich wrote:
> > >>> On 08.10.14 at 15:57, <andrew.cooper3@citrix.com> wrote:
> > > On 08/10/14 14:05, Zytaruk, Kelly wrote:
> > >> What if I am doing something time critical or sequence critical
> > >> that can't
> > > be interrupted?  Is there a way to tell the Xen scheduler that "I am
> > > in a critical section and can't be interrupted by another Dom" so
> > > that Xen won't task switch until I am done?
> > >
> > > No.
> >
> > But I'd like to note that suggestions to add some mechanism to allow
> > this have been made more than once in the past.
> >
> And it does seems sensible, as a mechanism, at least IMO. However, the
> problem, as often in scheduling, especially when dealing with a complex
> architecture, such as Xen's one, would be how to use it "right"...
> Assuming we even know what "right" means in this case.
> 
> In fact, this is the typical thing that can bring a lot of benefits in one
> situation/workload, and at the same time disrupt substantially another one!
> :-/
> 
> For instance, Kelly, if I can ask, how where you thinking to use it, if it were
> there?

I am thinking of the case where I have a sequence of hardware registers (can't be more specific yet at this point in time) that need to be accessed without being interrupted.  In some cases timing is critical and it can't start the sequence, be interrupt by a context switch and then wait until it gets its next time slice before continuing. 

Thanks,
Kelly
> 
> Regards,
> Dario
> 
> --
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software
> Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Re: Question regarding Xen scheduler

[Jan Beulich]

>>> On 24.10.14 at 22:26, <Kelly.Zytaruk@amd.com> wrote:
>> From: Dario Faggioli [mailto:dario.faggioli@citrix.com]
>> For instance, Kelly, if I can ask, how where you thinking to use it, if it were
>> there?
> 
> I am thinking of the case where I have a sequence of hardware registers 
> (can't be more specific yet at this point in time) that need to be accessed 
> without being interrupted.  In some cases timing is critical and it can't 
> start the sequence, be interrupt by a context switch and then wait until it 
> gets its next time slice before continuing. 

For this specific use case see the recently added XENPF_resource_op.

Jan