* RCU treating guest mode just like it does user-mode execution
@ 2011-08-17 20:43 Paul E. McKenney
2011-08-17 21:50 ` Gleb Natapov
0 siblings, 1 reply; 9+ messages in thread
From: Paul E. McKenney @ 2011-08-17 20:43 UTC (permalink / raw)
To: gleb; +Cc: linux-kernel, avi, mtosatti, kvm
Hello, Gleb,
I was looking at KVM's call to rcu_virt_note_context_switch()
in kvm_guest_enter(), and noting the comment talking about treating
guest mode like user-mode execution is. One difference between RCU's
treatment of KVM guest execution and user-mode execution is that RCU
notes a context switch only at the beginning of KVM guest execution,
but notes user-mode execution at every scheduling-clock interrupt.
Does it make sense to also note KVM guest execution on each
scheduling-clock interrupt? One reason it might not make sense is
if interrupts from KVM guest execution appear to rcu_check_callbacks()
as interrupts from user-mode execution. (Do they? Given that people
are reporting RCU CPU stall warnings in virtualized environments, I
am beginning to suspect that the answer is "no".)
If KVM guest execution does not appear as user-mode execution to
rcu_check_callback(), I would consider doing the following:
1. Rename rcu_virt_note_context_switch() to something like
rcu_guest_execution_start().
2. Place a call to a new rcu_guest_execution_end() in
kvm_guest_exit().
3. Make rcu_guest_execution_start() and rcu_guest_execution_end()
set and clear a new per-CPU variable.
4. Make rcu_check_callbacks() check this per-CPU variable in
much the same way that it currently checks its "user"
argument, aside from needing to check that the CPU is
not in an interrupt handler or some such.
Of course, some thought is required to make sure that the checks for
executing in an interrupt handler actually cover all of the needed
situations, but so it goes!
Thoughts?
Thanx, Paul
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RCU treating guest mode just like it does user-mode execution
2011-08-17 20:43 RCU treating guest mode just like it does user-mode execution Paul E. McKenney
@ 2011-08-17 21:50 ` Gleb Natapov
2011-08-17 22:05 ` Paul E. McKenney
0 siblings, 1 reply; 9+ messages in thread
From: Gleb Natapov @ 2011-08-17 21:50 UTC (permalink / raw)
To: Paul E. McKenney; +Cc: linux-kernel, avi, mtosatti, kvm
On Wed, Aug 17, 2011 at 01:43:27PM -0700, Paul E. McKenney wrote:
> Hello, Gleb,
>
> I was looking at KVM's call to rcu_virt_note_context_switch()
> in kvm_guest_enter(), and noting the comment talking about treating
> guest mode like user-mode execution is. One difference between RCU's
> treatment of KVM guest execution and user-mode execution is that RCU
> notes a context switch only at the beginning of KVM guest execution,
> but notes user-mode execution at every scheduling-clock interrupt.
>
> Does it make sense to also note KVM guest execution on each
> scheduling-clock interrupt? One reason it might not make sense is
> if interrupts from KVM guest execution appear to rcu_check_callbacks()
> as interrupts from user-mode execution. (Do they? Given that people
> are reporting RCU CPU stall warnings in virtualized environments, I
> am beginning to suspect that the answer is "no".)
>
The answer is "no" because any interrupt kicks cpu out of a guest mode, so
it appears to be in the kernel for RCU. Do people still reporting RCU
stalls even with the my patch?
> If KVM guest execution does not appear as user-mode execution to
> rcu_check_callback(), I would consider doing the following:
>
> 1. Rename rcu_virt_note_context_switch() to something like
> rcu_guest_execution_start().
>
> 2. Place a call to a new rcu_guest_execution_end() in
> kvm_guest_exit().
>
> 3. Make rcu_guest_execution_start() and rcu_guest_execution_end()
> set and clear a new per-CPU variable.
There is such variable already: current->flags & PF_VCPU.
>
> 4. Make rcu_check_callbacks() check this per-CPU variable in
> much the same way that it currently checks its "user"
> argument, aside from needing to check that the CPU is
> not in an interrupt handler or some such.
>
> Of course, some thought is required to make sure that the checks for
> executing in an interrupt handler actually cover all of the needed
> situations, but so it goes!
>
> Thoughts?
I wonder why it will be better than current situation. After cpu leaves
a guest mode there are only three options. It will either go to
userspace, execute schedule or go back to guest mode. At all those cases
RCU should note quiescent state.
--
Gleb.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RCU treating guest mode just like it does user-mode execution
2011-08-17 21:50 ` Gleb Natapov
@ 2011-08-17 22:05 ` Paul E. McKenney
2011-08-17 23:55 ` Gleb Natapov
0 siblings, 1 reply; 9+ messages in thread
From: Paul E. McKenney @ 2011-08-17 22:05 UTC (permalink / raw)
To: Gleb Natapov; +Cc: linux-kernel, avi, mtosatti, kvm
On Thu, Aug 18, 2011 at 12:50:15AM +0300, Gleb Natapov wrote:
> On Wed, Aug 17, 2011 at 01:43:27PM -0700, Paul E. McKenney wrote:
> > Hello, Gleb,
> >
> > I was looking at KVM's call to rcu_virt_note_context_switch()
> > in kvm_guest_enter(), and noting the comment talking about treating
> > guest mode like user-mode execution is. One difference between RCU's
> > treatment of KVM guest execution and user-mode execution is that RCU
> > notes a context switch only at the beginning of KVM guest execution,
> > but notes user-mode execution at every scheduling-clock interrupt.
> >
> > Does it make sense to also note KVM guest execution on each
> > scheduling-clock interrupt? One reason it might not make sense is
> > if interrupts from KVM guest execution appear to rcu_check_callbacks()
> > as interrupts from user-mode execution. (Do they? Given that people
> > are reporting RCU CPU stall warnings in virtualized environments, I
> > am beginning to suspect that the answer is "no".)
> >
> The answer is "no" because any interrupt kicks cpu out of a guest mode, so
> it appears to be in the kernel for RCU. Do people still reporting RCU
> stalls even with the my patch?
>
> > If KVM guest execution does not appear as user-mode execution to
> > rcu_check_callback(), I would consider doing the following:
> >
> > 1. Rename rcu_virt_note_context_switch() to something like
> > rcu_guest_execution_start().
> >
> > 2. Place a call to a new rcu_guest_execution_end() in
> > kvm_guest_exit().
> >
> > 3. Make rcu_guest_execution_start() and rcu_guest_execution_end()
> > set and clear a new per-CPU variable.
> There is such variable already: current->flags & PF_VCPU.
Good to know, thank you!
> > 4. Make rcu_check_callbacks() check this per-CPU variable in
> > much the same way that it currently checks its "user"
> > argument, aside from needing to check that the CPU is
> > not in an interrupt handler or some such.
> >
> > Of course, some thought is required to make sure that the checks for
> > executing in an interrupt handler actually cover all of the needed
> > situations, but so it goes!
> >
> > Thoughts?
>
> I wonder why it will be better than current situation. After cpu leaves
> a guest mode there are only three options. It will either go to
> userspace, execute schedule or go back to guest mode. At all those cases
> RCU should note quiescent state.
Might be that the current state is optimal. That would be a good thing.
But if a CPU stays in guest mode for (say) 30 seconds, it will have
called schedule() every jiffy in the meantime? In other words, if
a CPU stays in guest mode for a long time, how does RCU know that
this CPU is in an extended quiescent state?
Thanx, Paul
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RCU treating guest mode just like it does user-mode execution
2011-08-17 22:05 ` Paul E. McKenney
@ 2011-08-17 23:55 ` Gleb Natapov
2011-08-18 0:25 ` Paul E. McKenney
0 siblings, 1 reply; 9+ messages in thread
From: Gleb Natapov @ 2011-08-17 23:55 UTC (permalink / raw)
To: Paul E. McKenney; +Cc: linux-kernel, avi, mtosatti, kvm
On Wed, Aug 17, 2011 at 03:05:20PM -0700, Paul E. McKenney wrote:
> On Thu, Aug 18, 2011 at 12:50:15AM +0300, Gleb Natapov wrote:
> > On Wed, Aug 17, 2011 at 01:43:27PM -0700, Paul E. McKenney wrote:
> > > Hello, Gleb,
> > >
> > > I was looking at KVM's call to rcu_virt_note_context_switch()
> > > in kvm_guest_enter(), and noting the comment talking about treating
> > > guest mode like user-mode execution is. One difference between RCU's
> > > treatment of KVM guest execution and user-mode execution is that RCU
> > > notes a context switch only at the beginning of KVM guest execution,
> > > but notes user-mode execution at every scheduling-clock interrupt.
> > >
> > > Does it make sense to also note KVM guest execution on each
> > > scheduling-clock interrupt? One reason it might not make sense is
> > > if interrupts from KVM guest execution appear to rcu_check_callbacks()
> > > as interrupts from user-mode execution. (Do they? Given that people
> > > are reporting RCU CPU stall warnings in virtualized environments, I
> > > am beginning to suspect that the answer is "no".)
> > >
> > The answer is "no" because any interrupt kicks cpu out of a guest mode, so
> > it appears to be in the kernel for RCU. Do people still reporting RCU
> > stalls even with the my patch?
> >
> > > If KVM guest execution does not appear as user-mode execution to
> > > rcu_check_callback(), I would consider doing the following:
> > >
> > > 1. Rename rcu_virt_note_context_switch() to something like
> > > rcu_guest_execution_start().
> > >
> > > 2. Place a call to a new rcu_guest_execution_end() in
> > > kvm_guest_exit().
> > >
> > > 3. Make rcu_guest_execution_start() and rcu_guest_execution_end()
> > > set and clear a new per-CPU variable.
> > There is such variable already: current->flags & PF_VCPU.
>
> Good to know, thank you!
>
> > > 4. Make rcu_check_callbacks() check this per-CPU variable in
> > > much the same way that it currently checks its "user"
> > > argument, aside from needing to check that the CPU is
> > > not in an interrupt handler or some such.
> > >
> > > Of course, some thought is required to make sure that the checks for
> > > executing in an interrupt handler actually cover all of the needed
> > > situations, but so it goes!
> > >
> > > Thoughts?
> >
> > I wonder why it will be better than current situation. After cpu leaves
> > a guest mode there are only three options. It will either go to
> > userspace, execute schedule or go back to guest mode. At all those cases
> > RCU should note quiescent state.
>
> Might be that the current state is optimal. That would be a good thing.
>
> But if a CPU stays in guest mode for (say) 30 seconds, it will have
> called schedule() every jiffy in the meantime? In other words, if
> a CPU stays in guest mode for a long time, how does RCU know that
> this CPU is in an extended quiescent state?
>
Wouldn't scheduling-clock interrupt kick vcpu out of a guest mode much
earlier then 30 seconds?
--
Gleb.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RCU treating guest mode just like it does user-mode execution
2011-08-17 23:55 ` Gleb Natapov
@ 2011-08-18 0:25 ` Paul E. McKenney
2011-08-18 0:28 ` Avi Kivity
2011-08-18 0:44 ` Gleb Natapov
0 siblings, 2 replies; 9+ messages in thread
From: Paul E. McKenney @ 2011-08-18 0:25 UTC (permalink / raw)
To: Gleb Natapov; +Cc: linux-kernel, avi, mtosatti, kvm
On Thu, Aug 18, 2011 at 02:55:29AM +0300, Gleb Natapov wrote:
> On Wed, Aug 17, 2011 at 03:05:20PM -0700, Paul E. McKenney wrote:
> > On Thu, Aug 18, 2011 at 12:50:15AM +0300, Gleb Natapov wrote:
> > > On Wed, Aug 17, 2011 at 01:43:27PM -0700, Paul E. McKenney wrote:
> > > > Hello, Gleb,
> > > >
> > > > I was looking at KVM's call to rcu_virt_note_context_switch()
> > > > in kvm_guest_enter(), and noting the comment talking about treating
> > > > guest mode like user-mode execution is. One difference between RCU's
> > > > treatment of KVM guest execution and user-mode execution is that RCU
> > > > notes a context switch only at the beginning of KVM guest execution,
> > > > but notes user-mode execution at every scheduling-clock interrupt.
> > > >
> > > > Does it make sense to also note KVM guest execution on each
> > > > scheduling-clock interrupt? One reason it might not make sense is
> > > > if interrupts from KVM guest execution appear to rcu_check_callbacks()
> > > > as interrupts from user-mode execution. (Do they? Given that people
> > > > are reporting RCU CPU stall warnings in virtualized environments, I
> > > > am beginning to suspect that the answer is "no".)
> > > >
> > > The answer is "no" because any interrupt kicks cpu out of a guest mode, so
> > > it appears to be in the kernel for RCU. Do people still reporting RCU
> > > stalls even with the my patch?
> > >
> > > > If KVM guest execution does not appear as user-mode execution to
> > > > rcu_check_callback(), I would consider doing the following:
> > > >
> > > > 1. Rename rcu_virt_note_context_switch() to something like
> > > > rcu_guest_execution_start().
> > > >
> > > > 2. Place a call to a new rcu_guest_execution_end() in
> > > > kvm_guest_exit().
> > > >
> > > > 3. Make rcu_guest_execution_start() and rcu_guest_execution_end()
> > > > set and clear a new per-CPU variable.
> > > There is such variable already: current->flags & PF_VCPU.
> >
> > Good to know, thank you!
> >
> > > > 4. Make rcu_check_callbacks() check this per-CPU variable in
> > > > much the same way that it currently checks its "user"
> > > > argument, aside from needing to check that the CPU is
> > > > not in an interrupt handler or some such.
> > > >
> > > > Of course, some thought is required to make sure that the checks for
> > > > executing in an interrupt handler actually cover all of the needed
> > > > situations, but so it goes!
> > > >
> > > > Thoughts?
> > >
> > > I wonder why it will be better than current situation. After cpu leaves
> > > a guest mode there are only three options. It will either go to
> > > userspace, execute schedule or go back to guest mode. At all those cases
> > > RCU should note quiescent state.
> >
> > Might be that the current state is optimal. That would be a good thing.
> >
> > But if a CPU stays in guest mode for (say) 30 seconds, it will have
> > called schedule() every jiffy in the meantime? In other words, if
> > a CPU stays in guest mode for a long time, how does RCU know that
> > this CPU is in an extended quiescent state?
> >
> Wouldn't scheduling-clock interrupt kick vcpu out of a guest mode much
> earlier then 30 seconds?
The scheduling-clock interrupt would happen, but I do not know whether
or not it would kick the vcpu out of guest mode in such a way that
would result in RCU thinking that the CPU has passed through a quiescent
state.
Thanx, Paul
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RCU treating guest mode just like it does user-mode execution
2011-08-18 0:25 ` Paul E. McKenney
@ 2011-08-18 0:28 ` Avi Kivity
2011-08-18 0:44 ` Paul E. McKenney
2011-08-18 0:44 ` Gleb Natapov
1 sibling, 1 reply; 9+ messages in thread
From: Avi Kivity @ 2011-08-18 0:28 UTC (permalink / raw)
To: paulmck; +Cc: Gleb Natapov, linux-kernel, mtosatti, kvm
On 08/17/2011 05:25 PM, Paul E. McKenney wrote:
> > >
> > Wouldn't scheduling-clock interrupt kick vcpu out of a guest mode much
> > earlier then 30 seconds?
>
> The scheduling-clock interrupt would happen, but I do not know whether
> or not it would kick the vcpu out of guest mode in such a way that
> would result in RCU thinking that the CPU has passed through a quiescent
> state.
>
The result can only be one of
reenter guest (and rcu_virt_note_context_switch)
schedule()
exit to userspace
--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RCU treating guest mode just like it does user-mode execution
2011-08-18 0:25 ` Paul E. McKenney
2011-08-18 0:28 ` Avi Kivity
@ 2011-08-18 0:44 ` Gleb Natapov
1 sibling, 0 replies; 9+ messages in thread
From: Gleb Natapov @ 2011-08-18 0:44 UTC (permalink / raw)
To: Paul E. McKenney; +Cc: linux-kernel, avi, mtosatti, kvm
On Wed, Aug 17, 2011 at 05:25:13PM -0700, Paul E. McKenney wrote:
> On Thu, Aug 18, 2011 at 02:55:29AM +0300, Gleb Natapov wrote:
> > On Wed, Aug 17, 2011 at 03:05:20PM -0700, Paul E. McKenney wrote:
> > > On Thu, Aug 18, 2011 at 12:50:15AM +0300, Gleb Natapov wrote:
> > > > On Wed, Aug 17, 2011 at 01:43:27PM -0700, Paul E. McKenney wrote:
> > > > > Hello, Gleb,
> > > > >
> > > > > I was looking at KVM's call to rcu_virt_note_context_switch()
> > > > > in kvm_guest_enter(), and noting the comment talking about treating
> > > > > guest mode like user-mode execution is. One difference between RCU's
> > > > > treatment of KVM guest execution and user-mode execution is that RCU
> > > > > notes a context switch only at the beginning of KVM guest execution,
> > > > > but notes user-mode execution at every scheduling-clock interrupt.
> > > > >
> > > > > Does it make sense to also note KVM guest execution on each
> > > > > scheduling-clock interrupt? One reason it might not make sense is
> > > > > if interrupts from KVM guest execution appear to rcu_check_callbacks()
> > > > > as interrupts from user-mode execution. (Do they? Given that people
> > > > > are reporting RCU CPU stall warnings in virtualized environments, I
> > > > > am beginning to suspect that the answer is "no".)
> > > > >
> > > > The answer is "no" because any interrupt kicks cpu out of a guest mode, so
> > > > it appears to be in the kernel for RCU. Do people still reporting RCU
> > > > stalls even with the my patch?
> > > >
> > > > > If KVM guest execution does not appear as user-mode execution to
> > > > > rcu_check_callback(), I would consider doing the following:
> > > > >
> > > > > 1. Rename rcu_virt_note_context_switch() to something like
> > > > > rcu_guest_execution_start().
> > > > >
> > > > > 2. Place a call to a new rcu_guest_execution_end() in
> > > > > kvm_guest_exit().
> > > > >
> > > > > 3. Make rcu_guest_execution_start() and rcu_guest_execution_end()
> > > > > set and clear a new per-CPU variable.
> > > > There is such variable already: current->flags & PF_VCPU.
> > >
> > > Good to know, thank you!
> > >
> > > > > 4. Make rcu_check_callbacks() check this per-CPU variable in
> > > > > much the same way that it currently checks its "user"
> > > > > argument, aside from needing to check that the CPU is
> > > > > not in an interrupt handler or some such.
> > > > >
> > > > > Of course, some thought is required to make sure that the checks for
> > > > > executing in an interrupt handler actually cover all of the needed
> > > > > situations, but so it goes!
> > > > >
> > > > > Thoughts?
> > > >
> > > > I wonder why it will be better than current situation. After cpu leaves
> > > > a guest mode there are only three options. It will either go to
> > > > userspace, execute schedule or go back to guest mode. At all those cases
> > > > RCU should note quiescent state.
> > >
> > > Might be that the current state is optimal. That would be a good thing.
> > >
> > > But if a CPU stays in guest mode for (say) 30 seconds, it will have
> > > called schedule() every jiffy in the meantime? In other words, if
> > > a CPU stays in guest mode for a long time, how does RCU know that
> > > this CPU is in an extended quiescent state?
> > >
> > Wouldn't scheduling-clock interrupt kick vcpu out of a guest mode much
> > earlier then 30 seconds?
>
> The scheduling-clock interrupt would happen, but I do not know whether
> or not it would kick the vcpu out of guest mode in such a way that
> would result in RCU thinking that the CPU has passed through a quiescent
> state.
Then I think we are OK. Any interrupt will kick cpu out of guest mode.
After that vcpu thread will be either rescheduled or it will get back to
guest mode calling rcu_virt_note_context_switch() on the way there.
--
Gleb.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RCU treating guest mode just like it does user-mode execution
2011-08-18 0:28 ` Avi Kivity
@ 2011-08-18 0:44 ` Paul E. McKenney
2011-08-18 0:53 ` Avi Kivity
0 siblings, 1 reply; 9+ messages in thread
From: Paul E. McKenney @ 2011-08-18 0:44 UTC (permalink / raw)
To: Avi Kivity; +Cc: Gleb Natapov, linux-kernel, mtosatti, kvm
On Wed, Aug 17, 2011 at 05:28:46PM -0700, Avi Kivity wrote:
> On 08/17/2011 05:25 PM, Paul E. McKenney wrote:
> >> >
> >> Wouldn't scheduling-clock interrupt kick vcpu out of a guest mode much
> >> earlier then 30 seconds?
> >
> >The scheduling-clock interrupt would happen, but I do not know whether
> >or not it would kick the vcpu out of guest mode in such a way that
> >would result in RCU thinking that the CPU has passed through a quiescent
> >state.
> >
>
> The result can only be one of
>
> reenter guest (and rcu_virt_note_context_switch)
> schedule()
> exit to userspace
Very good, thank you both for bearing with me on this!
Thanx, Paul
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RCU treating guest mode just like it does user-mode execution
2011-08-18 0:44 ` Paul E. McKenney
@ 2011-08-18 0:53 ` Avi Kivity
0 siblings, 0 replies; 9+ messages in thread
From: Avi Kivity @ 2011-08-18 0:53 UTC (permalink / raw)
To: paulmck; +Cc: Gleb Natapov, linux-kernel, mtosatti, kvm
On 08/17/2011 05:44 PM, Paul E. McKenney wrote:
> >
> > The result can only be one of
> >
> > reenter guest (and rcu_virt_note_context_switch)
> > schedule()
> > exit to userspace
>
> Very good, thank you both for bearing with me on this!
>
Hey, it's our pleasure. Pester us as often as you'll like and we'll
happily "bear with you".
--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2011-08-18 0:54 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-08-17 20:43 RCU treating guest mode just like it does user-mode execution Paul E. McKenney
2011-08-17 21:50 ` Gleb Natapov
2011-08-17 22:05 ` Paul E. McKenney
2011-08-17 23:55 ` Gleb Natapov
2011-08-18 0:25 ` Paul E. McKenney
2011-08-18 0:28 ` Avi Kivity
2011-08-18 0:44 ` Paul E. McKenney
2011-08-18 0:53 ` Avi Kivity
2011-08-18 0:44 ` Gleb Natapov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox