* RCU treating guest mode just like it does user-mode execution @ 2011-08-17 20:43 Paul E. McKenney 2011-08-17 21:50 ` Gleb Natapov 0 siblings, 1 reply; 9+ messages in thread From: Paul E. McKenney @ 2011-08-17 20:43 UTC (permalink / raw) To: gleb; +Cc: linux-kernel, avi, mtosatti, kvm Hello, Gleb, I was looking at KVM's call to rcu_virt_note_context_switch() in kvm_guest_enter(), and noting the comment talking about treating guest mode like user-mode execution is. One difference between RCU's treatment of KVM guest execution and user-mode execution is that RCU notes a context switch only at the beginning of KVM guest execution, but notes user-mode execution at every scheduling-clock interrupt. Does it make sense to also note KVM guest execution on each scheduling-clock interrupt? One reason it might not make sense is if interrupts from KVM guest execution appear to rcu_check_callbacks() as interrupts from user-mode execution. (Do they? Given that people are reporting RCU CPU stall warnings in virtualized environments, I am beginning to suspect that the answer is "no".) If KVM guest execution does not appear as user-mode execution to rcu_check_callback(), I would consider doing the following: 1. Rename rcu_virt_note_context_switch() to something like rcu_guest_execution_start(). 2. Place a call to a new rcu_guest_execution_end() in kvm_guest_exit(). 3. Make rcu_guest_execution_start() and rcu_guest_execution_end() set and clear a new per-CPU variable. 4. Make rcu_check_callbacks() check this per-CPU variable in much the same way that it currently checks its "user" argument, aside from needing to check that the CPU is not in an interrupt handler or some such. Of course, some thought is required to make sure that the checks for executing in an interrupt handler actually cover all of the needed situations, but so it goes! Thoughts? Thanx, Paul ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RCU treating guest mode just like it does user-mode execution 2011-08-17 20:43 RCU treating guest mode just like it does user-mode execution Paul E. McKenney @ 2011-08-17 21:50 ` Gleb Natapov 2011-08-17 22:05 ` Paul E. McKenney 0 siblings, 1 reply; 9+ messages in thread From: Gleb Natapov @ 2011-08-17 21:50 UTC (permalink / raw) To: Paul E. McKenney; +Cc: linux-kernel, avi, mtosatti, kvm On Wed, Aug 17, 2011 at 01:43:27PM -0700, Paul E. McKenney wrote: > Hello, Gleb, > > I was looking at KVM's call to rcu_virt_note_context_switch() > in kvm_guest_enter(), and noting the comment talking about treating > guest mode like user-mode execution is. One difference between RCU's > treatment of KVM guest execution and user-mode execution is that RCU > notes a context switch only at the beginning of KVM guest execution, > but notes user-mode execution at every scheduling-clock interrupt. > > Does it make sense to also note KVM guest execution on each > scheduling-clock interrupt? One reason it might not make sense is > if interrupts from KVM guest execution appear to rcu_check_callbacks() > as interrupts from user-mode execution. (Do they? Given that people > are reporting RCU CPU stall warnings in virtualized environments, I > am beginning to suspect that the answer is "no".) > The answer is "no" because any interrupt kicks cpu out of a guest mode, so it appears to be in the kernel for RCU. Do people still reporting RCU stalls even with the my patch? > If KVM guest execution does not appear as user-mode execution to > rcu_check_callback(), I would consider doing the following: > > 1. Rename rcu_virt_note_context_switch() to something like > rcu_guest_execution_start(). > > 2. Place a call to a new rcu_guest_execution_end() in > kvm_guest_exit(). > > 3. Make rcu_guest_execution_start() and rcu_guest_execution_end() > set and clear a new per-CPU variable. There is such variable already: current->flags & PF_VCPU. > > 4. Make rcu_check_callbacks() check this per-CPU variable in > much the same way that it currently checks its "user" > argument, aside from needing to check that the CPU is > not in an interrupt handler or some such. > > Of course, some thought is required to make sure that the checks for > executing in an interrupt handler actually cover all of the needed > situations, but so it goes! > > Thoughts? I wonder why it will be better than current situation. After cpu leaves a guest mode there are only three options. It will either go to userspace, execute schedule or go back to guest mode. At all those cases RCU should note quiescent state. -- Gleb. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RCU treating guest mode just like it does user-mode execution 2011-08-17 21:50 ` Gleb Natapov @ 2011-08-17 22:05 ` Paul E. McKenney 2011-08-17 23:55 ` Gleb Natapov 0 siblings, 1 reply; 9+ messages in thread From: Paul E. McKenney @ 2011-08-17 22:05 UTC (permalink / raw) To: Gleb Natapov; +Cc: linux-kernel, avi, mtosatti, kvm On Thu, Aug 18, 2011 at 12:50:15AM +0300, Gleb Natapov wrote: > On Wed, Aug 17, 2011 at 01:43:27PM -0700, Paul E. McKenney wrote: > > Hello, Gleb, > > > > I was looking at KVM's call to rcu_virt_note_context_switch() > > in kvm_guest_enter(), and noting the comment talking about treating > > guest mode like user-mode execution is. One difference between RCU's > > treatment of KVM guest execution and user-mode execution is that RCU > > notes a context switch only at the beginning of KVM guest execution, > > but notes user-mode execution at every scheduling-clock interrupt. > > > > Does it make sense to also note KVM guest execution on each > > scheduling-clock interrupt? One reason it might not make sense is > > if interrupts from KVM guest execution appear to rcu_check_callbacks() > > as interrupts from user-mode execution. (Do they? Given that people > > are reporting RCU CPU stall warnings in virtualized environments, I > > am beginning to suspect that the answer is "no".) > > > The answer is "no" because any interrupt kicks cpu out of a guest mode, so > it appears to be in the kernel for RCU. Do people still reporting RCU > stalls even with the my patch? > > > If KVM guest execution does not appear as user-mode execution to > > rcu_check_callback(), I would consider doing the following: > > > > 1. Rename rcu_virt_note_context_switch() to something like > > rcu_guest_execution_start(). > > > > 2. Place a call to a new rcu_guest_execution_end() in > > kvm_guest_exit(). > > > > 3. Make rcu_guest_execution_start() and rcu_guest_execution_end() > > set and clear a new per-CPU variable. > There is such variable already: current->flags & PF_VCPU. Good to know, thank you! > > 4. Make rcu_check_callbacks() check this per-CPU variable in > > much the same way that it currently checks its "user" > > argument, aside from needing to check that the CPU is > > not in an interrupt handler or some such. > > > > Of course, some thought is required to make sure that the checks for > > executing in an interrupt handler actually cover all of the needed > > situations, but so it goes! > > > > Thoughts? > > I wonder why it will be better than current situation. After cpu leaves > a guest mode there are only three options. It will either go to > userspace, execute schedule or go back to guest mode. At all those cases > RCU should note quiescent state. Might be that the current state is optimal. That would be a good thing. But if a CPU stays in guest mode for (say) 30 seconds, it will have called schedule() every jiffy in the meantime? In other words, if a CPU stays in guest mode for a long time, how does RCU know that this CPU is in an extended quiescent state? Thanx, Paul ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RCU treating guest mode just like it does user-mode execution 2011-08-17 22:05 ` Paul E. McKenney @ 2011-08-17 23:55 ` Gleb Natapov 2011-08-18 0:25 ` Paul E. McKenney 0 siblings, 1 reply; 9+ messages in thread From: Gleb Natapov @ 2011-08-17 23:55 UTC (permalink / raw) To: Paul E. McKenney; +Cc: linux-kernel, avi, mtosatti, kvm On Wed, Aug 17, 2011 at 03:05:20PM -0700, Paul E. McKenney wrote: > On Thu, Aug 18, 2011 at 12:50:15AM +0300, Gleb Natapov wrote: > > On Wed, Aug 17, 2011 at 01:43:27PM -0700, Paul E. McKenney wrote: > > > Hello, Gleb, > > > > > > I was looking at KVM's call to rcu_virt_note_context_switch() > > > in kvm_guest_enter(), and noting the comment talking about treating > > > guest mode like user-mode execution is. One difference between RCU's > > > treatment of KVM guest execution and user-mode execution is that RCU > > > notes a context switch only at the beginning of KVM guest execution, > > > but notes user-mode execution at every scheduling-clock interrupt. > > > > > > Does it make sense to also note KVM guest execution on each > > > scheduling-clock interrupt? One reason it might not make sense is > > > if interrupts from KVM guest execution appear to rcu_check_callbacks() > > > as interrupts from user-mode execution. (Do they? Given that people > > > are reporting RCU CPU stall warnings in virtualized environments, I > > > am beginning to suspect that the answer is "no".) > > > > > The answer is "no" because any interrupt kicks cpu out of a guest mode, so > > it appears to be in the kernel for RCU. Do people still reporting RCU > > stalls even with the my patch? > > > > > If KVM guest execution does not appear as user-mode execution to > > > rcu_check_callback(), I would consider doing the following: > > > > > > 1. Rename rcu_virt_note_context_switch() to something like > > > rcu_guest_execution_start(). > > > > > > 2. Place a call to a new rcu_guest_execution_end() in > > > kvm_guest_exit(). > > > > > > 3. Make rcu_guest_execution_start() and rcu_guest_execution_end() > > > set and clear a new per-CPU variable. > > There is such variable already: current->flags & PF_VCPU. > > Good to know, thank you! > > > > 4. Make rcu_check_callbacks() check this per-CPU variable in > > > much the same way that it currently checks its "user" > > > argument, aside from needing to check that the CPU is > > > not in an interrupt handler or some such. > > > > > > Of course, some thought is required to make sure that the checks for > > > executing in an interrupt handler actually cover all of the needed > > > situations, but so it goes! > > > > > > Thoughts? > > > > I wonder why it will be better than current situation. After cpu leaves > > a guest mode there are only three options. It will either go to > > userspace, execute schedule or go back to guest mode. At all those cases > > RCU should note quiescent state. > > Might be that the current state is optimal. That would be a good thing. > > But if a CPU stays in guest mode for (say) 30 seconds, it will have > called schedule() every jiffy in the meantime? In other words, if > a CPU stays in guest mode for a long time, how does RCU know that > this CPU is in an extended quiescent state? > Wouldn't scheduling-clock interrupt kick vcpu out of a guest mode much earlier then 30 seconds? -- Gleb. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RCU treating guest mode just like it does user-mode execution 2011-08-17 23:55 ` Gleb Natapov @ 2011-08-18 0:25 ` Paul E. McKenney 2011-08-18 0:28 ` Avi Kivity 2011-08-18 0:44 ` Gleb Natapov 0 siblings, 2 replies; 9+ messages in thread From: Paul E. McKenney @ 2011-08-18 0:25 UTC (permalink / raw) To: Gleb Natapov; +Cc: linux-kernel, avi, mtosatti, kvm On Thu, Aug 18, 2011 at 02:55:29AM +0300, Gleb Natapov wrote: > On Wed, Aug 17, 2011 at 03:05:20PM -0700, Paul E. McKenney wrote: > > On Thu, Aug 18, 2011 at 12:50:15AM +0300, Gleb Natapov wrote: > > > On Wed, Aug 17, 2011 at 01:43:27PM -0700, Paul E. McKenney wrote: > > > > Hello, Gleb, > > > > > > > > I was looking at KVM's call to rcu_virt_note_context_switch() > > > > in kvm_guest_enter(), and noting the comment talking about treating > > > > guest mode like user-mode execution is. One difference between RCU's > > > > treatment of KVM guest execution and user-mode execution is that RCU > > > > notes a context switch only at the beginning of KVM guest execution, > > > > but notes user-mode execution at every scheduling-clock interrupt. > > > > > > > > Does it make sense to also note KVM guest execution on each > > > > scheduling-clock interrupt? One reason it might not make sense is > > > > if interrupts from KVM guest execution appear to rcu_check_callbacks() > > > > as interrupts from user-mode execution. (Do they? Given that people > > > > are reporting RCU CPU stall warnings in virtualized environments, I > > > > am beginning to suspect that the answer is "no".) > > > > > > > The answer is "no" because any interrupt kicks cpu out of a guest mode, so > > > it appears to be in the kernel for RCU. Do people still reporting RCU > > > stalls even with the my patch? > > > > > > > If KVM guest execution does not appear as user-mode execution to > > > > rcu_check_callback(), I would consider doing the following: > > > > > > > > 1. Rename rcu_virt_note_context_switch() to something like > > > > rcu_guest_execution_start(). > > > > > > > > 2. Place a call to a new rcu_guest_execution_end() in > > > > kvm_guest_exit(). > > > > > > > > 3. Make rcu_guest_execution_start() and rcu_guest_execution_end() > > > > set and clear a new per-CPU variable. > > > There is such variable already: current->flags & PF_VCPU. > > > > Good to know, thank you! > > > > > > 4. Make rcu_check_callbacks() check this per-CPU variable in > > > > much the same way that it currently checks its "user" > > > > argument, aside from needing to check that the CPU is > > > > not in an interrupt handler or some such. > > > > > > > > Of course, some thought is required to make sure that the checks for > > > > executing in an interrupt handler actually cover all of the needed > > > > situations, but so it goes! > > > > > > > > Thoughts? > > > > > > I wonder why it will be better than current situation. After cpu leaves > > > a guest mode there are only three options. It will either go to > > > userspace, execute schedule or go back to guest mode. At all those cases > > > RCU should note quiescent state. > > > > Might be that the current state is optimal. That would be a good thing. > > > > But if a CPU stays in guest mode for (say) 30 seconds, it will have > > called schedule() every jiffy in the meantime? In other words, if > > a CPU stays in guest mode for a long time, how does RCU know that > > this CPU is in an extended quiescent state? > > > Wouldn't scheduling-clock interrupt kick vcpu out of a guest mode much > earlier then 30 seconds? The scheduling-clock interrupt would happen, but I do not know whether or not it would kick the vcpu out of guest mode in such a way that would result in RCU thinking that the CPU has passed through a quiescent state. Thanx, Paul ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RCU treating guest mode just like it does user-mode execution 2011-08-18 0:25 ` Paul E. McKenney @ 2011-08-18 0:28 ` Avi Kivity 2011-08-18 0:44 ` Paul E. McKenney 2011-08-18 0:44 ` Gleb Natapov 1 sibling, 1 reply; 9+ messages in thread From: Avi Kivity @ 2011-08-18 0:28 UTC (permalink / raw) To: paulmck; +Cc: Gleb Natapov, linux-kernel, mtosatti, kvm On 08/17/2011 05:25 PM, Paul E. McKenney wrote: > > > > > Wouldn't scheduling-clock interrupt kick vcpu out of a guest mode much > > earlier then 30 seconds? > > The scheduling-clock interrupt would happen, but I do not know whether > or not it would kick the vcpu out of guest mode in such a way that > would result in RCU thinking that the CPU has passed through a quiescent > state. > The result can only be one of reenter guest (and rcu_virt_note_context_switch) schedule() exit to userspace -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RCU treating guest mode just like it does user-mode execution 2011-08-18 0:28 ` Avi Kivity @ 2011-08-18 0:44 ` Paul E. McKenney 2011-08-18 0:53 ` Avi Kivity 0 siblings, 1 reply; 9+ messages in thread From: Paul E. McKenney @ 2011-08-18 0:44 UTC (permalink / raw) To: Avi Kivity; +Cc: Gleb Natapov, linux-kernel, mtosatti, kvm On Wed, Aug 17, 2011 at 05:28:46PM -0700, Avi Kivity wrote: > On 08/17/2011 05:25 PM, Paul E. McKenney wrote: > >> > > >> Wouldn't scheduling-clock interrupt kick vcpu out of a guest mode much > >> earlier then 30 seconds? > > > >The scheduling-clock interrupt would happen, but I do not know whether > >or not it would kick the vcpu out of guest mode in such a way that > >would result in RCU thinking that the CPU has passed through a quiescent > >state. > > > > The result can only be one of > > reenter guest (and rcu_virt_note_context_switch) > schedule() > exit to userspace Very good, thank you both for bearing with me on this! Thanx, Paul ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RCU treating guest mode just like it does user-mode execution 2011-08-18 0:44 ` Paul E. McKenney @ 2011-08-18 0:53 ` Avi Kivity 0 siblings, 0 replies; 9+ messages in thread From: Avi Kivity @ 2011-08-18 0:53 UTC (permalink / raw) To: paulmck; +Cc: Gleb Natapov, linux-kernel, mtosatti, kvm On 08/17/2011 05:44 PM, Paul E. McKenney wrote: > > > > The result can only be one of > > > > reenter guest (and rcu_virt_note_context_switch) > > schedule() > > exit to userspace > > Very good, thank you both for bearing with me on this! > Hey, it's our pleasure. Pester us as often as you'll like and we'll happily "bear with you". -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RCU treating guest mode just like it does user-mode execution 2011-08-18 0:25 ` Paul E. McKenney 2011-08-18 0:28 ` Avi Kivity @ 2011-08-18 0:44 ` Gleb Natapov 1 sibling, 0 replies; 9+ messages in thread From: Gleb Natapov @ 2011-08-18 0:44 UTC (permalink / raw) To: Paul E. McKenney; +Cc: linux-kernel, avi, mtosatti, kvm On Wed, Aug 17, 2011 at 05:25:13PM -0700, Paul E. McKenney wrote: > On Thu, Aug 18, 2011 at 02:55:29AM +0300, Gleb Natapov wrote: > > On Wed, Aug 17, 2011 at 03:05:20PM -0700, Paul E. McKenney wrote: > > > On Thu, Aug 18, 2011 at 12:50:15AM +0300, Gleb Natapov wrote: > > > > On Wed, Aug 17, 2011 at 01:43:27PM -0700, Paul E. McKenney wrote: > > > > > Hello, Gleb, > > > > > > > > > > I was looking at KVM's call to rcu_virt_note_context_switch() > > > > > in kvm_guest_enter(), and noting the comment talking about treating > > > > > guest mode like user-mode execution is. One difference between RCU's > > > > > treatment of KVM guest execution and user-mode execution is that RCU > > > > > notes a context switch only at the beginning of KVM guest execution, > > > > > but notes user-mode execution at every scheduling-clock interrupt. > > > > > > > > > > Does it make sense to also note KVM guest execution on each > > > > > scheduling-clock interrupt? One reason it might not make sense is > > > > > if interrupts from KVM guest execution appear to rcu_check_callbacks() > > > > > as interrupts from user-mode execution. (Do they? Given that people > > > > > are reporting RCU CPU stall warnings in virtualized environments, I > > > > > am beginning to suspect that the answer is "no".) > > > > > > > > > The answer is "no" because any interrupt kicks cpu out of a guest mode, so > > > > it appears to be in the kernel for RCU. Do people still reporting RCU > > > > stalls even with the my patch? > > > > > > > > > If KVM guest execution does not appear as user-mode execution to > > > > > rcu_check_callback(), I would consider doing the following: > > > > > > > > > > 1. Rename rcu_virt_note_context_switch() to something like > > > > > rcu_guest_execution_start(). > > > > > > > > > > 2. Place a call to a new rcu_guest_execution_end() in > > > > > kvm_guest_exit(). > > > > > > > > > > 3. Make rcu_guest_execution_start() and rcu_guest_execution_end() > > > > > set and clear a new per-CPU variable. > > > > There is such variable already: current->flags & PF_VCPU. > > > > > > Good to know, thank you! > > > > > > > > 4. Make rcu_check_callbacks() check this per-CPU variable in > > > > > much the same way that it currently checks its "user" > > > > > argument, aside from needing to check that the CPU is > > > > > not in an interrupt handler or some such. > > > > > > > > > > Of course, some thought is required to make sure that the checks for > > > > > executing in an interrupt handler actually cover all of the needed > > > > > situations, but so it goes! > > > > > > > > > > Thoughts? > > > > > > > > I wonder why it will be better than current situation. After cpu leaves > > > > a guest mode there are only three options. It will either go to > > > > userspace, execute schedule or go back to guest mode. At all those cases > > > > RCU should note quiescent state. > > > > > > Might be that the current state is optimal. That would be a good thing. > > > > > > But if a CPU stays in guest mode for (say) 30 seconds, it will have > > > called schedule() every jiffy in the meantime? In other words, if > > > a CPU stays in guest mode for a long time, how does RCU know that > > > this CPU is in an extended quiescent state? > > > > > Wouldn't scheduling-clock interrupt kick vcpu out of a guest mode much > > earlier then 30 seconds? > > The scheduling-clock interrupt would happen, but I do not know whether > or not it would kick the vcpu out of guest mode in such a way that > would result in RCU thinking that the CPU has passed through a quiescent > state. Then I think we are OK. Any interrupt will kick cpu out of guest mode. After that vcpu thread will be either rescheduled or it will get back to guest mode calling rcu_virt_note_context_switch() on the way there. -- Gleb. ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2011-08-18 0:54 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-08-17 20:43 RCU treating guest mode just like it does user-mode execution Paul E. McKenney 2011-08-17 21:50 ` Gleb Natapov 2011-08-17 22:05 ` Paul E. McKenney 2011-08-17 23:55 ` Gleb Natapov 2011-08-18 0:25 ` Paul E. McKenney 2011-08-18 0:28 ` Avi Kivity 2011-08-18 0:44 ` Paul E. McKenney 2011-08-18 0:53 ` Avi Kivity 2011-08-18 0:44 ` Gleb Natapov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox