Atomicity & preemptive kernels

Atomicity & preemptive kernels

[Justin Carlson]

I know we're not there yet, but I'm trying to understand some issues
with rml's preemptive kernel and ASID's.

While doing a virtually-tagged hit invalidate of a cache, I was going to
write code something like this;

set_entryhi(CPU_CONTEXT(cpu, mm->vm_mm));
hit_invalidate_range(start, end);
set_entryhi(CPU_CONTEXT(cpu, current->mm));

Insofar as I understand current kernel scheduling guarantees, this is
safe because we won't reschedule while running in kernel mode.  But, if
I'm looking ahead to the preemptive kernel, then I think there is a
slight window for a race in between the reading of current->mm and 
the setting of entryhi.  Something like this:

current->mm->context is read
  * kernel reschedules.  
  * switch_mm() called
  * current->mm->context changes on return to this process
entryhi is set to the wrong context.

Is this a real race?  If so, is there any way around it other than
locally disabling interrupts around the restoration of the context?

-Justin
 

Re: Atomicity & preemptive kernels

[Jun Sun]

Justin Carlson wrote:

> I know we're not there yet, but I'm trying to understand some issues
> with rml's preemptive kernel and ASID's.
> 
> While doing a virtually-tagged hit invalidate of a cache, I was going to
> write code something like this;
> 
> set_entryhi(CPU_CONTEXT(cpu, mm->vm_mm));
> hit_invalidate_range(start, end);
> set_entryhi(CPU_CONTEXT(cpu, current->mm));
> 
> Insofar as I understand current kernel scheduling guarantees, this is
> safe because we won't reschedule while running in kernel mode.  But, if
> I'm looking ahead to the preemptive kernel, then I think there is a
> slight window for a race in between the reading of current->mm and 
> the setting of entryhi.  Something like this:
> 
> current->mm->context is read
>   * kernel reschedules.  
>   * switch_mm() called
>   * current->mm->context changes on return to this process
> entryhi is set to the wrong context.
> 
> Is this a real race? 


I am not sure if I am following your logic, but I don't see a race condition here.

Once current->mm is read into a register, the register is saved into stack 
when an interrupt happens (which later incurs a reschedule presumbably).  When 
the current preempted process comes back later, it goes back to the "tail" of 
do_IRQ(), followed by restoring the registers.  Since the register now holds 
the right value, set_entryhi() should be correct.

BTW, I have preemptiable kernel working fine under both UP and SMP.  If there 
is much interestes, I will publish it on the list.

Jun

Re: Atomicity & preemptive kernels

[Justin Carlson]

On Mon, 2002-06-10 at 15:14, Jun Sun wrote:
> I am not sure if I am following your logic, but I don't see a race condition here.
> 
> Once current->mm is read into a register, the register is saved into stack 
> when an interrupt happens (which later incurs a reschedule presumbably).  When 
> the current preempted process comes back later, it goes back to the "tail" of 
> do_IRQ(), followed by restoring the registers.  Since the register now holds 
> the right value, set_entryhi() should be correct.
> 

You've described exactly what happens.  The only problem is, it's
possible the underlying value for current->mm has changed.  It's a
*really* narrow window, at most a cycle or two, but I think it is
there.  In addition, even if you hit the window, to trigger wrong
behavior it requires that you also saturate the local ASID space,
invoking the tlb flush and asid reset in get_mmu_context().

The change that's introduced by the preemptive kernel is that
switch_mm() can be called after an interrupt.  So, with some
hypothetical assembly, the code flow looks like this:

	lw	$1, 120($29) ; Load current->mm->context into a register
         * Interrupt happens *
	 * reschedule happens, switch_mm() is called *
	   * get_new_mmu_context() invoked, starts a new ASID cycle.
	   * current->mm->context for the original process changes
	 * (sometime later) switch back to original process
	mtc0 	$entryhi, $1 ; stale context put back into entryhi!

Does that make more sense?  It's really a tiny race, but I think it's a
real one.  

-Justin
	   

Re: Atomicity & preemptive kernels

[Jun Sun]

Justin Carlson wrote:

> On Mon, 2002-06-10 at 15:14, Jun Sun wrote:
> 
>>I am not sure if I am following your logic, but I don't see a race condition here.
>>
>>Once current->mm is read into a register, the register is saved into stack 
>>when an interrupt happens (which later incurs a reschedule presumbably).  When 
>>the current preempted process comes back later, it goes back to the "tail" of 
>>do_IRQ(), followed by restoring the registers.  Since the register now holds 
>>the right value, set_entryhi() should be correct.
>>
>>
> 
> You've described exactly what happens.  The only problem is, it's
> possible the underlying value for current->mm has changed.  It's a
> *really* narrow window, at most a cycle or two, but I think it is
> there.  In addition, even if you hit the window, to trigger wrong
> behavior it requires that you also saturate the local ASID space,
> invoking the tlb flush and asid reset in get_mmu_context().
> 
> The change that's introduced by the preemptive kernel is that
> switch_mm() can be called after an interrupt.  So, with some
> hypothetical assembly, the code flow looks like this:
> 
> 	lw	$1, 120($29) ; Load current->mm->context into a register
>          * Interrupt happens *
> 	 * reschedule happens, switch_mm() is called *
> 	   * get_new_mmu_context() invoked, starts a new ASID cycle.
> 	   * current->mm->context for the original process changes
> 	 * (sometime later) switch back to original process
> 	mtc0 	$entryhi, $1 ; stale context put back into entryhi!
> 
> Does that make more sense?  It's really a tiny race, but I think it's a
> real one.  
> 


I see your point now.

However, the race is not there.  switch_mm() is only called from inside 
schedule() function, which as a whole is preemption-safe.  In other words, the 
above event sequence won't cause a context switch until we exit from 
schedule() function.

Jun

Re: Atomicity & preemptive kernels

[Justin Carlson]

On Tue, 2002-06-11 at 11:44, Jun Sun wrote:

> I see your point now.
> 
> However, the race is not there.  switch_mm() is only called from inside 
> schedule() function, which as a whole is preemption-safe.  In other words, the 
> above event sequence won't cause a context switch until we exit from 
> schedule() function.
> 

Yes, but the initial code I cited was explicitly not in the schedule
function.  It was an icache_flush_page() implementation.

Just something to be careful of.  I think it's going to bite more than a
few people as a bug that's about impossible to track down...

-Justin