Re: [PATCH v7 00/31] context_tracking,x86: Defer some IPIs until a user->kernel transition

Andy Lutomirski
2025-11-14T18:45:32Z

On Fri, Nov 14, 2025, at 10:14 AM, Paul E. McKenney wrote:
> On Fri, Nov 14, 2025 at 09:22:35AM -0800, Andy Lutomirski wrote:
>>

>> > Oh, any another primitive would be possible: one CPU could plausibly
>> > execute another CPU's interrupts or soft-irqs or whatever by taking a
>> > special lock that would effectively pin the remote CPU in user mode --
>> > you'd set a flag in the target cpu_flags saying "pin in USER mode" and
>> > the transition on that CPU to kernel mode would then spin on entry to
>> > kernel mode and wait for the lock to be released. This could plausibly
>> > get a lot of the on_each_cpu callers to switch over in one fell swoop:
>> > anything that needs to synchronize to the remote CPU but does not need
>> > to poke its actual architectural state could be executed locally while
>> > the remote CPU is pinned.
>
> It would be necessary to arrange for the remote CPU to remain pinned
> while the local CPU executed on its behalf. Does the above approach
> make that happen without re-introducing our current context-tracking
> overhead and complexity?

Using the pseudo-implementation farther down, I think this would be like:

if (my_state->status == INDETERMINATE) {
// we were lazy and we never promised to do work atomically.
atomic_set(&my_state->status, KERNEL);
this_entry_work = 0;
/* we are definitely not pinned in this path */
} else {
// we were not lazy and we promised we would do work atomically
atomic exchange the entire state to { .work = 0, .status = KERNEL }
this_entry_work = (whatever we just read);
if (this_entry_work & PINNED) {
u32 this_cpu_pin_count = this_cpu_ptr(pin_count);
while (atomic_read(&this_cpu_pin_count)) {
cpu_relax();
}
}
}

and we'd have something like:

bool try_pin_remote_cpu(int cpu)
{
u32 *remote_pin_count = ...;
struct fancy_cpu_state *remote_state = ...;
atomic_inc(remote_pin_count); // optimistic

// Hmm, we do not want that read to get reordered with the inc, so we probably
// need a full barrier or seq_cst. How does Linux spell that? C++ has atomic::load
// with seq_cst and maybe the optimizer can do the right thing. Maybe it's:
smp_mb__after_atomic();

if (atomic_read(&remote_state->status) == USER) {
// Okay, it's genuinely pinned.
return true;

// egads, if this is some arch with very weak ordering,
// do we need to be concerned that we just took a lock but we
// just did a relaxed read and therefore a subsequent access
// that thinks it's locked might appear to precede the load and therefore
// somehow get surprisingly seen out of order by the target cpu?
// maybe we wanted atomic_read_acquire above instead?
} else {
// We might not have successfully pinned it
atomic_dec(remote_pin_count);
}
}

void unpin_remote_cpu(int cpu)
{
atomic_dec(remote_pin_count();
}

and we'd use it like:

if (try_pin_remote_cpu(cpu)) {
// do something useful
} else {
send IPI;
}

but we'd really accumulate the set of CPUs that need the IPIs and do them all at once.

I ran the theorem prover that lives inside my head on this code using the assumption that the machine is a well-behaved x86 system and it said "yeah, looks like it might be correct". I trust an actual formalized system or someone like you who is genuinely very good at this stuff much more than I trust my initial impression :)

>
>> Following up, I think that x86 can do this all with a single atomic (in the common case) per usermode round trip. Imagine:
>>
>> struct fancy_cpu_state {
>> u32 work; // <-- writable by any CPU
>> u32 status; // <-- readable anywhere; writable locally
>> };
>>
>> status includes KERNEL, USER, and maybe INDETERMINATE. (INDETERMINATE means USER but we're not committing to doing work.)
>>
>> Exit to user mode:
>>
>> atomic_set(&my_state->status, USER);
>
> We need ordering in the RCU nohz_full case. If the grace-period kthread
> sees the status as USER, all the preceding KERNEL code's effects must
> be visible to the grace-period kthread.

Sorry, I'm speaking lazy x86 programmer here. Maybe I mean atomic_set_release. I want, roughly, the property that anyone who remotely observes USER can rely on the target cpu subsequently going through the atomic exchange path above. I think even relaxed ought to be good enough for that one most architectures, but there are some potentially nasty complications involving that fact that this mixes operations on a double word and a single word that's part of the double word.

>
>> (or, in the lazy case, set to INDETERMINATE instead.)
>>
>> Entry from user mode, with IRQs off, before switching to kernel CR3:
>>
>> if (my_state->status == INDETERMINATE) {
>> // we were lazy and we never promised to do work atomically.
>> atomic_set(&my_state->status, KERNEL);
>> this_entry_work = 0;
>> } else {
>> // we were not lazy and we promised we would do work atomically
>> atomic exchange the entire state to { .work = 0, .status = KERNEL }
>> this_entry_work = (whatever we just read);
>> }
>
> If this atomic exchange is fully ordered (as opposed to, say, _relaxed),
> then this works in that if the grace-period kthread sees USER, its prior
> references are guaranteed not to see later kernel-mode references from
> that CPU.

Yep, that's the intent of my pseudocode. On x86 this would be a plain 64-bit lock xchg -- I don't think cmpxchg is needed. (I like to write lock even when it's implicit to avoid needing to trust myself to remember precisely which instructions imply it.)

>
>> if (PTI) {
>> switch to kernel CR3 *and flush if this_entry_work says to flush*
>> } else {
>> flush if this_entry_work says to flush;
>> }
>>
>> do the rest of the work;
>>
>>
>>
>> I suppose that a lot of the stuff in ti_flags could merge into here, but it could be done one bit at a time when people feel like doing so. And I imagine, but I'm very far from confident, that RCU could use this instead of the current context tracking code.
>
> RCU currently needs pretty heavy-duty ordering to reliably detect the
> other CPUs' quiescent states without needing to wake them from idle, or,
> in the nohz_full case, interrupt their userspace execution. Not saying
> it is impossible, but it will need extreme care.

Is the thingy above heavy-duty enough? Perhaps more relevantly, could RCU do this *instead* of the current CT hooks on architectures that implement it, and/or could RCU arrange for the ct hooks to be cheap no-ops on architectures that support the thingy above. By "could" I mean "could be done without absolutely massive refactoring and without the resulting code being an unmaintainable disaster? I'm sure any sufficiently motivated human or LLM could pull off the unmaintainable disaster version. I bet I could even do it myself!)

>
>> The idea behind INDETERMINATE is that there are plenty of workloads that frequently switch between user and kernel mode and that would rather accept a few IPIs to avoid the heavyweight atomic operation on user -> kernel transitions. So the default behavior could be to do KERNEL -> INDETERMINATE instead of KERNEL -> USER, but code that wants to be in user mode for a long time could go all the way to USER. We could make it sort of automatic by noticing that we're returning from an IRQ without a context switch and go to USER (so we would get at most one unneeded IPI per normal user entry), and we could have some nice API for a program that intends to hang out in user mode for a very long time (cpu isolation users, for example) to tell the kernel to go immediately into USER mode. (Don't we already have something that could be used for this purpose?)
>
> RCU *could* do an smp_call_function_single() when the CPU failed
> to respond, perhaps in a manner similar to how it already forces a
> given CPU out of nohz_full state if that CPU has been executing in the
> kernel for too long. The real-time guys might not be amused, though.
> Especially those real-time guys hitting sub-microsecond latencies.

It wouldn't be outrageous to have real-time imply the full USER transition.

>
>> Hmm, now I wonder if it would make sense for the default behavior of Linux to be like that. We could call it ONEHZ. It's like NOHZ_FULL except that user threads that don't do syscalls get one single timer tick instead of many or none.
>>
>>
>> Anyway, I think my proposal is pretty good *if* RCU could be made to use it -- the existing context tracking code is fairly expensive, and I don't think we want to invent a new context-tracking-like mechanism if we still need to do the existing thing.
>
> If you build with CONFIG_NO_HZ_FULL=n, do you still get the heavyweight
> operations when transitioning between kernel and user execution?

No. And I even tested approximately this a couple weeks ago for unrelated reasons. The only unnecessary heavy-weight thing we're doing in a syscall loop with mitigations off is the RDTSC for stack randomization.

But I did that test on a machine that would absolutely benefit from the IPI suppression that the OP is talking about here, and I think it would be really quite nice if a default distro kernel with a more or less default distribution could be easily convinced to run a user thread without interrupting it. It's a little bit had that NO_HZ_FULL is still an exotic non-default thing, IMO.

--Andy