Re: [PATCH v3 sched_ext/for-7.3 25/36] sched_ext: Add the SCX_CAP_ENQ_IMMED cap

[Andrea Righi <arighi@nvidia.com>]

Hi Tejun,

On Mon, Jul 06, 2026 at 02:12:18PM -1000, Tejun Heo wrote:
> Replace the __SCX_CAP_DUMMY placeholder with SCX_CAP_ENQ_IMMED, which gates
> inserting IMMED tasks onto a cid's local DSQ. An IMMED enqueue is guaranteed
> to either get its task running on the cpu at once or hand it back to the
> scheduler, so IMMED work can never pile up on the cpu's queue and a cpu can
> be shared across sub-scheds through IMMED access without any of them
> swamping it.
> 
> That makes ENQ_IMMED the natural baseline, the minimal cap to make any use
> of a cpu. SCX_CAP_BASE aliases it so gates on basic cpu access can state the
> intention instead of naming ENQ_IMMED.
> 
> Enforcement covers inserts and queued tasks. An insert without the cap is
> diverted to the reject DSQ, and queued tasks are reenqueued when the cap is
> lost. scx_bpf_sub_dispatch() skips a child that lacks the cap on the cpu, as
> its inserts would only be rejected. Vacating the running task on cap loss
> lands in a later patch.

I was able to trigger the following RCU stall with scx_qmap attached to a "test"
cgroup and running `stress-ng --race-sched 0` inside it:

[  183.913108] sched_ext: BPF sub-scheduler "qmap" enabled
[  209.915162] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[  209.915521] rcu: 	6-....: (0 ticks this GP) idle=a794/1/0x4000000000000000 softirq=3550/3550 fqs=6500
[  209.916217] rcu: 	7-....: (2 GPs behind) idle=fc7c/1/0x4000000000000000 softirq=3550/3551 fqs=6500
[  209.916581] rcu: 	(detected by 0, t=26003 jiffies, g=35149, q=153 ncpus=16)
[  209.916799] Sending NMI from CPU 0 to CPUs 6:
[  209.916820] NMI backtrace for cpu 6
[  209.916827] CPU: 6 UID: 1000 PID: 12259 Comm: stress-ng-race- Not tainted 7.1.0-virtme #1 PREEMPT(full)
[  209.916833] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[  209.916836] Sched_ext: qmap (enabled+all), task: runnable_at=-26004ms
[  209.916839] RIP: 0010:check_chain_key+0x15e/0x1a0
[  209.916849] Code: f9 67 48 0f b9 3a eb 21 e8 af cb 50 00 48 8d 3d 38 3d 45 02 41 8b b6 40 0c 00 00 4d 8b 04 24 89 da 4c 89 f9 67 48 0f b9 3a 5b <41> 5c 41 5e 41 5f c3 cc cc cc cc cc 90 e8 80 cb 50 00 85 c0 74 1c
[  209.916852] RSP: 0018:ffffc900001d8de0 EFLAGS: 00000046
[  209.916856] RAX: ffff8881392253c8 RBX: ffff888139224780 RCX: ffff8881392253f0
[  209.916858] RDX: 0000000000000001 RSI: 00000000e23450be RDI: ffff888139224780
[  209.916860] RBP: 0000000000004051 R08: 000000000000e304 R09: 0000000000004000
[  209.916861] R10: ffff8882b381a000 R11: ffffffffc002f373 R12: ffff8881392253c8
[  209.916863] R13: 35590c0ea7cde75a R14: ffff888139224780 R15: 35590c0ea7cde75a
[  209.916870] FS:  00007fb7c6b86b00(0000) GS:ffff8882b381a000(0000) knlGS:0000000000000000
[  209.916876] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  209.916878] CR2: 00007ffd98a3b7de CR3: 0000000138ec8000 CR4: 0000000000750ef0
[  209.916885] PKRU: 55555554
[  209.916889] Call Trace:
[  209.916892]  <IRQ>
[  209.916896]  __lock_acquire+0xcf9/0x2a40
[  209.916905]  ? __irq_work_queue_local+0x67/0x120
[  209.916909]  ? arch_irq_work_raise+0x26/0x30
[  209.916913]  ? irq_work_queue_on+0x3e/0xa0
[  209.916915]  ? scx_dispatch_enqueue+0x2f4/0x460
[  209.916922]  ? ktime_get+0x1e/0xf0
[  209.916925]  lock_acquire+0xd3/0x280
[  209.916929]  ? ktime_get+0x1e/0xf0
[  209.916932]  ? irq_work_run_list+0x93/0xc0
[  209.916935]  ? ktime_get+0x1e/0xf0
[  209.916938]  seqcount_lockdep_reader_access+0x34/0xc0
[  209.916941]  ? ktime_get+0x1e/0xf0
[  209.916944]  ktime_get+0x1e/0xf0
[  209.916947]  tick_nohz_irq_exit+0x4c/0xb0
[  209.916951]  irq_exit_rcu+0xe/0x20
[  209.916956]  sysvec_irq_work+0x73/0x80
[  209.916961]  </IRQ>
[  209.916961]  <TASK>
[  209.916963]  asm_sysvec_irq_work+0x1a/0x20
[  209.916966] RIP: 0010:preempt_schedule_irq+0x4d/0x70

I bisected it and this was to be the offending commit.

IIUC, what's happening is that qmap grants the child CPUs 2–3, but race-sched
stressor can pin its tasks to CPU 1. Since the child has no enqueue capability
there, the tasks repeatedly land on the reject DSQ and eventually trigger the
RCU stall.

Maybe we need a forced fallback when a task's affinity mask has no active CPU on
which its scheduler holds SCX_CAP_BASE?

Checking this only during sched_setaffinity() won't work because capabilities
can be revoked after the affinity change succeeds. What do you think?

Thanks,
-Andrea

> 
> Signed-off-by: Tejun Heo <tj@kernel.org>
> ---
>  kernel/sched/ext/ext.c      |  2 +-
>  kernel/sched/ext/internal.h | 11 ++++++++---
>  kernel/sched/ext/sub.c      |  8 ++++++++
>  kernel/sched/ext/sub.h      |  4 ++--
>  4 files changed, 19 insertions(+), 6 deletions(-)
> 
> diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c
> index d0966412ee42..d4e4340fdcd8 100644
> --- a/kernel/sched/ext/ext.c
> +++ b/kernel/sched/ext/ext.c
> @@ -4752,7 +4752,7 @@ SCX_ATTR(events);
>  
>  #ifdef CONFIG_EXT_SUB_SCHED
>  static const char *scx_cap_names[__SCX_NR_CAPS] = {
> -	[__SCX_CAP_DUMMY]	= "dummy",
> +	[__SCX_CAP_ENQ_IMMED]	= "enq_immed",
>  };
>  
>  static ssize_t scx_attr_caps_show(struct kobject *kobj,
> diff --git a/kernel/sched/ext/internal.h b/kernel/sched/ext/internal.h
> index 7cc7e10457d7..5d4edd505ed1 100644
> --- a/kernel/sched/ext/internal.h
> +++ b/kernel/sched/ext/internal.h
> @@ -1270,17 +1270,22 @@ struct scx_sched_pnode {
>   * topology-aligned and likely to serve as the locality unit when cids are
>   * allocated to schedulers, so per-shard lock granularity scales naturally with
>   * the allocation pattern.
> + *
> + * ENQ_IMMED  insert an IMMED task onto the cid's local DSQ
>   */
>  enum scx_cap_flags {
> -	__SCX_CAP_DUMMY			= 0,
> +	__SCX_CAP_ENQ_IMMED		= 0,
>  
>  	__SCX_NR_CAPS,
>  	__SCX_CAP_ALL			= BIT_U64(__SCX_NR_CAPS) - 1,
>  
> -	SCX_CAP_DUMMY			= BIT_U64(__SCX_CAP_DUMMY),
> +	SCX_CAP_ENQ_IMMED		= BIT_U64(__SCX_CAP_ENQ_IMMED),
> +
> +	/* alias for minimal cap to make any use of a cpu */
> +	SCX_CAP_BASE			= SCX_CAP_ENQ_IMMED,
>  
>  	/* caps whose loss strands queued tasks, see scx_process_sync_ecaps() */
> -	SCX_CAPS_REENQ_ON_LOSS		= 0,
> +	SCX_CAPS_REENQ_ON_LOSS		= SCX_CAP_ENQ_IMMED,
>  };
>  
>  #ifdef CONFIG_EXT_SUB_SCHED
> diff --git a/kernel/sched/ext/sub.c b/kernel/sched/ext/sub.c
> index 23686c609336..778ae3f6f91f 100644
> --- a/kernel/sched/ext/sub.c
> +++ b/kernel/sched/ext/sub.c
> @@ -1270,6 +1270,14 @@ __bpf_kfunc bool scx_bpf_sub_dispatch(u64 cgroup_id, const struct bpf_prog_aux *
>  		return false;
>  	}
>  
> +	/*
> +	 * Skip a child that does not effectively hold the base cap on this cpu:
> +	 * its inserts would only be rejected. ecaps are synced at the top of
> +	 * balance_one() before dispatch, so this reflects the in-effect state.
> +	 */
> +	if (scx_missing_caps(child, cpu_of(this_rq), SCX_CAP_BASE))
> +		return false;
> +
>  	return scx_dispatch_sched(child, this_rq, this_rq->scx.sub_dispatch_prev,
>  				  true);
>  }
> diff --git a/kernel/sched/ext/sub.h b/kernel/sched/ext/sub.h
> index 51400e174cd0..f3548976800b 100644
> --- a/kernel/sched/ext/sub.h
> +++ b/kernel/sched/ext/sub.h
> @@ -106,13 +106,13 @@ static inline u64 scx_missing_caps(struct scx_sched *sch, s32 cpu, u64 needed)
>  /* map @enq_flags to the SCX_CAP_* bit required for the local-DSQ insert */
>  static inline u64 scx_caps_for_enq(u64 enq_flags)
>  {
> -	return 0;
> +	return SCX_CAP_ENQ_IMMED;
>  }
>  
>  /* map queued @p to the SCX_CAP_* bit required to stay on its local DSQ */
>  static inline u64 scx_caps_for_task(struct task_struct *p)
>  {
> -	return 0;
> +	return SCX_CAP_ENQ_IMMED;
>  }
>  
>  /* caps implied by holding @cap */
> -- 
> 2.54.0
>