* Re: [PATCH] scx: Fix maximal BPF selftest prog [not found] <20241209152924.4508-1-void@manifault.com> @ 2024-12-10 20:52 ` Ihor Solodrai 2024-12-11 21:01 ` [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix invalid irq restore in scx_ops_bypass() Tejun Heo 0 siblings, 1 reply; 20+ messages in thread From: Ihor Solodrai @ 2024-12-10 20:52 UTC (permalink / raw) To: David Vernet; +Cc: sched-ext, kernel-team, linux-kernel, bpf, tj On Monday, December 9th, 2024 at 7:29 AM, David Vernet <void@manifault.com> wrote: > > > maximal.bpf.c is still dispatching to and consuming from SCX_DSQ_GLOBAL. > Let's have it use its own DSQ to avoid any runtime errors. > > Signed-off-by: David Vernet void@manifault.com > > --- > tools/testing/selftests/sched_ext/maximal.bpf.c | 8 +++++--- > 1 file changed, 5 insertions(+), 3 deletions(-) Hi David, thanks for looking into the test failures. I ran BPF CI on kernel with this change on top of bpf-next (6e8ba494d87d) + tests build fix [1]. I ran the workflow twice. A second run on aarch64 failed: https://github.com/kernel-patches/vmtest/actions/runs/12263276418/job/34217056880 Failed tests are "dsp_local_on" and "exit". The difference in comparison to a successful run is a deadlock, which happens on attempt to attach a scheduler, assuming I read the log correctly. See a paste below. I'd like to note that the selftests/sched_ext/runner log is very verbose (~23M), although I can't tell how necessary this is for debugging. [1] https://lore.kernel.org/bpf/20241121214014.3346203-1-ihor.solodrai@pm.me/ 2024-12-10T20:20:55.6921607Z ##[group]selftests/sched_ext - Executing selftests/sched_ext/runner 2024-12-10T20:20:55.7556144Z 2024-12-10T20:20:55.7558626Z ================================ 2024-12-10T20:20:55.7560796Z WARNING: inconsistent lock state 2024-12-10T20:20:55.7564132Z 6.13.0-rc2-gb23d77932777-dirty #1 Not tainted 2024-12-10T20:20:55.7566359Z -------------------------------- 2024-12-10T20:20:55.7570239Z inconsistent {IN-HARDIRQ-W} -> {HARDIRQ-ON-W} usage. 2024-12-10T20:20:55.7573094Z runner/96 [HC0[0]:SC0[0]:HE1:SE1] takes: 2024-12-10T20:20:55.7579001Z ffff0000fb5f4858 (&rq->__lock){?.-.}-{2:2}, at: raw_spin_rq_lock_nested+0x2c/0x50 2024-12-10T20:20:55.7581586Z {IN-HARDIRQ-W} state was registered at: 2024-12-10T20:20:55.7583722Z __lock_acquire+0x3c0/0xb18 2024-12-10T20:20:55.7586124Z lock_acquire.part.0+0xe0/0x250 2024-12-10T20:20:55.7588061Z lock_acquire+0x88/0x160 2024-12-10T20:20:55.7590605Z _raw_spin_lock_nested+0x5c/0xb0 2024-12-10T20:20:55.7592640Z try_to_wake_up+0x244/0x560 2024-12-10T20:20:55.7594719Z wake_up_process+0x20/0x38 2024-12-10T20:20:55.7596673Z hrtimer_wakeup+0x28/0x48 2024-12-10T20:20:55.7598757Z __run_hrtimer+0x32c/0x3a0 2024-12-10T20:20:55.7601330Z __hrtimer_run_queues+0xb4/0x140 2024-12-10T20:20:55.7603618Z hrtimer_run_queues+0xe4/0x1d8 2024-12-10T20:20:55.7606009Z update_process_times+0x3c/0x160 2024-12-10T20:20:55.7607957Z tick_periodic+0x44/0x140 2024-12-10T20:20:55.7610565Z tick_handle_periodic+0x34/0xa8 2024-12-10T20:20:55.7613159Z arch_timer_handler_virt+0x34/0x58 2024-12-10T20:20:55.7615811Z handle_percpu_devid_irq+0xb0/0x2a0 2024-12-10T20:20:55.7618562Z generic_handle_domain_irq+0x34/0x58 2024-12-10T20:20:55.7620462Z gic_handle_irq+0x5c/0xd8 2024-12-10T20:20:55.7622667Z call_on_irq_stack+0x24/0x58 2024-12-10T20:20:55.7625060Z do_interrupt_handler+0x88/0x98 2024-12-10T20:20:55.7626970Z el1_interrupt+0x34/0x50 2024-12-10T20:20:55.7629327Z el1h_64_irq_handler+0x18/0x28 2024-12-10T20:20:55.7631094Z el1h_64_irq+0x6c/0x70 2024-12-10T20:20:55.7633416Z default_idle_call+0xac/0x254 2024-12-10T20:20:55.7635665Z default_idle_call+0xa8/0x254 2024-12-10T20:20:55.7637184Z do_idle+0xc4/0x118 2024-12-10T20:20:55.7639486Z cpu_startup_entry+0x3c/0x50 2024-12-10T20:20:55.7641200Z rest_init+0x118/0x1a8 2024-12-10T20:20:55.7643382Z start_kernel+0x59c/0x678 2024-12-10T20:20:55.7645786Z __primary_switched+0x88/0x98 2024-12-10T20:20:55.7647582Z irq event stamp: 33739 2024-12-10T20:20:55.7653885Z hardirqs last enabled at (33739): [<ffff8000800b699c>] scx_ops_bypass+0x174/0x3b8 2024-12-10T20:20:55.7660311Z hardirqs last disabled at (33738): [<ffff800080d48ad4>] _raw_spin_lock_irqsave+0xb4/0xd8 2024-12-10T20:20:55.7666393Z softirqs last enabled at (32824): [<ffff800080056a34>] handle_softirqs+0x46c/0x490 2024-12-10T20:20:55.7672087Z softirqs last disabled at (32815): [<ffff8000800101bc>] __do_softirq+0x1c/0x28 2024-12-10T20:20:55.7672575Z 2024-12-10T20:20:55.7675191Z other info that might help us debug this: 2024-12-10T20:20:55.7678564Z Possible unsafe locking scenario: 2024-12-10T20:20:55.7678867Z 2024-12-10T20:20:55.7680124Z CPU0 2024-12-10T20:20:55.7680709Z ---- 2024-12-10T20:20:55.7682424Z lock(&rq->__lock); 2024-12-10T20:20:55.7683722Z <Interrupt> 2024-12-10T20:20:55.7685546Z lock(&rq->__lock); 2024-12-10T20:20:55.7685788Z 2024-12-10T20:20:55.7687156Z *** DEADLOCK *** 2024-12-10T20:20:55.7687380Z 2024-12-10T20:20:55.7689485Z 3 locks held by runner/96: 2024-12-10T20:20:55.7696730Z #0: ffff8000821226f8 (update_mutex){+.+.}-{4:4}, at: bpf_struct_ops_link_create+0x13c/0x1b0 2024-12-10T20:20:55.7703773Z #1: ffff80008207e7f0 (scx_ops_enable_mutex){+.+.}-{4:4}, at: scx_ops_enable.isra.0+0x88/0xaa8 2024-12-10T20:20:55.7709594Z #2: ffff80008207e060 (bypass_lock){+.+.}-{2:2}, at: scx_ops_bypass+0x50/0x3b8 2024-12-10T20:20:55.7710078Z 2024-12-10T20:20:55.7710895Z stack backtrace: 2024-12-10T20:20:55.7717184Z CPU: 1 UID: 0 PID: 96 Comm: runner Not tainted 6.13.0-rc2-gb23d77932777-dirty #1 2024-12-10T20:20:55.7719782Z Hardware name: linux,dummy-virt (DT) 2024-12-10T20:20:55.7722429Z Sched_ext: create_dsq (enabling) 2024-12-10T20:20:55.7723286Z Call trace: 2024-12-10T20:20:55.7725440Z show_stack+0x20/0x38 (C) 2024-12-10T20:20:55.7727531Z dump_stack_lvl+0xa0/0xf0 2024-12-10T20:20:55.7729283Z dump_stack+0x18/0x28 2024-12-10T20:20:55.7732218Z print_usage_bug.part.0+0x270/0x320 2024-12-10T20:20:55.7734277Z mark_lock_irq+0x3e0/0x538 2024-12-10T20:20:55.7736126Z mark_lock+0x1c0/0x290 2024-12-10T20:20:55.7738113Z mark_usage+0x108/0x170 2024-12-10T20:20:55.7740306Z __lock_acquire+0x3c0/0xb18 2024-12-10T20:20:55.7742923Z lock_acquire.part.0+0xe0/0x250 2024-12-10T20:20:55.7744782Z lock_acquire+0x88/0x160 2024-12-10T20:20:55.7747510Z _raw_spin_lock_nested+0x5c/0xb0 2024-12-10T20:20:55.7750224Z raw_spin_rq_lock_nested+0x2c/0x50 2024-12-10T20:20:55.7752314Z scx_ops_bypass+0xfc/0x3b8 2024-12-10T20:20:55.7755093Z scx_ops_enable.isra.0+0x2c8/0xaa8 2024-12-10T20:20:55.7756819Z bpf_scx_reg+0x18/0x30 2024-12-10T20:20:55.7760072Z bpf_struct_ops_link_create+0x154/0x1b0 2024-12-10T20:20:55.7762017Z link_create+0x14c/0x350 2024-12-10T20:20:55.7763821Z __sys_bpf+0x3fc/0xb90 2024-12-10T20:20:55.7765976Z __arm64_sys_bpf+0x2c/0x48 2024-12-10T20:20:55.7768134Z invoke_syscall+0x50/0x120 2024-12-10T20:20:55.7771191Z el0_svc_common.constprop.0+0x48/0xf0 2024-12-10T20:20:55.7772796Z do_el0_svc+0x24/0x38 2024-12-10T20:20:55.7774452Z el0_svc+0x48/0x110 2024-12-10T20:20:55.7777321Z el0t_64_sync_handler+0x10c/0x138 2024-12-10T20:20:55.7779253Z el0t_64_sync+0x198/0x1a0 2024-12-10T20:20:55.7782673Z ------------[ cut here ]------------ 2024-12-10T20:20:55.7786247Z raw_local_irq_restore() called with IRQs enabled 2024-12-10T20:20:55.7792987Z WARNING: CPU: 1 PID: 96 at kernel/locking/irqflag-debug.c:10 warn_bogus_irq_restore+0x30/0x40 2024-12-10T20:20:55.7794104Z Modules linked in: 2024-12-10T20:20:55.7799949Z CPU: 1 UID: 0 PID: 96 Comm: runner Not tainted 6.13.0-rc2-gb23d77932777-dirty #1 2024-12-10T20:20:55.7802378Z Hardware name: linux,dummy-virt (DT) 2024-12-10T20:20:55.7804767Z Sched_ext: create_dsq (enabling) 2024-12-10T20:20:55.7809450Z pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) 2024-12-10T20:20:55.7812011Z pc : warn_bogus_irq_restore+0x30/0x40 2024-12-10T20:20:55.7814814Z lr : warn_bogus_irq_restore+0x30/0x40 2024-12-10T20:20:55.7816303Z sp : ffff800084d53a50 2024-12-10T20:20:55.7821165Z x29: ffff800084d53a50 x28: ffff0000c0960000 x27: ffff80008205d078 2024-12-10T20:20:55.7825770Z x26: ffff800081cf6840 x25: ffff800082059350 x24: ffff800082256958 2024-12-10T20:20:55.7830346Z x23: 0000000000000000 x22: ffff0000fb5f4858 x21: ffff80008207d6b8 2024-12-10T20:20:55.7834929Z x20: 0000000000000001 x19: ffff0000fb5f4840 x18: 00000000fffffffd 2024-12-10T20:20:55.7839534Z x17: 3764333262672d32 x16: 63722d302e33312e x15: ffff800084d52c90 2024-12-10T20:20:55.7844210Z x14: 0000000000000000 x13: 64656c62616e6520 x12: 7351524920687469 2024-12-10T20:20:55.7849054Z x11: 772064656c6c6163 x10: ffff8000820da8b0 x9 : ffff800080101b00 2024-12-10T20:20:55.7854102Z x8 : 00000000ffffefff x7 : ffff8000820da8b0 x6 : 80000000fffff000 2024-12-10T20:20:55.7858669Z x5 : 000000000000017f x4 : 0000000000000000 x3 : 0000000000000000 2024-12-10T20:20:55.7863487Z x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff0000c0960000 2024-12-10T20:20:55.7864284Z Call trace: 2024-12-10T20:20:55.7867341Z warn_bogus_irq_restore+0x30/0x40 (P) 2024-12-10T20:20:55.7870193Z warn_bogus_irq_restore+0x30/0x40 (L) 2024-12-10T20:20:55.7872308Z scx_ops_bypass+0x224/0x3b8 2024-12-10T20:20:55.7875067Z scx_ops_enable.isra.0+0x2c8/0xaa8 2024-12-10T20:20:55.7876915Z bpf_scx_reg+0x18/0x30 2024-12-10T20:20:55.7880088Z bpf_struct_ops_link_create+0x154/0x1b0 2024-12-10T20:20:55.7881875Z link_create+0x14c/0x350 2024-12-10T20:20:55.7883755Z __sys_bpf+0x3fc/0xb90 2024-12-10T20:20:55.7885864Z __arm64_sys_bpf+0x2c/0x48 2024-12-10T20:20:55.7887978Z invoke_syscall+0x50/0x120 2024-12-10T20:20:55.7890961Z el0_svc_common.constprop.0+0x48/0xf0 2024-12-10T20:20:55.7892546Z do_el0_svc+0x24/0x38 2024-12-10T20:20:55.7894224Z el0_svc+0x48/0x110 2024-12-10T20:20:55.7896940Z el0t_64_sync_handler+0x10c/0x138 2024-12-10T20:20:55.7898846Z el0t_64_sync+0x198/0x1a0 2024-12-10T20:20:55.7900627Z irq event stamp: 33739 2024-12-10T20:20:55.7906972Z hardirqs last enabled at (33739): [<ffff8000800b699c>] scx_ops_bypass+0x174/0x3b8 2024-12-10T20:20:55.7913438Z hardirqs last disabled at (33738): [<ffff800080d48ad4>] _raw_spin_lock_irqsave+0xb4/0xd8 2024-12-10T20:20:55.7919488Z softirqs last enabled at (32824): [<ffff800080056a34>] handle_softirqs+0x46c/0x490 2024-12-10T20:20:55.7925161Z softirqs last disabled at (32815): [<ffff8000800101bc>] __do_softirq+0x1c/0x28 2024-12-10T20:20:55.7927698Z ---[ end trace 0000000000000000 ]--- 2024-12-10T20:20:55.8646931Z sched_ext: BPF scheduler "create_dsq" enabled 2024-12-10T20:20:55.9047921Z sched_ext: BPF scheduler "create_dsq" disabled (unregistered from user space) 2024-12-10T20:20:55.9647423Z sched_ext: BPF scheduler "enq_last_no_enq_fails" disabled (runtime error) 2024-12-10T20:20:55.9653845Z sched_ext: enq_last_no_enq_fails: SCX_OPS_ENQ_LAST requires ops.enqueue() to be implemented 2024-12-10T20:20:55.9656238Z scx_ops_enable.isra.0+0xa98/0xaa8 2024-12-10T20:20:55.9658038Z bpf_scx_reg+0x18/0x30 2024-12-10T20:20:55.9661235Z bpf_struct_ops_link_create+0x154/0x1b0 2024-12-10T20:20:55.9663113Z link_create+0x14c/0x350 2024-12-10T20:20:55.9664999Z __sys_bpf+0x3fc/0xb90 2024-12-10T20:20:55.9667216Z __arm64_sys_bpf+0x2c/0x48 2024-12-10T20:20:55.9669376Z invoke_syscall+0x50/0x120 2024-12-10T20:20:55.9672393Z el0_svc_common.constprop.0+0x48/0xf0 2024-12-10T20:20:55.9674125Z do_el0_svc+0x24/0x38 2024-12-10T20:20:55.9675781Z el0_svc+0x48/0x110 2024-12-10T20:20:55.9678659Z el0t_64_sync_handler+0x10c/0x138 2024-12-10T20:20:55.9680560Z el0t_64_sync+0x198/0x1a0 2024-12-10T20:20:55.9864103Z sched_ext: BPF scheduler "enq_select_cpu_fails" enabled 2024-12-10T20:20:56.0347550Z sched_ext: BPF scheduler "enq_select_cpu_fails" disabled (runtime error) 2024-12-10T20:20:56.0354154Z sched_ext: enq_select_cpu_fails: kfunc with mask 0x8 called from an operation only allowing 0x4 2024-12-10T20:20:56.0356414Z scx_bpf_select_cpu_dfl+0x8c/0x98 2024-12-10T20:20:56.0361395Z bpf_prog_b68dda6ca71e9089_enq_select_cpu_fails_enqueue+0x54/0xb0 2024-12-10T20:20:56.0364138Z bpf__sched_ext_ops_enqueue+0x50/0x74 2024-12-10T20:20:56.0366225Z do_enqueue_task+0xf8/0x240 2024-12-10T20:20:56.0368567Z enqueue_task_scx+0x190/0x288 2024-12-10T20:20:56.0370532Z enqueue_task+0x44/0xe8 2024-12-10T20:20:56.0372845Z ttwu_do_activate+0x88/0x298 2024-12-10T20:20:56.0375059Z try_to_wake_up+0x2b8/0x560 2024-12-10T20:20:56.0377269Z wake_up_process+0x20/0x38 2024-12-10T20:20:56.0379030Z kick_pool+0xa4/0x190 2024-12-10T20:20:56.0381198Z __queue_work+0x540/0x600 2024-12-10T20:20:56.0383276Z queue_work_on+0xb8/0x100 2024-12-10T20:20:56.0385314Z bpf_prog_free+0x90/0xa8 2024-12-10T20:20:56.0387681Z __bpf_prog_put_rcu+0x44/0x60 2024-12-10T20:20:56.0389757Z rcu_do_batch+0x1e8/0xa90 2024-12-10T20:20:56.0391541Z rcu_core+0x174/0x378 2024-12-10T20:20:56.0393744Z rcu_core_si+0x18/0x30 2024-12-10T20:20:56.0395820Z handle_softirqs+0x12c/0x490 2024-12-10T20:20:56.0397644Z __do_softirq+0x1c/0x28 2024-12-10T20:20:56.0399798Z ____do_softirq+0x18/0x30 2024-12-10T20:20:56.0402104Z call_on_irq_stack+0x24/0x58 2024-12-10T20:20:56.0404662Z do_softirq_own_stack+0x24/0x38 2024-12-10T20:20:56.0406780Z __irq_exit_rcu+0x140/0x180 2024-12-10T20:20:56.0408766Z irq_exit_rcu+0x18/0x48 2024-12-10T20:20:56.0410801Z el1_interrupt+0x38/0x50 2024-12-10T20:20:56.0413210Z el1h_64_irq_handler+0x18/0x28 2024-12-10T20:20:56.0415052Z el1h_64_irq+0x6c/0x70 2024-12-10T20:20:56.0417472Z default_idle_call+0xac/0x254 2024-12-10T20:20:56.0419830Z default_idle_call+0xa8/0x254 2024-12-10T20:20:56.0421507Z do_idle+0xc4/0x118 2024-12-10T20:20:56.0423811Z cpu_startup_entry+0x3c/0x50 2024-12-10T20:20:56.0426578Z secondary_start_kernel+0xdc/0x108 2024-12-10T20:20:56.0428987Z __secondary_switched+0xc0/0xc8 2024-12-10T20:20:57.1047514Z sched_ext: BPF scheduler "ddsp_bogus_dsq_fail" enabled 2024-12-10T20:20:57.1647261Z sched_ext: BPF scheduler "ddsp_bogus_dsq_fail" disabled (runtime error) 2024-12-10T20:20:57.1652666Z sched_ext: ddsp_bogus_dsq_fail: non-existent DSQ 0xcafef00d for rcu_sched[16] 2024-12-10T20:20:57.1655003Z find_dsq_for_dispatch+0xf4/0x158 2024-12-10T20:20:57.1657196Z direct_dispatch+0x58/0x228 2024-12-10T20:20:57.1659464Z do_enqueue_task+0x17c/0x240 2024-12-10T20:20:57.1661801Z enqueue_task_scx+0x190/0x288 2024-12-10T20:20:57.1663937Z enqueue_task+0x44/0xe8 2024-12-10T20:20:57.1666068Z ttwu_do_activate+0x88/0x298 2024-12-10T20:20:57.1668231Z try_to_wake_up+0x2b8/0x560 2024-12-10T20:20:57.1670117Z swake_up_one+0x4c/0x98 2024-12-10T20:20:57.1672701Z swake_up_one_online+0x54/0xc0 2024-12-10T20:20:57.1674986Z rcu_gp_kthread_wake+0x68/0x98 2024-12-10T20:20:57.1678136Z rcu_accelerate_cbs_unlocked+0xf0/0x128 2024-12-10T20:20:57.1679813Z rcu_core+0x350/0x378 2024-12-10T20:20:57.1681646Z rcu_core_si+0x18/0x30 2024-12-10T20:20:57.1683953Z handle_softirqs+0x12c/0x490 2024-12-10T20:20:57.1685841Z __do_softirq+0x1c/0x28 2024-12-10T20:20:57.1687901Z ____do_softirq+0x18/0x30 2024-12-10T20:20:57.1690172Z call_on_irq_stack+0x24/0x58 2024-12-10T20:20:57.1692602Z do_softirq_own_stack+0x24/0x38 2024-12-10T20:20:57.1694778Z __irq_exit_rcu+0x140/0x180 2024-12-10T20:20:57.1696640Z irq_exit_rcu+0x18/0x48 2024-12-10T20:20:57.1698672Z el1_interrupt+0x38/0x50 2024-12-10T20:20:57.1701082Z el1h_64_irq_handler+0x18/0x28 2024-12-10T20:20:57.1702858Z el1h_64_irq+0x6c/0x70 2024-12-10T20:20:57.1705280Z default_idle_call+0xac/0x254 2024-12-10T20:20:57.1707600Z default_idle_call+0xa8/0x254 2024-12-10T20:20:57.1709171Z do_idle+0xc4/0x118 2024-12-10T20:20:57.1711481Z cpu_startup_entry+0x40/0x50 2024-12-10T20:20:57.1714251Z secondary_start_kernel+0xdc/0x108 2024-12-10T20:20:57.1716568Z __secondary_switched+0xc0/0xc8 2024-12-10T20:20:58.2547951Z sched_ext: BPF scheduler "ddsp_vtimelocal_fail" enabled 2024-12-10T20:20:58.3147251Z sched_ext: BPF scheduler "ddsp_vtimelocal_fail" disabled (runtime error) 2024-12-10T20:20:58.3152469Z sched_ext: ddsp_vtimelocal_fail: cannot use vtime ordering for built-in DSQs 2024-12-10T20:20:58.3154552Z dispatch_enqueue+0x350/0x360 2024-12-10T20:20:58.3156792Z direct_dispatch+0x144/0x228 2024-12-10T20:20:58.3159057Z do_enqueue_task+0x17c/0x240 2024-12-10T20:20:58.3161402Z enqueue_task_scx+0x190/0x288 2024-12-10T20:20:58.3163484Z enqueue_task+0x44/0xe8 2024-12-10T20:20:58.3165655Z ttwu_do_activate+0x88/0x298 2024-12-10T20:20:58.3167841Z try_to_wake_up+0x2b8/0x560 2024-12-10T20:20:58.3169741Z swake_up_one+0x4c/0x98 2024-12-10T20:20:58.3172304Z swake_up_one_online+0x54/0xc0 2024-12-10T20:20:58.3174591Z rcu_gp_kthread_wake+0x68/0x98 2024-12-10T20:20:58.3177003Z rcu_report_qs_rsp+0xc0/0x110 2024-12-10T20:20:58.3179438Z rcu_report_qs_rnp+0x324/0x380 2024-12-10T20:20:58.3181831Z rcu_report_qs_rdp+0x1ac/0x1c8 2024-12-10T20:20:58.3183880Z rcu_core+0x264/0x378 2024-12-10T20:20:58.3185468Z rcu_core_si+0x18/0x30 2024-12-10T20:20:58.3187895Z handle_softirqs+0x12c/0x490 2024-12-10T20:20:58.3189738Z __do_softirq+0x1c/0x28 2024-12-10T20:20:58.3191821Z ____do_softirq+0x18/0x30 2024-12-10T20:20:58.3194130Z call_on_irq_stack+0x24/0x58 2024-12-10T20:20:58.3196588Z do_softirq_own_stack+0x24/0x38 2024-12-10T20:20:58.3198745Z __irq_exit_rcu+0x140/0x180 2024-12-10T20:20:58.3200627Z irq_exit_rcu+0x18/0x48 2024-12-10T20:20:58.3202698Z el1_interrupt+0x38/0x50 2024-12-10T20:20:58.3205112Z el1h_64_irq_handler+0x18/0x28 2024-12-10T20:20:58.3206944Z el1h_64_irq+0x6c/0x70 2024-12-10T20:20:58.3209342Z default_idle_call+0xac/0x254 2024-12-10T20:20:58.3211702Z default_idle_call+0xa8/0x254 2024-12-10T20:20:58.3213312Z do_idle+0xc4/0x118 2024-12-10T20:20:58.3215643Z cpu_startup_entry+0x40/0x50 2024-12-10T20:20:58.3218478Z secondary_start_kernel+0xdc/0x108 2024-12-10T20:20:58.3220846Z __secondary_switched+0xc0/0xc8 2024-12-10T20:20:59.3646877Z sched_ext: BPF scheduler "dsp_local_on" enabled 2024-12-10T20:21:00.3650453Z ERR: dsp_local_on.c:37 2024-12-10T20:21:00.3652199Z Expected skel->data->uei.kind == EXIT_KIND(SCX_EXIT_ERROR) (0 == 1024) 2024-12-10T20:21:00.3873220Z ERR: exit.c:30 2024-12-10T20:21:00.3873570Z Failed to attach scheduler 2024-12-10T20:21:00.4746585Z psci: CPU1 killed (polled 0 ms) 2024-12-10T20:21:00.4946799Z sched_ext: BPF scheduler "dsp_local_on" disabled (runtime error) 2024-12-10T20:21:00.4953445Z sched_ext: dsp_local_on: SCX_DSQ_LOCAL[_ON] verdict target cpu 0 not allowed for ksoftirqd/1[23] 2024-12-10T20:21:00.4955934Z task_can_run_on_remote_rq+0xdc/0x108 2024-12-10T20:21:00.4958580Z dispatch_to_local_dsq+0x74/0x1d0 2024-12-10T20:21:00.4960911Z flush_dispatch_buf+0x18c/0x1d8 2024-12-10T20:21:00.4962903Z balance_one+0x154/0x2f8 2024-12-10T20:21:00.4964749Z balance_scx+0x58/0x98 2024-12-10T20:21:00.4966663Z __schedule+0x390/0x838 2024-12-10T20:21:00.4968348Z schedule+0x54/0x138 2024-12-10T20:21:00.4970453Z worker_thread+0xec/0x360 2024-12-10T20:21:00.4972076Z kthread+0x100/0x110 2024-12-10T20:21:00.4974136Z ret_from_fork+0x10/0x20 2024-12-10T20:21:00.5545933Z sched_ext: BPF scheduler "hotplug_cbs" enabled 2024-12-10T20:21:00.6145106Z Detected PIPT I-cache on CPU1 2024-12-10T20:21:00.6149673Z GICv3: CPU1: found redistributor 1 region 0:0x00000000080c0000 2024-12-10T20:21:00.6154461Z CPU1: Booted secondary processor 0x0000000001 [0x413fd0c1] 2024-12-10T20:21:00.8046976Z sched_ext: BPF scheduler "hotplug_cbs" disabled (unregistered from BPF) 2024-12-10T20:21:00.8047755Z EXIT: unregistered from BPF (hotplug event detected (1 going online)) 2024-12-10T20:21:00.9246328Z sched_ext: BPF scheduler "hotplug_cbs" enabled 2024-12-10T20:21:01.0246233Z psci: CPU1 killed (polled 0 ms) 2024-12-10T20:21:01.0946840Z sched_ext: BPF scheduler "hotplug_cbs" disabled (unregistered from BPF) 2024-12-10T20:21:01.0947621Z EXIT: unregistered from BPF (hotplug event detected (1 going offline)) 2024-12-10T20:21:01.1545076Z Detected PIPT I-cache on CPU1 2024-12-10T20:21:01.1549600Z GICv3: CPU1: found redistributor 1 region 0:0x00000000080c0000 2024-12-10T20:21:01.1554406Z CPU1: Booted secondary processor 0x0000000001 [0x413fd0c1] 2024-12-10T20:21:01.2245966Z psci: CPU1 killed (polled 0 ms) > > diff --git a/tools/testing/selftests/sched_ext/maximal.bpf.c b/tools/testing/selftests/sched_ext/maximal.bpf.c > index 4c005fa71810..430f5e13bf55 100644 > --- a/tools/testing/selftests/sched_ext/maximal.bpf.c > +++ b/tools/testing/selftests/sched_ext/maximal.bpf.c > @@ -12,6 +12,8 @@ > > char _license[] SEC("license") = "GPL"; > > +#define DSQ_ID 0 > + > s32 BPF_STRUCT_OPS(maximal_select_cpu, struct task_struct *p, s32 prev_cpu, > u64 wake_flags) > { > @@ -20,7 +22,7 @@ s32 BPF_STRUCT_OPS(maximal_select_cpu, struct task_struct *p, s32 prev_cpu, > > void BPF_STRUCT_OPS(maximal_enqueue, struct task_struct *p, u64 enq_flags) > { > - scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags); > + scx_bpf_dsq_insert(p, DSQ_ID, SCX_SLICE_DFL, enq_flags); > } > > void BPF_STRUCT_OPS(maximal_dequeue, struct task_struct *p, u64 deq_flags) > @@ -28,7 +30,7 @@ void BPF_STRUCT_OPS(maximal_dequeue, struct task_struct *p, u64 deq_flags) > > void BPF_STRUCT_OPS(maximal_dispatch, s32 cpu, struct task_struct *prev) > { > - scx_bpf_dsq_move_to_local(SCX_DSQ_GLOBAL); > + scx_bpf_dsq_move_to_local(DSQ_ID); > } > > void BPF_STRUCT_OPS(maximal_runnable, struct task_struct *p, u64 enq_flags) > @@ -123,7 +125,7 @@ void BPF_STRUCT_OPS(maximal_cgroup_set_weight, struct cgroup *cgrp, u32 weight) > > s32 BPF_STRUCT_OPS_SLEEPABLE(maximal_init) > { > - return 0; > + return scx_bpf_create_dsq(DSQ_ID, -1); > } > > void BPF_STRUCT_OPS(maximal_exit, struct scx_exit_info *info) > -- > 2.46.1 ^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix invalid irq restore in scx_ops_bypass() 2024-12-10 20:52 ` [PATCH] scx: Fix maximal BPF selftest prog Ihor Solodrai @ 2024-12-11 21:01 ` Tejun Heo 2024-12-11 21:03 ` Tejun Heo 2024-12-12 18:12 ` Ihor Solodrai 0 siblings, 2 replies; 20+ messages in thread From: Tejun Heo @ 2024-12-11 21:01 UTC (permalink / raw) To: Ihor Solodrai; +Cc: David Vernet, sched-ext, kernel-team, linux-kernel, bpf While adding outer irqsave/restore locking, 0e7ffff1b811 ("scx: Fix raciness in scx_ops_bypass()") forgot to convert an inner rq_unlock_irqrestore() to rq_unlock() which could re-enable IRQ prematurely leading to the following warning: raw_local_irq_restore() called with IRQs enabled WARNING: CPU: 1 PID: 96 at kernel/locking/irqflag-debug.c:10 warn_bogus_irq_restore+0x30/0x40 ... Sched_ext: create_dsq (enabling) pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) pc : warn_bogus_irq_restore+0x30/0x40 lr : warn_bogus_irq_restore+0x30/0x40 ... Call trace: warn_bogus_irq_restore+0x30/0x40 (P) warn_bogus_irq_restore+0x30/0x40 (L) scx_ops_bypass+0x224/0x3b8 scx_ops_enable.isra.0+0x2c8/0xaa8 bpf_scx_reg+0x18/0x30 ... irq event stamp: 33739 hardirqs last enabled at (33739): [<ffff8000800b699c>] scx_ops_bypass+0x174/0x3b8 hardirqs last disabled at (33738): [<ffff800080d48ad4>] _raw_spin_lock_irqsave+0xb4/0xd8 Drop the stray _irqrestore(). Signed-off-by: Tejun Heo <tj@kernel.org> Reported-by: Ihor Solodrai <ihor.solodrai@pm.me> Link: http://lkml.kernel.org/r/qC39k3UsonrBYD_SmuxHnZIQLsuuccoCrkiqb_BT7DvH945A1_LZwE4g-5Pu9FcCtqZt4lY1HhIPi0homRuNWxkgo1rgP3bkxa0donw8kV4=@pm.me Fixes: 0e7ffff1b811 ("scx: Fix raciness in scx_ops_bypass()") Cc: stable@vger.kernel.org # v6.12 --- kernel/sched/ext.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c index 7fff1d045477..98519e6d0dcd 100644 --- a/kernel/sched/ext.c +++ b/kernel/sched/ext.c @@ -4763,7 +4763,7 @@ static void scx_ops_bypass(bool bypass) * sees scx_rq_bypassing() before moving tasks to SCX. */ if (!scx_enabled()) { - rq_unlock_irqrestore(rq, &rf); + rq_unlock(rq, &rf); continue; } ^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix invalid irq restore in scx_ops_bypass() 2024-12-11 21:01 ` [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix invalid irq restore in scx_ops_bypass() Tejun Heo @ 2024-12-11 21:03 ` Tejun Heo 2024-12-12 18:12 ` Ihor Solodrai 1 sibling, 0 replies; 20+ messages in thread From: Tejun Heo @ 2024-12-11 21:03 UTC (permalink / raw) To: Ihor Solodrai; +Cc: David Vernet, sched-ext, kernel-team, linux-kernel, bpf On Wed, Dec 11, 2024 at 11:01:51AM -1000, Tejun Heo wrote: > While adding outer irqsave/restore locking, 0e7ffff1b811 ("scx: Fix raciness > in scx_ops_bypass()") forgot to convert an inner rq_unlock_irqrestore() to > rq_unlock() which could re-enable IRQ prematurely leading to the following > warning: > > raw_local_irq_restore() called with IRQs enabled > WARNING: CPU: 1 PID: 96 at kernel/locking/irqflag-debug.c:10 warn_bogus_irq_restore+0x30/0x40 > ... > Sched_ext: create_dsq (enabling) > pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) > pc : warn_bogus_irq_restore+0x30/0x40 > lr : warn_bogus_irq_restore+0x30/0x40 > ... > Call trace: > warn_bogus_irq_restore+0x30/0x40 (P) > warn_bogus_irq_restore+0x30/0x40 (L) > scx_ops_bypass+0x224/0x3b8 > scx_ops_enable.isra.0+0x2c8/0xaa8 > bpf_scx_reg+0x18/0x30 > ... > irq event stamp: 33739 > hardirqs last enabled at (33739): [<ffff8000800b699c>] scx_ops_bypass+0x174/0x3b8 > hardirqs last disabled at (33738): [<ffff800080d48ad4>] _raw_spin_lock_irqsave+0xb4/0xd8 > > Drop the stray _irqrestore(). > > Signed-off-by: Tejun Heo <tj@kernel.org> > Reported-by: Ihor Solodrai <ihor.solodrai@pm.me> > Link: http://lkml.kernel.org/r/qC39k3UsonrBYD_SmuxHnZIQLsuuccoCrkiqb_BT7DvH945A1_LZwE4g-5Pu9FcCtqZt4lY1HhIPi0homRuNWxkgo1rgP3bkxa0donw8kV4=@pm.me > Fixes: 0e7ffff1b811 ("scx: Fix raciness in scx_ops_bypass()") > Cc: stable@vger.kernel.org # v6.12 Applying to sched_ext/for-6.13-fixes. Thanks. -- tejun ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix invalid irq restore in scx_ops_bypass() 2024-12-11 21:01 ` [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix invalid irq restore in scx_ops_bypass() Tejun Heo 2024-12-11 21:03 ` Tejun Heo @ 2024-12-12 18:12 ` Ihor Solodrai 2024-12-17 23:44 ` Ihor Solodrai 1 sibling, 1 reply; 20+ messages in thread From: Ihor Solodrai @ 2024-12-12 18:12 UTC (permalink / raw) To: Tejun Heo; +Cc: David Vernet, sched-ext, kernel-team, linux-kernel, bpf On Wednesday, December 11th, 2024 at 1:01 PM, Tejun Heo <tj@kernel.org> wrote: > > > While adding outer irqsave/restore locking, 0e7ffff1b811 ("scx: Fix raciness > in scx_ops_bypass()") forgot to convert an inner rq_unlock_irqrestore() to > rq_unlock() which could re-enable IRQ prematurely leading to the following > warning: > > raw_local_irq_restore() called with IRQs enabled > WARNING: CPU: 1 PID: 96 at kernel/locking/irqflag-debug.c:10 warn_bogus_irq_restore+0x30/0x40 > ... > Sched_ext: create_dsq (enabling) > pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) > pc : warn_bogus_irq_restore+0x30/0x40 > lr : warn_bogus_irq_restore+0x30/0x40 > ... > Call trace: > warn_bogus_irq_restore+0x30/0x40 (P) > warn_bogus_irq_restore+0x30/0x40 (L) > scx_ops_bypass+0x224/0x3b8 > scx_ops_enable.isra.0+0x2c8/0xaa8 > bpf_scx_reg+0x18/0x30 > ... > irq event stamp: 33739 > hardirqs last enabled at (33739): [<ffff8000800b699c>] scx_ops_bypass+0x174/0x3b8 > > hardirqs last disabled at (33738): [<ffff800080d48ad4>] _raw_spin_lock_irqsave+0xb4/0xd8 > > > Drop the stray _irqrestore(). > > Signed-off-by: Tejun Heo tj@kernel.org > > Reported-by: Ihor Solodrai ihor.solodrai@pm.me > > Link: http://lkml.kernel.org/r/qC39k3UsonrBYD_SmuxHnZIQLsuuccoCrkiqb_BT7DvH945A1_LZwE4g-5Pu9FcCtqZt4lY1HhIPi0homRuNWxkgo1rgP3bkxa0donw8kV4=@pm.me > Fixes: 0e7ffff1b811 ("scx: Fix raciness in scx_ops_bypass()") > Cc: stable@vger.kernel.org # v6.12 > --- > kernel/sched/ext.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c > index 7fff1d045477..98519e6d0dcd 100644 > --- a/kernel/sched/ext.c > +++ b/kernel/sched/ext.c > @@ -4763,7 +4763,7 @@ static void scx_ops_bypass(bool bypass) > * sees scx_rq_bypassing() before moving tasks to SCX. > */ > if (!scx_enabled()) { > - rq_unlock_irqrestore(rq, &rf); > + rq_unlock(rq, &rf); > continue; > } Hi Tejun, I tried this patch on BPF CI: the pipeline ran 3 times successfully. That's 12 selftests/sched_ext runs in total. https://github.com/kernel-patches/vmtest/actions/runs/12301284063 Tested-by: Ihor Solodrai ihor.solodrai@pm.me Thanks for the fix! ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix invalid irq restore in scx_ops_bypass() 2024-12-12 18:12 ` Ihor Solodrai @ 2024-12-17 23:44 ` Ihor Solodrai 2024-12-18 18:34 ` Tejun Heo 0 siblings, 1 reply; 20+ messages in thread From: Ihor Solodrai @ 2024-12-17 23:44 UTC (permalink / raw) To: Tejun Heo; +Cc: David Vernet, sched-ext, kernel-team, linux-kernel, bpf Hi Tejun, I re-enabled selftests/sched_ext on BPF CI today. The kernel on CI includes this patch. Sometimes there is a failure on attempt to attach a dsp_local_on scheduler. Examples of failed jobs: * https://github.com/kernel-patches/bpf/actions/runs/12379720791/job/34555104994 * https://github.com/kernel-patches/bpf/actions/runs/12382862660/job/34564648924 * https://github.com/kernel-patches/bpf/actions/runs/12381361846/job/34560047798 Here is a piece of log that is present in failed run, but not in a successful run: 2024-12-17T19:30:12.9010943Z [ 5.285022] sched_ext: BPF scheduler "dsp_local_on" enabled 2024-12-17T19:30:13.9022892Z ERR: dsp_local_on.c:37 2024-12-17T19:30:13.9025841Z Expected skel->data->uei.kind == EXIT_KIND(SCX_EXIT_ERROR) (0 == 1024) 2024-12-17T19:30:13.9256108Z ERR: exit.c:30 2024-12-17T19:30:13.9256641Z Failed to attach scheduler 2024-12-17T19:30:13.9611443Z [ 6.345087] smpboot: CPU 1 is now offline Could you please investigate? Thanks. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix invalid irq restore in scx_ops_bypass() 2024-12-17 23:44 ` Ihor Solodrai @ 2024-12-18 18:34 ` Tejun Heo 2024-12-19 22:51 ` Ihor Solodrai 0 siblings, 1 reply; 20+ messages in thread From: Tejun Heo @ 2024-12-18 18:34 UTC (permalink / raw) To: Ihor Solodrai; +Cc: David Vernet, sched-ext, kernel-team, linux-kernel, bpf Hello, On Tue, Dec 17, 2024 at 11:44:08PM +0000, Ihor Solodrai wrote: > I re-enabled selftests/sched_ext on BPF CI today. The kernel on CI > includes this patch. Sometimes there is a failure on attempt to attach > a dsp_local_on scheduler. > > Examples of failed jobs: > > * https://github.com/kernel-patches/bpf/actions/runs/12379720791/job/34555104994 > * https://github.com/kernel-patches/bpf/actions/runs/12382862660/job/34564648924 > * https://github.com/kernel-patches/bpf/actions/runs/12381361846/job/34560047798 > > Here is a piece of log that is present in failed run, but not in > a successful run: > > 2024-12-17T19:30:12.9010943Z [ 5.285022] sched_ext: BPF scheduler "dsp_local_on" enabled > 2024-12-17T19:30:13.9022892Z ERR: dsp_local_on.c:37 > 2024-12-17T19:30:13.9025841Z Expected skel->data->uei.kind == EXIT_KIND(SCX_EXIT_ERROR) (0 == 1024) > 2024-12-17T19:30:13.9256108Z ERR: exit.c:30 > 2024-12-17T19:30:13.9256641Z Failed to attach scheduler > 2024-12-17T19:30:13.9611443Z [ 6.345087] smpboot: CPU 1 is now offline > > Could you please investigate? The test prog is wrong in assuming all possible CPUs to be consecutive and online but I'm not sure whether that's what's making the test flaky. Do you have dmesg from a failed run? Thanks. -- tejun ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix invalid irq restore in scx_ops_bypass() 2024-12-18 18:34 ` Tejun Heo @ 2024-12-19 22:51 ` Ihor Solodrai 2024-12-20 19:26 ` Ihor Solodrai 2024-12-25 0:09 ` [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix dsq_local_on selftest Tejun Heo 0 siblings, 2 replies; 20+ messages in thread From: Ihor Solodrai @ 2024-12-19 22:51 UTC (permalink / raw) To: Tejun Heo; +Cc: David Vernet, sched-ext, kernel-team, linux-kernel, bpf On Wednesday, December 18th, 2024 at 10:34 AM, Tejun Heo <tj@kernel.org> wrote: > > > Hello, > > On Tue, Dec 17, 2024 at 11:44:08PM +0000, Ihor Solodrai wrote: > > > I re-enabled selftests/sched_ext on BPF CI today. The kernel on CI > > includes this patch. Sometimes there is a failure on attempt to attach > > a dsp_local_on scheduler. > > > > Examples of failed jobs: > > > > * https://github.com/kernel-patches/bpf/actions/runs/12379720791/job/34555104994 > > * https://github.com/kernel-patches/bpf/actions/runs/12382862660/job/34564648924 > > * https://github.com/kernel-patches/bpf/actions/runs/12381361846/job/34560047798 > > > > Here is a piece of log that is present in failed run, but not in > > a successful run: > > > > 2024-12-17T19:30:12.9010943Z [ 5.285022] sched_ext: BPF scheduler "dsp_local_on" enabled > > 2024-12-17T19:30:13.9022892Z ERR: dsp_local_on.c:37 > > 2024-12-17T19:30:13.9025841Z Expected skel->data->uei.kind == EXIT_KIND(SCX_EXIT_ERROR) (0 == 1024) > > 2024-12-17T19:30:13.9256108Z ERR: exit.c:30 > > 2024-12-17T19:30:13.9256641Z Failed to attach scheduler > > 2024-12-17T19:30:13.9611443Z [ 6.345087] smpboot: CPU 1 is now offline > > > > Could you please investigate? > > > The test prog is wrong in assuming all possible CPUs to be consecutive and > online but I'm not sure whether that's what's making the test flaky. Do you > have dmesg from a failed run? Tejun, can you elaborate on what you're looking for in the logs? My understanding is that QEMU prints some of the dmesg messages. QEMU output is available in raw logs. Here is a link (you have to login to github to open): https://productionresultssa1.blob.core.windows.net/actions-results/99cd995e-679f-4180-872b-d31e1f564837/workflow-job-run-7216a7c9-5129-5959-a45a-28d6f9b737e2/logs/job/job-logs.txt?rsct=text%2Fplain&se=2024-12-19T22%3A57%3A01Z&sig=z%2B%2FUtIIhli4VG%2FCCVxawBnubNwfIIsl9Q2FlTVvM8q0%3D&ske=2024-12-20T07%3A00%3A35Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2024-12-19T19%3A00%3A35Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2024-11-04&sp=r&spr=https&sr=b&st=2024-12-19T22%3A46%3A56Z&sv=2024-11-04 Generally, you can access raw logs by going to the job, and clicking the gear on the topright -> "View raw logs". > > Thanks. > > -- > tejun ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix invalid irq restore in scx_ops_bypass() 2024-12-19 22:51 ` Ihor Solodrai @ 2024-12-20 19:26 ` Ihor Solodrai 2024-12-25 0:09 ` [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix dsq_local_on selftest Tejun Heo 1 sibling, 0 replies; 20+ messages in thread From: Ihor Solodrai @ 2024-12-20 19:26 UTC (permalink / raw) To: Tejun Heo; +Cc: David Vernet, sched-ext, kernel-team, linux-kernel, bpf On Thursday, December 19th, 2024 at 2:51 PM, Ihor Solodrai <ihor.solodrai@pm.me> wrote: > [...] > > Tejun, can you elaborate on what you're looking for in the logs? > My understanding is that QEMU prints some of the dmesg messages. > QEMU output is available in raw logs. I made changes to the CI scripts to explicitly dump dmesg in case of a failure. It looks like most of that log was already printed. Job: https://github.com/kernel-patches/bpf/actions/runs/12436924307/job/34726070343 Raw log: https://productionresultssa11.blob.core.windows.net/actions-results/a10f57cb-19e3-487a-9fb0-69742cfbef1b/workflow-job-run-4c580b44-6466-54d8-b922-6f707064e5ca/logs/job/job-logs.txt?rsct=text%2Fplain&se=2024-12-20T19%3A34%3A55Z&sig=kQ09k9r01VtP4p%2FgYvvCmm2FUuOHfsLjU3ARzks4xmE%3D&ske=2024-12-21T07%3A00%3A50Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2024-12-20T19%3A00%3A50Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2024-11-04&sp=r&spr=https&sr=b&st=2024-12-20T19%3A24%3A50Z&sv=2024-11-04 Search for "dmesg start". > [...] ^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix dsq_local_on selftest 2024-12-19 22:51 ` Ihor Solodrai 2024-12-20 19:26 ` Ihor Solodrai @ 2024-12-25 0:09 ` Tejun Heo 2024-12-25 0:10 ` Tejun Heo 2025-01-15 23:50 ` Ihor Solodrai 1 sibling, 2 replies; 20+ messages in thread From: Tejun Heo @ 2024-12-25 0:09 UTC (permalink / raw) To: Ihor Solodrai; +Cc: David Vernet, sched-ext, kernel-team, linux-kernel, bpf The dsp_local_on selftest expects the scheduler to fail by trying to schedule an e.g. CPU-affine task to the wrong CPU. However, this isn't guaranteed to happen in the 1 second window that the test is running. Besides, it's odd to have this particular exception path tested when there are no other tests that verify that the interface is working at all - e.g. the test would pass if dsp_local_on interface is completely broken and fails on any attempt. Flip the test so that it verifies that the feature works. While at it, fix a typo in the info message. Signed-off-by: Tejun Heo <tj@kernel.org> Reported-by: Ihor Solodrai <ihor.solodrai@pm.me> Link: http://lkml.kernel.org/r/Z1n9v7Z6iNJ-wKmq@slm.duckdns.org --- tools/testing/selftests/sched_ext/dsp_local_on.bpf.c | 5 ++++- tools/testing/selftests/sched_ext/dsp_local_on.c | 5 +++-- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/sched_ext/dsp_local_on.bpf.c b/tools/testing/selftests/sched_ext/dsp_local_on.bpf.c index 6325bf76f47e..fbda6bf54671 100644 --- a/tools/testing/selftests/sched_ext/dsp_local_on.bpf.c +++ b/tools/testing/selftests/sched_ext/dsp_local_on.bpf.c @@ -43,7 +43,10 @@ void BPF_STRUCT_OPS(dsp_local_on_dispatch, s32 cpu, struct task_struct *prev) if (!p) return; - target = bpf_get_prandom_u32() % nr_cpus; + if (p->nr_cpus_allowed == nr_cpus) + target = bpf_get_prandom_u32() % nr_cpus; + else + target = scx_bpf_task_cpu(p); scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL_ON | target, SCX_SLICE_DFL, 0); bpf_task_release(p); diff --git a/tools/testing/selftests/sched_ext/dsp_local_on.c b/tools/testing/selftests/sched_ext/dsp_local_on.c index 472851b56854..0ff27e57fe43 100644 --- a/tools/testing/selftests/sched_ext/dsp_local_on.c +++ b/tools/testing/selftests/sched_ext/dsp_local_on.c @@ -34,9 +34,10 @@ static enum scx_test_status run(void *ctx) /* Just sleeping is fine, plenty of scheduling events happening */ sleep(1); - SCX_EQ(skel->data->uei.kind, EXIT_KIND(SCX_EXIT_ERROR)); bpf_link__destroy(link); + SCX_EQ(skel->data->uei.kind, EXIT_KIND(SCX_EXIT_UNREG)); + return SCX_TEST_PASS; } @@ -50,7 +51,7 @@ static void cleanup(void *ctx) struct scx_test dsp_local_on = { .name = "dsp_local_on", .description = "Verify we can directly dispatch tasks to a local DSQs " - "from osp.dispatch()", + "from ops.dispatch()", .setup = setup, .run = run, .cleanup = cleanup, ^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix dsq_local_on selftest 2024-12-25 0:09 ` [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix dsq_local_on selftest Tejun Heo @ 2024-12-25 0:10 ` Tejun Heo 2025-01-15 23:50 ` Ihor Solodrai 1 sibling, 0 replies; 20+ messages in thread From: Tejun Heo @ 2024-12-25 0:10 UTC (permalink / raw) To: Ihor Solodrai; +Cc: David Vernet, sched-ext, kernel-team, linux-kernel, bpf On Tue, Dec 24, 2024 at 02:09:15PM -1000, Tejun Heo wrote: > The dsp_local_on selftest expects the scheduler to fail by trying to > schedule an e.g. CPU-affine task to the wrong CPU. However, this isn't > guaranteed to happen in the 1 second window that the test is running. > Besides, it's odd to have this particular exception path tested when there > are no other tests that verify that the interface is working at all - e.g. > the test would pass if dsp_local_on interface is completely broken and fails > on any attempt. > > Flip the test so that it verifies that the feature works. While at it, fix a > typo in the info message. > > Signed-off-by: Tejun Heo <tj@kernel.org> > Reported-by: Ihor Solodrai <ihor.solodrai@pm.me> > Link: http://lkml.kernel.org/r/Z1n9v7Z6iNJ-wKmq@slm.duckdns.org Applied to sched_ext/for-6.13-fixes. Thanks. -- tejun ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix dsq_local_on selftest 2024-12-25 0:09 ` [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix dsq_local_on selftest Tejun Heo 2024-12-25 0:10 ` Tejun Heo @ 2025-01-15 23:50 ` Ihor Solodrai 2025-01-22 1:40 ` Tejun Heo 1 sibling, 1 reply; 20+ messages in thread From: Ihor Solodrai @ 2025-01-15 23:50 UTC (permalink / raw) To: Tejun Heo; +Cc: sched-ext, kernel-team, linux-kernel, bpf On Tuesday, December 24th, 2024 at 4:09 PM, Tejun Heo <tj@kernel.org> wrote: > > > The dsp_local_on selftest expects the scheduler to fail by trying to > schedule an e.g. CPU-affine task to the wrong CPU. However, this isn't > guaranteed to happen in the 1 second window that the test is running. > Besides, it's odd to have this particular exception path tested when there > are no other tests that verify that the interface is working at all - e.g. > the test would pass if dsp_local_on interface is completely broken and fails > on any attempt. > > Flip the test so that it verifies that the feature works. While at it, fix a > typo in the info message. > > Signed-off-by: Tejun Heo tj@kernel.org > > Reported-by: Ihor Solodrai ihor.solodrai@pm.me > > Link: http://lkml.kernel.org/r/Z1n9v7Z6iNJ-wKmq@slm.duckdns.org > --- > tools/testing/selftests/sched_ext/dsp_local_on.bpf.c | 5 ++++- > tools/testing/selftests/sched_ext/dsp_local_on.c | 5 +++-- > 2 files changed, 7 insertions(+), 3 deletions(-) Hi Tejun. I've tried running sched_ext selftests on BPF CI today, applying a set of patches from sched_ext/for-6.13-fixes, including this one. You can see the list of patches I added here: https://github.com/kernel-patches/vmtest/pull/332/files With that, dsq_local_on has failed on x86_64 (llvm-18), although it passed with other configurations: https://github.com/kernel-patches/vmtest/actions/runs/12798804552/job/35683769806 Here is a piece of log that appears to be relevant: 2025-01-15T23:28:55.8238375Z [ 5.334631] sched_ext: BPF scheduler "dsp_local_on" disabled (runtime error) 2025-01-15T23:28:55.8243034Z [ 5.335420] sched_ext: dsp_local_on: SCX_DSQ_LOCAL[_ON] verdict target cpu 1 not allowed for kworker/u8:1[33] 2025-01-15T23:28:55.8246187Z [ 5.336139] dispatch_to_local_dsq+0x13e/0x1f0 2025-01-15T23:28:55.8249296Z [ 5.336474] flush_dispatch_buf+0x13d/0x170 2025-01-15T23:28:55.8252083Z [ 5.336793] balance_scx+0x225/0x3e0 2025-01-15T23:28:55.8254695Z [ 5.337065] __schedule+0x406/0xe80 2025-01-15T23:28:55.8257121Z [ 5.337330] schedule+0x41/0xb0 2025-01-15T23:28:55.8260146Z [ 5.337574] schedule_timeout+0xe5/0x160 2025-01-15T23:28:55.8263080Z [ 5.337875] rcu_tasks_kthread+0xb1/0xc0 2025-01-15T23:28:55.8265477Z [ 5.338169] kthread+0xfa/0x120 2025-01-15T23:28:55.8268202Z [ 5.338410] ret_from_fork+0x37/0x50 2025-01-15T23:28:55.8271272Z [ 5.338690] ret_from_fork_asm+0x1a/0x30 2025-01-15T23:28:56.7349562Z ERR: dsp_local_on.c:39 2025-01-15T23:28:56.7350182Z Expected skel->data->uei.kind == EXIT_KIND(SCX_EXIT_UNREG) (1024 == 64) Could you please take a look? Thank you. > > [...] ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix dsq_local_on selftest 2025-01-15 23:50 ` Ihor Solodrai @ 2025-01-22 1:40 ` Tejun Heo 2025-01-22 19:10 ` Ihor Solodrai 0 siblings, 1 reply; 20+ messages in thread From: Tejun Heo @ 2025-01-22 1:40 UTC (permalink / raw) To: Ihor Solodrai; +Cc: sched-ext, kernel-team, linux-kernel, bpf Hello, sorry about the delay. On Wed, Jan 15, 2025 at 11:50:37PM +0000, Ihor Solodrai wrote: ... > 2025-01-15T23:28:55.8238375Z [ 5.334631] sched_ext: BPF scheduler "dsp_local_on" disabled (runtime error) > 2025-01-15T23:28:55.8243034Z [ 5.335420] sched_ext: dsp_local_on: SCX_DSQ_LOCAL[_ON] verdict target cpu 1 not allowed for kworker/u8:1[33] That's a head scratcher. It's a single node 2 cpu instance and all unbound kworkers should be allowed on all CPUs. I'll update the test to test the actual cpumask but can you see whether this failure is consistent or flaky? Thanks. -- tejun ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix dsq_local_on selftest 2025-01-22 1:40 ` Tejun Heo @ 2025-01-22 19:10 ` Ihor Solodrai 2025-01-23 9:40 ` Andrea Righi 0 siblings, 1 reply; 20+ messages in thread From: Ihor Solodrai @ 2025-01-22 19:10 UTC (permalink / raw) To: Tejun Heo; +Cc: sched-ext, kernel-team, linux-kernel, bpf On Tuesday, January 21st, 2025 at 5:40 PM, Tejun Heo <tj@kernel.org> wrote: > > > Hello, sorry about the delay. > > On Wed, Jan 15, 2025 at 11:50:37PM +0000, Ihor Solodrai wrote: > ... > > > 2025-01-15T23:28:55.8238375Z [ 5.334631] sched_ext: BPF scheduler "dsp_local_on" disabled (runtime error) > > 2025-01-15T23:28:55.8243034Z [ 5.335420] sched_ext: dsp_local_on: SCX_DSQ_LOCAL[_ON] verdict target cpu 1 not allowed for kworker/u8:1[33] > > > That's a head scratcher. It's a single node 2 cpu instance and all unbound > kworkers should be allowed on all CPUs. I'll update the test to test the > actual cpumask but can you see whether this failure is consistent or flaky? I re-ran all the jobs, and all sched_ext jobs have failed (3/3). Previous time only 1 of 3 runs failed. https://github.com/kernel-patches/vmtest/actions/runs/12798804552/job/36016405680 > > Thanks. > > -- > tejun ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix dsq_local_on selftest 2025-01-22 19:10 ` Ihor Solodrai @ 2025-01-23 9:40 ` Andrea Righi 2025-01-23 16:57 ` Tejun Heo 0 siblings, 1 reply; 20+ messages in thread From: Andrea Righi @ 2025-01-23 9:40 UTC (permalink / raw) To: Ihor Solodrai; +Cc: Tejun Heo, sched-ext, kernel-team, linux-kernel, bpf On Wed, Jan 22, 2025 at 07:10:00PM +0000, Ihor Solodrai wrote: > > On Tuesday, January 21st, 2025 at 5:40 PM, Tejun Heo <tj@kernel.org> wrote: > > > > > > > Hello, sorry about the delay. > > > > On Wed, Jan 15, 2025 at 11:50:37PM +0000, Ihor Solodrai wrote: > > ... > > > > > 2025-01-15T23:28:55.8238375Z [ 5.334631] sched_ext: BPF scheduler "dsp_local_on" disabled (runtime error) > > > 2025-01-15T23:28:55.8243034Z [ 5.335420] sched_ext: dsp_local_on: SCX_DSQ_LOCAL[_ON] verdict target cpu 1 not allowed for kworker/u8:1[33] > > > > > > That's a head scratcher. It's a single node 2 cpu instance and all unbound > > kworkers should be allowed on all CPUs. I'll update the test to test the > > actual cpumask but can you see whether this failure is consistent or flaky? > > I re-ran all the jobs, and all sched_ext jobs have failed (3/3). > Previous time only 1 of 3 runs failed. > > https://github.com/kernel-patches/vmtest/actions/runs/12798804552/job/36016405680 Oh I see what happens, SCX_DSQ_LOCAL_ON is (incorrectly) resolved to 0. More exactly, none of the enum values are being resolved correctly, likely due to the CO:RE enum refactoring. There’s probably something broken in tools/testing/selftests/sched_ext/Makefile, I’ll take a look. Thanks, -Andrea ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix dsq_local_on selftest 2025-01-23 9:40 ` Andrea Righi @ 2025-01-23 16:57 ` Tejun Heo 2025-01-23 18:45 ` Andrea Righi 0 siblings, 1 reply; 20+ messages in thread From: Tejun Heo @ 2025-01-23 16:57 UTC (permalink / raw) To: Andrea Righi; +Cc: Ihor Solodrai, sched-ext, kernel-team, linux-kernel, bpf On Thu, Jan 23, 2025 at 10:40:52AM +0100, Andrea Righi wrote: > On Wed, Jan 22, 2025 at 07:10:00PM +0000, Ihor Solodrai wrote: > > > > On Tuesday, January 21st, 2025 at 5:40 PM, Tejun Heo <tj@kernel.org> wrote: > > > > > > > > > > > Hello, sorry about the delay. > > > > > > On Wed, Jan 15, 2025 at 11:50:37PM +0000, Ihor Solodrai wrote: > > > ... > > > > > > > 2025-01-15T23:28:55.8238375Z [ 5.334631] sched_ext: BPF scheduler "dsp_local_on" disabled (runtime error) > > > > 2025-01-15T23:28:55.8243034Z [ 5.335420] sched_ext: dsp_local_on: SCX_DSQ_LOCAL[_ON] verdict target cpu 1 not allowed for kworker/u8:1[33] > > > > > > > > > That's a head scratcher. It's a single node 2 cpu instance and all unbound > > > kworkers should be allowed on all CPUs. I'll update the test to test the > > > actual cpumask but can you see whether this failure is consistent or flaky? > > > > I re-ran all the jobs, and all sched_ext jobs have failed (3/3). > > Previous time only 1 of 3 runs failed. > > > > https://github.com/kernel-patches/vmtest/actions/runs/12798804552/job/36016405680 > > Oh I see what happens, SCX_DSQ_LOCAL_ON is (incorrectly) resolved to 0. > > More exactly, none of the enum values are being resolved correctly, likely > due to the CO:RE enum refactoring. There’s probably something broken in > tools/testing/selftests/sched_ext/Makefile, I’ll take a look. Yeah, we need to add SCX_ENUM_INIT() to each test. Will do that once the pending pull request is merged. The original report is a separate issue tho. I'm still a bit baffled by it. Thanks. -- tejun ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix dsq_local_on selftest 2025-01-23 16:57 ` Tejun Heo @ 2025-01-23 18:45 ` Andrea Righi 2025-01-23 22:26 ` Andrea Righi 0 siblings, 1 reply; 20+ messages in thread From: Andrea Righi @ 2025-01-23 18:45 UTC (permalink / raw) To: Tejun Heo; +Cc: Ihor Solodrai, sched-ext, kernel-team, linux-kernel, bpf On Thu, Jan 23, 2025 at 06:57:58AM -1000, Tejun Heo wrote: > On Thu, Jan 23, 2025 at 10:40:52AM +0100, Andrea Righi wrote: > > On Wed, Jan 22, 2025 at 07:10:00PM +0000, Ihor Solodrai wrote: > > > > > > On Tuesday, January 21st, 2025 at 5:40 PM, Tejun Heo <tj@kernel.org> wrote: > > > > > > > > > > > > > > > Hello, sorry about the delay. > > > > > > > > On Wed, Jan 15, 2025 at 11:50:37PM +0000, Ihor Solodrai wrote: > > > > ... > > > > > > > > > 2025-01-15T23:28:55.8238375Z [ 5.334631] sched_ext: BPF scheduler "dsp_local_on" disabled (runtime error) > > > > > 2025-01-15T23:28:55.8243034Z [ 5.335420] sched_ext: dsp_local_on: SCX_DSQ_LOCAL[_ON] verdict target cpu 1 not allowed for kworker/u8:1[33] > > > > > > > > > > > > That's a head scratcher. It's a single node 2 cpu instance and all unbound > > > > kworkers should be allowed on all CPUs. I'll update the test to test the > > > > actual cpumask but can you see whether this failure is consistent or flaky? > > > > > > I re-ran all the jobs, and all sched_ext jobs have failed (3/3). > > > Previous time only 1 of 3 runs failed. > > > > > > https://github.com/kernel-patches/vmtest/actions/runs/12798804552/job/36016405680 > > > > Oh I see what happens, SCX_DSQ_LOCAL_ON is (incorrectly) resolved to 0. > > > > More exactly, none of the enum values are being resolved correctly, likely > > due to the CO:RE enum refactoring. There’s probably something broken in > > tools/testing/selftests/sched_ext/Makefile, I’ll take a look. > > Yeah, we need to add SCX_ENUM_INIT() to each test. Will do that once the > pending pull request is merged. The original report is a separate issue tho. > I'm still a bit baffled by it. For the enum part: https://lore.kernel.org/all/20250123124606.242115-1-arighi@nvidia.com/ And yeah, I missed that the original bug report was about the unbound kworker not allowed to be dispatched on cpu 1. Weird... I'm wondering if we need to do the cpumask_cnt / scx_bpf_dsq_cancel() game, like we did with scx_rustland to handle concurrent affinity changes, but in this case the kworker shouldn't have its affinity changed... -Andrea ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix dsq_local_on selftest 2025-01-23 18:45 ` Andrea Righi @ 2025-01-23 22:26 ` Andrea Righi 2025-01-24 22:00 ` [PATCH sched_ext/for-6.14-fixes] sched_ext: selftests/dsp_local_on: Fix sporadic failures Tejun Heo 0 siblings, 1 reply; 20+ messages in thread From: Andrea Righi @ 2025-01-23 22:26 UTC (permalink / raw) To: Tejun Heo; +Cc: Ihor Solodrai, sched-ext, kernel-team, linux-kernel, bpf On Thu, Jan 23, 2025 at 07:45:08PM +0100, Andrea Righi wrote: > On Thu, Jan 23, 2025 at 06:57:58AM -1000, Tejun Heo wrote: > > On Thu, Jan 23, 2025 at 10:40:52AM +0100, Andrea Righi wrote: > > > On Wed, Jan 22, 2025 at 07:10:00PM +0000, Ihor Solodrai wrote: > > > > > > > > On Tuesday, January 21st, 2025 at 5:40 PM, Tejun Heo <tj@kernel.org> wrote: > > > > > > > > > > > > > > > > > > > Hello, sorry about the delay. > > > > > > > > > > On Wed, Jan 15, 2025 at 11:50:37PM +0000, Ihor Solodrai wrote: > > > > > ... > > > > > > > > > > > 2025-01-15T23:28:55.8238375Z [ 5.334631] sched_ext: BPF scheduler "dsp_local_on" disabled (runtime error) > > > > > > 2025-01-15T23:28:55.8243034Z [ 5.335420] sched_ext: dsp_local_on: SCX_DSQ_LOCAL[_ON] verdict target cpu 1 not allowed for kworker/u8:1[33] > > > > > > > > > > > > > > > That's a head scratcher. It's a single node 2 cpu instance and all unbound > > > > > kworkers should be allowed on all CPUs. I'll update the test to test the > > > > > actual cpumask but can you see whether this failure is consistent or flaky? > > > > > > > > I re-ran all the jobs, and all sched_ext jobs have failed (3/3). > > > > Previous time only 1 of 3 runs failed. > > > > > > > > https://github.com/kernel-patches/vmtest/actions/runs/12798804552/job/36016405680 > > > > > > Oh I see what happens, SCX_DSQ_LOCAL_ON is (incorrectly) resolved to 0. > > > > > > More exactly, none of the enum values are being resolved correctly, likely > > > due to the CO:RE enum refactoring. There’s probably something broken in > > > tools/testing/selftests/sched_ext/Makefile, I’ll take a look. > > > > Yeah, we need to add SCX_ENUM_INIT() to each test. Will do that once the > > pending pull request is merged. The original report is a separate issue tho. > > I'm still a bit baffled by it. > > For the enum part: https://lore.kernel.org/all/20250123124606.242115-1-arighi@nvidia.com/ > > And yeah, I missed that the original bug report was about the unbound > kworker not allowed to be dispatched on cpu 1. Weird... I'm wondering if we > need to do the cpumask_cnt / scx_bpf_dsq_cancel() game, like we did with > scx_rustland to handle concurrent affinity changes, but in this case the > kworker shouldn't have its affinity changed... Thinking more about this, scx_bpf_task_cpu(p) returns the last known CPU where the task p was running, but it doesn't necessarily give a CPU where the task can run at any time. In general it's probably a safer choice to rely on p->cpus_ptr, maybe doing bpf_cpumask_any_distribute(p->cpus_ptr) for this test case. However, I still don't see why the unbound kworker couldn't be dispatched on cpu 1 in this particular case... -Andrea ^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH sched_ext/for-6.14-fixes] sched_ext: selftests/dsp_local_on: Fix sporadic failures 2025-01-23 22:26 ` Andrea Righi @ 2025-01-24 22:00 ` Tejun Heo 2025-01-25 4:54 ` Andrea Righi 0 siblings, 1 reply; 20+ messages in thread From: Tejun Heo @ 2025-01-24 22:00 UTC (permalink / raw) To: Andrea Righi; +Cc: Ihor Solodrai, sched-ext, kernel-team, linux-kernel, bpf From e9fe182772dcb2630964724fd93e9c90b68ea0fd Mon Sep 17 00:00:00 2001 From: Tejun Heo <tj@kernel.org> Date: Fri, 24 Jan 2025 10:48:25 -1000 dsp_local_on has several incorrect assumptions, one of which is that p->nr_cpus_allowed always tracks p->cpus_ptr. This is not true when a task is scheduled out while migration is disabled - p->cpus_ptr is temporarily overridden to the previous CPU while p->nr_cpus_allowed remains unchanged. This led to sporadic test faliures when dsp_local_on_dispatch() tries to put a migration disabled task to a different CPU. Fix it by keeping the previous CPU when migration is disabled. There are SCX schedulers that make use of p->nr_cpus_allowed. They should also implement explicit handling for p->migration_disabled. Signed-off-by: Tejun Heo <tj@kernel.org> Reported-by: Ihor Solodrai <ihor.solodrai@pm.me> Cc: Andrea Righi <arighi@nvidia.com> Cc: Changwoo Min <changwoo@igalia.com> --- Applying to sched_ext/for-6.14-fixes. Thanks. tools/testing/selftests/sched_ext/dsp_local_on.bpf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/sched_ext/dsp_local_on.bpf.c b/tools/testing/selftests/sched_ext/dsp_local_on.bpf.c index fbda6bf54671..758b479bd1ee 100644 --- a/tools/testing/selftests/sched_ext/dsp_local_on.bpf.c +++ b/tools/testing/selftests/sched_ext/dsp_local_on.bpf.c @@ -43,7 +43,7 @@ void BPF_STRUCT_OPS(dsp_local_on_dispatch, s32 cpu, struct task_struct *prev) if (!p) return; - if (p->nr_cpus_allowed == nr_cpus) + if (p->nr_cpus_allowed == nr_cpus && !p->migration_disabled) target = bpf_get_prandom_u32() % nr_cpus; else target = scx_bpf_task_cpu(p); -- 2.48.1 ^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [PATCH sched_ext/for-6.14-fixes] sched_ext: selftests/dsp_local_on: Fix sporadic failures 2025-01-24 22:00 ` [PATCH sched_ext/for-6.14-fixes] sched_ext: selftests/dsp_local_on: Fix sporadic failures Tejun Heo @ 2025-01-25 4:54 ` Andrea Righi 2025-01-27 18:45 ` Tejun Heo 0 siblings, 1 reply; 20+ messages in thread From: Andrea Righi @ 2025-01-25 4:54 UTC (permalink / raw) To: Tejun Heo; +Cc: Ihor Solodrai, sched-ext, kernel-team, linux-kernel, bpf On Fri, Jan 24, 2025 at 12:00:38PM -1000, Tejun Heo wrote: > From e9fe182772dcb2630964724fd93e9c90b68ea0fd Mon Sep 17 00:00:00 2001 > From: Tejun Heo <tj@kernel.org> > Date: Fri, 24 Jan 2025 10:48:25 -1000 > > dsp_local_on has several incorrect assumptions, one of which is that > p->nr_cpus_allowed always tracks p->cpus_ptr. This is not true when a task > is scheduled out while migration is disabled - p->cpus_ptr is temporarily > overridden to the previous CPU while p->nr_cpus_allowed remains unchanged. > > This led to sporadic test faliures when dsp_local_on_dispatch() tries to put > a migration disabled task to a different CPU. Fix it by keeping the previous > CPU when migration is disabled. > > There are SCX schedulers that make use of p->nr_cpus_allowed. They should > also implement explicit handling for p->migration_disabled. > > Signed-off-by: Tejun Heo <tj@kernel.org> > Reported-by: Ihor Solodrai <ihor.solodrai@pm.me> > Cc: Andrea Righi <arighi@nvidia.com> > Cc: Changwoo Min <changwoo@igalia.com> > --- > Applying to sched_ext/for-6.14-fixes. Thanks. > > tools/testing/selftests/sched_ext/dsp_local_on.bpf.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/tools/testing/selftests/sched_ext/dsp_local_on.bpf.c b/tools/testing/selftests/sched_ext/dsp_local_on.bpf.c > index fbda6bf54671..758b479bd1ee 100644 > --- a/tools/testing/selftests/sched_ext/dsp_local_on.bpf.c > +++ b/tools/testing/selftests/sched_ext/dsp_local_on.bpf.c > @@ -43,7 +43,7 @@ void BPF_STRUCT_OPS(dsp_local_on_dispatch, s32 cpu, struct task_struct *prev) > if (!p) > return; > > - if (p->nr_cpus_allowed == nr_cpus) > + if (p->nr_cpus_allowed == nr_cpus && !p->migration_disabled) This doesn't work with !CONFIG_SMP, maybe we can introduce a helper like: static bool is_migration_disabled(const struct task_struct *p) { if (bpf_core_field_exists(p->migration_disabled)) return p->migration_disabled; return false; } > target = bpf_get_prandom_u32() % nr_cpus; > else > target = scx_bpf_task_cpu(p); > -- > 2.48.1 > Thanks, -Andrea ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH sched_ext/for-6.14-fixes] sched_ext: selftests/dsp_local_on: Fix sporadic failures 2025-01-25 4:54 ` Andrea Righi @ 2025-01-27 18:45 ` Tejun Heo 0 siblings, 0 replies; 20+ messages in thread From: Tejun Heo @ 2025-01-27 18:45 UTC (permalink / raw) To: Andrea Righi; +Cc: Ihor Solodrai, sched-ext, kernel-team, linux-kernel, bpf On Sat, Jan 25, 2025 at 05:54:23AM +0100, Andrea Righi wrote: ... > > - if (p->nr_cpus_allowed == nr_cpus) > > + if (p->nr_cpus_allowed == nr_cpus && !p->migration_disabled) > > This doesn't work with !CONFIG_SMP, maybe we can introduce a helper like: > > static bool is_migration_disabled(const struct task_struct *p) > { > if (bpf_core_field_exists(p->migration_disabled)) > return p->migration_disabled; > return false; Ah, right. Would you care to send the patch? Thanks. -- tejun ^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2025-01-27 18:45 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20241209152924.4508-1-void@manifault.com>
2024-12-10 20:52 ` [PATCH] scx: Fix maximal BPF selftest prog Ihor Solodrai
2024-12-11 21:01 ` [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix invalid irq restore in scx_ops_bypass() Tejun Heo
2024-12-11 21:03 ` Tejun Heo
2024-12-12 18:12 ` Ihor Solodrai
2024-12-17 23:44 ` Ihor Solodrai
2024-12-18 18:34 ` Tejun Heo
2024-12-19 22:51 ` Ihor Solodrai
2024-12-20 19:26 ` Ihor Solodrai
2024-12-25 0:09 ` [PATCH sched_ext/for-6.13-fixes] sched_ext: Fix dsq_local_on selftest Tejun Heo
2024-12-25 0:10 ` Tejun Heo
2025-01-15 23:50 ` Ihor Solodrai
2025-01-22 1:40 ` Tejun Heo
2025-01-22 19:10 ` Ihor Solodrai
2025-01-23 9:40 ` Andrea Righi
2025-01-23 16:57 ` Tejun Heo
2025-01-23 18:45 ` Andrea Righi
2025-01-23 22:26 ` Andrea Righi
2025-01-24 22:00 ` [PATCH sched_ext/for-6.14-fixes] sched_ext: selftests/dsp_local_on: Fix sporadic failures Tejun Heo
2025-01-25 4:54 ` Andrea Righi
2025-01-27 18:45 ` Tejun Heo
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox