Re: [RFC PATCH v2 0/7] Defer throttle when task exits to user

Florian Bezdeka
2025-04-22T14:54:32Z

On Tue, 2025-04-22 at 08:24 +0530, K Prateek Nayak wrote:
> Hello Aaron,
>
> On 4/22/2025 7:40 AM, Aaron Lu wrote:
> > > anon_pipe_write()
> > > __wake_up_common()
> > > ep_poll_callback() {
> > > read_lock_irq(&ep->lock) /* Read lock acquired here */
> > I was confused by this function's name. I had thought irq is off but
> > then realized under PREEMPT_RT, read_lock_irq() doesn't disable irq...
>
> Yup! Most of the interrupt handlers are run by the IRQ threads on
> PREEMPT_RT and the ones that do run in the interrupt context have all
> been adapted to use non-blocking locks whose *_irq variants disables
> interrupts on PREEMPT_RT too.
>
> >
> > > __wake_up_common()
> > > ep_autoremove_wake_function()
> > > try_to_wake_up() /* Wakes up "epoll-stall" */
> > > preempt_schedule()
> > > ...
> > >
> > > # "epoll-stall-writer" has run out of bandwidth, needs replenish to run
> > Luckily in this "only throttle when ret2user" model, epoll-stall-writer
> > does not need replenish to run again(and then unblock the others).
>
> I can confirm that throttle deferral solves this issue. I have run Jan's
> reproducer for a long time without seeing any hangs on your series. I
> hope Florian can confirm the same.
>

Partially, yes.

First, let me clarify what I am testing: I'm testing with PREEMPT_RT
enabled, as that is the setup that makes problems in the field. For
those setups it's not a performance/jitter optimization it's a critical
fix. The system locks up completely.

I ported the series to 6.14. Background was stability and the
possibility to replace one of the devices in the field with a patched
version. We do not trust anything newer yet.

The test results: 6.14 + backport is still running fine for ~10 days
now on a system where the reproducer (that Jan posted already) crashed
a unpatched 6.14 in a couple of minutes. Success.

But: I also started a test with 6.14 vanilla (so unpatched) on a
different system. This one crashes within a couple of minutes. This is
a completely different story - as the series we're discussing here is
not even applied - but to be complete, this is the last message we get
from the device:

The device is completely locked up afterwards. PID 34 is ktimers on
CPU1.

kernel: ------------[ cut here ]------------
kernel: !se->on_rq
kernel: WARNING: CPU: 1 PID: 34 at kernel/sched/fair.c:699 update_entity_lag+0x7d/0x90
kernel: Modules linked in: veth xt_nat nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink xfr>
kernel: sd_mod mptspi ata_generic mptscsih mptbase psmouse scsi_transport_spi ata_piix libata scs>
kernel: CPU: 1 UID: 0 PID: 34 Comm: ktimers/1 Not tainted 6.14.0 #1
kernel: Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.242>
kernel: RIP: 0010:update_entity_lag+0x7d/0x90
kernel: Code: 0f 4d d7 48 89 53 78 5b 5d c3 cc cc cc cc 80 3d e7 f4 dd 01 00 75 a9 48 c7 c7 d0 81 >
kernel: RSP: 0018:ffffacf58012fbe8 EFLAGS: 00010082
kernel: RAX: 0000000000000000 RBX: ffff9ee43ca00080 RCX: 0000000000000027
kernel: RDX: ffff9ee6efd21988 RSI: 0000000000000001 RDI: ffff9ee6efd21980
kernel: RBP: ffff9ee421929800 R08: 00000000462951bd R09: ffffffff8e654811
kernel: R10: ffffffff8e654811 R11: ffffffff8e608a2a R12: 000000000000000e
kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 000000000000000e
kernel: FS: 0000000000000000(0000) GS:ffff9ee6efd00000(0000) knlGS:0000000000000000
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 000000c00082a000 CR3: 0000000113416002 CR4: 00000000007706f0
kernel: PKRU: 55555554
kernel: Call Trace:
kernel: <TASK>
kernel: ? __warn+0x91/0x190
kernel: ? update_entity_lag+0x7d/0x90
kernel: ? report_bug+0x164/0x190
kernel: ? handle_bug+0x58/0x90
kernel: ? exc_invalid_op+0x17/0x70
kernel: ? asm_exc_invalid_op+0x1a/0x20
kernel: ? ret_from_fork_asm+0x1a/0x30
kernel: ? ret_from_fork+0x31/0x50
kernel: ? ret_from_fork+0x31/0x50
kernel: ? update_entity_lag+0x7d/0x90
kernel: ? update_entity_lag+0x7d/0x90
kernel: dequeue_entity+0x90/0x5a0
kernel: dequeue_entities+0x121/0x640
kernel: dequeue_task_fair+0xbf/0x290
kernel: rt_mutex_setprio+0x37c/0x690
kernel: rtlock_slowlock_locked+0xca1/0x1860
kernel: ? lock_acquire+0xcb/0x2e0
kernel: ? run_ktimerd+0xe/0x80
kernel: ? __pfx_smpboot_thread_fn+0x10/0x10
kernel: rt_spin_lock+0x86/0x160
kernel: __local_bh_disable_ip+0x9d/0x190
kernel: ksoftirqd_run_begin+0xe/0x30
kernel: run_ktimerd+0xe/0x80
kernel: smpboot_thread_fn+0xda/0x1d0