Re: [RFC PATCH v2 0/7] Defer throttle when task exits to user

Aaron Lu
2025-04-22T02:11:02Z

Hi Prateek,

On Tue, Apr 15, 2025 at 09:19:20PM +0530, K Prateek Nayak wrote:
> Hello Jan,
>
> Sorry for the noise.
>
> On 4/15/2025 4:46 PM, K Prateek Nayak wrote:
> > Hello Jan,
> >
> > On 4/15/2025 3:51 PM, Jan Kiszka wrote:
> > > > Is this in line with what you are seeing?
> > > >
> > >
> > > Yes, and if you wait a bit longer for the second reporting round, you
> > > should get more task backtraces as well.
> >
> > So looking at the backtrace [1], Aaron's patch should help with the
> > stalls you are seeing.
> >
> > timerfd that queues a hrtimer also uses ep_poll_callback() to wakeup
> > the epoll waiter which queues ahead of the bandwidth timer and
> > requires the read lock but now since the writer tried to grab the
> > lock pushing readers on the slowpath. if epoll-stall-writer is now
> > throttled, it needs ktimer to replenish its bandwidth which cannot
> > happen without it grabbing the read lock first.
> >
> > # epoll-stall-writer
>
> So I got confused between "epoll-stall" and "epoll-stall-writer" here.
> Turns out the actual series of events (based on traces, and hopefully
> correct this time) are slightly longer. The correct series of events
> are:
>
> # epoll-stall-writer
>
> anon_pipe_write()
> __wake_up_common()
> ep_poll_callback() {
> read_lock_irq(&ep->lock) /* Read lock acquired here */

I was confused by this function's name. I had thought irq is off but
then realized under PREEMPT_RT, read_lock_irq() doesn't disable irq...

> __wake_up_common()
> ep_autoremove_wake_function()
> try_to_wake_up() /* Wakes up "epoll-stall" */
> preempt_schedule()
> ...
>
> # "epoll-stall-writer" has run out of bandwidth, needs replenish to run

Luckily in this "only throttle when ret2user" model, epoll-stall-writer
does not need replenish to run again(and then unblock the others).

> # sched_switch: "epoll-stall-writer" => "epoll-stall"
>
> ... /* Resumes from epoll_wait() */
> epoll_wait() => 1 /* Write to FIFO */
> read() /* Reads one byte of data */
> epoll_wait()
> write_lock_irq() /* Tries to grab write lock; "epoll-stall-writer" still has read lock */
> schedule_rtlock() /* Sleeps but put next readers on slowpath */
> ...
>
> # sched_switch: "epoll-stall" => "swapper"
> # CPU is idle
>
> ...
>
> # Timer interrupt schedules ktimers
> # sched_switch: "swapper" => "ktimers"
>
> hrtimer_run_softirq()
> timerfd_tmrproc()
> __wake_up_common()
> ep_poll_callback() {
> read_lock_irq(&ep->lock) /* Blocks since we are in rwlock slowpath */
> schedule_rtlock()
> ...
>
> # sched_switch: "ktimers" => "swapper"
> # Bandwidth replenish never happens
> # Stall
>
> From a second look at trace, this should be the right series of
> events since "epoll-stall-writer" with bandwidth control seems
> to have cut off during while doing the wakeup and hasn't run
> again.
>
> Sorry for the noise.
>

Thanks for the analysis.

I'm testing this reprod with this series and didn't notice any issue
yet, I'll report if anything unexpected happened.

Best wishes,
Aaron