Re: [RFC PATCH v2 0/7] Defer throttle when task exits to user

[Aaron Lu <ziqianlu@bytedance.com>]

Hi Prateek,

On Tue, Apr 15, 2025 at 09:19:20PM +0530, K Prateek Nayak wrote:
> Hello Jan,
> 
> Sorry for the noise.
> 
> On 4/15/2025 4:46 PM, K Prateek Nayak wrote:
> > Hello Jan,
> > 
> > On 4/15/2025 3:51 PM, Jan Kiszka wrote:
> > > > Is this in line with what you are seeing?
> > > > 
> > > 
> > > Yes, and if you wait a bit longer for the second reporting round, you
> > > should get more task backtraces as well.
> > 
> > So looking at the backtrace [1], Aaron's patch should help with the
> > stalls you are seeing.
> > 
> > timerfd that queues a hrtimer also uses ep_poll_callback() to wakeup
> > the epoll waiter which queues ahead of the bandwidth timer and
> > requires the read lock but now since the writer tried to grab the
> > lock pushing readers on the slowpath. if epoll-stall-writer is now
> > throttled, it needs ktimer to replenish its bandwidth which cannot
> > happen without it grabbing the read lock first.
> > 
> > # epoll-stall-writer
> 
> So I got confused between "epoll-stall" and "epoll-stall-writer" here.
> Turns out the actual series of events (based on traces, and hopefully
> correct this time) are slightly longer. The correct series of events
> are:
> 
> # epoll-stall-writer
> 
> anon_pipe_write()
>   __wake_up_common()
>     ep_poll_callback() {
>       read_lock_irq(&ep->lock)		/* Read lock acquired here */

I was confused by this function's name. I had thought irq is off but
then realized under PREEMPT_RT, read_lock_irq() doesn't disable irq...

>       __wake_up_common()
>         ep_autoremove_wake_function()
>           try_to_wake_up()		/* Wakes up "epoll-stall" */
>             preempt_schedule()
>             ...
> 
> # "epoll-stall-writer" has run out of bandwidth, needs replenish to run

Luckily in this "only throttle when ret2user" model, epoll-stall-writer
does not need replenish to run again(and then unblock the others).

> # sched_switch: "epoll-stall-writer" => "epoll-stall"
> 
>     ... /* Resumes from epoll_wait() */
>     epoll_wait() => 1			/* Write to FIFO */
>     read() 				/* Reads one byte of data */
>     epoll_wait()
>       write_lock_irq()			/* Tries to grab write lock; "epoll-stall-writer" still has read lock */
>         schedule_rtlock()		/* Sleeps but put next readers on slowpath */
>         ...
> 
> # sched_switch: "epoll-stall" => "swapper"
> # CPU is idle
> 
> ...
> 
> # Timer interrupt schedules ktimers
> # sched_switch: "swapper" => "ktimers"
> 
> hrtimer_run_softirq()
>   timerfd_tmrproc()
>     __wake_up_common()
>       ep_poll_callback() {
>         read_lock_irq(&ep->lock)	/* Blocks since we are in rwlock slowpath */
>           schedule_rtlock()
>           ...
> 
> # sched_switch: "ktimers" => "swapper"
> # Bandwidth replenish never happens
> # Stall
> 
> From a second look at trace, this should be the right series of
> events since "epoll-stall-writer" with bandwidth control seems
> to have cut off during while doing the wakeup and hasn't run
> again.
> 
> Sorry for the noise.
> 

Thanks for the analysis.

I'm testing this reprod with this series and didn't notice any issue
yet, I'll report if anything unexpected happened.

Best wishes,
Aaron