Re: [PATCH net-next v6 1/3] net: tcp/dcpp: prepare for tw_timer un-pinning

[Florian Westphal <fw@strlen.de>]

Florian Westphal <fw@strlen.de> wrote:
> Eric Dumazet <edumazet@google.com> wrote:
> > On Mon, Jun 3, 2024 at 11:37 AM Florian Westphal <fw@strlen.de> wrote:
> > > +       spin_lock(lock);
> > > +       if (timer_shutdown(&tw->tw_timer)) {
> > > +               /* releases @lock */
> > > +               __inet_twsk_kill(tw, lock);
> > > +       } else {
> > 
> > If we do not have a sync variant here, I think that inet_twsk_purge()
> > could return while ongoing timers are alive.
> 
> Yes.
> 
> We can't use sync variant, it would deadlock on ehash spinlock.
> 
> > tcp_sk_exit_batch() would then possibly hit :
> > 
> > WARN_ON_ONCE(!refcount_dec_and_test(&net->ipv4.tcp_death_row.tw_refcount));
> > 
> > The alive timer are releasing tw->tw_dr->tw_refcount at the end of
> > inet_twsk_kill()
> 
> Theoretically the tw socket can be unlinked from the tw hash already
> (inet_twsk_purge won't encounter it), but timer is still running.
> 
> Only solution I see is to schedule() in tcp_sk_exit_batch() until
> tw_refcount has dropped to the expected value, i.e. something like
> 
> static void tcp_wait_for_tw_timers(struct net *n)
> {
> 	while (refcount_read(&n->ipv4.tcp_death_row.tw_refcount) > 1))
> 		schedule();
> }
> 
> Any better idea?

Actually, I think we can solve this in a much simpler way.

Instead of replacing:

void inet_twsk_deschedule_put(struct inet_timewait_sock *tw)
{
 if (del_timer_sync(&tw->tw_timer))
    inet_twsk_kill(tw);
 inet_twsk_put(tw);
}

With:
 spinlock_t *lock = inet_ehash_lockp(hashinfo, tw->tw_hash);
 spin_lock(lock);
 if (timer_shutdown(&tw->tw_timer)) {

(Which gets us into the tcp_sk_exit_batch trouble Eric points out),
we can simply add "empty" ehash lock unlock pair before calling
del_timer_sync():

void inet_twsk_deschedule_put(struct inet_timewait_sock *tw)
{
+	spinlock_t *lock = inet_ehash_lockp(hashinfo, tw->tw_hash);
+	spin_lock(lock)
+	spin_unlock(lock)

        if (del_timer_sync(&tw->tw_timer))
                inet_twsk_kill(tw);
        inet_twsk_put(tw);
}

Rationale:
inet_twsk_deschedule_put() cannot be called before hashdance_schedule
calls refcount_set(&tw->tw_refcnt, 3).

Before this any refcount_inc_not_zero fails so we never get into
deschedule_put.

Hashdance_schedule holds the ehash lock when it sets the tw refcount.
The lock is released only after the timer is up and running.

When inet_twsk_deschedule_put() is called, and hashdance_schedule
is not yet done, the spinlock/unlock pair will guarantee that
the timer is up after the spin_unlock.

I think this is much better than the schedule loop waiting for tw_dr
refcount to drop, it mainly needs a comment to explain what this is
doing.

Thoughts?