Re: [PATCH] net: reduce number of reference taken on sk_refcnt

[Eric Dumazet <dada1@cosmosbay.com>]

David Miller a écrit :
> From: David Miller <davem@davemloft.net>
> Date: Sat, 09 May 2009 13:34:54 -0700 (PDT)
> 
>> Consider the case where we always send some message on CPU A and
>> then process the ACK on CPU B.  We'll always be cancelling the
>> timer on a foreign cpu.
> 
> I should also mention that TCP has a peculiar optimization of timers
> that is likely being thwarted by your workload.  It never deletes
> timers under normal operation, it simply lets them still expire
> and the handler notices that there is "nothing to do" and returns.

Yes, you refer to INET_CSK_CLEAR_TIMERS condition, never set.

> 
> But when the connection does shut down, we have to purge all of
> these timers.
> 
> That could be another part of why you see timers in your profile.
> 
> 

Well, in my workload they should never expire, since application exchange
enough data on both direction, and they are no losses (Gigabit LAN context)

On machine acting as a server (the one I am focusing to, of course),
each incoming frame :

- Contains ACK for the previous sent frame
- Contains data provided by the client.
- Starts a timer for delayed ACK

Then server applications reacts and sends a new payload, and TCP stack
- Sends a frame including ACK for previous received frame
- Contains data provided by server application
- Starts a timer for retransmiting this frame if no ACK is received later.

So yes, each incoming and each outgoing frame is going to call mod_timer()

Problem is that incoming process is done by CPU 0 (the one that is dedicated
to NAPI processing because of stress situation, cpu 100% in softirq land),
and outgoing processing done by other cpus in the machine.

offsetof(struct inet_connection_sock, icsk_retransmit_timer)=0x208
offsetof(struct inet_connection_sock, icsk_delack_timer)=0x238

So there are cache line ping-pongs, but oprofile seems to point
to a spinlock contention in lock_timer_base(), I dont know why...
shouldnt (in my workload) delack_timer all belongs to cpu 0, and 
retransmit_timers to other cpus ? 

Or is mod_timer never migrates an already established timer ?

That would explain the lock contention on timer_base, we should
take care of it if possible.

Thanks David