Re: [PATCH 00/16] Remove the ipv4 routing cache

[Eric Dumazet <eric.dumazet@gmail.com>]

On Wed, 2012-07-25 at 17:54 -0700, David Miller wrote:
> From: David Miller <davem@davemloft.net>
> Date: Wed, 25 Jul 2012 16:39:39 -0700 (PDT)
> 
> > From: David Miller <davem@davemloft.net>
> > Date: Wed, 25 Jul 2012 16:17:32 -0700 (PDT)
> > 
> >> From: Alexander Duyck <alexander.duyck@gmail.com>
> >> Date: Wed, 25 Jul 2012 16:02:45 -0700
> >> 
> >>> Since your patches are in I have started to re-run my tests. I am
> >>> seeing a significant drop in throughput with 8 flows which I expected,
> >>> however it looks like one of the biggest issues I am seeing is that
> >>> the dst_hold and dst_release calls seem to be causing some serious
> >>> cache thrash.  I was at 12.5Mpps w/ 8 flows before the patches, after
> >>> your patches it drops to 8.3Mpps.
> >> 
> >> Yes, this is something we knew would start happening.
> >> 
> >> One idea is to make cached dsts be per-cpu in the nexthops.
> > 
> > Actually I think what really kills your case is the removal of the
> > noref path for route lookups.  I'll work on a patch to restore that
> > in the case where we use cached routes from the FIB nexthops.
> 
> Alex, here is something I tossed together, does it help with the
> dst_hold()/dst_release() overhead at all?
> 

seems good to me, only one question :


>  static void rt_cache_route(struct fib_nh *nh, struct rtable *rt)
> @@ -1216,9 +1215,15 @@ static void rt_cache_route(struct fib_nh *nh, struct rtable *rt)
>  
>  	prev = cmpxchg(p, orig, rt);
>  	if (prev == orig) {
> -		dst_clone(&rt->dst);
>  		if (orig)
> -			call_rcu_bh(&orig->dst.rcu_head, rt_release_rcu);
> +			rt_free(orig);
> +	} else {
> +		/* Routes we intend to cache in the FIB nexthop have
> +		 * the DST_NOCACHE bit set.  However, if we are
> +		 * unsuccessful at storing this route into the cache
> +		 * we really need to clear that bit.
> +		 */
> +		rt->dst.flags &= ~DST_NOCACHE;
>  	}
>  }
>  

Not sure why you removed the dst_clone(&rt->dst) ?

If it is not needed, we might need to release a reference in the else {}
clause, no ?