Re: Strange latency spikes/TX network stalls on Sun Fire X4150(x86) and e1000e

[Eric Dumazet <eric.dumazet@gmail.com>]

On Wed, 2012-05-30 at 09:06 +0900, Hiroaki SHIMODA wrote:
> While reading the bql code, I have some questions.
> 
> 1) dql_completed() and dql_queued() can be called concurrently,
>    so dql->num_queued could change while processing
>    dql_completed().
>    Is it intentional to refer num_queued from "dql->" each time ?
> 

not sure it can have problems, but doing the read once is indeed a good
plan.

> 2) From the comment in the code
>    *   - The queue was over-limit in the previous interval and
>    *     when enqueuing it was possible that all queued data
>    *     had been consumed.
> 
>    and
> 
>    * Queue was not starved, check if the limit can be decreased.
>    * A decrease is only considered if the queue has been busy in
>    * the whole interval (the check above). 
> 
>    the calculation of all_prev_completed should take into account
>    completed == dql->prev_num_queued case ?
>    On current implementation, limit shrinks easily and some NIC
>    hit TX stalls.
>    To mitigate TX stalls, should we fix all_prev_completed rather
>    than individual driver ?
> 

Not sure what you mean

> 3) limit calculation fails to consider integer wrap around in
>    one place ?
> 

Yes

> Here is the patch what I meant.
> 
> diff --git a/lib/dynamic_queue_limits.c b/lib/dynamic_queue_limits.c
> @@ -11,22 +11,27 @@
>  #include <linux/dynamic_queue_limits.h>
>  
>  #define POSDIFF(A, B) ((A) > (B) ? (A) - (B) : 0)
> +#define POSDIFFI(A, B) ((int)((A) - (B)) > 0 ? (A) - (B) : 0)
> +#define AFTER_EQ(A, B) ((int)((A) - (B)) >= 0)
>  
>  /* Records completed count and recalculates the queue limit */
>  void dql_completed(struct dql *dql, unsigned int count)
>  {
>  	unsigned int inprogress, prev_inprogress, limit;
> -	unsigned int ovlimit, all_prev_completed, completed;
> +	unsigned int ovlimit, completed, num_queued;
> +	bool all_prev_completed;
> +
> +	num_queued = dql->num_queued;


I suggest :

	num_queued = ACCESS_ONCE(dql->num_queued);
	
Or else compiler is free to do whatever he wants.