Re: Strange latency spikes/TX network stalls on Sun Fire X4150(x86) and e1000e

[Hiroaki SHIMODA <shimoda.hiroaki@gmail.com>]

While reading the bql code, I have some questions.

1) dql_completed() and dql_queued() can be called concurrently,
   so dql->num_queued could change while processing
   dql_completed().
   Is it intentional to refer num_queued from "dql->" each time ?

2) From the comment in the code
   *   - The queue was over-limit in the previous interval and
   *     when enqueuing it was possible that all queued data
   *     had been consumed.

   and

   * Queue was not starved, check if the limit can be decreased.
   * A decrease is only considered if the queue has been busy in
   * the whole interval (the check above). 

   the calculation of all_prev_completed should take into account
   completed == dql->prev_num_queued case ?
   On current implementation, limit shrinks easily and some NIC
   hit TX stalls.
   To mitigate TX stalls, should we fix all_prev_completed rather
   than individual driver ?

3) limit calculation fails to consider integer wrap around in
   one place ?

Here is the patch what I meant.

diff --git a/lib/dynamic_queue_limits.c b/lib/dynamic_queue_limits.c
@@ -11,22 +11,27 @@
 #include <linux/dynamic_queue_limits.h>
 
 #define POSDIFF(A, B) ((A) > (B) ? (A) - (B) : 0)
+#define POSDIFFI(A, B) ((int)((A) - (B)) > 0 ? (A) - (B) : 0)
+#define AFTER_EQ(A, B) ((int)((A) - (B)) >= 0)
 
 /* Records completed count and recalculates the queue limit */
 void dql_completed(struct dql *dql, unsigned int count)
 {
 	unsigned int inprogress, prev_inprogress, limit;
-	unsigned int ovlimit, all_prev_completed, completed;
+	unsigned int ovlimit, completed, num_queued;
+	bool all_prev_completed;
+
+	num_queued = dql->num_queued;
 
 	/* Can't complete more than what's in queue */
-	BUG_ON(count > dql->num_queued - dql->num_completed);
+	BUG_ON(count > num_queued - dql->num_completed);
 
 	completed = dql->num_completed + count;
 	limit = dql->limit;
-	ovlimit = POSDIFF(dql->num_queued - dql->num_completed, limit);
-	inprogress = dql->num_queued - completed;
+	ovlimit = POSDIFF(num_queued - dql->num_completed, limit);
+	inprogress = num_queued - completed;
 	prev_inprogress = dql->prev_num_queued - dql->num_completed;
-	all_prev_completed = POSDIFF(completed, dql->prev_num_queued);
+	all_prev_completed = AFTER_EQ(completed, dql->prev_num_queued);
 
 	if ((ovlimit && !inprogress) ||
 	    (dql->prev_ovlimit && all_prev_completed)) {
@@ -45,7 +50,7 @@ void dql_completed(struct dql *dql, unsigned int count)
 		 *     of bytes both sent and completed in the last interval,
 		 *     plus any previous over-limit.
 		 */
-		limit += POSDIFF(completed, dql->prev_num_queued) +
+		limit += POSDIFFI(completed, dql->prev_num_queued) +
 		     dql->prev_ovlimit;
 		dql->slack_start_time = jiffies;
 		dql->lowest_slack = UINT_MAX;
@@ -104,7 +109,7 @@ void dql_completed(struct dql *dql, unsigned int count)
 	dql->prev_ovlimit = ovlimit;
 	dql->prev_last_obj_cnt = dql->last_obj_cnt;
 	dql->num_completed = completed;
-	dql->prev_num_queued = dql->num_queued;
+	dql->prev_num_queued = num_queued;
 }
 EXPORT_SYMBOL(dql_completed);