[PATCH 0/3] NIU: Implement discard counters

[PATCH 0/3] NIU: Implement discard counters

[Jesper Dangaard Brouer]

This patch series implements discard counter stats for the NIU driver.

This work has been done together with Robert Olsson.  We have come to a
stage where the patches needs public review, and possible discussion.

Tested with hardware:
 Sun Quad GbE x8 PCIe
 Sun Dual 10GbE XFP PCIe

-- 
Med venlig hilsen / Best regards
  Jesper Brouer
  ComX Networks A/S
  Linux Network developer
  Cand. Scient Datalog / MSc.
  Author of http://adsl-optimizer.dk
  LinkedIn: http://www.linkedin.com/in/brouer

[PATCH 1/3] NIU: Implement discard counters

[Jesper Dangaard Brouer]

Implementing discard counters for the NIU driver turned out to be more
complicated than first assumed.

The discard counters for the NIU neptune chip is only 16-bit
(eventhough this is a 64-bit chip).  These 16-bit counters can
overflow quickly, especially considering this is a 10Gbit/s ethernet
card.

The overflow indication bit is, unfortunatly, not usable as the
counter value does not wrap, but remains at max value 0xFFFF.
Resulting in lost counts until the counter is reset.

The read and reset scheme also poses a problem. Both in theory and in
praxis counters can be lost in between reading nr64() and clearing the
counter nw64().  For this reason, the number of counter clearings
nw64() is limited/reduced.  On the fash-path the counters are only
syncronized once it exceeds 0x7FFF.  When read by userspace, 
its syncronized fully.

Signed-off-by: Jesper Dangaard Brouer <hawk@comx.dk>
---

 drivers/net/niu.c |   51 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 51 insertions(+), 0 deletions(-)

diff --git a/drivers/net/niu.c b/drivers/net/niu.c
index 1b6f548..1bd7018 100644
--- a/drivers/net/niu.c
+++ b/drivers/net/niu.c
@@ -3529,6 +3529,51 @@ out:
 	}
 }
 
+static inline void niu_sync_rx_discard_stats(struct niu *np,
+					     struct rx_ring_info *rp,
+					     const int limit)
+{
+	/* This elaborate scheme is needed for reading the RX discard
+	 * counters, as they are only 16-bit and can overflow quickly,
+	 * and because the overflow indication bit is not usable as
+	 * the counter value does not wrap, but remains at max value
+	 * 0xFFFF.
+	 *
+	 * In theory and in praxis counters can be lost in between
+	 * reading nr64() and clearing the counter nw64().  For this
+	 * reason, the number of counter clearings nw64() is
+	 * limited/reduced though the limit parameter.
+	 */
+	int rx_channel = rp->rx_channel;
+	u32 misc, wred;
+
+	/* RXMISC (Receive Miscellaneous Discard Count), covers the
+	 * following discard events: IPP (Input Port Process),
+	 * FFLP/TCAM, Full RCR (Receive Completion Ring) RBR (Receive
+	 * Block Ring) prefetch buffer is empty.
+	 */
+	misc = nr64(RXMISC(rx_channel));
+	if (unlikely((misc & RXMISC_COUNT) > limit)) {
+		nw64(RXMISC(rx_channel), 0);
+		rp->rx_errors += misc & RXMISC_COUNT;
+
+		if (unlikely(misc & RXMISC_OFLOW))
+			dev_err(np->device, "rx-%d: Counter overflow "
+				"RXMISC discard\n", rx_channel);
+	}
+
+	/* WRED (Weighted Random Early Discard) by hardware */
+	wred = nr64(RED_DIS_CNT(rx_channel));
+	if (unlikely((wred & RED_DIS_CNT_COUNT) > limit)) {
+		nw64(RED_DIS_CNT(rx_channel), 0);
+		rp->rx_dropped += wred & RED_DIS_CNT_COUNT;
+
+		if (unlikely(wred & RED_DIS_CNT_OFLOW))
+			dev_err(np->device, "rx-%d: Counter overflow "
+				"WRED discard\n", rx_channel);
+	}
+}
+
 static int niu_rx_work(struct niu *np, struct rx_ring_info *rp, int budget)
 {
 	int qlen, rcr_done = 0, work_done = 0;
@@ -3569,6 +3614,8 @@ static int niu_rx_work(struct niu *np, struct rx_ring_info *rp, int budget)
 
 	nw64(RX_DMA_CTL_STAT(rp->rx_channel), stat);
 
+	niu_sync_rx_discard_stats(np, rp, 0x7FFF);
+
 	return work_done;
 }
 
@@ -6050,6 +6097,8 @@ static void niu_get_rx_stats(struct niu *np)
 	for (i = 0; i < np->num_rx_rings; i++) {
 		struct rx_ring_info *rp = &np->rx_rings[i];
 
+		niu_sync_rx_discard_stats(np, rp, 0);
+
 		pkts += rp->rx_packets;
 		bytes += rp->rx_bytes;
 		dropped += rp->rx_dropped;
@@ -6991,6 +7040,8 @@ static void niu_get_ethtool_stats(struct net_device *dev,
 	for (i = 0; i < np->num_rx_rings; i++) {
 		struct rx_ring_info *rp = &np->rx_rings[i];
 
+		niu_sync_rx_discard_stats(np, rp, 0);
+
 		data[0] = rp->rx_channel;
 		data[1] = rp->rx_packets;
 		data[2] = rp->rx_bytes;

[PATCH 2/3] NIU: Implement discard counters, info/debug statements.

[Jesper Dangaard Brouer]

Discard packet counter debug statements that can be turned on
runtime, by users to assist debugging of the driver code.

Signed-off-by: Jesper Dangaard Brouer <hawk@comx.dk>
---

 drivers/net/niu.c |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/drivers/net/niu.c b/drivers/net/niu.c
index 1bd7018..9f6a98f 100644
--- a/drivers/net/niu.c
+++ b/drivers/net/niu.c
@@ -3560,6 +3560,9 @@ static inline void niu_sync_rx_discard_stats(struct niu *np,
 		if (unlikely(misc & RXMISC_OFLOW))
 			dev_err(np->device, "rx-%d: Counter overflow "
 				"RXMISC discard\n", rx_channel);
+
+		niudbg(RX_ERR, "%s-rx-%d: MISC drop=%u over=%u\n",
+		       np->dev->name, rx_channel, misc, misc-limit);
 	}
 
 	/* WRED (Weighted Random Early Discard) by hardware */
@@ -3571,6 +3574,9 @@ static inline void niu_sync_rx_discard_stats(struct niu *np,
 		if (unlikely(wred & RED_DIS_CNT_OFLOW))
 			dev_err(np->device, "rx-%d: Counter overflow "
 				"WRED discard\n", rx_channel);
+
+		niudbg(RX_ERR, "%s-rx-%d: WRED drop=%u over=%u\n",
+		       np->dev->name, rx_channel, wred, wred-limit);
 	}
 }
 

[PATCH 3/3] NIU: Implement discard counters, optimize

[Jesper Dangaard Brouer]

Optimize the lightly loaded case, by only syncronizing discards stats
when qlen > 10 indicate potential for drops.

Notice Robert Olsson might disagree with this patch.

Signed-off-by: Jesper Dangaard Brouer <hawk@comx.dk>
---

 drivers/net/niu.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/drivers/net/niu.c b/drivers/net/niu.c
index 9f6a98f..7746230 100644
--- a/drivers/net/niu.c
+++ b/drivers/net/niu.c
@@ -3620,7 +3620,9 @@ static int niu_rx_work(struct niu *np, struct rx_ring_info *rp, int budget)
 
 	nw64(RX_DMA_CTL_STAT(rp->rx_channel), stat);
 
-	niu_sync_rx_discard_stats(np, rp, 0x7FFF);
+	/* Only sync discards stats when qlen indicate potential for drops */
+	if (qlen > 10)
+		niu_sync_rx_discard_stats(np, rp, 0x7FFF);
 
 	return work_done;
 }

Re: [PATCH 0/3] NIU: Implement discard counters

[David Miller]

From: Jesper Dangaard Brouer <hawk@comx.dk>
Date: Thu, 18 Dec 2008 14:52:11 +0100

> 
> This patch series implements discard counter stats for the NIU driver.
> 
> This work has been done together with Robert Olsson.  We have come to a
> stage where the patches needs public review, and possible discussion.
> 
> Tested with hardware:
>  Sun Quad GbE x8 PCIe
>  Sun Dual 10GbE XFP PCIe

To be honest these patches look great, I'll add them to
net-next-2.6

Thanks!