macb napi strange behavior

macb napi strange behavior

[Nicolae Rosia]

Hello,

I'm trying to determine why I have a huge number of IRQs for only a
macb interface and the other one works just fine (low IRQ activity). I
have activated IP forward and I'm just forwarding packets from eth0 to
eth1.
The platform is Zynq7, Linux kernel 4.0, vanilla macb.

cat /proc/interrupts:
[...]
144:     679425     0       GIC  54  eth0
145:   17867097          0       GIC  77  eth1
[...]

Any ideas?

Best regards,
Nicolae Rosia

Re: macb napi strange behavior

[Jaeden Amero]

On 06/17/2015 11:09 AM, Nicolae Rosia wrote:
> I'm trying to determine why I have a huge number of IRQs for only a
> macb interface and the other one works just fine (low IRQ activity). I
> have activated IP forward and I'm just forwarding packets from eth0 to
> eth1.
> The platform is Zynq7, Linux kernel 4.0, vanilla macb.
> 
> cat /proc/interrupts:
> [...]
> 144:     679425     0       GIC  54  eth0
> 145:   17867097          0       GIC  77  eth1
> [...]
> 
> Any ideas?

The times we've seen tons of interrupts on Ethernet with interrupts
routed through the PL was when the FPGA was unprogrammed (or in the
process of being reprogrammed), or was configured with the interrupt
line tied to asserted.

In the latter case, Linux would eventually stop handling any more
interrupts for that port due to the interrupt storm.

In the former case, there isn't much one can do except make sure that
any FPGA-routed interrupts are unregistered and disabled before FPGA
reprogramming and then to re-enable those interrupts after reprogramming.

It'd be nice to have some sort of notification to drivers, given when
the FPGA state changes, when hardware the drivers are responsible for
disappears or gets disconnected. This is an area of research for us at NI.

Cheers,
Jaeden

Re: macb napi strange behavior

[Nicolae Rosia]

Hi,

On Wed, Jun 17, 2015 at 9:54 PM, Jaeden Amero <jaeden.amero@ni.com> wrote:
> On 06/17/2015 11:09 AM, Nicolae Rosia wrote:
> The times we've seen tons of interrupts on Ethernet with interrupts
> routed through the PL was when the FPGA was unprogrammed (or in the
> process of being reprogrammed), or was configured with the interrupt
> line tied to asserted.
>
> In the latter case, Linux would eventually stop handling any more
> interrupts for that port due to the interrupt storm.
>
This isn't the case. The FPGA is programmed, and indeed I'm using the
second MAC routed through PL to SFP.
The interesting thing is that I'm seeing the exact behavior on the
other side (another Zynq7 board), with eth0 having lots of interrupts.
It seems that the interface receiving packets doesn't have a high IRQ
activity in contrast to the one sending packets.

Re: macb napi strange behavior

[Florian Fainelli]

2015-06-17 12:00 GMT-07:00 Nicolae Rosia <nicolae.rosia@gmail.com>:
> Hi,
>
> On Wed, Jun 17, 2015 at 9:54 PM, Jaeden Amero <jaeden.amero@ni.com> wrote:
>> On 06/17/2015 11:09 AM, Nicolae Rosia wrote:
>> The times we've seen tons of interrupts on Ethernet with interrupts
>> routed through the PL was when the FPGA was unprogrammed (or in the
>> process of being reprogrammed), or was configured with the interrupt
>> line tied to asserted.
>>
>> In the latter case, Linux would eventually stop handling any more
>> interrupts for that port due to the interrupt storm.
>>
> This isn't the case. The FPGA is programmed, and indeed I'm using the
> second MAC routed through PL to SFP.
> The interesting thing is that I'm seeing the exact behavior on the
> other side (another Zynq7 board), with eth0 having lots of interrupts.
> It seems that the interface receiving packets doesn't have a high IRQ
> activity in contrast to the one sending packets.

Typically, NAPI is used at the receive side of the Ethernet NIC/driver
to lower the hard/soft interrupt context switch, although there is
nothing that prevent you to implement a similar scheme for the
transmit side. Usually, for transmit you will be submitting one packet
for transmission and get a completion interrupt, so without interrupt
coalescing (software or hardware) you can end-up with 1 interrupt per
packet transmitted.
-- 
Florian

Re: macb napi strange behavior

[Francois Romieu]

Florian Fainelli <f.fainelli@gmail.com> :
[...]
> Typically, NAPI is used at the receive side of the Ethernet NIC/driver
> to lower the hard/soft interrupt context switch, although there is
> nothing that prevent you to implement a similar scheme for the
> transmit side. Usually, for transmit you will be submitting one packet
> for transmission and get a completion interrupt, so without interrupt
> coalescing (software or hardware) you can end-up with 1 interrupt per
> packet transmitted.

The wording is a bit shy: there is a long standing policy to move
everything to NAPI context (as well as go mostly lockless, etc.).

Any taker to move macb Tx processing to NAPI context or should I consider it ?

-- 
Ueimor

Re: macb napi strange behavior

[Nicolae Rosia]

On Sat, Jun 20, 2015 at 7:43 PM, Francois Romieu <romieu@fr.zoreil.com> wrote:
> Florian Fainelli <f.fainelli@gmail.com> :
> [...]
>> Typically, NAPI is used at the receive side of the Ethernet NIC/driver
>> to lower the hard/soft interrupt context switch, although there is
>> nothing that prevent you to implement a similar scheme for the
>> transmit side. Usually, for transmit you will be submitting one packet
>> for transmission and get a completion interrupt, so without interrupt
>> coalescing (software or hardware) you can end-up with 1 interrupt per
>> packet transmitted.
>
> The wording is a bit shy: there is a long standing policy to move
> everything to NAPI context (as well as go mostly lockless, etc.).
>
> Any taker to move macb Tx processing to NAPI context or should I consider it ?
>
I can help with testing!

Re: macb napi strange behavior

[Nicolae Rosia]

Hi,

I gave it a shot. Can you please take a look?
I don't know how to deal with multiple queues since Zynq 7000 has one
queue per interface.
I get a performance improvement of over 110 Mbps in IP forwarding (680
Mbps vs 570 Mbps) and a massive reduction of interrupts.

Patch below.

From: Nicolae Rosia <nicolae.rosia@certsign.ro>
Date: Fri, 26 Jun 2015 19:44:50 +0300
Subject: [PATCH] macb: use tx napi

---
 drivers/net/ethernet/cadence/macb.c | 37 ++++++++++++++++++++++++++++++++++---
 1 file changed, 34 insertions(+), 3 deletions(-)

diff --git a/drivers/net/ethernet/cadence/macb.c
b/drivers/net/ethernet/cadence/macb.c
index 30eedb1..2a3ecf7 100644
--- a/drivers/net/ethernet/cadence/macb.c
+++ b/drivers/net/ethernet/cadence/macb.c
@@ -602,10 +602,13 @@ static void macb_tx_interrupt(struct macb_queue *queue)
 {
     unsigned int tail;
     unsigned int head;
+#if 0
     u32 status;
+#endif
     struct macb *bp = queue->bp;
     u16 queue_index = queue - bp->queues;

+#if 0
     status = macb_readl(bp, TSR);
     macb_writel(bp, TSR, status);

@@ -614,6 +617,7 @@ static void macb_tx_interrupt(struct macb_queue *queue)

     netdev_vdbg(bp->dev, "macb_tx_interrupt status = 0x%03lx\n",
         (unsigned long)status);
+#endif

     head = queue->tx_head;
     for (tail = queue->tx_tail; tail != head; tail++) {
@@ -951,12 +955,17 @@ static int macb_poll(struct napi_struct *napi, int budget)
     status = macb_readl(bp, RSR);
     macb_writel(bp, RSR, status);

-    work_done = 0;
+    status = macb_readl(bp, TSR);
+    if (status) {
+        macb_writel(bp, TSR, status);
+        macb_tx_interrupt(&bp->queues[0]);
+    }

     netdev_vdbg(bp->dev, "poll: status = %08lx, budget = %d\n",
            (unsigned long)status, budget);

     work_done = bp->macbgem_ops.mog_rx(bp, budget);
+
     if (work_done < budget) {
         napi_complete(napi);

@@ -969,6 +978,16 @@ static int macb_poll(struct napi_struct *napi, int budget)
         } else {
             macb_writel(bp, IER, MACB_RX_INT_FLAGS);
         }
+
+        /* Packets received while interrupts were disabled */
+        status = macb_readl(bp, TSR);
+        if (status) {
+            if (bp->caps & MACB_CAPS_ISR_CLEAR_ON_WRITE)
+                macb_writel(bp, ISR, MACB_BIT(TCOMP));
+            napi_reschedule(napi);
+        } else {
+            macb_writel(bp, IER, MACB_TX_INT_FLAGS);
+        }
     }

     /* TODO: Handle errors */
@@ -1029,8 +1048,17 @@ static irqreturn_t macb_interrupt(int irq, void *dev_id)
             break;
         }

-        if (status & MACB_BIT(TCOMP))
-            macb_tx_interrupt(queue);
+        if (status & MACB_BIT(TCOMP)) {
+            /* disable tx interrupts */
+            queue_writel(queue, IDR, MACB_TX_INT_FLAGS);
+            if (bp->caps & MACB_CAPS_ISR_CLEAR_ON_WRITE)
+                queue_writel(queue, ISR, MACB_BIT(TCOMP));
+
+            if (napi_schedule_prep(&bp->napi)) {
+                netdev_vdbg(bp->dev, "scheduling TX softirq\n");
+                __napi_schedule(&bp->napi);
+            }
+        }

         /*
          * Link change detection isn't possible with RMII, so we'll
@@ -2348,6 +2376,7 @@ static int macb_probe(struct platform_device *pdev)
     }

     macb_probe_queues(mem, &queue_mask, &num_queues);
+    dev_err(&pdev->dev, "queue count: %d.\n", num_queues);
     dev = alloc_etherdev_mq(sizeof(*bp), num_queues);
     if (!dev)
         goto err_out_disable_clocks;
@@ -2377,6 +2406,8 @@ static int macb_probe(struct platform_device *pdev)
         if (!(queue_mask & (1 << hw_q)))
             continue;

+        dev_err(&pdev->dev, "queue index: %d\n", q);
+
         queue = &bp->queues[q];
         queue->bp = bp;
         if (hw_q) {
-- 
2.1.0

On Sat, Jun 20, 2015 at 7:43 PM, Francois Romieu <romieu@fr.zoreil.com> wrote:
> Florian Fainelli <f.fainelli@gmail.com> :
> [...]
>> Typically, NAPI is used at the receive side of the Ethernet NIC/driver
>> to lower the hard/soft interrupt context switch, although there is
>> nothing that prevent you to implement a similar scheme for the
>> transmit side. Usually, for transmit you will be submitting one packet
>> for transmission and get a completion interrupt, so without interrupt
>> coalescing (software or hardware) you can end-up with 1 interrupt per
>> packet transmitted.
>
> The wording is a bit shy: there is a long standing policy to move
> everything to NAPI context (as well as go mostly lockless, etc.).
>
> Any taker to move macb Tx processing to NAPI context or should I consider it ?
>
> --
> Ueimor

Re: macb napi strange behavior

[Francois Romieu]

Nicolae Rosia <nicolae.rosia@gmail.com> :
[...]
> I gave it a shot. Can you please take a look?
> I don't know how to deal with multiple queues since Zynq 7000 has one
> queue per interface.

macb_interrupt knows the queue. macb_poll doesn't. Either you store it
somewhere or you go for per queue napi struct.

-- 
Ueimor