[PATCH 0/6] net: thunderx: Setting IRQ affinity hints and other optimizations

[PATCH 0/6] net: thunderx: Setting IRQ affinity hints and other optimizations

[sunil.kovvuri at gmail.com]

From: Sunil Goutham <sgoutham@cavium.com>

This patch series contains changes
- To add support for virtual function's irq affinity hint
- Replace napi_schedule() with napi_schedule_irqoff()
- Reduce page allocation overhead by allocating pages
  of higher order when pagesize is 4KB.
- Add couple of stats which helps in debugging
- Some miscellaneous changes to BGX driver.


David Daney (1):
  net: thunderx: bgx: Use standard firmware node infrastructure.

Robert Richter (1):
  net: thunderx: bgx: Add log message when setting mac address

Sunil Goutham (3):
  net: thunderx: Use napi_schedule_irqoff()
  net: thunderx: Assign affinity hints to vf's interrupts
  net: thunderx: Alloc higher order pages when pagesize is small

Thanneeru Srinivasulu (1):
  net, thunderx: Add TX timeout and RX buffer alloc failure stats.

 drivers/net/ethernet/cavium/thunder/nic.h          |    4 ++
 .../net/ethernet/cavium/thunder/nicvf_ethtool.c    |    2 +
 drivers/net/ethernet/cavium/thunder/nicvf_main.c   |   40 ++++++++++++++++++-
 drivers/net/ethernet/cavium/thunder/nicvf_queues.c |    5 +-
 drivers/net/ethernet/cavium/thunder/thunder_bgx.c  |   38 +++++++++++--------
 5 files changed, 67 insertions(+), 22 deletions(-)

[PATCH 1/6] net, thunderx: Add TX timeout and RX buffer alloc failure stats.

[sunil.kovvuri at gmail.com]

From: Thanneeru Srinivasulu <tsrinivasulu@caviumnetworks.com>

When system is low on atomic memory, too many error messages are logged.
Since this is not a total failure but a simple switch to non-atomic allocation
better to have a stat.

Also add a stat for reset, kicked due to transmit watchdog timeout.

Signed-off-by: Thanneeru Srinivasulu <tsrinivasulu@caviumnetworks.com>
Signed-off-by: Sunil Goutham <sgoutham@cavium.com>
---
 drivers/net/ethernet/cavium/thunder/nic.h          |    3 +++
 .../net/ethernet/cavium/thunder/nicvf_ethtool.c    |    2 ++
 drivers/net/ethernet/cavium/thunder/nicvf_main.c   |    1 +
 drivers/net/ethernet/cavium/thunder/nicvf_queues.c |    3 +--
 4 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/cavium/thunder/nic.h b/drivers/net/ethernet/cavium/thunder/nic.h
index 6888288..8af363a 100644
--- a/drivers/net/ethernet/cavium/thunder/nic.h
+++ b/drivers/net/ethernet/cavium/thunder/nic.h
@@ -248,10 +248,13 @@ struct nicvf_drv_stats {
 	u64 rx_frames_jumbo;
 	u64 rx_drops;
 
+	u64 rcv_buffer_alloc_failures;
+
 	/* Tx */
 	u64 tx_frames_ok;
 	u64 tx_drops;
 	u64 tx_tso;
+	u64 tx_timeout;
 	u64 txq_stop;
 	u64 txq_wake;
 };
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_ethtool.c b/drivers/net/ethernet/cavium/thunder/nicvf_ethtool.c
index a12b2e3..d2d8ef2 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_ethtool.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_ethtool.c
@@ -89,9 +89,11 @@ static const struct nicvf_stat nicvf_drv_stats[] = {
 	NICVF_DRV_STAT(rx_frames_1518),
 	NICVF_DRV_STAT(rx_frames_jumbo),
 	NICVF_DRV_STAT(rx_drops),
+	NICVF_DRV_STAT(rcv_buffer_alloc_failures),
 	NICVF_DRV_STAT(tx_frames_ok),
 	NICVF_DRV_STAT(tx_tso),
 	NICVF_DRV_STAT(tx_drops),
+	NICVF_DRV_STAT(tx_timeout),
 	NICVF_DRV_STAT(txq_stop),
 	NICVF_DRV_STAT(txq_wake),
 };
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_main.c b/drivers/net/ethernet/cavium/thunder/nicvf_main.c
index c24cb2a..95db6b7 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_main.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_main.c
@@ -1394,6 +1394,7 @@ static void nicvf_tx_timeout(struct net_device *dev)
 		netdev_warn(dev, "%s: Transmit timed out, resetting\n",
 			    dev->name);
 
+	nic->drv_stats.tx_timeout++;
 	schedule_work(&nic->reset_task);
 }
 
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
index d0d1b54..50ab6f4 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
@@ -96,8 +96,7 @@ static inline int nicvf_alloc_rcv_buffer(struct nicvf *nic, gfp_t gfp,
 		nic->rb_page = alloc_pages(gfp | __GFP_COMP | __GFP_NOWARN,
 					   order);
 		if (!nic->rb_page) {
-			netdev_err(nic->netdev,
-				   "Failed to allocate new rcv buffer\n");
+			nic->drv_stats.rcv_buffer_alloc_failures++;
 			return -ENOMEM;
 		}
 		nic->rb_page_offset = 0;
-- 
1.7.1

[PATCH 2/6] net: thunderx: Use napi_schedule_irqoff()

[sunil.kovvuri at gmail.com]

From: Sunil Goutham <sgoutham@cavium.com>

napi_schedule is being called from hard irq context, hence
switch to napi_schedule_irqoff which avoids unneeded call
to local_irq_save and local_irq_restore.

Signed-off-by: Sunil Goutham <sgoutham@cavium.com>
---
 drivers/net/ethernet/cavium/thunder/nicvf_main.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_main.c b/drivers/net/ethernet/cavium/thunder/nicvf_main.c
index 95db6b7..c6f146c 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_main.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_main.c
@@ -828,7 +828,7 @@ static irqreturn_t nicvf_intr_handler(int irq, void *cq_irq)
 	nicvf_disable_intr(nic, NICVF_INTR_CQ, qidx);
 
 	/* Schedule NAPI */
-	napi_schedule(&cq_poll->napi);
+	napi_schedule_irqoff(&cq_poll->napi);
 
 	/* Clear interrupt */
 	nicvf_clear_intr(nic, NICVF_INTR_CQ, qidx);
-- 
1.7.1

[PATCH 3/6] net: thunderx: Assign affinity hints to vf's interrupts

[sunil.kovvuri at gmail.com]

From: Sunil Goutham <sgoutham@cavium.com>

This affinity hint can be used by user space irqbalance tool to set
preferred CPU mask for irqs registered by this VF. Irqbalance needs
to be in 'exact' mode to set irq affinity same as indicated by
affinity hint.

Signed-off-by: Sunil Goutham <sgoutham@cavium.com>
---
 drivers/net/ethernet/cavium/thunder/nic.h        |    1 +
 drivers/net/ethernet/cavium/thunder/nicvf_main.c |   37 ++++++++++++++++++++-
 2 files changed, 36 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/cavium/thunder/nic.h b/drivers/net/ethernet/cavium/thunder/nic.h
index 8af363a..00cc915 100644
--- a/drivers/net/ethernet/cavium/thunder/nic.h
+++ b/drivers/net/ethernet/cavium/thunder/nic.h
@@ -309,6 +309,7 @@ struct nicvf {
 	struct msix_entry	msix_entries[NIC_VF_MSIX_VECTORS];
 	char			irq_name[NIC_VF_MSIX_VECTORS][20];
 	bool			irq_allocated[NIC_VF_MSIX_VECTORS];
+	cpumask_var_t		affinity_mask[NIC_VF_MSIX_VECTORS];
 
 	/* VF <-> PF mailbox communication */
 	bool			pf_acked;
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_main.c b/drivers/net/ethernet/cavium/thunder/nicvf_main.c
index c6f146c..90ce93e 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_main.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_main.c
@@ -899,6 +899,31 @@ static void nicvf_disable_msix(struct nicvf *nic)
 	}
 }
 
+static void nicvf_set_irq_affinity(struct nicvf *nic)
+{
+	int vec, cpu;
+	int irqnum;
+
+	for (vec = 0; vec < nic->num_vec; vec++) {
+		if (!nic->irq_allocated[vec])
+			continue;
+
+		if (!zalloc_cpumask_var(&nic->affinity_mask[vec], GFP_KERNEL))
+			return;
+		 /* CQ interrupts */
+		if (vec < NICVF_INTR_ID_SQ)
+			/* Leave CPU0 for RBDR and other interrupts */
+			cpu = nicvf_netdev_qidx(nic, vec) + 1;
+		else
+			cpu = 0;
+
+		cpumask_set_cpu(cpumask_local_spread(cpu, nic->node),
+				nic->affinity_mask[vec]);
+		irqnum = nic->msix_entries[vec].vector;
+		irq_set_affinity_hint(irqnum, nic->affinity_mask[vec]);
+	}
+}
+
 static int nicvf_register_interrupts(struct nicvf *nic)
 {
 	int irq, ret = 0;
@@ -944,8 +969,13 @@ static int nicvf_register_interrupts(struct nicvf *nic)
 	ret = request_irq(nic->msix_entries[irq].vector,
 			  nicvf_qs_err_intr_handler,
 			  0, nic->irq_name[irq], nic);
-	if (!ret)
-		nic->irq_allocated[irq] = true;
+	if (ret)
+		goto err;
+
+	nic->irq_allocated[irq] = true;
+
+	/* Set IRQ affinities */
+	nicvf_set_irq_affinity(nic);
 
 err:
 	if (ret)
@@ -963,6 +993,9 @@ static void nicvf_unregister_interrupts(struct nicvf *nic)
 		if (!nic->irq_allocated[irq])
 			continue;
 
+		irq_set_affinity_hint(nic->msix_entries[irq].vector, NULL);
+		free_cpumask_var(nic->affinity_mask[irq]);
+
 		if (irq < NICVF_INTR_ID_SQ)
 			free_irq(nic->msix_entries[irq].vector, nic->napi[irq]);
 		else
-- 
1.7.1

[PATCH 4/6] net: thunderx: bgx: Use standard firmware node infrastructure.

[sunil.kovvuri at gmail.com]

From: David Daney <david.daney@cavium.com>

In the case of OF device tree, the firmware information is attached to
the BGX device structure in the standard manner, so use the firmware
iterators and accessors where possible.

Signed-off-by: David Daney <david.daney@cavium.com>
Signed-off-by: Sunil Goutham <sgoutham@cavium.com>
---
 drivers/net/ethernet/cavium/thunder/thunder_bgx.c |   27 +++++++++++----------
 1 files changed, 14 insertions(+), 13 deletions(-)

diff --git a/drivers/net/ethernet/cavium/thunder/thunder_bgx.c b/drivers/net/ethernet/cavium/thunder/thunder_bgx.c
index 9df26c2..111835b 100644
--- a/drivers/net/ethernet/cavium/thunder/thunder_bgx.c
+++ b/drivers/net/ethernet/cavium/thunder/thunder_bgx.c
@@ -968,26 +968,27 @@ static int bgx_init_acpi_phy(struct bgx *bgx)
 
 static int bgx_init_of_phy(struct bgx *bgx)
 {
-	struct device_node *np;
-	struct device_node *np_child;
+	struct fwnode_handle *fwn;
 	u8 lmac = 0;
-	char bgx_sel[5];
 	const char *mac;
 
-	/* Get BGX node from DT */
-	snprintf(bgx_sel, 5, "bgx%d", bgx->bgx_id);
-	np = of_find_node_by_name(NULL, bgx_sel);
-	if (!np)
-		return -ENODEV;
+	device_for_each_child_node(&bgx->pdev->dev, fwn) {
+		struct device_node *phy_np;
+		struct device_node *node = to_of_node(fwn);
+
+		/* If it is not an OF node we cannot handle it yet, so
+		 * exit the loop.
+		 */
+		if (!node)
+			break;
 
-	for_each_child_of_node(np, np_child) {
-		struct device_node *phy_np = of_parse_phandle(np_child,
-							      "phy-handle", 0);
+		phy_np = of_parse_phandle(node, "phy-handle", 0);
 		if (!phy_np)
 			continue;
+
 		bgx->lmac[lmac].phydev = of_phy_find_device(phy_np);
 
-		mac = of_get_mac_address(np_child);
+		mac = of_get_mac_address(node);
 		if (mac)
 			ether_addr_copy(bgx->lmac[lmac].mac, mac);
 
@@ -995,7 +996,7 @@ static int bgx_init_of_phy(struct bgx *bgx)
 		bgx->lmac[lmac].lmacid = lmac;
 		lmac++;
 		if (lmac == MAX_LMAC_PER_BGX) {
-			of_node_put(np_child);
+			of_node_put(node);
 			break;
 		}
 	}
-- 
1.7.1

[PATCH 5/6] net: thunderx: bgx: Add log message when setting mac address

[sunil.kovvuri at gmail.com]

From: Robert Richter <rrichter@cavium.com>

Signed-off-by: Robert Richter <rrichter@cavium.com>
Signed-off-by: Sunil Goutham <sgoutham@cavium.com>
---
 drivers/net/ethernet/cavium/thunder/thunder_bgx.c |   11 ++++++++---
 1 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/drivers/net/ethernet/cavium/thunder/thunder_bgx.c b/drivers/net/ethernet/cavium/thunder/thunder_bgx.c
index 111835b..cfee496 100644
--- a/drivers/net/ethernet/cavium/thunder/thunder_bgx.c
+++ b/drivers/net/ethernet/cavium/thunder/thunder_bgx.c
@@ -886,7 +886,8 @@ static void bgx_get_qlm_mode(struct bgx *bgx)
 
 #ifdef CONFIG_ACPI
 
-static int acpi_get_mac_address(struct acpi_device *adev, u8 *dst)
+static int acpi_get_mac_address(struct device *dev, struct acpi_device *adev,
+				u8 *dst)
 {
 	u8 mac[ETH_ALEN];
 	int ret;
@@ -897,10 +898,13 @@ static int acpi_get_mac_address(struct acpi_device *adev, u8 *dst)
 		goto out;
 
 	if (!is_valid_ether_addr(mac)) {
+		dev_warn(dev, "MAC address invalid: %pM\n", mac);
 		ret = -EINVAL;
 		goto out;
 	}
 
+	dev_info(dev, "MAC address set to: %pM\n", mac);
+
 	memcpy(dst, mac, ETH_ALEN);
 out:
 	return ret;
@@ -911,14 +915,15 @@ static acpi_status bgx_acpi_register_phy(acpi_handle handle,
 					 u32 lvl, void *context, void **rv)
 {
 	struct bgx *bgx = context;
+	struct device *dev = &bgx->pdev->dev;
 	struct acpi_device *adev;
 
 	if (acpi_bus_get_device(handle, &adev))
 		goto out;
 
-	acpi_get_mac_address(adev, bgx->lmac[bgx->lmac_count].mac);
+	acpi_get_mac_address(dev, adev, bgx->lmac[bgx->lmac_count].mac);
 
-	SET_NETDEV_DEV(&bgx->lmac[bgx->lmac_count].netdev, &bgx->pdev->dev);
+	SET_NETDEV_DEV(&bgx->lmac[bgx->lmac_count].netdev, dev);
 
 	bgx->lmac[bgx->lmac_count].lmacid = bgx->lmac_count;
 out:
-- 
1.7.1

[PATCH 5/6] net: thunderx: bgx: Add log message when setting mac address

[Sergei Shtylyov]

Hello.

On 2/8/2016 3:07 PM, sunil.kovvuri at gmail.com wrote:

> From: Robert Richter <rrichter@cavium.com>
>
> Signed-off-by: Robert Richter <rrichter@cavium.com>
> Signed-off-by: Sunil Goutham <sgoutham@cavium.com>
> ---
>   drivers/net/ethernet/cavium/thunder/thunder_bgx.c |   11 ++++++++---
>   1 files changed, 8 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/net/ethernet/cavium/thunder/thunder_bgx.c b/drivers/net/ethernet/cavium/thunder/thunder_bgx.c
> index 111835b..cfee496 100644
> --- a/drivers/net/ethernet/cavium/thunder/thunder_bgx.c
> +++ b/drivers/net/ethernet/cavium/thunder/thunder_bgx.c
[...]
> @@ -897,10 +898,13 @@ static int acpi_get_mac_address(struct acpi_device *adev, u8 *dst)
>   		goto out;
>
>   	if (!is_valid_ether_addr(mac)) {
> +		dev_warn(dev, "MAC address invalid: %pM\n", mac);

    dev_er(), maybe?

>   		ret = -EINVAL;
>   		goto out;
>   	}
>
> +	dev_info(dev, "MAC address set to: %pM\n", mac);
> +
>   	memcpy(dst, mac, ETH_ALEN);
>   out:
>   	return ret;
[...]

MBR, Sergei

[PATCH 5/6] net: thunderx: bgx: Add log message when setting mac address

[Robert Richter]

On 08.02.16 16:30:37, Sergei Shtylyov wrote:
> >@@ -897,10 +898,13 @@ static int acpi_get_mac_address(struct acpi_device *adev, u8 *dst)
> >  		goto out;
> >
> >  	if (!is_valid_ether_addr(mac)) {
> >+		dev_warn(dev, "MAC address invalid: %pM\n", mac);
> 
>    dev_er(), maybe?

Since the driver may continue, my choice was a warning only.

-Robert

> 
> >  		ret = -EINVAL;
> >  		goto out;
> >  	}
> >
> >+	dev_info(dev, "MAC address set to: %pM\n", mac);
> >+
> >  	memcpy(dst, mac, ETH_ALEN);
> >  out:
> >  	return ret;
> [...]

[PATCH 5/6] net: thunderx: bgx: Add log message when setting mac address

[Bjørn Mork]

Sergei Shtylyov <sergei.shtylyov@cogentembedded.com> writes:

>    dev_er(), maybe?

I like that!

It's often hard to know whether to print something or be quiet.
pr_er(), dev_er(), netdev_er() etc would be the perfect solution to that
problem.


Bj?rn

[PATCH 5/6] net: thunderx: bgx: Add log message when setting mac address

[Sergei Shtylyov]

On 02/08/2016 04:49 PM, Bj?rn Mork wrote:

>>     dev_er(), maybe?
>
> I like that!
>
> It's often hard to know whether to print something or be quiet.
> pr_er(), dev_er(), netdev_er() etc would be the perfect solution to that
> problem.

    :-D
    Sorry for the typo...

> Bj?rn

MBR, Sergei

[PATCH 6/6] net: thunderx: Alloc higher order pages when pagesize is small

[sunil.kovvuri at gmail.com]

From: Sunil Goutham <sgoutham@cavium.com>

Allocate higher order pages when pagesize is small, this will
reduce number of calls to page allocator and wastage of memory.

Signed-off-by: Sunil Goutham <sgoutham@cavium.com>
---
 drivers/net/ethernet/cavium/thunder/nicvf_queues.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
index 50ab6f4..5adb208 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
@@ -78,7 +78,7 @@ static void nicvf_free_q_desc_mem(struct nicvf *nic, struct q_desc_mem *dmem)
 static inline int nicvf_alloc_rcv_buffer(struct nicvf *nic, gfp_t gfp,
 					 u32 buf_len, u64 **rbuf)
 {
-	int order = get_order(buf_len);
+	int order = (PAGE_SIZE <= 4096) ?  PAGE_ALLOC_COSTLY_ORDER : 0;
 
 	/* Check if request can be accomodated in previous allocated page */
 	if (nic->rb_page) {
-- 
1.7.1

[PATCH 0/6] net: thunderx: Setting IRQ affinity hints and other optimizations

[Sunil Kovvuri]

Hi David,

If time permits, can you please look at this patchset.

Thanks,
Sunil.

On Mon, Feb 8, 2016 at 5:37 PM,  <sunil.kovvuri@gmail.com> wrote:
> From: Sunil Goutham <sgoutham@cavium.com>
>
> This patch series contains changes
> - To add support for virtual function's irq affinity hint
> - Replace napi_schedule() with napi_schedule_irqoff()
> - Reduce page allocation overhead by allocating pages
>   of higher order when pagesize is 4KB.
> - Add couple of stats which helps in debugging
> - Some miscellaneous changes to BGX driver.
>
>
> David Daney (1):
>   net: thunderx: bgx: Use standard firmware node infrastructure.
>
> Robert Richter (1):
>   net: thunderx: bgx: Add log message when setting mac address
>
> Sunil Goutham (3):
>   net: thunderx: Use napi_schedule_irqoff()
>   net: thunderx: Assign affinity hints to vf's interrupts
>   net: thunderx: Alloc higher order pages when pagesize is small
>
> Thanneeru Srinivasulu (1):
>   net, thunderx: Add TX timeout and RX buffer alloc failure stats.
>
>  drivers/net/ethernet/cavium/thunder/nic.h          |    4 ++
>  .../net/ethernet/cavium/thunder/nicvf_ethtool.c    |    2 +
>  drivers/net/ethernet/cavium/thunder/nicvf_main.c   |   40 ++++++++++++++++++-
>  drivers/net/ethernet/cavium/thunder/nicvf_queues.c |    5 +-
>  drivers/net/ethernet/cavium/thunder/thunder_bgx.c  |   38 +++++++++++--------
>  5 files changed, 67 insertions(+), 22 deletions(-)
>

[PATCH 0/6] net: thunderx: Setting IRQ affinity hints and other optimizations

[David Miller]

From: Sunil Kovvuri <sunil.kovvuri@gmail.com>
Date: Thu, 11 Feb 2016 18:56:48 +0530

> If time permits, can you please look at this patchset.

You were given feedback and I expect you to address that feedback
and resubmit this series.

This is what the "Changed Requested" state in patchwork means.

[PATCH 0/6] net: thunderx: Setting IRQ affinity hints and other optimizations

[Robert Richter]

On 11.02.16 09:33:41, David Miller wrote:
> From: Sunil Kovvuri <sunil.kovvuri@gmail.com>
> Date: Thu, 11 Feb 2016 18:56:48 +0530
> 
> > If time permits, can you please look at this patchset.
> 
> You were given feedback and I expect you to address that feedback
> and resubmit this series.
> 
> This is what the "Changed Requested" state in patchwork means.

Sunil, should I send a reworked version of my patch to you, or do you
just want to change it to dev_err() for me?

Thanks,

-Robert