[BUG][mvebu] mvneta: cannot request irq 25 on openblocks-ax3

[BUG][mvebu] mvneta: cannot request irq 25 on openblocks-ax3

[Masami Hiramatsu]

Hi,

Here I've hit a bug on the recent kernel. As far as I know, this bug
exists on 3.9-rc1 too.

When I tried the latest mvebu for-next tree
(git://git.infradead.org/users/jcooper/linux.git mvebu/for-next),
I got below warning at bootup time and mvneta didn't work (link was never up).
I ensured that "ifconfig ethX up" always caused that.

Does anyone succeed to boot openblocks-ax3 recently or hit same
trouble?

------------[ cut here ]------------
WARNING: at /ssd/ksrc/linux-3/kernel/irq/manage.c:1417
request_threaded_irq+0x64/0x124()
Modules linked in:
Backtrace:
[<c00119a4>] (dump_backtrace+0x0/0x110) from [<c04b2bac>] (dump_stack+0x18/0x1c)
 r6:c05ad11a r5:00000589 r4:00000000 r3:ed80e000
[<c04b2b94>] (dump_stack+0x0/0x1c) from [<c0021484>]
(warn_slowpath_common+0x54/0x70)
[<c0021430>] (warn_slowpath_common+0x0/0x70) from [<c00214c4>]
(warn_slowpath_null+0x24/0x2c)
 r8:00000000 r7:ef0a4600 r6:ee9c4d40 r5:00000000 r4:c037378c
r3:00000009
[<c00214a0>] (warn_slowpath_null+0x0/0x2c) from [<c0084bb8>]
(request_threaded_irq+0x64/0x124)
[<c0084b54>] (request_threaded_irq+0x0/0x124) from [<c037418c>]
(mvneta_open+0x80/0x15c)
[<c037410c>] (mvneta_open+0x0/0x15c) from [<c03c9e3c>] (__dev_open+0xa0/0xf4)
 r6:00001002 r5:c050bb24 r4:ee9c4800
[<c03c9d9c>] (__dev_open+0x0/0xf4) from [<c03ca070>] (__dev_change_flags+0x94/0x118)
 r7:00000001 r6:00001002 r5:00001003 r4:ee9c4800
[<c03c9fdc>] (__dev_change_flags+0x0/0x118) from [<c03ca170>]
(dev_change_flags+0x18/0x4c)
 r7:00000000 r6:edd0d380 r5:00001002 r4:ee9c4800
[<c03ca158>] (dev_change_flags+0x0/0x4c) from [<c03d6168>] (do_setlink+0x2a4/0x7a0)
 r6:edd0d380 r5:ed80fb4c r4:ee9c4800 r3:00001002
[<c03d5ec4>] (do_setlink+0x0/0x7a0) from [<c03d72d8>] (rtnl_newlink+0x248/0x45c)
[<c03d7090>] (rtnl_newlink+0x0/0x45c) from [<c03d7060>]
(rtnetlink_rcv_msg+0x248/0x278)
[<c03d6e18>] (rtnetlink_rcv_msg+0x0/0x278) from [<c03ead78>]
(netlink_rcv_skb+0x58/0xb4)
[<c03ead20>] (netlink_rcv_skb+0x0/0xb4) from [<c03d5d28>] (rtnetlink_rcv+0x20/0x2c)
 r6:eea75c00 r5:eea87240 r4:eea87240 r3:c03d5d08
[<c03d5d08>] (rtnetlink_rcv+0x0/0x2c) from [<c03ea780>]
(netlink_unicast+0x15c/0x214)
 r4:ef0a3000 r3:c03d5d08
[<c03ea624>] (netlink_unicast+0x0/0x214) from [<c03eab94>]
(netlink_sendmsg+0x2b8/0x348)
[<c03ea8dc>] (netlink_sendmsg+0x0/0x348) from [<c03b2dbc>] (sock_sendmsg+0x88/0xa4)
[<c03b2d34>] (sock_sendmsg+0x0/0xa4) from [<c03b3330>] (__sys_sendmsg+0x1c8/0x25c)
 r7:00000000 r6:00000020 r5:eefccc80 r4:ed80ff64
[<c03b3168>] (__sys_sendmsg+0x0/0x25c) from [<c03b4f30>] (sys_sendmsg+0x44/0x68)
[<c03b4eec>] (sys_sendmsg+0x0/0x68) from [<c000daa0>] (ret_fast_syscall+0x0/0x30)
 r6:be9f367c r5:00000000 r4:5146f675
---[ end trace a8ceb54233e388b5 ]---
mvneta d0070000.ethernet eth0: cannot request irq 25

Thank you,
-- 
Masami HIRAMATSU
IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [BUG][mvebu] mvneta: cannot request irq 25 on openblocks-ax3

[Jason Cooper]

On Tue, Mar 19, 2013 at 10:12:37PM +0900, Masami Hiramatsu wrote:
> Hi,
> 
> Here I've hit a bug on the recent kernel. As far as I know, this bug
> exists on 3.9-rc1 too.
> 
> When I tried the latest mvebu for-next tree
> (git://git.infradead.org/users/jcooper/linux.git mvebu/for-next),

FYI: that branch isn't stable, it's used as a merge-test of
arm-soc/for-next (also not stable) and any branches I am trying to push
upstream that day.

Gregory has a patch in the works for this.  Hopefully he'll submit
it by the end of the week.

thx,

Jason.

Re: Re: [BUG][mvebu] mvneta: cannot request irq 25 on openblocks-ax3

[Masami Hiramatsu]

Hi Jason,

(2013/03/19 22:33), Jason Cooper wrote:
> On Tue, Mar 19, 2013 at 10:12:37PM +0900, Masami Hiramatsu wrote:
>> Hi,
>>
>> Here I've hit a bug on the recent kernel. As far as I know, this bug
>> exists on 3.9-rc1 too.
>>
>> When I tried the latest mvebu for-next tree
>> (git://git.infradead.org/users/jcooper/linux.git mvebu/for-next),
> 
> FYI: that branch isn't stable, it's used as a merge-test of
> arm-soc/for-next (also not stable) and any branches I am trying to push
> upstream that day.

Thanks! could you tell me which branch is stable?
(however, I'd like to try new fixes/features on my device too :))

> Gregory has a patch in the works for this.  Hopefully he'll submit
> it by the end of the week.

Thank you,

-- 
Masami HIRAMATSU
IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: Re: [BUG][mvebu] mvneta: cannot request irq 25 on openblocks-ax3

[Jason Cooper]

On Tue, Mar 19, 2013 at 11:19:05PM +0900, Masami Hiramatsu wrote:
> Hi Jason,
> 
> (2013/03/19 22:33), Jason Cooper wrote:
> > On Tue, Mar 19, 2013 at 10:12:37PM +0900, Masami Hiramatsu wrote:
> >> Hi,
> >>
> >> Here I've hit a bug on the recent kernel. As far as I know, this bug
> >> exists on 3.9-rc1 too.
> >>
> >> When I tried the latest mvebu for-next tree
> >> (git://git.infradead.org/users/jcooper/linux.git mvebu/for-next),
> > 
> > FYI: that branch isn't stable, it's used as a merge-test of
> > arm-soc/for-next (also not stable) and any branches I am trying to push
> > upstream that day.
> 
> Thanks! could you tell me which branch is stable?
> (however, I'd like to try new fixes/features on my device too :))

Generally, we advise using one of Linus' tags (eg v3.9-rc3).  Only if
that doesn't work, seek out a branch containing the fix/feature.  When
in doubt, I'll do:

$ ./scripts/get_maintainer.pl -f drivers/net/ethernet/marvell/mvneta.c

and ask those listed which branch might contain a fix or feature.

In this case, it'll probably go through David Miller and the netdev
mailinglist, but it isn't there yet.  ;-)

hth,

Jason.

Re: [BUG][mvebu] mvneta: cannot request irq 25 on openblocks-ax3

[Ezequiel Garcia]

[-- Attachment #1: Type: text/plain, Size: 901 bytes --]

Hi Masami,

On Tue, Mar 19, 2013 at 10:12:37PM +0900, Masami Hiramatsu wrote:
> 
> Here I've hit a bug on the recent kernel. As far as I know, this bug
> exists on 3.9-rc1 too.
> 
> When I tried the latest mvebu for-next tree
> (git://git.infradead.org/users/jcooper/linux.git mvebu/for-next),
> I got below warning at bootup time and mvneta didn't work (link was never up).
> I ensured that "ifconfig ethX up" always caused that.
> 
> Does anyone succeed to boot openblocks-ax3 recently or hit same
> trouble?

This is a known bug. Gregory Clement already has a fix and he
will submit it soon. In case you need this fixed ASAP, I'm attaching
you a patch with a fix.

Please note the attached patch is not ready for mainline inclusion,
as I said Gregory will submit a cleaner version soon.

-- 
Ezequiel García, Free Electrons
Embedded Linux, Kernel and Android Engineering
http://free-electrons.com

[-- Attachment #2: 0001-net-mvneta-convert-to-percpu-interrupt.patch --]
[-- Type: text/plain, Size: 1799 bytes --]

>From 03080b4e459b103b97b658789658f118053de522 Mon Sep 17 00:00:00 2001
From: Gregory CLEMENT <gregory.clement@free-electrons.com>
Date: Sat, 9 Feb 2013 22:07:54 +0100
Subject: [PATCH] net: mvneta: convert to percpu interrupt

Signed-off-by: Gregory CLEMENT <gregory.clement@free-electrons.com>
---
 drivers/net/ethernet/marvell/mvneta.c |   12 +++++++++---
 1 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c
index b6025c3..7f63dd4 100644
--- a/drivers/net/ethernet/marvell/mvneta.c
+++ b/drivers/net/ethernet/marvell/mvneta.c
@@ -1800,7 +1800,7 @@ static void mvneta_set_rx_mode(struct net_device *dev)
 /* Interrupt handling - the callback for request_irq() */
 static irqreturn_t mvneta_isr(int irq, void *dev_id)
 {
-	struct mvneta_port *pp = (struct mvneta_port *)dev_id;
+	struct mvneta_port *pp = *(struct mvneta_port **)dev_id;
 
 	/* Mask all interrupts */
 	mvreg_write(pp, MVNETA_INTR_NEW_MASK, 0);
@@ -2368,6 +2368,7 @@ static void mvneta_mdio_remove(struct mvneta_port *pp)
 	phy_disconnect(pp->phy_dev);
 	pp->phy_dev = NULL;
 }
+static struct mvneta_port __percpu **percpu_pp;
 
 static int mvneta_open(struct net_device *dev)
 {
@@ -2386,9 +2387,14 @@ static int mvneta_open(struct net_device *dev)
 	if (ret)
 		goto err_cleanup_rxqs;
 
+	percpu_pp = alloc_percpu(struct mvneta_port *);
+	*__this_cpu_ptr(percpu_pp) = pp;
+
 	/* Connect to port interrupt line */
-	ret = request_irq(pp->dev->irq, mvneta_isr, 0,
-			  MVNETA_DRIVER_NAME, pp);
+	ret = request_percpu_irq(pp->dev->irq, mvneta_isr,
+			  MVNETA_DRIVER_NAME, percpu_pp);
+	enable_percpu_irq(pp->dev->irq, 0);
+
 	if (ret) {
 		netdev_err(pp->dev, "cannot request irq %d\n", pp->dev->irq);
 		goto err_cleanup_txqs;
-- 
1.7.8.6

Re: Re: [BUG][mvebu] mvneta: cannot request irq 25 on openblocks-ax3

[Masami Hiramatsu]

Hi Ezequiel,

(2013/03/19 22:39), Ezequiel Garcia wrote:
> Hi Masami,
> 
> On Tue, Mar 19, 2013 at 10:12:37PM +0900, Masami Hiramatsu wrote:
>>
>> Here I've hit a bug on the recent kernel. As far as I know, this bug
>> exists on 3.9-rc1 too.
>>
>> When I tried the latest mvebu for-next tree
>> (git://git.infradead.org/users/jcooper/linux.git mvebu/for-next),
>> I got below warning at bootup time and mvneta didn't work (link was never up).
>> I ensured that "ifconfig ethX up" always caused that.
>>
>> Does anyone succeed to boot openblocks-ax3 recently or hit same
>> trouble?
> 
> This is a known bug. Gregory Clement already has a fix and he
> will submit it soon. In case you need this fixed ASAP, I'm attaching
> you a patch with a fix.

Thanks! I'll try that.

> Please note the attached patch is not ready for mainline inclusion,
> as I said Gregory will submit a cleaner version soon.

Yeah, I look forward to it :)

Thank you,

-- 
Masami HIRAMATSU
IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [BUG][mvebu] mvneta: cannot request irq 25 on openblocks-ax3

[Gregory CLEMENT]

[-- Attachment #1: Type: text/plain, Size: 1401 bytes --]

On 03/19/2013 03:19 PM, Masami Hiramatsu wrote:
> Hi Ezequiel,
> 
> (2013/03/19 22:39), Ezequiel Garcia wrote:
>> Hi Masami,
>>
>> On Tue, Mar 19, 2013 at 10:12:37PM +0900, Masami Hiramatsu wrote:
>>>
>>> Here I've hit a bug on the recent kernel. As far as I know, this bug
>>> exists on 3.9-rc1 too.
>>>
>>> When I tried the latest mvebu for-next tree
>>> (git://git.infradead.org/users/jcooper/linux.git mvebu/for-next),
>>> I got below warning at bootup time and mvneta didn't work (link was never up).
>>> I ensured that "ifconfig ethX up" always caused that.
>>>
>>> Does anyone succeed to boot openblocks-ax3 recently or hit same
>>> trouble?
>>
>> This is a known bug. Gregory Clement already has a fix and he
>> will submit it soon. In case you need this fixed ASAP, I'm attaching
>> you a patch with a fix.
> 
> Thanks! I'll try that.
> 
>> Please note the attached patch is not ready for mainline inclusion,
>> as I said Gregory will submit a cleaner version soon.
> 
> Yeah, I look forward to it :)

Hi Masami,

You can try this patch if you want.
I don't have the hardware today so I didn't test it.
If you (and also Florian and Ezequiel) can test it and if it fixed
the bug, then I will be able send a proper email for it,

Thanks,
-- 
Gregory Clement, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com


[-- Attachment #2: 0001-net-mvneta-convert-to-local-interrupt.patch --]
[-- Type: text/x-diff, Size: 3140 bytes --]

>From a82800cbd4f2ff34a4a03c8caa688149b8770ab7 Mon Sep 17 00:00:00 2001
From: Gregory CLEMENT <gregory.clement@free-electrons.com>
Date: Tue, 19 Mar 2013 15:11:48 +0100
Subject: [PATCH] net: mvneta: convert to local interrupt

Since commit 3a6f08a37 "arm: mvebu: Add support for local interrupt",
the mvneta interrupt is now managed as a local interrupt. That means
that the driver have to use the request_percpu_irq() function instead
of request_irq().

Signed-off-by: Gregory CLEMENT <gregory.clement@free-electrons.com>
---
 drivers/net/ethernet/marvell/mvneta.c |   26 ++++++++++++++++++++++----
 1 file changed, 22 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c
index cd345b8..ad64a50 100644
--- a/drivers/net/ethernet/marvell/mvneta.c
+++ b/drivers/net/ethernet/marvell/mvneta.c
@@ -256,6 +256,8 @@ struct mvneta_port {
 	unsigned int link;
 	unsigned int duplex;
 	unsigned int speed;
+
+	struct mvneta_port __percpu **percpu_pp;
 };
 
 /* The mvneta_tx_desc and mvneta_rx_desc structures describe the
@@ -1799,7 +1801,7 @@ static void mvneta_set_rx_mode(struct net_device *dev)
 /* Interrupt handling - the callback for request_irq() */
 static irqreturn_t mvneta_isr(int irq, void *dev_id)
 {
-	struct mvneta_port *pp = (struct mvneta_port *)dev_id;
+	struct mvneta_port *pp = *(struct mvneta_port **)dev_id;
 
 	/* Mask all interrupts */
 	mvreg_write(pp, MVNETA_INTR_NEW_MASK, 0);
@@ -2371,8 +2373,19 @@ static void mvneta_mdio_remove(struct mvneta_port *pp)
 static int mvneta_open(struct net_device *dev)
 {
 	struct mvneta_port *pp = netdev_priv(dev);
+
 	int ret;
 
+	/* As the mvneta interrupts are locals, we need to create a
+	 * percpu variable
+	 */
+	pp->percpu_pp = alloc_percpu(struct mvneta_port *);
+	if (!pp) {
+		ret = -ENOMEM;
+		goto err_percpu_alloc;
+	}
+	*__this_cpu_ptr(pp->percpu_pp) = pp;
+
 	mvneta_mac_addr_set(pp, dev->dev_addr, rxq_def);
 
 	pp->pkt_size = MVNETA_RX_PKT_SIZE(pp->dev->mtu);
@@ -2385,13 +2398,15 @@ static int mvneta_open(struct net_device *dev)
 	if (ret)
 		goto err_cleanup_rxqs;
 
+
 	/* Connect to port interrupt line */
-	ret = request_irq(pp->dev->irq, mvneta_isr, 0,
-			  MVNETA_DRIVER_NAME, pp);
+	ret = request_percpu_irq(pp->dev->irq, mvneta_isr,
+				MVNETA_DRIVER_NAME, pp->percpu_pp);
 	if (ret) {
 		netdev_err(pp->dev, "cannot request irq %d\n", pp->dev->irq);
 		goto err_cleanup_txqs;
 	}
+	enable_percpu_irq(pp->dev->irq, 0);
 
 	/* In default link is down */
 	netif_carrier_off(pp->dev);
@@ -2407,11 +2422,13 @@ static int mvneta_open(struct net_device *dev)
 	return 0;
 
 err_free_irq:
+	free_percpu(pp->percpu_pp);
 	free_irq(pp->dev->irq, pp);
 err_cleanup_txqs:
 	mvneta_cleanup_txqs(pp);
 err_cleanup_rxqs:
 	mvneta_cleanup_rxqs(pp);
+err_percpu_alloc:
 	return ret;
 }
 
@@ -2422,7 +2439,8 @@ static int mvneta_stop(struct net_device *dev)
 
 	mvneta_stop_dev(pp);
 	mvneta_mdio_remove(pp);
-	free_irq(dev->irq, pp);
+	free_percpu(pp->percpu_pp);
+	free_percpu_irq(dev->irq, pp);
 	mvneta_cleanup_rxqs(pp);
 	mvneta_cleanup_txqs(pp);
 	del_timer(&pp->tx_done_timer);
-- 
1.7.9.5

Re: [BUG][mvebu] mvneta: cannot request irq 25 on openblocks-ax3

[Florian Fainelli]

On Tuesday 19 March 2013 17:43:49 Gregory CLEMENT wrote:
> 
> Hi Masami,
> 
> You can try this patch if you want.
> I don't have the hardware today so I didn't test it.
> If you (and also Florian and Ezequiel) can test it and if it fixed
> the bug, then I will be able send a proper email for it,

I just tested your patch on RD-A370-A1 and DB-MV784MP-GP and it works fine, 
thanks! Feel free to add my Tested-by: Florian Fainelli <florian@openwrt.org> 
to your next submission.

Thanks!
-- 
Florian

Re: [BUG][mvebu] mvneta: cannot request irq 25 on openblocks-ax3

[Gregory CLEMENT]

On 03/19/2013 08:48 PM, Florian Fainelli wrote:
> On Tuesday 19 March 2013 17:43:49 Gregory CLEMENT wrote:
>>
>> Hi Masami,
>>
>> You can try this patch if you want.
>> I don't have the hardware today so I didn't test it.
>> If you (and also Florian and Ezequiel) can test it and if it fixed
>> the bug, then I will be able send a proper email for it,
> 
> I just tested your patch on RD-A370-A1 and DB-MV784MP-GP and it works fine, 
> thanks! Feel free to add my Tested-by: Florian Fainelli <florian@openwrt.org> 
> to your next submission.
> 
> Thanks!
> 
Thanks for having tested it, but with this patch there is still an issue when
we try to release the network interface. In this case the the interrupt line
is not properly freed.
Unfortunately the state of the percpu_irq API prevents the driver to use it.

Indeed the interrupts have to be freed when the .stop() function is
called. As the free_percpu_irq() function don't disable the interrupt
line, we have to do it on each CPU before calling this. The function
disable_percpu_irq() only disable the percpu on the current CPU and
there is no function which allows to disable a percpu irq on a given
CPU. Waiting for the extension of the percpu_irq API, this fix allows
to use again the mvneta driver.

I have just wrote and tested a new patch to fix this issue by just go back
on the old way we managed the irq for the mvenat as global irq and not
as local ones.

I am going to send the fix on the LAKML in a few minutes.

Thanks,

Gregory

Re: Re: [BUG][mvebu] mvneta: cannot request irq 25 on openblocks-ax3

[Masami Hiramatsu]

(2013/03/20 1:43), Gregory CLEMENT wrote:
> Hi Masami,
> 
> You can try this patch if you want.
> I don't have the hardware today so I didn't test it.
> If you (and also Florian and Ezequiel) can test it and if it fixed
> the bug, then I will be able send a proper email for it,

Hi Gregory,

Thank you for giving me the latest patch.
I've tested it on OpenBlocks-AX4 and that fixed the problem!

Tested-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>

Thanks,
-- 
Masami HIRAMATSU
IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com