[Intel-wired-lan] stopping 10GC(x540T2) port does not down the link

[Intel-wired-lan] stopping 10GC(x540T2) port does not down the link

[Wen Chiu]

Hi DKDK maintainers,

We are seeing an issue where if a port is shutdown in linux using "ip link set <dev> down" command on baremetal. Then use "ip link show" to show the port which got shutdown which is in DOWN state as expected. But, the port on the other end still shows UP state which should be DOWN. Following is the excerpt from DKDK notes which indicates the similar issue with 40G. But, we are seeing the problem with 10GC(x540T2) adapter. Could you confirm this? It would be highly appreciated if you could shed some lights regarding this issue, fixes or workaround.

Attached has the screen traces to show how to create and show the issue using "ip link".  Eth8 and Eth9 are directly connected. Eth8 is the interface that got disabled/down via admin. Eth9 is the other end which should be down but is still UP and hence the issue. Attached also has output for "lspci -vv" in that device 42:00.0 is eth8 and 42:00.1 is eth9.

PS. ixgbe_dev_stop() is called but looks phy is not turn off is it is copper. I did see laser tx is turned off if it is fiber.

Thanks,
Wen Chiu

6.24. Stopping the port does not down the link on Intel(r) 40G Ethernet controller
Description:
On Intel(r) 40G Ethernet Controller stopping the port does not really down the port link.
Implication:
The port link will be still up after stopping the port.
Resolution/Workaround:
None
Affected Environment/Platform:
All.
Driver/Module:
Poll Mode Driver (PMD).


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20150910/704bf06b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: output
Type: application/octet-stream
Size: 56948 bytes
Desc: output
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20150910/704bf06b/attachment-0001.obj>

[Intel-wired-lan] stopping 10GC(x540T2) port does not down the link

[Rustad, Mark D]

Wen Chiu,

> On Sep 10, 2015, at 3:19 PM, Wen Chiu <Wen.Chiu@brocade.com> wrote:
> 
> Hi DKDK maintainers,

I assume that you are referring to DPDK. I am not aware of any DPDK maintainers on this mailing list, though there could be.

> We are seeing an issue where if a port is shutdown in linux using ?ip link set <dev> down? command on baremetal. Then use ?ip link show? to show the port which got shutdown which is in DOWN state as expected. But, the port on the other end still shows UP state which should be DOWN. Following is the excerpt from DKDK notes which indicates the similar issue with 40G. But, we are seeing the problem with 10GC(x540T2) adapter. Could you confirm this? It would be highly appreciated if you could shed some lights regarding this issue, fixes or workaround.

Although I do not know the internals of DPDK, something general to look at is whether Wake-On-LAN is enabled. Ethtool can show you the state. If it reports anything other than d, then it is enabled and that will prevent the phy from powering down. You should be able to change the WOL setting using the command:
	ethtool -s eth8 wol d

You would want to do that for each port that you want to be able to power down administratively. This is all kind of assuming that the DPDK driver supports these operations, which I do not know, but WOL is the first thing that I think of to check.

--
Mark Rustad, Networking Division, Intel Corporation

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20150911/24d66115/attachment-0001.asc>

[Intel-wired-lan] stopping 10GC(x540T2) port does not down the link

[Wen Chiu]

Hi Mark,

Thanks for your prompted response. First of all, let me clarify the issue. The issue was first seen with our vRouter running DPDK. To isolate it, we setup a Ubuntu 13.4 based baremetal box to try the same and the issue was also seen. So, it is now genuine linux (driver?) issue.

I did try what you suggest and used ethtool to see the eth8 status below. Both ports are already set to "d".
wen at wen-PowerEdge-R620:~$ sudo ethtool eth8 |grep Wake
        Supports Wake-on: d
        Wake-on: d
wen at wen-PowerEdge-R620:~$ sudo ethtool eth9 |grep Wake
        Supports Wake-on: d
        Wake-on: d

After I administratively disable eth8, eth9 (the other end of the connected port) still shows UP. Any other ideas? Please let me know.

Thanks,
Wen Chiu

-----Original Message-----
From: Rustad, Mark D [mailto:mark.d.rustad at intel.com] 
Sent: Friday, September 11, 2015 9:17 AM
To: Wen Chiu
Cc: Kirsher, Jeffrey T; Brandeburg, Jesse; Nelson, Shannon; Wyborny, Carolyn; Skidmore, Donald C; Vick, Matthew; Ronciak, John; Williams, Mitch A; intel-wired-lan at lists.osuosl.org; Stephen Hemminger
Subject: Re: [Intel-wired-lan] stopping 10GC(x540T2) port does not down the link

Wen Chiu,

> On Sep 10, 2015, at 3:19 PM, Wen Chiu <Wen.Chiu@brocade.com> wrote:
> 
> Hi DKDK maintainers,

I assume that you are referring to DPDK. I am not aware of any DPDK maintainers on this mailing list, though there could be.

> We are seeing an issue where if a port is shutdown in linux using ?ip link set <dev> down? command on baremetal. Then use ?ip link show? to show the port which got shutdown which is in DOWN state as expected. But, the port on the other end still shows UP state which should be DOWN. Following is the excerpt from DKDK notes which indicates the similar issue with 40G. But, we are seeing the problem with 10GC(x540T2) adapter. Could you confirm this? It would be highly appreciated if you could shed some lights regarding this issue, fixes or workaround.

Although I do not know the internals of DPDK, something general to look at is whether Wake-On-LAN is enabled. Ethtool can show you the state. If it reports anything other than d, then it is enabled and that will prevent the phy from powering down. You should be able to change the WOL setting using the command:
	ethtool -s eth8 wol d

You would want to do that for each port that you want to be able to power down administratively. This is all kind of assuming that the DPDK driver supports these operations, which I do not know, but WOL is the first thing that I think of to check.

--
Mark Rustad, Networking Division, Intel Corporation

[Intel-wired-lan] stopping 10GC(x540T2) port does not down the link

[Rustad, Mark D]

Wen Chiu,

> On Sep 11, 2015, at 5:28 PM, Wen Chiu <Wen.Chiu@brocade.com> wrote:
> 
> Thanks for your prompted response. First of all, let me clarify the issue. The issue was first seen with our vRouter running DPDK. To isolate it, we setup a Ubuntu 13.4 based baremetal box to try the same and the issue was also seen. So, it is now genuine linux (driver?) issue.

Ah. Ok. Can you tell me what the kernel version is on Ubuntu 13.4? I just checked and found that code to control the phy power did not get into the kernel driver until kernel version 4.2.

> I did try what you suggest and used ethtool to see the eth8 status below. Both ports are already set to "d".
> wen at wen-PowerEdge-R620:~$ sudo ethtool eth8 |grep Wake
>        Supports Wake-on: d
>        Wake-on: d
> wen at wen-PowerEdge-R620:~$ sudo ethtool eth9 |grep Wake
>        Supports Wake-on: d
>        Wake-on: d

Yup. WOL is disabled.

> After I administratively disable eth8, eth9 (the other end of the connected port) still shows UP. Any other ideas? Please let me know.

As I indicated above, the kernel driver did not get code to control phy power until version 4.2. The out-of-tree Source Forge driver got that code earlier, so it is in the current version posted there. You could try installing that driver and see if it works for you if that is easier than getting to the 4.2 kernel.

--
Mark Rustad, Networking Division, Intel Corporation

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20150912/251592c0/attachment.asc>

[Intel-wired-lan] stopping 10GC(x540T2) port does not down the link

[Stephen Hemminger]

On Sat, 12 Sep 2015 00:54:48 +0000
"Rustad, Mark D" <mark.d.rustad@intel.com> wrote:

> Wen Chiu,
> 
> > On Sep 11, 2015, at 5:28 PM, Wen Chiu <Wen.Chiu@brocade.com> wrote:
> > 
> > Thanks for your prompted response. First of all, let me clarify the issue. The issue was first seen with our vRouter running DPDK. To isolate it, we setup a Ubuntu 13.4 based baremetal box to try the same and the issue was also seen. So, it is now genuine linux (driver?) issue.  
> 
> Ah. Ok. Can you tell me what the kernel version is on Ubuntu 13.4? I just checked and found that code to control the phy power did not get into the kernel driver until kernel version 4.2.
> 
> > I did try what you suggest and used ethtool to see the eth8 status below. Both ports are already set to "d".
> > wen at wen-PowerEdge-R620:~$ sudo ethtool eth8 |grep Wake
> >        Supports Wake-on: d
> >        Wake-on: d
> > wen at wen-PowerEdge-R620:~$ sudo ethtool eth9 |grep Wake
> >        Supports Wake-on: d
> >        Wake-on: d  
> 
> Yup. WOL is disabled.
> 
> > After I administratively disable eth8, eth9 (the other end of the connected port) still shows UP. Any other ideas? Please let me know.  
> 
> As I indicated above, the kernel driver did not get code to control phy power until version 4.2. The out-of-tree Source Forge driver got that code earlier, so it is in the current version posted there. You could try installing that driver and see if it works for you if that is easier than getting to the 4.2 kernel.

Please identify the commit, and tell David Miller (and netdev) to merge that into the stable tree.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20150911/e99eef3e/attachment-0001.asc>