Sporadic errors while initializing NICs in example applications, dpdk-1.5.0r1

Sporadic errors while initializing NICs in example applications, dpdk-1.5.0r1

[Dmitry Vyal]

Hi, I'm experiencing weird problems with running dpdk examples on my 
server running ubuntu-12.04. Application either manages to use ethernet 
ports or doesn't. For example, this is results of two identical 
sequental runs of l2fwd:

******************************************************************************
dev@tiny-one:~/dpdk-1.5.0r1/examples/l2fwd$ s -E ./build/l2fwd -c 0x3 -n 2
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 0 on socket 0
EAL: Detected lcore 5 as core 1 on socket 0
EAL: Detected lcore 6 as core 2 on socket 0
EAL: Detected lcore 7 as core 3 on socket 0
EAL: Skip lcore 8 (not detected)
<LINES DELETED>
EAL: Setting up memory...
EAL: Ask a virtual area of 0x4194304 bytes
EAL: Virtual area found at 0x7f6b82400000 (size = 0x400000)
EAL: Ask a virtual area of 0x2097152 bytes
EAL: Virtual area found at 0x7f6b82000000 (size = 0x200000)
EAL: Ask a virtual area of 0x2097152 bytes
EAL: Virtual area found at 0x7f6b81c00000 (size = 0x200000)
EAL: Ask a virtual area of 0x1056964608 bytes
EAL: Virtual area found at 0x7f6b42a00000 (size = 0x3f000000)
EAL: Ask a virtual area of 0x2097152 bytes
EAL: Virtual area found at 0x7f6b42600000 (size = 0x200000)
EAL: Ask a virtual area of 0x2097152 bytes
EAL: Virtual area found at 0x7f6b42200000 (size = 0x200000)
EAL: Ask a virtual area of 0x2097152 bytes
EAL: Virtual area found at 0x7f6b41e00000 (size = 0x200000)
EAL: Ask a virtual area of 0x2097152 bytes
EAL: Virtual area found at 0x7f6b41a00000 (size = 0x200000)
EAL: Requesting 512 pages of size 2MB from socket 0
EAL: TSC frequency is ~1600000 KHz
EAL: Master core 0 is ready (tid=836da800)
EAL: Core 1 is ready (tid=40ff8700)
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10fb rte_ixgbe_pmd
EAL:   PCI memory mapped at 0x7f6b83687000
EAL:   PCI memory mapped at 0x7f6b83683000
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL:   probe driver: 8086:10fb rte_ixgbe_pmd
EAL:   PCI memory mapped at 0x7f6b83663000
EAL:   PCI memory mapped at 0x7f6b8365f000
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10d3 rte_em_pmd
EAL:   0000:03:00.0 not managed by UIO driver, skipping
EAL: PCI device 0000:04:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10d3 rte_em_pmd
EAL:   0000:04:00.0 not managed by UIO driver, skipping
EAL: PCI device 0000:05:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10d3 rte_em_pmd
EAL:   0000:05:00.0 not managed by UIO driver, skipping
EAL: PCI device 0000:06:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10d3 rte_em_pmd
EAL:   0000:06:00.0 not managed by UIO driver, skipping
EAL: PCI device 0000:07:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10d3 rte_em_pmd
EAL:   0000:07:00.0 not managed by UIO driver, skipping
EAL: PCI device 0000:08:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10d3 rte_em_pmd
EAL:   0000:08:00.0 not managed by UIO driver, skipping
EAL: PCI device 0000:09:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10d3 rte_em_pmd
EAL:   0000:09:00.0 not managed by UIO driver, skipping
EAL: Error - exiting with code: 1
   Cause: No Ethernet ports - bye

************************************************************************************

dev@econat-tiny-one:~/dpdk-1.5.0r1/examples/l2fwd$ s -E ./build/l2fwd -c 
0x3 -n 2
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 0 on socket 0
EAL: Detected lcore 5 as core 1 on socket 0
EAL: Detected lcore 6 as core 2 on socket 0
EAL: Detected lcore 7 as core 3 on socket 0
EAL: Skip lcore 8 (not detected)
<LINES DELETED>
EAL: Setting up memory...
EAL: Ask a virtual area of 0x4194304 bytes
EAL: Virtual area found at 0x7f538a600000 (size = 0x400000)
EAL: Ask a virtual area of 0x2097152 bytes
EAL: Virtual area found at 0x7f538a200000 (size = 0x200000)
EAL: Ask a virtual area of 0x2097152 bytes
EAL: Virtual area found at 0x7f5389e00000 (size = 0x200000)
EAL: Ask a virtual area of 0x1056964608 bytes
EAL: Virtual area found at 0x7f534ac00000 (size = 0x3f000000)
EAL: Ask a virtual area of 0x2097152 bytes
EAL: Virtual area found at 0x7f534a800000 (size = 0x200000)
EAL: Ask a virtual area of 0x2097152 bytes
EAL: Virtual area found at 0x7f534a400000 (size = 0x200000)
EAL: Ask a virtual area of 0x2097152 bytes
EAL: Virtual area found at 0x7f534a000000 (size = 0x200000)
EAL: Ask a virtual area of 0x2097152 bytes
EAL: Virtual area found at 0x7f5349c00000 (size = 0x200000)
EAL: Requesting 512 pages of size 2MB from socket 0
EAL: TSC frequency is ~3301000 KHz
EAL: Master core 0 is ready (tid=8b98d800)
EAL: Core 1 is ready (tid=491f8700)
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10fb rte_ixgbe_pmd
EAL:   PCI memory mapped at 0x7f538b93a000
EAL:   PCI memory mapped at 0x7f538b936000
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL:   probe driver: 8086:10fb rte_ixgbe_pmd
EAL:   PCI memory mapped at 0x7f538b916000
EAL:   PCI memory mapped at 0x7f538b912000
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10d3 rte_em_pmd
EAL:   0000:03:00.0 not managed by UIO driver, skipping
EAL: PCI device 0000:04:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10d3 rte_em_pmd
EAL:   0000:04:00.0 not managed by UIO driver, skipping
EAL: PCI device 0000:05:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10d3 rte_em_pmd
EAL:   0000:05:00.0 not managed by UIO driver, skipping
EAL: PCI device 0000:06:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10d3 rte_em_pmd
EAL:   0000:06:00.0 not managed by UIO driver, skipping
EAL: PCI device 0000:07:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10d3 rte_em_pmd
EAL:   0000:07:00.0 not managed by UIO driver, skipping
EAL: PCI device 0000:08:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10d3 rte_em_pmd
EAL:   0000:08:00.0 not managed by UIO driver, skipping
EAL: PCI device 0000:09:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10d3 rte_em_pmd
EAL:   0000:09:00.0 not managed by UIO driver, skipping
Skipping disabled port 0
Skipping disabled port 1
EAL: Error - exiting with code: 1
   Cause: All available ports are disabled. Please set portmask.

******************************************************************

You see, final messages differ. Almost all the time I get "Cause: No 
Ethernet ports - bye" but sometimes (with probability of 1/10 or so) NIC 
is initialized successfully.

Some info about the system:

dev@tiny-one:~/dpdk-1.5.0r1/examples/l2fwd$ cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 58
model name      : Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz
stepping        : 9
microcode       : 0x10
cpu MHz         : 1600.000
cache size      : 8192 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe 
syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl 
xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx 
smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt 
tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb 
xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase 
smep erms
bogomips        : 6584.75
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:


dev@tiny-one:~/dpdk-1.5.0r1/examples/l2fwd$ lspci
00:00.0 Host bridge: Intel Corporation Ivy Bridge DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Ivy Bridge PCI Express Root Port 
(rev 09)
00:01.1 PCI bridge: Intel Corporation Ivy Bridge PCI Express Root Port 
(rev 09)
00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series 
Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset 
Family USB Enhanced Host Controller #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset 
Family PCI Express Root Port 1 (rev b5)
00:1c.1 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset 
Family PCI Express Root Port 2 (rev b5)
00:1c.2 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset 
Family PCI Express Root Port 3 (rev b5)
00:1c.3 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset 
Family PCI Express Root Port 4 (rev b5)
00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset 
Family PCI Express Root Port 5 (rev b5)
00:1c.5 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset 
Family PCI Express Root Port 6 (rev b5)
00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset 
Family PCI Express Root Port 7 (rev b5)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset 
Family USB Enhanced Host Controller #1 (rev 05)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5)
00:1f.0 ISA bridge: Intel Corporation C206 Chipset Family LPC Controller 
(rev 05)
00:1f.2 IDE interface: Intel Corporation 6 Series/C200 Series Chipset 
Family 4 port SATA IDE Controller (rev 05)
00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family 
SMBus Controller (rev 05)
00:1f.5 IDE interface: Intel Corporation 6 Series/C200 Series Chipset 
Family 2 port SATA IDE Controller (rev 05)
02:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
02:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
03:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network 
Connection
04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network 
Connection
05:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network 
Connection
06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network 
Connection
07:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network 
Connection
08:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network 
Connection
09:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network 
Connection


dev@tiny-one:~/dpdk-1.5.0r1/examples/l2fwd$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 12.04.2 LTS
Release:        12.04
Codename:       precise

Any ideas how to investigate this?

Regards,
Dmitry

Re: Sporadic errors while initializing NICs in example applications, dpdk-1.5.0r1

[Thomas Monjalon]

Hello,

22/11/2013 13:29, Dmitry Vyal :
> EAL: PCI device 0000:02:00.0 on NUMA socket -1
> EAL:   probe driver: 8086:10fb rte_ixgbe_pmd
> EAL:   PCI memory mapped at 0x7f6b83687000
> EAL:   PCI memory mapped at 0x7f6b83683000
> EAL: PCI device 0000:02:00.1 on NUMA socket -1
> EAL:   probe driver: 8086:10fb rte_ixgbe_pmd
> EAL:   PCI memory mapped at 0x7f6b83663000
> EAL:   PCI memory mapped at 0x7f6b8365f000
[...]
> EAL: Error - exiting with code: 1
>    Cause: No Ethernet ports - bye
> 
> Any ideas how to investigate this?

Could you try this patch in order to see the root cause of your issue ?

--- a/lib/librte_pmd_ixgbe/ixgbe_logs.h
+++ b/lib/librte_pmd_ixgbe/ixgbe_logs.h
@@ -34,41 +34,44 @@
 #ifndef _IXGBE_LOGS_H_
 #define _IXGBE_LOGS_H_
 
+#define PMD_LOG(level, fmt, args...) \
+	RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ##args)
+
 #ifdef RTE_LIBRTE_IXGBE_DEBUG_INIT
-#define PMD_INIT_LOG(level, fmt, args...) \
-	RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args)
+#define PMD_INIT_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)
 #define PMD_INIT_FUNC_TRACE() PMD_INIT_LOG(DEBUG, " >>")
 #else
-#define PMD_INIT_LOG(level, fmt, args...) do { } while(0)
+#define PMD_INIT_LOG(level, fmt, args...) \
+	(void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)
 #define PMD_INIT_FUNC_TRACE() do { } while(0)
 #endif
 
 #ifdef RTE_LIBRTE_IXGBE_DEBUG_RX
-#define PMD_RX_LOG(level, fmt, args...) \
-	RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args)
+#define PMD_RX_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)
 #else
-#define PMD_RX_LOG(level, fmt, args...) do { } while(0)
+#define PMD_RX_LOG(level, fmt, args...) \
+	(void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)
 #endif
 
 #ifdef RTE_LIBRTE_IXGBE_DEBUG_TX
-#define PMD_TX_LOG(level, fmt, args...) \
-	RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args)
+#define PMD_TX_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)
 #else
-#define PMD_TX_LOG(level, fmt, args...) do { } while(0)
+#define PMD_TX_LOG(level, fmt, args...) \
+	(void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)
 #endif
 
 #ifdef RTE_LIBRTE_IXGBE_DEBUG_TX_FREE
-#define PMD_TX_FREE_LOG(level, fmt, args...) \
-	RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args)
+#define PMD_TX_FREE_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)
 #else
-#define PMD_TX_FREE_LOG(level, fmt, args...) do { } while(0)
+#define PMD_TX_FREE_LOG(level, fmt, args...) \
+	(void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)
 #endif
 
 #ifdef RTE_LIBRTE_IXGBE_DEBUG_DRIVER
-#define PMD_DRV_LOG(level, fmt, args...) \
-	RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args)
+#define PMD_DRV_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)
 #else
-#define PMD_DRV_LOG(level, fmt, args...) do { } while(0)
+#define PMD_DRV_LOG(level, fmt, args...) \
+	(void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)
 #endif
 
 #endif /* _IXGBE_LOGS_H_ */

-- 
Thomas

Re: Sporadic errors while initializing NICs in example applications, dpdk-1.5.0r1

[Dmitry Vyal]

Looks like I finally found the reason. After applying this patch I can 
no longer reproduce the error.

diff --git a/lib/librte_pmd_ixgbe/ixgbe/ixgbe_82599.c 
b/lib/librte_pmd_ixgbe/ixgbe/ixgbe_82599.c
index db07789..5f825fa 100644
--- a/lib/librte_pmd_ixgbe/ixgbe/ixgbe_82599.c
+++ b/lib/librte_pmd_ixgbe/ixgbe/ixgbe_82599.c
@@ -2347,7 +2347,7 @@ s32 ixgbe_reset_pipeline_82599(struct ixgbe_hw *hw)
         /* Write AUTOC register with toggled LMS[2] bit and Restart_AN */
         IXGBE_WRITE_REG(hw, IXGBE_AUTOC, autoc_reg ^ 
IXGBE_AUTOC_LMS_1G_AN);
         /* Wait for AN to leave state 0 */
-       for (i = 0; i < 10; i++) {
+       for (i = 0; i < 100; i++) {
                 msec_delay(4);
                 anlp1_reg = IXGBE_READ_REG(hw, IXGBE_ANLP1);
                 if (anlp1_reg & IXGBE_ANLP1_AN_STATE_MASK)

On 11/22/2013 04:48 PM, Thomas Monjalon wrote:
> Hello,
>
> 22/11/2013 13:29, Dmitry Vyal :
>> EAL: PCI device 0000:02:00.0 on NUMA socket -1
>> EAL:   probe driver: 8086:10fb rte_ixgbe_pmd
>> EAL:   PCI memory mapped at 0x7f6b83687000
>> EAL:   PCI memory mapped at 0x7f6b83683000
>> EAL: PCI device 0000:02:00.1 on NUMA socket -1
>> EAL:   probe driver: 8086:10fb rte_ixgbe_pmd
>> EAL:   PCI memory mapped at 0x7f6b83663000
>> EAL:   PCI memory mapped at 0x7f6b8365f000
> [...]
>> EAL: Error - exiting with code: 1
>>     Cause: No Ethernet ports - bye
>>
>> Any ideas how to investigate this?
> Could you try this patch in order to see the root cause of your issue ?
>
> --- a/lib/librte_pmd_ixgbe/ixgbe_logs.h
> +++ b/lib/librte_pmd_ixgbe/ixgbe_logs.h
> @@ -34,41 +34,44 @@
>   #ifndef _IXGBE_LOGS_H_
>   #define _IXGBE_LOGS_H_
>   
> +#define PMD_LOG(level, fmt, args...) \
> +	RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ##args)
> +
>   #ifdef RTE_LIBRTE_IXGBE_DEBUG_INIT
> -#define PMD_INIT_LOG(level, fmt, args...) \
> -	RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args)
> +#define PMD_INIT_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)
>   #define PMD_INIT_FUNC_TRACE() PMD_INIT_LOG(DEBUG, " >>")
>   #else
> -#define PMD_INIT_LOG(level, fmt, args...) do { } while(0)
> +#define PMD_INIT_LOG(level, fmt, args...) \
> +	(void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)
>   #define PMD_INIT_FUNC_TRACE() do { } while(0)
>   #endif
>   
>   #ifdef RTE_LIBRTE_IXGBE_DEBUG_RX
> -#define PMD_RX_LOG(level, fmt, args...) \
> -	RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args)
> +#define PMD_RX_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)
>   #else
> -#define PMD_RX_LOG(level, fmt, args...) do { } while(0)
> +#define PMD_RX_LOG(level, fmt, args...) \
> +	(void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)
>   #endif
>   
>   #ifdef RTE_LIBRTE_IXGBE_DEBUG_TX
> -#define PMD_TX_LOG(level, fmt, args...) \
> -	RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args)
> +#define PMD_TX_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)
>   #else
> -#define PMD_TX_LOG(level, fmt, args...) do { } while(0)
> +#define PMD_TX_LOG(level, fmt, args...) \
> +	(void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)
>   #endif
>   
>   #ifdef RTE_LIBRTE_IXGBE_DEBUG_TX_FREE
> -#define PMD_TX_FREE_LOG(level, fmt, args...) \
> -	RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args)
> +#define PMD_TX_FREE_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)
>   #else
> -#define PMD_TX_FREE_LOG(level, fmt, args...) do { } while(0)
> +#define PMD_TX_FREE_LOG(level, fmt, args...) \
> +	(void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)
>   #endif
>   
>   #ifdef RTE_LIBRTE_IXGBE_DEBUG_DRIVER
> -#define PMD_DRV_LOG(level, fmt, args...) \
> -	RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args)
> +#define PMD_DRV_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)
>   #else
> -#define PMD_DRV_LOG(level, fmt, args...) do { } while(0)
> +#define PMD_DRV_LOG(level, fmt, args...) \
> +	(void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)
>   #endif
>   
>   #endif /* _IXGBE_LOGS_H_ */
>

Re: Sporadic errors while initializing NICs in example applications, dpdk-1.5.0r1

[jigsaw]

I had exactly same problem and fixed it with same patch since release
1.4. But somehow it stopped to appear even without the patch. I have
no idea what I have done since both the kernel and the driver have
been updated during these days.

On Wed, Nov 27, 2013 at 4:06 PM, Dmitry Vyal <dmitryvyal-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Looks like I finally found the reason. After applying this patch I can no
> longer reproduce the error.
>
> diff --git a/lib/librte_pmd_ixgbe/ixgbe/ixgbe_82599.c
> b/lib/librte_pmd_ixgbe/ixgbe/ixgbe_82599.c
> index db07789..5f825fa 100644
> --- a/lib/librte_pmd_ixgbe/ixgbe/ixgbe_82599.c
> +++ b/lib/librte_pmd_ixgbe/ixgbe/ixgbe_82599.c
> @@ -2347,7 +2347,7 @@ s32 ixgbe_reset_pipeline_82599(struct ixgbe_hw *hw)
>         /* Write AUTOC register with toggled LMS[2] bit and Restart_AN */
>         IXGBE_WRITE_REG(hw, IXGBE_AUTOC, autoc_reg ^ IXGBE_AUTOC_LMS_1G_AN);
>         /* Wait for AN to leave state 0 */
> -       for (i = 0; i < 10; i++) {
> +       for (i = 0; i < 100; i++) {
>                 msec_delay(4);
>                 anlp1_reg = IXGBE_READ_REG(hw, IXGBE_ANLP1);
>                 if (anlp1_reg & IXGBE_ANLP1_AN_STATE_MASK)
>
> On 11/22/2013 04:48 PM, Thomas Monjalon wrote:
>>
>> Hello,
>>
>> 22/11/2013 13:29, Dmitry Vyal :
>>>
>>> EAL: PCI device 0000:02:00.0 on NUMA socket -1
>>> EAL:   probe driver: 8086:10fb rte_ixgbe_pmd
>>> EAL:   PCI memory mapped at 0x7f6b83687000
>>> EAL:   PCI memory mapped at 0x7f6b83683000
>>> EAL: PCI device 0000:02:00.1 on NUMA socket -1
>>> EAL:   probe driver: 8086:10fb rte_ixgbe_pmd
>>> EAL:   PCI memory mapped at 0x7f6b83663000
>>> EAL:   PCI memory mapped at 0x7f6b8365f000
>>
>> [...]
>>>
>>> EAL: Error - exiting with code: 1
>>>     Cause: No Ethernet ports - bye
>>>
>>> Any ideas how to investigate this?
>>
>> Could you try this patch in order to see the root cause of your issue ?
>>
>> --- a/lib/librte_pmd_ixgbe/ixgbe_logs.h
>> +++ b/lib/librte_pmd_ixgbe/ixgbe_logs.h
>> @@ -34,41 +34,44 @@
>>   #ifndef _IXGBE_LOGS_H_
>>   #define _IXGBE_LOGS_H_
>>   +#define PMD_LOG(level, fmt, args...) \
>> +       RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ##args)
>> +
>>   #ifdef RTE_LIBRTE_IXGBE_DEBUG_INIT
>> -#define PMD_INIT_LOG(level, fmt, args...) \
>> -       RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args)
>> +#define PMD_INIT_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)
>>   #define PMD_INIT_FUNC_TRACE() PMD_INIT_LOG(DEBUG, " >>")
>>   #else
>> -#define PMD_INIT_LOG(level, fmt, args...) do { } while(0)
>> +#define PMD_INIT_LOG(level, fmt, args...) \
>> +       (void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt,
>> ##args) : 0)
>>   #define PMD_INIT_FUNC_TRACE() do { } while(0)
>>   #endif
>>     #ifdef RTE_LIBRTE_IXGBE_DEBUG_RX
>> -#define PMD_RX_LOG(level, fmt, args...) \
>> -       RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args)
>> +#define PMD_RX_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)
>>   #else
>> -#define PMD_RX_LOG(level, fmt, args...) do { } while(0)
>> +#define PMD_RX_LOG(level, fmt, args...) \
>> +       (void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt,
>> ##args) : 0)
>>   #endif
>>     #ifdef RTE_LIBRTE_IXGBE_DEBUG_TX
>> -#define PMD_TX_LOG(level, fmt, args...) \
>> -       RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args)
>> +#define PMD_TX_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)
>>   #else
>> -#define PMD_TX_LOG(level, fmt, args...) do { } while(0)
>> +#define PMD_TX_LOG(level, fmt, args...) \
>> +       (void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt,
>> ##args) : 0)
>>   #endif
>>     #ifdef RTE_LIBRTE_IXGBE_DEBUG_TX_FREE
>> -#define PMD_TX_FREE_LOG(level, fmt, args...) \
>> -       RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args)
>> +#define PMD_TX_FREE_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)
>>   #else
>> -#define PMD_TX_FREE_LOG(level, fmt, args...) do { } while(0)
>> +#define PMD_TX_FREE_LOG(level, fmt, args...) \
>> +       (void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt,
>> ##args) : 0)
>>   #endif
>>     #ifdef RTE_LIBRTE_IXGBE_DEBUG_DRIVER
>> -#define PMD_DRV_LOG(level, fmt, args...) \
>> -       RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args)
>> +#define PMD_DRV_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)
>>   #else
>> -#define PMD_DRV_LOG(level, fmt, args...) do { } while(0)
>> +#define PMD_DRV_LOG(level, fmt, args...) \
>> +       (void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt,
>> ##args) : 0)
>>   #endif
>>     #endif /* _IXGBE_LOGS_H_ */
>>
>

Re: Sporadic errors while initializing NICs in example applications, dpdk-1.5.0r1

[Thomas Monjalon]

27/11/2013 15:06, Dmitry Vyal :
> Looks like I finally found the reason. After applying this patch I can
> no longer reproduce the error.
> 
> --- a/lib/librte_pmd_ixgbe/ixgbe/ixgbe_82599.c
> +++ b/lib/librte_pmd_ixgbe/ixgbe/ixgbe_82599.c
>          /* Wait for AN to leave state 0 */
> -       for (i = 0; i < 10; i++) {
> +       for (i = 0; i < 100; i++) {
>                  msec_delay(4);
>                  anlp1_reg = IXGBE_READ_REG(hw, IXGBE_ANLP1);
>                  if (anlp1_reg & IXGBE_ANLP1_AN_STATE_MASK)

It's probably due to a frequency scaling.
The timer based is initialized when DPDK initialize
and the CPU can change its frequency, breaking next timers.

The fix is to control the CPU frequency.
Please try this, without your patch:
	for g in /sys/devices/system/cpu/*/cpufreq/scaling_governor; do echo performance >$g; done
The right fix for applications (examples and testpmd included)
could be to call rte_power_init(). Patches are welcomed.

Thank you for your analysis
-- 
Thomas

Re: Sporadic errors while initializing NICs in example applications, dpdk-1.5.0r1

[Richardson, Bruce]

> 
> It's probably due to a frequency scaling.
> The timer based is initialized when DPDK initialize and the CPU can change
> its frequency, breaking next timers.
> 
> The fix is to control the CPU frequency.
> Please try this, without your patch:
> 	for g in /sys/devices/system/cpu/*/cpufreq/scaling_governor; do
> echo performance >$g; done The right fix for applications (examples and
> testpmd included) could be to call rte_power_init(). Patches are welcomed.
> 
[BR] Frequency changes should not affect timers for modern Intel CPUs. Please see the " Intel(r) 64 and IA-32 Architectures Software Developer's Manual" Volume 3 (http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-system-programming-manual-325384.pdf) , Section 17.13 for more details on this. 

Re: Sporadic errors while initializing NICs in example applications, dpdk-1.5.0r1

[Dmitry Vyal]

Hmm, that's strange. I don't know how to interpret my observations then. 
I have access to two platforms, one is based on Intel(R) Xeon(R) CPU 
E3-1230 V2 @ 3.30GHz and another on Intel(R) Xeon(R) CPU E3-1270 v3 @ 
3.50GHz. Both running ubuntu-12.04 server. I see repeating errors on NIC 
initialisation phase. The error frequency greatly reduces if I patch 
loop limit as I described earlier or if I call rte_power_init and 
rte_power_freq_max as Thomas suggested.

But the only way to get rid of them completely is to set performance 
governor.

On 11/28/2013 03:01 PM, Richardson, Bruce wrote:
>> It's probably due to a frequency scaling.
>> The timer based is initialized when DPDK initialize and the CPU can change
>> its frequency, breaking next timers.
>>
>> The fix is to control the CPU frequency.
>> Please try this, without your patch:
>> 	for g in /sys/devices/system/cpu/*/cpufreq/scaling_governor; do
>> echo performance >$g; done The right fix for applications (examples and
>> testpmd included) could be to call rte_power_init(). Patches are welcomed.
>>
> [BR] Frequency changes should not affect timers for modern Intel CPUs. Please see the " Intel(r) 64 and IA-32 Architectures Software Developer's Manual" Volume 3 (http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-system-programming-manual-325384.pdf) , Section 17.13 for more details on this.
>

Re: Sporadic errors while initializing NICs in example applications, dpdk-1.5.0r1

[Thomas Monjalon]

29/11/2013 14:53, Dmitry Vyal :
> On 11/28/2013 03:01 PM, Richardson, Bruce wrote:
> >> It's probably due to a frequency scaling.
> >> The timer based is initialized when DPDK initialize and the CPU can
> >> change
> >> its frequency, breaking next timers.
> >> 
> >> The fix is to control the CPU frequency.
> >> 
> >> Please try this, without your patch:
> >> 	for g in /sys/devices/system/cpu/*/cpufreq/scaling_governor; do
> >> 
> >> echo performance >$g; done The right fix for applications (examples and
> >> testpmd included) could be to call rte_power_init(). Patches are
> >> welcomed.
> > 
> > [BR] Frequency changes should not affect timers for modern Intel CPUs.
> > Please see the " Intel(r) 64 and IA-32 Architectures Software Developer's
> > Manual" Volume 3
> > (http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-i
> > a-32-architectures-software-developer-system-programming-manual-325384.pdf
> > ) , Section 17.13 for more details on this.
>
> Hmm, that's strange. I don't know how to interpret my observations then.
> I have access to two platforms, one is based on Intel(R) Xeon(R) CPU
> E3-1230 V2 @ 3.30GHz and another on Intel(R) Xeon(R) CPU E3-1270 v3 @
> 3.50GHz. Both running ubuntu-12.04 server. I see repeating errors on NIC
> initialisation phase. The error frequency greatly reduces if I patch
> loop limit as I described earlier or if I call rte_power_init and
> rte_power_freq_max as Thomas suggested.
>
> But the only way to get rid of them completely is to set performance
> governor.

Please check that your hardware do not support invariant TSC.
It would explain why you need to fix frequency.

I attach a simple code to test CPU feature "Invariant TSC".

-- 
Thomas

Re: Sporadic errors while initializing NICs in example applications, dpdk-1.5.0r1

[Thomas Monjalon]

29/11/2013 13:25, Thomas Monjalon :
> 29/11/2013 14:53, Dmitry Vyal :
> > On 11/28/2013 03:01 PM, Richardson, Bruce wrote:
> > > [BR] Frequency changes should not affect timers for modern Intel CPUs.
> > > Please see the " Intel(r) 64 and IA-32 Architectures Software
> > > Developer's
> > > Manual" Volume 3
> > > (http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-> > > i
> > > a-32-architectures-software-developer-system-programming-manual-325384.p
> > > df
> > > ) , Section 17.13 for more details on this.
> > 
> > Hmm, that's strange. I don't know how to interpret my observations then.
> > I have access to two platforms, one is based on Intel(R) Xeon(R) CPU
> > E3-1230 V2 @ 3.30GHz and another on Intel(R) Xeon(R) CPU E3-1270 v3 @
> > 3.50GHz. Both running ubuntu-12.04 server. I see repeating errors on NIC
> > initialisation phase. The error frequency greatly reduces if I patch
> > loop limit as I described earlier or if I call rte_power_init and
> > rte_power_freq_max as Thomas suggested.
> > 
> > But the only way to get rid of them completely is to set performance
> > governor.
> 
> Please check that your hardware do not support invariant TSC.
> It would explain why you need to fix frequency.
> 
> I attach a simple code to test CPU feature "Invariant TSC".

It seems that the file is stripped on the mailing-list.
Code inlined:

#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <stdint.h>


int main()
{
    uint32_t a = 0x80000000;
    uint32_t b, d;

    __asm__("cpuid;"
            :"=a"(b)
            :"0"(a));

    if (b >= 0x80000007) {

        a = 0x80000007;
        __asm__("cpuid;"
                :"=a"(b), "=d"(d)
                :"0"(a));

        if (d & (1<<8)) {
            printf("Invariant TSC is supported\n");
        } else{
            printf("Invariant TSC is NOT supported\n");
        }
    } else {
        printf("No support for Advanced Power Management Information in 
CPUID\n");
    }
    return 0;
}

Re: Sporadic errors while initializing NICs in example applications, dpdk-1.5.0r1

[Thomas Monjalon]

Thomas29/11/2013 13:25,  Monjalon :
> 29/11/2013 14:53, Dmitry Vyal :
> > On 11/28/2013 03:01 PM, Richardson, Bruce wrote:
> > > [BR] Frequency changes should not affect timers for modern Intel CPUs.
> > 
> > The error frequency greatly reduces if I patch
> > loop limit as I described earlier or if I call rte_power_init and
> > rte_power_freq_max as Thomas suggested.
> > 
> > But the only way to get rid of them completely is to set performance
> > governor.
> 
> Please check that your hardware do not support invariant TSC.
> It would explain why you need to fix frequency.
> 
> I attach a simple code to test CPU feature "Invariant TSC".

Note that this feature is called constant_tsc in /proc/cpuinfo:
	grep --color -m1 constant_tsc /proc/cpuinfo
It is checked by check_tsc_flags() at DPDK initialization and should print a 
warning if missing:
http://dpdk.org/browse/dpdk/commit/?id=e7c8996e13b9abb706ea0de53271346f4f86ca03

-- 
Thomas

Re: Sporadic errors while initializing NICs in example applications, dpdk-1.5.0r1

[Dmitry Vyal]

On 11/29/2013 04:39 PM, Thomas Monjalon wrote:
> 29/11/2013 13:25, Thomas Monjalon :
>
> Please check that your hardware do not support invariant TSC.
> It would explain why you need to fix frequency.
>
> I attach a simple code to test CPU feature "Invariant TSC".

I compiled and ran the code on all the platforms I had troubles on. 
Invariant TSC is supported everywhere.

> It seems that the file is stripped on the mailing-list.
> Code inlined:
>
> #include <stdlib.h>
> #include <stdio.h>
> #include <unistd.h>
> #include <stdint.h>
>
>
> int main()
> {
>      uint32_t a = 0x80000000;
>      uint32_t b, d;
>
>      __asm__("cpuid;"
>              :"=a"(b)
>              :"0"(a));
>
>      if (b >= 0x80000007) {
>
>          a = 0x80000007;
>          __asm__("cpuid;"
>                  :"=a"(b), "=d"(d)
>                  :"0"(a));
>
>          if (d & (1<<8)) {
>              printf("Invariant TSC is supported\n");
>          } else{
>              printf("Invariant TSC is NOT supported\n");
>          }
>      } else {
>          printf("No support for Advanced Power Management Information in
> CPUID\n");
>      }
>      return 0;
> }
>