Need help with strange mlx4_core error

Need help with strange mlx4_core error

[Vasiliy Tolstov]

Hi all. After switching to kernel 3.10 i get sometimes errors in dmesg
like this:
[Sun Jan 26 09:58:50 2014] <mlx4_ib> destroy_qp_common: modify QP
007def to RESET failed.
[Sun Jan 26 20:27:52 2014] <mlx4_ib> destroy_qp_common: modify QP
0221ca to RESET failed.
[Mon Jan 27 03:44:20 2014] <mlx4_ib> destroy_qp_common: modify QP
0232ad to RESET failed.
[Mon Jan 27 14:23:25 2014] mlx4_core 0000:03:00.0: command 0x19
failed: fw status = 0x9
[Mon Jan 27 14:23:25 2014] ib0: failed to modify QP to INIT: -9
[Mon Jan 27 16:37:00 2014] <mlx4_ib> destroy_qp_common: modify QP
00258a to RESET failed.
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0: Internal error detected:
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0:   buf[00]: 001805a5
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0:   buf[01]: 00000000
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0:   buf[02]: 20060384
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0:   buf[03]: 00000000
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0:   buf[04]: 0018050c
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0:   buf[05]: 00000001
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0:   buf[06]: 00002cd4
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0:   buf[07]: 00000084
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0:   buf[08]: 0000f8af
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0:   buf[09]: 00004000
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0:   buf[0a]: 00000000
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0:   buf[0b]: 00000000
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0:   buf[0c]: 00000000
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0:   buf[0d]: 00000000
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0:   buf[0e]: 00000000
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0:   buf[0f]: 00000000
[Mon Jan 27 16:58:26 2014] mlx4_en 0000:03:00.0: Internal error
detected, restarting device
[Mon Jan 27 16:58:35 2014] <mlx4_ib> destroy_qp_common: modify QP
002cd4 to RESET failed.
[Mon Jan 27 16:58:35 2014] ib0: dev_queue_xmit failed to requeue packet
[Mon Jan 27 16:58:35 2014] ib0: dev_queue_xmit failed to requeue packet
[Mon Jan 27 16:58:35 2014] ib0: dev_queue_xmit failed to requeue packet
[Mon Jan 27 16:58:36 2014] mlx4_core: Initializing 0000:03:00.0
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 59 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 60 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 61 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 62 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 63 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 64 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 65 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 66 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 67 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 68 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 69 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 70 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 71 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: command 0xc failed:
fw status = 0x40
[Mon Jan 27 16:58:38 2014] mlx4_en 0000:03:00.0: UDP RSS is not
supported on this device.

I don't know what traffic can trigger this (i'm using IPoIB with
connected mode) but i think this can happening then someone send
massive udp traffic.
What can i do to fix this issue? When error appears ib0 device (IPoIB)
goes to down.

Very big thanks for all help.

-- 
Vasiliy Tolstov,
e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
jabber: vase-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Need help with strange mlx4_core error

[Vasiliy Tolstov]

2014-02-03 Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
> What kernel you used before? can you please send here the dmesg output when
> the mlx4_core driver loads with debug_level=1


Before that i'm using 2.6.32 from debian squeeze...
dmesg:
[583789.683677] mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011)
[583789.683710] mlx4_core: Initializing 0000:03:00.0
[583790.687071] mlx4_core 0000:03:00.0: FW version 2.6.900 (cmd intf
rev 3), max commands 16
[583790.687077] mlx4_core 0000:03:00.0: Catastrophic error buffer at
0x1f020, size 0x10, BAR 0
[583790.687081] mlx4_core 0000:03:00.0: Communication vector bar:0 offset:0x0
[583790.687084] mlx4_core 0000:03:00.0: FW size 385 KB
[583790.687088] mlx4_core 0000:03:00.0: Internal clock bar:0 offset:0x0
[583790.687091] mlx4_core 0000:03:00.0: Clear int @ f0058, BAR 0
[583790.688781] mlx4_core 0000:03:00.0: Mapped 26 chunks/6168 KB for FW.
[583791.325846] mlx4_core 0000:03:00.0: BlueFlame available (reg size
512, regs/page 8)
[583791.325952] mlx4_core 0000:03:00.0: Base MM extensions: flags
00000cc0, rsvd L_Key 00000500
[583791.325954] mlx4_core 0000:03:00.0: Max ICM size 4294967296 MB
[583791.325957] mlx4_core 0000:03:00.0: Max QPs: 16777216, reserved
QPs: 64, entry size: 256
[583791.325959] mlx4_core 0000:03:00.0: Max SRQs: 16777216, reserved
SRQs: 64, entry size: 128
[583791.325962] mlx4_core 0000:03:00.0: Max CQs: 16777216, reserved
CQs: 128, entry size: 128
[583791.325964] mlx4_core 0000:03:00.0: Max EQs: 512, reserved EQs: 4,
entry size: 128
[583791.325967] mlx4_core 0000:03:00.0: reserved MPTs: 16, reserved MTTs: 16
[583791.325969] mlx4_core 0000:03:00.0: Max PDs: 8388608, reserved
PDs: 4, reserved UARs: 1
[583791.325971] mlx4_core 0000:03:00.0: Max QP/MCG: 8388608, reserved MGMs: 0
[583791.325974] mlx4_core 0000:03:00.0: Max CQEs: 4194304, max WQEs:
16384, max SRQ WQEs: 16384
[583791.325976] mlx4_core 0000:03:00.0: Local CA ACK delay: 15, max
MTU: 4096, port width cap: 3
[583791.325978] mlx4_core 0000:03:00.0: Max SQ desc size: 1008, max SQ S/G: 62
[583791.325981] mlx4_core 0000:03:00.0: Max RQ desc size: 512, max RQ S/G: 32
[583791.325983] mlx4_core 0000:03:00.0: Max GSO size: 131072
[583791.325985] mlx4_core 0000:03:00.0: Max counters: 0
[583791.325987] mlx4_core 0000:03:00.0: Max RSS Table size: 256
[583791.325989] mlx4_core 0000:03:00.0: DEV_CAP flags:
[583791.325991] mlx4_core 0000:03:00.0:     RC transport
[583791.325993] mlx4_core 0000:03:00.0:     UC transport
[583791.325995] mlx4_core 0000:03:00.0:     UD transport
[583791.325997] mlx4_core 0000:03:00.0:     XRC transport
[583791.325999] mlx4_core 0000:03:00.0:     FCoIB support
[583791.326001] mlx4_core 0000:03:00.0:     SRQ support
[583791.326003] mlx4_core 0000:03:00.0:     IPoIB checksum offload
[583791.326005] mlx4_core 0000:03:00.0:     P_Key violation counter
[583791.326007] mlx4_core 0000:03:00.0:     Q_Key violation counter
[583791.326009] mlx4_core 0000:03:00.0:     Big LSO headers
[583791.326011] mlx4_core 0000:03:00.0:     APM support
[583791.326013] mlx4_core 0000:03:00.0:     Atomic ops support
[583791.326015] mlx4_core 0000:03:00.0:     Address vector port checking support
[583791.326017] mlx4_core 0000:03:00.0:     UD multicast support
[583791.326019] mlx4_core 0000:03:00.0:     Router support
[583791.326021] mlx4_core 0000:03:00.0:     IBoE support
[583791.326023] mlx4_core 0000:03:00.0:     RSS support
[583791.326025] mlx4_core 0000:03:00.0:     RSS Toeplitz Hash Function support
[583791.326027] mlx4_core 0000:03:00.0:     RSS XOR Hash Function support
[583791.326030] mlx4_core 0000:03:00.0: Steering mode is: A0 steering,
oper_log_mgm_entry_size = 10, modparam log_num_mgm_entry_size = 10
[583791.326037] mlx4_core 0000:03:00.0:   profile[ 0] (  CMPT): 2^26
entries @ 0x         0, size 0x 100000000
[583791.326040] mlx4_core 0000:03:00.0:   profile[ 1] (RDMARC): 2^22
entries @ 0x 100000000, size 0x   8000000
[583791.326043] mlx4_core 0000:03:00.0:   profile[ 2] (    QP): 2^18
entries @ 0x 108000000, size 0x   4000000
[583791.326045] mlx4_core 0000:03:00.0:   profile[ 3] (   MTT): 2^23
entries @ 0x 10c000000, size 0x   4000000
[583791.326048] mlx4_core 0000:03:00.0:   profile[ 4] (  DMPT): 2^19
entries @ 0x 110000000, size 0x   2000000
[583791.326051] mlx4_core 0000:03:00.0:   profile[ 5] (  ALTC): 2^18
entries @ 0x 112000000, size 0x   1000000
[583791.326054] mlx4_core 0000:03:00.0:   profile[ 6] (   SRQ): 2^16
entries @ 0x 113000000, size 0x    800000
[583791.326057] mlx4_core 0000:03:00.0:   profile[ 7] (    CQ): 2^16
entries @ 0x 113800000, size 0x    800000
[583791.326060] mlx4_core 0000:03:00.0:   profile[ 8] (   MCG): 2^13
entries @ 0x 114000000, size 0x    800000
[583791.326063] mlx4_core 0000:03:00.0:   profile[ 9] (  AUXC): 2^18
entries @ 0x 114800000, size 0x     40000
[583791.326066] mlx4_core 0000:03:00.0:   profile[10] (    EQ): 2^06
entries @ 0x 114840000, size 0x      2000
[583791.326068] mlx4_core 0000:03:00.0: HCA context memory: reserving 4530440 KB
[583791.326089] mlx4_core 0000:03:00.0: 4530440 KB of HCA context
requires 8888 KB aux memory.
[583791.348513] mlx4_core 0000:03:00.0: Mapped 38 chunks/8888 KB for ICM aux.
[583791.349649] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 0 for ICM.
[583791.350756] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
40000000 for ICM.
[583791.351862] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
80000000 for ICM.
[583791.351894] mlx4_core 0000:03:00.0: Mapped 1 chunks/4 KB at
c0000000 for ICM.
[583791.351942] mlx4_core 0000:03:00.0: Mapped 1 chunks/8 KB at
114840000 for ICM.
[583791.353028] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
10c000000 for ICM.
[583791.354151] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
110000000 for ICM.
[583791.355235] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
108000000 for ICM.
[583791.356319] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114800000 for ICM.
[583791.357404] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
112000000 for ICM.
[583791.358489] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
100000000 for ICM.
[583791.359571] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
113800000 for ICM.
[583791.360657] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
113000000 for ICM.
[583791.361741] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114000000 for ICM.
[583791.362824] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114040000 for ICM.
[583791.363907] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114080000 for ICM.
[583791.364991] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
1140c0000 for ICM.
[583791.366074] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114100000 for ICM.
[583791.367158] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114140000 for ICM.
[583791.368240] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114180000 for ICM.
[583791.369324] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
1141c0000 for ICM.
[583791.370409] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114200000 for ICM.
[583791.371491] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114240000 for ICM.
[583791.372574] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114280000 for ICM.
[583791.373657] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
1142c0000 for ICM.
[583791.374741] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114300000 for ICM.
[583791.375823] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114340000 for ICM.
[583791.376908] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114380000 for ICM.
[583791.377992] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
1143c0000 for ICM.
[583791.379073] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114400000 for ICM.
[583791.380158] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114440000 for ICM.
[583791.381242] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114480000 for ICM.
[583791.382326] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
1144c0000 for ICM.
[583791.383408] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114500000 for ICM.
[583791.384493] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114540000 for ICM.
[583791.385577] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114580000 for ICM.
[583791.386659] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
1145c0000 for ICM.
[583791.387742] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114600000 for ICM.
[583791.388824] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114640000 for ICM.
[583791.389909] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114680000 for ICM.
[583791.390992] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
1146c0000 for ICM.
[583791.392076] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114700000 for ICM.
[583791.393159] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114740000 for ICM.
[583791.394243] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
114780000 for ICM.
[583791.395325] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at
1147c0000 for ICM.
[583791.905303] mlx4_core 0000:03:00.0: irq 59 for MSI/MSI-X
[583791.905313] mlx4_core 0000:03:00.0: irq 60 for MSI/MSI-X
[583791.905319] mlx4_core 0000:03:00.0: irq 61 for MSI/MSI-X
[583791.905327] mlx4_core 0000:03:00.0: irq 62 for MSI/MSI-X
[583791.905333] mlx4_core 0000:03:00.0: irq 63 for MSI/MSI-X
[583791.905340] mlx4_core 0000:03:00.0: irq 64 for MSI/MSI-X
[583791.905346] mlx4_core 0000:03:00.0: irq 65 for MSI/MSI-X
[583791.905354] mlx4_core 0000:03:00.0: irq 66 for MSI/MSI-X
[583791.905360] mlx4_core 0000:03:00.0: irq 67 for MSI/MSI-X
[583791.905366] mlx4_core 0000:03:00.0: irq 68 for MSI/MSI-X
[583791.905373] mlx4_core 0000:03:00.0: irq 69 for MSI/MSI-X
[583791.905379] mlx4_core 0000:03:00.0: irq 70 for MSI/MSI-X
[583791.905386] mlx4_core 0000:03:00.0: irq 71 for MSI/MSI-X
[583791.970917] mlx4_core 0000:03:00.0: NOP command IRQ test passed
[583791.971212] mlx4_core 0000:03:00.0: command 0xc failed: fw status = 0x40



-- 
Vasiliy Tolstov,
e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
jabber: vase-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Need help with strange mlx4_core error

[Or Gerlitz]

On Mon, Feb 3, 2014 at 9:27 AM, Vasiliy Tolstov <v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org> wrote:
> 2014-02-03 Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
>> What kernel you used before? can you please send here the dmesg output when
>> the mlx4_core driver loads with debug_level=1
>
>
> Before that i'm using 2.6.32 from debian squeeze... dmesg:
> [583789.683677] mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011)
> [583789.683710] mlx4_core: Initializing 0000:03:00.0
> [583790.687071] mlx4_core 0000:03:00.0: FW version 2.6.900 (cmd intf
> rev 3), max commands 16


This is pretty much very old firmware, could you update it please with
the latest GA from the mellanox site and see if things smile back?
also send the "lspci | grep -i Mellanox"
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Need help with strange mlx4_core error

[Vasiliy Tolstov]

2014-02-03 Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
> This is pretty much very old firmware, could you update it please with
> the latest GA from the mellanox site and see if things smile back?
> also send the "lspci | grep -i Mellanox"


Thanks. I'm try to connect with supermicro team, because blade from
supermicro and mellanox mftlint can't upgrade firmware =(.
lspci output:
03:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe
2.0 5GT/s - IB QDR / 10GigE] (rev a0)

-- 
Vasiliy Tolstov,
e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
jabber: vase-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Need help with strange mlx4_core error

[Vasiliy Tolstov]

2014-02-04 Vasiliy Tolstov <v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org>:
>> This is pretty much very old firmware, could you update it please with
>> the latest GA from the mellanox site and see if things smile back?
>> also send the "lspci | grep -i Mellanox"


I'm update firmware to 2.7.200 (that is the latest from ftp
supermicro). Now i'm try to test it.

-- 
Vasiliy Tolstov,
e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
jabber: vase-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Need help with strange mlx4_core error

[Vasiliy Tolstov]

2014-02-04 18:19 GMT+04:00 Vasiliy Tolstov <v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org>:
> 2014-02-04 Vasiliy Tolstov <v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org>:
>>> This is pretty much very old firmware, could you update it please with
>>> the latest GA from the mellanox site and see if things smile back?
>>> also send the "lspci | grep -i Mellanox"
>
>
> I'm update firmware to 2.7.200 (that is the latest from ftp
> supermicro). Now i'm try to test it.

Sorry for bumping old thread. Now i again have this error:
ibstat:
CA 'mlx4_0'
        CA type: MT26428
        Number of ports: 1
        Firmware version: 2.7.5558
        Hardware version: a0
        Node GUID: 0x003048ffff9dc9d8
        System image GUID: 0x003048ffff9dc9db
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 7
                LMC: 0
                SM lid: 6
                Capability mask: 0x02510868

dmesg:
[1067167.780266] <mlx4_ib> destroy_qp_common: modify QP 00110e to RESET failed.
[1068708.243893] <mlx4_ib> destroy_qp_common: modify QP 001118 to RESET failed.
[1174994.897394] mlx4_core 0000:03:00.0: Internal error detected:
[1174994.897484] mlx4_core 0000:03:00.0:   buf[00]: 001805a5
[1174994.897516] mlx4_core 0000:03:00.0:   buf[01]: 00000000
[1174994.897544] mlx4_core 0000:03:00.0:   buf[02]: 200715b6
[1174994.897577] mlx4_core 0000:03:00.0:   buf[03]: 00000000
[1174994.897607] mlx4_core 0000:03:00.0:   buf[04]: 0018050c
[1174994.897637] mlx4_core 0000:03:00.0:   buf[05]: 00000001
[1174994.897666] mlx4_core 0000:03:00.0:   buf[06]: 00002fd8
[1174994.897697] mlx4_core 0000:03:00.0:   buf[07]: 00000084
[1174994.897727] mlx4_core 0000:03:00.0:   buf[08]: 0000db9f
[1174994.897756] mlx4_core 0000:03:00.0:   buf[09]: 00004000
[1174994.897786] mlx4_core 0000:03:00.0:   buf[0a]: 00000000
[1174994.897815] mlx4_core 0000:03:00.0:   buf[0b]: 00000000
[1174994.897845] mlx4_core 0000:03:00.0:   buf[0c]: 00000000
[1174994.897874] mlx4_core 0000:03:00.0:   buf[0d]: 00000000
[1174994.897904] mlx4_core 0000:03:00.0:   buf[0e]: 00000000
[1174994.897934] mlx4_core 0000:03:00.0:   buf[0f]: 00000000
[1174994.897963] mlx4_en 0000:03:00.0: Internal error detected,
restarting device
[1175000.294830] <mlx4_ib> destroy_qp_common: modify QP 001eca to RESET failed.
[1175000.352029] bonding: bond1: releasing active interface ib0
[1175000.352072] bonding: bond1: Warning: clearing HW address of bond1
while it still has VLANs.
[1175000.352182] bonding: bond1: When re-adding slaves, make sure the
bond's HW address matches its VLANs'.
[1175000.352517] bonding: bond1: destroying bond bond1.
[1175000.354034] bonding: bond1: released all slaves
[1175000.354694] ib0: ib_cq_destroy (recv) failed
[1175000.354721] ib0: ib_destroy_srq failed: -16
[1175000.355780] ib0: ib_dealloc_pd failed




-- 
Vasiliy Tolstov,
e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
jabber: vase-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Need help with strange mlx4_core error

[Vasiliy Tolstov]

2014-02-28 22:24 GMT+04:00 Vasiliy Tolstov <v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org>:
>>>> This is pretty much very old firmware, could you update it please with
>>>> the latest GA from the mellanox site and see if things smile back?
>>>> also send the "lspci | grep -i Mellanox"
>>


And after looking dmesg i see sometime early:
[271701.934646] device ib0 left promiscuous mode
[292670.940271] ------------[ cut here ]------------
[292670.940309] WARNING: at
/tmp/buildd/linux-3.10.11/net/sched/sch_generic.c:255
dev_watchdog+0xe3/0x153()
[292670.940359] NETDEV WATCHDOG: ib0 (mlx4_core): transmit queue 0 timed out
[292670.940388] Modules linked in: tcp_diag inet_diag ip_set_hash_net
xt_set ip_set nfnetlink xt_tcpudp xt_pkttype ip6table_mangle
ip6table_raw iptable_mangle iptable_raw
 ip6table_filter ip6_tables iptable_filter ip_tables 8021q garp stp
mrp llc bonding ib_ucm ib_uverbs ib_addr ib_umad ib_ipoib ib_srp
scsi_transport_srp ib_cm scsi_tgt mlx
4_ib ib_sa ib_mad ib_core mlx4_en ipmi_si ipmi_devintf ipmi_msghandler
ipt_NETFLOW(O) x_tables dm_mod md_mod iTCO_wdt coretemp acpi_cpufreq
i7core_edac snd_pcm mperf iTCO
_vendor_support snd_page_alloc snd_timer edac_core snd soundcore
psmouse kvm_intel i2c_i801 ioatdma pcspkr serio_raw kvm lpc_ich evdev
joydev mfd_core crc32c_intel proces
sor button thermal_sys squashfs loop aufs(C) hid_generic usbhid hid
ata_generic ata_piix microcode uhci_hcd ehci_pci mpt2sas libata
ehci_hcd raid_class scsi_transport_sas
 usbcore usb_common scsi_mod mlx4_core igb i2c_algo_bit i2c_core dca
ptp pps_core [last unloaded: nf_conntrack]
[292670.940991] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         CIO
3.10-3-amd64 #1 Debian
3.10.11-3+0~20131204153435.21+wheezy~1.gbpa177bd
[292670.941045] Hardware name: Supermicro B8DT6/B8DT6, BIOS 080015  03/03/2010
[292670.941075]  ffffffff8138d323 0000000000000000 ffffffff8103be31
ffff8801bfc0dc00
[292670.941130]  ffff8801bfc03e08 0000000000000000 ffff8801b8688000
0000000000000100
[292670.941185]  ffffffff812e5e78 ffff8801b8688348 ffffffff8103bee1
ffffffff8153628f
[292670.941240] Call Trace:
[292670.941262]  <IRQ>  [<ffffffff8138d323>] ? dump_stack+0xd/0x17
[292670.941303]  [<ffffffff8103be31>] ? warn_slowpath_common+0x5f/0x77
[292670.941334]  [<ffffffff812e5e78>] ? netif_tx_lock+0x7a/0x7a
[292670.941364]  [<ffffffff8103bee1>] ? warn_slowpath_fmt+0x45/0x4a
[292670.941394]  [<ffffffff812e5e65>] ? netif_tx_lock+0x67/0x7a
[292670.941426]  [<ffffffff812e5f5b>] ? dev_watchdog+0xe3/0x153
[292670.941458]  [<ffffffff810471ee>] ? call_timer_fn+0x4b/0xf6
[292670.941488]  [<ffffffff812e5e78>] ? netif_tx_lock+0x7a/0x7a
[292670.941518]  [<ffffffff810482fe>] ? run_timer_softirq+0x18d/0x1d6
[292670.941548]  [<ffffffff81042576>] ? __do_softirq+0xec/0x209
[292670.941578]  [<ffffffff8104275e>] ? irq_exit+0x3f/0x83
[292670.941608]  [<ffffffff8100e6f3>] ? do_IRQ+0x81/0x97
[292670.941639]  [<ffffffff81390a6d>] ? common_interrupt+0x6d/0x6d
[292670.941666]  <EOI>  [<ffffffff812a70ec>] ? arch_local_irq_enable+0x4/0x8
[292670.941708]  [<ffffffff812a74af>] ? cpuidle_enter_state+0x46/0xb1
[292670.941739]  [<ffffffff812a75f0>] ? cpuidle_idle_call+0xd6/0x147
[292670.941770]  [<ffffffff81013add>] ? arch_cpu_idle+0x6/0x1a
[292670.941802]  [<ffffffff810734d3>] ? cpu_startup_entry+0x114/0x191
[292670.941835]  [<ffffffff816b3d51>] ? start_kernel+0x3e8/0x3f3
[292670.941864]  [<ffffffff816b377f>] ? repair_env_string+0x57/0x57
[292670.941895]  [<ffffffff816b359a>] ? x86_64_start_kernel+0xf2/0xfd
[292670.941924] ---[ end trace 119aa908393ed9b3 ]---
[292670.941950] ib0: transmit timeout: latency 1356 msecs
[292670.941977] ib0: queue stopped 1, tx_head 595756, tx_tail 595756


-- 
Vasiliy Tolstov,
e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
jabber: vase-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Need help with strange mlx4_core error

[Bart Van Assche]

On 02/28/14 19:35, Vasiliy Tolstov wrote:
> [292670.940309] WARNING: at
> /tmp/buildd/linux-3.10.11/net/sched/sch_generic.c:255
> dev_watchdog+0xe3/0x153()
> [292670.940359] NETDEV WATCHDOG: ib0 (mlx4_core): transmit queue 0 timed out

I'm not sure but I think there is a fix in kernel 3.11 for this issue.

Bart.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Need help with strange mlx4_core error

[Vasiliy Tolstov]

2014-03-01 16:50 GMT+04:00 Bart Van Assche <bvanassche-HInyCGIudOg@public.gmane.org>:
> I'm not sure but I think there is a fix in kernel 3.11 for this issue.


Thanks, but as i understand, 3.10 lts and need to get this updates..
But thanks! I'm try to search changelogs for 3.11.

-- 
Vasiliy Tolstov,
e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
jabber: vase-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Need help with strange mlx4_core error

[Vasiliy Tolstov]

2014-03-01 20:55 GMT+04:00 Vasiliy Tolstov <v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org>:
>
> Thanks, but as i understand, 3.10 lts and need to get this updates..
> But thanks! I'm try to search changelogs for 3.11.


Hmm. May me i miss something, but i can't see in changelogs for 3.11
(all) fixes for this error.

-- 
Vasiliy Tolstov,
e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
jabber: vase-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Need help with strange mlx4_core error

[Bart Van Assche]

On 03/01/14 18:16, Vasiliy Tolstov wrote:
> 2014-03-01 20:55 GMT+04:00 Vasiliy Tolstov <v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org>:
>> Thanks, but as i understand, 3.10 lts and need to get this updates..
>> But thanks! I'm try to search changelogs for 3.11.
> 
> Hmm. May me i miss something, but i can't see in changelogs for 3.11
> (all) fixes for this error.

I have seen this issue myself with the 3.0 mlx4_en driver but do not see
it anymore with the latest version of this driver. I do not know which
patch fixes this issue. But I have been wondering whether it could have
been the following patch:

commit 0cc5c8bf11852dec3225fda2f53a599243095d23

net/mlx4_en: Fix a race between napi poll function and RX ring cleanup

The RX rings were cleaned while there was still possible RX traffic
completion handling. Change the sequence of events so that the port is
closed and the QPs are being stopped before RX cleanup.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Need help with strange mlx4_core error

[Vasiliy Tolstov]

2014-03-01 21:23 GMT+04:00 Bart Van Assche <bvanassche-HInyCGIudOg@public.gmane.org>:
> I have seen this issue myself with the 3.0 mlx4_en driver but do not see
> it anymore with the latest version of this driver. I do not know which
> patch fixes this issue. But I have been wondering whether it could have
> been the following patch:
>
> commit 0cc5c8bf11852dec3225fda2f53a599243095d23
>
> net/mlx4_en: Fix a race between napi poll function and RX ring cleanup
>
> The RX rings were cleaned while there was still possible RX traffic
> completion handling. Change the sequence of events so that the port is
> closed and the QPs are being stopped before RX cleanup.


Thanks! I'm try it.

-- 
Vasiliy Tolstov,
e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
jabber: vase-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Need help with strange mlx4_core error

[Mark Lehrer]

On Feb 28, 2014, Vasiliy Tolstov wrote:

> I'm update firmware to 2.7.200 (that is the latest from ftp
> supermicro). Now i'm try to test it.

Which type of HCA do you have?  The firmware I downloaded from
ftp.supermicro.com was 2.9.1000, and once I upgraded to this version
from 2.7.200 I was able to use mlx4_en.

lspci shows my card as: Mellanox Technologies MT26428 [ConnectX VPI
PCIe 2.0 5GT/s - IB QDR / 10GigE]


Mark
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html