* Need help with strange mlx4_core error
@ 2014-02-03 3:35 Vasiliy Tolstov
[not found] ` <CAJZOPZJmdGutUQZ4icBrKRq-YPwD8TqtoLH9SgqNVx_CDjL0TQ@mail.gmail.com>
0 siblings, 1 reply; 13+ messages in thread
From: Vasiliy Tolstov @ 2014-02-03 3:35 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
Hi all. After switching to kernel 3.10 i get sometimes errors in dmesg
like this:
[Sun Jan 26 09:58:50 2014] <mlx4_ib> destroy_qp_common: modify QP
007def to RESET failed.
[Sun Jan 26 20:27:52 2014] <mlx4_ib> destroy_qp_common: modify QP
0221ca to RESET failed.
[Mon Jan 27 03:44:20 2014] <mlx4_ib> destroy_qp_common: modify QP
0232ad to RESET failed.
[Mon Jan 27 14:23:25 2014] mlx4_core 0000:03:00.0: command 0x19
failed: fw status = 0x9
[Mon Jan 27 14:23:25 2014] ib0: failed to modify QP to INIT: -9
[Mon Jan 27 16:37:00 2014] <mlx4_ib> destroy_qp_common: modify QP
00258a to RESET failed.
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0: Internal error detected:
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0: buf[00]: 001805a5
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0: buf[01]: 00000000
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0: buf[02]: 20060384
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0: buf[03]: 00000000
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0: buf[04]: 0018050c
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0: buf[05]: 00000001
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0: buf[06]: 00002cd4
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0: buf[07]: 00000084
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0: buf[08]: 0000f8af
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0: buf[09]: 00004000
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0: buf[0a]: 00000000
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0: buf[0b]: 00000000
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0: buf[0c]: 00000000
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0: buf[0d]: 00000000
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0: buf[0e]: 00000000
[Mon Jan 27 16:58:26 2014] mlx4_core 0000:03:00.0: buf[0f]: 00000000
[Mon Jan 27 16:58:26 2014] mlx4_en 0000:03:00.0: Internal error
detected, restarting device
[Mon Jan 27 16:58:35 2014] <mlx4_ib> destroy_qp_common: modify QP
002cd4 to RESET failed.
[Mon Jan 27 16:58:35 2014] ib0: dev_queue_xmit failed to requeue packet
[Mon Jan 27 16:58:35 2014] ib0: dev_queue_xmit failed to requeue packet
[Mon Jan 27 16:58:35 2014] ib0: dev_queue_xmit failed to requeue packet
[Mon Jan 27 16:58:36 2014] mlx4_core: Initializing 0000:03:00.0
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 59 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 60 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 61 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 62 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 63 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 64 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 65 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 66 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 67 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 68 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 69 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 70 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: irq 71 for MSI/MSI-X
[Mon Jan 27 16:58:38 2014] mlx4_core 0000:03:00.0: command 0xc failed:
fw status = 0x40
[Mon Jan 27 16:58:38 2014] mlx4_en 0000:03:00.0: UDP RSS is not
supported on this device.
I don't know what traffic can trigger this (i'm using IPoIB with
connected mode) but i think this can happening then someone send
massive udp traffic.
What can i do to fix this issue? When error appears ib0 device (IPoIB)
goes to down.
Very big thanks for all help.
--
Vasiliy Tolstov,
e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
jabber: vase-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 13+ messages in thread[parent not found: <CAJZOPZJmdGutUQZ4icBrKRq-YPwD8TqtoLH9SgqNVx_CDjL0TQ@mail.gmail.com>]
[parent not found: <CAJZOPZJmdGutUQZ4icBrKRq-YPwD8TqtoLH9SgqNVx_CDjL0TQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: Need help with strange mlx4_core error [not found] ` <CAJZOPZJmdGutUQZ4icBrKRq-YPwD8TqtoLH9SgqNVx_CDjL0TQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2014-02-03 7:27 ` Vasiliy Tolstov [not found] ` <CACaajQtHzF9g2kp9MrAaQsvKUQdr10YVa+Np1danmAcR_rvLqg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 13+ messages in thread From: Vasiliy Tolstov @ 2014-02-03 7:27 UTC (permalink / raw) To: Or Gerlitz; +Cc: linux-rdma, Jack Morgenstein 2014-02-03 Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>: > What kernel you used before? can you please send here the dmesg output when > the mlx4_core driver loads with debug_level=1 Before that i'm using 2.6.32 from debian squeeze... dmesg: [583789.683677] mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011) [583789.683710] mlx4_core: Initializing 0000:03:00.0 [583790.687071] mlx4_core 0000:03:00.0: FW version 2.6.900 (cmd intf rev 3), max commands 16 [583790.687077] mlx4_core 0000:03:00.0: Catastrophic error buffer at 0x1f020, size 0x10, BAR 0 [583790.687081] mlx4_core 0000:03:00.0: Communication vector bar:0 offset:0x0 [583790.687084] mlx4_core 0000:03:00.0: FW size 385 KB [583790.687088] mlx4_core 0000:03:00.0: Internal clock bar:0 offset:0x0 [583790.687091] mlx4_core 0000:03:00.0: Clear int @ f0058, BAR 0 [583790.688781] mlx4_core 0000:03:00.0: Mapped 26 chunks/6168 KB for FW. [583791.325846] mlx4_core 0000:03:00.0: BlueFlame available (reg size 512, regs/page 8) [583791.325952] mlx4_core 0000:03:00.0: Base MM extensions: flags 00000cc0, rsvd L_Key 00000500 [583791.325954] mlx4_core 0000:03:00.0: Max ICM size 4294967296 MB [583791.325957] mlx4_core 0000:03:00.0: Max QPs: 16777216, reserved QPs: 64, entry size: 256 [583791.325959] mlx4_core 0000:03:00.0: Max SRQs: 16777216, reserved SRQs: 64, entry size: 128 [583791.325962] mlx4_core 0000:03:00.0: Max CQs: 16777216, reserved CQs: 128, entry size: 128 [583791.325964] mlx4_core 0000:03:00.0: Max EQs: 512, reserved EQs: 4, entry size: 128 [583791.325967] mlx4_core 0000:03:00.0: reserved MPTs: 16, reserved MTTs: 16 [583791.325969] mlx4_core 0000:03:00.0: Max PDs: 8388608, reserved PDs: 4, reserved UARs: 1 [583791.325971] mlx4_core 0000:03:00.0: Max QP/MCG: 8388608, reserved MGMs: 0 [583791.325974] mlx4_core 0000:03:00.0: Max CQEs: 4194304, max WQEs: 16384, max SRQ WQEs: 16384 [583791.325976] mlx4_core 0000:03:00.0: Local CA ACK delay: 15, max MTU: 4096, port width cap: 3 [583791.325978] mlx4_core 0000:03:00.0: Max SQ desc size: 1008, max SQ S/G: 62 [583791.325981] mlx4_core 0000:03:00.0: Max RQ desc size: 512, max RQ S/G: 32 [583791.325983] mlx4_core 0000:03:00.0: Max GSO size: 131072 [583791.325985] mlx4_core 0000:03:00.0: Max counters: 0 [583791.325987] mlx4_core 0000:03:00.0: Max RSS Table size: 256 [583791.325989] mlx4_core 0000:03:00.0: DEV_CAP flags: [583791.325991] mlx4_core 0000:03:00.0: RC transport [583791.325993] mlx4_core 0000:03:00.0: UC transport [583791.325995] mlx4_core 0000:03:00.0: UD transport [583791.325997] mlx4_core 0000:03:00.0: XRC transport [583791.325999] mlx4_core 0000:03:00.0: FCoIB support [583791.326001] mlx4_core 0000:03:00.0: SRQ support [583791.326003] mlx4_core 0000:03:00.0: IPoIB checksum offload [583791.326005] mlx4_core 0000:03:00.0: P_Key violation counter [583791.326007] mlx4_core 0000:03:00.0: Q_Key violation counter [583791.326009] mlx4_core 0000:03:00.0: Big LSO headers [583791.326011] mlx4_core 0000:03:00.0: APM support [583791.326013] mlx4_core 0000:03:00.0: Atomic ops support [583791.326015] mlx4_core 0000:03:00.0: Address vector port checking support [583791.326017] mlx4_core 0000:03:00.0: UD multicast support [583791.326019] mlx4_core 0000:03:00.0: Router support [583791.326021] mlx4_core 0000:03:00.0: IBoE support [583791.326023] mlx4_core 0000:03:00.0: RSS support [583791.326025] mlx4_core 0000:03:00.0: RSS Toeplitz Hash Function support [583791.326027] mlx4_core 0000:03:00.0: RSS XOR Hash Function support [583791.326030] mlx4_core 0000:03:00.0: Steering mode is: A0 steering, oper_log_mgm_entry_size = 10, modparam log_num_mgm_entry_size = 10 [583791.326037] mlx4_core 0000:03:00.0: profile[ 0] ( CMPT): 2^26 entries @ 0x 0, size 0x 100000000 [583791.326040] mlx4_core 0000:03:00.0: profile[ 1] (RDMARC): 2^22 entries @ 0x 100000000, size 0x 8000000 [583791.326043] mlx4_core 0000:03:00.0: profile[ 2] ( QP): 2^18 entries @ 0x 108000000, size 0x 4000000 [583791.326045] mlx4_core 0000:03:00.0: profile[ 3] ( MTT): 2^23 entries @ 0x 10c000000, size 0x 4000000 [583791.326048] mlx4_core 0000:03:00.0: profile[ 4] ( DMPT): 2^19 entries @ 0x 110000000, size 0x 2000000 [583791.326051] mlx4_core 0000:03:00.0: profile[ 5] ( ALTC): 2^18 entries @ 0x 112000000, size 0x 1000000 [583791.326054] mlx4_core 0000:03:00.0: profile[ 6] ( SRQ): 2^16 entries @ 0x 113000000, size 0x 800000 [583791.326057] mlx4_core 0000:03:00.0: profile[ 7] ( CQ): 2^16 entries @ 0x 113800000, size 0x 800000 [583791.326060] mlx4_core 0000:03:00.0: profile[ 8] ( MCG): 2^13 entries @ 0x 114000000, size 0x 800000 [583791.326063] mlx4_core 0000:03:00.0: profile[ 9] ( AUXC): 2^18 entries @ 0x 114800000, size 0x 40000 [583791.326066] mlx4_core 0000:03:00.0: profile[10] ( EQ): 2^06 entries @ 0x 114840000, size 0x 2000 [583791.326068] mlx4_core 0000:03:00.0: HCA context memory: reserving 4530440 KB [583791.326089] mlx4_core 0000:03:00.0: 4530440 KB of HCA context requires 8888 KB aux memory. [583791.348513] mlx4_core 0000:03:00.0: Mapped 38 chunks/8888 KB for ICM aux. [583791.349649] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 0 for ICM. [583791.350756] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 40000000 for ICM. [583791.351862] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 80000000 for ICM. [583791.351894] mlx4_core 0000:03:00.0: Mapped 1 chunks/4 KB at c0000000 for ICM. [583791.351942] mlx4_core 0000:03:00.0: Mapped 1 chunks/8 KB at 114840000 for ICM. [583791.353028] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 10c000000 for ICM. [583791.354151] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 110000000 for ICM. [583791.355235] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 108000000 for ICM. [583791.356319] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114800000 for ICM. [583791.357404] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 112000000 for ICM. [583791.358489] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 100000000 for ICM. [583791.359571] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 113800000 for ICM. [583791.360657] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 113000000 for ICM. [583791.361741] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114000000 for ICM. [583791.362824] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114040000 for ICM. [583791.363907] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114080000 for ICM. [583791.364991] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 1140c0000 for ICM. [583791.366074] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114100000 for ICM. [583791.367158] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114140000 for ICM. [583791.368240] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114180000 for ICM. [583791.369324] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 1141c0000 for ICM. [583791.370409] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114200000 for ICM. [583791.371491] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114240000 for ICM. [583791.372574] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114280000 for ICM. [583791.373657] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 1142c0000 for ICM. [583791.374741] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114300000 for ICM. [583791.375823] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114340000 for ICM. [583791.376908] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114380000 for ICM. [583791.377992] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 1143c0000 for ICM. [583791.379073] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114400000 for ICM. [583791.380158] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114440000 for ICM. [583791.381242] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114480000 for ICM. [583791.382326] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 1144c0000 for ICM. [583791.383408] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114500000 for ICM. [583791.384493] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114540000 for ICM. [583791.385577] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114580000 for ICM. [583791.386659] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 1145c0000 for ICM. [583791.387742] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114600000 for ICM. [583791.388824] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114640000 for ICM. [583791.389909] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114680000 for ICM. [583791.390992] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 1146c0000 for ICM. [583791.392076] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114700000 for ICM. [583791.393159] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114740000 for ICM. [583791.394243] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 114780000 for ICM. [583791.395325] mlx4_core 0000:03:00.0: Mapped 1 chunks/256 KB at 1147c0000 for ICM. [583791.905303] mlx4_core 0000:03:00.0: irq 59 for MSI/MSI-X [583791.905313] mlx4_core 0000:03:00.0: irq 60 for MSI/MSI-X [583791.905319] mlx4_core 0000:03:00.0: irq 61 for MSI/MSI-X [583791.905327] mlx4_core 0000:03:00.0: irq 62 for MSI/MSI-X [583791.905333] mlx4_core 0000:03:00.0: irq 63 for MSI/MSI-X [583791.905340] mlx4_core 0000:03:00.0: irq 64 for MSI/MSI-X [583791.905346] mlx4_core 0000:03:00.0: irq 65 for MSI/MSI-X [583791.905354] mlx4_core 0000:03:00.0: irq 66 for MSI/MSI-X [583791.905360] mlx4_core 0000:03:00.0: irq 67 for MSI/MSI-X [583791.905366] mlx4_core 0000:03:00.0: irq 68 for MSI/MSI-X [583791.905373] mlx4_core 0000:03:00.0: irq 69 for MSI/MSI-X [583791.905379] mlx4_core 0000:03:00.0: irq 70 for MSI/MSI-X [583791.905386] mlx4_core 0000:03:00.0: irq 71 for MSI/MSI-X [583791.970917] mlx4_core 0000:03:00.0: NOP command IRQ test passed [583791.971212] mlx4_core 0000:03:00.0: command 0xc failed: fw status = 0x40 -- Vasiliy Tolstov, e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org jabber: vase-+9FY0jupvH6HXe+LvDLADg@public.gmane.org -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
[parent not found: <CACaajQtHzF9g2kp9MrAaQsvKUQdr10YVa+Np1danmAcR_rvLqg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: Need help with strange mlx4_core error [not found] ` <CACaajQtHzF9g2kp9MrAaQsvKUQdr10YVa+Np1danmAcR_rvLqg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2014-02-03 10:54 ` Or Gerlitz 2014-02-03 21:19 ` Vasiliy Tolstov 0 siblings, 1 reply; 13+ messages in thread From: Or Gerlitz @ 2014-02-03 10:54 UTC (permalink / raw) To: Vasiliy Tolstov; +Cc: linux-rdma, Jack Morgenstein On Mon, Feb 3, 2014 at 9:27 AM, Vasiliy Tolstov <v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org> wrote: > 2014-02-03 Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>: >> What kernel you used before? can you please send here the dmesg output when >> the mlx4_core driver loads with debug_level=1 > > > Before that i'm using 2.6.32 from debian squeeze... dmesg: > [583789.683677] mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011) > [583789.683710] mlx4_core: Initializing 0000:03:00.0 > [583790.687071] mlx4_core 0000:03:00.0: FW version 2.6.900 (cmd intf > rev 3), max commands 16 This is pretty much very old firmware, could you update it please with the latest GA from the mellanox site and see if things smile back? also send the "lspci | grep -i Mellanox" -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Need help with strange mlx4_core error 2014-02-03 10:54 ` Or Gerlitz @ 2014-02-03 21:19 ` Vasiliy Tolstov [not found] ` <CACaajQs9ANqLX7m=TcR2wT6wkiS-M5uVvAnQrvGWPmukZnLqcg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 13+ messages in thread From: Vasiliy Tolstov @ 2014-02-03 21:19 UTC (permalink / raw) To: Or Gerlitz; +Cc: linux-rdma, Jack Morgenstein 2014-02-03 Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>: > This is pretty much very old firmware, could you update it please with > the latest GA from the mellanox site and see if things smile back? > also send the "lspci | grep -i Mellanox" Thanks. I'm try to connect with supermicro team, because blade from supermicro and mellanox mftlint can't upgrade firmware =(. lspci output: 03:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev a0) -- Vasiliy Tolstov, e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org jabber: vase-+9FY0jupvH6HXe+LvDLADg@public.gmane.org -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
[parent not found: <CACaajQs9ANqLX7m=TcR2wT6wkiS-M5uVvAnQrvGWPmukZnLqcg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: Need help with strange mlx4_core error [not found] ` <CACaajQs9ANqLX7m=TcR2wT6wkiS-M5uVvAnQrvGWPmukZnLqcg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2014-02-04 14:19 ` Vasiliy Tolstov [not found] ` <CACaajQuRzBy3oV+-3+NjyE9muGKPN3YQUd5wnkkcoak0GSP=uw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 13+ messages in thread From: Vasiliy Tolstov @ 2014-02-04 14:19 UTC (permalink / raw) To: Vasiliy Tolstov; +Cc: Or Gerlitz, linux-rdma, Jack Morgenstein 2014-02-04 Vasiliy Tolstov <v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org>: >> This is pretty much very old firmware, could you update it please with >> the latest GA from the mellanox site and see if things smile back? >> also send the "lspci | grep -i Mellanox" I'm update firmware to 2.7.200 (that is the latest from ftp supermicro). Now i'm try to test it. -- Vasiliy Tolstov, e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org jabber: vase-+9FY0jupvH6HXe+LvDLADg@public.gmane.org -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
[parent not found: <CACaajQuRzBy3oV+-3+NjyE9muGKPN3YQUd5wnkkcoak0GSP=uw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: Need help with strange mlx4_core error [not found] ` <CACaajQuRzBy3oV+-3+NjyE9muGKPN3YQUd5wnkkcoak0GSP=uw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2014-02-28 18:24 ` Vasiliy Tolstov [not found] ` <CACaajQtKvihFu6X-+9N3VvVKK4Jw3eADBOENhuM+-4s8s=PsUQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 13+ messages in thread From: Vasiliy Tolstov @ 2014-02-28 18:24 UTC (permalink / raw) To: Vasiliy Tolstov; +Cc: Or Gerlitz, linux-rdma, Jack Morgenstein 2014-02-04 18:19 GMT+04:00 Vasiliy Tolstov <v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org>: > 2014-02-04 Vasiliy Tolstov <v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org>: >>> This is pretty much very old firmware, could you update it please with >>> the latest GA from the mellanox site and see if things smile back? >>> also send the "lspci | grep -i Mellanox" > > > I'm update firmware to 2.7.200 (that is the latest from ftp > supermicro). Now i'm try to test it. Sorry for bumping old thread. Now i again have this error: ibstat: CA 'mlx4_0' CA type: MT26428 Number of ports: 1 Firmware version: 2.7.5558 Hardware version: a0 Node GUID: 0x003048ffff9dc9d8 System image GUID: 0x003048ffff9dc9db Port 1: State: Active Physical state: LinkUp Rate: 40 Base lid: 7 LMC: 0 SM lid: 6 Capability mask: 0x02510868 dmesg: [1067167.780266] <mlx4_ib> destroy_qp_common: modify QP 00110e to RESET failed. [1068708.243893] <mlx4_ib> destroy_qp_common: modify QP 001118 to RESET failed. [1174994.897394] mlx4_core 0000:03:00.0: Internal error detected: [1174994.897484] mlx4_core 0000:03:00.0: buf[00]: 001805a5 [1174994.897516] mlx4_core 0000:03:00.0: buf[01]: 00000000 [1174994.897544] mlx4_core 0000:03:00.0: buf[02]: 200715b6 [1174994.897577] mlx4_core 0000:03:00.0: buf[03]: 00000000 [1174994.897607] mlx4_core 0000:03:00.0: buf[04]: 0018050c [1174994.897637] mlx4_core 0000:03:00.0: buf[05]: 00000001 [1174994.897666] mlx4_core 0000:03:00.0: buf[06]: 00002fd8 [1174994.897697] mlx4_core 0000:03:00.0: buf[07]: 00000084 [1174994.897727] mlx4_core 0000:03:00.0: buf[08]: 0000db9f [1174994.897756] mlx4_core 0000:03:00.0: buf[09]: 00004000 [1174994.897786] mlx4_core 0000:03:00.0: buf[0a]: 00000000 [1174994.897815] mlx4_core 0000:03:00.0: buf[0b]: 00000000 [1174994.897845] mlx4_core 0000:03:00.0: buf[0c]: 00000000 [1174994.897874] mlx4_core 0000:03:00.0: buf[0d]: 00000000 [1174994.897904] mlx4_core 0000:03:00.0: buf[0e]: 00000000 [1174994.897934] mlx4_core 0000:03:00.0: buf[0f]: 00000000 [1174994.897963] mlx4_en 0000:03:00.0: Internal error detected, restarting device [1175000.294830] <mlx4_ib> destroy_qp_common: modify QP 001eca to RESET failed. [1175000.352029] bonding: bond1: releasing active interface ib0 [1175000.352072] bonding: bond1: Warning: clearing HW address of bond1 while it still has VLANs. [1175000.352182] bonding: bond1: When re-adding slaves, make sure the bond's HW address matches its VLANs'. [1175000.352517] bonding: bond1: destroying bond bond1. [1175000.354034] bonding: bond1: released all slaves [1175000.354694] ib0: ib_cq_destroy (recv) failed [1175000.354721] ib0: ib_destroy_srq failed: -16 [1175000.355780] ib0: ib_dealloc_pd failed -- Vasiliy Tolstov, e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org jabber: vase-+9FY0jupvH6HXe+LvDLADg@public.gmane.org -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
[parent not found: <CACaajQtKvihFu6X-+9N3VvVKK4Jw3eADBOENhuM+-4s8s=PsUQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: Need help with strange mlx4_core error [not found] ` <CACaajQtKvihFu6X-+9N3VvVKK4Jw3eADBOENhuM+-4s8s=PsUQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2014-02-28 18:35 ` Vasiliy Tolstov [not found] ` <CACaajQuQ+mbwhp4jxUt+9r4Rq=14oBUqC-=xx6J3EGpAfecG-Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 13+ messages in thread From: Vasiliy Tolstov @ 2014-02-28 18:35 UTC (permalink / raw) To: Vasiliy Tolstov; +Cc: Or Gerlitz, linux-rdma, Jack Morgenstein 2014-02-28 22:24 GMT+04:00 Vasiliy Tolstov <v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org>: >>>> This is pretty much very old firmware, could you update it please with >>>> the latest GA from the mellanox site and see if things smile back? >>>> also send the "lspci | grep -i Mellanox" >> And after looking dmesg i see sometime early: [271701.934646] device ib0 left promiscuous mode [292670.940271] ------------[ cut here ]------------ [292670.940309] WARNING: at /tmp/buildd/linux-3.10.11/net/sched/sch_generic.c:255 dev_watchdog+0xe3/0x153() [292670.940359] NETDEV WATCHDOG: ib0 (mlx4_core): transmit queue 0 timed out [292670.940388] Modules linked in: tcp_diag inet_diag ip_set_hash_net xt_set ip_set nfnetlink xt_tcpudp xt_pkttype ip6table_mangle ip6table_raw iptable_mangle iptable_raw ip6table_filter ip6_tables iptable_filter ip_tables 8021q garp stp mrp llc bonding ib_ucm ib_uverbs ib_addr ib_umad ib_ipoib ib_srp scsi_transport_srp ib_cm scsi_tgt mlx 4_ib ib_sa ib_mad ib_core mlx4_en ipmi_si ipmi_devintf ipmi_msghandler ipt_NETFLOW(O) x_tables dm_mod md_mod iTCO_wdt coretemp acpi_cpufreq i7core_edac snd_pcm mperf iTCO _vendor_support snd_page_alloc snd_timer edac_core snd soundcore psmouse kvm_intel i2c_i801 ioatdma pcspkr serio_raw kvm lpc_ich evdev joydev mfd_core crc32c_intel proces sor button thermal_sys squashfs loop aufs(C) hid_generic usbhid hid ata_generic ata_piix microcode uhci_hcd ehci_pci mpt2sas libata ehci_hcd raid_class scsi_transport_sas usbcore usb_common scsi_mod mlx4_core igb i2c_algo_bit i2c_core dca ptp pps_core [last unloaded: nf_conntrack] [292670.940991] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G CIO 3.10-3-amd64 #1 Debian 3.10.11-3+0~20131204153435.21+wheezy~1.gbpa177bd [292670.941045] Hardware name: Supermicro B8DT6/B8DT6, BIOS 080015 03/03/2010 [292670.941075] ffffffff8138d323 0000000000000000 ffffffff8103be31 ffff8801bfc0dc00 [292670.941130] ffff8801bfc03e08 0000000000000000 ffff8801b8688000 0000000000000100 [292670.941185] ffffffff812e5e78 ffff8801b8688348 ffffffff8103bee1 ffffffff8153628f [292670.941240] Call Trace: [292670.941262] <IRQ> [<ffffffff8138d323>] ? dump_stack+0xd/0x17 [292670.941303] [<ffffffff8103be31>] ? warn_slowpath_common+0x5f/0x77 [292670.941334] [<ffffffff812e5e78>] ? netif_tx_lock+0x7a/0x7a [292670.941364] [<ffffffff8103bee1>] ? warn_slowpath_fmt+0x45/0x4a [292670.941394] [<ffffffff812e5e65>] ? netif_tx_lock+0x67/0x7a [292670.941426] [<ffffffff812e5f5b>] ? dev_watchdog+0xe3/0x153 [292670.941458] [<ffffffff810471ee>] ? call_timer_fn+0x4b/0xf6 [292670.941488] [<ffffffff812e5e78>] ? netif_tx_lock+0x7a/0x7a [292670.941518] [<ffffffff810482fe>] ? run_timer_softirq+0x18d/0x1d6 [292670.941548] [<ffffffff81042576>] ? __do_softirq+0xec/0x209 [292670.941578] [<ffffffff8104275e>] ? irq_exit+0x3f/0x83 [292670.941608] [<ffffffff8100e6f3>] ? do_IRQ+0x81/0x97 [292670.941639] [<ffffffff81390a6d>] ? common_interrupt+0x6d/0x6d [292670.941666] <EOI> [<ffffffff812a70ec>] ? arch_local_irq_enable+0x4/0x8 [292670.941708] [<ffffffff812a74af>] ? cpuidle_enter_state+0x46/0xb1 [292670.941739] [<ffffffff812a75f0>] ? cpuidle_idle_call+0xd6/0x147 [292670.941770] [<ffffffff81013add>] ? arch_cpu_idle+0x6/0x1a [292670.941802] [<ffffffff810734d3>] ? cpu_startup_entry+0x114/0x191 [292670.941835] [<ffffffff816b3d51>] ? start_kernel+0x3e8/0x3f3 [292670.941864] [<ffffffff816b377f>] ? repair_env_string+0x57/0x57 [292670.941895] [<ffffffff816b359a>] ? x86_64_start_kernel+0xf2/0xfd [292670.941924] ---[ end trace 119aa908393ed9b3 ]--- [292670.941950] ib0: transmit timeout: latency 1356 msecs [292670.941977] ib0: queue stopped 1, tx_head 595756, tx_tail 595756 -- Vasiliy Tolstov, e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org jabber: vase-+9FY0jupvH6HXe+LvDLADg@public.gmane.org -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
[parent not found: <CACaajQuQ+mbwhp4jxUt+9r4Rq=14oBUqC-=xx6J3EGpAfecG-Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: Need help with strange mlx4_core error [not found] ` <CACaajQuQ+mbwhp4jxUt+9r4Rq=14oBUqC-=xx6J3EGpAfecG-Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2014-03-01 12:50 ` Bart Van Assche [not found] ` <5311D799.2090607-HInyCGIudOg@public.gmane.org> 0 siblings, 1 reply; 13+ messages in thread From: Bart Van Assche @ 2014-03-01 12:50 UTC (permalink / raw) To: Vasiliy Tolstov; +Cc: Or Gerlitz, linux-rdma, Jack Morgenstein On 02/28/14 19:35, Vasiliy Tolstov wrote: > [292670.940309] WARNING: at > /tmp/buildd/linux-3.10.11/net/sched/sch_generic.c:255 > dev_watchdog+0xe3/0x153() > [292670.940359] NETDEV WATCHDOG: ib0 (mlx4_core): transmit queue 0 timed out I'm not sure but I think there is a fix in kernel 3.11 for this issue. Bart. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
[parent not found: <5311D799.2090607-HInyCGIudOg@public.gmane.org>]
* Re: Need help with strange mlx4_core error [not found] ` <5311D799.2090607-HInyCGIudOg@public.gmane.org> @ 2014-03-01 16:55 ` Vasiliy Tolstov [not found] ` <CACaajQuPrAFNAR8d4NphuVRH+pVGC-Yv=afS1cbkrg=48M8ROg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 13+ messages in thread From: Vasiliy Tolstov @ 2014-03-01 16:55 UTC (permalink / raw) To: Bart Van Assche; +Cc: Or Gerlitz, linux-rdma, Jack Morgenstein 2014-03-01 16:50 GMT+04:00 Bart Van Assche <bvanassche-HInyCGIudOg@public.gmane.org>: > I'm not sure but I think there is a fix in kernel 3.11 for this issue. Thanks, but as i understand, 3.10 lts and need to get this updates.. But thanks! I'm try to search changelogs for 3.11. -- Vasiliy Tolstov, e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org jabber: vase-+9FY0jupvH6HXe+LvDLADg@public.gmane.org -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
[parent not found: <CACaajQuPrAFNAR8d4NphuVRH+pVGC-Yv=afS1cbkrg=48M8ROg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: Need help with strange mlx4_core error [not found] ` <CACaajQuPrAFNAR8d4NphuVRH+pVGC-Yv=afS1cbkrg=48M8ROg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2014-03-01 17:16 ` Vasiliy Tolstov [not found] ` <CACaajQuXn5+-KQh_XtB3bb-1q9C+5ezAEh26C1TYT=5ncYJvCw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 13+ messages in thread From: Vasiliy Tolstov @ 2014-03-01 17:16 UTC (permalink / raw) To: Vasiliy Tolstov; +Cc: Bart Van Assche, Or Gerlitz, linux-rdma, Jack Morgenstein 2014-03-01 20:55 GMT+04:00 Vasiliy Tolstov <v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org>: > > Thanks, but as i understand, 3.10 lts and need to get this updates.. > But thanks! I'm try to search changelogs for 3.11. Hmm. May me i miss something, but i can't see in changelogs for 3.11 (all) fixes for this error. -- Vasiliy Tolstov, e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org jabber: vase-+9FY0jupvH6HXe+LvDLADg@public.gmane.org -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
[parent not found: <CACaajQuXn5+-KQh_XtB3bb-1q9C+5ezAEh26C1TYT=5ncYJvCw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: Need help with strange mlx4_core error [not found] ` <CACaajQuXn5+-KQh_XtB3bb-1q9C+5ezAEh26C1TYT=5ncYJvCw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2014-03-01 17:23 ` Bart Van Assche [not found] ` <531217AD.3030807-HInyCGIudOg@public.gmane.org> 0 siblings, 1 reply; 13+ messages in thread From: Bart Van Assche @ 2014-03-01 17:23 UTC (permalink / raw) To: Vasiliy Tolstov; +Cc: Or Gerlitz, linux-rdma, Jack Morgenstein On 03/01/14 18:16, Vasiliy Tolstov wrote: > 2014-03-01 20:55 GMT+04:00 Vasiliy Tolstov <v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org>: >> Thanks, but as i understand, 3.10 lts and need to get this updates.. >> But thanks! I'm try to search changelogs for 3.11. > > Hmm. May me i miss something, but i can't see in changelogs for 3.11 > (all) fixes for this error. I have seen this issue myself with the 3.0 mlx4_en driver but do not see it anymore with the latest version of this driver. I do not know which patch fixes this issue. But I have been wondering whether it could have been the following patch: commit 0cc5c8bf11852dec3225fda2f53a599243095d23 net/mlx4_en: Fix a race between napi poll function and RX ring cleanup The RX rings were cleaned while there was still possible RX traffic completion handling. Change the sequence of events so that the port is closed and the QPs are being stopped before RX cleanup. Bart. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
[parent not found: <531217AD.3030807-HInyCGIudOg@public.gmane.org>]
* Re: Need help with strange mlx4_core error [not found] ` <531217AD.3030807-HInyCGIudOg@public.gmane.org> @ 2014-03-01 20:06 ` Vasiliy Tolstov 0 siblings, 0 replies; 13+ messages in thread From: Vasiliy Tolstov @ 2014-03-01 20:06 UTC (permalink / raw) To: Bart Van Assche; +Cc: Or Gerlitz, linux-rdma, Jack Morgenstein 2014-03-01 21:23 GMT+04:00 Bart Van Assche <bvanassche-HInyCGIudOg@public.gmane.org>: > I have seen this issue myself with the 3.0 mlx4_en driver but do not see > it anymore with the latest version of this driver. I do not know which > patch fixes this issue. But I have been wondering whether it could have > been the following patch: > > commit 0cc5c8bf11852dec3225fda2f53a599243095d23 > > net/mlx4_en: Fix a race between napi poll function and RX ring cleanup > > The RX rings were cleaned while there was still possible RX traffic > completion handling. Change the sequence of events so that the port is > closed and the QPs are being stopped before RX cleanup. Thanks! I'm try it. -- Vasiliy Tolstov, e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org jabber: vase-+9FY0jupvH6HXe+LvDLADg@public.gmane.org -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Need help with strange mlx4_core error
@ 2014-06-26 16:59 Mark Lehrer
0 siblings, 0 replies; 13+ messages in thread
From: Mark Lehrer @ 2014-06-26 16:59 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
On Feb 28, 2014, Vasiliy Tolstov wrote:
> I'm update firmware to 2.7.200 (that is the latest from ftp
> supermicro). Now i'm try to test it.
Which type of HCA do you have? The firmware I downloaded from
ftp.supermicro.com was 2.9.1000, and once I upgraded to this version
from 2.7.200 I was able to use mlx4_en.
lspci shows my card as: Mellanox Technologies MT26428 [ConnectX VPI
PCIe 2.0 5GT/s - IB QDR / 10GigE]
Mark
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 13+ messages in threadend of thread, other threads:[~2014-06-26 16:59 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-02-03 3:35 Need help with strange mlx4_core error Vasiliy Tolstov
[not found] ` <CAJZOPZJmdGutUQZ4icBrKRq-YPwD8TqtoLH9SgqNVx_CDjL0TQ@mail.gmail.com>
[not found] ` <CAJZOPZJmdGutUQZ4icBrKRq-YPwD8TqtoLH9SgqNVx_CDjL0TQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-02-03 7:27 ` Vasiliy Tolstov
[not found] ` <CACaajQtHzF9g2kp9MrAaQsvKUQdr10YVa+Np1danmAcR_rvLqg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-02-03 10:54 ` Or Gerlitz
2014-02-03 21:19 ` Vasiliy Tolstov
[not found] ` <CACaajQs9ANqLX7m=TcR2wT6wkiS-M5uVvAnQrvGWPmukZnLqcg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-02-04 14:19 ` Vasiliy Tolstov
[not found] ` <CACaajQuRzBy3oV+-3+NjyE9muGKPN3YQUd5wnkkcoak0GSP=uw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-02-28 18:24 ` Vasiliy Tolstov
[not found] ` <CACaajQtKvihFu6X-+9N3VvVKK4Jw3eADBOENhuM+-4s8s=PsUQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-02-28 18:35 ` Vasiliy Tolstov
[not found] ` <CACaajQuQ+mbwhp4jxUt+9r4Rq=14oBUqC-=xx6J3EGpAfecG-Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-03-01 12:50 ` Bart Van Assche
[not found] ` <5311D799.2090607-HInyCGIudOg@public.gmane.org>
2014-03-01 16:55 ` Vasiliy Tolstov
[not found] ` <CACaajQuPrAFNAR8d4NphuVRH+pVGC-Yv=afS1cbkrg=48M8ROg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-03-01 17:16 ` Vasiliy Tolstov
[not found] ` <CACaajQuXn5+-KQh_XtB3bb-1q9C+5ezAEh26C1TYT=5ncYJvCw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-03-01 17:23 ` Bart Van Assche
[not found] ` <531217AD.3030807-HInyCGIudOg@public.gmane.org>
2014-03-01 20:06 ` Vasiliy Tolstov
-- strict thread matches above, loose matches on Subject: below --
2014-06-26 16:59 Mark Lehrer
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox