* Re: 4.5-rc1 multipath regression
[not found] <56AAA6AE.1060802@sandisk.com>
@ 2016-01-29 18:42 ` Bart Van Assche
2016-01-30 0:06 ` Mike Snitzer
0 siblings, 1 reply; 4+ messages in thread
From: Bart Van Assche @ 2016-01-29 18:42 UTC (permalink / raw)
To: device-mapper development
On 01/28/2016 03:39 PM, Bart Van Assche wrote:
> There is a regression in the 4.5-rc1 kernel with regard to multipath
> setup. On my SRP I usually use for these tests after a few minutes a
> kernel crash occurs and the console freezes. A screenshot has been attached.
(replying to my own e-mail)
Before that crash was triggered, the following appeared in the system log, a
message I had not yet seen during tests with previous kernel versions:
INFO: task multipathd:3889 blocked for more than 120 seconds.
Tainted: G E 4.5.0-rc1+ #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
multipathd D ffff88045266b828 0 3889 1 0x00000000
ffff88045266b828 ffffffff81a0f500 ffff8804254d9f80 ffffffff82145850
ffffffff810a518d ffffffff82132890 ffff8804254dabc8 ffff8804254d9f80
0000000000000002 ffff880400000001 0000000000000002 ffff880400000000
Call Trace:
[<ffffffff8152ea42>] schedule+0x42/0xb0
[<ffffffff81532c9f>] schedule_timeout+0x17f/0x1d0
[<ffffffff8152fae5>] wait_for_completion+0xe5/0x120
[<ffffffff81070736>] flush_workqueue+0x256/0x6e0
[<ffffffffa0352804>] flush_multipath_work+0x34/0x80 [dm_multipath]
[<ffffffffa035287f>] multipath_postsuspend+0x2f/0x50 [dm_multipath]
[<ffffffffa000580b>] suspend_targets+0x3b/0x90 [dm_mod]
[<ffffffffa00058b3>] dm_table_postsuspend_targets+0x13/0x20 [dm_mod]
[<ffffffffa00042f4>] dm_suspend+0xf4/0x130 [dm_mod]
[<ffffffffa0009d5f>] do_resume+0x15f/0x1d0 [dm_mod]
[<ffffffffa0009e5a>] dev_suspend+0x8a/0xc0 [dm_mod]
[<ffffffffa000ad3c>] ctl_ioctl+0x18c/0x280 [dm_mod]
[<ffffffffa000ae3e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
[<ffffffff81196903>] vfs_ioctl+0x13/0x30
[<ffffffff811972e2>] do_vfs_ioctl+0x82/0x420
[<ffffffff811976ce>] SyS_ioctl+0x4e/0x80
[<ffffffff81534197>] entry_SYSCALL_64_fastpath+0x12/0x6f
2 locks held by multipathd/3889:
#0: (&md->suspend_lock/1){+.+...}, at: [<ffffffffa0004232>] dm_suspend+0x32/0x130 [dm_mod]
#1: (&m->work_mutex){+.+.+.}, at: [<ffffffffa0352877>] multipath_postsuspend+0x27/0x50 [dm_multipath]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: 4.5-rc1 multipath regression
2016-01-29 18:42 ` 4.5-rc1 multipath regression Bart Van Assche
@ 2016-01-30 0:06 ` Mike Snitzer
2016-02-08 18:16 ` Bart Van Assche
0 siblings, 1 reply; 4+ messages in thread
From: Mike Snitzer @ 2016-01-30 0:06 UTC (permalink / raw)
To: Bart Van Assche; +Cc: device-mapper development
On Fri, Jan 29 2016 at 1:42pm -0500,
Bart Van Assche <bart.vanassche@sandisk.com> wrote:
> On 01/28/2016 03:39 PM, Bart Van Assche wrote:
> > There is a regression in the 4.5-rc1 kernel with regard to multipath
> > setup. On my SRP I usually use for these tests after a few minutes a
> > kernel crash occurs and the console freezes. A screenshot has been attached.
>
> (replying to my own e-mail)
Not sure where you sent your first email.. not seeing it on dm-devel
archives.
So I don't have the original screenshot you attached.
The 4.5 merge window didn't see any changes to DM mpath or DM core. So
any regression is very likely outside DM and rooted in SRP or whatever
other dependencies your setup relies on.
Mike
> Before that crash was triggered, the following appeared in the system log, a
> message I had not yet seen during tests with previous kernel versions:
>
> INFO: task multipathd:3889 blocked for more than 120 seconds.
> Tainted: G E 4.5.0-rc1+ #2
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> multipathd D ffff88045266b828 0 3889 1 0x00000000
> ffff88045266b828 ffffffff81a0f500 ffff8804254d9f80 ffffffff82145850
> ffffffff810a518d ffffffff82132890 ffff8804254dabc8 ffff8804254d9f80
> 0000000000000002 ffff880400000001 0000000000000002 ffff880400000000
> Call Trace:
> [<ffffffff8152ea42>] schedule+0x42/0xb0
> [<ffffffff81532c9f>] schedule_timeout+0x17f/0x1d0
> [<ffffffff8152fae5>] wait_for_completion+0xe5/0x120
> [<ffffffff81070736>] flush_workqueue+0x256/0x6e0
> [<ffffffffa0352804>] flush_multipath_work+0x34/0x80 [dm_multipath]
> [<ffffffffa035287f>] multipath_postsuspend+0x2f/0x50 [dm_multipath]
> [<ffffffffa000580b>] suspend_targets+0x3b/0x90 [dm_mod]
> [<ffffffffa00058b3>] dm_table_postsuspend_targets+0x13/0x20 [dm_mod]
> [<ffffffffa00042f4>] dm_suspend+0xf4/0x130 [dm_mod]
> [<ffffffffa0009d5f>] do_resume+0x15f/0x1d0 [dm_mod]
> [<ffffffffa0009e5a>] dev_suspend+0x8a/0xc0 [dm_mod]
> [<ffffffffa000ad3c>] ctl_ioctl+0x18c/0x280 [dm_mod]
> [<ffffffffa000ae3e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
> [<ffffffff81196903>] vfs_ioctl+0x13/0x30
> [<ffffffff811972e2>] do_vfs_ioctl+0x82/0x420
> [<ffffffff811976ce>] SyS_ioctl+0x4e/0x80
> [<ffffffff81534197>] entry_SYSCALL_64_fastpath+0x12/0x6f
> 2 locks held by multipathd/3889:
> #0: (&md->suspend_lock/1){+.+...}, at: [<ffffffffa0004232>] dm_suspend+0x32/0x130 [dm_mod]
> #1: (&m->work_mutex){+.+.+.}, at: [<ffffffffa0352877>] multipath_postsuspend+0x27/0x50 [dm_multipath]
>
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: 4.5-rc1 multipath regression
2016-01-30 0:06 ` Mike Snitzer
@ 2016-02-08 18:16 ` Bart Van Assche
2016-02-09 16:40 ` Mike Snitzer
0 siblings, 1 reply; 4+ messages in thread
From: Bart Van Assche @ 2016-02-08 18:16 UTC (permalink / raw)
To: Mike Snitzer; +Cc: device-mapper development
On 01/29/2016 04:07 PM, Mike Snitzer wrote:
> On Fri, Jan 29 2016 at 1:42pm -0500,
> Bart Van Assche <bart.vanassche@sandisk.com> wrote:
>> On 01/28/2016 03:39 PM, Bart Van Assche wrote:
>>> There is a regression in the 4.5-rc1 kernel with regard to multipath
>>> setup. On my SRP I usually use for these tests after a few minutes a
>>> kernel crash occurs and the console freezes. A screenshot has been attached.
>>
>> (replying to my own e-mail)
>
> Not sure where you sent your first email.. not seeing it on dm-devel
> archives.
>
> So I don't have the original screenshot you attached.
>
> The 4.5 merge window didn't see any changes to DM mpath or DM core. So
> any regression is very likely outside DM and rooted in SRP or whatever
> other dependencies your setup relies on.
Hello Mike,
The behavior I see with kernel v4.5-rc3 is different of what I saw with
v4.5-rc1 but it still is not the behavior I expect. The call trace that
was triggered this morning on my test setup can be found below. I assume
the information below means that the tio->ti->type is NULL in dm_done() ?
Bart.
BUG: unable to handle kernel NULL pointer dereference at 0000000000000060
IP: [<ffffffffa00020e5>] dm_done+0x35/0x1b0 [dm_mod]
PGD 456993067 PUD 40c76a067 PMD 0
Oops: 0000 [#1] SMP
Modules linked in: scsi_dh_alua dm_queue_length netconsole autofs4 ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm configfs ib_cm iw_cm dm_round_robin dm_multipath iTCO_wdt iTCO_vendor_support ipmi_devintf dcdbas ipmi_si ipmi_msghandler sb_edac edac_core lpc_ich mfd_core tg3 libphy ptp pps_core sg wmi ext4(E) jbd2(E) mbcache(E) sr_mod(E) cdrom(E) sd_mod(E) ahci(E) libahci(E) mlx4_ib(E) ib_sa(E) ib_mad(E) ib_core(E) ib_addr(E) ipv6(E) mlx4_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
CPU: 0 PID: 618 Comm: kworker/0:1H Tainted: G E 4.5.0-rc3+ #3
Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
Workqueue: kblockd blk_mq_run_work_fn
task: ffff880437fa5e80 ti: ffff880437a6c000 task.ti: ffff880437a6c000
RIP: 0010:[<ffffffffa00020e5>] [<ffffffffa00020e5>] dm_done+0x35/0x1b0 [dm_mod]
RSP: 0018:ffff88046e403e38 EFLAGS: 00010202
RAX: 0000000000000000 RBX: ffff8803f6a98d70 RCX: dead000000000200
RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffc9000933c040
sd 23:0:0:1: Asymmetric access state changed
device-mapper: multipath: Failing path 67:176.
device-mapper: multipath: Failing path 68:16.
sd 24:0:0:1: Asymmetric access state changed
RBP: ffff88046e403e78 R08: ffff8803f6a98c78 R09: 0000000000000001
R10: 0000000000000000 R11: 0000000000000000 R12: ffff88006c0f2680
R13: ffff8803f6a98c00 R14: ffff88046e403ec8 R15: 0000000000000005
FS: 0000000000000000(0000) GS:ffff88046e400000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000060 CR3: 000000041defd000 CR4: 00000000001406f0
Stack:
0000000000000003 0000000000000002 ffff88046e403e78 ffff8803f6a98d70
ffff8803f6a98c00 ffff8803f6a98c00 ffff88046e403ec8 0000000000000005
ffff88046e403ea8 ffffffffa00022ac ffffffff81a090e0 ffff8803f6a98c78
Call Trace:
<IRQ>
[<ffffffffa00022ac>] dm_softirq_done+0x4c/0xd0 [dm_mod]
[<ffffffff812476ac>] blk_done_softirq+0x8c/0xb0
[<ffffffff8105be66>] __do_softirq+0xf6/0x240
[<ffffffff8105c0bc>] irq_exit+0xac/0xc0
[<ffffffff8103afde>] smp_call_function_single_interrupt+0x2e/0x40
[<ffffffff81535779>] call_function_single_interrupt+0x89/0x90
<EOI>
[<ffffffff8153422d>] ? _raw_spin_unlock_irqrestore+0x3d/0x60
[<ffffffffa03515bc>] multipath_busy+0xcc/0xf0 [dm_multipath]
[<ffffffffa00045bd>] dm_mq_queue_rq+0x7d/0x180 [dm_mod]
[<ffffffff81249cdb>] __blk_mq_run_hw_queue+0x29b/0x490
[<ffffffff810a5fd3>] ? __lock_acquire+0x3b3/0x560
[<ffffffff81249f10>] blk_mq_run_work_fn+0x10/0x20
[<ffffffff810723ea>] process_one_work+0x1da/0x480
[<ffffffff8107237a>] ? process_one_work+0x16a/0x480
[<ffffffff810a62c4>] ? __lock_release+0xc4/0x3a0
[<ffffffff81072f39>] worker_thread+0x169/0x520
[<ffffffff81099d58>] ? complete+0x48/0x60
[<ffffffff8153422b>] ? _raw_spin_unlock_irqrestore+0x3b/0x60
[<ffffffff81072dd0>] ? maybe_create_worker+0x110/0x110
[<ffffffff81072dd0>] ? maybe_create_worker+0x110/0x110
[<ffffffff8152ee92>] ? schedule+0x42/0xb0
[<ffffffff81072dd0>] ? maybe_create_worker+0x110/0x110
[<ffffffff81078f94>] kthread+0xe4/0x100
[<ffffffff810a4dcd>] ? trace_hardirqs_on+0xd/0x10
[<ffffffff81081c99>] ? schedule_tail+0x19/0xd0
[<ffffffff81078eb0>] ? __init_kthread_worker+0x70/0x70
[<ffffffff8153497f>] ret_from_fork+0x3f/0x70
[<ffffffff81078eb0>] ? __init_kthread_worker+0x70/0x70
Code: 65 e0 48 89 5d d8 49 89 fc 4c 89 6d e8 4c 89 75 f0 4c 89 7d f8 48 8b 9f 60 01 00 00 48 8b 7b 08 48 85 ff 74 0c 48 8b 47 08 84 d2 <4c> 8b 40 60 75 44 41 89 f5 41 83 fd 87 0f 84 f2 00 00 00 45 85
RIP [<ffffffffa00020e5>] dm_done+0x35/0x1b0 [dm_mod]
RSP <ffff88046e403e38>
CR2: 0000000000000060
---[ end trace f47c39416952f73a ]---
sd 31:0:0:1: Asymmetric access state changed
Kernel panic - not syncing: Fatal exception in interrupt
Kernel Offset: disabled
---[ end Kernel panic - not syncing: Fatal exception in interrupt
$ gdb drivers/md/dm-mod.o
(gdb) list *(dm_done+0x35)
0x20e5 is in dm_done (drivers/md/dm.c:1273).
1268 int r = error;
1269 struct dm_rq_target_io *tio = clone->end_io_data;
1270 dm_request_endio_fn rq_end_io = NULL;
1271
1272 if (tio->ti) {
1273 rq_end_io = tio->ti->type->rq_end_io;
1274
1275 if (mapped && rq_end_io)
1276 r = rq_end_io(tio->ti, clone, error, &tio->info);
1277 }
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: 4.5-rc1 multipath regression
2016-02-08 18:16 ` Bart Van Assche
@ 2016-02-09 16:40 ` Mike Snitzer
0 siblings, 0 replies; 4+ messages in thread
From: Mike Snitzer @ 2016-02-09 16:40 UTC (permalink / raw)
To: Bart Van Assche; +Cc: device-mapper development
On Mon, Feb 08 2016 at 1:16pm -0500,
Bart Van Assche <bart.vanassche@sandisk.com> wrote:
> On 01/29/2016 04:07 PM, Mike Snitzer wrote:
> > On Fri, Jan 29 2016 at 1:42pm -0500,
> > Bart Van Assche <bart.vanassche@sandisk.com> wrote:
> >> On 01/28/2016 03:39 PM, Bart Van Assche wrote:
> >>> There is a regression in the 4.5-rc1 kernel with regard to multipath
> >>> setup. On my SRP I usually use for these tests after a few minutes a
> >>> kernel crash occurs and the console freezes. A screenshot has been attached.
> >>
> >> (replying to my own e-mail)
> >
> > Not sure where you sent your first email.. not seeing it on dm-devel
> > archives.
> >
> > So I don't have the original screenshot you attached.
> >
> > The 4.5 merge window didn't see any changes to DM mpath or DM core. So
> > any regression is very likely outside DM and rooted in SRP or whatever
> > other dependencies your setup relies on.
>
> Hello Mike,
>
> The behavior I see with kernel v4.5-rc3 is different of what I saw with
> v4.5-rc1 but it still is not the behavior I expect. The call trace that
> was triggered this morning on my test setup can be found below. I assume
> the information below means that the tio->ti->type is NULL in dm_done() ?
Yes, looks like it:
crash> struct -o target_type
struct target_type {
[0x0] uint64_t features;
[0x8] const char *name;
[0x10] struct module *module;
[0x18] unsigned int version[3];
[0x28] dm_ctr_fn ctr;
[0x30] dm_dtr_fn dtr;
[0x38] dm_map_fn map;
[0x40] dm_map_request_fn map_rq;
[0x48] dm_clone_and_map_request_fn clone_and_map_rq;
[0x50] dm_release_clone_request_fn release_clone_rq;
[0x58] dm_endio_fn end_io;
[0x60] dm_request_endio_fn rq_end_io;
...
Not aware of any use-after-free issues in request-based DM. But this
report clearly speaks to one. If you're using blk-mq then the tio is
part of the pdu (so that explains why dereferencing tio isn't a
problem). But somehow tio->ti is being reset to NULL early (init_tio
does reset it, but not until a new request comes in via dm_mq_queue_rq).
Anyway, certainly strange.
> BUG: unable to handle kernel NULL pointer dereference at 0000000000000060
> IP: [<ffffffffa00020e5>] dm_done+0x35/0x1b0 [dm_mod]
> PGD 456993067 PUD 40c76a067 PMD 0
> Oops: 0000 [#1] SMP
> Modules linked in: scsi_dh_alua dm_queue_length netconsole autofs4 ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm configfs ib_cm iw_cm dm_round_robin dm_multipath iTCO_wdt iTCO_vendor_support ipmi_devintf dcdbas ipmi_si ipmi_msghandler sb_edac edac_core lpc_ich mfd_core tg3 libphy ptp pps_core sg wmi ext4(E) jbd2(E) mbcache(E) sr_mod(E) cdrom(E) sd_mod(E) ahci(E) libahci(E) mlx4_ib(E) ib_sa(E) ib_mad(E) ib_core(E) ib_addr(E) ipv6(E) mlx4_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
> CPU: 0 PID: 618 Comm: kworker/0:1H Tainted: G E 4.5.0-rc3+ #3
> Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
> Workqueue: kblockd blk_mq_run_work_fn
> task: ffff880437fa5e80 ti: ffff880437a6c000 task.ti: ffff880437a6c000
> RIP: 0010:[<ffffffffa00020e5>] [<ffffffffa00020e5>] dm_done+0x35/0x1b0 [dm_mod]
> RSP: 0018:ffff88046e403e38 EFLAGS: 00010202
> RAX: 0000000000000000 RBX: ffff8803f6a98d70 RCX: dead000000000200
> RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffc9000933c040
> sd 23:0:0:1: Asymmetric access state changed
> device-mapper: multipath: Failing path 67:176.
> device-mapper: multipath: Failing path 68:16.
> sd 24:0:0:1: Asymmetric access state changed
> RBP: ffff88046e403e78 R08: ffff8803f6a98c78 R09: 0000000000000001
> R10: 0000000000000000 R11: 0000000000000000 R12: ffff88006c0f2680
> R13: ffff8803f6a98c00 R14: ffff88046e403ec8 R15: 0000000000000005
> FS: 0000000000000000(0000) GS:ffff88046e400000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000000000060 CR3: 000000041defd000 CR4: 00000000001406f0
> Stack:
> 0000000000000003 0000000000000002 ffff88046e403e78 ffff8803f6a98d70
> ffff8803f6a98c00 ffff8803f6a98c00 ffff88046e403ec8 0000000000000005
> ffff88046e403ea8 ffffffffa00022ac ffffffff81a090e0 ffff8803f6a98c78
> Call Trace:
> <IRQ>
> [<ffffffffa00022ac>] dm_softirq_done+0x4c/0xd0 [dm_mod]
> [<ffffffff812476ac>] blk_done_softirq+0x8c/0xb0
> [<ffffffff8105be66>] __do_softirq+0xf6/0x240
> [<ffffffff8105c0bc>] irq_exit+0xac/0xc0
> [<ffffffff8103afde>] smp_call_function_single_interrupt+0x2e/0x40
> [<ffffffff81535779>] call_function_single_interrupt+0x89/0x90
> <EOI>
> [<ffffffff8153422d>] ? _raw_spin_unlock_irqrestore+0x3d/0x60
> [<ffffffffa03515bc>] multipath_busy+0xcc/0xf0 [dm_multipath]
> [<ffffffffa00045bd>] dm_mq_queue_rq+0x7d/0x180 [dm_mod]
> [<ffffffff81249cdb>] __blk_mq_run_hw_queue+0x29b/0x490
> [<ffffffff810a5fd3>] ? __lock_acquire+0x3b3/0x560
> [<ffffffff81249f10>] blk_mq_run_work_fn+0x10/0x20
> [<ffffffff810723ea>] process_one_work+0x1da/0x480
> [<ffffffff8107237a>] ? process_one_work+0x16a/0x480
> [<ffffffff810a62c4>] ? __lock_release+0xc4/0x3a0
> [<ffffffff81072f39>] worker_thread+0x169/0x520
> [<ffffffff81099d58>] ? complete+0x48/0x60
> [<ffffffff8153422b>] ? _raw_spin_unlock_irqrestore+0x3b/0x60
> [<ffffffff81072dd0>] ? maybe_create_worker+0x110/0x110
> [<ffffffff81072dd0>] ? maybe_create_worker+0x110/0x110
> [<ffffffff8152ee92>] ? schedule+0x42/0xb0
> [<ffffffff81072dd0>] ? maybe_create_worker+0x110/0x110
> [<ffffffff81078f94>] kthread+0xe4/0x100
> [<ffffffff810a4dcd>] ? trace_hardirqs_on+0xd/0x10
> [<ffffffff81081c99>] ? schedule_tail+0x19/0xd0
> [<ffffffff81078eb0>] ? __init_kthread_worker+0x70/0x70
> [<ffffffff8153497f>] ret_from_fork+0x3f/0x70
> [<ffffffff81078eb0>] ? __init_kthread_worker+0x70/0x70
> Code: 65 e0 48 89 5d d8 49 89 fc 4c 89 6d e8 4c 89 75 f0 4c 89 7d f8 48 8b 9f 60 01 00 00 48 8b 7b 08 48 85 ff 74 0c 48 8b 47 08 84 d2 <4c> 8b 40 60 75 44 41 89 f5 41 83 fd 87 0f 84 f2 00 00 00 45 85
> RIP [<ffffffffa00020e5>] dm_done+0x35/0x1b0 [dm_mod]
> RSP <ffff88046e403e38>
> CR2: 0000000000000060
> ---[ end trace f47c39416952f73a ]---
> sd 31:0:0:1: Asymmetric access state changed
> Kernel panic - not syncing: Fatal exception in interrupt
> Kernel Offset: disabled
> ---[ end Kernel panic - not syncing: Fatal exception in interrupt
>
>
> $ gdb drivers/md/dm-mod.o
> (gdb) list *(dm_done+0x35)
> 0x20e5 is in dm_done (drivers/md/dm.c:1273).
> 1268 int r = error;
> 1269 struct dm_rq_target_io *tio = clone->end_io_data;
> 1270 dm_request_endio_fn rq_end_io = NULL;
> 1271
> 1272 if (tio->ti) {
> 1273 rq_end_io = tio->ti->type->rq_end_io;
> 1274
> 1275 if (mapped && rq_end_io)
> 1276 r = rq_end_io(tio->ti, clone, error, &tio->info);
> 1277 }
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2016-02-09 16:40 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <56AAA6AE.1060802@sandisk.com>
2016-01-29 18:42 ` 4.5-rc1 multipath regression Bart Van Assche
2016-01-30 0:06 ` Mike Snitzer
2016-02-08 18:16 ` Bart Van Assche
2016-02-09 16:40 ` Mike Snitzer
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).