From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mike Snitzer Subject: Re: v3.15 dm-mpath regression: cable pull test causes I/O hang Date: Fri, 27 Jun 2014 09:33:45 -0400 Message-ID: <20140627133345.GA6150@redhat.com> References: <53AD6B62.2020407@acm.org> Reply-To: device-mapper development Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: <53AD6B62.2020407@acm.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: Bart Van Assche Cc: Jun'ichi Nomura , device-mapper development List-Id: dm-devel.ids On Fri, Jun 27 2014 at 9:02am -0400, Bart Van Assche wrote: > Hello, > > While running a cable pull simulation test with dm_multipath on top of > the SRP initiator driver I noticed that after a few iterations I/O locks > up instead of dm_multipath processing the path failure properly (see also > below for a call trace). At least kernel versions 3.15 and 3.16-rc2 are > vulnerable. This issue does not occur with kernel 3.14. I have tried to > bisect this but gave up when I noticed that I/O locked up completely with > a kernel built from git commit ID e809917735ebf1b9a56c24e877ce0d320baee2ec > (dm mpath: push back requests instead of queueing). But with the bisect I > have been able to narrow down this issue to one of the patches in "Merge > tag 'dm-3.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/ > device-mapper/linux-dm". Does anyone have a suggestion how to analyze this > further or how to fix this ? > > Thanks, > > Bart. > > systemd-udevd D ffff880831b58000 0 9926 356 0x00000006 > ffff8807f8bb79b8 0000000000000002 ffff880831b58000 ffff8807f8bb7fd8 > 00000000000131c0 00000000000131c0 ffff88083b490000 ffff88085fc53ad0 > ffff88085ff6bf38 ffff8807f8bb7a40 0000000000000002 ffffffff81135bd0 > Call Trace: > [] io_schedule+0x9d/0x130 > [] sleep_on_page+0xe/0x20 > [] __wait_on_bit_lock+0x48/0xb0 > [] __lock_page+0x6a/0x70 > [] truncate_inode_pages_range+0x3ff/0x690 > [] truncate_inode_pages+0x15/0x20 > [] kill_bdev+0x35/0x40 > [] __blkdev_put+0x69/0x1b0 > [] blkdev_put+0x50/0x160 > [] blkdev_close+0x25/0x30 > [] __fput+0xea/0x1f0 > [] ____fput+0xe/0x10 > [] task_work_run+0xac/0xe0 > [] do_exit+0x2c7/0xc60 > [] do_group_exit+0x4c/0xc0 > [] get_signal_to_deliver+0x2e1/0x940 > [] do_signal+0x48/0x630 > [] do_notify_resume+0x71/0xc0 > [] int_signal+0x12/0x17 (we've seen sync on last close cause problems when the block device isn't reachable). Any other threads that look suspect in output from?: echo t > /proc/sysrq-trigger Can you provide your dmsetup table output for the relevant mpath device? Are you using queue_if_no_path? Also, AFAIK you don't use multipath-tools, but if by some chance you do please provide your multipath.conf. I'll attempt to reproduce. But I'm almost tempted to just revert _all_ of 3.15's dm-mpath changes, and only reintroduce them once they can pass your testing. I'd like to avoid that, so Hannes and/or Jun'ichi, it is time for looking at this seriously.. any help would be very appreciated. For starters, have you guys done cable pull tests with > 3.15 ?