From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ryan Harper Subject: Re: [REGRESSION][BISECTED] virtio-blk serial attribute causes guest to hang [Was: Re: [PATCH UPDATED 4/5] dm: implement REQ_FLUSH/FUA support for request-based dm] Date: Thu, 9 Sep 2010 10:44:42 -0500 Message-ID: <20100909154442.GI30086@us.ibm.com> References: <20100902032246.GA31484@redhat.com> <20100909152658.GA8118@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Tejun Heo , Mikulas Patocka , dm-devel@redhat.com, Vivek Goyal , ryanh@us.ibm.com, john.cooper@redhat.com, rusty@rustcorp.com.au, hch@infradead.org, kvm@vger.kernel.org To: Mike Snitzer Return-path: Received: from e8.ny.us.ibm.com ([32.97.182.138]:54417 "EHLO e8.ny.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752587Ab0IIPou (ORCPT ); Thu, 9 Sep 2010 11:44:50 -0400 Received: from d01relay06.pok.ibm.com (d01relay06.pok.ibm.com [9.56.227.116]) by e8.ny.us.ibm.com (8.14.4/8.13.1) with ESMTP id o89FQOXM015018 for ; Thu, 9 Sep 2010 11:26:24 -0400 Received: from d01av02.pok.ibm.com (d01av02.pok.ibm.com [9.56.224.216]) by d01relay06.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id o89FinxH1937594 for ; Thu, 9 Sep 2010 11:44:49 -0400 Received: from d01av02.pok.ibm.com (loopback [127.0.0.1]) by d01av02.pok.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id o89FilYH020871 for ; Thu, 9 Sep 2010 12:44:48 -0300 Content-Disposition: inline In-Reply-To: <20100909152658.GA8118@redhat.com> Sender: kvm-owner@vger.kernel.org List-ID: * Mike Snitzer [2010-09-09 10:29]: > On Wed, Sep 01 2010 at 11:22pm -0400, > Mike Snitzer wrote: > > > On Wed, Sep 01 2010 at 2:59pm -0400, > > Mike Snitzer wrote: > > > > > My hope was that the request-based deadlock I'm seeing would disappear > > > if that relaxed ordering patch wasn't applied. Unfortunately, I still > > > see the hang. > > > > Turns out I can reproduce the hang on a stock 2.6.36-rc3 (without _any_ > > FLUSH+FUA patches)! > > > > I'll try to pin-point the root cause but I think my test is somehow > > exposing a bug in my virt setup. > > [my virt setup == single kvm guest (RHEL6) with F13 host] What's your kvm guest command line? And the guest is using stock RHEL6 kernel? What KVM userspace are you using on the host? What comes with F13 or some updated version? > > My gut turned out to be correct. I finally tracked down the regression > point to the following commit (cc'ing appropriate people): > > commit a5eb9e4ff18a33e43557d44b205f953b0c1efade > Author: Ryan Harper > Date: Wed Jun 23 22:19:57 2010 -0500 > > virtio_blk: Add 'serial' attribute to virtio-blk devices (v2) > > Create a new attribute for virtio-blk devices that will fetch the serial number > of the block device. This attribute can be used by udev to create disk/by-id > symlinks for devices that don't have a UUID (filesystem) associated with them. > > ATA_IDENTIFY strings are special in that they can be up to 20 chars long > and aren't required to be nul-terminated. The buffer is also zero-padded > meaning that if the serial is 19 chars or less that we get a nul-terminated > string. When copying this value into a string buffer, we must be careful to > copy up to the nul (if it present) and only 20 if it is longer and not to > attempt to nul terminate; this isn't needed. > > Changes since v1: > - Added BUILD_BUG_ON() for PAGE_SIZE check > - Removed min() since BUILD_BUG_ON() handles the check > - Replaced serial_sysfs() by copying id directly to buffer > > Signed-off-by: Ryan Harper > Signed-off-by: john cooper > Signed-off-by: Rusty Russell > > So the first released kernel to have this regression is 2.6.36-rc1. > > Some background: > I have been working with Tejun to test the barrier to FLUSH+FUA > conversion patchset. I crafted the attached script to test the DM > changes that are part of the FLUSH+FUA patchset. > > Using this script with: > while true ; do ./test_dm_discard_mpath_scsi_debug.sh ; done > > I can reliably trigger the following hang, always on the 5th iteration > in my testing, IFF commit a5eb9e4ff18a33e43557d44b205f953b0c1efade is > applied: > > INFO: task lvcreate:2484 blocked for more than 120 seconds. > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this > message. > lvcreate D 0000000100064871 4960 2484 2350 0x00000080 > ffff88007b87b978 0000000000000046 ffff88007b87b8e8 ffff880000000000 > ffff88007b87bfd8 ffff8800724fa400 00000000001d4040 ffff88007b87bfd8 > 00000000001d4040 00000000001d4040 00000000001d4040 00000000001d4040 > Call Trace: > [] io_schedule+0x73/0xb5 > [] get_request_wait+0xf2/0x180 > [] ? autoremove_wake_function+0x0/0x39 > [] __make_request+0x310/0x434 > [] generic_make_request+0x2f1/0x36e > [] ? cpu_clock+0x43/0x5e > [] submit_bio+0xde/0xfb > [] ? trace_hardirqs_on+0xd/0xf > [] dio_bio_submit+0x7b/0x9c > [] dio_send_cur_page+0x4a/0xb0 > [] __blockdev_direct_IO_newtrunc+0x7c5/0x97d > [] blkdev_direct_IO+0x57/0x59 > [] ? blkdev_get_blocks+0x0/0x90 > [] generic_file_aio_read+0xed/0x5b4 > [] ? might_fault+0x5c/0xac > [] ? pvclock_clocksource_read+0x50/0xb9 > [] do_sync_read+0xcb/0x108 > [] ? __mutex_unlock_slowpath+0x119/0x12b > [] ? trace_hardirqs_on_caller+0x11d/0x141 > [] ? trace_hardirqs_on+0xd/0xf > [] ? security_file_permission+0x16/0x18 > [] vfs_read+0xab/0x108 > [] ? trace_hardirqs_on_caller+0x11d/0x141 > [] sys_read+0x4a/0x6e > [] system_call_fastpath+0x16/0x1b > no locks held by lvcreate/2484. > > > lvcreate is just the first victim (sometimes it is the vgcreate). But > if the guest is left running other new processes get hung with > comparable traces (w/ get_request_wait). Until eventually the guest is > completely unresponsive. > > Mike -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx ryanh@us.ibm.com