From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ozlabs.org (ozlabs.org [IPv6:2401:3900:2:1::2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 009501A001D for ; Tue, 1 Jul 2014 07:30:13 +1000 (EST) Date: Tue, 1 Jul 2014 07:30:08 +1000 From: Paul Mackerras To: Hannes Reinecke Subject: Re: Regression in 3.15 on POWER8 with multipath SCSI Message-ID: <20140630213008.GC20819@iris.ozlabs.ibm.com> References: <20140630103058.GA17747@iris.ozlabs.ibm.com> <53B1416D.8020305@suse.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii In-Reply-To: <53B1416D.8020305@suse.de> Cc: Vladimir Davydov , Kay Sievers , linux-kernel@vger.kernel.org, linuxppc-dev@ozlabs.org, dm-devel@redhat.com, Andrew Morton , Linus Torvalds List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , On Mon, Jun 30, 2014 at 12:52:29PM +0200, Hannes Reinecke wrote: > On 06/30/2014 12:30 PM, Paul Mackerras wrote: > >I have a machine on which 3.15 usually fails to boot, and 3.14 boots > >every time. The machine is a POWER8 2-socket server with 20 cores > >(thus 160 CPUs), 128GB of RAM, and 7 SCSI disks connected via a > >hardware-RAID-capable adapter which appears as two IPR controllers > >which are both connected to each disk. I am booting from a disk that > >has Fedora 20 installed on it. > > > >After over two weeks of bisections, I can finally point to the commits > >that cause the problems. The culprits are: > > > >3e9f1be1 dm mpath: remove process_queued_ios() > >e8099177 dm mpath: push back requests instead of queueing > >bcccff93 kobject: don't block for each kobject_uevent > > > >The interesting thing is that neither e8099177 nor bcccff93 cause > >failures on their own, but with both commits in there are failures > >where the system will fail to find /home on some occasions. > > > >With 3e9f1be1 included, the system appears to be prone to a deadlock > >condition which typically causes the boot process to hang with this > >message showing: > > > >A start job is running for Monitoring of LVM2 mirror...rogress polling > > > >(with a [*** ] thing before it where the asterisks move back and > >forth). > > > >If I revert 63d832c3 ("dm mpath: really fix lockdep warning") , > >4cdd2ad7 ("dm mpath: fix lock order inconsistency in > >multipath_ioctl"), 3e9f1be1 and bcccff93, in that order, I get a > >kernel that will boot every time. The first two are later commits > >that fix some problems with 3e9f1be1 (though not the problems I am > >seeing). > > > >Can anyone see any reason why e8099177 and bcccff93 would interfere > >with each other? > > > It might be running afoul with the 'cookie' mechanism. > Device-mapper is using inserting a 'cookie' with the ioctl, and listens to > any event containing the cookie to ensure udev has finished processing that > device and hence the device node is accessible. Added to this is the problem > that we don't have any good means of detecting any changes to device-mapper > devices. How does that relate to e8099177? Did e8099177 introduce this cookie mechanism? If not, what is it about e8099177 that makes the async processing problematic? Paul.