From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mike Snitzer Subject: Re: x86 VM Boot hang with latest linux-next Date: Mon, 4 Mar 2019 23:07:15 -0500 Message-ID: <20190305040714.GB21739@redhat.com> References: <20190303034847.GA8699@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: Alexander Duyck Cc: dm-devel@redhat.com, linux-next@vger.kernel.org, LKML List-Id: dm-devel.ids T24gU3VuLCBNYXIgMDMgMjAxOSBhdCAxMjowNnBtIC0wNTAwLApBbGV4YW5kZXIgRHV5Y2sgPGFs ZXhhbmRlci5kdXlja0BnbWFpbC5jb20+IHdyb3RlOgoKPiBPbiBTYXQsIE1hciAyLCAyMDE5IGF0 IDc6NDggUE0gTWlrZSBTbml0emVyIDxzbml0emVyQHJlZGhhdC5jb20+IHdyb3RlOgo+ID4KPiA+ IE9uIFNhdCwgTWFyIDAyIDIwMTkgYXQgIDY6MzRwbSAtMDUwMCwKPiA+IEFsZXhhbmRlciBEdXlj ayA8YWxleGFuZGVyLmR1eWNrQGdtYWlsLmNvbT4gd3JvdGU6Cj4gPgo+ID4gPiBTbyBJIGhhdmUg YmVlbiBzZWVpbmcgYW4gaXNzdWUgd2l0aCBhbiBpbnRlcm1pdHRlbnQgYm9vdCBoYW5nIG9uIG15 Cj4gPiA+IHg4NiBLVk0gVk0gd2l0aCB0aGUgbGF0ZXN0IGxpbnV4LW5leHQgYW5kIGhhdmUgYmlz ZWN0ZWQgaXQgZG93biB0byB0aGUKPiA+ID4gZm9sbG93aW5nIGNvbW1pdDoKPiA+ID4gMWVmYTNi Yjc5ZDNkZThjYTFiN2Y2NzcwMzEzYTFmYzBiZWJlMjVjNyBpcyB0aGUgZmlyc3QgYmFkIGNvbW1p dAo+ID4gPiBjb21taXQgMWVmYTNiYjc5ZDNkZThjYTFiN2Y2NzcwMzEzYTFmYzBiZWJlMjVjNwo+ ID4gPiBBdXRob3I6IE1pa2UgU25pdHplciA8c25pdHplckByZWRoYXQuY29tPgo+ID4gPiBEYXRl OiAgIEZyaSBGZWIgMjIgMTE6MjM6MDEgMjAxOSAtMDUwMAo+ID4gPgo+ID4gPiAgICAgZG06IG11 c3QgYWxsb2NhdGUgZG1fbm9jbG9uZSBmb3Igc3RhY2tlZCBub2Nsb25lIGRldmljZXMKPiA+ID4K PiA+ID4gICAgIE90aGVyd2lzZSB2YXJpb3VzIGx2bTIgdGVzdHN1aXRlIHRlc3RzIGZhaWwgYmVj YXVzZSB0aGUgbG93ZXIgbGF5ZXJzIG9mCj4gPiA+ICAgICB0aGUgc3RhY2tlZCBub2Nsb25lIGRl dmljZSBhcmVuJ3QgdXBkYXRlZCB0byBhbGxvY2F0ZSBhIG5ldyAnc3RydWN0Cj4gPiA+ICAgICBk bV9jbG9uZScgdGhhdCByZWZsZWN0cyB0aGUgdXBwZXIgbGF5ZXIgYmlvIHRoYXQgd2FzIGlzc3Vl ZCB0byBpdC4KPiA+ID4KPiA+ID4gICAgIEZpeGVzOiA5N2E4OTQ1ODAyMGIzOCAoImRtOiBpbXBy b3ZlIG5vY2xvbmUgYmlvIHN1cHBvcnQiKQo+ID4gPiAgICAgUmVwb3J0ZWQtYnk6IE1pa3VsYXMg UGF0b2NrYSA8bXBhdG9ja2FAcmVkaGF0LmNvbT4KPiA+ID4gICAgIFNpZ25lZC1vZmYtYnk6IE1p a2UgU25pdHplciA8c25pdHplckByZWRoYXQuY29tPgo+ID4gPgo+ID4gPiBXaGF0IEkgYW0gc2Vl aW5nIGlzIGluIGFib3V0IDMgb3V0IG9mIDQgYm9vdHMgdGhlIHN0YXJ0dXAganVzdCBoYW5ncwo+ ID4gPiBhdCB0aGUgZmlsZXN5c3RlbSBjaGVjayBzdGFnZSB3aXRoIHRoZSBmb2xsb3dpbmcgbWVz c2FnZToKPiA+ID4gWyAgT0sgIF0gUmVhY2hlZCB0YXJnZXQgTG9jYWwgRmlsZSBTeXN0ZW1zIChQ cmUpLgo+ID4gPiAgICAgICAgICBTdGFydGluZyBGaWxlIFN5c3RlbSBDaGVjayBvbiAvZGV2L+KA pjEyNy1hZDU3LTQyNmYtYmI0NS0zNjM5NTA1NDRjMGMuLi4KPiA+ID4gWyAgICAqKl0gKDEgb2Yg MikgQSBzdGFydCBqb2IgaXMgcnVubmluZyBmb3LigKZuIG9uIGRldmljZSAyNTI6MiAoMTlzIC8g bm8gbGltaXQpCj4gPiA+Cj4gPiA+IEkgZGlkIHNvbWUgZ29vZ2xpbmcgYW5kIGl0IGxvb2tzIGxp a2UgYSBzaW1pbGFyIGlzc3VlIGhhcyBiZWVuCj4gPiA+IHJlcG9ydGVkIGZvciBzMzkwLiBCYXNl ZCBvbiB0aGUgcmVxdWVzdCBmb3IgZGF0YSB0aGVyZSBJIGhhdmUgdGhlCj4gPiA+IGZvbGxvd2lu ZyBpbmZvOgo+ID4gPiBbcm9vdEBsb2NhbGhvc3Qgfl0jIGRtc2V0dXAgbHMgLS10cmVlCj4gPiA+ IGZlZG9yYS1zd2FwICgyNTM6MSkKPiA+ID4gIOKUlOKUgCAoMjUyOjIpCj4gPiA+IGZlZG9yYS1y b290ICgyNTM6MCkKPiA+ID4gIOKUlOKUgCAoMjUyOjIpCj4gPiA+Cj4gPiA+IFtyb290QGxvY2Fs aG9zdCB+XSMgZG1zZXR1cCB0YWJsZQo+ID4gPiBmZWRvcmEtc3dhcDogMCA0MTk0MzA0IGxpbmVh ciAyNTI6MiAyMDQ4Cj4gPiA+IGZlZG9yYS1yb290OiAwIDMxNDU3MjgwIGxpbmVhciAyNTI6MiA0 MTk2MzUyCj4gPgo+ID4gVGhhbmtzLCB3aGljaCB2ZXJzaW9uIG9mIEZlZG9yYSBhcmUgeW91IHJ1 bm5pbmc/Cj4gCj4gVGhlIFZNIGlzIHJ1bm5pbmcgRmVkb3JhIDI3IHdpdGggYSBrZXJuZWwgYnVp bHQgb2ZmIG9mIGxhdGVzdAo+IGxpbnV4LW5leHQgYXMgb2YgTWFyY2ggMXN0Lgo+IAo+ID4gWW91 ciBjYXNlIGlzIG1vcmUgc3RyYWlnaHRmb3J3YXJkIGluIHRoYXQgeW91J3JlIGNsZWFybHkgdXNp bmcgYmlvLWJhc2VkCj4gPiBETSBsaW5lYXIgKHdoaWNoIHdhcyB1cGRhdGVkIHRvIGxldmVyYWdl ICJub2Nsb25lIiBzdXBwb3J0KTsgd2hlcmVhcyB0aGUKPiA+IHMzOTAgY2FzZSBpcyB1c2luZyBy ZXF1ZXN0LWJhc2VkIERNIHdoaWNoIGlzbid0IGltcGFjdGVkIGJ5IHRoZSBjb21taXQKPiA+IGlu IHF1ZXN0aW9uIGF0IGFsbC4KPiA+Cj4gPiBJJ2xsIGF0dGVtcHQgdG8gcmVwcm9kdWNlIGZpcnN0 IHRoaW5nIE1vbmRheS4KPiA+Cj4gPiBNaWtlCj4gCj4gVGhhbmtzLiBUaGUgYmVoYXZpb3Igb2Yg aXQgaGFzIG1lIHdvbmRlcmluZyBpZiB3ZSBhcmUgbG9va2luZyBhdAo+IHNvbWV0aGluZyBsaWtl IGFuIHVuaW5pdGlhbGl6ZWQgZGF0YSBpc3N1ZSBvciBzb21ldGhpbmcgbGlrZSB0aGF0Cj4gc2lu Y2UgYXMgSSBtZW50aW9uZWQgSSBkb24ndCBzZWUgdGhpcyBvY2N1ciBvbiBldmVyeSBib290LCBq dXN0IG9uCj4gbW9zdCBvZiB0aGVtLiBTbyBldmVyeSBub3cgYW5kIHRoZW4gSSBjYW4gYm9vdCB1 cCB0aGUgVk0gd2l0aG91dCBhbnkKPiBpc3N1ZXMsIGJ1dCBtb3N0IG9mIHRoZSB0aW1lIGl0IHdp bGwgYm9vdCBhbmQgdGhlbiBnZXQgc3R1Y2sgd2FpdGluZwo+IG9uIGpvYnMgdGhhdCB0YWtlIGZv cmV2ZXIuCgpJIGp1c3QgY29waWVkIHlvdSBvbiBhbm90aGVyIHJlbGF0ZWQgdGhyZWFkLCBidXQg Zm9yIHRoZSBiZW5lZml0IG9mCmFueW9uZSBvbiBMS01MLCBwbGVhc2Ugc2VlIHRoZSBmb2xsb3dp bmcgZm9yIGEgZml4IHRoYXQgd29ya3MgZm9yIG1lOgoKaHR0cHM6Ly93d3cucmVkaGF0LmNvbS9h cmNoaXZlcy9kbS1kZXZlbC8yMDE5LU1hcmNoL21zZzAwMDI3Lmh0bWwKCi0tCmRtLWRldmVsIG1h aWxpbmcgbGlzdApkbS1kZXZlbEByZWRoYXQuY29tCmh0dHBzOi8vd3d3LnJlZGhhdC5jb20vbWFp bG1hbi9saXN0aW5mby9kbS1kZXZlbA== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_PASS,USER_AGENT_MUTT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34964C43381 for ; Tue, 5 Mar 2019 04:07:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1054D20684 for ; Tue, 5 Mar 2019 04:07:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727092AbfCEEHR (ORCPT ); Mon, 4 Mar 2019 23:07:17 -0500 Received: from mx1.redhat.com ([209.132.183.28]:45336 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726942AbfCEEHQ (ORCPT ); Mon, 4 Mar 2019 23:07:16 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 269F359449; Tue, 5 Mar 2019 04:07:16 +0000 (UTC) Received: from localhost (unknown [10.18.25.174]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E6A3060F8D; Tue, 5 Mar 2019 04:07:15 +0000 (UTC) Date: Mon, 4 Mar 2019 23:07:15 -0500 From: Mike Snitzer To: Alexander Duyck Cc: linux-next@vger.kernel.org, LKML , dm-devel@redhat.com Subject: Re: x86 VM Boot hang with latest linux-next Message-ID: <20190305040714.GB21739@redhat.com> References: <20190303034847.GA8699@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Tue, 05 Mar 2019 04:07:16 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sun, Mar 03 2019 at 12:06pm -0500, Alexander Duyck wrote: > On Sat, Mar 2, 2019 at 7:48 PM Mike Snitzer wrote: > > > > On Sat, Mar 02 2019 at 6:34pm -0500, > > Alexander Duyck wrote: > > > > > So I have been seeing an issue with an intermittent boot hang on my > > > x86 KVM VM with the latest linux-next and have bisected it down to the > > > following commit: > > > 1efa3bb79d3de8ca1b7f6770313a1fc0bebe25c7 is the first bad commit > > > commit 1efa3bb79d3de8ca1b7f6770313a1fc0bebe25c7 > > > Author: Mike Snitzer > > > Date: Fri Feb 22 11:23:01 2019 -0500 > > > > > > dm: must allocate dm_noclone for stacked noclone devices > > > > > > Otherwise various lvm2 testsuite tests fail because the lower layers of > > > the stacked noclone device aren't updated to allocate a new 'struct > > > dm_clone' that reflects the upper layer bio that was issued to it. > > > > > > Fixes: 97a89458020b38 ("dm: improve noclone bio support") > > > Reported-by: Mikulas Patocka > > > Signed-off-by: Mike Snitzer > > > > > > What I am seeing is in about 3 out of 4 boots the startup just hangs > > > at the filesystem check stage with the following message: > > > [ OK ] Reached target Local File Systems (Pre). > > > Starting File System Check on /dev/…127-ad57-426f-bb45-363950544c0c... > > > [ **] (1 of 2) A start job is running for…n on device 252:2 (19s / no limit) > > > > > > I did some googling and it looks like a similar issue has been > > > reported for s390. Based on the request for data there I have the > > > following info: > > > [root@localhost ~]# dmsetup ls --tree > > > fedora-swap (253:1) > > > └─ (252:2) > > > fedora-root (253:0) > > > └─ (252:2) > > > > > > [root@localhost ~]# dmsetup table > > > fedora-swap: 0 4194304 linear 252:2 2048 > > > fedora-root: 0 31457280 linear 252:2 4196352 > > > > Thanks, which version of Fedora are you running? > > The VM is running Fedora 27 with a kernel built off of latest > linux-next as of March 1st. > > > Your case is more straightforward in that you're clearly using bio-based > > DM linear (which was updated to leverage "noclone" support); whereas the > > s390 case is using request-based DM which isn't impacted by the commit > > in question at all. > > > > I'll attempt to reproduce first thing Monday. > > > > Mike > > Thanks. The behavior of it has me wondering if we are looking at > something like an uninitialized data issue or something like that > since as I mentioned I don't see this occur on every boot, just on > most of them. So every now and then I can boot up the VM without any > issues, but most of the time it will boot and then get stuck waiting > on jobs that take forever. I just copied you on another related thread, but for the benefit of anyone on LKML, please see the following for a fix that works for me: https://www.redhat.com/archives/dm-devel/2019-March/msg00027.html