From mboxrd@z Thu Jan 1 00:00:00 1970 From: Joonas Lahtinen Subject: Re: [PATCH V6] drm/i915: Disable stolen memory when i915 runs in guest vm Date: Mon, 08 May 2017 13:07:10 +0300 Message-ID: <1494238030.3367.10.camel@linux.intel.com> References: <1493116501-29327-1-git-send-email-xiong.y.zhang@intel.com> <1493122003.3731.27.camel@linux.intel.com> <8082FF9BCB2B054996454E47167FF4EC1C4D0CAA@SHSMSX104.ccr.corp.intel.com> <8082FF9BCB2B054996454E47167FF4EC1C4D77EC@SHSMSX104.ccr.corp.intel.com> <1493976072.3551.19.camel@linux.intel.com> <8082FF9BCB2B054996454E47167FF4EC1C4DD744@SHSMSX104.ccr.corp.intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <8082FF9BCB2B054996454E47167FF4EC1C4DD744@SHSMSX104.ccr.corp.intel.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" To: "Zhang, Xiong Y" , "Tian, Kevin" , "Vetter, Daniel" , "zhenyuw@linux.intel.com" , "jani.nikula@linux.intel.com" , "alex.williamson@redhat.com" , David Woodhouse , "Bloomfield, Jon" Cc: "intel-gfx@lists.freedesktop.org" , "intel-gvt-dev@lists.freedesktop.org" , "stable@vger.kernel.org" List-Id: intel-gfx@lists.freedesktop.org T24gbGEsIDIwMTctMDUtMDYgYXQgMDI6NTggKzAwMDAsIFpoYW5nLCBYaW9uZyBZIHdyb3RlOgo+ ID4gCj4gPiBPbiBrZSwgMjAxNy0wNS0wMyBhdCAwOToyMiArMDAwMCwgWmhhbmcsIFhpb25nIFkg d3JvdGU6Cj4gPiA+IAo+ID4gPiA+IAo+ID4gPiA+IAo+ID4gPiA+ID4gCj4gPiA+ID4gPiAKPiA+ ID4gPiA+ICsgRGF2aWQgYW5kIEpvbgo+ID4gPiA+ID4gCj4gPiA+ID4gPiBPbiB0aSwgMjAxNy0w NC0yNSBhdCAxODozNCArMDgwMCwgWGlvbmcgWmhhbmcgd3JvdGU6Cj4gPiA+ID4gPiAKPiA+ID4g PiA+IFRoZSBibG9ja2luZyBpc3N1ZSBJIHNlZSBpcyB0aGF0IGJpc2VjdGluZyBpcyBzdGlsbCBu b3QgcG9pbnRpbmcgYXQKPiA+ID4gPiA+IHJlbGV2YW50IGNvbW1pdHMuIEJvdGggYmlzZWN0ZWQg Y29tbWl0cyBmcm9tIEJ1Z3ppbGxhIGFyZSBub3QgcmVsYXRlZAo+ID4gPiA+ID4gdG8gY2hhbmdl cyBpbiBzdG9sZW4gbWVtb3J5IHVzYWdlIGJlaGF2aW9yLiBJJ2QgYXNzdW1lIGEgc3VjY2Vzc2Z1 bAo+ID4gPiA+ID4gYmlzZWN0IHRvIGxhbmQgYXQgdGhlIHBhdGNoZXMgd2hlcmUgd2Ugc3RhcnQg Y3JlYXRpbmcga2VybmVsIGludGVybmFsCj4gPiA+ID4gPiBvYmplY3RzIGZyb20gc3RvbGVuIG1l bW9yeS4gT3RoZXJ3aXNlIHdlIGNvdWxkIGJlIGlnbm9yaW5nIGEgYnVnCj4gPiA+ID4gPiBlbHNl d2hlcmUuIElmIGl0IGNvbnNpc3RlbnRseSBsYW5kcyBvbiB0aG9zZSBwYXRjaGVzLCB0aGVuIHRo ZXJlIG1pZ2h0Cj4gPiA+ID4gPiBiZSBzb21ldGhpbmcgd3Jvbmcgd2l0aCB0aGVtLCBpbiBhZGRp dGlvbiB0byBzdG9sZW4gbWVtb3J5IHByb2JsZW1zLgo+ID4gPiA+IFtaaGFuZywgWGlvbmcgWV0g SSBvbmx5IHRyeSBrZXJuZWwgNC44IGFuZCA0LjkgYWJvdmUsIGFzIHRoZSBidWd6aWxsYQo+ID4g ZGVzY3JpcHRlZCwKPiA+ID4gCj4gPiA+ID4gCj4gPiA+ID4gZ3Vlc3QgNC44IGtlcm5lbCBkb2Vz bid0IHNlZSBncHUgaGFuZyBpbiBndWVzdCBkbWVzZywgNC45IGtlcm5lbCBoYXMgZ3B1Cj4gPiBo YW5nCj4gPiA+IAo+ID4gPiA+IAo+ID4gPiA+IGluIGd1ZXN0IGRtZXNnLiBGcm9tIHRoaXMgcG9p bnQsIHdlIGNvdWxkIGRvIGdpdCBiaXNlY3QuCj4gPiA+ID4gQnV0IHRvbnMgb2YgSU9NTVUgRE1B IFIvVyBleGNlcHRpb24gdG8gc3RvbGVuIG1lbW9yeSBleGlzdCBpbiBob3N0Cj4gPiBkbWVzZwo+ ID4gPiAKPiA+ID4gPiAKPiA+ID4gPiB3aGVuIGd1ZXN0IGtlcm5lbCBpcyA0LjggYW5kIDQuOS4g VGhpcyBtZWFucyBndWVzdCBkb21haW4gaW9tbXUgdGFibGUKPiA+ID4gPiBkb2Vzbid0Cj4gPiA+ ID4gaGF2ZSBtYXBwaW5nIGZvciBzdG9sZW4gbWVtb3J5IGFuZCBJR0QgZmFpbCBpbiBhY2Nlc3Np bmcgc3RvbGVuIG1lbW9yeQo+ID4gPiA+IGZyb20gZ3Vlc3Qga2VybmVsIDQuOCBhbmQgNC45LiBG cm9tIHRoaXMgcG9pbnQsIHRoaXMgaXNzdWUgaXNuJ3QgYSByZWdyZXNzaW9uCj4gPiBhbmQKPiA+ ID4gCj4gPiA+ID4gCj4gPiA+ID4gc2hvdWxkbid0IGdvIGdpdCBiaXNlY3QuIFlvdSBjb3VsZCBj aGVjayB0aGlzIGhvc3QgZXJyb3IgbWVzc2FnZSBmcm9tIHRoZQo+ID4gPiA+IGJ1Z3ppbGxhCj4g PiA+ID4gYXR0YWNobWVudC4gQW5kIHRoaXMgc2hvdWxkIGJlIGZpeGVkIGZpcnN0Lgo+ID4gPiA+ IEFueXdheSwgSSB3aWxsIHRyeSBteSBiZXN0IHRvIGdldCB0aGUgaWRlYWwgY29tbWl0IHRocm91 Z2ggZ2l0IGJpc2VjdCwgYnV0Cj4gPiBJJ20KPiA+ID4gCj4gPiA+ID4gCj4gPiA+ID4gYWZyYWlk Cj4gPiA+ID4gdGhlIHJlc3VsdCBpcyB0aGUgc2FtZSBhcyBwYXN0IGJlY2F1c2Ugd2UgZG9uJ3Qg aGF2ZSBhIHN0YWJsZSBnb29kIHBvaW50IHRvCj4gPiA+ID4gc3RhcnQgZ2l0Cj4gPiA+ID4gYmlz ZWN0Lgo+ID4gPiBbWmhhbmcsIFhpb25nIFldIGhpLCBKb29uYXM6Cj4gPiA+IEFzIHlvdSBzYWlk LCB0aGUgZ3B1IGhhbmcgZXhpc3QgYmVjYXVzZSBpOTE1IGNyZWF0ZSByaW5nIGJ1ZmZlciBmcm9t IHN0b2xlbgo+ID4gbWVtb3J5Lgo+ID4gPiAKPiA+ID4gSSBkaWQgZ2l0IGJpc2VjdCBhZ2Fpbiwg YW5kIHRoZSBmb2xsb3dpbmcgY29tbWl0IGlzIHRoZSBmaXJzdCBiYWQgY29tbWl0Ogo+ID4gPiBj b21taXQgYzU4YjczNWZjNzYyZTg5MTQ4MWU5MmFmNzEyNGI4NWNiMGE1MWZjZQo+ID4gPiA+ID4g PiBBdXRob3I6IENocmlzIFdpbHNvbiA8Y2hyaXNAY2hyaXMtd2lsc29uLmNvLnVrPgo+ID4gPiBE YXRlOsKgwqDCoFRodSBBdWcgMTggMTc6MTY6NTcgMjAxNiArMDEwMAo+ID4gPiAKPiA+ID4gwqDC oMKgwqBkcm0vaTkxNTogQWxsb2NhdGUgcmluZ3MgZnJvbSBzdG9sZW4KPiA+ID4gCj4gPiA+IMKg wqDCoMKgSWYgd2UgaGF2ZSBzdG9sZW4gYXZhaWxhYmxlLCBtYWtlIHVzZSBvZiBpdCBmb3Igcmlu Z2J1ZmZlciBhbGxvY2F0aW9uLgo+ID4gPiDCoMKgwqDCoFByZXZpb3VzbHkgdGhpcyB3YXMgcmVz dHJpY3RlZCB0byAhbGxjIHBsYXRmb3JtcywgYXMgd3JpdGluZyB0byBzdG9sZW4KPiA+ID4gwqDC oMKgwqByZXF1aXJlcyBhIEdHVFQgbWFwcGluZyAtIGJ1dCBub3cgdGhhdCB3ZSBoYXZlIHBhcnRp YWwgbWFwcGFibGUKPiA+IHN1cHBvcnQsCj4gPiA+IAo+ID4gPiDCoMKgwqDCoHRoZSBtYXBwYWJs ZSBhcGVydHVyZSBpc24ndCBxdWl0ZSBzbyBwcmVjaW91cyBzbyB3ZSBjYW4gdXNlIGl0IG1vcmUK PiA+ID4gwqDCoMKgwqBmcmVlbHkgYW5kIHJpbmdidWZmZXJzIGFyZSBhIGdvb2QgdXNlciBmb3Ig dGhlIG90aGVyd2lzZSB3YXN0ZWQgc3RvbGVuLgo+ID4gPiAKPiA+ID4gQWZ0ZXIgcmV2ZXJ0aW5n IHRoaXMgcGF0Y2ggZnJvbSBkcm0taW50ZWwtbmlnaHRseSwgSSBkaWRuJ3Qgc2VlIGdwdSBoYW5n IGR1cmluZwo+ID4gZ3Vlc3QgYm9vdCBwcm9jZXNzLgo+ID4gPiAKPiA+ID4gU28gd2hhdCdzIG91 ciBuZXh0IHN0ZXAgPwo+ID4gCj4gPiBBbiBhcHByb3ByaWF0ZSBuZXh0IHN0ZXAgd291bGQgYmUg dG8gZXZhbHVhdGUgaG93IG11Y2ggd29yayBpdCBpcyB0bwo+ID4gc3VwcG9ydCB0aGUgUk1SUiBw YXNzdGhyb3VnaCBEYXZpZCBtZW50aW9uZWQgYWJvdXQgaW4gaGlzIGNvbW1pdC4KPiBbWmhhbmcs IFhpb25nIFldIEFzIEtldmluIGV4cGxhaW5lZCwgS1ZNIGNvbW11bml0eSBmb3VuZCB0aGUgZGlz YWR2YW50YWdlCj4gT2YgUk1SUiBhbmQgaGF2ZSBkZWNpZGVkIHRvIG5vdCBzdXBwb3J0IFJNUlIg cGFzc3Rocm91Z2gsIHNvIGl0IGlzIHJlYWxseSBoYXJkCj4gZm9yIHVzIHRvIHB1c2ggc3VjaCBz b2x1dGlvbiBhbmQgaXNuJ3QgcmVsYXRlZCB0byB0aGUgd29ya2xvYWQuCj4gRXhjZXB0IHVzYiBh bmQgZ3JhcGhpYyBjYXJkLCBhbGwgb3RoZXIgZGV2aWNlcyB3aXRoIFJNUlIgY291bGRuJ3QgcGFz c3Rocm91Z2gKPiB0byBndWVzdC4gQnV0IHRoZSBkcml2ZXIgb2YgdXNiIGFuZCBncmFwaGljIGNh cmQgY291bGRuJ3QgYWNjZXNzIFJNUlIgaW4gc3VjaAo+IGVudmlyb25tZW50Lgo+IGh0dHBzOi8v YWNjZXNzLnJlZGhhdC5jb20vc2l0ZXMvZGVmYXVsdC9maWxlcy9hdHRhY2htZW50cy9ybXJyLXdw MS5wZGYKCkRvZXMgdGhpcyBwYXRjaCBoYXZlIHRoZSByaWdodCBDYydzIGZyb20gS1ZNIHRlYW0/ IEknZCBsaWtlIHRvIGhlYXIKZGlyZWN0bHkgZnJvbSB0aGVtIHRoYXQgZXZlbiB0aGUgdXNhZ2Ug b2YgUk1SUnMgdGhhdCBmb2xsb3cgdGhlCmludGVudGlvbiBvZiBWVC1kIHNwZWMgYXJlIG5vdCBn b2luZyB0byBiZSBzdXBwb3J0ZWQuIFRoYXQgZG9jdW1lbnQKcHJlZGF0ZXMgdGhlIHBhdGNoZXMg dG8gYWRkIHRoZSBleGNsdXNpb24gZm9yIGdyYXBoaWNzLgoKPiA+IEknZCBhbHNvIGdvIHRhbGsg d2l0aCB0aGUgSUdEIHRlYW0sIHdoeSB0aGV5IHJlZnVzZSB0byBsb2FkIHRoZSBkcml2ZXIKPiA+ IHdoZW4gc3RvbGVuIG1lbW9yeSBpcyBjb3JyZWN0bHkgcmVwb3J0ZWQgYXMgemVybywgYW5kIGlu c2lzdCBvbiBiZWluZwo+ID4gbGllZCB0by4KPiBbWmhhbmcsIFhpb25nIFldIHRoYW5rcyBhIGxv dCBmb3IgZG9pbmcgc28uCgpJIGRvbid0IGhhdmUgdGhlIGNvbnRhY3RzLCBzbyBJIGFzc3VtZSB5 b3UgdG8gcHVyc3VlIHRoYXQuCgpSZWdhcmRzLCBKb29uYXMKLS0gCkpvb25hcyBMYWh0aW5lbgpP cGVuIFNvdXJjZSBUZWNobm9sb2d5IENlbnRlcgpJbnRlbCBDb3Jwb3JhdGlvbgpfX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpJbnRlbC1nZnggbWFpbGluZyBs aXN0CkludGVsLWdmeEBsaXN0cy5mcmVlZGVza3RvcC5vcmcKaHR0cHM6Ly9saXN0cy5mcmVlZGVz a3RvcC5vcmcvbWFpbG1hbi9saXN0aW5mby9pbnRlbC1nZngK From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga04.intel.com ([192.55.52.120]:29235 "EHLO mga04.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751373AbdEHKHR (ORCPT ); Mon, 8 May 2017 06:07:17 -0400 Message-ID: <1494238030.3367.10.camel@linux.intel.com> Subject: Re: [PATCH V6] drm/i915: Disable stolen memory when i915 runs in guest vm From: Joonas Lahtinen To: "Zhang, Xiong Y" , "Tian, Kevin" , "Vetter, Daniel" , "zhenyuw@linux.intel.com" , "jani.nikula@linux.intel.com" , "alex.williamson@redhat.com" , David Woodhouse , "Bloomfield, Jon" Cc: "intel-gfx@lists.freedesktop.org" , "intel-gvt-dev@lists.freedesktop.org" , "stable@vger.kernel.org" Date: Mon, 08 May 2017 13:07:10 +0300 In-Reply-To: <8082FF9BCB2B054996454E47167FF4EC1C4DD744@SHSMSX104.ccr.corp.intel.com> References: <1493116501-29327-1-git-send-email-xiong.y.zhang@intel.com> <1493122003.3731.27.camel@linux.intel.com> <8082FF9BCB2B054996454E47167FF4EC1C4D0CAA@SHSMSX104.ccr.corp.intel.com> <8082FF9BCB2B054996454E47167FF4EC1C4D77EC@SHSMSX104.ccr.corp.intel.com> <1493976072.3551.19.camel@linux.intel.com> <8082FF9BCB2B054996454E47167FF4EC1C4DD744@SHSMSX104.ccr.corp.intel.com> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: stable-owner@vger.kernel.org List-ID: On la, 2017-05-06 at 02:58 +0000, Zhang, Xiong Y wrote: > > > > On ke, 2017-05-03 at 09:22 +0000, Zhang, Xiong Y wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > + David and Jon > > > > > > > > > > On ti, 2017-04-25 at 18:34 +0800, Xiong Zhang wrote: > > > > > > > > > > The blocking issue I see is that bisecting is still not pointing at > > > > > relevant commits. Both bisected commits from Bugzilla are not related > > > > > to changes in stolen memory usage behavior. I'd assume a successful > > > > > bisect to land at the patches where we start creating kernel internal > > > > > objects from stolen memory. Otherwise we could be ignoring a bug > > > > > elsewhere. If it consistently lands on those patches, then there might > > > > > be something wrong with them, in addition to stolen memory problems. > > > > [Zhang, Xiong Y] I only try kernel 4.8 and 4.9 above, as the bugzilla > > descripted, > > > > > > > > > > > guest 4.8 kernel doesn't see gpu hang in guest dmesg, 4.9 kernel has gpu > > hang > > > > > > > > > > > in guest dmesg. From this point, we could do git bisect. > > > > But tons of IOMMU DMA R/W exception to stolen memory exist in host > > dmesg > > > > > > > > > > > when guest kernel is 4.8 and 4.9. This means guest domain iommu table > > > > doesn't > > > > have mapping for stolen memory and IGD fail in accessing stolen memory > > > > from guest kernel 4.8 and 4.9. From this point, this issue isn't a regression > > and > > > > > > > > > > > shouldn't go git bisect. You could check this host error message from the > > > > bugzilla > > > > attachment. And this should be fixed first. > > > > Anyway, I will try my best to get the ideal commit through git bisect, but > > I'm > > > > > > > > > > > afraid > > > > the result is the same as past because we don't have a stable good point to > > > > start git > > > > bisect. > > > [Zhang, Xiong Y] hi, Joonas: > > > As you said, the gpu hang exist because i915 create ring buffer from stolen > > memory. > > > > > > I did git bisect again, and the following commit is the first bad commit: > > > commit c58b735fc762e891481e92af7124b85cb0a51fce > > > > > > Author: Chris Wilson > > > Date:   Thu Aug 18 17:16:57 2016 +0100 > > > > > >     drm/i915: Allocate rings from stolen > > > > > >     If we have stolen available, make use of it for ringbuffer allocation. > > >     Previously this was restricted to !llc platforms, as writing to stolen > > >     requires a GGTT mapping - but now that we have partial mappable > > support, > > > > > >     the mappable aperture isn't quite so precious so we can use it more > > >     freely and ringbuffers are a good user for the otherwise wasted stolen. > > > > > > After reverting this patch from drm-intel-nightly, I didn't see gpu hang during > > guest boot process. > > > > > > So what's our next step ? > > > > An appropriate next step would be to evaluate how much work it is to > > support the RMRR passthrough David mentioned about in his commit. > [Zhang, Xiong Y] As Kevin explained, KVM community found the disadvantage > Of RMRR and have decided to not support RMRR passthrough, so it is really hard > for us to push such solution and isn't related to the workload. > Except usb and graphic card, all other devices with RMRR couldn't passthrough > to guest. But the driver of usb and graphic card couldn't access RMRR in such > environment. > https://access.redhat.com/sites/default/files/attachments/rmrr-wp1.pdf Does this patch have the right Cc's from KVM team? I'd like to hear directly from them that even the usage of RMRRs that follow the intention of VT-d spec are not going to be supported. That document predates the patches to add the exclusion for graphics. > > I'd also go talk with the IGD team, why they refuse to load the driver > > when stolen memory is correctly reported as zero, and insist on being > > lied to. > [Zhang, Xiong Y] thanks a lot for doing so. I don't have the contacts, so I assume you to pursue that. Regards, Joonas -- Joonas Lahtinen Open Source Technology Center Intel Corporation