From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:44911) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cxH0h-00051T-JZ for qemu-devel@nongnu.org; Sun, 09 Apr 2017 13:52:24 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cxH0d-00088n-Lm for qemu-devel@nongnu.org; Sun, 09 Apr 2017 13:52:23 -0400 Received: from mail-lf0-x243.google.com ([2a00:1450:4010:c07::243]:36304) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1cxH0d-00087g-EI for qemu-devel@nongnu.org; Sun, 09 Apr 2017 13:52:19 -0400 Received: by mail-lf0-x243.google.com with SMTP id 75so1339897lfs.3 for ; Sun, 09 Apr 2017 10:52:15 -0700 (PDT) Date: Mon, 10 Apr 2017 03:52:05 +1000 From: Alexey G Message-ID: <20170410035205.000050b1@gmail.com> In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] [Xen-devel] [RFC/BUG] xen-mapcache: buggy invalidate map cache? List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: hrg Cc: anthony.perard@citrix.com, xen-devel@lists.xensource.com, qemu-devel@nongnu.org, jun.nakajima@intel.com, agraf@suse.de, sstabellini@kernel.org, xen-devel@lists.xenproject.org, wangxinxin.wang@huawei.com, "Herongguang (Stephen)" , xen-devel@lists.xen.org On Mon, 10 Apr 2017 00:36:02 +0800 hrg wrote: Hi, > On Sun, Apr 9, 2017 at 11:55 PM, hrg wrote: > > On Sun, Apr 9, 2017 at 11:52 PM, hrg wrote: =20 > >> Hi, > >> > >> In xen_map_cache_unlocked(), map to guest memory maybe in entry->next > >> instead of first level entry (if map to rom other than guest memory > >> comes first), while in xen_invalidate_map_cache(), when VM ballooned > >> out memory, qemu did not invalidate cache entries in linked > >> list(entry->next), so when VM balloon back in memory, gfns probably > >> mapped to different mfns, thus if guest asks device to DMA to these > >> GPA, qemu may DMA to stale MFNs. > >> > >> So I think in xen_invalidate_map_cache() linked lists should also be > >> checked and invalidated. > >> > >> What=E2=80=99s your opinion? Is this a bug? Is my analyze correct? =20 > > > > Added Jun Nakajima and Alexander Graf =20 > And correct Stefano Stabellini's email address. There is a real issue with the xen-mapcache corruption in fact. I encounter= ed it a few months ago while experimenting with Q35 support on Xen. Q35 emulat= ion uses an AHCI controller by default, along with NCQ mode enabled. The issue = can be (somewhat) easily reproduced there, though using a normal i440 emulation might possibly allow to reproduce the issue as well, using a dedicated test code from a guest side. In case of Q35+NCQ the issue can be reproduced "as = is". The issue occurs when a guest domain performs an intensive disk I/O, ex. wh= ile guest OS booting. QEMU crashes with "Bad ram offset 980aa000" message logged, where the address is different each time. The hard thing wi= th this issue is that it has a very low reproducibility rate. The corruption happens when there are multiple I/O commands in the NCQ queu= e. So there are overlapping emulated DMA operations in flight and QEMU uses a sequence of mapcache actions which can be executed in the "wrong" order thus leading to an inconsistent xen-mapcache - so a bad address from the wrong entry is returned. The bad thing with this issue is that QEMU crash due to "Bad ram offset" appearance is a relatively good situation in the sense that this is a caught error. But there might be a much worse (artificial) situation where the ret= urned address looks valid but points to a different mapped memory. The fix itself is not hard (ex. an additional checked field in MapCacheEntr= y), but there is a need of some reliable way to test it considering the low reproducibility rate. Regards, Alex From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alexey G Subject: Re: [Qemu-devel] [RFC/BUG] xen-mapcache: buggy invalidate map cache? Date: Mon, 10 Apr 2017 03:52:05 +1000 Message-ID: <20170410035205.000050b1@gmail.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cxH0a-0008KP-4Q for xen-devel@lists.xenproject.org; Sun, 09 Apr 2017 17:52:16 +0000 Received: by mail-lf0-f65.google.com with SMTP id r36so9668030lfi.0 for ; Sun, 09 Apr 2017 10:52:14 -0700 (PDT) In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: hrg Cc: xen-devel@lists.xensource.com, wangxinxin.wang@huawei.com, qemu-devel@nongnu.org, agraf@suse.de, sstabellini@kernel.org, jun.nakajima@intel.com, anthony.perard@citrix.com, xen-devel@lists.xenproject.org, xen-devel@lists.xen.org, "Herongguang (Stephen)" List-Id: xen-devel@lists.xenproject.org T24gTW9uLCAxMCBBcHIgMjAxNyAwMDozNjowMiArMDgwMApocmcgPGhyZ3N0ZXBoZW5AZ21haWwu Y29tPiB3cm90ZToKCkhpLAoKPiBPbiBTdW4sIEFwciA5LCAyMDE3IGF0IDExOjU1IFBNLCBocmcg PGhyZ3N0ZXBoZW5AZ21haWwuY29tPiB3cm90ZToKPiA+IE9uIFN1biwgQXByIDksIDIwMTcgYXQg MTE6NTIgUE0sIGhyZyA8aHJnc3RlcGhlbkBnbWFpbC5jb20+IHdyb3RlOiAgCj4gPj4gSGksCj4g Pj4KPiA+PiBJbiB4ZW5fbWFwX2NhY2hlX3VubG9ja2VkKCksIG1hcCB0byBndWVzdCBtZW1vcnkg bWF5YmUgaW4gZW50cnktPm5leHQKPiA+PiBpbnN0ZWFkIG9mIGZpcnN0IGxldmVsIGVudHJ5IChp ZiBtYXAgdG8gcm9tIG90aGVyIHRoYW4gZ3Vlc3QgbWVtb3J5Cj4gPj4gY29tZXMgZmlyc3QpLCB3 aGlsZSBpbiB4ZW5faW52YWxpZGF0ZV9tYXBfY2FjaGUoKSwgd2hlbiBWTSBiYWxsb29uZWQKPiA+ PiBvdXQgbWVtb3J5LCBxZW11IGRpZCBub3QgaW52YWxpZGF0ZSBjYWNoZSBlbnRyaWVzIGluIGxp bmtlZAo+ID4+IGxpc3QoZW50cnktPm5leHQpLCBzbyB3aGVuIFZNIGJhbGxvb24gYmFjayBpbiBt ZW1vcnksIGdmbnMgcHJvYmFibHkKPiA+PiBtYXBwZWQgdG8gZGlmZmVyZW50IG1mbnMsIHRodXMg aWYgZ3Vlc3QgYXNrcyBkZXZpY2UgdG8gRE1BIHRvIHRoZXNlCj4gPj4gR1BBLCBxZW11IG1heSBE TUEgdG8gc3RhbGUgTUZOcy4KPiA+Pgo+ID4+IFNvIEkgdGhpbmsgaW4geGVuX2ludmFsaWRhdGVf bWFwX2NhY2hlKCkgbGlua2VkIGxpc3RzIHNob3VsZCBhbHNvIGJlCj4gPj4gY2hlY2tlZCBhbmQg aW52YWxpZGF0ZWQuCj4gPj4KPiA+PiBXaGF04oCZcyB5b3VyIG9waW5pb24/IElzIHRoaXMgYSBi dWc/IElzIG15IGFuYWx5emUgY29ycmVjdD8gIAo+ID4KPiA+IEFkZGVkIEp1biBOYWthamltYSBh bmQgQWxleGFuZGVyIEdyYWYgIAo+IEFuZCBjb3JyZWN0IFN0ZWZhbm8gU3RhYmVsbGluaSdzIGVt YWlsIGFkZHJlc3MuCgpUaGVyZSBpcyBhIHJlYWwgaXNzdWUgd2l0aCB0aGUgeGVuLW1hcGNhY2hl IGNvcnJ1cHRpb24gaW4gZmFjdC4gSSBlbmNvdW50ZXJlZAppdCBhIGZldyBtb250aHMgYWdvIHdo aWxlIGV4cGVyaW1lbnRpbmcgd2l0aCBRMzUgc3VwcG9ydCBvbiBYZW4uIFEzNSBlbXVsYXRpb24K dXNlcyBhbiBBSENJIGNvbnRyb2xsZXIgYnkgZGVmYXVsdCwgYWxvbmcgd2l0aCBOQ1EgbW9kZSBl bmFibGVkLiBUaGUgaXNzdWUgY2FuCmJlIChzb21ld2hhdCkgZWFzaWx5IHJlcHJvZHVjZWQgdGhl cmUsIHRob3VnaCB1c2luZyBhIG5vcm1hbCBpNDQwIGVtdWxhdGlvbgptaWdodCBwb3NzaWJseSBh bGxvdyB0byByZXByb2R1Y2UgdGhlIGlzc3VlIGFzIHdlbGwsIHVzaW5nIGEgZGVkaWNhdGVkIHRl c3QKY29kZSBmcm9tIGEgZ3Vlc3Qgc2lkZS4gSW4gY2FzZSBvZiBRMzUrTkNRIHRoZSBpc3N1ZSBj YW4gYmUgcmVwcm9kdWNlZCAiYXMgaXMiLgoKVGhlIGlzc3VlIG9jY3VycyB3aGVuIGEgZ3Vlc3Qg ZG9tYWluIHBlcmZvcm1zIGFuIGludGVuc2l2ZSBkaXNrIEkvTywgZXguIHdoaWxlCmd1ZXN0IE9T IGJvb3RpbmcuIFFFTVUgY3Jhc2hlcyB3aXRoICJCYWQgcmFtIG9mZnNldCA5ODBhYTAwMCIKbWVz c2FnZSBsb2dnZWQsIHdoZXJlIHRoZSBhZGRyZXNzIGlzIGRpZmZlcmVudCBlYWNoIHRpbWUuIFRo ZSBoYXJkIHRoaW5nIHdpdGgKdGhpcyBpc3N1ZSBpcyB0aGF0IGl0IGhhcyBhIHZlcnkgbG93IHJl cHJvZHVjaWJpbGl0eSByYXRlLgoKVGhlIGNvcnJ1cHRpb24gaGFwcGVucyB3aGVuIHRoZXJlIGFy ZSBtdWx0aXBsZSBJL08gY29tbWFuZHMgaW4gdGhlIE5DUSBxdWV1ZS4KU28gdGhlcmUgYXJlIG92 ZXJsYXBwaW5nIGVtdWxhdGVkIERNQSBvcGVyYXRpb25zIGluIGZsaWdodCBhbmQgUUVNVSB1c2Vz IGEKc2VxdWVuY2Ugb2YgbWFwY2FjaGUgYWN0aW9ucyB3aGljaCBjYW4gYmUgZXhlY3V0ZWQgaW4g dGhlICJ3cm9uZyIgb3JkZXIgdGh1cwpsZWFkaW5nIHRvIGFuIGluY29uc2lzdGVudCB4ZW4tbWFw Y2FjaGUgLSBzbyBhIGJhZCBhZGRyZXNzIGZyb20gdGhlIHdyb25nCmVudHJ5IGlzIHJldHVybmVk LgoKVGhlIGJhZCB0aGluZyB3aXRoIHRoaXMgaXNzdWUgaXMgdGhhdCBRRU1VIGNyYXNoIGR1ZSB0 byAiQmFkIHJhbSBvZmZzZXQiCmFwcGVhcmFuY2UgaXMgYSByZWxhdGl2ZWx5IGdvb2Qgc2l0dWF0 aW9uIGluIHRoZSBzZW5zZSB0aGF0IHRoaXMgaXMgYSBjYXVnaHQKZXJyb3IuIEJ1dCB0aGVyZSBt aWdodCBiZSBhIG11Y2ggd29yc2UgKGFydGlmaWNpYWwpIHNpdHVhdGlvbiB3aGVyZSB0aGUgcmV0 dXJuZWQKYWRkcmVzcyBsb29rcyB2YWxpZCBidXQgcG9pbnRzIHRvIGEgZGlmZmVyZW50IG1hcHBl ZCBtZW1vcnkuCgpUaGUgZml4IGl0c2VsZiBpcyBub3QgaGFyZCAoZXguIGFuIGFkZGl0aW9uYWwg Y2hlY2tlZCBmaWVsZCBpbiBNYXBDYWNoZUVudHJ5KSwKYnV0IHRoZXJlIGlzIGEgbmVlZCBvZiBz b21lIHJlbGlhYmxlIHdheSB0byB0ZXN0IGl0IGNvbnNpZGVyaW5nIHRoZSBsb3cKcmVwcm9kdWNp YmlsaXR5IHJhdGUuCgpSZWdhcmRzLApBbGV4CgpfX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fXwpYZW4tZGV2ZWwgbWFpbGluZyBsaXN0Clhlbi1kZXZlbEBsaXN0 cy54ZW4ub3JnCmh0dHBzOi8vbGlzdHMueGVuLm9yZy94ZW4tZGV2ZWwK