From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 44936C433F5 for ; Fri, 17 Dec 2021 02:01:13 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id DC12884B32; Fri, 17 Dec 2021 02:01:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eaEqffQJgcOE; Fri, 17 Dec 2021 02:01:11 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id B221784B31; Fri, 17 Dec 2021 02:01:10 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 96C4EC002F; Fri, 17 Dec 2021 02:01:10 +0000 (UTC) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2A70FC0012 for ; Fri, 17 Dec 2021 02:01:10 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 0AE9684B32 for ; Fri, 17 Dec 2021 02:01:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id A1MVzbIyVG4q for ; Fri, 17 Dec 2021 02:01:08 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id 44AE284B31 for ; Fri, 17 Dec 2021 02:01:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1639706467; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gbhzR/ta5wJM0NZ8+LJz1rln0Vg8UKqxlqmsFHgHsdY=; b=BhX/Sk0q7+Rj63zIzOvmlVhbjyZHhhCBdEyM/DI3T/K74dg66hr++9EA2VjOGZhqqkBrGs oONozh2Bwv3KbbVvTa1ooXk6f7xGuM/AwjupA4FPkj1abWPmQeGAw0lQjHU3IT8I4tGe4A MbD8jswbL634nwEG3wSF4oAD55wZ1/Q= Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-19-93X4MzN_MOOIo6FmfjGBiw-1; Thu, 16 Dec 2021 21:01:05 -0500 X-MC-Unique: 93X4MzN_MOOIo6FmfjGBiw-1 Received: by mail-ed1-f69.google.com with SMTP id y11-20020a056402358b00b003f7ce63b89eso584993edc.3 for ; Thu, 16 Dec 2021 18:01:05 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=gbhzR/ta5wJM0NZ8+LJz1rln0Vg8UKqxlqmsFHgHsdY=; b=amY6+t1DO0WXYvaENOHQCA/mcDpERC8+eQxCjONuahcE2FxlvvoAo7fO9Wsb6SEyAW UgDRr4JSGDZ4zGQd3v8rrgLT1fqQm0EtjpoMPb1933eRjG+8i6hgsx9e2Jkr/uc5EkkB 2JbrXZN9qRu6Y2vWcw/ItCgglECHA1i8+S5WbIZFWt2CiheQQFQl4VRHGou5ncPdYfj4 h0qIq2PpeuGyZP3lqkEmsrTMWYnT+fj74auJ9+dXrMVhGhU7ehiHVCLfDXocGYMiJG2U M6eJ6r9QTMS6lPowsvP9O+o+3om4LDlxpOPZuGlVofv56akRK4lHB4wV1Jd7m0LqXxjR SOJQ== X-Gm-Message-State: AOAM5318gSrT/0mQXhar1mDhk5o8mzg7/HjBi1P19HlJXQTI0iIQ+LBJ HhBNdjI5nSl9AoezbwV0t8HKxKCZUJ9F5bIt/HO9nVGVFwT84EAHT6LVUnT0w2GoRNfSSSNGXlB TGYL+cr3DbFYnvi8VUKAei6SJJMMX9YusM81yU9kd6Q== X-Received: by 2002:a17:907:1c89:: with SMTP id nb9mr309666ejc.408.1639706464287; Thu, 16 Dec 2021 18:01:04 -0800 (PST) X-Google-Smtp-Source: ABdhPJyJOVB8t1MYtjdN8SJk8prhfOCZMdXtMw9SPSd0gMBtUijDWi/l/Iwya9JaM7Wm6Vd6RmlDzw== X-Received: by 2002:a17:907:1c89:: with SMTP id nb9mr309645ejc.408.1639706463952; Thu, 16 Dec 2021 18:01:03 -0800 (PST) Received: from redhat.com ([2.55.30.56]) by smtp.gmail.com with ESMTPSA id nc24sm2401836ejc.94.2021.12.16.18.01.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Dec 2021 18:01:03 -0800 (PST) Date: Thu, 16 Dec 2021 21:00:59 -0500 From: "Michael S. Tsirkin" To: Jason Wang Subject: Re: vdpa legacy guest support (was Re: [PATCH] vdpa/mlx5: set_features should allow reset to zero) Message-ID: <20211216205958-mutt-send-email-mst@kernel.org> References: <20211214000245-mutt-send-email-mst@kernel.org> <4fc43d0f-da9e-ce16-1f26-9f0225239b75@oracle.com> <6eaf672c-cc86-b5bf-5b74-c837affeb6e1@oracle.com> <20211215162917-mutt-send-email-mst@kernel.org> <71d2a69c-94a7-76b5-2971-570026760bf0@oracle.com> MIME-Version: 1.0 In-Reply-To: Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=mst@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Disposition: inline Cc: Si-Wei Liu , netdev , Eli Cohen , linux-kernel , virtualization X-BeenThere: virtualization@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Linux virtualization List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: virtualization-bounces@lists.linux-foundation.org Sender: "Virtualization" T24gRnJpLCBEZWMgMTcsIDIwMjEgYXQgMDk6NTc6MzhBTSArMDgwMCwgSmFzb24gV2FuZyB3cm90 ZToKPiBPbiBGcmksIERlYyAxNywgMjAyMSBhdCA2OjMyIEFNIFNpLVdlaSBMaXUgPHNpLXdlaS5s aXVAb3JhY2xlLmNvbT4gd3JvdGU6Cj4gPgo+ID4KPiA+Cj4gPiBPbiAxMi8xNS8yMDIxIDY6NTMg UE0sIEphc29uIFdhbmcgd3JvdGU6Cj4gPiA+IE9uIFRodSwgRGVjIDE2LCAyMDIxIGF0IDEwOjAy IEFNIFNpLVdlaSBMaXUgPHNpLXdlaS5saXVAb3JhY2xlLmNvbT4gd3JvdGU6Cj4gPiA+Pgo+ID4g Pj4KPiA+ID4+IE9uIDEyLzE1LzIwMjEgMTozMyBQTSwgTWljaGFlbCBTLiBUc2lya2luIHdyb3Rl Ogo+ID4gPj4+IE9uIFdlZCwgRGVjIDE1LCAyMDIxIGF0IDEyOjUyOjIwUE0gLTA4MDAsIFNpLVdl aSBMaXUgd3JvdGU6Cj4gPiA+Pj4+IE9uIDEyLzE0LzIwMjEgNjowNiBQTSwgSmFzb24gV2FuZyB3 cm90ZToKPiA+ID4+Pj4+IE9uIFdlZCwgRGVjIDE1LCAyMDIxIGF0IDk6MDUgQU0gU2ktV2VpIExp dSA8c2ktd2VpLmxpdUBvcmFjbGUuY29tPiB3cm90ZToKPiA+ID4+Pj4+PiBPbiAxMi8xMy8yMDIx IDk6MDYgUE0sIE1pY2hhZWwgUy4gVHNpcmtpbiB3cm90ZToKPiA+ID4+Pj4+Pj4gT24gTW9uLCBE ZWMgMTMsIDIwMjEgYXQgMDU6NTk6NDVQTSAtMDgwMCwgU2ktV2VpIExpdSB3cm90ZToKPiA+ID4+ Pj4+Pj4+IE9uIDEyLzEyLzIwMjEgMToyNiBBTSwgTWljaGFlbCBTLiBUc2lya2luIHdyb3RlOgo+ ID4gPj4+Pj4+Pj4+IE9uIEZyaSwgRGVjIDEwLCAyMDIxIGF0IDA1OjQ0OjE1UE0gLTA4MDAsIFNp LVdlaSBMaXUgd3JvdGU6Cj4gPiA+Pj4+Pj4+Pj4+IFNvcnJ5IGZvciByZXZpdmluZyB0aGlzIGFu Y2llbnQgdGhyZWFkLiBJIHdhcyBraW5kYSBsb3N0IGZvciB0aGUgY29uY2x1c2lvbgo+ID4gPj4+ Pj4+Pj4+PiBpdCBlbmRlZCB1cCB3aXRoLiBJIGhhdmUgdGhlIGZvbGxvd2luZyBxdWVzdGlvbnMs Cj4gPiA+Pj4+Pj4+Pj4+Cj4gPiA+Pj4+Pj4+Pj4+IDEuIGxlZ2FjeSBndWVzdCBzdXBwb3J0OiBm cm9tIHRoZSBwYXN0IGNvbnZlcnNhdGlvbnMgaXQgZG9lc24ndCBzZWVtIHRoZQo+ID4gPj4+Pj4+ Pj4+PiBzdXBwb3J0IHdpbGwgYmUgY29tcGxldGVseSBkcm9wcGVkIGZyb20gdGhlIHRhYmxlLCBp cyBteSB1bmRlcnN0YW5kaW5nCj4gPiA+Pj4+Pj4+Pj4+IGNvcnJlY3Q/IEFjdHVhbGx5IHdlJ3Jl IGludGVyZXN0ZWQgaW4gc3VwcG9ydGluZyB2aXJ0aW8gdjAuOTUgZ3Vlc3QgZm9yIHg4NiwKPiA+ ID4+Pj4+Pj4+Pj4gd2hpY2ggaXMgYmFja2VkIGJ5IHRoZSBzcGVjIGF0Cj4gPiA+Pj4+Pj4+Pj4+ IGh0dHBzOi8vdXJsZGVmZW5zZS5jb20vdjMvX19odHRwczovL296bGFicy5vcmcvKnJ1c3R5L3Zp cnRpby1zcGVjL3ZpcnRpby0wLjkuNS5wZGZfXztmZyEhQUNXVjVOOU0yUlY5OWhRIWRUS216Snd3 UnNGTTdCdFN1VER1MWNObHk1bjRYQ290SDBXWW1pZHpHcUhTWHQ0MGk3WlU0M1VjTmc3R1l4Wmck IC4gVGhvdWdoIEknbSBub3Qgc3VyZQo+ID4gPj4+Pj4+Pj4+PiBpZiB0aGVyZSdzIHJlcXVlc3Qv bmVlZCB0byBzdXBwb3J0IHdpbGRlciBsZWdhY3kgdmlydGlvIHZlcnNpb25zIGVhcmxpZXIKPiA+ ID4+Pj4+Pj4+Pj4gYmV5b25kLgo+ID4gPj4+Pj4+Pj4+IEkgcGVyc29uYWxseSBmZWVsIGl0J3Mg bGVzcyB3b3JrIHRvIGFkZCBpbiBrZXJuZWwgdGhhbiB0cnkgdG8KPiA+ID4+Pj4+Pj4+PiB3b3Jr IGFyb3VuZCBpdCBpbiB1c2Vyc3BhY2UuIEphc29uIGZlZWxzIGRpZmZlcmVudGx5Lgo+ID4gPj4+ Pj4+Pj4+IE1heWJlIHBvc3QgdGhlIHBhdGNoZXMgYW5kIHRoaXMgd2lsbCBwcm92ZSB0byBKYXNv biBpdCdzIG5vdAo+ID4gPj4+Pj4+Pj4+IHRvbyB0ZXJyaWJsZT8KPiA+ID4+Pj4+Pj4+IEkgc3Vw cG9zZSBpZiB0aGUgdmRwYSB2ZW5kb3IgZG9lcyBzdXBwb3J0IDAuOTUgaW4gdGhlIGRhdGFwYXRo IGFuZCByaW5nCj4gPiA+Pj4+Pj4+PiBsYXlvdXQgbGV2ZWwgYW5kIGlzIGxpbWl0ZWQgdG8geDg2 IG9ubHksIHRoZXJlIHNob3VsZCBiZSBlYXN5IHdheSBvdXQuCj4gPiA+Pj4+Pj4+IE5vdGUgYSBz dWJ0bGUgZGlmZmVyZW5jZTogd2hhdCBtYXR0ZXJzIGlzIHRoYXQgZ3Vlc3QsIG5vdCBob3N0IGlz IHg4Ni4KPiA+ID4+Pj4+Pj4gTWF0dGVycyBmb3IgZW11bGF0b3JzIHdoaWNoIG1pZ2h0IHJlb3Jk ZXIgbWVtb3J5IGFjY2Vzc2VzLgo+ID4gPj4+Pj4+PiBJIGd1ZXNzIHRoaXMgZW5mb3JjZW1lbnQg YmVsb25ncyBpbiBRRU1VIHRoZW4/Cj4gPiA+Pj4+Pj4gUmlnaHQsIEkgbWVhbiB0byBnZXQgc3Rh cnRlZCwgdGhlIGluaXRpYWwgZ3Vlc3QgZHJpdmVyIHN1cHBvcnQgYW5kIHRoZQo+ID4gPj4+Pj4+ IGNvcnJlc3BvbmRpbmcgUUVNVSBzdXBwb3J0IGZvciB0cmFuc2l0aW9uYWwgdmRwYSBiYWNrZW5k IGNhbiBiZSBsaW1pdGVkCj4gPiA+Pj4+Pj4gdG8geDg2IGd1ZXN0L2hvc3Qgb25seS4gU2luY2Ug dGhlIGNvbmZpZyBzcGFjZSBpcyBlbXVsYXRlZCBpbiBRRU1VLCBJCj4gPiA+Pj4+Pj4gc3VwcG9z ZSBpdCdzIG5vdCBoYXJkIHRvIGVuZm9yY2UgaW4gUUVNVS4KPiA+ID4+Pj4+IEl0J3MgbW9yZSB0 aGFuIGp1c3QgY29uZmlnIHNwYWNlLCBtb3N0IGRldmljZXMgaGF2ZSBoZWFkZXJzIGJlZm9yZSB0 aGUgYnVmZmVyLgo+ID4gPj4+PiBUaGUgb3JkZXJpbmcgaW4gZGF0YXBhdGggKGRhdGEgVlFzKSB3 b3VsZCBoYXZlIHRvIHJlbHkgb24gdmVuZG9yJ3Mgc3VwcG9ydC4KPiA+ID4+Pj4gU2luY2UgT1JE RVJfUExBVEZPUk0gaXMgcHJldHR5IG5ldyAodjEuMSksIEkgZ3Vlc3MgdmRwYSBoL3cgdmVuZG9y IG5vd2FkYXlzCj4gPiA+Pj4+IGNhbi9zaG91bGQgd2VsbCBzdXBwb3J0IHRoZSBjYXNlIHdoZW4g T1JERVJfUExBVEZPUk0gaXMgbm90IGFja2VkIGJ5IHRoZQo+ID4gPj4+PiBkcml2ZXIgKGFjdHVh bGx5IHRoaXMgZmVhdHVyZSBpcyBmaWx0ZXJlZCBvdXQgYnkgdGhlIFFFTVUgdmhvc3QtdmRwYSBk cml2ZXIKPiA+ID4+Pj4gdG9kYXkpLCBldmVuIHdpdGggdjEuMCBzcGVjIGNvbmZvcm1pbmcgYW5k IG1vZGVybiBvbmx5IHZEUEEgZGV2aWNlLiBUaGUKPiA+ID4+Pj4gY29udHJvbCBWUSBpcyBpbXBs ZW1lbnRlZCBpbiBzb2Z0d2FyZSBpbiB0aGUga2VybmVsLCB3aGljaCBjYW4gYmUgZWFzaWx5Cj4g PiA+Pj4+IGFjY29tbW9kYXRlZC9maXhlZCB3aGVuIG5lZWRlZC4KPiA+ID4+Pj4KPiA+ID4+Pj4+ PiBRRU1VIGNhbiBkcml2ZSBHRVRfTEVHQUNZLAo+ID4gPj4+Pj4+IEdFVF9FTkRJQU4gZXQgYWwg aW9jdGxzIGluIGFkdmFuY2UgdG8gZ2V0IHRoZSBjYXBhYmlsaXR5IGZyb20gdGhlCj4gPiA+Pj4+ Pj4gaW5kaXZpZHVhbCB2ZW5kb3IgZHJpdmVyLiBGb3IgdGhhdCwgd2UgbmVlZCBhbm90aGVyIG5l Z290aWF0aW9uIHByb3RvY29sCj4gPiA+Pj4+Pj4gc2ltaWxhciB0byB2aG9zdF91c2VyJ3MgcHJv dG9jb2xfZmVhdHVyZXMgYmV0d2VlbiB0aGUgdmRwYSBrZXJuZWwgYW5kCj4gPiA+Pj4+Pj4gUUVN VSwgd2F5IGJlZm9yZSB0aGUgZ3Vlc3QgZHJpdmVyIGlzIGV2ZXIgcHJvYmVkIGFuZCBpdHMgZmVh dHVyZQo+ID4gPj4+Pj4+IG5lZ290aWF0aW9uIGtpY2tzIGluLiBOb3Qgc3VyZSB3ZSBuZWVkIGEg R0VUX01FTU9SWV9PUkRFUiBpb2N0bCBjYWxsCj4gPiA+Pj4+Pj4gZnJvbSB0aGUgZGV2aWNlLCBi dXQgd2UgY2FuIGFzc3VtZSB3ZWFrIG9yZGVyaW5nIGZvciBsZWdhY3kgYXQgdGhpcwo+ID4gPj4+ Pj4+IHBvaW50ICh4ODYgb25seSk/Cj4gPiA+Pj4+PiBJJ20gbG9zdCBoZXJlLCB3ZSBoYXZlIGdl dF9mZWF0dXJlcygpIHNvOgo+ID4gPj4+PiBJIGFzc3VtZSBoZXJlIHlvdSByZWZlciB0byBnZXRf ZGV2aWNlX2ZlYXR1cmVzKCkgdGhhdCBFbGkganVzdCBjaGFuZ2VkIHRoZQo+ID4gPj4+PiBuYW1l Lgo+ID4gPj4+Pj4gMSkgVkVSU0lPTl8xIG1lYW5zIHRoZSBkZXZpY2UgdXNlcyBMRSBpZiBwcm92 aWRlZCwgb3RoZXJ3aXNlIG5hdHZpZQo+ID4gPj4+Pj4gMikgT1JERVJfUExBVEZPUk0gbWVhbnMg ZGV2aWNlIHJlcXVpcmVzIHBsYXRmb3JtIG9yZGVyaW5nCj4gPiA+Pj4+Pgo+ID4gPj4+Pj4gQW55 IHJlYXNvbiBmb3IgaGF2aW5nIGEgbmV3IEFQSSBmb3IgdGhpcz8KPiA+ID4+Pj4gQXJlIHlvdSBn b2luZyB0byBlbmZvcmNlIGFsbCB2RFBBIGhhcmR3YXJlIHZlbmRvcnMgdG8gc3VwcG9ydCB0aGUK PiA+ID4+Pj4gdHJhbnNpdGlvbmFsIG1vZGVsIGZvciBsZWdhY3kgZ3Vlc3Q/Cj4gPiA+IERvIHdl IHJlYWxseSBoYXZlIG90aGVyIGNob2ljZXM/Cj4gPiA+Cj4gPiA+IEkgc3VzcGVjdCB0aGUgbGVn YWN5IGRldmljZSBpcyBuZXZlciBpbXBsZW1lbnRlZCBieSBhbnkgdmVuZG9yOgo+ID4gPgo+ID4g PiAxKSBubyB2aXJ0aW8gd2F5IHRvIGRldGVjdCBob3N0IGVuZGlhbgo+ID4gVGhpcyBpcyBldmVu IHRydWUgZm9yIHRyYW5zaXRpb25hbCBkZXZpY2UgdGhhdCBpcyBjb25mb3JtaW5nIHRvIHRoZQo+ ID4gc3BlYywgcmlnaHQ/Cj4gCj4gRm9yIGhhcmR3YXJlLCB5ZXMuCj4gCj4gPiBUaGUgdHJhbnNw b3J0IHNwZWNpZmljIHdheSB0byBkZXRlY3QgaG9zdCBlbmRpYW4gaXMgc3RpbGwKPiA+IGJlaW5n IGRpc2N1c3NlZCBhbmQgdGhlIHNwZWMgcmV2aXNpb24gaXMgbm90IGZpbmFsaXplZCB5ZXQgc28g ZmFyIGFzIEkKPiA+IHNlZS4gV2h5IHRoaXMgc3VkZGVubHkgYmVjb21lcyBhIHJlcXVpcmVtZW50 L2Jsb2NrZXIgZm9yIGgvdyB2ZW5kb3JzIHRvCj4gPiBpbXBsZW1lbnQgdGhlIHRyYW5zaXRpb25h bCBtb2RlbD8KPiAKPiBJdCdzIG5vdCBhIHN1ZGRlbiBibG9ja2VyLCB0aGUgcHJvYmxlbSBoYXMg ZXhpc3RlZCBzaW5jZSBkYXkgMCBpZiBJCj4gd2FzIG5vdCB3cm9uZy4gVGhhdCdzIHdoeSB0aGUg cHJvYmxlbSBsb29rcyBhIGxpdHRsZSBiaXQgY29tcGxpY2F0ZWQKPiBhbmQgd2h5IGl0IHdvdWxk IGJlIG11Y2ggc2ltcGxlciBpZiB3ZSBzdGljayB0byBtb2Rlcm4gZGV2aWNlcy4KPiAKPiA+IEV2 ZW4gaWYgdGhlIHNwZWMgaXMgb3V0LCB0aGlzIGlzCj4gPiBwcmV0dHkgbmV3IGFuZCBJIHN1c3Bl Y3Qgbm90IGFsbCB2ZW5kb3Igd291bGQgZm9sbG93IHJpZ2h0IGF3YXkuIEkgaG9wZQo+ID4gdGhl IHNvZnR3YXJlIGZyYW1ld29yayBjYW4gYmUgdG9sZXJhbnQgd2l0aCBoL3cgdmVuZG9ycyBub3Qg c3VwcG9ydGluZwo+ID4gaG9zdCBlbmRpYW5lc3MgKEJFIHNwZWNpZmljYWxseSkgb3Igbm90IGRl dGVjdGluZyBpdCBpZiB0aGV5IHdvdWxkIGxpa2UKPiA+IHRvIHN1cHBvcnQgYSB0cmFuc2l0aW9u YWwgZGV2aWNlIGZvciBsZWdhY3kuCj4gCj4gV2VsbCwgaWYgd2Uga25vdyB3ZSBkb24ndCB3YW50 IHRvIHN1cHBvcnQgdGhlIEJFIGhvc3QgaXQgd291bGQgYmUgZmluZS4KCkkgdGhpbmsgeW91IGd1 eXMgbWVhbiBndWVzdCBub3QgaG9zdCBoZXJlLiBTYW1lIGZvciBtZW1vcnkgb3JkZXJpbmcgZXRj LgpXaGF0IG1hdHRlcnMgaXMgd2hldGhlciBndWVzdCBoYXMgYmFycmllcnMgZXRjLgoKPiA+Cj4g PiA+IDIpIGJ5cGFzcyBJT01NVSB3aXRoIHRyYW5zbGF0ZWQgcmVxdWVzdHMKPiA+ID4gMykgUElP IHBvcnQKPiA+ID4KPiA+ID4gWWVzIHdlIGhhdmUgZW5wX3ZkcGEsIGJ1dCBpdCdzIG1vcmUgbGlr ZSBhICJ0cmFuc2l0aW9uYWwgZGV2aWNlIiBmb3IKPiA+ID4gbGVnYWN5IG9ubHkgZ3Vlc3RzLgo+ ID4gPgo+ID4gPj4gbWVhbmluZyBndWVzdCBub3QgYWNrbm93bGVkZ2luZwo+ID4gPj4+PiBWRVJT SU9OXzEgd291bGQgdXNlIHRoZSBsZWdhY3kgaW50ZXJmYWNlcyBjYXB0dXJlZCBpbiB0aGUgc3Bl YyBzZWN0aW9uIDcuNAo+ID4gPj4+PiAocmVnYXJkaW5nIHJpbmcgbGF5b3V0LCBuYXRpdmUgZW5k aWFubmVzcywgbWVzc2FnZSBmcmFtaW5nLCB2cSBhbGlnbm1lbnQgb2YKPiA+ID4+Pj4gNDA5Niwg MzJiaXQgZmVhdHVyZSwgbm8gZmVhdHVyZXNfb2sgYml0IGluIHN0YXR1cywgSU8gcG9ydCBpbnRl cmZhY2UgaS5lLgo+ID4gPj4+PiBhbGwgdGhlIHRoaW5ncykgaW5zdGVhZD8KPiA+ID4gTm90ZSB0 aGF0IHdlIG9ubHkgY2FyZSBhYm91dCB0aGUgZGF0YXBhdGgsIGNvbnRyb2wgcGF0aCBpcyBtZWRp YXRlZCBhbnlob3cuCj4gPiA+Cj4gPiA+IFNvIGZlYXR1cmVfb2sgYW5kIElPIHBvcnQgaXNuJ3Qg YW4gaXNzdWUuIFRoZSByZXN0IGxvb2tzIGxpa2UgYSBtdXN0Cj4gPiA+IGZvciB0aGUgaGFyZHdh cmUuCj4gPiBIL1cgdmVuZG9ycyBjYW4gb3B0IG91dCBub3QgaW1wbGVtZW50aW5nIHRyYW5zaXRp b25hbCBpbnRlcmZhY2VzIGF0IGFsbAo+ID4gd2hpY2ggbGltaXRzIGl0c2VsZiBhIG1vZGVybiBv bmx5IGRldmljZS4gU2V0IGVuZGlhbmVzcyBkZXRlY3Rpb24gKHZpYQo+ID4gdHJhbnNwb3J0IHNw ZWNpZmljIG1lYW5zKSBhc2lkZSwgZm9yIHZlbmRvcnMgdGhhdCB3aXNoZXMgdG8gc3VwcG9ydAo+ ID4gdHJhbnNpdGlvbmFsIGRldmljZSB3aXRoIGxlZ2FjeSBpbnRlcmZhY2UsIGlzIGl0IGEgaGFy ZCBzdG9wIHRvIGRyb3AKPiA+IHN1cHBvcnRpbmcgQkUgaG9zdCBpZiBldmVyeXRoaW5nIGVsc2Ug aXMgdGhlcmU/IFRoZSBzcGVjIHRvZGF5IGRvZXNuJ3QKPiA+IGRlZmluZSB2aXJ0aW8gc3BlY2lm aWMgbWVhbnMgdG8gZGV0ZWN0IGhvc3QgbWVtb3J5IG9yZGVyaW5nIG9yIGRldmljZQo+ID4gbWVt b3J5IGNvaGVyZW5jeSwKPiAKPiBBbnkgcmVhc29uIHRoYXQgd2UgbmVlZCB0byBjYXJlIGFib3V0 IG1lbW9yeSBjb2hlcmVuY3kgYXQgdGhlIHZpcnRpbwo+IGxldmVsLiBJJ2QgZXhwZWN0IGl0J3Mg dGhlIHRhc2sgb2YgdHJhbnNwb3J0Lgo+IAo+ID4gd2lsbCBpdCB5ZXQgYmVjb21lIGEgc3RvcHBl ciBhbm90aGVyIGRheSBmb3IgaC93Cj4gPiB2ZW5kb3IgdG8gc3VwcG9ydCBtb3JlIHBsYXRmb3Jt cz8KPiAKPiBMZXQncyBkaWZmZXJlbnRpYXRlIHZpcnRpbyBmcm9tIHZkcGEgaGVyZS4gRm9yIHZp cnRpbywgdGhlcmUncyBubyB3YXkKPiB0byBhZGQgYW55IGZlYXR1cmUgZm9yIGxlZ2FjeSBkZXZp Y2VzLiBXZSBjYW4gb25seSBhZGQgbWVtb3J5IGZlYXR1cmVzCj4gZGV0ZWN0aW5nIGZvciBtb2Rl cm4gZGV2aWNlcy4KPiAKPiBCdXQgZm9yIHZEUEEsIHdlIGNhbiBpbnRyb2R1Y2UgYW55IEFQSSB0 aGF0IGNhbiBoZWxwIHZlbmRvcnMgdG8KPiBwcmVzZW50IGEgdHJhbnNpdGlvbmFsIGRldmljZS4g QnV0IHdlIGNhbiBmb3JjZSB0aG9zZSBBUElzIHNpbmNlIGl0J3MKPiB0b28gbGF0ZSB0byBkbyB0 aGF0LiBTbyB0cmFuc2l0aW9uYWwgZGV2aWNlcyBzdXBwb3J0IGlzIG9wdGlvbmFsIGZvcgo+IHN1 cmUuCj4gCj4gPgo+ID4gPgo+ID4gPj4gTm90ZWQgd2UgZG9uJ3QgeWV0IGhhdmUgYSBzZXRfZGV2 aWNlX2ZlYXR1cmVzKCkKPiA+ID4+Pj4gdGhhdCBhbGxvd3MgdGhlIHZkcGEgZGV2aWNlIHRvIHRl bGwgd2hldGhlciBpdCBpcyBvcGVyYXRpbmcgaW4gdHJhbnNpdGlvbmFsCj4gPiA+Pj4+IG9yIG1v ZGVybi1vbmx5IG1vZGUuCj4gPiA+IFNvIHRoZSBkZXZpY2UgZmVhdHVyZSBzaG91bGQgYmUgcHJv dmlzaW9uZWQgdmlhIHRoZSBuZXRsaW5rIHByb3RvY29sLgo+ID4gU3VjaCBuZXRsaW5rIGludGVy ZmFjZSB3aWxsIG9ubHkgYmUgdXNlZCB0byBsaW1pdCBmZWF0dXJlIGV4cG9zdXJlLAo+ID4gcmln aHQ/IGkuZS4geW91IGNhbiBsaW1pdCBhIHRyYW5zaXRpb25hbCBzdXBwb3J0aW5nIHZlbmRvciBk cml2ZXIgdG8KPiA+IG9mZmVyaW5nIG1vZGVybi1vbmx5IGludGVyZmFjZSwKPiAKPiBUaGVyZSdz IG5vIHdheSBmb3IgdGhlIG1hbmFnZW1lbnQgdG8gZm9yY2UgYSBmZWF0dXJlLCBsaWtlIFZFUlNJ T05fMQo+IHZpYSB0aGUgY3VycmVudCBwcm90b2NvbC4KPiAKPiA+IGJ1dCB5b3UgbmV2ZXIgd2Fu dCB0byBtYWtlIGEgbW9kZXJuLW9ubHkKPiA+IHZlbmRvciBkcml2ZXIgdG8gc3VwcG9ydCB0cmFu c2l0aW9uYWwgKEknbSBub3Qgc3VyZSBpZiBpdCdzIGEgZ29vZCBpZGVhCj4gPiB0byBzdXBwb3J0 IGFsbCB0aGUgdHJhbnNsYXRpb24gaW4gc29mdHdhcmUsIGVzcC4gZm9yIGRhdGFwYXRoKS4KPiAK PiBZb3UgbWF5IGhpdCB0aGlzIHByb2JsZW0gZm9yIHN1cmUsIHlvdSBjYW4ndCBmb3JjZSBhbGwg dmVuZG9ycyB0bwo+IHN1cHBvcnQgdHJhbnNpdGlvbmFsIGRldmljZXMgZXNwZWNpYWxseSBjb25z aWRlcmluZyBzcGVjIHNhaWQgbGVnYWN5Cj4gaXMgb3B0aW9uYWwuIFdlIGRvbid0IHdhbnQgdG8g ZW5kIHVwIHdpdGggYSB1c2Vyc3BhY2UgY29kZSB0aGF0IGNhbgo+IG9ubHkgd29yayBmb3Igc29t ZSBzcGVjaWZpYyB2ZW5kb3JzLgo+IAo+ID4gPiBBbmQgd2hhdCB3ZSB3YW50IGlzIG5vdCAic2V0 X2RldmljZV9mZWF0dXJlKCkiIGJ1dAo+ID4gPiAic2V0X2RldmljZV9tYW5kYXRvcnlfZmVhdXRy ZSgpIiwgdGhlbiB0aGUgcGFyZW50IGNhbiBjaG9vc2UgdG8gZmFpbAo+ID4gPiB0aGUgbmVnb3Rp YXRpb24gd2hlbiBWRVJTSU9OXzEgaXMgbm90IG5lZ290aWF0ZWQuCj4gPiBUaGlzIGFzc3VtZXMg dGhlIHRyYW5zcG9ydCBzcGVjaWZpYyBkZXRlY3Rpb24gb2YgQkUgaG9zdCBpcyBpbiBwbGFjZSwK PiA+IHJpZ2h0Pwo+IAo+IEFnYWluLCB0aGUgcG9pbnQgaXMsIHdlIGNhbiBub3QgYXNzdW1lIHN1 Y2ggZGV0ZWN0aW9uIHdvcmtzIGZvciBhbGwgb2YKPiB0aGUgdmVuZG9ycy4gQW5kIGFzc3VtZSBC RSBkZXRlY3Rpb24gaXMgcmVhZHksIHdlIHN0aWxsIG5lZWQgdGhpcyBmb3IKPiBtb2Rlcm4gZGV2 aWNlcywgaXNuJ3QgaXQ/Cj4gCj4gPiBJIGFtIG5vdCBjbGVhciB3aG8gaW5pdGlhdGVzIHRoZSBz ZXRfZGV2aWNlX21hbmRhdG9yeV9mZWF1dHJlKCkKPiA+IGNhbGwsIFFFTVUgZHVyaW5nIGd1ZXN0 IGZlYXR1cmUgbmVnb3RpYXRpb24sIG9yIGFkbWluIHVzZXIgc2V0dGluZyBpdAo+ID4gYWhlYWQg dmlhIG5ldGxpbms/Cj4gCj4gTmV0bGluaywgYWN0dWFsbHksIHRoZSBzcGVjIG5lZWRzIHRvIGJl IGV4dGVuZGVkIGFzIHdlbGwsIHdlIHNhdwo+IHNpbWlsYXIgcmVxdWVzdHMgaW4gdGhlIHBhc3Qu IEUuZyB0aGVyZSBjb3VsZCBiZSBhIGRldmljZSB0aGF0IHdvcmtzCj4gaW4gYSBwYWNrZWQgbGF5 b3V0IG9ubHkuCj4gCj4gVGhhbmtzCj4gCj4gPgo+ID4gVGhhbmtzLAo+ID4gLVNpd2VpCj4gPgo+ ID4gPiAgIFFlbXUgdGhlbiBrbm93cyBmb3IKPiA+ID4gc3VyZSBpdCB0YWxrcyB0byBhIHRyYW5z aXRpb25hbCBkZXZpY2Ugb3IgbW9kZXJuIG9ubHkgZGV2aWNlLgo+ID4gPgo+ID4gPiBUaGFua3MK PiA+ID4KPiA+ID4+IEZvciBzb2Z0d2FyZSB2aXJ0aW8sIGFsbCBzdXBwb3J0IGZvciB0aGUgbGVn YWN5IHBhcnQgaW4KPiA+ID4+Pj4gYSB0cmFuc2l0aW9uYWwgbW9kZWwgaGFzIGJlZW4gYnVpbHQg dXAgdGhlcmUgYWxyZWFkeSwgaG93ZXZlciwgaXQncyBub3QgZWFzeQo+ID4gPj4+PiBmb3IgdkRQ QSB2ZW5kb3JzIHRvIGltcGxlbWVudCBhbGwgdGhlIHJlcXVpcmVtZW50cyBmb3IgYW4gYWxsLW9y LW5vdGhpbmcKPiA+ID4+Pj4gbGVnYWN5IGd1ZXN0IHN1cHBvcnQgKGJpZyBlbmRpYW4gZ3Vlc3Qg Zm9yIGV4YW1wbGUpLiBUbyB0aGVzZSB2ZW5kb3JzLCB0aGUKPiA+ID4+Pj4gbGVnYWN5IHN1cHBv cnQgd2l0aGluIGEgdHJhbnNpdGlvbmFsIG1vZGVsIGlzIG1vcmUgb2YgZmVhdHVyZSB0byB0aGVt IGFuZAo+ID4gPj4+PiBpdCdzIGJlc3QgdG8gbGVhdmUgc29tZSBmbGV4aWJpbGl0eSBmb3IgdGhl bSB0byBpbXBsZW1lbnQgcGFydGlhbCBzdXBwb3J0Cj4gPiA+Pj4+IGZvciBsZWdhY3kuIFRoYXQg aW4gdHVybiBjYWxscyBvdXQgdGhlIG5lZWQgZm9yIGEgdmhvc3QtdXNlciBwcm90b2NvbAo+ID4g Pj4+PiBmZWF0dXJlIGxpa2UgbmVnb3RpYXRpb24gQVBJIHRoYXQgY2FuIHByb2hpYml0IHRob3Nl IHVuc3VwcG9ydGVkIGd1ZXN0Cj4gPiA+Pj4+IHNldHVwcyB0byBhcyBlYXJseSBhcyBiYWNrZW5k X2luaXQgYmVmb3JlIGxhdW5jaGluZyB0aGUgVk0uCj4gPiA+Pj4gUmlnaHQuIE9mIG5vdGUgaXMg dGhlIGZhY3QgdGhhdCBpdCdzIGEgc3BlYyBidWcgd2hpY2ggSQo+ID4gPj4+IGhvcGUgeWV0IHRv IGZpeCwgdGhvdWdoIGR1ZSB0byBleGlzdGluZyBndWVzdCBjb2RlIHRoZQo+ID4gPj4+IGZpeCB3 b24ndCBiZSBjb21wbGV0ZS4KPiA+ID4+IEkgdGhvdWdodCBhdCBvbmUgcG9pbnQgeW91IHBvaW50 ZWQgb3V0IHRvIG1lIHRoYXQgdGhlIHNwZWMgZG9lcyBhbGxvdwo+ID4gPj4gY29uZmlnIHNwYWNl IHJlYWQgYmVmb3JlIGNsYWltaW5nIGZlYXR1cmVzX29rLCBhbmQgb25seSBjb25maWcgd3JpdGUK PiA+ID4+IGJlZm9yZSBmZWF0dXJlc19vayBpcyBwcm9oaWJpdGVkLiBJIGhhdmVuJ3QgcmVhZCB1 cCB0aGUgZnVsbCB0aHJlYWQgb2YKPiA+ID4+IEhhbGlsJ3MgVkVSU0lPTl8xIGZvciB0cmFuc2l0 aW9uYWwgYmlnIGVuZGlhbiBkZXZpY2UgeWV0LCBidXQgd2hhdCBpcwo+ID4gPj4gdGhlIHNwZWMg YnVnIHlvdSBob3BlIHRvIGZpeD8KPiA+ID4+Cj4gPiA+Pj4gV1JUIGlvY3RscywgT25lIHRoaW5n IHdlIGNhbiBkbyB0aG91Z2ggaXMgYWJ1c2Ugc2V0X2ZlYXR1cmVzCj4gPiA+Pj4gd2hlcmUgaXQn cyBjYWxsZWQgYnkgUUVNVSBlYXJseSBvbiB3aXRoIGp1c3QgdGhlIFZFUlNJT05fMQo+ID4gPj4+ IGJpdCBzZXQsIHRvIGRpc3Rpbmd1aXNoIGJldHdlZW4gbGVnYWN5IGFuZCBtb2Rlcm4KPiA+ID4+ PiBpbnRlcmZhY2UuIFRoaXMgYmVmb3JlIGNvbmZpZyBzcGFjZSBhY2Nlc3NlcyBhbmQgRkVBVFVS RVNfT0suCj4gPiA+Pj4KPiA+ID4+PiBIYWxpbCBoYXMgYmVlbiB3b3JraW5nIG9uIHRoaXMsIHBs cyB0YWtlIGEgbG9vayBhbmQgbWF5YmUgaGVscCBoaW0gb3V0Lgo+ID4gPj4gSW50ZXJlc3Rpbmcg dGhyZWFkLCBhbSByZWFkaW5nIG5vdyBhbmQgc2VlIGhvdyBJIG1heSBsZXZlcmFnZSBvciBoZWxw IHRoZXJlLgo+ID4gPj4KPiA+ID4+Pj4+Pj4+IEkKPiA+ID4+Pj4+Pj4+IGNoZWNrZWQgd2l0aCBF bGkgYW5kIG90aGVyIE1lbGxhbm94L05WRElBIGZvbGtzIGZvciBoYXJkd2FyZS9maXJtd2FyZSBs ZXZlbAo+ID4gPj4+Pj4+Pj4gMC45NSBzdXBwb3J0LCBpdCBzZWVtcyBhbGwgdGhlIGluZ3JlZGll bnQgaGFkIGJlZW4gdGhlcmUgYWxyZWFkeSBkYXRlZCBiYWNrCj4gPiA+Pj4+Pj4+PiB0byB0aGUg RFBESyBkYXlzLiBUaGUgb25seSBtYWpvciB0aGluZyBsaW1pdGluZyBpcyBpbiB0aGUgdkRQQSBz b2Z0d2FyZSB0aGF0Cj4gPiA+Pj4+Pj4+PiB0aGUgY3VycmVudCB2ZHBhIGNvcmUgaGFzIHRoZSBh c3N1bXB0aW9uIGFyb3VuZCBWSVJUSU9fRl9BQ0NFU1NfUExBVEZPUk0gZm9yCj4gPiA+Pj4+Pj4+ PiBhIGZldyBETUEgc2V0dXAgb3BzLCB3aGljaCBpcyB2aXJ0aW8gMS4wIG9ubHkuCj4gPiA+Pj4+ Pj4+Pgo+ID4gPj4+Pj4+Pj4+PiAyLiBzdXBwb3NlIHNvbWUgZm9ybSBvZiBsZWdhY3kgZ3Vlc3Qg c3VwcG9ydCBuZWVkcyB0byBiZSB0aGVyZSwgaG93IGRvIHdlCj4gPiA+Pj4+Pj4+Pj4+IGRlYWwg d2l0aCB0aGUgYm9ndXMgYXNzdW1wdGlvbiBiZWxvdyBpbiB2ZHBhX2dldF9jb25maWcoKSBpbiB0 aGUgc2hvcnQgdGVybT8KPiA+ID4+Pj4+Pj4+Pj4gSXQgbG9va3Mgb25lIG9mIHRoZSBpbnR1aXRp dmUgZml4IGlzIHRvIG1vdmUgdGhlIHZkcGFfc2V0X2ZlYXR1cmVzIGNhbGwgb3V0Cj4gPiA+Pj4+ Pj4+Pj4+IG9mIHZkcGFfZ2V0X2NvbmZpZygpIHRvIHZkcGFfc2V0X2NvbmZpZygpLgo+ID4gPj4+ Pj4+Pj4+Pgo+ID4gPj4+Pj4+Pj4+PiAgICAgICAgICAgICAgLyoKPiA+ID4+Pj4+Pj4+Pj4gICAg ICAgICAgICAgICAqIENvbmZpZyBhY2Nlc3NlcyBhcmVuJ3Qgc3VwcG9zZWQgdG8gdHJpZ2dlciBi ZWZvcmUgZmVhdHVyZXMgYXJlCj4gPiA+Pj4+Pj4+Pj4+IHNldC4KPiA+ID4+Pj4+Pj4+Pj4gICAg ICAgICAgICAgICAqIElmIGl0IGRvZXMgaGFwcGVuIHdlIGFzc3VtZSBhIGxlZ2FjeSBndWVzdC4K PiA+ID4+Pj4+Pj4+Pj4gICAgICAgICAgICAgICAqLwo+ID4gPj4+Pj4+Pj4+PiAgICAgICAgICAg ICAgaWYgKCF2ZGV2LT5mZWF0dXJlc192YWxpZCkKPiA+ID4+Pj4+Pj4+Pj4gICAgICAgICAgICAg ICAgICAgICAgdmRwYV9zZXRfZmVhdHVyZXModmRldiwgMCk7Cj4gPiA+Pj4+Pj4+Pj4+ICAgICAg ICAgICAgICBvcHMtPmdldF9jb25maWcodmRldiwgb2Zmc2V0LCBidWYsIGxlbik7Cj4gPiA+Pj4+ Pj4+Pj4+Cj4gPiA+Pj4+Pj4+Pj4+IEkgY2FuIHBvc3QgYSBwYXRjaCB0byBmaXggMikgaWYgdGhl cmUncyBjb25zZW5zdXMgYWxyZWFkeSByZWFjaGVkLgo+ID4gPj4+Pj4+Pj4+Pgo+ID4gPj4+Pj4+ Pj4+PiBUaGFua3MsCj4gPiA+Pj4+Pj4+Pj4+IC1TaXdlaQo+ID4gPj4+Pj4+Pj4+IEknbSBub3Qg c3VyZSBob3cgaW1wb3J0YW50IGl0IGlzIHRvIGNoYW5nZSB0aGF0Lgo+ID4gPj4+Pj4+Pj4+IElu IGFueSBjYXNlIGl0IG9ubHkgYWZmZWN0cyB0cmFuc2l0aW9uYWwgZGV2aWNlcywgcmlnaHQ/Cj4g PiA+Pj4+Pj4+Pj4gTGVnYWN5IG9ubHkgc2hvdWxkIG5vdCBjYXJlIC4uLgo+ID4gPj4+Pj4+Pj4g WWVzIEknZCBsaWtlIHRvIGRpc3Rpbmd1aXNoIGxlZ2FjeSBkcml2ZXIgKHN1cHBvc2UgaXQgaXMg MC45NSkgYWdhaW5zdCB0aGUKPiA+ID4+Pj4+Pj4+IG1vZGVybiBvbmUgaW4gYSB0cmFuc2l0aW9u YWwgZGV2aWNlIG1vZGVsIHJhdGhlciB0aGFuIGJlaW5nIGxlZ2FjeSBvbmx5Lgo+ID4gPj4+Pj4+ Pj4gVGhhdCB3YXkgYSB2MC45NSBhbmQgdjEuMCBzdXBwb3J0aW5nIHZkcGEgcGFyZW50IGNhbiBz dXBwb3J0IGJvdGggdHlwZXMgb2YKPiA+ID4+Pj4+Pj4+IGd1ZXN0cyB3aXRob3V0IGhhdmluZyB0 byByZWNvbmZpZ3VyZS4gT3IgYXJlIHlvdSBzdWdnZXN0aW5nIGxpbWl0IHRvIGxlZ2FjeQo+ID4g Pj4+Pj4+Pj4gb25seSBhdCB0aGUgdGltZSBvZiB2ZHBhIGNyZWF0aW9uIHdvdWxkIHNpbXBsaWZ5 IHRoZSBpbXBsZW1lbnRhdGlvbiBhIGxvdD8KPiA+ID4+Pj4+Pj4+Cj4gPiA+Pj4+Pj4+PiBUaGFu a3MsCj4gPiA+Pj4+Pj4+PiAtU2l3ZWkKPiA+ID4+Pj4+Pj4gSSBkb24ndCBrbm93IGZvciBzdXJl LiBUYWtlIGEgbG9vayBhdCB0aGUgd29yayBIYWxpbCB3YXMgZG9pbmcKPiA+ID4+Pj4+Pj4gdG8g dHJ5IGFuZCBzdXBwb3J0IHRyYW5zaXRpb25hbCBkZXZpY2VzIHdpdGggQkUgZ3Vlc3RzLgo+ID4g Pj4+Pj4+IEhtbW0sIHdlIGNhbiBoYXZlIHRob3NlIGVuZGlhbm5lc3MgaW9jdGxzIGRlZmluZWQg YnV0IHRoZSBpbml0aWFsIFFFTVUKPiA+ID4+Pj4+PiBpbXBsZW1lbnRhdGlvbiBjYW4gYmUgc3Rh cnRlZCB0byBzdXBwb3J0IHg4NiBndWVzdC9ob3N0IHdpdGggbGl0dGxlCj4gPiA+Pj4+Pj4gZW5k aWFuIGFuZCB3ZWFrIG1lbW9yeSBvcmRlcmluZyBmaXJzdC4gVGhlIHJlYWwgdHJpY2sgaXMgdG8g ZGV0ZWN0Cj4gPiA+Pj4+Pj4gbGVnYWN5IGd1ZXN0IC0gSSBhbSBub3Qgc3VyZSBpZiBpdCdzIGZl YXNpYmxlIHRvIHNoaWZ0IGFsbCB0aGUgbGVnYWN5Cj4gPiA+Pj4+Pj4gZGV0ZWN0aW9uIHdvcmsg dG8gUUVNVSwgb3IgdGhlIGtlcm5lbCBoYXMgdG8gYmUgcGFydCBvZiB0aGUgZGV0ZWN0aW9uCj4g PiA+Pj4+Pj4gKGUuZy4gdGhlIGtpY2sgYmVmb3JlIERSSVZFUl9PSyB0aGluZyB3ZSBoYXZlIHRv IGR1cGxpY2F0ZSB0aGUgdHJhY2tpbmcKPiA+ID4+Pj4+PiBlZmZvcnQgaW4gUUVNVSkgYXMgd2Vs bC4gTGV0IG1lIHRha2UgYSBmdXJ0aGVyIGxvb2sgYW5kIGdldCBiYWNrLgo+ID4gPj4+Pj4gTWlj aGFlbCBtYXkgdGhpbmsgZGlmZmVyZW50bHkgYnV0IEkgdGhpbmsgZG9pbmcgdGhpcyBpbiBRZW11 IGlzIG11Y2ggZWFzaWVyLgo+ID4gPj4+PiBJIHRoaW5rIHRoZSBrZXkgaXMgd2hldGhlciB3ZSBw b3NpdGlvbiBlbXVsYXRpbmcgbGVnYWN5IGludGVyZmFjZXMgaW4gUUVNVQo+ID4gPj4+PiBkb2lu ZyB0cmFuc2xhdGlvbiBvbiB0b3Agb2YgYSB2MS4wIG1vZGVybi1vbmx5IGRldmljZSBpbiB0aGUg a2VybmVsLCBvciB3ZQo+ID4gPj4+PiBhbGxvdyB2ZHBhIGNvcmUgKG9yIHlvdSBjYW4gc2F5IHZo b3N0LXZkcGEpIGFuZCB2ZW5kb3IgZHJpdmVyIHRvIHN1cHBvcnQgYQo+ID4gPj4+PiB0cmFuc2l0 aW9uYWwgbW9kZWwgaW4gdGhlIGtlcm5lbCB0aGF0IGlzIGFibGUgdG8gd29yayBmb3IgYm90aCB2 MC45NSBhbmQKPiA+ID4+Pj4gdjEuMCBkcml2ZXJzLCB3aXRoIHNvbWUgc2xpZ2h0IGFpZCBmcm9t IFFFTVUgZm9yCj4gPiA+Pj4+IGRldGVjdGluZy9lbXVsYXRpb24vc2hhZG93aW5nIChmb3IgZS5n IENWUSwgSS9PIHBvcnQgcmVsYXkpLiBJIGd1ZXNzIGZvciB0aGUKPiA+ID4+Pj4gZm9ybWVyIHdl IHN0aWxsIHJlbHkgb24gdmVuZG9yIGZvciBhIHBlcmZvcm1hbnQgZGF0YSB2cXMgaW1wbGVtZW50 YXRpb24sCj4gPiA+Pj4+IGxlYXZpbmcgdGhlIHF1ZXN0aW9uIHRvIHdoYXQgaXQgbWF5IGVuZCB1 cCBldmVudHVhbGx5IGluIHRoZSBrZXJuZWwgaXMKPiA+ID4+Pj4gZWZmZWN0aXZlbHkgdGhlIGxh dHRlcikuCj4gPiA+Pj4+Cj4gPiA+Pj4+IFRoYW5rcywKPiA+ID4+Pj4gLVNpd2VpCj4gPiA+Pj4g TXkgc3VnZ2VzdGlvbiBpcyBwb3N0IHRoZSBrZXJuZWwgcGF0Y2hlcywgYW5kIHdlIGNhbiBldmFs dWF0ZQo+ID4gPj4+IGhvdyBtdWNoIHdvcmsgdGhleSBhcmUuCj4gPiA+PiBUaGFua3MgZm9yIHRo ZSBmZWVkYmFjay4gSSB3aWxsIHRha2Ugc29tZSByZWFkIHRoZW4gZ2V0IGJhY2ssIHByb2JhYmx5 Cj4gPiA+PiBhZnRlciB0aGUgd2ludGVyIGJyZWFrLiBTdGF5IHR1bmVkLgo+ID4gPj4KPiA+ID4+ IFRoYW5rcywKPiA+ID4+IC1TaXdlaQo+ID4gPj4KPiA+ID4+Pj4+IFRoYW5rcwo+ID4gPj4+Pj4K PiA+ID4+Pj4+Cj4gPiA+Pj4+Pgo+ID4gPj4+Pj4+IE1lYW53aGlsZSwgSSdsbCBjaGVjayBpbnRl cm5hbGx5IHRvIHNlZSBpZiBhIGxlZ2FjeSBvbmx5IG1vZGVsIHdvdWxkCj4gPiA+Pj4+Pj4gd29y ay4gVGhhbmtzLgo+ID4gPj4+Pj4+Cj4gPiA+Pj4+Pj4gVGhhbmtzLAo+ID4gPj4+Pj4+IC1TaXdl aQo+ID4gPj4+Pj4+Cj4gPiA+Pj4+Pj4KPiA+ID4+Pj4+Pj4+Pj4gT24gMy8yLzIwMjEgMjo1MyBB TSwgSmFzb24gV2FuZyB3cm90ZToKPiA+ID4+Pj4+Pj4+Pj4+IE9uIDIwMjEvMy8yIDU6NDcg5LiL 5Y2ILCBNaWNoYWVsIFMuIFRzaXJraW4gd3JvdGU6Cj4gPiA+Pj4+Pj4+Pj4+Pj4gT24gTW9uLCBN YXIgMDEsIDIwMjEgYXQgMTE6NTY6NTBBTSArMDgwMCwgSmFzb24gV2FuZyB3cm90ZToKPiA+ID4+ Pj4+Pj4+Pj4+Pj4gT24gMjAyMS8zLzEgNTozNCDkuIrljYgsIE1pY2hhZWwgUy4gVHNpcmtpbiB3 cm90ZToKPiA+ID4+Pj4+Pj4+Pj4+Pj4+IE9uIFdlZCwgRmViIDI0LCAyMDIxIGF0IDEwOjI0OjQx QU0gLTA4MDAsIFNpLVdlaSBMaXUgd3JvdGU6Cj4gPiA+Pj4+Pj4+Pj4+Pj4+Pj4+IERldGVjdGlu ZyBpdCBpc24ndCBlbm91Z2ggdGhvdWdoLCB3ZSB3aWxsIG5lZWQgYSBuZXcgaW9jdGwgdG8gbm90 aWZ5Cj4gPiA+Pj4+Pj4+Pj4+Pj4+Pj4+IHRoZSBrZXJuZWwgdGhhdCBpdCdzIGEgbGVnYWN5IGd1 ZXN0LiBVZ2ggOigKPiA+ID4+Pj4+Pj4+Pj4+Pj4+PiBXZWxsLCBhbHRob3VnaCBJIHRoaW5rIGFk ZGluZyBhbiBpb2N0bCBpcyBkb2FibGUsIG1heSBJCj4gPiA+Pj4+Pj4+Pj4+Pj4+Pj4ga25vdyB3 aGF0IHRoZSB1c2UKPiA+ID4+Pj4+Pj4+Pj4+Pj4+PiBjYXNlIHRoZXJlIHdpbGwgYmUgZm9yIGtl cm5lbCB0byBsZXZlcmFnZSBzdWNoIGluZm8KPiA+ID4+Pj4+Pj4+Pj4+Pj4+PiBkaXJlY3RseT8g SXMgdGhlcmUgYQo+ID4gPj4+Pj4+Pj4+Pj4+Pj4+IGNhc2UgUUVNVSBjYW4ndCBkbyB3aXRoIGRl ZGljYXRlIGlvY3RscyBsYXRlciBpZiB0aGVyZSdzIGluZGVlZAo+ID4gPj4+Pj4+Pj4+Pj4+Pj4+ IGRpZmZlcmVudGlhdGlvbiAobGVnYWN5IHYucy4gbW9kZXJuKSBuZWVkZWQ/Cj4gPiA+Pj4+Pj4+ Pj4+Pj4+PiBCVFcgYSBnb29kIEFQSSBjb3VsZCBiZQo+ID4gPj4+Pj4+Pj4+Pj4+Pj4KPiA+ID4+ Pj4+Pj4+Pj4+Pj4+ICNkZWZpbmUgVkhPU1RfU0VUX0VORElBTiBfSU9XKFZIT1NUX1ZJUlRJTywg PywgaW50KQo+ID4gPj4+Pj4+Pj4+Pj4+Pj4gI2RlZmluZSBWSE9TVF9HRVRfRU5ESUFOIF9JT1co VkhPU1RfVklSVElPLCA/LCBpbnQpCj4gPiA+Pj4+Pj4+Pj4+Pj4+Pgo+ID4gPj4+Pj4+Pj4+Pj4+ Pj4gd2UgZGlkIGl0IHBlciB2cmluZyBidXQgbWF5YmUgdGhhdCB3YXMgYSBtaXN0YWtlIC4uLgo+ ID4gPj4+Pj4+Pj4+Pj4+PiBBY3R1YWxseSwgSSB3b25kZXIgd2hldGhlciBpdCdzIGdvb2QgdGlt ZSB0byBqdXN0IG5vdCBzdXBwb3J0Cj4gPiA+Pj4+Pj4+Pj4+Pj4+IGxlZ2FjeSBkcml2ZXIKPiA+ ID4+Pj4+Pj4+Pj4+Pj4gZm9yIHZEUEEuIENvbnNpZGVyOgo+ID4gPj4+Pj4+Pj4+Pj4+Pgo+ID4g Pj4+Pj4+Pj4+Pj4+PiAxKSBJdCdzIGRlZmluaXRpb24gaXMgbm8tbm9ybWF0aXZlCj4gPiA+Pj4+ Pj4+Pj4+Pj4+IDIpIEEgbG90IG9mIGJ1ZHJlbiBvZiBjb2Rlcwo+ID4gPj4+Pj4+Pj4+Pj4+Pgo+ ID4gPj4+Pj4+Pj4+Pj4+PiBTbyBxZW11IGNhbiBzdGlsbCBwcmVzZW50IHRoZSBsZWdhY3kgZGV2 aWNlIHNpbmNlIHRoZSBjb25maWcKPiA+ID4+Pj4+Pj4+Pj4+Pj4gc3BhY2Ugb3Igb3RoZXIKPiA+ ID4+Pj4+Pj4+Pj4+Pj4gc3R1ZmZzIHRoYXQgaXMgcHJlc2VudGVkIGJ5IHZob3N0LXZEUEEgaXMg bm90IGV4cGVjdGVkIHRvIGJlCj4gPiA+Pj4+Pj4+Pj4+Pj4+IGFjY2Vzc2VkIGJ5Cj4gPiA+Pj4+ Pj4+Pj4+Pj4+IGd1ZXN0IGRpcmVjdGx5LiBRZW11IGNhbiBkbyB0aGUgZW5kaWFuIGNvbnZlcnNp b24gd2hlbiBuZWNlc3NhcnkKPiA+ID4+Pj4+Pj4+Pj4+Pj4gaW4gdGhpcwo+ID4gPj4+Pj4+Pj4+ Pj4+PiBjYXNlPwo+ID4gPj4+Pj4+Pj4+Pj4+Pgo+ID4gPj4+Pj4+Pj4+Pj4+PiBUaGFua3MKPiA+ ID4+Pj4+Pj4+Pj4+Pj4KPiA+ID4+Pj4+Pj4+Pj4+PiBPdmVyYWxsIEkgd291bGQgYmUgZmluZSB3 aXRoIHRoaXMgYXBwcm9hY2ggYnV0IHdlIG5lZWQgdG8gYXZvaWQgYnJlYWtpbmcKPiA+ID4+Pj4+ Pj4+Pj4+PiB3b3JraW5nIHVzZXJzcGFjZSwgcWVtdSByZWxlYXNlcyB3aXRoIHZkcGEgc3VwcG9y dCBhcmUgb3V0IHRoZXJlIGFuZAo+ID4gPj4+Pj4+Pj4+Pj4+IHNlZW0gdG8gd29yayBmb3IgcGVv cGxlLiBBbnkgY2hhbmdlcyBuZWVkIHRvIHRha2UgdGhhdCBpbnRvIGFjY291bnQKPiA+ID4+Pj4+ Pj4+Pj4+PiBhbmQgZG9jdW1lbnQgY29tcGF0aWJpbGl0eSBjb25jZXJucy4KPiA+ID4+Pj4+Pj4+ Pj4+IEFncmVlLCBsZXQgbWUgY2hlY2suCj4gPiA+Pj4+Pj4+Pj4+Pgo+ID4gPj4+Pj4+Pj4+Pj4K PiA+ID4+Pj4+Pj4+Pj4+PiAgICAgICAgSSBub3RlIHRoYXQgYW55IGhhcmR3YXJlCj4gPiA+Pj4+ Pj4+Pj4+Pj4gaW1wbGVtZW50YXRpb24gaXMgYWxyZWFkeSBicm9rZW4gZm9yIGxlZ2FjeSBleGNl cHQgb24gcGxhdGZvcm1zIHdpdGgKPiA+ID4+Pj4+Pj4+Pj4+PiBzdHJvbmcgb3JkZXJpbmcgd2hp Y2ggbWlnaHQgYmUgaGVscGZ1bCBpbiByZWR1Y2luZyB0aGUgc2NvcGUuCj4gPiA+Pj4+Pj4+Pj4+ PiBZZXMuCj4gPiA+Pj4+Pj4+Pj4+Pgo+ID4gPj4+Pj4+Pj4+Pj4gVGhhbmtzCj4gPiA+Pj4+Pj4+ Pj4+Pgo+ID4gPj4+Pj4+Pj4+Pj4KPiA+CgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fXwpWaXJ0dWFsaXphdGlvbiBtYWlsaW5nIGxpc3QKVmlydHVhbGl6YXRp b25AbGlzdHMubGludXgtZm91bmRhdGlvbi5vcmcKaHR0cHM6Ly9saXN0cy5saW51eGZvdW5kYXRp b24ub3JnL21haWxtYW4vbGlzdGluZm8vdmlydHVhbGl6YXRpb24= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE489C433F5 for ; Fri, 17 Dec 2021 02:01:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232180AbhLQCBJ (ORCPT ); Thu, 16 Dec 2021 21:01:09 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:55514 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231336AbhLQCBH (ORCPT ); Thu, 16 Dec 2021 21:01:07 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1639706467; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gbhzR/ta5wJM0NZ8+LJz1rln0Vg8UKqxlqmsFHgHsdY=; b=BhX/Sk0q7+Rj63zIzOvmlVhbjyZHhhCBdEyM/DI3T/K74dg66hr++9EA2VjOGZhqqkBrGs oONozh2Bwv3KbbVvTa1ooXk6f7xGuM/AwjupA4FPkj1abWPmQeGAw0lQjHU3IT8I4tGe4A MbD8jswbL634nwEG3wSF4oAD55wZ1/Q= Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-277-WL4NXVYQP-WoSKVr1xQjPw-1; Thu, 16 Dec 2021 21:01:05 -0500 X-MC-Unique: WL4NXVYQP-WoSKVr1xQjPw-1 Received: by mail-ed1-f71.google.com with SMTP id l11-20020a056402254b00b003f6a9bd7e81so552275edb.13 for ; Thu, 16 Dec 2021 18:01:05 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=gbhzR/ta5wJM0NZ8+LJz1rln0Vg8UKqxlqmsFHgHsdY=; b=3oaq1fJi46vw9N1CBy4+JYq5k2qVGY2Buv4RZUdLTYoUyhyQBPJH/ny/qQ+PBTbdm1 cYWV9OKO1mOS/zCb2DW/O/LfO3xi6jeZqSwtIqX532HNSRtSfVpd+VpqONLwaPGMGka7 6G2srO1hZHdPjA2XXq2k1sOUP1MFomj/6Bjf5vMRb3oNTLN/KQL3zLgxVjQcEsQM+hYK 5RYpUYGwtW5O0Fvd2II0tx1P9hr5b0uSoT7j1Ni1v4vUisQiWknbT/cQg2RGw5Hm0Enh Bl8hrHNRhMM2MJ0MsxiFjikW1cv1Ypoe5eDINmP75w4wmdybPakM2jM0Ra7wEUcnbb4Q 1Y+A== X-Gm-Message-State: AOAM533J7mQqx3vVHakhEerSLS4TEUgXVfBvVhCn/1ugoX5pRs+k+tAa cSWx6gLexPTso8oVLpf+hh6Ah77W0tD5OnxBIi+O3o0F2uV5etmGeS/lsQxOu+cUaKBgZMCm490 sW2gI6qUnegimvowJViKf/4ed X-Received: by 2002:a17:907:1c89:: with SMTP id nb9mr309670ejc.408.1639706464314; Thu, 16 Dec 2021 18:01:04 -0800 (PST) X-Google-Smtp-Source: ABdhPJyJOVB8t1MYtjdN8SJk8prhfOCZMdXtMw9SPSd0gMBtUijDWi/l/Iwya9JaM7Wm6Vd6RmlDzw== X-Received: by 2002:a17:907:1c89:: with SMTP id nb9mr309645ejc.408.1639706463952; Thu, 16 Dec 2021 18:01:03 -0800 (PST) Received: from redhat.com ([2.55.30.56]) by smtp.gmail.com with ESMTPSA id nc24sm2401836ejc.94.2021.12.16.18.01.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Dec 2021 18:01:03 -0800 (PST) Date: Thu, 16 Dec 2021 21:00:59 -0500 From: "Michael S. Tsirkin" To: Jason Wang Cc: Si-Wei Liu , Eli Cohen , linux-kernel , virtualization , netdev Subject: Re: vdpa legacy guest support (was Re: [PATCH] vdpa/mlx5: set_features should allow reset to zero) Message-ID: <20211216205958-mutt-send-email-mst@kernel.org> References: <20211214000245-mutt-send-email-mst@kernel.org> <4fc43d0f-da9e-ce16-1f26-9f0225239b75@oracle.com> <6eaf672c-cc86-b5bf-5b74-c837affeb6e1@oracle.com> <20211215162917-mutt-send-email-mst@kernel.org> <71d2a69c-94a7-76b5-2971-570026760bf0@oracle.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Dec 17, 2021 at 09:57:38AM +0800, Jason Wang wrote: > On Fri, Dec 17, 2021 at 6:32 AM Si-Wei Liu wrote: > > > > > > > > On 12/15/2021 6:53 PM, Jason Wang wrote: > > > On Thu, Dec 16, 2021 at 10:02 AM Si-Wei Liu wrote: > > >> > > >> > > >> On 12/15/2021 1:33 PM, Michael S. Tsirkin wrote: > > >>> On Wed, Dec 15, 2021 at 12:52:20PM -0800, Si-Wei Liu wrote: > > >>>> On 12/14/2021 6:06 PM, Jason Wang wrote: > > >>>>> On Wed, Dec 15, 2021 at 9:05 AM Si-Wei Liu wrote: > > >>>>>> On 12/13/2021 9:06 PM, Michael S. Tsirkin wrote: > > >>>>>>> On Mon, Dec 13, 2021 at 05:59:45PM -0800, Si-Wei Liu wrote: > > >>>>>>>> On 12/12/2021 1:26 AM, Michael S. Tsirkin wrote: > > >>>>>>>>> On Fri, Dec 10, 2021 at 05:44:15PM -0800, Si-Wei Liu wrote: > > >>>>>>>>>> Sorry for reviving this ancient thread. I was kinda lost for the conclusion > > >>>>>>>>>> it ended up with. I have the following questions, > > >>>>>>>>>> > > >>>>>>>>>> 1. legacy guest support: from the past conversations it doesn't seem the > > >>>>>>>>>> support will be completely dropped from the table, is my understanding > > >>>>>>>>>> correct? Actually we're interested in supporting virtio v0.95 guest for x86, > > >>>>>>>>>> which is backed by the spec at > > >>>>>>>>>> https://urldefense.com/v3/__https://ozlabs.org/*rusty/virtio-spec/virtio-0.9.5.pdf__;fg!!ACWV5N9M2RV99hQ!dTKmzJwwRsFM7BtSuTDu1cNly5n4XCotH0WYmidzGqHSXt40i7ZU43UcNg7GYxZg$ . Though I'm not sure > > >>>>>>>>>> if there's request/need to support wilder legacy virtio versions earlier > > >>>>>>>>>> beyond. > > >>>>>>>>> I personally feel it's less work to add in kernel than try to > > >>>>>>>>> work around it in userspace. Jason feels differently. > > >>>>>>>>> Maybe post the patches and this will prove to Jason it's not > > >>>>>>>>> too terrible? > > >>>>>>>> I suppose if the vdpa vendor does support 0.95 in the datapath and ring > > >>>>>>>> layout level and is limited to x86 only, there should be easy way out. > > >>>>>>> Note a subtle difference: what matters is that guest, not host is x86. > > >>>>>>> Matters for emulators which might reorder memory accesses. > > >>>>>>> I guess this enforcement belongs in QEMU then? > > >>>>>> Right, I mean to get started, the initial guest driver support and the > > >>>>>> corresponding QEMU support for transitional vdpa backend can be limited > > >>>>>> to x86 guest/host only. Since the config space is emulated in QEMU, I > > >>>>>> suppose it's not hard to enforce in QEMU. > > >>>>> It's more than just config space, most devices have headers before the buffer. > > >>>> The ordering in datapath (data VQs) would have to rely on vendor's support. > > >>>> Since ORDER_PLATFORM is pretty new (v1.1), I guess vdpa h/w vendor nowadays > > >>>> can/should well support the case when ORDER_PLATFORM is not acked by the > > >>>> driver (actually this feature is filtered out by the QEMU vhost-vdpa driver > > >>>> today), even with v1.0 spec conforming and modern only vDPA device. The > > >>>> control VQ is implemented in software in the kernel, which can be easily > > >>>> accommodated/fixed when needed. > > >>>> > > >>>>>> QEMU can drive GET_LEGACY, > > >>>>>> GET_ENDIAN et al ioctls in advance to get the capability from the > > >>>>>> individual vendor driver. For that, we need another negotiation protocol > > >>>>>> similar to vhost_user's protocol_features between the vdpa kernel and > > >>>>>> QEMU, way before the guest driver is ever probed and its feature > > >>>>>> negotiation kicks in. Not sure we need a GET_MEMORY_ORDER ioctl call > > >>>>>> from the device, but we can assume weak ordering for legacy at this > > >>>>>> point (x86 only)? > > >>>>> I'm lost here, we have get_features() so: > > >>>> I assume here you refer to get_device_features() that Eli just changed the > > >>>> name. > > >>>>> 1) VERSION_1 means the device uses LE if provided, otherwise natvie > > >>>>> 2) ORDER_PLATFORM means device requires platform ordering > > >>>>> > > >>>>> Any reason for having a new API for this? > > >>>> Are you going to enforce all vDPA hardware vendors to support the > > >>>> transitional model for legacy guest? > > > Do we really have other choices? > > > > > > I suspect the legacy device is never implemented by any vendor: > > > > > > 1) no virtio way to detect host endian > > This is even true for transitional device that is conforming to the > > spec, right? > > For hardware, yes. > > > The transport specific way to detect host endian is still > > being discussed and the spec revision is not finalized yet so far as I > > see. Why this suddenly becomes a requirement/blocker for h/w vendors to > > implement the transitional model? > > It's not a sudden blocker, the problem has existed since day 0 if I > was not wrong. That's why the problem looks a little bit complicated > and why it would be much simpler if we stick to modern devices. > > > Even if the spec is out, this is > > pretty new and I suspect not all vendor would follow right away. I hope > > the software framework can be tolerant with h/w vendors not supporting > > host endianess (BE specifically) or not detecting it if they would like > > to support a transitional device for legacy. > > Well, if we know we don't want to support the BE host it would be fine. I think you guys mean guest not host here. Same for memory ordering etc. What matters is whether guest has barriers etc. > > > > > 2) bypass IOMMU with translated requests > > > 3) PIO port > > > > > > Yes we have enp_vdpa, but it's more like a "transitional device" for > > > legacy only guests. > > > > > >> meaning guest not acknowledging > > >>>> VERSION_1 would use the legacy interfaces captured in the spec section 7.4 > > >>>> (regarding ring layout, native endianness, message framing, vq alignment of > > >>>> 4096, 32bit feature, no features_ok bit in status, IO port interface i.e. > > >>>> all the things) instead? > > > Note that we only care about the datapath, control path is mediated anyhow. > > > > > > So feature_ok and IO port isn't an issue. The rest looks like a must > > > for the hardware. > > H/W vendors can opt out not implementing transitional interfaces at all > > which limits itself a modern only device. Set endianess detection (via > > transport specific means) aside, for vendors that wishes to support > > transitional device with legacy interface, is it a hard stop to drop > > supporting BE host if everything else is there? The spec today doesn't > > define virtio specific means to detect host memory ordering or device > > memory coherency, > > Any reason that we need to care about memory coherency at the virtio > level. I'd expect it's the task of transport. > > > will it yet become a stopper another day for h/w > > vendor to support more platforms? > > Let's differentiate virtio from vdpa here. For virtio, there's no way > to add any feature for legacy devices. We can only add memory features > detecting for modern devices. > > But for vDPA, we can introduce any API that can help vendors to > present a transitional device. But we can force those APIs since it's > too late to do that. So transitional devices support is optional for > sure. > > > > > > > > >> Noted we don't yet have a set_device_features() > > >>>> that allows the vdpa device to tell whether it is operating in transitional > > >>>> or modern-only mode. > > > So the device feature should be provisioned via the netlink protocol. > > Such netlink interface will only be used to limit feature exposure, > > right? i.e. you can limit a transitional supporting vendor driver to > > offering modern-only interface, > > There's no way for the management to force a feature, like VERSION_1 > via the current protocol. > > > but you never want to make a modern-only > > vendor driver to support transitional (I'm not sure if it's a good idea > > to support all the translation in software, esp. for datapath). > > You may hit this problem for sure, you can't force all vendors to > support transitional devices especially considering spec said legacy > is optional. We don't want to end up with a userspace code that can > only work for some specific vendors. > > > > And what we want is not "set_device_feature()" but > > > "set_device_mandatory_feautre()", then the parent can choose to fail > > > the negotiation when VERSION_1 is not negotiated. > > This assumes the transport specific detection of BE host is in place, > > right? > > Again, the point is, we can not assume such detection works for all of > the vendors. And assume BE detection is ready, we still need this for > modern devices, isn't it? > > > I am not clear who initiates the set_device_mandatory_feautre() > > call, QEMU during guest feature negotiation, or admin user setting it > > ahead via netlink? > > Netlink, actually, the spec needs to be extended as well, we saw > similar requests in the past. E.g there could be a device that works > in a packed layout only. > > Thanks > > > > > Thanks, > > -Siwei > > > > > Qemu then knows for > > > sure it talks to a transitional device or modern only device. > > > > > > Thanks > > > > > >> For software virtio, all support for the legacy part in > > >>>> a transitional model has been built up there already, however, it's not easy > > >>>> for vDPA vendors to implement all the requirements for an all-or-nothing > > >>>> legacy guest support (big endian guest for example). To these vendors, the > > >>>> legacy support within a transitional model is more of feature to them and > > >>>> it's best to leave some flexibility for them to implement partial support > > >>>> for legacy. That in turn calls out the need for a vhost-user protocol > > >>>> feature like negotiation API that can prohibit those unsupported guest > > >>>> setups to as early as backend_init before launching the VM. > > >>> Right. Of note is the fact that it's a spec bug which I > > >>> hope yet to fix, though due to existing guest code the > > >>> fix won't be complete. > > >> I thought at one point you pointed out to me that the spec does allow > > >> config space read before claiming features_ok, and only config write > > >> before features_ok is prohibited. I haven't read up the full thread of > > >> Halil's VERSION_1 for transitional big endian device yet, but what is > > >> the spec bug you hope to fix? > > >> > > >>> WRT ioctls, One thing we can do though is abuse set_features > > >>> where it's called by QEMU early on with just the VERSION_1 > > >>> bit set, to distinguish between legacy and modern > > >>> interface. This before config space accesses and FEATURES_OK. > > >>> > > >>> Halil has been working on this, pls take a look and maybe help him out. > > >> Interesting thread, am reading now and see how I may leverage or help there. > > >> > > >>>>>>>> I > > >>>>>>>> checked with Eli and other Mellanox/NVDIA folks for hardware/firmware level > > >>>>>>>> 0.95 support, it seems all the ingredient had been there already dated back > > >>>>>>>> to the DPDK days. The only major thing limiting is in the vDPA software that > > >>>>>>>> the current vdpa core has the assumption around VIRTIO_F_ACCESS_PLATFORM for > > >>>>>>>> a few DMA setup ops, which is virtio 1.0 only. > > >>>>>>>> > > >>>>>>>>>> 2. suppose some form of legacy guest support needs to be there, how do we > > >>>>>>>>>> deal with the bogus assumption below in vdpa_get_config() in the short term? > > >>>>>>>>>> It looks one of the intuitive fix is to move the vdpa_set_features call out > > >>>>>>>>>> of vdpa_get_config() to vdpa_set_config(). > > >>>>>>>>>> > > >>>>>>>>>> /* > > >>>>>>>>>> * Config accesses aren't supposed to trigger before features are > > >>>>>>>>>> set. > > >>>>>>>>>> * If it does happen we assume a legacy guest. > > >>>>>>>>>> */ > > >>>>>>>>>> if (!vdev->features_valid) > > >>>>>>>>>> vdpa_set_features(vdev, 0); > > >>>>>>>>>> ops->get_config(vdev, offset, buf, len); > > >>>>>>>>>> > > >>>>>>>>>> I can post a patch to fix 2) if there's consensus already reached. > > >>>>>>>>>> > > >>>>>>>>>> Thanks, > > >>>>>>>>>> -Siwei > > >>>>>>>>> I'm not sure how important it is to change that. > > >>>>>>>>> In any case it only affects transitional devices, right? > > >>>>>>>>> Legacy only should not care ... > > >>>>>>>> Yes I'd like to distinguish legacy driver (suppose it is 0.95) against the > > >>>>>>>> modern one in a transitional device model rather than being legacy only. > > >>>>>>>> That way a v0.95 and v1.0 supporting vdpa parent can support both types of > > >>>>>>>> guests without having to reconfigure. Or are you suggesting limit to legacy > > >>>>>>>> only at the time of vdpa creation would simplify the implementation a lot? > > >>>>>>>> > > >>>>>>>> Thanks, > > >>>>>>>> -Siwei > > >>>>>>> I don't know for sure. Take a look at the work Halil was doing > > >>>>>>> to try and support transitional devices with BE guests. > > >>>>>> Hmmm, we can have those endianness ioctls defined but the initial QEMU > > >>>>>> implementation can be started to support x86 guest/host with little > > >>>>>> endian and weak memory ordering first. The real trick is to detect > > >>>>>> legacy guest - I am not sure if it's feasible to shift all the legacy > > >>>>>> detection work to QEMU, or the kernel has to be part of the detection > > >>>>>> (e.g. the kick before DRIVER_OK thing we have to duplicate the tracking > > >>>>>> effort in QEMU) as well. Let me take a further look and get back. > > >>>>> Michael may think differently but I think doing this in Qemu is much easier. > > >>>> I think the key is whether we position emulating legacy interfaces in QEMU > > >>>> doing translation on top of a v1.0 modern-only device in the kernel, or we > > >>>> allow vdpa core (or you can say vhost-vdpa) and vendor driver to support a > > >>>> transitional model in the kernel that is able to work for both v0.95 and > > >>>> v1.0 drivers, with some slight aid from QEMU for > > >>>> detecting/emulation/shadowing (for e.g CVQ, I/O port relay). I guess for the > > >>>> former we still rely on vendor for a performant data vqs implementation, > > >>>> leaving the question to what it may end up eventually in the kernel is > > >>>> effectively the latter). > > >>>> > > >>>> Thanks, > > >>>> -Siwei > > >>> My suggestion is post the kernel patches, and we can evaluate > > >>> how much work they are. > > >> Thanks for the feedback. I will take some read then get back, probably > > >> after the winter break. Stay tuned. > > >> > > >> Thanks, > > >> -Siwei > > >> > > >>>>> Thanks > > >>>>> > > >>>>> > > >>>>> > > >>>>>> Meanwhile, I'll check internally to see if a legacy only model would > > >>>>>> work. Thanks. > > >>>>>> > > >>>>>> Thanks, > > >>>>>> -Siwei > > >>>>>> > > >>>>>> > > >>>>>>>>>> On 3/2/2021 2:53 AM, Jason Wang wrote: > > >>>>>>>>>>> On 2021/3/2 5:47 下午, Michael S. Tsirkin wrote: > > >>>>>>>>>>>> On Mon, Mar 01, 2021 at 11:56:50AM +0800, Jason Wang wrote: > > >>>>>>>>>>>>> On 2021/3/1 5:34 上午, Michael S. Tsirkin wrote: > > >>>>>>>>>>>>>> On Wed, Feb 24, 2021 at 10:24:41AM -0800, Si-Wei Liu wrote: > > >>>>>>>>>>>>>>>> Detecting it isn't enough though, we will need a new ioctl to notify > > >>>>>>>>>>>>>>>> the kernel that it's a legacy guest. Ugh :( > > >>>>>>>>>>>>>>> Well, although I think adding an ioctl is doable, may I > > >>>>>>>>>>>>>>> know what the use > > >>>>>>>>>>>>>>> case there will be for kernel to leverage such info > > >>>>>>>>>>>>>>> directly? Is there a > > >>>>>>>>>>>>>>> case QEMU can't do with dedicate ioctls later if there's indeed > > >>>>>>>>>>>>>>> differentiation (legacy v.s. modern) needed? > > >>>>>>>>>>>>>> BTW a good API could be > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> #define VHOST_SET_ENDIAN _IOW(VHOST_VIRTIO, ?, int) > > >>>>>>>>>>>>>> #define VHOST_GET_ENDIAN _IOW(VHOST_VIRTIO, ?, int) > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> we did it per vring but maybe that was a mistake ... > > >>>>>>>>>>>>> Actually, I wonder whether it's good time to just not support > > >>>>>>>>>>>>> legacy driver > > >>>>>>>>>>>>> for vDPA. Consider: > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> 1) It's definition is no-normative > > >>>>>>>>>>>>> 2) A lot of budren of codes > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> So qemu can still present the legacy device since the config > > >>>>>>>>>>>>> space or other > > >>>>>>>>>>>>> stuffs that is presented by vhost-vDPA is not expected to be > > >>>>>>>>>>>>> accessed by > > >>>>>>>>>>>>> guest directly. Qemu can do the endian conversion when necessary > > >>>>>>>>>>>>> in this > > >>>>>>>>>>>>> case? > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Thanks > > >>>>>>>>>>>>> > > >>>>>>>>>>>> Overall I would be fine with this approach but we need to avoid breaking > > >>>>>>>>>>>> working userspace, qemu releases with vdpa support are out there and > > >>>>>>>>>>>> seem to work for people. Any changes need to take that into account > > >>>>>>>>>>>> and document compatibility concerns. > > >>>>>>>>>>> Agree, let me check. > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>>> I note that any hardware > > >>>>>>>>>>>> implementation is already broken for legacy except on platforms with > > >>>>>>>>>>>> strong ordering which might be helpful in reducing the scope. > > >>>>>>>>>>> Yes. > > >>>>>>>>>>> > > >>>>>>>>>>> Thanks > > >>>>>>>>>>> > > >>>>>>>>>>> > >