From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 38C5EC433F5 for ; Thu, 16 Dec 2021 06:35:21 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id CB11A60F98; Thu, 16 Dec 2021 06:35:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HO2vTMWJ6Mh9; Thu, 16 Dec 2021 06:35:19 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id E053060FBF; Thu, 16 Dec 2021 06:35:18 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id B0B01C002F; Thu, 16 Dec 2021 06:35:18 +0000 (UTC) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2E56EC0012 for ; Thu, 16 Dec 2021 06:35:17 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 1C1EC83218 for ; Thu, 16 Dec 2021 06:35:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kz9GkpA0E5QV for ; Thu, 16 Dec 2021 06:35:15 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id D031582F3D for ; Thu, 16 Dec 2021 06:35:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1639636513; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Xtjxk7vbEoRGCIi4LCQQtU/KJOvEczYA81RP/eWBh9Y=; b=hGoUoMzTwuH8eEcCYRE8vjh+CVtMc/NKEbtSUAZ+kaz/EL2srJ1C0p25GeAUnnB3DOgn0k S0oGj0BKyKaV9Dl/4rFm78aH38Y2a6Lecs/uoQxRac9SoWKJQ0jgBDZ3AFLzmw8z0PQ7EY acRnf/qKkDJM6rc2HtDHenLaENwGipI= Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-167-FWPxW4eTPVynyBVKwo3Rvg-1; Thu, 16 Dec 2021 01:35:10 -0500 X-MC-Unique: FWPxW4eTPVynyBVKwo3Rvg-1 Received: by mail-ed1-f71.google.com with SMTP id y11-20020a056402358b00b003f7ce63b89eso5538574edc.3 for ; Wed, 15 Dec 2021 22:35:09 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=Xtjxk7vbEoRGCIi4LCQQtU/KJOvEczYA81RP/eWBh9Y=; b=OHnTIQPBjiODLUhiH0VcvhCMusvKsFO36SWy1f+3gEMQ/SuWUUmgtaUDvAmx/HFDnB SfLFeujR8VyRSloXOByzmyHl3QxBDz6TmoLzAz2NvIJOnBqkLyLnAeT/SrrgOnctvojP ciCLMur93fkAApl7HzNRT8Qp1R/LcV6Rn/dkCP7z3htACstFaMJgGlDxTBKE2W/z2/Qb 3jaeUgjnMCQ8N+G/840fTdw+jWGkRTi4uWIgSwzabRVj5kYl1iyahSJ5aJT2zynj2tUd mNV5sZTDFIobQ3S+DcNwmq4UPIzv4fq4Ke8ZMT1h31hH2lOxyK+/zTExAvOFy46s55lr YvGw== X-Gm-Message-State: AOAM532klakLDAv1cDjLEDkmCbHxzerUtMQ6mB9XNq597wUQ7KbBxPZv n4vShXBkFW3Q7NFsbQhFdxLVTNAPvhewLCUDUxw36MlLNXqtc9oy9D2of74cZjlR78JEUyedN0N lXkUODcMAwwrJ76Ks9dvp7B2XPsj14O6te1KV/MFqEg== X-Received: by 2002:a05:6402:11c7:: with SMTP id j7mr18802915edw.83.1639636508816; Wed, 15 Dec 2021 22:35:08 -0800 (PST) X-Google-Smtp-Source: ABdhPJwWCUmlRiKevknUPqmIQUHsZ+5y6PNB1J0axe2lqls/B6DVNAlfPTQH5R9PundfbCHdLNAHzQ== X-Received: by 2002:a05:6402:11c7:: with SMTP id j7mr18802899edw.83.1639636508514; Wed, 15 Dec 2021 22:35:08 -0800 (PST) Received: from redhat.com ([2.55.22.18]) by smtp.gmail.com with ESMTPSA id hv13sm1441483ejc.75.2021.12.15.22.35.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Dec 2021 22:35:07 -0800 (PST) Date: Thu, 16 Dec 2021 01:35:04 -0500 From: "Michael S. Tsirkin" To: Si-Wei Liu Subject: Re: vdpa legacy guest support (was Re: [PATCH] vdpa/mlx5: set_features should allow reset to zero) Message-ID: <20211216013403-mutt-send-email-mst@kernel.org> References: <178f8ea7-cebd-0e81-3dc7-10a058d22c07@redhat.com> <20211212042311-mutt-send-email-mst@kernel.org> <20211214000245-mutt-send-email-mst@kernel.org> <4fc43d0f-da9e-ce16-1f26-9f0225239b75@oracle.com> <6eaf672c-cc86-b5bf-5b74-c837affeb6e1@oracle.com> <20211215162917-mutt-send-email-mst@kernel.org> <71d2a69c-94a7-76b5-2971-570026760bf0@oracle.com> MIME-Version: 1.0 In-Reply-To: <71d2a69c-94a7-76b5-2971-570026760bf0@oracle.com> Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=mst@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Disposition: inline Cc: netdev , virtualization , Eli Cohen , linux-kernel X-BeenThere: virtualization@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Linux virtualization List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: virtualization-bounces@lists.linux-foundation.org Sender: "Virtualization" T24gV2VkLCBEZWMgMTUsIDIwMjEgYXQgMDY6MDE6NTVQTSAtMDgwMCwgU2ktV2VpIExpdSB3cm90 ZToKPiAKPiAKPiBPbiAxMi8xNS8yMDIxIDE6MzMgUE0sIE1pY2hhZWwgUy4gVHNpcmtpbiB3cm90 ZToKPiA+IE9uIFdlZCwgRGVjIDE1LCAyMDIxIGF0IDEyOjUyOjIwUE0gLTA4MDAsIFNpLVdlaSBM aXUgd3JvdGU6Cj4gPiA+IAo+ID4gPiBPbiAxMi8xNC8yMDIxIDY6MDYgUE0sIEphc29uIFdhbmcg d3JvdGU6Cj4gPiA+ID4gT24gV2VkLCBEZWMgMTUsIDIwMjEgYXQgOTowNSBBTSBTaS1XZWkgTGl1 IDxzaS13ZWkubGl1QG9yYWNsZS5jb20+IHdyb3RlOgo+ID4gPiA+ID4gCj4gPiA+ID4gPiBPbiAx Mi8xMy8yMDIxIDk6MDYgUE0sIE1pY2hhZWwgUy4gVHNpcmtpbiB3cm90ZToKPiA+ID4gPiA+ID4g T24gTW9uLCBEZWMgMTMsIDIwMjEgYXQgMDU6NTk6NDVQTSAtMDgwMCwgU2ktV2VpIExpdSB3cm90 ZToKPiA+ID4gPiA+ID4gPiBPbiAxMi8xMi8yMDIxIDE6MjYgQU0sIE1pY2hhZWwgUy4gVHNpcmtp biB3cm90ZToKPiA+ID4gPiA+ID4gPiA+IE9uIEZyaSwgRGVjIDEwLCAyMDIxIGF0IDA1OjQ0OjE1 UE0gLTA4MDAsIFNpLVdlaSBMaXUgd3JvdGU6Cj4gPiA+ID4gPiA+ID4gPiA+IFNvcnJ5IGZvciBy ZXZpdmluZyB0aGlzIGFuY2llbnQgdGhyZWFkLiBJIHdhcyBraW5kYSBsb3N0IGZvciB0aGUgY29u Y2x1c2lvbgo+ID4gPiA+ID4gPiA+ID4gPiBpdCBlbmRlZCB1cCB3aXRoLiBJIGhhdmUgdGhlIGZv bGxvd2luZyBxdWVzdGlvbnMsCj4gPiA+ID4gPiA+ID4gPiA+IAo+ID4gPiA+ID4gPiA+ID4gPiAx LiBsZWdhY3kgZ3Vlc3Qgc3VwcG9ydDogZnJvbSB0aGUgcGFzdCBjb252ZXJzYXRpb25zIGl0IGRv ZXNuJ3Qgc2VlbSB0aGUKPiA+ID4gPiA+ID4gPiA+ID4gc3VwcG9ydCB3aWxsIGJlIGNvbXBsZXRl bHkgZHJvcHBlZCBmcm9tIHRoZSB0YWJsZSwgaXMgbXkgdW5kZXJzdGFuZGluZwo+ID4gPiA+ID4g PiA+ID4gPiBjb3JyZWN0PyBBY3R1YWxseSB3ZSdyZSBpbnRlcmVzdGVkIGluIHN1cHBvcnRpbmcg dmlydGlvIHYwLjk1IGd1ZXN0IGZvciB4ODYsCj4gPiA+ID4gPiA+ID4gPiA+IHdoaWNoIGlzIGJh Y2tlZCBieSB0aGUgc3BlYyBhdAo+ID4gPiA+ID4gPiA+ID4gPiBodHRwczovL3VybGRlZmVuc2Uu Y29tL3YzL19faHR0cHM6Ly9vemxhYnMub3JnLypydXN0eS92aXJ0aW8tc3BlYy92aXJ0aW8tMC45 LjUucGRmX187ZmchIUFDV1Y1TjlNMlJWOTloUSFkVEttekp3d1JzRk03QnRTdVREdTFjTmx5NW40 WENvdEgwV1ltaWR6R3FIU1h0NDBpN1pVNDNVY05nN0dZeFpnJCAuIFRob3VnaCBJJ20gbm90IHN1 cmUKPiA+ID4gPiA+ID4gPiA+ID4gaWYgdGhlcmUncyByZXF1ZXN0L25lZWQgdG8gc3VwcG9ydCB3 aWxkZXIgbGVnYWN5IHZpcnRpbyB2ZXJzaW9ucyBlYXJsaWVyCj4gPiA+ID4gPiA+ID4gPiA+IGJl eW9uZC4KPiA+ID4gPiA+ID4gPiA+IEkgcGVyc29uYWxseSBmZWVsIGl0J3MgbGVzcyB3b3JrIHRv IGFkZCBpbiBrZXJuZWwgdGhhbiB0cnkgdG8KPiA+ID4gPiA+ID4gPiA+IHdvcmsgYXJvdW5kIGl0 IGluIHVzZXJzcGFjZS4gSmFzb24gZmVlbHMgZGlmZmVyZW50bHkuCj4gPiA+ID4gPiA+ID4gPiBN YXliZSBwb3N0IHRoZSBwYXRjaGVzIGFuZCB0aGlzIHdpbGwgcHJvdmUgdG8gSmFzb24gaXQncyBu b3QKPiA+ID4gPiA+ID4gPiA+IHRvbyB0ZXJyaWJsZT8KPiA+ID4gPiA+ID4gPiBJIHN1cHBvc2Ug aWYgdGhlIHZkcGEgdmVuZG9yIGRvZXMgc3VwcG9ydCAwLjk1IGluIHRoZSBkYXRhcGF0aCBhbmQg cmluZwo+ID4gPiA+ID4gPiA+IGxheW91dCBsZXZlbCBhbmQgaXMgbGltaXRlZCB0byB4ODYgb25s eSwgdGhlcmUgc2hvdWxkIGJlIGVhc3kgd2F5IG91dC4KPiA+ID4gPiA+ID4gTm90ZSBhIHN1YnRs ZSBkaWZmZXJlbmNlOiB3aGF0IG1hdHRlcnMgaXMgdGhhdCBndWVzdCwgbm90IGhvc3QgaXMgeDg2 Lgo+ID4gPiA+ID4gPiBNYXR0ZXJzIGZvciBlbXVsYXRvcnMgd2hpY2ggbWlnaHQgcmVvcmRlciBt ZW1vcnkgYWNjZXNzZXMuCj4gPiA+ID4gPiA+IEkgZ3Vlc3MgdGhpcyBlbmZvcmNlbWVudCBiZWxv bmdzIGluIFFFTVUgdGhlbj8KPiA+ID4gPiA+IFJpZ2h0LCBJIG1lYW4gdG8gZ2V0IHN0YXJ0ZWQs IHRoZSBpbml0aWFsIGd1ZXN0IGRyaXZlciBzdXBwb3J0IGFuZCB0aGUKPiA+ID4gPiA+IGNvcnJl c3BvbmRpbmcgUUVNVSBzdXBwb3J0IGZvciB0cmFuc2l0aW9uYWwgdmRwYSBiYWNrZW5kIGNhbiBi ZSBsaW1pdGVkCj4gPiA+ID4gPiB0byB4ODYgZ3Vlc3QvaG9zdCBvbmx5LiBTaW5jZSB0aGUgY29u ZmlnIHNwYWNlIGlzIGVtdWxhdGVkIGluIFFFTVUsIEkKPiA+ID4gPiA+IHN1cHBvc2UgaXQncyBu b3QgaGFyZCB0byBlbmZvcmNlIGluIFFFTVUuCj4gPiA+ID4gSXQncyBtb3JlIHRoYW4ganVzdCBj b25maWcgc3BhY2UsIG1vc3QgZGV2aWNlcyBoYXZlIGhlYWRlcnMgYmVmb3JlIHRoZSBidWZmZXIu Cj4gPiA+IFRoZSBvcmRlcmluZyBpbiBkYXRhcGF0aCAoZGF0YSBWUXMpIHdvdWxkIGhhdmUgdG8g cmVseSBvbiB2ZW5kb3IncyBzdXBwb3J0Lgo+ID4gPiBTaW5jZSBPUkRFUl9QTEFURk9STSBpcyBw cmV0dHkgbmV3ICh2MS4xKSwgSSBndWVzcyB2ZHBhIGgvdyB2ZW5kb3Igbm93YWRheXMKPiA+ID4g Y2FuL3Nob3VsZCB3ZWxsIHN1cHBvcnQgdGhlIGNhc2Ugd2hlbiBPUkRFUl9QTEFURk9STSBpcyBu b3QgYWNrZWQgYnkgdGhlCj4gPiA+IGRyaXZlciAoYWN0dWFsbHkgdGhpcyBmZWF0dXJlIGlzIGZp bHRlcmVkIG91dCBieSB0aGUgUUVNVSB2aG9zdC12ZHBhIGRyaXZlcgo+ID4gPiB0b2RheSksIGV2 ZW4gd2l0aCB2MS4wIHNwZWMgY29uZm9ybWluZyBhbmQgbW9kZXJuIG9ubHkgdkRQQSBkZXZpY2Uu IFRoZQo+ID4gPiBjb250cm9sIFZRIGlzIGltcGxlbWVudGVkIGluIHNvZnR3YXJlIGluIHRoZSBr ZXJuZWwsIHdoaWNoIGNhbiBiZSBlYXNpbHkKPiA+ID4gYWNjb21tb2RhdGVkL2ZpeGVkIHdoZW4g bmVlZGVkLgo+ID4gPiAKPiA+ID4gPiA+IFFFTVUgY2FuIGRyaXZlIEdFVF9MRUdBQ1ksCj4gPiA+ ID4gPiBHRVRfRU5ESUFOIGV0IGFsIGlvY3RscyBpbiBhZHZhbmNlIHRvIGdldCB0aGUgY2FwYWJp bGl0eSBmcm9tIHRoZQo+ID4gPiA+ID4gaW5kaXZpZHVhbCB2ZW5kb3IgZHJpdmVyLiBGb3IgdGhh dCwgd2UgbmVlZCBhbm90aGVyIG5lZ290aWF0aW9uIHByb3RvY29sCj4gPiA+ID4gPiBzaW1pbGFy IHRvIHZob3N0X3VzZXIncyBwcm90b2NvbF9mZWF0dXJlcyBiZXR3ZWVuIHRoZSB2ZHBhIGtlcm5l bCBhbmQKPiA+ID4gPiA+IFFFTVUsIHdheSBiZWZvcmUgdGhlIGd1ZXN0IGRyaXZlciBpcyBldmVy IHByb2JlZCBhbmQgaXRzIGZlYXR1cmUKPiA+ID4gPiA+IG5lZ290aWF0aW9uIGtpY2tzIGluLiBO b3Qgc3VyZSB3ZSBuZWVkIGEgR0VUX01FTU9SWV9PUkRFUiBpb2N0bCBjYWxsCj4gPiA+ID4gPiBm cm9tIHRoZSBkZXZpY2UsIGJ1dCB3ZSBjYW4gYXNzdW1lIHdlYWsgb3JkZXJpbmcgZm9yIGxlZ2Fj eSBhdCB0aGlzCj4gPiA+ID4gPiBwb2ludCAoeDg2IG9ubHkpPwo+ID4gPiA+IEknbSBsb3N0IGhl cmUsIHdlIGhhdmUgZ2V0X2ZlYXR1cmVzKCkgc286Cj4gPiA+IEkgYXNzdW1lIGhlcmUgeW91IHJl ZmVyIHRvIGdldF9kZXZpY2VfZmVhdHVyZXMoKSB0aGF0IEVsaSBqdXN0IGNoYW5nZWQgdGhlCj4g PiA+IG5hbWUuCj4gPiA+ID4gMSkgVkVSU0lPTl8xIG1lYW5zIHRoZSBkZXZpY2UgdXNlcyBMRSBp ZiBwcm92aWRlZCwgb3RoZXJ3aXNlIG5hdHZpZQo+ID4gPiA+IDIpIE9SREVSX1BMQVRGT1JNIG1l YW5zIGRldmljZSByZXF1aXJlcyBwbGF0Zm9ybSBvcmRlcmluZwo+ID4gPiA+IAo+ID4gPiA+IEFu eSByZWFzb24gZm9yIGhhdmluZyBhIG5ldyBBUEkgZm9yIHRoaXM/Cj4gPiA+IEFyZSB5b3UgZ29p bmcgdG8gZW5mb3JjZSBhbGwgdkRQQSBoYXJkd2FyZSB2ZW5kb3JzIHRvIHN1cHBvcnQgdGhlCj4g PiA+IHRyYW5zaXRpb25hbCBtb2RlbCBmb3IgbGVnYWN5IGd1ZXN0PyBtZWFuaW5nIGd1ZXN0IG5v dCBhY2tub3dsZWRnaW5nCj4gPiA+IFZFUlNJT05fMSB3b3VsZCB1c2UgdGhlIGxlZ2FjeSBpbnRl cmZhY2VzIGNhcHR1cmVkIGluIHRoZSBzcGVjIHNlY3Rpb24gNy40Cj4gPiA+IChyZWdhcmRpbmcg cmluZyBsYXlvdXQsIG5hdGl2ZSBlbmRpYW5uZXNzLCBtZXNzYWdlIGZyYW1pbmcsIHZxIGFsaWdu bWVudCBvZgo+ID4gPiA0MDk2LCAzMmJpdCBmZWF0dXJlLCBubyBmZWF0dXJlc19vayBiaXQgaW4g c3RhdHVzLCBJTyBwb3J0IGludGVyZmFjZSBpLmUuCj4gPiA+IGFsbCB0aGUgdGhpbmdzKSBpbnN0 ZWFkPyBOb3RlZCB3ZSBkb24ndCB5ZXQgaGF2ZSBhIHNldF9kZXZpY2VfZmVhdHVyZXMoKQo+ID4g PiB0aGF0IGFsbG93cyB0aGUgdmRwYSBkZXZpY2UgdG8gdGVsbCB3aGV0aGVyIGl0IGlzIG9wZXJh dGluZyBpbiB0cmFuc2l0aW9uYWwKPiA+ID4gb3IgbW9kZXJuLW9ubHkgbW9kZS4gRm9yIHNvZnR3 YXJlIHZpcnRpbywgYWxsIHN1cHBvcnQgZm9yIHRoZSBsZWdhY3kgcGFydCBpbgo+ID4gPiBhIHRy YW5zaXRpb25hbCBtb2RlbCBoYXMgYmVlbiBidWlsdCB1cCB0aGVyZSBhbHJlYWR5LCBob3dldmVy LCBpdCdzIG5vdCBlYXN5Cj4gPiA+IGZvciB2RFBBIHZlbmRvcnMgdG8gaW1wbGVtZW50IGFsbCB0 aGUgcmVxdWlyZW1lbnRzIGZvciBhbiBhbGwtb3Itbm90aGluZwo+ID4gPiBsZWdhY3kgZ3Vlc3Qg c3VwcG9ydCAoYmlnIGVuZGlhbiBndWVzdCBmb3IgZXhhbXBsZSkuIFRvIHRoZXNlIHZlbmRvcnMs IHRoZQo+ID4gPiBsZWdhY3kgc3VwcG9ydCB3aXRoaW4gYSB0cmFuc2l0aW9uYWwgbW9kZWwgaXMg bW9yZSBvZiBmZWF0dXJlIHRvIHRoZW0gYW5kCj4gPiA+IGl0J3MgYmVzdCB0byBsZWF2ZSBzb21l IGZsZXhpYmlsaXR5IGZvciB0aGVtIHRvIGltcGxlbWVudCBwYXJ0aWFsIHN1cHBvcnQKPiA+ID4g Zm9yIGxlZ2FjeS4gVGhhdCBpbiB0dXJuIGNhbGxzIG91dCB0aGUgbmVlZCBmb3IgYSB2aG9zdC11 c2VyIHByb3RvY29sCj4gPiA+IGZlYXR1cmUgbGlrZSBuZWdvdGlhdGlvbiBBUEkgdGhhdCBjYW4g cHJvaGliaXQgdGhvc2UgdW5zdXBwb3J0ZWQgZ3Vlc3QKPiA+ID4gc2V0dXBzIHRvIGFzIGVhcmx5 IGFzIGJhY2tlbmRfaW5pdCBiZWZvcmUgbGF1bmNoaW5nIHRoZSBWTS4KPiA+IFJpZ2h0LiBPZiBu b3RlIGlzIHRoZSBmYWN0IHRoYXQgaXQncyBhIHNwZWMgYnVnIHdoaWNoIEkKPiA+IGhvcGUgeWV0 IHRvIGZpeCwgdGhvdWdoIGR1ZSB0byBleGlzdGluZyBndWVzdCBjb2RlIHRoZQo+ID4gZml4IHdv bid0IGJlIGNvbXBsZXRlLgo+IEkgdGhvdWdodCBhdCBvbmUgcG9pbnQgeW91IHBvaW50ZWQgb3V0 IHRvIG1lIHRoYXQgdGhlIHNwZWMgZG9lcyBhbGxvdyBjb25maWcKPiBzcGFjZSByZWFkIGJlZm9y ZSBjbGFpbWluZyBmZWF0dXJlc19vaywgYW5kIG9ubHkgY29uZmlnIHdyaXRlIGJlZm9yZQo+IGZl YXR1cmVzX29rIGlzIHByb2hpYml0ZWQuIEkgaGF2ZW4ndCByZWFkIHVwIHRoZSBmdWxsIHRocmVh ZCBvZiBIYWxpbCdzCj4gVkVSU0lPTl8xIGZvciB0cmFuc2l0aW9uYWwgYmlnIGVuZGlhbiBkZXZp Y2UgeWV0LCBidXQgd2hhdCBpcyB0aGUgc3BlYyBidWcKPiB5b3UgaG9wZSB0byBmaXg/CgpBbGxv d2luZyBjb25maWcgc3BhY2UgcmVhZHMgYmVmb3JlIGZlYXR1cmVzX29rIHNlZW1lZCB1c2VmdWwg eWVhcnMgYWdvCmJ1dCBpbiBwcmFjdGljZSBpcyBvbmx5IGNhdXNpbmcgYnVncyBhbmQgY29tcGxp Y2F0aW5nIGRldmljZSBkZXNpZ24uCgo+IAo+ID4gCj4gPiBXUlQgaW9jdGxzLCBPbmUgdGhpbmcg d2UgY2FuIGRvIHRob3VnaCBpcyBhYnVzZSBzZXRfZmVhdHVyZXMKPiA+IHdoZXJlIGl0J3MgY2Fs bGVkIGJ5IFFFTVUgZWFybHkgb24gd2l0aCBqdXN0IHRoZSBWRVJTSU9OXzEKPiA+IGJpdCBzZXQs IHRvIGRpc3Rpbmd1aXNoIGJldHdlZW4gbGVnYWN5IGFuZCBtb2Rlcm4KPiA+IGludGVyZmFjZS4g VGhpcyBiZWZvcmUgY29uZmlnIHNwYWNlIGFjY2Vzc2VzIGFuZCBGRUFUVVJFU19PSy4KPiA+IAo+ ID4gSGFsaWwgaGFzIGJlZW4gd29ya2luZyBvbiB0aGlzLCBwbHMgdGFrZSBhIGxvb2sgYW5kIG1h eWJlIGhlbHAgaGltIG91dC4KPiBJbnRlcmVzdGluZyB0aHJlYWQsIGFtIHJlYWRpbmcgbm93IGFu ZCBzZWUgaG93IEkgbWF5IGxldmVyYWdlIG9yIGhlbHAgdGhlcmUuCj4gCj4gPiA+ID4gPiA+ID4g SQo+ID4gPiA+ID4gPiA+IGNoZWNrZWQgd2l0aCBFbGkgYW5kIG90aGVyIE1lbGxhbm94L05WRElB IGZvbGtzIGZvciBoYXJkd2FyZS9maXJtd2FyZSBsZXZlbAo+ID4gPiA+ID4gPiA+IDAuOTUgc3Vw cG9ydCwgaXQgc2VlbXMgYWxsIHRoZSBpbmdyZWRpZW50IGhhZCBiZWVuIHRoZXJlIGFscmVhZHkg ZGF0ZWQgYmFjawo+ID4gPiA+ID4gPiA+IHRvIHRoZSBEUERLIGRheXMuIFRoZSBvbmx5IG1ham9y IHRoaW5nIGxpbWl0aW5nIGlzIGluIHRoZSB2RFBBIHNvZnR3YXJlIHRoYXQKPiA+ID4gPiA+ID4g PiB0aGUgY3VycmVudCB2ZHBhIGNvcmUgaGFzIHRoZSBhc3N1bXB0aW9uIGFyb3VuZCBWSVJUSU9f Rl9BQ0NFU1NfUExBVEZPUk0gZm9yCj4gPiA+ID4gPiA+ID4gYSBmZXcgRE1BIHNldHVwIG9wcywg d2hpY2ggaXMgdmlydGlvIDEuMCBvbmx5Lgo+ID4gPiA+ID4gPiA+IAo+ID4gPiA+ID4gPiA+ID4g PiAyLiBzdXBwb3NlIHNvbWUgZm9ybSBvZiBsZWdhY3kgZ3Vlc3Qgc3VwcG9ydCBuZWVkcyB0byBi ZSB0aGVyZSwgaG93IGRvIHdlCj4gPiA+ID4gPiA+ID4gPiA+IGRlYWwgd2l0aCB0aGUgYm9ndXMg YXNzdW1wdGlvbiBiZWxvdyBpbiB2ZHBhX2dldF9jb25maWcoKSBpbiB0aGUgc2hvcnQgdGVybT8K PiA+ID4gPiA+ID4gPiA+ID4gSXQgbG9va3Mgb25lIG9mIHRoZSBpbnR1aXRpdmUgZml4IGlzIHRv IG1vdmUgdGhlIHZkcGFfc2V0X2ZlYXR1cmVzIGNhbGwgb3V0Cj4gPiA+ID4gPiA+ID4gPiA+IG9m IHZkcGFfZ2V0X2NvbmZpZygpIHRvIHZkcGFfc2V0X2NvbmZpZygpLgo+ID4gPiA+ID4gPiA+ID4g PiAKPiA+ID4gPiA+ID4gPiA+ID4gICAgICAgICAgICAgLyoKPiA+ID4gPiA+ID4gPiA+ID4gICAg ICAgICAgICAgICogQ29uZmlnIGFjY2Vzc2VzIGFyZW4ndCBzdXBwb3NlZCB0byB0cmlnZ2VyIGJl Zm9yZSBmZWF0dXJlcyBhcmUKPiA+ID4gPiA+ID4gPiA+ID4gc2V0Lgo+ID4gPiA+ID4gPiA+ID4g PiAgICAgICAgICAgICAgKiBJZiBpdCBkb2VzIGhhcHBlbiB3ZSBhc3N1bWUgYSBsZWdhY3kgZ3Vl c3QuCj4gPiA+ID4gPiA+ID4gPiA+ICAgICAgICAgICAgICAqLwo+ID4gPiA+ID4gPiA+ID4gPiAg ICAgICAgICAgICBpZiAoIXZkZXYtPmZlYXR1cmVzX3ZhbGlkKQo+ID4gPiA+ID4gPiA+ID4gPiAg ICAgICAgICAgICAgICAgICAgIHZkcGFfc2V0X2ZlYXR1cmVzKHZkZXYsIDApOwo+ID4gPiA+ID4g PiA+ID4gPiAgICAgICAgICAgICBvcHMtPmdldF9jb25maWcodmRldiwgb2Zmc2V0LCBidWYsIGxl bik7Cj4gPiA+ID4gPiA+ID4gPiA+IAo+ID4gPiA+ID4gPiA+ID4gPiBJIGNhbiBwb3N0IGEgcGF0 Y2ggdG8gZml4IDIpIGlmIHRoZXJlJ3MgY29uc2Vuc3VzIGFscmVhZHkgcmVhY2hlZC4KPiA+ID4g PiA+ID4gPiA+ID4gCj4gPiA+ID4gPiA+ID4gPiA+IFRoYW5rcywKPiA+ID4gPiA+ID4gPiA+ID4g LVNpd2VpCj4gPiA+ID4gPiA+ID4gPiBJJ20gbm90IHN1cmUgaG93IGltcG9ydGFudCBpdCBpcyB0 byBjaGFuZ2UgdGhhdC4KPiA+ID4gPiA+ID4gPiA+IEluIGFueSBjYXNlIGl0IG9ubHkgYWZmZWN0 cyB0cmFuc2l0aW9uYWwgZGV2aWNlcywgcmlnaHQ/Cj4gPiA+ID4gPiA+ID4gPiBMZWdhY3kgb25s eSBzaG91bGQgbm90IGNhcmUgLi4uCj4gPiA+ID4gPiA+ID4gWWVzIEknZCBsaWtlIHRvIGRpc3Rp bmd1aXNoIGxlZ2FjeSBkcml2ZXIgKHN1cHBvc2UgaXQgaXMgMC45NSkgYWdhaW5zdCB0aGUKPiA+ ID4gPiA+ID4gPiBtb2Rlcm4gb25lIGluIGEgdHJhbnNpdGlvbmFsIGRldmljZSBtb2RlbCByYXRo ZXIgdGhhbiBiZWluZyBsZWdhY3kgb25seS4KPiA+ID4gPiA+ID4gPiBUaGF0IHdheSBhIHYwLjk1 IGFuZCB2MS4wIHN1cHBvcnRpbmcgdmRwYSBwYXJlbnQgY2FuIHN1cHBvcnQgYm90aCB0eXBlcyBv Zgo+ID4gPiA+ID4gPiA+IGd1ZXN0cyB3aXRob3V0IGhhdmluZyB0byByZWNvbmZpZ3VyZS4gT3Ig YXJlIHlvdSBzdWdnZXN0aW5nIGxpbWl0IHRvIGxlZ2FjeQo+ID4gPiA+ID4gPiA+IG9ubHkgYXQg dGhlIHRpbWUgb2YgdmRwYSBjcmVhdGlvbiB3b3VsZCBzaW1wbGlmeSB0aGUgaW1wbGVtZW50YXRp b24gYSBsb3Q/Cj4gPiA+ID4gPiA+ID4gCj4gPiA+ID4gPiA+ID4gVGhhbmtzLAo+ID4gPiA+ID4g PiA+IC1TaXdlaQo+ID4gPiA+ID4gPiBJIGRvbid0IGtub3cgZm9yIHN1cmUuIFRha2UgYSBsb29r IGF0IHRoZSB3b3JrIEhhbGlsIHdhcyBkb2luZwo+ID4gPiA+ID4gPiB0byB0cnkgYW5kIHN1cHBv cnQgdHJhbnNpdGlvbmFsIGRldmljZXMgd2l0aCBCRSBndWVzdHMuCj4gPiA+ID4gPiBIbW1tLCB3 ZSBjYW4gaGF2ZSB0aG9zZSBlbmRpYW5uZXNzIGlvY3RscyBkZWZpbmVkIGJ1dCB0aGUgaW5pdGlh bCBRRU1VCj4gPiA+ID4gPiBpbXBsZW1lbnRhdGlvbiBjYW4gYmUgc3RhcnRlZCB0byBzdXBwb3J0 IHg4NiBndWVzdC9ob3N0IHdpdGggbGl0dGxlCj4gPiA+ID4gPiBlbmRpYW4gYW5kIHdlYWsgbWVt b3J5IG9yZGVyaW5nIGZpcnN0LiBUaGUgcmVhbCB0cmljayBpcyB0byBkZXRlY3QKPiA+ID4gPiA+ IGxlZ2FjeSBndWVzdCAtIEkgYW0gbm90IHN1cmUgaWYgaXQncyBmZWFzaWJsZSB0byBzaGlmdCBh bGwgdGhlIGxlZ2FjeQo+ID4gPiA+ID4gZGV0ZWN0aW9uIHdvcmsgdG8gUUVNVSwgb3IgdGhlIGtl cm5lbCBoYXMgdG8gYmUgcGFydCBvZiB0aGUgZGV0ZWN0aW9uCj4gPiA+ID4gPiAoZS5nLiB0aGUg a2ljayBiZWZvcmUgRFJJVkVSX09LIHRoaW5nIHdlIGhhdmUgdG8gZHVwbGljYXRlIHRoZSB0cmFj a2luZwo+ID4gPiA+ID4gZWZmb3J0IGluIFFFTVUpIGFzIHdlbGwuIExldCBtZSB0YWtlIGEgZnVy dGhlciBsb29rIGFuZCBnZXQgYmFjay4KPiA+ID4gPiBNaWNoYWVsIG1heSB0aGluayBkaWZmZXJl bnRseSBidXQgSSB0aGluayBkb2luZyB0aGlzIGluIFFlbXUgaXMgbXVjaCBlYXNpZXIuCj4gPiA+ IEkgdGhpbmsgdGhlIGtleSBpcyB3aGV0aGVyIHdlIHBvc2l0aW9uIGVtdWxhdGluZyBsZWdhY3kg aW50ZXJmYWNlcyBpbiBRRU1VCj4gPiA+IGRvaW5nIHRyYW5zbGF0aW9uIG9uIHRvcCBvZiBhIHYx LjAgbW9kZXJuLW9ubHkgZGV2aWNlIGluIHRoZSBrZXJuZWwsIG9yIHdlCj4gPiA+IGFsbG93IHZk cGEgY29yZSAob3IgeW91IGNhbiBzYXkgdmhvc3QtdmRwYSkgYW5kIHZlbmRvciBkcml2ZXIgdG8g c3VwcG9ydCBhCj4gPiA+IHRyYW5zaXRpb25hbCBtb2RlbCBpbiB0aGUga2VybmVsIHRoYXQgaXMg YWJsZSB0byB3b3JrIGZvciBib3RoIHYwLjk1IGFuZAo+ID4gPiB2MS4wIGRyaXZlcnMsIHdpdGgg c29tZSBzbGlnaHQgYWlkIGZyb20gUUVNVSBmb3IKPiA+ID4gZGV0ZWN0aW5nL2VtdWxhdGlvbi9z aGFkb3dpbmcgKGZvciBlLmcgQ1ZRLCBJL08gcG9ydCByZWxheSkuIEkgZ3Vlc3MgZm9yIHRoZQo+ ID4gPiBmb3JtZXIgd2Ugc3RpbGwgcmVseSBvbiB2ZW5kb3IgZm9yIGEgcGVyZm9ybWFudCBkYXRh IHZxcyBpbXBsZW1lbnRhdGlvbiwKPiA+ID4gbGVhdmluZyB0aGUgcXVlc3Rpb24gdG8gd2hhdCBp dCBtYXkgZW5kIHVwIGV2ZW50dWFsbHkgaW4gdGhlIGtlcm5lbCBpcwo+ID4gPiBlZmZlY3RpdmVs eSB0aGUgbGF0dGVyKS4KPiA+ID4gCj4gPiA+IFRoYW5rcywKPiA+ID4gLVNpd2VpCj4gPiAKPiA+ IE15IHN1Z2dlc3Rpb24gaXMgcG9zdCB0aGUga2VybmVsIHBhdGNoZXMsIGFuZCB3ZSBjYW4gZXZh bHVhdGUKPiA+IGhvdyBtdWNoIHdvcmsgdGhleSBhcmUuCj4gVGhhbmtzIGZvciB0aGUgZmVlZGJh Y2suIEkgd2lsbCB0YWtlIHNvbWUgcmVhZCB0aGVuIGdldCBiYWNrLCBwcm9iYWJseSBhZnRlcgo+ IHRoZSB3aW50ZXIgYnJlYWsuIFN0YXkgdHVuZWQuCj4gCj4gVGhhbmtzLAo+IC1TaXdlaQo+IAo+ ID4gCj4gPiA+ID4gVGhhbmtzCj4gPiA+ID4gCj4gPiA+ID4gCj4gPiA+ID4gCj4gPiA+ID4gPiBN ZWFud2hpbGUsIEknbGwgY2hlY2sgaW50ZXJuYWxseSB0byBzZWUgaWYgYSBsZWdhY3kgb25seSBt b2RlbCB3b3VsZAo+ID4gPiA+ID4gd29yay4gVGhhbmtzLgo+ID4gPiA+ID4gCj4gPiA+ID4gPiBU aGFua3MsCj4gPiA+ID4gPiAtU2l3ZWkKPiA+ID4gPiA+IAo+ID4gPiA+ID4gCj4gPiA+ID4gPiA+ ID4gPiA+IE9uIDMvMi8yMDIxIDI6NTMgQU0sIEphc29uIFdhbmcgd3JvdGU6Cj4gPiA+ID4gPiA+ ID4gPiA+ID4gT24gMjAyMS8zLzIgNTo0NyDkuIvljYgsIE1pY2hhZWwgUy4gVHNpcmtpbiB3cm90 ZToKPiA+ID4gPiA+ID4gPiA+ID4gPiA+IE9uIE1vbiwgTWFyIDAxLCAyMDIxIGF0IDExOjU2OjUw QU0gKzA4MDAsIEphc29uIFdhbmcgd3JvdGU6Cj4gPiA+ID4gPiA+ID4gPiA+ID4gPiA+IE9uIDIw MjEvMy8xIDU6MzQg5LiK5Y2ILCBNaWNoYWVsIFMuIFRzaXJraW4gd3JvdGU6Cj4gPiA+ID4gPiA+ ID4gPiA+ID4gPiA+ID4gT24gV2VkLCBGZWIgMjQsIDIwMjEgYXQgMTA6MjQ6NDFBTSAtMDgwMCwg U2ktV2VpIExpdSB3cm90ZToKPiA+ID4gPiA+ID4gPiA+ID4gPiA+ID4gPiA+ID4gRGV0ZWN0aW5n IGl0IGlzbid0IGVub3VnaCB0aG91Z2gsIHdlIHdpbGwgbmVlZCBhIG5ldyBpb2N0bCB0byBub3Rp ZnkKPiA+ID4gPiA+ID4gPiA+ID4gPiA+ID4gPiA+ID4gdGhlIGtlcm5lbCB0aGF0IGl0J3MgYSBs ZWdhY3kgZ3Vlc3QuIFVnaCA6KAo+ID4gPiA+ID4gPiA+ID4gPiA+ID4gPiA+ID4gV2VsbCwgYWx0 aG91Z2ggSSB0aGluayBhZGRpbmcgYW4gaW9jdGwgaXMgZG9hYmxlLCBtYXkgSQo+ID4gPiA+ID4g PiA+ID4gPiA+ID4gPiA+ID4ga25vdyB3aGF0IHRoZSB1c2UKPiA+ID4gPiA+ID4gPiA+ID4gPiA+ ID4gPiA+IGNhc2UgdGhlcmUgd2lsbCBiZSBmb3Iga2VybmVsIHRvIGxldmVyYWdlIHN1Y2ggaW5m bwo+ID4gPiA+ID4gPiA+ID4gPiA+ID4gPiA+ID4gZGlyZWN0bHk/IElzIHRoZXJlIGEKPiA+ID4g PiA+ID4gPiA+ID4gPiA+ID4gPiA+IGNhc2UgUUVNVSBjYW4ndCBkbyB3aXRoIGRlZGljYXRlIGlv Y3RscyBsYXRlciBpZiB0aGVyZSdzIGluZGVlZAo+ID4gPiA+ID4gPiA+ID4gPiA+ID4gPiA+ID4g ZGlmZmVyZW50aWF0aW9uIChsZWdhY3kgdi5zLiBtb2Rlcm4pIG5lZWRlZD8KPiA+ID4gPiA+ID4g PiA+ID4gPiA+ID4gPiBCVFcgYSBnb29kIEFQSSBjb3VsZCBiZQo+ID4gPiA+ID4gPiA+ID4gPiA+ ID4gPiA+IAo+ID4gPiA+ID4gPiA+ID4gPiA+ID4gPiA+ICNkZWZpbmUgVkhPU1RfU0VUX0VORElB TiBfSU9XKFZIT1NUX1ZJUlRJTywgPywgaW50KQo+ID4gPiA+ID4gPiA+ID4gPiA+ID4gPiA+ICNk ZWZpbmUgVkhPU1RfR0VUX0VORElBTiBfSU9XKFZIT1NUX1ZJUlRJTywgPywgaW50KQo+ID4gPiA+ ID4gPiA+ID4gPiA+ID4gPiA+IAo+ID4gPiA+ID4gPiA+ID4gPiA+ID4gPiA+IHdlIGRpZCBpdCBw ZXIgdnJpbmcgYnV0IG1heWJlIHRoYXQgd2FzIGEgbWlzdGFrZSAuLi4KPiA+ID4gPiA+ID4gPiA+ ID4gPiA+ID4gQWN0dWFsbHksIEkgd29uZGVyIHdoZXRoZXIgaXQncyBnb29kIHRpbWUgdG8ganVz dCBub3Qgc3VwcG9ydAo+ID4gPiA+ID4gPiA+ID4gPiA+ID4gPiBsZWdhY3kgZHJpdmVyCj4gPiA+ ID4gPiA+ID4gPiA+ID4gPiA+IGZvciB2RFBBLiBDb25zaWRlcjoKPiA+ID4gPiA+ID4gPiA+ID4g PiA+ID4gCj4gPiA+ID4gPiA+ID4gPiA+ID4gPiA+IDEpIEl0J3MgZGVmaW5pdGlvbiBpcyBuby1u b3JtYXRpdmUKPiA+ID4gPiA+ID4gPiA+ID4gPiA+ID4gMikgQSBsb3Qgb2YgYnVkcmVuIG9mIGNv ZGVzCj4gPiA+ID4gPiA+ID4gPiA+ID4gPiA+IAo+ID4gPiA+ID4gPiA+ID4gPiA+ID4gPiBTbyBx ZW11IGNhbiBzdGlsbCBwcmVzZW50IHRoZSBsZWdhY3kgZGV2aWNlIHNpbmNlIHRoZSBjb25maWcK PiA+ID4gPiA+ID4gPiA+ID4gPiA+ID4gc3BhY2Ugb3Igb3RoZXIKPiA+ID4gPiA+ID4gPiA+ID4g PiA+ID4gc3R1ZmZzIHRoYXQgaXMgcHJlc2VudGVkIGJ5IHZob3N0LXZEUEEgaXMgbm90IGV4cGVj dGVkIHRvIGJlCj4gPiA+ID4gPiA+ID4gPiA+ID4gPiA+IGFjY2Vzc2VkIGJ5Cj4gPiA+ID4gPiA+ ID4gPiA+ID4gPiA+IGd1ZXN0IGRpcmVjdGx5LiBRZW11IGNhbiBkbyB0aGUgZW5kaWFuIGNvbnZl cnNpb24gd2hlbiBuZWNlc3NhcnkKPiA+ID4gPiA+ID4gPiA+ID4gPiA+ID4gaW4gdGhpcwo+ID4g PiA+ID4gPiA+ID4gPiA+ID4gPiBjYXNlPwo+ID4gPiA+ID4gPiA+ID4gPiA+ID4gPiAKPiA+ID4g PiA+ID4gPiA+ID4gPiA+ID4gVGhhbmtzCj4gPiA+ID4gPiA+ID4gPiA+ID4gPiA+IAo+ID4gPiA+ ID4gPiA+ID4gPiA+ID4gT3ZlcmFsbCBJIHdvdWxkIGJlIGZpbmUgd2l0aCB0aGlzIGFwcHJvYWNo IGJ1dCB3ZSBuZWVkIHRvIGF2b2lkIGJyZWFraW5nCj4gPiA+ID4gPiA+ID4gPiA+ID4gPiB3b3Jr aW5nIHVzZXJzcGFjZSwgcWVtdSByZWxlYXNlcyB3aXRoIHZkcGEgc3VwcG9ydCBhcmUgb3V0IHRo ZXJlIGFuZAo+ID4gPiA+ID4gPiA+ID4gPiA+ID4gc2VlbSB0byB3b3JrIGZvciBwZW9wbGUuIEFu eSBjaGFuZ2VzIG5lZWQgdG8gdGFrZSB0aGF0IGludG8gYWNjb3VudAo+ID4gPiA+ID4gPiA+ID4g PiA+ID4gYW5kIGRvY3VtZW50IGNvbXBhdGliaWxpdHkgY29uY2VybnMuCj4gPiA+ID4gPiA+ID4g PiA+ID4gQWdyZWUsIGxldCBtZSBjaGVjay4KPiA+ID4gPiA+ID4gPiA+ID4gPiAKPiA+ID4gPiA+ ID4gPiA+ID4gPiAKPiA+ID4gPiA+ID4gPiA+ID4gPiA+ICAgICAgIEkgbm90ZSB0aGF0IGFueSBo YXJkd2FyZQo+ID4gPiA+ID4gPiA+ID4gPiA+ID4gaW1wbGVtZW50YXRpb24gaXMgYWxyZWFkeSBi cm9rZW4gZm9yIGxlZ2FjeSBleGNlcHQgb24gcGxhdGZvcm1zIHdpdGgKPiA+ID4gPiA+ID4gPiA+ ID4gPiA+IHN0cm9uZyBvcmRlcmluZyB3aGljaCBtaWdodCBiZSBoZWxwZnVsIGluIHJlZHVjaW5n IHRoZSBzY29wZS4KPiA+ID4gPiA+ID4gPiA+ID4gPiBZZXMuCj4gPiA+ID4gPiA+ID4gPiA+ID4g Cj4gPiA+ID4gPiA+ID4gPiA+ID4gVGhhbmtzCj4gPiA+ID4gPiA+ID4gPiA+ID4gCj4gPiA+ID4g PiA+ID4gPiA+ID4gCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fXwpWaXJ0dWFsaXphdGlvbiBtYWlsaW5nIGxpc3QKVmlydHVhbGl6YXRpb25AbGlzdHMubGlu dXgtZm91bmRhdGlvbi5vcmcKaHR0cHM6Ly9saXN0cy5saW51eGZvdW5kYXRpb24ub3JnL21haWxt YW4vbGlzdGluZm8vdmlydHVhbGl6YXRpb24= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD843C433F5 for ; Thu, 16 Dec 2021 06:35:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234129AbhLPGfR (ORCPT ); Thu, 16 Dec 2021 01:35:17 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:51581 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229747AbhLPGfM (ORCPT ); Thu, 16 Dec 2021 01:35:12 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1639636511; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Xtjxk7vbEoRGCIi4LCQQtU/KJOvEczYA81RP/eWBh9Y=; b=NIcDvFtwSkcNozntoKA9AMHjfTAYLr+983ITlY8XPgH0UXKDZaPiApvr50NFs9h36gg7rQ Y70J6u0wQnE3MBKy0VN5s3YgyK+NwJBrDemsR2zzUGewXBWcZDpWjPbbCAtEgON/hRno0v nkHgQzXnA8pnYWIUjipF4DsXAUw+9zM= Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-20-hhxO7WB2M0-Zs4KRyZiaKg-1; Thu, 16 Dec 2021 01:35:10 -0500 X-MC-Unique: hhxO7WB2M0-Zs4KRyZiaKg-1 Received: by mail-ed1-f72.google.com with SMTP id z14-20020a05640235ce00b003f7e90c32b6so3864203edc.4 for ; Wed, 15 Dec 2021 22:35:09 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=Xtjxk7vbEoRGCIi4LCQQtU/KJOvEczYA81RP/eWBh9Y=; b=IgKpkrkUeCDgl9xzm833LbHZwKe1hysDefZ8p6Rg5twQUjKKvNFh7fqz5N6G+0UFeJ ETEG21WV5sjEzrOz/UB94PPuhSk4L2/HZNXjjqRnjJPjcSSUrXUj5c+8XQgOvRZm4OlN 4Fgwkz7ujA/pgFCUlV7B7KSLJpvyi+/+zJEYIJitzoVC9bWmLo+2cdXYFBCNLvyxIoW8 1MEW1/h/bcnhaZkpQzFAY7YGvRrZnaRf+W1790tzYPr/JS5jaDdMQApHT/hA9683pqAZ 8UnphrNwe5MlwpllVS5gMG11uaZVb9lR7Fob5nkiH9aoxdaFNRKtTCrh2iO4SETs1k5Z QwIQ== X-Gm-Message-State: AOAM530UV6kXNRtfT2Bo7U+mFwQWWv9ZYqjfI4k5hoTou26ZRy9sRu/k n6yJ4cfidFp4YBdgOxOmA9+cxa970TvMLc3Hhvk6EXQRSmKMA1go5L9W70eaID26lC93C8aLc2R n4TSbj9i89nSoUY6vAnifla4e X-Received: by 2002:a05:6402:11c7:: with SMTP id j7mr18802919edw.83.1639636508834; Wed, 15 Dec 2021 22:35:08 -0800 (PST) X-Google-Smtp-Source: ABdhPJwWCUmlRiKevknUPqmIQUHsZ+5y6PNB1J0axe2lqls/B6DVNAlfPTQH5R9PundfbCHdLNAHzQ== X-Received: by 2002:a05:6402:11c7:: with SMTP id j7mr18802899edw.83.1639636508514; Wed, 15 Dec 2021 22:35:08 -0800 (PST) Received: from redhat.com ([2.55.22.18]) by smtp.gmail.com with ESMTPSA id hv13sm1441483ejc.75.2021.12.15.22.35.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Dec 2021 22:35:07 -0800 (PST) Date: Thu, 16 Dec 2021 01:35:04 -0500 From: "Michael S. Tsirkin" To: Si-Wei Liu Cc: Jason Wang , Eli Cohen , linux-kernel , virtualization , netdev Subject: Re: vdpa legacy guest support (was Re: [PATCH] vdpa/mlx5: set_features should allow reset to zero) Message-ID: <20211216013403-mutt-send-email-mst@kernel.org> References: <178f8ea7-cebd-0e81-3dc7-10a058d22c07@redhat.com> <20211212042311-mutt-send-email-mst@kernel.org> <20211214000245-mutt-send-email-mst@kernel.org> <4fc43d0f-da9e-ce16-1f26-9f0225239b75@oracle.com> <6eaf672c-cc86-b5bf-5b74-c837affeb6e1@oracle.com> <20211215162917-mutt-send-email-mst@kernel.org> <71d2a69c-94a7-76b5-2971-570026760bf0@oracle.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <71d2a69c-94a7-76b5-2971-570026760bf0@oracle.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Dec 15, 2021 at 06:01:55PM -0800, Si-Wei Liu wrote: > > > On 12/15/2021 1:33 PM, Michael S. Tsirkin wrote: > > On Wed, Dec 15, 2021 at 12:52:20PM -0800, Si-Wei Liu wrote: > > > > > > On 12/14/2021 6:06 PM, Jason Wang wrote: > > > > On Wed, Dec 15, 2021 at 9:05 AM Si-Wei Liu wrote: > > > > > > > > > > On 12/13/2021 9:06 PM, Michael S. Tsirkin wrote: > > > > > > On Mon, Dec 13, 2021 at 05:59:45PM -0800, Si-Wei Liu wrote: > > > > > > > On 12/12/2021 1:26 AM, Michael S. Tsirkin wrote: > > > > > > > > On Fri, Dec 10, 2021 at 05:44:15PM -0800, Si-Wei Liu wrote: > > > > > > > > > Sorry for reviving this ancient thread. I was kinda lost for the conclusion > > > > > > > > > it ended up with. I have the following questions, > > > > > > > > > > > > > > > > > > 1. legacy guest support: from the past conversations it doesn't seem the > > > > > > > > > support will be completely dropped from the table, is my understanding > > > > > > > > > correct? Actually we're interested in supporting virtio v0.95 guest for x86, > > > > > > > > > which is backed by the spec at > > > > > > > > > https://urldefense.com/v3/__https://ozlabs.org/*rusty/virtio-spec/virtio-0.9.5.pdf__;fg!!ACWV5N9M2RV99hQ!dTKmzJwwRsFM7BtSuTDu1cNly5n4XCotH0WYmidzGqHSXt40i7ZU43UcNg7GYxZg$ . Though I'm not sure > > > > > > > > > if there's request/need to support wilder legacy virtio versions earlier > > > > > > > > > beyond. > > > > > > > > I personally feel it's less work to add in kernel than try to > > > > > > > > work around it in userspace. Jason feels differently. > > > > > > > > Maybe post the patches and this will prove to Jason it's not > > > > > > > > too terrible? > > > > > > > I suppose if the vdpa vendor does support 0.95 in the datapath and ring > > > > > > > layout level and is limited to x86 only, there should be easy way out. > > > > > > Note a subtle difference: what matters is that guest, not host is x86. > > > > > > Matters for emulators which might reorder memory accesses. > > > > > > I guess this enforcement belongs in QEMU then? > > > > > Right, I mean to get started, the initial guest driver support and the > > > > > corresponding QEMU support for transitional vdpa backend can be limited > > > > > to x86 guest/host only. Since the config space is emulated in QEMU, I > > > > > suppose it's not hard to enforce in QEMU. > > > > It's more than just config space, most devices have headers before the buffer. > > > The ordering in datapath (data VQs) would have to rely on vendor's support. > > > Since ORDER_PLATFORM is pretty new (v1.1), I guess vdpa h/w vendor nowadays > > > can/should well support the case when ORDER_PLATFORM is not acked by the > > > driver (actually this feature is filtered out by the QEMU vhost-vdpa driver > > > today), even with v1.0 spec conforming and modern only vDPA device. The > > > control VQ is implemented in software in the kernel, which can be easily > > > accommodated/fixed when needed. > > > > > > > > QEMU can drive GET_LEGACY, > > > > > GET_ENDIAN et al ioctls in advance to get the capability from the > > > > > individual vendor driver. For that, we need another negotiation protocol > > > > > similar to vhost_user's protocol_features between the vdpa kernel and > > > > > QEMU, way before the guest driver is ever probed and its feature > > > > > negotiation kicks in. Not sure we need a GET_MEMORY_ORDER ioctl call > > > > > from the device, but we can assume weak ordering for legacy at this > > > > > point (x86 only)? > > > > I'm lost here, we have get_features() so: > > > I assume here you refer to get_device_features() that Eli just changed the > > > name. > > > > 1) VERSION_1 means the device uses LE if provided, otherwise natvie > > > > 2) ORDER_PLATFORM means device requires platform ordering > > > > > > > > Any reason for having a new API for this? > > > Are you going to enforce all vDPA hardware vendors to support the > > > transitional model for legacy guest? meaning guest not acknowledging > > > VERSION_1 would use the legacy interfaces captured in the spec section 7.4 > > > (regarding ring layout, native endianness, message framing, vq alignment of > > > 4096, 32bit feature, no features_ok bit in status, IO port interface i.e. > > > all the things) instead? Noted we don't yet have a set_device_features() > > > that allows the vdpa device to tell whether it is operating in transitional > > > or modern-only mode. For software virtio, all support for the legacy part in > > > a transitional model has been built up there already, however, it's not easy > > > for vDPA vendors to implement all the requirements for an all-or-nothing > > > legacy guest support (big endian guest for example). To these vendors, the > > > legacy support within a transitional model is more of feature to them and > > > it's best to leave some flexibility for them to implement partial support > > > for legacy. That in turn calls out the need for a vhost-user protocol > > > feature like negotiation API that can prohibit those unsupported guest > > > setups to as early as backend_init before launching the VM. > > Right. Of note is the fact that it's a spec bug which I > > hope yet to fix, though due to existing guest code the > > fix won't be complete. > I thought at one point you pointed out to me that the spec does allow config > space read before claiming features_ok, and only config write before > features_ok is prohibited. I haven't read up the full thread of Halil's > VERSION_1 for transitional big endian device yet, but what is the spec bug > you hope to fix? Allowing config space reads before features_ok seemed useful years ago but in practice is only causing bugs and complicating device design. > > > > > WRT ioctls, One thing we can do though is abuse set_features > > where it's called by QEMU early on with just the VERSION_1 > > bit set, to distinguish between legacy and modern > > interface. This before config space accesses and FEATURES_OK. > > > > Halil has been working on this, pls take a look and maybe help him out. > Interesting thread, am reading now and see how I may leverage or help there. > > > > > > > > I > > > > > > > checked with Eli and other Mellanox/NVDIA folks for hardware/firmware level > > > > > > > 0.95 support, it seems all the ingredient had been there already dated back > > > > > > > to the DPDK days. The only major thing limiting is in the vDPA software that > > > > > > > the current vdpa core has the assumption around VIRTIO_F_ACCESS_PLATFORM for > > > > > > > a few DMA setup ops, which is virtio 1.0 only. > > > > > > > > > > > > > > > > 2. suppose some form of legacy guest support needs to be there, how do we > > > > > > > > > deal with the bogus assumption below in vdpa_get_config() in the short term? > > > > > > > > > It looks one of the intuitive fix is to move the vdpa_set_features call out > > > > > > > > > of vdpa_get_config() to vdpa_set_config(). > > > > > > > > > > > > > > > > > > /* > > > > > > > > > * Config accesses aren't supposed to trigger before features are > > > > > > > > > set. > > > > > > > > > * If it does happen we assume a legacy guest. > > > > > > > > > */ > > > > > > > > > if (!vdev->features_valid) > > > > > > > > > vdpa_set_features(vdev, 0); > > > > > > > > > ops->get_config(vdev, offset, buf, len); > > > > > > > > > > > > > > > > > > I can post a patch to fix 2) if there's consensus already reached. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > -Siwei > > > > > > > > I'm not sure how important it is to change that. > > > > > > > > In any case it only affects transitional devices, right? > > > > > > > > Legacy only should not care ... > > > > > > > Yes I'd like to distinguish legacy driver (suppose it is 0.95) against the > > > > > > > modern one in a transitional device model rather than being legacy only. > > > > > > > That way a v0.95 and v1.0 supporting vdpa parent can support both types of > > > > > > > guests without having to reconfigure. Or are you suggesting limit to legacy > > > > > > > only at the time of vdpa creation would simplify the implementation a lot? > > > > > > > > > > > > > > Thanks, > > > > > > > -Siwei > > > > > > I don't know for sure. Take a look at the work Halil was doing > > > > > > to try and support transitional devices with BE guests. > > > > > Hmmm, we can have those endianness ioctls defined but the initial QEMU > > > > > implementation can be started to support x86 guest/host with little > > > > > endian and weak memory ordering first. The real trick is to detect > > > > > legacy guest - I am not sure if it's feasible to shift all the legacy > > > > > detection work to QEMU, or the kernel has to be part of the detection > > > > > (e.g. the kick before DRIVER_OK thing we have to duplicate the tracking > > > > > effort in QEMU) as well. Let me take a further look and get back. > > > > Michael may think differently but I think doing this in Qemu is much easier. > > > I think the key is whether we position emulating legacy interfaces in QEMU > > > doing translation on top of a v1.0 modern-only device in the kernel, or we > > > allow vdpa core (or you can say vhost-vdpa) and vendor driver to support a > > > transitional model in the kernel that is able to work for both v0.95 and > > > v1.0 drivers, with some slight aid from QEMU for > > > detecting/emulation/shadowing (for e.g CVQ, I/O port relay). I guess for the > > > former we still rely on vendor for a performant data vqs implementation, > > > leaving the question to what it may end up eventually in the kernel is > > > effectively the latter). > > > > > > Thanks, > > > -Siwei > > > > My suggestion is post the kernel patches, and we can evaluate > > how much work they are. > Thanks for the feedback. I will take some read then get back, probably after > the winter break. Stay tuned. > > Thanks, > -Siwei > > > > > > > Thanks > > > > > > > > > > > > > > > > > Meanwhile, I'll check internally to see if a legacy only model would > > > > > work. Thanks. > > > > > > > > > > Thanks, > > > > > -Siwei > > > > > > > > > > > > > > > > > > > On 3/2/2021 2:53 AM, Jason Wang wrote: > > > > > > > > > > On 2021/3/2 5:47 下午, Michael S. Tsirkin wrote: > > > > > > > > > > > On Mon, Mar 01, 2021 at 11:56:50AM +0800, Jason Wang wrote: > > > > > > > > > > > > On 2021/3/1 5:34 上午, Michael S. Tsirkin wrote: > > > > > > > > > > > > > On Wed, Feb 24, 2021 at 10:24:41AM -0800, Si-Wei Liu wrote: > > > > > > > > > > > > > > > Detecting it isn't enough though, we will need a new ioctl to notify > > > > > > > > > > > > > > > the kernel that it's a legacy guest. Ugh :( > > > > > > > > > > > > > > Well, although I think adding an ioctl is doable, may I > > > > > > > > > > > > > > know what the use > > > > > > > > > > > > > > case there will be for kernel to leverage such info > > > > > > > > > > > > > > directly? Is there a > > > > > > > > > > > > > > case QEMU can't do with dedicate ioctls later if there's indeed > > > > > > > > > > > > > > differentiation (legacy v.s. modern) needed? > > > > > > > > > > > > > BTW a good API could be > > > > > > > > > > > > > > > > > > > > > > > > > > #define VHOST_SET_ENDIAN _IOW(VHOST_VIRTIO, ?, int) > > > > > > > > > > > > > #define VHOST_GET_ENDIAN _IOW(VHOST_VIRTIO, ?, int) > > > > > > > > > > > > > > > > > > > > > > > > > > we did it per vring but maybe that was a mistake ... > > > > > > > > > > > > Actually, I wonder whether it's good time to just not support > > > > > > > > > > > > legacy driver > > > > > > > > > > > > for vDPA. Consider: > > > > > > > > > > > > > > > > > > > > > > > > 1) It's definition is no-normative > > > > > > > > > > > > 2) A lot of budren of codes > > > > > > > > > > > > > > > > > > > > > > > > So qemu can still present the legacy device since the config > > > > > > > > > > > > space or other > > > > > > > > > > > > stuffs that is presented by vhost-vDPA is not expected to be > > > > > > > > > > > > accessed by > > > > > > > > > > > > guest directly. Qemu can do the endian conversion when necessary > > > > > > > > > > > > in this > > > > > > > > > > > > case? > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > Overall I would be fine with this approach but we need to avoid breaking > > > > > > > > > > > working userspace, qemu releases with vdpa support are out there and > > > > > > > > > > > seem to work for people. Any changes need to take that into account > > > > > > > > > > > and document compatibility concerns. > > > > > > > > > > Agree, let me check. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I note that any hardware > > > > > > > > > > > implementation is already broken for legacy except on platforms with > > > > > > > > > > > strong ordering which might be helpful in reducing the scope. > > > > > > > > > > Yes. > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > >