From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0678DC433EF for ; Mon, 10 Jan 2022 15:44:30 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id B172C605A2; Mon, 10 Jan 2022 15:44:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zXA-gUvGAc3g; Mon, 10 Jan 2022 15:44:29 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 7D9F760C2B; Mon, 10 Jan 2022 15:44:29 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id DF43EC006F; Mon, 10 Jan 2022 15:44:28 +0000 (UTC) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id EF165C002F for ; Mon, 10 Jan 2022 15:44:26 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id D0FF84018B for ; Mon, 10 Jan 2022 15:44:26 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DaswWFl4h1L3 for ; Mon, 10 Jan 2022 15:44:25 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 8B15A400B9 for ; Mon, 10 Jan 2022 15:44:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1641829464; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1GwsjTdK6v2LnBFhWI+JJOy9OQSY+yI1I9CwqbFn3SQ=; b=DvnUYEvpnzTAeUwOpuyl1FA4lSU6D58bvsSauGti+BwmRcfx32Tvyqesnmx6aqQ2Rkiuzf GUVZ0bLMiFlQfXXw4YZpepIPoYc9QsC8EMiv3lyYaIW5msmujJ+K9YhVp2m25ZKHl2bqwp b6kq25ecLpJnhNVjJmKrdvVrMIm1ucs= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-636-hohxOSwDO9irZT5IHUKETg-1; Mon, 10 Jan 2022 10:44:22 -0500 X-MC-Unique: hohxOSwDO9irZT5IHUKETg-1 Received: by mail-wm1-f71.google.com with SMTP id i81-20020a1c3b54000000b003467c58cbddso8727671wma.5 for ; Mon, 10 Jan 2022 07:44:21 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=1GwsjTdK6v2LnBFhWI+JJOy9OQSY+yI1I9CwqbFn3SQ=; b=JdZfDaCdoIQ8D2bKCHc+NaKrK2ElbZ0XvD6x7SSzBJ8nRHioXq6GdaX0MMq03TSGrt pQKcrB7V3lleeicB33zFUO800uZUqPyKwiS5wPaBJLye3z9AtpOn4ZVQONXTKxJrqkq/ b6c8amdF+3vxcXubmxdNHjf0EH+26jtX9kmsat8Z5L2tWUhjpJtwsmFQZSjRSaGqeZS2 r1u2EHyyyX9SLGmw00L62XTHHLJmzfexlR4jXWNOf0h3z/jNbvIli3qpC7S3r+SgLn6B tMpTT23uUkJKJlbr+h/MnU5XM4KVtagifMhRwUPcPGcYNfdpwJqnZ3wtqBB35AouZpTI oNnw== X-Gm-Message-State: AOAM533PTrAhZ44W3Z0HqlJCtOvlpvC/2aU5NHQhd7GFidKBcaQDihfK MJNZjuuxyQI+QG+Li3HuQ+SEXqMVHeMtLxF9l715ODJAp1K2qqA10zh13F4dZyyf9yQ1MZ4XJJJ LWXd+372qSiXsM2FjPEDLxCK52/U2rA== X-Received: by 2002:adf:f807:: with SMTP id s7mr184303wrp.638.1641829460437; Mon, 10 Jan 2022 07:44:20 -0800 (PST) X-Google-Smtp-Source: ABdhPJwGlziNjGEYg/AKCCVR6dHJcBIW6ndhJ3HU0fYXJaw1aGDnRw1EG3ab/feKEGmla/24mvGOMw== X-Received: by 2002:adf:f807:: with SMTP id s7mr184278wrp.638.1641829460157; Mon, 10 Jan 2022 07:44:20 -0800 (PST) Received: from redhat.com ([2.55.148.228]) by smtp.gmail.com with ESMTPSA id s1sm6891060wmh.35.2022.01.10.07.44.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 07:44:19 -0800 (PST) Date: Mon, 10 Jan 2022 10:44:14 -0500 From: "Michael S. Tsirkin" To: Yongji Xie Subject: Re: [PATCH v12 00/13] Introduce VDUSE - vDPA Device in Userspace Message-ID: <20220110103938-mutt-send-email-mst@kernel.org> References: <20210830141737.181-1-xieyongji@bytedance.com> <20220110075546-mutt-send-email-mst@kernel.org> <20220110100911-mutt-send-email-mst@kernel.org> MIME-Version: 1.0 In-Reply-To: Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=mst@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Disposition: inline Cc: kvm , Jason Wang , virtualization , Christian Brauner , Will Deacon , Jonathan Corbet , Matthew Wilcox , Christoph Hellwig , Dan Carpenter , Stefano Garzarella , Liu Xiaodong , linux-fsdevel@vger.kernel.org, Al Viro , Stefan Hajnoczi , songmuchun@bytedance.com, Jens Axboe , He Zhe , Greg KH , Randy Dunlap , linux-kernel , iommu@lists.linux-foundation.org, bcrl@kvack.org, Netdev , Joe Perches , Robin Murphy , Mika =?iso-8859-1?Q?Penttil=E4?= X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" T24gTW9uLCBKYW4gMTAsIDIwMjIgYXQgMTE6MjQ6NDBQTSArMDgwMCwgWW9uZ2ppIFhpZSB3cm90 ZToKPiBPbiBNb24sIEphbiAxMCwgMjAyMiBhdCAxMToxMCBQTSBNaWNoYWVsIFMuIFRzaXJraW4g PG1zdEByZWRoYXQuY29tPiB3cm90ZToKPiA+Cj4gPiBPbiBNb24sIEphbiAxMCwgMjAyMiBhdCAw OTo1NDowOFBNICswODAwLCBZb25namkgWGllIHdyb3RlOgo+ID4gPiBPbiBNb24sIEphbiAxMCwg MjAyMiBhdCA4OjU3IFBNIE1pY2hhZWwgUy4gVHNpcmtpbiA8bXN0QHJlZGhhdC5jb20+IHdyb3Rl Ogo+ID4gPiA+Cj4gPiA+ID4gT24gTW9uLCBBdWcgMzAsIDIwMjEgYXQgMTA6MTc6MjRQTSArMDgw MCwgWGllIFlvbmdqaSB3cm90ZToKPiA+ID4gPiA+IFRoaXMgc2VyaWVzIGludHJvZHVjZXMgYSBm cmFtZXdvcmsgdGhhdCBtYWtlcyBpdCBwb3NzaWJsZSB0byBpbXBsZW1lbnQKPiA+ID4gPiA+IHNv ZnR3YXJlLWVtdWxhdGVkIHZEUEEgZGV2aWNlcyBpbiB1c2Vyc3BhY2UuIEFuZCB0byBtYWtlIHRo ZSBkZXZpY2UKPiA+ID4gPiA+IGVtdWxhdGlvbiBtb3JlIHNlY3VyZSwgdGhlIGVtdWxhdGVkIHZE UEEgZGV2aWNlJ3MgY29udHJvbCBwYXRoIGlzIGhhbmRsZWQKPiA+ID4gPiA+IGluIHRoZSBrZXJu ZWwgYW5kIG9ubHkgdGhlIGRhdGEgcGF0aCBpcyBpbXBsZW1lbnRlZCBpbiB0aGUgdXNlcnNwYWNl Lgo+ID4gPiA+ID4KPiA+ID4gPiA+IFNpbmNlIHRoZSBlbXVsZGF0ZWQgdkRQQSBkZXZpY2UncyBj b250cm9sIHBhdGggaXMgaGFuZGxlZCBpbiB0aGUga2VybmVsLAo+ID4gPiA+ID4gYSBtZXNzYWdl IG1lY2huaXNtIGlzIGludHJvZHVjZWQgdG8gbWFrZSB1c2Vyc3BhY2UgYmUgYXdhcmUgb2YgdGhl IGRhdGEKPiA+ID4gPiA+IHBhdGggcmVsYXRlZCBjaGFuZ2VzLiBVc2Vyc3BhY2UgY2FuIHVzZSBy ZWFkKCkvd3JpdGUoKSB0byByZWNlaXZlL3JlcGx5Cj4gPiA+ID4gPiB0aGUgY29udHJvbCBtZXNz YWdlcy4KPiA+ID4gPiA+Cj4gPiA+ID4gPiBJbiB0aGUgZGF0YSBwYXRoLCB0aGUgY29yZSBpcyBt YXBwaW5nIGRtYSBidWZmZXIgaW50byBWRFVTRSBkYWVtb24ncwo+ID4gPiA+ID4gYWRkcmVzcyBz cGFjZSwgd2hpY2ggY2FuIGJlIGltcGxlbWVudGVkIGluIGRpZmZlcmVudCB3YXlzIGRlcGVuZGlu ZyBvbgo+ID4gPiA+ID4gdGhlIHZkcGEgYnVzIHRvIHdoaWNoIHRoZSB2RFBBIGRldmljZSBpcyBh dHRhY2hlZC4KPiA+ID4gPiA+Cj4gPiA+ID4gPiBJbiB2aXJ0aW8tdmRwYSBjYXNlLCB3ZSBpbXBs ZW1lbnRzIGEgTU1VLWJhc2VkIHNvZnR3YXJlIElPVExCIHdpdGgKPiA+ID4gPiA+IGJvdW5jZS1i dWZmZXJpbmcgbWVjaGFuaXNtIHRvIGFjaGlldmUgdGhhdC4gQW5kIGluIHZob3N0LXZkcGEgY2Fz ZSwgdGhlIGRtYQo+ID4gPiA+ID4gYnVmZmVyIGlzIHJlc2lkZSBpbiBhIHVzZXJzcGFjZSBtZW1v cnkgcmVnaW9uIHdoaWNoIGNhbiBiZSBzaGFyZWQgdG8gdGhlCj4gPiA+ID4gPiBWRFVTRSB1c2Vy c3BhY2UgcHJvY2Vzc3MgdmlhIHRyYW5zZmVycmluZyB0aGUgc2htZmQuCj4gPiA+ID4gPgo+ID4g PiA+ID4gVGhlIGRldGFpbHMgYW5kIG91ciB1c2VyIGNhc2UgaXMgc2hvd24gYmVsb3c6Cj4gPiA+ ID4gPgo+ID4gPiA+ID4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tICAgIC0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0gICAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tCj4gPiA+ID4gPiB8ICAgICAgICAgICAgQ29udGFpbmVyIHwgICAgfCAgICAgICAgICAgICAg UUVNVShWTSkgfCAgIHwgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgVkRVU0UgZGFlbW9u IHwKPiA+ID4gPiA+IHwgICAgICAgLS0tLS0tLS0tICAgICAgfCAgICB8ICAtLS0tLS0tLS0tLS0t LS0tLS0tICB8ICAgfCAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tIC0tLS0tLS0tLS0tLS0tLS0g fAo+ID4gPiA+ID4gfCAgICAgICB8ZGV2L3ZkeHwgICAgICB8ICAgIHwgIHwvZGV2L3Zob3N0LXZk cGEteHwgIHwgICB8IHwgdkRQQSBkZXZpY2UgZW11bGF0aW9uIHwgfCBibG9jayBkcml2ZXIgfCB8 Cj4gPiA+ID4gPiAtLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0gICAgIC0tLS0tLS0tLS0tKy0tLS0t LS0tLS0tLSAgIC0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0K PiA+ID4gPiA+ICAgICAgICAgICAgIHwgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAg ICAgICAgICAgICAgICAgICAgICAgIHwgICAgICAgICAgICAgICAgICAgICAgfAo+ID4gPiA+ID4g ICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgIHwgICAgICAgICAgICAgICAg ICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICB8Cj4gPiA+ID4gPiAtLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0KPiA+ID4gPiA+IHwgICAgfCBibG9j ayBkZXZpY2UgfCAgICAgICAgICAgfCAgdmhvc3QgZGV2aWNlIHwgICAgICAgICAgICB8IHZkdXNl IGRyaXZlciB8ICAgICAgICAgIHwgVENQL0lQIHwgICAgfAo+ID4gPiA+ID4gfCAgICAtLS0tLS0t Ky0tLS0tLS0tICAgICAgICAgICAtLS0tLS0tLSstLS0tLS0tLSAgICAgICAgICAgIC0tLS0tLS0r LS0tLS0tLS0gICAgICAgICAgLS0tLS0rLS0tLSAgICB8Cj4gPiA+ID4gPiB8ICAgICAgICAgICB8 ICAgICAgICAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgIHwg ICAgICAgICAgICAgICAgICAgICAgIHwgICAgICAgIHwKPiA+ID4gPiA+IHwgLS0tLS0tLS0tLSst LS0tLS0tLS0tICAgICAgIC0tLS0tLS0tLS0rLS0tLS0tLS0tLS0gICAgICAgICAtLS0tLS0tKy0t LS0tLS0gICAgICAgICAgICAgICAgfCAgICAgICAgfAo+ID4gPiA+ID4gfCB8IHZpcnRpby1ibGsg ZHJpdmVyIHwgICAgICAgfCAgdmhvc3QtdmRwYSBkcml2ZXIgfCAgICAgICAgIHwgdmRwYSBkZXZp Y2UgfCAgICAgICAgICAgICAgICB8ICAgICAgICB8Cj4gPiA+ID4gPiB8IC0tLS0tLS0tLS0rLS0t LS0tLS0tLSAgICAgICAtLS0tLS0tLS0tKy0tLS0tLS0tLS0tICAgICAgICAgLS0tLS0tLSstLS0t LS0tICAgICAgICAgICAgICAgIHwgICAgICAgIHwKPiA+ID4gPiA+IHwgICAgICAgICAgIHwgICAg ICB2aXJ0aW8gYnVzICAgICAgICAgICB8ICAgICAgICAgICAgICAgICAgICAgICAgICAgfCAgICAg ICAgICAgICAgICAgICAgICAgfCAgICAgICAgfAo+ID4gPiA+ID4gfCAgIC0tLS0tLS0tKy0tLS0r LS0tLS0tLS0tLS0gICAgICAgICAgIHwgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAg ICAgICAgICAgICAgICAgICB8ICAgICAgICB8Cj4gPiA+ID4gPiB8ICAgICAgICAgICAgICAgIHwg ICAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgIHwgICAgICAg ICAgICAgICAgICAgICAgIHwgICAgICAgIHwKPiA+ID4gPiA+IHwgICAgICAtLS0tLS0tLS0tKy0t LS0tLS0tLS0gICAgICAgICAgICB8ICAgICAgICAgICAgICAgICAgICAgICAgICAgfCAgICAgICAg ICAgICAgICAgICAgICAgfCAgICAgICAgfAo+ID4gPiA+ID4gfCAgICAgIHwgdmlydGlvLWJsayBk ZXZpY2UgfCAgICAgICAgICAgIHwgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAg ICAgICAgICAgICAgICB8ICAgICAgICB8Cj4gPiA+ID4gPiB8ICAgICAgLS0tLS0tLS0tLSstLS0t LS0tLS0tICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgIHwgICAgICAgICAg ICAgICAgICAgICAgIHwgICAgICAgIHwKPiA+ID4gPiA+IHwgICAgICAgICAgICAgICAgfCAgICAg ICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAg ICAgICAgICAgICAgfCAgICAgICAgfAo+ID4gPiA+ID4gfCAgICAgLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0gICAgICAgICAgIHwgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAg ICAgICAgICAgICB8ICAgICAgICB8Cj4gPiA+ID4gPiB8ICAgICB8ICB2aXJ0aW8tdmRwYSBkcml2 ZXIgfCAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgIHwgICAgICAgICAgICAg ICAgICAgICAgIHwgICAgICAgIHwKPiA+ID4gPiA+IHwgICAgIC0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tICAgICAgICAgICB8ICAgICAgICAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAg ICAgICAgICAgfCAgICAgICAgfAo+ID4gPiA+ID4gfCAgICAgICAgICAgICAgICB8ICAgICAgICAg ICAgICAgICAgICAgIHwgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgIHZkcGEgYnVzICAg ICAgICAgICB8ICAgICAgICB8Cj4gPiA+ID4gPiB8ICAgICAtLS0tLS0tLS0tLSstLS0tLS0tLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0gICAg ICAgICAgIHwgICAgICAgIHwKPiA+ID4gPiA+IHwgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgLS0tKy0tLSAgICAgfAo+ID4gPiA+ID4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS18IE5JQyB8LS0tLS0tCj4gPiA+ID4gPiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IC0tLSstLS0KPiA+ID4gPiA+ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfAo+ ID4gPiA+ID4gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAtLS0tLS0tLS0rLS0tLS0tLS0tCj4g PiA+ID4gPiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwgUmVtb3RlIFN0b3JhZ2VzIHwKPiA+ ID4gPiA+ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLS0tLS0tLS0tLS0tLS0tLS0tLQo+ID4g PiA+ID4KPiA+ID4gPiA+IFdlIG1ha2UgdXNlIG9mIGl0IHRvIGltcGxlbWVudCBhIGJsb2NrIGRl dmljZSBjb25uZWN0aW5nIHRvCj4gPiA+ID4gPiBvdXIgZGlzdHJpYnV0ZWQgc3RvcmFnZSwgd2hp Y2ggY2FuIGJlIHVzZWQgYm90aCBpbiBjb250YWluZXJzIGFuZAo+ID4gPiA+ID4gVk1zLiBUaHVz LCB3ZSBjYW4gaGF2ZSBhbiB1bmlmaWVkIHRlY2hub2xvZ3kgc3RhY2sgaW4gdGhpcyB0d28gY2Fz ZXMuCj4gPiA+ID4gPgo+ID4gPiA+ID4gVG8gdGVzdCBpdCB3aXRoIG51bGwtYmxrOgo+ID4gPiA+ ID4KPiA+ID4gPiA+ICAgJCBxZW11LXN0b3JhZ2UtZGFlbW9uIFwKPiA+ID4gPiA+ICAgICAgIC0t Y2hhcmRldiBzb2NrZXQsaWQ9Y2hhcm1vbml0b3IscGF0aD0vdG1wL3FtcC5zb2NrLHNlcnZlcixu b3dhaXQgXAo+ID4gPiA+ID4gICAgICAgLS1tb25pdG9yIGNoYXJkZXY9Y2hhcm1vbml0b3IgXAo+ ID4gPiA+ID4gICAgICAgLS1ibG9ja2RldiBkcml2ZXI9aG9zdF9kZXZpY2UsY2FjaGUuZGlyZWN0 PW9uLGFpbz1uYXRpdmUsZmlsZW5hbWU9L2Rldi9udWxsYjAsbm9kZS1uYW1lPWRpc2swIFwKPiA+ ID4gPiA+ICAgICAgIC0tZXhwb3J0IHR5cGU9dmR1c2UtYmxrLGlkPXRlc3Qsbm9kZS1uYW1lPWRp c2swLHdyaXRhYmxlPW9uLG5hbWU9dmR1c2UtbnVsbCxudW0tcXVldWVzPTE2LHF1ZXVlLXNpemU9 MTI4Cj4gPiA+ID4gPgo+ID4gPiA+ID4gVGhlIHFlbXUtc3RvcmFnZS1kYWVtb24gY2FuIGJlIGZv dW5kIGF0IGh0dHBzOi8vZ2l0aHViLmNvbS9ieXRlZGFuY2UvcWVtdS90cmVlL3ZkdXNlCj4gPiA+ ID4KPiA+ID4gPiBJdCdzIGJlZW4gaGFsZiBhIHllYXIgLSBhbnkgcGxhbnMgdG8gdXBzdHJlYW0g dGhpcz8KPiA+ID4KPiA+ID4gWWVhaCwgdGhpcyBpcyBvbiBteSB0by1kbyBsaXN0IHRoaXMgbW9u dGguCj4gPiA+Cj4gPiA+IFNvcnJ5IGZvciB0YWtpbmcgc28gbG9uZy4uLiBJJ3ZlIGJlZW4gd29y a2luZyBvbiBhbm90aGVyIHByb2plY3QKPiA+ID4gZW5hYmxpbmcgdXNlcnNwYWNlIFJETUEgd2l0 aCBWRFVTRSBmb3IgdGhlIHBhc3QgZmV3IG1vbnRocy4gU28gSQo+ID4gPiBkaWRuJ3QgaGF2ZSBt dWNoIHRpbWUgZm9yIHRoaXMuIEFueXdheSwgSSB3aWxsIHN1Ym1pdCB0aGUgZmlyc3QKPiA+ID4g dmVyc2lvbiBhcyBzb29uIGFzIHBvc3NpYmxlLgo+ID4gPgo+ID4gPiBUaGFua3MsCj4gPiA+IFlv bmdqaQo+ID4KPiA+IE9oIGZ1bi4gWW91IG1lYW4gbGlrZSB2aXJ0aW8tcmRtYT8gT3IgUkRNQSBh cyBhIGJhY2tlbmQgZm9yIHJlZ3VsYXIKPiA+IHZpcnRpbz8KPiA+Cj4gCj4gWWVzLCBsaWtlIHZp cnRpby1yZG1hLiBUaGVuIHdlIGNhbiBkZXZlbG9wIHNvbWV0aGluZyBsaWtlIHVzZXJzcGFjZQo+ IHJ4ZeOAgXNpdyBvciBjdXN0b20gcHJvdG9jb2wgd2l0aCBWRFVTRS4KPiAKPiBUaGFua3MsCj4g WW9uZ2ppCgpXb3VsZCBiZSBpbnRlcmVzdGluZyB0byBzZWUgdGhlIHNwZWMgZm9yIHRoYXQuClRo ZSBpc3N1ZXMgd2l0aCBSRE1BIHJldm9sdmVkIGFyb3VuZCB0aGUgZmFjdCB0aGF0IGN1cnJlbnQK YXBwcyB0ZW5kIHRvIGVpdGhlciB1c2Ugbm9uLXN0YW5kYXJkIHByb3BvY29scyBmb3IgY29ubmVj dGlvbgplc3RhYmxpc2htZW50IG9yIHVzZSBVRCB3aGVyZSB0aGVyZSdzIElJUkMgbm8gc3RhbmRh cmQKYXQgYWxsLiBTbyBRUCBudW1iZXJzIGFyZSBoYXJkIHRvIHZpcnR1YWxpemUuClNpbWlsYXJs eSBtYW55IHVzZSBMSURzIGRpcmVjdGx5IHdpdGggdGhlIHNhbWUgZWZmZWN0LgpHVUlEcyBtaWdo dCBiZSB2aXJ0dWFsaXplYWJsZSBidXQgbm8gb25lIHdlbnQgdG8gdGhlIGVmZm9ydC4KClRvIHNh eSBub3RoaW5nIGFib3V0IHRoZSBpbnRlcmFjdGlvbiB3aXRoIG1lbW9yeSBvdmVyY29tbWl0LgoK LS0gCk1TVAoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18K aW9tbXUgbWFpbGluZyBsaXN0CmlvbW11QGxpc3RzLmxpbnV4LWZvdW5kYXRpb24ub3JnCmh0dHBz Oi8vbGlzdHMubGludXhmb3VuZGF0aW9uLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2lvbW11 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C251BC433F5 for ; Mon, 10 Jan 2022 15:44:30 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 476DD8148F; Mon, 10 Jan 2022 15:44:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pfrYWalsXs3C; Mon, 10 Jan 2022 15:44:29 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id CD263813E0; Mon, 10 Jan 2022 15:44:28 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id A522CC002F; Mon, 10 Jan 2022 15:44:28 +0000 (UTC) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id E91D9C001E for ; Mon, 10 Jan 2022 15:44:26 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id C0C62813F8 for ; Mon, 10 Jan 2022 15:44:26 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3igCf0xbgXkZ for ; Mon, 10 Jan 2022 15:44:25 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id CCBF8813E0 for ; Mon, 10 Jan 2022 15:44:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1641829464; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1GwsjTdK6v2LnBFhWI+JJOy9OQSY+yI1I9CwqbFn3SQ=; b=DvnUYEvpnzTAeUwOpuyl1FA4lSU6D58bvsSauGti+BwmRcfx32Tvyqesnmx6aqQ2Rkiuzf GUVZ0bLMiFlQfXXw4YZpepIPoYc9QsC8EMiv3lyYaIW5msmujJ+K9YhVp2m25ZKHl2bqwp b6kq25ecLpJnhNVjJmKrdvVrMIm1ucs= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-662-qq18qusGPmK1sPW0hMucZg-1; Mon, 10 Jan 2022 10:44:21 -0500 X-MC-Unique: qq18qusGPmK1sPW0hMucZg-1 Received: by mail-wm1-f72.google.com with SMTP id az9-20020a05600c600900b0034692565ca8so8069932wmb.9 for ; Mon, 10 Jan 2022 07:44:21 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=1GwsjTdK6v2LnBFhWI+JJOy9OQSY+yI1I9CwqbFn3SQ=; b=AXu77wIGJN1H8H9yu7o+I6dHfMOihRDuzQOc4twBcvNDvHDaSypsl/Who4Mn8iZZ5a 9GBdYt33YEHluM+6Ypv4P8RLSg6ChCWS1R0s6QWJkV2BffpoFKBYO150Jp+AauO5K7uq L9I5zkwUZw+W8YNlfLoMclXrE9n7UH/7Dbtyx9YMsahVb+R1c7yU3+nSjNswv8zEoa5y njdaqQ6RshYRbP3dIz0EsaBx+brGGGq6xAxqhBPNhpUs96QJGTXDGqTfy/YUraGb2EjR 6RKoNvGNh1+0fuJx8pgtXEvuPnsgpKRDrHg25vIJ+fBiGvXFt43BjipC1u2BZlcLfSQ/ Zu4A== X-Gm-Message-State: AOAM530TVVlo/orNHeFuUZvRmr78ba+Zq3P4J706pBLEVIz3URX7jI13 xKEcLBPeWhOS5Vzd0GOQDRbWqkOLV/2o74Vc5VU6ocOwn8m3M+tCOE94POTU9E9bwVttkrg/EPs BqZawxdnuu7Xs7d6nLGeGCs2Ka84UEyxp5hVGmMDw8A== X-Received: by 2002:adf:f807:: with SMTP id s7mr184299wrp.638.1641829460435; Mon, 10 Jan 2022 07:44:20 -0800 (PST) X-Google-Smtp-Source: ABdhPJwGlziNjGEYg/AKCCVR6dHJcBIW6ndhJ3HU0fYXJaw1aGDnRw1EG3ab/feKEGmla/24mvGOMw== X-Received: by 2002:adf:f807:: with SMTP id s7mr184278wrp.638.1641829460157; Mon, 10 Jan 2022 07:44:20 -0800 (PST) Received: from redhat.com ([2.55.148.228]) by smtp.gmail.com with ESMTPSA id s1sm6891060wmh.35.2022.01.10.07.44.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 07:44:19 -0800 (PST) Date: Mon, 10 Jan 2022 10:44:14 -0500 From: "Michael S. Tsirkin" To: Yongji Xie Subject: Re: [PATCH v12 00/13] Introduce VDUSE - vDPA Device in Userspace Message-ID: <20220110103938-mutt-send-email-mst@kernel.org> References: <20210830141737.181-1-xieyongji@bytedance.com> <20220110075546-mutt-send-email-mst@kernel.org> <20220110100911-mutt-send-email-mst@kernel.org> MIME-Version: 1.0 In-Reply-To: Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=mst@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Disposition: inline Cc: kvm , virtualization , Christian Brauner , Will Deacon , Jonathan Corbet , joro@8bytes.org, Matthew Wilcox , Christoph Hellwig , Dan Carpenter , John Garry , Liu Xiaodong , linux-fsdevel@vger.kernel.org, Al Viro , Stefan Hajnoczi , songmuchun@bytedance.com, Jens Axboe , He Zhe , Greg KH , Randy Dunlap , linux-kernel , iommu@lists.linux-foundation.org, bcrl@kvack.org, Netdev , Joe Perches , Robin Murphy , Mika =?iso-8859-1?Q?Penttil=E4?= X-BeenThere: virtualization@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Linux virtualization List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: virtualization-bounces@lists.linux-foundation.org Sender: "Virtualization" T24gTW9uLCBKYW4gMTAsIDIwMjIgYXQgMTE6MjQ6NDBQTSArMDgwMCwgWW9uZ2ppIFhpZSB3cm90 ZToKPiBPbiBNb24sIEphbiAxMCwgMjAyMiBhdCAxMToxMCBQTSBNaWNoYWVsIFMuIFRzaXJraW4g PG1zdEByZWRoYXQuY29tPiB3cm90ZToKPiA+Cj4gPiBPbiBNb24sIEphbiAxMCwgMjAyMiBhdCAw OTo1NDowOFBNICswODAwLCBZb25namkgWGllIHdyb3RlOgo+ID4gPiBPbiBNb24sIEphbiAxMCwg MjAyMiBhdCA4OjU3IFBNIE1pY2hhZWwgUy4gVHNpcmtpbiA8bXN0QHJlZGhhdC5jb20+IHdyb3Rl Ogo+ID4gPiA+Cj4gPiA+ID4gT24gTW9uLCBBdWcgMzAsIDIwMjEgYXQgMTA6MTc6MjRQTSArMDgw MCwgWGllIFlvbmdqaSB3cm90ZToKPiA+ID4gPiA+IFRoaXMgc2VyaWVzIGludHJvZHVjZXMgYSBm cmFtZXdvcmsgdGhhdCBtYWtlcyBpdCBwb3NzaWJsZSB0byBpbXBsZW1lbnQKPiA+ID4gPiA+IHNv ZnR3YXJlLWVtdWxhdGVkIHZEUEEgZGV2aWNlcyBpbiB1c2Vyc3BhY2UuIEFuZCB0byBtYWtlIHRo ZSBkZXZpY2UKPiA+ID4gPiA+IGVtdWxhdGlvbiBtb3JlIHNlY3VyZSwgdGhlIGVtdWxhdGVkIHZE UEEgZGV2aWNlJ3MgY29udHJvbCBwYXRoIGlzIGhhbmRsZWQKPiA+ID4gPiA+IGluIHRoZSBrZXJu ZWwgYW5kIG9ubHkgdGhlIGRhdGEgcGF0aCBpcyBpbXBsZW1lbnRlZCBpbiB0aGUgdXNlcnNwYWNl Lgo+ID4gPiA+ID4KPiA+ID4gPiA+IFNpbmNlIHRoZSBlbXVsZGF0ZWQgdkRQQSBkZXZpY2UncyBj b250cm9sIHBhdGggaXMgaGFuZGxlZCBpbiB0aGUga2VybmVsLAo+ID4gPiA+ID4gYSBtZXNzYWdl IG1lY2huaXNtIGlzIGludHJvZHVjZWQgdG8gbWFrZSB1c2Vyc3BhY2UgYmUgYXdhcmUgb2YgdGhl IGRhdGEKPiA+ID4gPiA+IHBhdGggcmVsYXRlZCBjaGFuZ2VzLiBVc2Vyc3BhY2UgY2FuIHVzZSBy ZWFkKCkvd3JpdGUoKSB0byByZWNlaXZlL3JlcGx5Cj4gPiA+ID4gPiB0aGUgY29udHJvbCBtZXNz YWdlcy4KPiA+ID4gPiA+Cj4gPiA+ID4gPiBJbiB0aGUgZGF0YSBwYXRoLCB0aGUgY29yZSBpcyBt YXBwaW5nIGRtYSBidWZmZXIgaW50byBWRFVTRSBkYWVtb24ncwo+ID4gPiA+ID4gYWRkcmVzcyBz cGFjZSwgd2hpY2ggY2FuIGJlIGltcGxlbWVudGVkIGluIGRpZmZlcmVudCB3YXlzIGRlcGVuZGlu ZyBvbgo+ID4gPiA+ID4gdGhlIHZkcGEgYnVzIHRvIHdoaWNoIHRoZSB2RFBBIGRldmljZSBpcyBh dHRhY2hlZC4KPiA+ID4gPiA+Cj4gPiA+ID4gPiBJbiB2aXJ0aW8tdmRwYSBjYXNlLCB3ZSBpbXBs ZW1lbnRzIGEgTU1VLWJhc2VkIHNvZnR3YXJlIElPVExCIHdpdGgKPiA+ID4gPiA+IGJvdW5jZS1i dWZmZXJpbmcgbWVjaGFuaXNtIHRvIGFjaGlldmUgdGhhdC4gQW5kIGluIHZob3N0LXZkcGEgY2Fz ZSwgdGhlIGRtYQo+ID4gPiA+ID4gYnVmZmVyIGlzIHJlc2lkZSBpbiBhIHVzZXJzcGFjZSBtZW1v cnkgcmVnaW9uIHdoaWNoIGNhbiBiZSBzaGFyZWQgdG8gdGhlCj4gPiA+ID4gPiBWRFVTRSB1c2Vy c3BhY2UgcHJvY2Vzc3MgdmlhIHRyYW5zZmVycmluZyB0aGUgc2htZmQuCj4gPiA+ID4gPgo+ID4g PiA+ID4gVGhlIGRldGFpbHMgYW5kIG91ciB1c2VyIGNhc2UgaXMgc2hvd24gYmVsb3c6Cj4gPiA+ ID4gPgo+ID4gPiA+ID4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tICAgIC0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0gICAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tCj4gPiA+ID4gPiB8ICAgICAgICAgICAgQ29udGFpbmVyIHwgICAgfCAgICAgICAgICAgICAg UUVNVShWTSkgfCAgIHwgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgVkRVU0UgZGFlbW9u IHwKPiA+ID4gPiA+IHwgICAgICAgLS0tLS0tLS0tICAgICAgfCAgICB8ICAtLS0tLS0tLS0tLS0t LS0tLS0tICB8ICAgfCAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tIC0tLS0tLS0tLS0tLS0tLS0g fAo+ID4gPiA+ID4gfCAgICAgICB8ZGV2L3ZkeHwgICAgICB8ICAgIHwgIHwvZGV2L3Zob3N0LXZk cGEteHwgIHwgICB8IHwgdkRQQSBkZXZpY2UgZW11bGF0aW9uIHwgfCBibG9jayBkcml2ZXIgfCB8 Cj4gPiA+ID4gPiAtLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0gICAgIC0tLS0tLS0tLS0tKy0tLS0t LS0tLS0tLSAgIC0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0K PiA+ID4gPiA+ICAgICAgICAgICAgIHwgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAg ICAgICAgICAgICAgICAgICAgICAgIHwgICAgICAgICAgICAgICAgICAgICAgfAo+ID4gPiA+ID4g ICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgIHwgICAgICAgICAgICAgICAg ICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICB8Cj4gPiA+ID4gPiAtLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0KPiA+ID4gPiA+IHwgICAgfCBibG9j ayBkZXZpY2UgfCAgICAgICAgICAgfCAgdmhvc3QgZGV2aWNlIHwgICAgICAgICAgICB8IHZkdXNl IGRyaXZlciB8ICAgICAgICAgIHwgVENQL0lQIHwgICAgfAo+ID4gPiA+ID4gfCAgICAtLS0tLS0t Ky0tLS0tLS0tICAgICAgICAgICAtLS0tLS0tLSstLS0tLS0tLSAgICAgICAgICAgIC0tLS0tLS0r LS0tLS0tLS0gICAgICAgICAgLS0tLS0rLS0tLSAgICB8Cj4gPiA+ID4gPiB8ICAgICAgICAgICB8 ICAgICAgICAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgIHwg ICAgICAgICAgICAgICAgICAgICAgIHwgICAgICAgIHwKPiA+ID4gPiA+IHwgLS0tLS0tLS0tLSst LS0tLS0tLS0tICAgICAgIC0tLS0tLS0tLS0rLS0tLS0tLS0tLS0gICAgICAgICAtLS0tLS0tKy0t LS0tLS0gICAgICAgICAgICAgICAgfCAgICAgICAgfAo+ID4gPiA+ID4gfCB8IHZpcnRpby1ibGsg ZHJpdmVyIHwgICAgICAgfCAgdmhvc3QtdmRwYSBkcml2ZXIgfCAgICAgICAgIHwgdmRwYSBkZXZp Y2UgfCAgICAgICAgICAgICAgICB8ICAgICAgICB8Cj4gPiA+ID4gPiB8IC0tLS0tLS0tLS0rLS0t LS0tLS0tLSAgICAgICAtLS0tLS0tLS0tKy0tLS0tLS0tLS0tICAgICAgICAgLS0tLS0tLSstLS0t LS0tICAgICAgICAgICAgICAgIHwgICAgICAgIHwKPiA+ID4gPiA+IHwgICAgICAgICAgIHwgICAg ICB2aXJ0aW8gYnVzICAgICAgICAgICB8ICAgICAgICAgICAgICAgICAgICAgICAgICAgfCAgICAg ICAgICAgICAgICAgICAgICAgfCAgICAgICAgfAo+ID4gPiA+ID4gfCAgIC0tLS0tLS0tKy0tLS0r LS0tLS0tLS0tLS0gICAgICAgICAgIHwgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAg ICAgICAgICAgICAgICAgICB8ICAgICAgICB8Cj4gPiA+ID4gPiB8ICAgICAgICAgICAgICAgIHwg ICAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgIHwgICAgICAg ICAgICAgICAgICAgICAgIHwgICAgICAgIHwKPiA+ID4gPiA+IHwgICAgICAtLS0tLS0tLS0tKy0t LS0tLS0tLS0gICAgICAgICAgICB8ICAgICAgICAgICAgICAgICAgICAgICAgICAgfCAgICAgICAg ICAgICAgICAgICAgICAgfCAgICAgICAgfAo+ID4gPiA+ID4gfCAgICAgIHwgdmlydGlvLWJsayBk ZXZpY2UgfCAgICAgICAgICAgIHwgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAg ICAgICAgICAgICAgICB8ICAgICAgICB8Cj4gPiA+ID4gPiB8ICAgICAgLS0tLS0tLS0tLSstLS0t LS0tLS0tICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgIHwgICAgICAgICAg ICAgICAgICAgICAgIHwgICAgICAgIHwKPiA+ID4gPiA+IHwgICAgICAgICAgICAgICAgfCAgICAg ICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAg ICAgICAgICAgICAgfCAgICAgICAgfAo+ID4gPiA+ID4gfCAgICAgLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0gICAgICAgICAgIHwgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAg ICAgICAgICAgICB8ICAgICAgICB8Cj4gPiA+ID4gPiB8ICAgICB8ICB2aXJ0aW8tdmRwYSBkcml2 ZXIgfCAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgIHwgICAgICAgICAgICAg ICAgICAgICAgIHwgICAgICAgIHwKPiA+ID4gPiA+IHwgICAgIC0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tICAgICAgICAgICB8ICAgICAgICAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAg ICAgICAgICAgfCAgICAgICAgfAo+ID4gPiA+ID4gfCAgICAgICAgICAgICAgICB8ICAgICAgICAg ICAgICAgICAgICAgIHwgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgIHZkcGEgYnVzICAg ICAgICAgICB8ICAgICAgICB8Cj4gPiA+ID4gPiB8ICAgICAtLS0tLS0tLS0tLSstLS0tLS0tLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0gICAg ICAgICAgIHwgICAgICAgIHwKPiA+ID4gPiA+IHwgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgLS0tKy0tLSAgICAgfAo+ID4gPiA+ID4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS18IE5JQyB8LS0tLS0tCj4gPiA+ID4gPiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IC0tLSstLS0KPiA+ID4gPiA+ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfAo+ ID4gPiA+ID4gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAtLS0tLS0tLS0rLS0tLS0tLS0tCj4g PiA+ID4gPiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwgUmVtb3RlIFN0b3JhZ2VzIHwKPiA+ ID4gPiA+ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLS0tLS0tLS0tLS0tLS0tLS0tLQo+ID4g PiA+ID4KPiA+ID4gPiA+IFdlIG1ha2UgdXNlIG9mIGl0IHRvIGltcGxlbWVudCBhIGJsb2NrIGRl dmljZSBjb25uZWN0aW5nIHRvCj4gPiA+ID4gPiBvdXIgZGlzdHJpYnV0ZWQgc3RvcmFnZSwgd2hp Y2ggY2FuIGJlIHVzZWQgYm90aCBpbiBjb250YWluZXJzIGFuZAo+ID4gPiA+ID4gVk1zLiBUaHVz LCB3ZSBjYW4gaGF2ZSBhbiB1bmlmaWVkIHRlY2hub2xvZ3kgc3RhY2sgaW4gdGhpcyB0d28gY2Fz ZXMuCj4gPiA+ID4gPgo+ID4gPiA+ID4gVG8gdGVzdCBpdCB3aXRoIG51bGwtYmxrOgo+ID4gPiA+ ID4KPiA+ID4gPiA+ICAgJCBxZW11LXN0b3JhZ2UtZGFlbW9uIFwKPiA+ID4gPiA+ICAgICAgIC0t Y2hhcmRldiBzb2NrZXQsaWQ9Y2hhcm1vbml0b3IscGF0aD0vdG1wL3FtcC5zb2NrLHNlcnZlcixu b3dhaXQgXAo+ID4gPiA+ID4gICAgICAgLS1tb25pdG9yIGNoYXJkZXY9Y2hhcm1vbml0b3IgXAo+ ID4gPiA+ID4gICAgICAgLS1ibG9ja2RldiBkcml2ZXI9aG9zdF9kZXZpY2UsY2FjaGUuZGlyZWN0 PW9uLGFpbz1uYXRpdmUsZmlsZW5hbWU9L2Rldi9udWxsYjAsbm9kZS1uYW1lPWRpc2swIFwKPiA+ ID4gPiA+ICAgICAgIC0tZXhwb3J0IHR5cGU9dmR1c2UtYmxrLGlkPXRlc3Qsbm9kZS1uYW1lPWRp c2swLHdyaXRhYmxlPW9uLG5hbWU9dmR1c2UtbnVsbCxudW0tcXVldWVzPTE2LHF1ZXVlLXNpemU9 MTI4Cj4gPiA+ID4gPgo+ID4gPiA+ID4gVGhlIHFlbXUtc3RvcmFnZS1kYWVtb24gY2FuIGJlIGZv dW5kIGF0IGh0dHBzOi8vZ2l0aHViLmNvbS9ieXRlZGFuY2UvcWVtdS90cmVlL3ZkdXNlCj4gPiA+ ID4KPiA+ID4gPiBJdCdzIGJlZW4gaGFsZiBhIHllYXIgLSBhbnkgcGxhbnMgdG8gdXBzdHJlYW0g dGhpcz8KPiA+ID4KPiA+ID4gWWVhaCwgdGhpcyBpcyBvbiBteSB0by1kbyBsaXN0IHRoaXMgbW9u dGguCj4gPiA+Cj4gPiA+IFNvcnJ5IGZvciB0YWtpbmcgc28gbG9uZy4uLiBJJ3ZlIGJlZW4gd29y a2luZyBvbiBhbm90aGVyIHByb2plY3QKPiA+ID4gZW5hYmxpbmcgdXNlcnNwYWNlIFJETUEgd2l0 aCBWRFVTRSBmb3IgdGhlIHBhc3QgZmV3IG1vbnRocy4gU28gSQo+ID4gPiBkaWRuJ3QgaGF2ZSBt dWNoIHRpbWUgZm9yIHRoaXMuIEFueXdheSwgSSB3aWxsIHN1Ym1pdCB0aGUgZmlyc3QKPiA+ID4g dmVyc2lvbiBhcyBzb29uIGFzIHBvc3NpYmxlLgo+ID4gPgo+ID4gPiBUaGFua3MsCj4gPiA+IFlv bmdqaQo+ID4KPiA+IE9oIGZ1bi4gWW91IG1lYW4gbGlrZSB2aXJ0aW8tcmRtYT8gT3IgUkRNQSBh cyBhIGJhY2tlbmQgZm9yIHJlZ3VsYXIKPiA+IHZpcnRpbz8KPiA+Cj4gCj4gWWVzLCBsaWtlIHZp cnRpby1yZG1hLiBUaGVuIHdlIGNhbiBkZXZlbG9wIHNvbWV0aGluZyBsaWtlIHVzZXJzcGFjZQo+ IHJ4ZeOAgXNpdyBvciBjdXN0b20gcHJvdG9jb2wgd2l0aCBWRFVTRS4KPiAKPiBUaGFua3MsCj4g WW9uZ2ppCgpXb3VsZCBiZSBpbnRlcmVzdGluZyB0byBzZWUgdGhlIHNwZWMgZm9yIHRoYXQuClRo ZSBpc3N1ZXMgd2l0aCBSRE1BIHJldm9sdmVkIGFyb3VuZCB0aGUgZmFjdCB0aGF0IGN1cnJlbnQK YXBwcyB0ZW5kIHRvIGVpdGhlciB1c2Ugbm9uLXN0YW5kYXJkIHByb3BvY29scyBmb3IgY29ubmVj dGlvbgplc3RhYmxpc2htZW50IG9yIHVzZSBVRCB3aGVyZSB0aGVyZSdzIElJUkMgbm8gc3RhbmRh cmQKYXQgYWxsLiBTbyBRUCBudW1iZXJzIGFyZSBoYXJkIHRvIHZpcnR1YWxpemUuClNpbWlsYXJs eSBtYW55IHVzZSBMSURzIGRpcmVjdGx5IHdpdGggdGhlIHNhbWUgZWZmZWN0LgpHVUlEcyBtaWdo dCBiZSB2aXJ0dWFsaXplYWJsZSBidXQgbm8gb25lIHdlbnQgdG8gdGhlIGVmZm9ydC4KClRvIHNh eSBub3RoaW5nIGFib3V0IHRoZSBpbnRlcmFjdGlvbiB3aXRoIG1lbW9yeSBvdmVyY29tbWl0LgoK LS0gCk1TVAoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18K VmlydHVhbGl6YXRpb24gbWFpbGluZyBsaXN0ClZpcnR1YWxpemF0aW9uQGxpc3RzLmxpbnV4LWZv dW5kYXRpb24ub3JnCmh0dHBzOi8vbGlzdHMubGludXhmb3VuZGF0aW9uLm9yZy9tYWlsbWFuL2xp c3RpbmZvL3ZpcnR1YWxpemF0aW9u From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 852A8C433F5 for ; Mon, 10 Jan 2022 15:44:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236401AbiAJPoY (ORCPT ); Mon, 10 Jan 2022 10:44:24 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:46632 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236361AbiAJPoX (ORCPT ); Mon, 10 Jan 2022 10:44:23 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1641829462; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1GwsjTdK6v2LnBFhWI+JJOy9OQSY+yI1I9CwqbFn3SQ=; b=BBe4NaoKQSbWyzM93tQDsHo2EmSH51X5nnoZWp5KwDypX/ekOsoHDd7XDz+dTiWJ6R4jDO ZjbTtht5MYmc7FRRkiiVF79G+poA9M1GyIjY9wC+93DDqcgFLE91hz+czZQhxoMpaDB6pn AGkgOu6dPrY5MLVMPX8CGLZgv99WFes= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-652-tv6vxxLkNEynoM9BibVYhw-1; Mon, 10 Jan 2022 10:44:21 -0500 X-MC-Unique: tv6vxxLkNEynoM9BibVYhw-1 Received: by mail-wm1-f71.google.com with SMTP id i81-20020a1c3b54000000b003467c58cbddso8727673wma.5 for ; Mon, 10 Jan 2022 07:44:21 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=1GwsjTdK6v2LnBFhWI+JJOy9OQSY+yI1I9CwqbFn3SQ=; b=UUxtMEIbOP/ujO49Luj2BX0a0QYOgEE9kkc6ejC+ImqPYU7wP49zXP36khv7a4m6IV GcCwNaHn14UMDNTLwZK7lC75wjPvJO2bJdUvj63Mn/YFUONA6E/jeo5iwm6SKz0Dcizv wRYOT6pqw1h7DXArSj6Z+dlvz17yfNtuI7ORyfGhI1Iau6zwZSl/TzBkHudDxaon1lDz 14mtMKAleiOK7HBRVY7copx/ZNTIfXI1oHjpvRjMEUX6eQy+eAdcRHr2K3Av+5tJ3anF RqYBIRI/SBOPW3jp9YTdMnXwjIyLfMHzPNRk+jL7fivmvPmk+eU8agmxBQ1+TW3Pb90T /m2g== X-Gm-Message-State: AOAM533gaQKZUvX9eDppNNAX8+PWtZETmFd5P1g/quVGWAQAXtj99rpX 5rvNI9CIP5Wxk/4axr2HWK+RqPpKnjNHkgDbxJjQOdrVCJfYWuRjg+lVVW0kCEmpI2C3fcciC4g 1xn1FMM49qqG2 X-Received: by 2002:adf:f807:: with SMTP id s7mr184307wrp.638.1641829460437; Mon, 10 Jan 2022 07:44:20 -0800 (PST) X-Google-Smtp-Source: ABdhPJwGlziNjGEYg/AKCCVR6dHJcBIW6ndhJ3HU0fYXJaw1aGDnRw1EG3ab/feKEGmla/24mvGOMw== X-Received: by 2002:adf:f807:: with SMTP id s7mr184278wrp.638.1641829460157; Mon, 10 Jan 2022 07:44:20 -0800 (PST) Received: from redhat.com ([2.55.148.228]) by smtp.gmail.com with ESMTPSA id s1sm6891060wmh.35.2022.01.10.07.44.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 07:44:19 -0800 (PST) Date: Mon, 10 Jan 2022 10:44:14 -0500 From: "Michael S. Tsirkin" To: Yongji Xie Cc: Jason Wang , Stefan Hajnoczi , Stefano Garzarella , Parav Pandit , Christoph Hellwig , Christian Brauner , Randy Dunlap , Matthew Wilcox , Al Viro , Jens Axboe , bcrl@kvack.org, Jonathan Corbet , Mika =?iso-8859-1?Q?Penttil=E4?= , Dan Carpenter , joro@8bytes.org, Greg KH , He Zhe , Liu Xiaodong , Joe Perches , Robin Murphy , Will Deacon , John Garry , songmuchun@bytedance.com, virtualization , Netdev , kvm , linux-fsdevel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-kernel Subject: Re: [PATCH v12 00/13] Introduce VDUSE - vDPA Device in Userspace Message-ID: <20220110103938-mutt-send-email-mst@kernel.org> References: <20210830141737.181-1-xieyongji@bytedance.com> <20220110075546-mutt-send-email-mst@kernel.org> <20220110100911-mutt-send-email-mst@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On Mon, Jan 10, 2022 at 11:24:40PM +0800, Yongji Xie wrote: > On Mon, Jan 10, 2022 at 11:10 PM Michael S. Tsirkin wrote: > > > > On Mon, Jan 10, 2022 at 09:54:08PM +0800, Yongji Xie wrote: > > > On Mon, Jan 10, 2022 at 8:57 PM Michael S. Tsirkin wrote: > > > > > > > > On Mon, Aug 30, 2021 at 10:17:24PM +0800, Xie Yongji wrote: > > > > > This series introduces a framework that makes it possible to implement > > > > > software-emulated vDPA devices in userspace. And to make the device > > > > > emulation more secure, the emulated vDPA device's control path is handled > > > > > in the kernel and only the data path is implemented in the userspace. > > > > > > > > > > Since the emuldated vDPA device's control path is handled in the kernel, > > > > > a message mechnism is introduced to make userspace be aware of the data > > > > > path related changes. Userspace can use read()/write() to receive/reply > > > > > the control messages. > > > > > > > > > > In the data path, the core is mapping dma buffer into VDUSE daemon's > > > > > address space, which can be implemented in different ways depending on > > > > > the vdpa bus to which the vDPA device is attached. > > > > > > > > > > In virtio-vdpa case, we implements a MMU-based software IOTLB with > > > > > bounce-buffering mechanism to achieve that. And in vhost-vdpa case, the dma > > > > > buffer is reside in a userspace memory region which can be shared to the > > > > > VDUSE userspace processs via transferring the shmfd. > > > > > > > > > > The details and our user case is shown below: > > > > > > > > > > ------------------------ ------------------------- ---------------------------------------------- > > > > > | Container | | QEMU(VM) | | VDUSE daemon | > > > > > | --------- | | ------------------- | | ------------------------- ---------------- | > > > > > | |dev/vdx| | | |/dev/vhost-vdpa-x| | | | vDPA device emulation | | block driver | | > > > > > ------------+----------- -----------+------------ -------------+----------------------+--------- > > > > > | | | | > > > > > | | | | > > > > > ------------+---------------------------+----------------------------+----------------------+--------- > > > > > | | block device | | vhost device | | vduse driver | | TCP/IP | | > > > > > | -------+-------- --------+-------- -------+-------- -----+---- | > > > > > | | | | | | > > > > > | ----------+---------- ----------+----------- -------+------- | | > > > > > | | virtio-blk driver | | vhost-vdpa driver | | vdpa device | | | > > > > > | ----------+---------- ----------+----------- -------+------- | | > > > > > | | virtio bus | | | | > > > > > | --------+----+----------- | | | | > > > > > | | | | | | > > > > > | ----------+---------- | | | | > > > > > | | virtio-blk device | | | | | > > > > > | ----------+---------- | | | | > > > > > | | | | | | > > > > > | -----------+----------- | | | | > > > > > | | virtio-vdpa driver | | | | | > > > > > | -----------+----------- | | | | > > > > > | | | | vdpa bus | | > > > > > | -----------+----------------------+---------------------------+------------ | | > > > > > | ---+--- | > > > > > -----------------------------------------------------------------------------------------| NIC |------ > > > > > ---+--- > > > > > | > > > > > ---------+--------- > > > > > | Remote Storages | > > > > > ------------------- > > > > > > > > > > We make use of it to implement a block device connecting to > > > > > our distributed storage, which can be used both in containers and > > > > > VMs. Thus, we can have an unified technology stack in this two cases. > > > > > > > > > > To test it with null-blk: > > > > > > > > > > $ qemu-storage-daemon \ > > > > > --chardev socket,id=charmonitor,path=/tmp/qmp.sock,server,nowait \ > > > > > --monitor chardev=charmonitor \ > > > > > --blockdev driver=host_device,cache.direct=on,aio=native,filename=/dev/nullb0,node-name=disk0 \ > > > > > --export type=vduse-blk,id=test,node-name=disk0,writable=on,name=vduse-null,num-queues=16,queue-size=128 > > > > > > > > > > The qemu-storage-daemon can be found at https://github.com/bytedance/qemu/tree/vduse > > > > > > > > It's been half a year - any plans to upstream this? > > > > > > Yeah, this is on my to-do list this month. > > > > > > Sorry for taking so long... I've been working on another project > > > enabling userspace RDMA with VDUSE for the past few months. So I > > > didn't have much time for this. Anyway, I will submit the first > > > version as soon as possible. > > > > > > Thanks, > > > Yongji > > > > Oh fun. You mean like virtio-rdma? Or RDMA as a backend for regular > > virtio? > > > > Yes, like virtio-rdma. Then we can develop something like userspace > rxe、siw or custom protocol with VDUSE. > > Thanks, > Yongji Would be interesting to see the spec for that. The issues with RDMA revolved around the fact that current apps tend to either use non-standard propocols for connection establishment or use UD where there's IIRC no standard at all. So QP numbers are hard to virtualize. Similarly many use LIDs directly with the same effect. GUIDs might be virtualizeable but no one went to the effort. To say nothing about the interaction with memory overcommit. -- MST