From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BA4FC43381 for ; Tue, 12 Mar 2019 05:14:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2AC9D2087C for ; Tue, 12 Mar 2019 05:14:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=hansenpartnership.com header.i=@hansenpartnership.com header.b="xYRhAyEt" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726218AbfCLFOs (ORCPT ); Tue, 12 Mar 2019 01:14:48 -0400 Received: from bedivere.hansenpartnership.com ([66.63.167.143]:59170 "EHLO bedivere.hansenpartnership.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725792AbfCLFOr (ORCPT ); Tue, 12 Mar 2019 01:14:47 -0400 Received: from localhost (localhost [127.0.0.1]) by bedivere.hansenpartnership.com (Postfix) with ESMTP id 13D308EE14F; Mon, 11 Mar 2019 22:14:47 -0700 (PDT) Received: from bedivere.hansenpartnership.com ([127.0.0.1]) by localhost (bedivere.hansenpartnership.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7WtYXw-QTR3T; Mon, 11 Mar 2019 22:14:46 -0700 (PDT) Received: from [153.66.254.194] (unknown [50.35.68.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bedivere.hansenpartnership.com (Postfix) with ESMTPSA id 3BBB58EE130; Mon, 11 Mar 2019 22:14:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=hansenpartnership.com; s=20151216; t=1552367686; bh=AW3dolA78+lzEd83tz6QQ+NipDZU+vh1dNrlssMHnVU=; h=Subject:From:To:Cc:Date:In-Reply-To:References:From; b=xYRhAyEtbC9eyaeABxwkOllNzY+9kjpj9FhbZaQeAldW3lO5vxwuVmjyIp3KNFqBG LD4cAqF1Nc+zpH6hIwFgFhhJvaNSxmq46EBcnVR1m2djSf239RlCYCBinxR2v67gmY JHvGMLPDK4DrxvhtR46iCuydqs9rTvtKEn+Y8lKk= Message-ID: <1552367685.23859.22.camel@HansenPartnership.com> Subject: Re: [RFC PATCH V2 0/5] vhost: accelerate metadata access through vmap() From: James Bottomley To: Jason Wang , David Miller , mst@redhat.com Cc: hch@infradead.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, peterx@redhat.com, linux-mm@kvack.org, aarcange@redhat.com, linux-arm-kernel@lists.infradead.org, linux-parisc@vger.kernel.org Date: Mon, 11 Mar 2019 22:14:45 -0700 In-Reply-To: <6b6dcc4a-2f08-ba67-0423-35787f3b966c@redhat.com> References: <20190308141220.GA21082@infradead.org> <56374231-7ba7-0227-8d6d-4d968d71b4d6@redhat.com> <20190311095405-mutt-send-email-mst@kernel.org> <20190311.111413.1140896328197448401.davem@davemloft.net> <6b6dcc4a-2f08-ba67-0423-35787f3b966c@redhat.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.26.6 Mime-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: linux-parisc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-parisc@vger.kernel.org On Tue, 2019-03-12 at 10:59 +0800, Jason Wang wrote: > On 2019/3/12 上午2:14, David Miller wrote: > > From: "Michael S. Tsirkin" > > Date: Mon, 11 Mar 2019 09:59:28 -0400 > > > > > On Mon, Mar 11, 2019 at 03:13:17PM +0800, Jason Wang wrote: > > > > On 2019/3/8 下午10:12, Christoph Hellwig wrote: > > > > > On Wed, Mar 06, 2019 at 02:18:07AM -0500, Jason Wang wrote: > > > > > > This series tries to access virtqueue metadata through > > > > > > kernel virtual > > > > > > address instead of copy_user() friends since they had too > > > > > > much > > > > > > overheads like checks, spec barriers or even hardware > > > > > > feature > > > > > > toggling. This is done through setup kernel address through > > > > > > vmap() and > > > > > > resigter MMU notifier for invalidation. > > > > > > > > > > > > Test shows about 24% improvement on TX PPS. TCP_STREAM > > > > > > doesn't see > > > > > > obvious improvement. > > > > > > > > > > How is this going to work for CPUs with virtually tagged > > > > > caches? > > > > > > > > Anything different that you worry? > > > > > > If caches have virtual tags then kernel and userspace view of > > > memory > > > might not be automatically in sync if they access memory > > > through different virtual addresses. You need to do things like > > > flush_cache_page, probably multiple times. > > > > "flush_dcache_page()" > > > I get this. Then I think the current set_bit_to_user() is suspicious, > we > probably miss a flush_dcache_page() there: > > > static int set_bit_to_user(int nr, void __user *addr) > { > unsigned long log = (unsigned long)addr; > struct page *page; > void *base; > int bit = nr + (log % PAGE_SIZE) * 8; > int r; > > r = get_user_pages_fast(log, 1, 1, &page); > if (r < 0) > return r; > BUG_ON(r != 1); > base = kmap_atomic(page); > set_bit(bit, base); > kunmap_atomic(base); This sequence should be OK. get_user_pages() contains a flush which clears the cache above the user virtual address, so on kmap, the page is coherent at the new alias. On parisc at least, kunmap embodies a flush_dcache_page() which pushes any changes in the cache above the kernel virtual address back to main memory and makes it coherent again for the user alias to pick it up. James From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CFC9C43381 for ; Tue, 12 Mar 2019 05:14:57 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 529882087C for ; Tue, 12 Mar 2019 05:14:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="o/SXzH/U"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=hansenpartnership.com header.i=@hansenpartnership.com header.b="xYRhAyEt" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 529882087C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=HansenPartnership.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Mime-Version:References:In-Reply-To: Date:To:From:Subject:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=1K7xHAAjEkX4qxPUL8OtdB9DR1ZE6tzvlI0cu+mF90I=; b=o/SXzH/U4yQCz1 Hr/mJzBrkMs9d2jLP0fAnKHZ3x7j/UGCj+YGgMiD0xaOqeQwBSGYe1/OVeTEvTsV3XYGOaxHHdG0w 4e2wH+CAWjxk3iSrkSIOiMwa3eShsbU6XZkK8HB4itoobz8gMtQR+hiOl1qxjHWaM/JlpMjhscaVH W/NM7FEC0firM5xU9J/AgMzqQcA+btEBjNXSXTx3efPNrUW4pEknJVNJtaxlxPWJACtqUgP8j/vzg PvrTw/13V7TLEoEbnJ3xndIrn5f2XQlkZNXotEbXdiUbeXkGlLKLiQGzst1tkXUvnXwMa/NvBNGPe doVizAEsJ2ijaK9JkXog==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1h3Zka-0000aN-T4; Tue, 12 Mar 2019 05:14:52 +0000 Received: from bedivere.hansenpartnership.com ([66.63.167.143]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1h3ZkX-0000ZQ-Dw for linux-arm-kernel@lists.infradead.org; Tue, 12 Mar 2019 05:14:50 +0000 Received: from localhost (localhost [127.0.0.1]) by bedivere.hansenpartnership.com (Postfix) with ESMTP id 13D308EE14F; Mon, 11 Mar 2019 22:14:47 -0700 (PDT) Received: from bedivere.hansenpartnership.com ([127.0.0.1]) by localhost (bedivere.hansenpartnership.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7WtYXw-QTR3T; Mon, 11 Mar 2019 22:14:46 -0700 (PDT) Received: from [153.66.254.194] (unknown [50.35.68.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bedivere.hansenpartnership.com (Postfix) with ESMTPSA id 3BBB58EE130; Mon, 11 Mar 2019 22:14:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=hansenpartnership.com; s=20151216; t=1552367686; bh=AW3dolA78+lzEd83tz6QQ+NipDZU+vh1dNrlssMHnVU=; h=Subject:From:To:Cc:Date:In-Reply-To:References:From; b=xYRhAyEtbC9eyaeABxwkOllNzY+9kjpj9FhbZaQeAldW3lO5vxwuVmjyIp3KNFqBG LD4cAqF1Nc+zpH6hIwFgFhhJvaNSxmq46EBcnVR1m2djSf239RlCYCBinxR2v67gmY JHvGMLPDK4DrxvhtR46iCuydqs9rTvtKEn+Y8lKk= Message-ID: <1552367685.23859.22.camel@HansenPartnership.com> Subject: Re: [RFC PATCH V2 0/5] vhost: accelerate metadata access through vmap() From: James Bottomley To: Jason Wang , David Miller , mst@redhat.com Date: Mon, 11 Mar 2019 22:14:45 -0700 In-Reply-To: <6b6dcc4a-2f08-ba67-0423-35787f3b966c@redhat.com> References: <20190308141220.GA21082@infradead.org> <56374231-7ba7-0227-8d6d-4d968d71b4d6@redhat.com> <20190311095405-mutt-send-email-mst@kernel.org> <20190311.111413.1140896328197448401.davem@davemloft.net> <6b6dcc4a-2f08-ba67-0423-35787f3b966c@redhat.com> X-Mailer: Evolution 3.26.6 Mime-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190311_221449_484434_34277582 X-CRM114-Status: GOOD ( 19.03 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com, linux-parisc@vger.kernel.org, kvm@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, peterx@redhat.com, virtualization@lists.linux-foundation.org, hch@infradead.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gVHVlLCAyMDE5LTAzLTEyIGF0IDEwOjU5ICswODAwLCBKYXNvbiBXYW5nIHdyb3RlOgo+IE9u IDIwMTkvMy8xMiDkuIrljYgyOjE0LCBEYXZpZCBNaWxsZXIgd3JvdGU6Cj4gPiBGcm9tOiAiTWlj aGFlbCBTLiBUc2lya2luIiA8bXN0QHJlZGhhdC5jb20+Cj4gPiBEYXRlOiBNb24sIDExIE1hciAy MDE5IDA5OjU5OjI4IC0wNDAwCj4gPiAKPiA+ID4gT24gTW9uLCBNYXIgMTEsIDIwMTkgYXQgMDM6 MTM6MTdQTSArMDgwMCwgSmFzb24gV2FuZyB3cm90ZToKPiA+ID4gPiBPbiAyMDE5LzMvOCDkuIvl jYgxMDoxMiwgQ2hyaXN0b3BoIEhlbGx3aWcgd3JvdGU6Cj4gPiA+ID4gPiBPbiBXZWQsIE1hciAw NiwgMjAxOSBhdCAwMjoxODowN0FNIC0wNTAwLCBKYXNvbiBXYW5nIHdyb3RlOgo+ID4gPiA+ID4g PiBUaGlzIHNlcmllcyB0cmllcyB0byBhY2Nlc3MgdmlydHF1ZXVlIG1ldGFkYXRhIHRocm91Z2gK PiA+ID4gPiA+ID4ga2VybmVsIHZpcnR1YWwKPiA+ID4gPiA+ID4gYWRkcmVzcyBpbnN0ZWFkIG9m IGNvcHlfdXNlcigpIGZyaWVuZHMgc2luY2UgdGhleSBoYWQgdG9vCj4gPiA+ID4gPiA+IG11Y2gK PiA+ID4gPiA+ID4gb3ZlcmhlYWRzIGxpa2UgY2hlY2tzLCBzcGVjIGJhcnJpZXJzIG9yIGV2ZW4g aGFyZHdhcmUKPiA+ID4gPiA+ID4gZmVhdHVyZQo+ID4gPiA+ID4gPiB0b2dnbGluZy4gVGhpcyBp cyBkb25lIHRocm91Z2ggc2V0dXAga2VybmVsIGFkZHJlc3MgdGhyb3VnaAo+ID4gPiA+ID4gPiB2 bWFwKCkgYW5kCj4gPiA+ID4gPiA+IHJlc2lndGVyIE1NVSBub3RpZmllciBmb3IgaW52YWxpZGF0 aW9uLgo+ID4gPiA+ID4gPiAKPiA+ID4gPiA+ID4gVGVzdCBzaG93cyBhYm91dCAyNCUgaW1wcm92 ZW1lbnQgb24gVFggUFBTLiBUQ1BfU1RSRUFNCj4gPiA+ID4gPiA+IGRvZXNuJ3Qgc2VlCj4gPiA+ ID4gPiA+IG9idmlvdXMgaW1wcm92ZW1lbnQuCj4gPiA+ID4gPiAKPiA+ID4gPiA+IEhvdyBpcyB0 aGlzIGdvaW5nIHRvIHdvcmsgZm9yIENQVXMgd2l0aCB2aXJ0dWFsbHkgdGFnZ2VkCj4gPiA+ID4g PiBjYWNoZXM/Cj4gPiA+ID4gCj4gPiA+ID4gQW55dGhpbmcgZGlmZmVyZW50IHRoYXQgeW91IHdv cnJ5Pwo+ID4gPiAKPiA+ID4gSWYgY2FjaGVzIGhhdmUgdmlydHVhbCB0YWdzIHRoZW4ga2VybmVs IGFuZCB1c2Vyc3BhY2UgdmlldyBvZgo+ID4gPiBtZW1vcnkKPiA+ID4gbWlnaHQgbm90IGJlIGF1 dG9tYXRpY2FsbHkgaW4gc3luYyBpZiB0aGV5IGFjY2VzcyBtZW1vcnkKPiA+ID4gdGhyb3VnaCBk aWZmZXJlbnQgdmlydHVhbCBhZGRyZXNzZXMuIFlvdSBuZWVkIHRvIGRvIHRoaW5ncyBsaWtlCj4g PiA+IGZsdXNoX2NhY2hlX3BhZ2UsIHByb2JhYmx5IG11bHRpcGxlIHRpbWVzLgo+ID4gCj4gPiAi Zmx1c2hfZGNhY2hlX3BhZ2UoKSIKPiAKPiAKPiBJIGdldCB0aGlzLiBUaGVuIEkgdGhpbmsgdGhl IGN1cnJlbnQgc2V0X2JpdF90b191c2VyKCkgaXMgc3VzcGljaW91cywKPiB3ZSAKPiBwcm9iYWJs eSBtaXNzIGEgZmx1c2hfZGNhY2hlX3BhZ2UoKSB0aGVyZToKPiAKPiAKPiBzdGF0aWMgaW50IHNl dF9iaXRfdG9fdXNlcihpbnQgbnIsIHZvaWQgX191c2VyICphZGRyKQo+IHsKPiAgICAgICAgICB1 bnNpZ25lZCBsb25nIGxvZyA9ICh1bnNpZ25lZCBsb25nKWFkZHI7Cj4gICAgICAgICAgc3RydWN0 IHBhZ2UgKnBhZ2U7Cj4gICAgICAgICAgdm9pZCAqYmFzZTsKPiAgICAgICAgICBpbnQgYml0ID0g bnIgKyAobG9nICUgUEFHRV9TSVpFKSAqIDg7Cj4gICAgICAgICAgaW50IHI7Cj4gCj4gICAgICAg ICAgciA9IGdldF91c2VyX3BhZ2VzX2Zhc3QobG9nLCAxLCAxLCAmcGFnZSk7Cj4gICAgICAgICAg aWYgKHIgPCAwKQo+ICAgICAgICAgICAgICAgICAgcmV0dXJuIHI7Cj4gICAgICAgICAgQlVHX09O KHIgIT0gMSk7Cj4gICAgICAgICAgYmFzZSA9IGttYXBfYXRvbWljKHBhZ2UpOwo+ICAgICAgICAg IHNldF9iaXQoYml0LCBiYXNlKTsKPiAgICAgICAgICBrdW5tYXBfYXRvbWljKGJhc2UpOwoKVGhp cyBzZXF1ZW5jZSBzaG91bGQgYmUgT0suICBnZXRfdXNlcl9wYWdlcygpIGNvbnRhaW5zIGEgZmx1 c2ggd2hpY2gKY2xlYXJzIHRoZSBjYWNoZSBhYm92ZSB0aGUgdXNlciB2aXJ0dWFsIGFkZHJlc3Ms IHNvIG9uIGttYXAsIHRoZSBwYWdlCmlzIGNvaGVyZW50IGF0IHRoZSBuZXcgYWxpYXMuICBPbiBw YXJpc2MgYXQgbGVhc3QsIGt1bm1hcCBlbWJvZGllcyBhCmZsdXNoX2RjYWNoZV9wYWdlKCkgd2hp Y2ggcHVzaGVzIGFueSBjaGFuZ2VzIGluIHRoZSBjYWNoZSBhYm92ZSB0aGUKa2VybmVsIHZpcnR1 YWwgYWRkcmVzcyBiYWNrIHRvIG1haW4gbWVtb3J5IGFuZCBtYWtlcyBpdCBjb2hlcmVudCBhZ2Fp bgpmb3IgdGhlIHVzZXIgYWxpYXMgdG8gcGljayBpdCB1cC4KCkphbWVzCgoKX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbGludXgtYXJtLWtlcm5lbCBtYWls aW5nIGxpc3QKbGludXgtYXJtLWtlcm5lbEBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0 cy5pbmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtYXJtLWtlcm5lbAo=