From mboxrd@z Thu Jan 1 00:00:00 1970 From: Rusty Russell Subject: Re: [RFCv2 00/12] Introduce host-side virtio queue and CAIF Virtio. Date: Fri, 11 Jan 2013 10:05:11 +1030 Message-ID: <87pq1c1uj4.fsf@rustcorp.com.au> References: <871ug75vp1.fsf@rustcorp.com.au> <1354718230-4486-1-git-send-email-sjur@brendeland.net> <20121206102750.GF10837@redhat.com> <877goc0wac.fsf@rustcorp.com.au> <87pq1f2rj0.fsf@rustcorp.com.au> <87wqvl1g9s.fsf@rustcorp.com.au> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: virtualization-bounces@lists.linux-foundation.org Errors-To: virtualization-bounces@lists.linux-foundation.org To: Sjur =?utf-8?Q?Br=C3=A6ndeland?= Cc: "Michael S. Tsirkin" , Linus Walleij , LKML , virtualization@lists.linux-foundation.org, Sjur =?utf-8?Q?Br=C3=A6ndeland?= List-Id: virtualization@lists.linuxfoundation.org U2p1ciBCcsOmbmRlbGFuZCA8c2p1cmJyZW5AZ21haWwuY29tPiB3cml0ZXM6Cj4gSGkgUnVzdHks Cj4KPiBPbiBUaHUsIEphbiAxMCwgMjAxMyBhdCAxMTozMCBBTSwgUnVzdHkgUnVzc2VsbCA8cnVz dHlAcnVzdGNvcnAuY29tLmF1PiB3cm90ZToKPiAuLi4KPj5JIG5vdyBoYXZlIHNvbWUgbGlnaHRs eS10ZXN0ZWQgY29kZSAodmlhIGEgdXNlcnNwYWNlIGhhcm5lc3MpLgo+Cj4gR3JlYXQgLSB0aGFu ayB5b3UgZm9yIGxvb2tpbmcgaW50byB0aGlzLiBJIHdpbGwgc3RhcnQgaW50ZWdyYXRpbmcgdGhp cwo+IHdpdGggbXkgcGF0Y2hlcwo+IHdoZW4geW91IHNlbmQgb3V0IGEgcHJvcGVyIHBhdGNoLgoK SGkgU2p1ciEKCiAgICAgICAgT0ssIHRoZSBJbnRlcm5ldCB3YXMgbm8gaGVscCBoZXJlLCBob3cg ZG8geW91IHByb25vdW5jZSBTanVyPwpJJ20gZ3Vlc3NpbmcgInNob29yIiByaHltaW5nIHdpdGgg dG91ciB1bnRpbCBJIGtub3cgYmV0dGVyLgoKPj4gZGlmZiAtLWdpdCBhL2RyaXZlcnMvdmlydGlv L0tjb25maWcgYi9kcml2ZXJzL3ZpcnRpby9LY29uZmlnCj4+IGluZGV4IDhkNWJkZGIuLmZkOTVk M2UgMTAwNjQ0Cj4+IC0tLSBhL2RyaXZlcnMvdmlydGlvL0tjb25maWcKPj4gKysrIGIvZHJpdmVy cy92aXJ0aW8vS2NvbmZpZwo+PiBAQCAtNSw2ICs1LDEyIEBAIGNvbmZpZyBWSVJUSU8KPj4gICAg ICAgICAgIGJ1cywgc3VjaCBhcyBDT05GSUdfVklSVElPX1BDSSwgQ09ORklHX1ZJUlRJT19NTUlP LCBDT05GSUdfTEdVRVNULAo+PiAgICAgICAgICAgQ09ORklHX1JQTVNHIG9yIENPTkZJR19TMzkw X0dVRVNULgo+Pgo+PiArY29uZmlnIFZIT1NUCj4+ICsgICAgICAgdHJpc3RhdGUKPgo+IEluY2x1 c2lvbiBvZiBkcml2ZXJzL3ZpcnRpbyBmcm9tIGRyaXZlcnMvTWFrZWZpbGUgZGVwZW5kcyBvbiBW SVJUSU8uCj4gU28gSSBndWVzcyBWSE9TVCBzaG91bGQgc2VsZWN0IFZJUlRJTyB0byBlbnN1cmUg dGhhdAo+IGRyaXZlcnMvdmlydGlvL3ZpcnRpb19ob3N0LmMKPiBpcyBwYXJ0IG9mIHRoZSBidWls ZC4KCk1heWJlIEkgc2hvdWxkIG1vdmUgZHJpdmVycy92aXJ0aW8vdmlydGlvX2hvc3QuYyB0bwpk cml2ZXJzL3Zob3N0L3ZyaW5naC5jOyBJJ2xsIGxvb2sgYXQgaXQuCgpJdCBtYWtlcyBzZW5zZSBm b3Igdmhvc3QvIHRvIGNvbnRhaW4gdGhlIGhvc3Qtc2lkZSBzdHVmZiwgc2luY2UgaXQKYWxyZWFk eSBleGlzdHMuCgo+PiArICAgICAgIGlmICgqbGFzdF9hdmFpbF9pZHggPT0gYXZhaWxfaWR4KQo+ PiArICAgICAgICAgICAgICAgcmV0dXJuIHZyaC0+dnJpbmcubnVtOwo+PiArCj4+ICsgICAgICAg LyogT25seSBnZXQgYXZhaWwgcmluZyBlbnRyaWVzIGFmdGVyIHRoZXkgaGF2ZSBiZWVuIGV4cG9z ZWQgYnkgZ3Vlc3QuICovCj4+ICsgICAgICAgc21wX3JtYigpOwo+Cj4gV2UgYXJlIGFjY2Vzc2lu ZyBtZW1vcnkgc2hhcmVkIHdpdGggYSByZW1vdGUgZGV2aWNlIChtb2RlbSksIHNvIHdlIHByb2Jh Ymx5Cj4gbmVlZCBtYW5kYXRvcnkgYmFycmllcnMgaGVyZSwgZS5nLiBzb21ldGhpbmcgbGlrZSB0 aGUgdmlydGlvX3JtYgo+IGRlZmluZWQgaW4gdmlydGlvX3JpbmcuYy4KCkZhaXIgZW5vdWdoLCB3 ZSBjYW4gcHV0IHRob3NlIGluIGEgaGVhZGVyLgoKPj4gKyAgICAgICAgICAgICAgIC8qIEFwcGVu ZCB0aGUgcGFnZXMgaW50byB0aGUgc2cuICovCj4+ICsgICAgICAgICAgICAgICBlcnIgPSBhZGRf dG9fc2coJnZzZywgKHZvaWQgKikobG9uZylhY2MtPmRlc2MuYWRkciwKPj4gKyAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICBhY2MtPmRlc2MubGVuLCBnZnApOwo+Cj4gSSB3b3VsZCBwcmVm ZXIgbm90IHRvIHNwbGl0IGludG8gcGFnZXMgYXQgdGhpcyBwb2ludCwgYnV0IHJhdGhlciBwcm92 aWRlIGFuCj4gaXRlcmF0b3Igb3IgdGhlIG9yaWdpbmFsIGxpc3QgZm91bmQgaW4gdGhlIGRlc2Ny aXB0b3IgdG8gdGhlIGNsaWVudC4KPgo+IEluIG91ciBjYXNlIHdlIHVzZSB2aXJ0aW8gcmluZ3Mg dG8gdGFsayB0byBhIExURS1tb2RlbSBvdmVyIHNoYXJlZCBtZW1vcnkuCj4gVGhlIElQIHRyYWZm aWMgaXMgcmVjZWl2ZWQgb3ZlciB0aGUgYWlyLCBpbnRlcmxlYXZlZCBhbmQgYXJyaXZlcyBpbgo+ IHRoZSB2aXJ0aW8gZHJpdmVyIGluCj4gbGFyZ2UgYnVyc3RzLiBTbyB2aXJ0aW8gZHJpdmVyIG9u IHRoZSBtb2RlbSByZWNlaXZlcyBtdWx0aXBsZSBkYXRhZ3JhbXMKPiBoZWxkIGluIGxhcmdlIGNv bnRpZ3VvdXMgYnVmZmVycy4gT3VyIGN1cnJlbnQgYXBwcm9hY2ggaXMgdG8gaGFuZGxlIGVhY2gK PiBidWZmZXIgYXMgYSBjaGFpbmVkIGRlc2NyaXB0b3IgbGlzdCwgd2hlcmUgZWFjaCBkYXRhZ3Jh bSBpcyBrZXB0IGluCj4gc2VwYXJhdGUgY2hhaW5lZCBkZXNjcmlwdG9ycy4gV2hlbiB0aGUgYnVm ZmVycyBhcmUgY29uc3VtZWQgb24gdGhlIGxpbnV4Cj4gaG9zdCwgdGhlIG1vZGVtIHdpbGwgcmVh ZCB0aGUgY2hhaW5lZCBkZXNjcmlwdG9ycyBmcm9tIHRoZSB1c2VkLXJpbmcgYW5kCj4gZnJlZSB0 aGUgZW50aXJlIGNvbnRpZ3VvdXMgYnVmZmVyIGluIG9uZSBvcGVyYXRpb24uCgpJbiBvdGhlciB3 b3JkcywgYm91bmRhcmllcyBtYXR0ZXI/CgpXaGlsZSB0aGUgc2ctaW4tcGxhY2UgaGFjayBpcyBj bG9zZSB0byBvcHRpbWFsIGZvciBUQ01fVkhPU1QsIG5laXRoZXIKbmV0IG5vdCB5b3UgY2FuIHVz ZSBpdCBkaXJlY3RseS4gIEknbGwgc3dpdGNoIHRvIGFuIGlvdmVjICh3aXRoIGEKc2ltaWxhciB1 c2UtY2FsbGVyLXN1cHBsaWVkLWlmLWl0LWZpdHMgdHJpY2spOyB0aGV5J3JlIHNtYWxsZXIgYW55 d2F5LgoKTW9yZSBjb2RlIGNvbWluZy4uLgoKVGhhbmtzLApSdXN0eS4KX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KVmlydHVhbGl6YXRpb24gbWFpbGluZyBs aXN0ClZpcnR1YWxpemF0aW9uQGxpc3RzLmxpbnV4LWZvdW5kYXRpb24ub3JnCmh0dHBzOi8vbGlz dHMubGludXhmb3VuZGF0aW9uLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3ZpcnR1YWxpemF0aW9u From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754450Ab3AJXjR (ORCPT ); Thu, 10 Jan 2013 18:39:17 -0500 Received: from ozlabs.org ([203.10.76.45]:50406 "EHLO ozlabs.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754032Ab3AJXjP convert rfc822-to-8bit (ORCPT ); Thu, 10 Jan 2013 18:39:15 -0500 From: Rusty Russell To: Sjur =?utf-8?Q?Br=C3=A6ndeland?= Cc: "Michael S. Tsirkin" , Linus Walleij , virtualization@lists.linux-foundation.org, LKML , Sjur =?utf-8?Q?Br=C3=A6ndeland?= , Ohad Ben-Cohen Subject: Re: [RFCv2 00/12] Introduce host-side virtio queue and CAIF Virtio. In-Reply-To: References: <871ug75vp1.fsf@rustcorp.com.au> <1354718230-4486-1-git-send-email-sjur@brendeland.net> <20121206102750.GF10837@redhat.com> <877goc0wac.fsf@rustcorp.com.au> <87pq1f2rj0.fsf@rustcorp.com.au> <87wqvl1g9s.fsf@rustcorp.com.au> User-Agent: Notmuch/0.14 (http://notmuchmail.org) Emacs/23.4.1 (i686-pc-linux-gnu) Date: Fri, 11 Jan 2013 10:05:11 +1030 Message-ID: <87pq1c1uj4.fsf@rustcorp.com.au> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8BIT Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Sjur Brændeland writes: > Hi Rusty, > > On Thu, Jan 10, 2013 at 11:30 AM, Rusty Russell wrote: > ... >>I now have some lightly-tested code (via a userspace harness). > > Great - thank you for looking into this. I will start integrating this > with my patches > when you send out a proper patch. Hi Sjur! OK, the Internet was no help here, how do you pronounce Sjur? I'm guessing "shoor" rhyming with tour until I know better. >> diff --git a/drivers/virtio/Kconfig b/drivers/virtio/Kconfig >> index 8d5bddb..fd95d3e 100644 >> --- a/drivers/virtio/Kconfig >> +++ b/drivers/virtio/Kconfig >> @@ -5,6 +5,12 @@ config VIRTIO >> bus, such as CONFIG_VIRTIO_PCI, CONFIG_VIRTIO_MMIO, CONFIG_LGUEST, >> CONFIG_RPMSG or CONFIG_S390_GUEST. >> >> +config VHOST >> + tristate > > Inclusion of drivers/virtio from drivers/Makefile depends on VIRTIO. > So I guess VHOST should select VIRTIO to ensure that > drivers/virtio/virtio_host.c > is part of the build. Maybe I should move drivers/virtio/virtio_host.c to drivers/vhost/vringh.c; I'll look at it. It makes sense for vhost/ to contain the host-side stuff, since it already exists. >> + if (*last_avail_idx == avail_idx) >> + return vrh->vring.num; >> + >> + /* Only get avail ring entries after they have been exposed by guest. */ >> + smp_rmb(); > > We are accessing memory shared with a remote device (modem), so we probably > need mandatory barriers here, e.g. something like the virtio_rmb > defined in virtio_ring.c. Fair enough, we can put those in a header. >> + /* Append the pages into the sg. */ >> + err = add_to_sg(&vsg, (void *)(long)acc->desc.addr, >> + acc->desc.len, gfp); > > I would prefer not to split into pages at this point, but rather provide an > iterator or the original list found in the descriptor to the client. > > In our case we use virtio rings to talk to a LTE-modem over shared memory. > The IP traffic is received over the air, interleaved and arrives in > the virtio driver in > large bursts. So virtio driver on the modem receives multiple datagrams > held in large contiguous buffers. Our current approach is to handle each > buffer as a chained descriptor list, where each datagram is kept in > separate chained descriptors. When the buffers are consumed on the linux > host, the modem will read the chained descriptors from the used-ring and > free the entire contiguous buffer in one operation. In other words, boundaries matter? While the sg-in-place hack is close to optimal for TCM_VHOST, neither net not you can use it directly. I'll switch to an iovec (with a similar use-caller-supplied-if-it-fits trick); they're smaller anyway. More code coming... Thanks, Rusty.