From: Bart Van Assche <Bart.VanAssche@wdc.com>
To: "lsf-pc@lists.linux-foundation.org"
<lsf-pc@lists.linux-foundation.org>,
"willy@infradead.org" <willy@infradead.org>
Cc: "linux-mm@kvack.org" <linux-mm@kvack.org>,
"linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
"linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>
Subject: Re: [LSF/MM TOPIC] A high-performance userspace block driver
Date: Wed, 17 Jan 2018 00:41:46 +0000 [thread overview]
Message-ID: <1516149705.2844.79.camel@wdc.com> (raw)
In-Reply-To: <20180116145240.GD30073@bombadil.infradead.org>
T24gVHVlLCAyMDE4LTAxLTE2IGF0IDA2OjUyIC0wODAwLCBNYXR0aGV3IFdpbGNveCB3cm90ZToN
Cj4gSSBzZWUgdGhlIGltcHJvdmVtZW50cyB0aGF0IEZhY2Vib29rIGhhdmUgYmVlbiBtYWtpbmcg
dG8gdGhlIG5iZCBkcml2ZXIsDQo+IGFuZCBJIHRoaW5rIHRoYXQncyBhIHdvbmRlcmZ1bCB0aGlu
Zy4gIE1heWJlIHRoZSBvdXRjb21lIG9mIHRoaXMgdG9waWMNCj4gaXMgc2ltcGx5OiAiU2h1dCB1
cCwgTWF0dGhldywgdGhpcyBpcyBnb29kIGVub3VnaCIuDQo+IA0KPiBJdCdzIGNsZWFyIHRoYXQg
dGhlcmUncyBhbiBhcHBldGl0ZSBmb3IgdXNlcnNwYWNlIGJsb2NrIGRldmljZXM7IG5vdCBmb3IN
Cj4gc3dhcCBkZXZpY2VzIG9yIHRoZSByb290IGRldmljZSwgYnV0IGZvciBhY2Nlc3NpbmcgZGF0
YSB0aGF0J3Mgc3RvcmVkDQo+IGluIHRoYXQgc2lsbyBvdmVyIHRoZXJlLCBhbmQgSSByZWFsbHkg
ZG9uJ3Qgd2FudCB0byBicmluZyB0aGF0IGVudGlyZQ0KPiBtZXNzIG9mIENPUkJBIC8gR28gLyBS
dXN0IC8gd2hhdGV2ZXIgaW50byB0aGUga2VybmVsIHRvIGdldCB0byBpdCwNCj4gYnV0IGl0IHdv
dWxkIGJlIHJlYWxseSBoYW5keSB0byBwcmVzZW50IGl0IGFzIGEgYmxvY2sgZGV2aWNlLg0KPiAN
Cj4gSSd2ZSBsb29rZWQgYXQgYSBmZXcgYmxvY2stZHJpdmVyLWluLXVzZXJzcGFjZSBwcm9qZWN0
cyB0aGF0IGV4aXN0LCBhbmQNCj4gdGhleSBhbGwgc2VlbSBwcmV0dHkgYmFkLiAgRm9yIGV4YW1w
bGUsIG9uZSBBUEkgbWFwcyBhIGZldyBnaWdhYnl0ZXMgb2YNCj4gYWRkcmVzcyBzcGFjZSBhbmQg
cGxheXMgZ2FtZXMgd2l0aCB2bV9pbnNlcnRfcGFnZSgpIHRvIHB1dCBwYWdlIGNhY2hlDQo+IHBh
Z2VzIGludG8gdGhlIGFkZHJlc3Mgc3BhY2Ugb2YgdGhlIGNsaWVudCBwcm9jZXNzLiAgT2YgY291
cnNlLCB0aGUgVExCDQo+IGZsdXNoIG92ZXJoZWFkIG9mIHRoYXQgc29sdXRpb24gaXMgY3JpbWlu
YWwuDQo+IA0KPiBJJ3ZlIGxvb2tlZCBhdCBwaXBlcywgYW5kIHRoZXkncmUgbm90IGFuIGF3ZnVs
IHNvbHV0aW9uLiAgV2UndmUgYWxtb3N0DQo+IGdvdCBlbm91Z2ggc3lzY2FsbHMgdG8gdHJlYXQg
b3RoZXIgb2JqZWN0cyBhcyBwaXBlcy4gIFRoZSBwcm9ibGVtIGlzDQo+IHRoYXQgdGhleSdyZSBu
b3Qgc2Vla2FibGUuICBTbyBlc3NlbnRpYWxseSB5b3UncmUgbG9va2luZyBhdCBoYXZpbmcgb25l
DQo+IHBpcGUgcGVyIG91dHN0YW5kaW5nIGNvbW1hbmQuICBJZiB5dSB3YW50IHRvIG1ha2UgZ29v
ZCB1c2Ugb2YgYSBtb2Rlcm4NCj4gTkFORCBkZXZpY2UsIHlvdSB3YW50IGEgZmV3IGh1bmRyZWQg
b3V0c3RhbmRpbmcgY29tbWFuZHMsIGFuZCB0aGF0J3MgYQ0KPiBiaXQgb2YgYSBzaG9kZHkgaW50
ZXJmYWNlLg0KPiANCj4gUmlnaHQgbm93LCBJJ20gbGVhbmluZyB0b3dhcmRzIGNvbWJpbmluZyB0
aGVzZSB0d28gYXBwcm9hY2hlczsgYWRkaW5nDQo+IGEgVk1fTk9UTEIgZmxhZyBzbyB0aGUgbW1h
cGVkIGJpdHMgb2YgdGhlIHBhZ2UgY2FjaGUgbmV2ZXIgbWFrZSBpdCBpbnRvDQo+IHRoZSBwcm9j
ZXNzJ3MgYWRkcmVzcyBzcGFjZSwgc28gdGhlIFRMQiBzaG9vdGRvd24gY2FuIGJlIHNhZmVseSBz
a2lwcGVkLg0KPiBUaGVuIGNoZWNrIGl0IGluIGZvbGxvd19wYWdlX21hc2soKSBhbmQgcmV0dXJu
IHRoZSBhcHByb3ByaWF0ZSBzdHJ1Y3QNCj4gcGFnZS4gIEFzIGxvbmcgYXMgdGhlIHVzZXJzcGFj
ZSBwcm9jZXNzIGRvZXMgZXZlcnl0aGluZyB1c2luZyBPX0RJUkVDVCwNCj4gSSB0aGluayB0aGlz
IHdpbGwgd29yay4NCj4gDQo+IEl0J3MgZWl0aGVyIHRoYXQgb3IgbWFrZSBwaXBlcyBzZWVrYWJs
ZSAuLi4NCg0KSG93IGFib3V0IHVzaW5nIHRoZSBSRE1BIEFQSSBhbmQgdGhlIHJkbWFfcnhlIGRy
aXZlciBvdmVyIGxvb3BiYWNrPyBUaGUgUkRNQQ0KQVBJIHN1cHBvcnRzIHplcm8tY29weSBjb21t
dW5pY2F0aW9uIHdoaWNoIGlzIHNvbWV0aGluZyB0aGUgQlNEIHNvY2tldCBBUEkNCmRvZXMgbm90
IHN1cHBvcnQuIFRoZSBSRE1BIEFQSSBhbHNvIHN1cHBvcnRzIGJ5dGUtbGV2ZWwgZ3JhbnVsYXJp
dHkgYW5kIHRoZQ0KaG90IHBhdGggKGliX3Bvc3Rfc2VuZCgpLCBpYl9wb3N0X3JlY3YoKSwgaWJf
cG9sbF9jcSgpKSBkb2VzIG5vdCByZXF1aXJlIGFueQ0Kc3lzdGVtIGNhbGxzIGZvciBQQ0llIFJE
TUEgYWRhcHRlcnMuIFRoZSByZG1hX3J4ZSBkcml2ZXIgaG93ZXZlciB1c2VzIGEgc3lzdGVtDQpj
YWxsIHRvIHRyaWdnZXIgdGhlIHNlbmQgZG9vcmJlbGwuDQoNCkJhcnQuDQo=
WARNING: multiple messages have this Message-ID (diff)
From: Bart Van Assche <Bart.VanAssche@wdc.com>
To: "lsf-pc@lists.linux-foundation.org"
<lsf-pc@lists.linux-foundation.org>,
"willy@infradead.org" <willy@infradead.org>
Cc: "linux-mm@kvack.org" <linux-mm@kvack.org>,
"linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
"linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>
Subject: Re: [LSF/MM TOPIC] A high-performance userspace block driver
Date: Wed, 17 Jan 2018 00:41:46 +0000 [thread overview]
Message-ID: <1516149705.2844.79.camel@wdc.com> (raw)
In-Reply-To: <20180116145240.GD30073@bombadil.infradead.org>
On Tue, 2018-01-16 at 06:52 -0800, Matthew Wilcox wrote:
> I see the improvements that Facebook have been making to the nbd driver,
> and I think that's a wonderful thing. Maybe the outcome of this topic
> is simply: "Shut up, Matthew, this is good enough".
>
> It's clear that there's an appetite for userspace block devices; not for
> swap devices or the root device, but for accessing data that's stored
> in that silo over there, and I really don't want to bring that entire
> mess of CORBA / Go / Rust / whatever into the kernel to get to it,
> but it would be really handy to present it as a block device.
>
> I've looked at a few block-driver-in-userspace projects that exist, and
> they all seem pretty bad. For example, one API maps a few gigabytes of
> address space and plays games with vm_insert_page() to put page cache
> pages into the address space of the client process. Of course, the TLB
> flush overhead of that solution is criminal.
>
> I've looked at pipes, and they're not an awful solution. We've almost
> got enough syscalls to treat other objects as pipes. The problem is
> that they're not seekable. So essentially you're looking at having one
> pipe per outstanding command. If yu want to make good use of a modern
> NAND device, you want a few hundred outstanding commands, and that's a
> bit of a shoddy interface.
>
> Right now, I'm leaning towards combining these two approaches; adding
> a VM_NOTLB flag so the mmaped bits of the page cache never make it into
> the process's address space, so the TLB shootdown can be safely skipped.
> Then check it in follow_page_mask() and return the appropriate struct
> page. As long as the userspace process does everything using O_DIRECT,
> I think this will work.
>
> It's either that or make pipes seekable ...
How about using the RDMA API and the rdma_rxe driver over loopback? The RDMA
API supports zero-copy communication which is something the BSD socket API
does not support. The RDMA API also supports byte-level granularity and the
hot path (ib_post_send(), ib_post_recv(), ib_poll_cq()) does not require any
system calls for PCIe RDMA adapters. The rdma_rxe driver however uses a system
call to trigger the send doorbell.
Bart.
next prev parent reply other threads:[~2018-01-17 0:41 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-01-16 14:52 [LSF/MM TOPIC] A high-performance userspace block driver Matthew Wilcox
2018-01-16 14:52 ` Matthew Wilcox
2018-01-16 23:04 ` Viacheslav Dubeyko
2018-01-16 23:04 ` Viacheslav Dubeyko
2018-01-16 23:23 ` [Lsf-pc] " Theodore Ts'o
2018-01-16 23:23 ` Theodore Ts'o
2018-01-16 23:28 ` [Lsf-pc] " James Bottomley
2018-01-16 23:28 ` James Bottomley
2018-01-16 23:28 ` James Bottomley
2018-01-16 23:57 ` Bart Van Assche
2018-01-16 23:57 ` Bart Van Assche
2018-01-17 0:41 ` Bart Van Assche [this message]
2018-01-17 0:41 ` Bart Van Assche
2018-01-17 2:49 ` Ming Lei
2018-01-17 2:49 ` Ming Lei
2018-01-17 21:21 ` Matthew Wilcox
2018-01-17 21:21 ` Matthew Wilcox
2018-01-22 12:02 ` Mike Rapoport
2018-01-22 12:02 ` Mike Rapoport
2018-01-22 12:18 ` [Lsf-pc] " Ming Lei
2018-01-22 12:18 ` Ming Lei
2018-01-18 5:27 ` [Lsf-pc] " Figo.zhang
2018-01-18 5:27 ` Figo.zhang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1516149705.2844.79.camel@wdc.com \
--to=bart.vanassche@wdc.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=lsf-pc@lists.linux-foundation.org \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.