RFC: kcrypto - (yet another) user space interface

RFC: kcrypto - (yet another) user space interface

[Phil Sutter]

Hello everyone,

my employer wants to have a lightweight, zero-copy user space interface
to the Crypto-API and I *think* I'm on the right way to realising this.

What I've got so far is just a proof-of-concept, tested only with
cbc(aes), merely as generic as I'd like it to be, but with zero-copy of
the actual data buffers (which reside in user space) and asynchronous
operation.

You can check it out via git://nwl.cc/~n0-1/kcrypto.git or simply have a
look at http://nwl.cc/cgi-bin/git/gitweb.cgi?p=kcrypto.git;a=summary. If
you do so, you may in return flame me as much as you like to. :)

I know that it's far from being ready for use as the sole crypto API
user space interface, but I'm convinced as well that it will never be
unless someone with Crypto-API skills points out at least the very basic
design flaws I've already created.

As this is not yet being used in production (of course), I'm totally
open to changes. So taking this as start of a collaborative project
would be perfect for me, too.

Greetings (and sorry for yet another interface approach),
Phil

Re: RFC: kcrypto - (yet another) user space interface

[Sebastian Andrzej Siewior]

* Phil Sutter | 2010-06-10 20:22:29 [+0200]:

>Hello everyone,
Hi Phil,

please take look at [0] and [1]. From README I can tell that those two
posts are different from you have so far.
You might want to take a look at AF_PACKET interface. It does zero copy
via a ring buffer interface of pre-mmaped user memory. So no
get_user_pages() then :)

I think that is the way to go.

[0] http://article.gmane.org/gmane.linux.kernel.cryptoapi/2656
[1] http://article.gmane.org/gmane.linux.kernel.cryptoapi/2658

>Phil

Sebastian

Re: RFC: kcrypto - (yet another) user space interface

[Nikos Mavrogiannopoulos]

Sebastian Andrzej Siewior wrote:
> * Phil Sutter | 2010-06-10 20:22:29 [+0200]:
> 
>> Hello everyone,
> Hi Phil,
> 
> please take look at [0] and [1]. From README I can tell that those two
> posts are different from you have so far.
> You might want to take a look at AF_PACKET interface. It does zero copy
> via a ring buffer interface of pre-mmaped user memory. So no
> get_user_pages() then :)
> 
> I think that is the way to go.

The problem with right or wrong is that they are only known afterwards.
For me the right way to go is _to go_. I can see discussions in this
least, years ago on talks about the "perfect" userspace crypto api and
rejections implementations because they are not perfect enough. I don't
believe there is such thing as a perfect crypto api. Other operating
systems have a userspace crypto API (maybe not perfect) but linux
hasn't. I don't think this is the way to go.

regards,
Nikos

Re: RFC: kcrypto - (yet another) user space interface

[Sebastian Andrzej Siewior]

* Nikos Mavrogiannopoulos | 2010-06-11 09:47:15 [+0200]:

>Sebastian Andrzej Siewior wrote:
>> * Phil Sutter | 2010-06-10 20:22:29 [+0200]:
>
>The problem with right or wrong is that they are only known afterwards.
>For me the right way to go is _to go_. I can see discussions in this
>least, years ago on talks about the "perfect" userspace crypto api and
>rejections implementations because they are not perfect enough. I don't
>believe there is such thing as a perfect crypto api. Other operating
>systems have a userspace crypto API (maybe not perfect) but linux
>hasn't. I don't think this is the way to go.

Phil asked me for my opinion and he got it. The fundumention problems
from what I've seen was the interface:
- kernel structs which are exposed to userland which limit the
  parameters. For instance the iv was limited to 16 bytes while we have
  allready algos with a much longer iv.
- the interface was using write()/poll()/read() and get_user_pages(). I
  pointed out Herbert's opinion about this and the alternative. So this
  _was_ allready discsussed.

>regards,
>Nikos

Sebastian

Re: RFC: kcrypto - (yet another) user space interface

[Phil Sutter]

Hey,

Seems like I'm stabbing into open wounds. :) First of all, thanks a lot
for your comments.

On Fri, Jun 11, 2010 at 11:08:56AM +0200, Sebastian Andrzej Siewior wrote:
> * Nikos Mavrogiannopoulos | 2010-06-11 09:47:15 [+0200]:
> 
> >Sebastian Andrzej Siewior wrote:
> >> * Phil Sutter | 2010-06-10 20:22:29 [+0200]:
> >
> >The problem with right or wrong is that they are only known afterwards.
> >For me the right way to go is _to go_. I can see discussions in this
> >least, years ago on talks about the "perfect" userspace crypto api and
> >rejections implementations because they are not perfect enough. I don't
> >believe there is such thing as a perfect crypto api. Other operating
> >systems have a userspace crypto API (maybe not perfect) but linux
> >hasn't. I don't think this is the way to go.
> 
> Phil asked me for my opinion and he got it. The fundumention problems
> from what I've seen was the interface:
> - kernel structs which are exposed to userland which limit the
>   parameters. For instance the iv was limited to 16 bytes while we have
>   allready algos with a much longer iv.
> - the interface was using write()/poll()/read() and get_user_pages(). I
>   pointed out Herbert's opinion about this and the alternative. So this
>   _was_ allready discsussed.

For me, this project is a rather pragmatical one - this just needs to
get done, and it has to be just perfect enough so my employer finds it
usable. Nice to have if I happen to create the perfect CryptoAPI user
space interface ever (yeah, right ...) but this is unlikely to happen.
For me it's enough to first get the concept right and next make it
stable and functional. After that I'm sure we all can tell better if
it's worth pushing it towards the kernel or leave it as (yet another)
niche product.

Greetings, Phil

Re: RFC: kcrypto - (yet another) user space interface

[Phil Sutter]

Hey Bigeasy,

On Thu, Jun 10, 2010 at 11:14:33PM +0200, Sebastian Andrzej Siewior wrote:
> please take look at [0] and [1]. From README I can tell that those two
> posts are different from you have so far.

Hmm. Indeed, using something like AF_CRYPTO didn't come to my mind so
far. Though I'm not sure if this is good or bad - what's the big
advantage in introducing an address family for something which doesn't
even know addressing as such? No offense here, but all I have is a bunch
of bytes which should be transformed by the kernel. Using socket(),
connect() and sendmsg() for just that purpose seems a bit too fancy to
me.

> You might want to take a look at AF_PACKET interface. It does zero copy
> via a ring buffer interface of pre-mmaped user memory. So no
> get_user_pages() then :)

Yes, I've already thought about using just mmap() for the buffer
exchange. But what I don't like about it is that the shared buffer is
allocated from within the kernel, leading to two preconditions:

1) Unless the user anyway has to fill a locally allocated buffer with
the data to transform, at least a single copy is needed to get the data
into the kernel buffer. Although get_user_pages() is quite ugly to use,
it's flexible enough to take any buffer directly from user space to
operate on. (Page alignment constraints, especially with hardware crypto
engines, should be another interesting topic in this context.)

2) Space constraints. I can take a hundred 1.5k buffers along with a
single, 64M one. Despite that my PoC actually doesn't work with buffers
above 64k, using only an in-kernel buffer would make things quite a bit
more complicated.

> 
> I think that is the way to go.
> 
> [0] http://article.gmane.org/gmane.linux.kernel.cryptoapi/2656
> [1] http://article.gmane.org/gmane.linux.kernel.cryptoapi/2658

Reading a bit further from there, splice() is mentioned as another way
of exchanging the data buffers. But despite that it's doing about what
I've implemented (i.e., using get_user_pages() to fetch the userspace
data), there seems to be now sane way back, at least not according to
the comments in fs/splice.c.

This is actually a limitation of my implementation: all data
transformation is done in situ. Fine for stream ciphers, acceptable for
block ciphers, but probably FUBAR for hashes, I guess.

Greetings, Phil

Re: RFC: kcrypto - (yet another) user space interface

[Phil Sutter]

Hey Bigeasy,

On Thu, Jun 10, 2010 at 11:14:33PM +0200, Sebastian Andrzej Siewior wrote:
> please take look at [0] and [1]. From README I can tell that those two
> posts are different from you have so far.

Hmm. Indeed, using something like AF_CRYPTO didn't come to my mind so
far. Though I'm not sure if this is good or bad - what's the big
advantage in introducing an address family for something which doesn't
even know addressing as such? No offense here, but all I have is a bunch
of bytes which should be transformed by the kernel. Using socket(),
connect() and sendmsg() for just that purpose seems a bit too fancy to
me.

> You might want to take a look at AF_PACKET interface. It does zero copy
> via a ring buffer interface of pre-mmaped user memory. So no
> get_user_pages() then :)

Yes, I've already thought about using just mmap() for the buffer
exchange. But what I don't like about it is that the shared buffer is
allocated from within the kernel, leading to two preconditions:

1) Unless the user anyway has to fill a locally allocated buffer with
the data to transform, at least a single copy is needed to get the data
into the kernel buffer. Although get_user_pages() is quite ugly to use,
it's flexible enough to take any buffer directly from user space to
operate on. (Page alignment constraints, especially with hardware crypto
engines, should be another interesting topic in this context.)

2) Space constraints. I can take a hundred 1.5k buffers along with a
single, 64M one. Despite that my PoC actually doesn't work with buffers
above 64k, using only an in-kernel buffer would make things quite a bit
more complicated.

> 
> I think that is the way to go.
> 
> [0] http://article.gmane.org/gmane.linux.kernel.cryptoapi/2656
> [1] http://article.gmane.org/gmane.linux.kernel.cryptoapi/2658

Reading a bit further from there, splice() is mentioned as another way
of exchanging the data buffers. But despite that it's doing about what
I've implemented (i.e., using get_user_pages() to fetch the userspace
data), there seems to be now sane way back, at least not according to
the comments in fs/splice.c.

This is actually a limitation of my implementation: all data
transformation is done in situ. Fine for stream ciphers, acceptable for
block ciphers, but probably FUBAR for hashes, I guess.

Greetings, Phil

Re: RFC: kcrypto - (yet another) user space interface

[Uri Simchoni]

Hi Phil,

Note on  implementation: I see that you're using the synchronous (blkcipher) interface and not the async (ablkcipher) one. I'm not 100% sure about this, but it appears to me that you can't utilize async HW drivers this way (there's adapter code from async client to sync driver but not the other way around).

With regards to interface, I don't know enough to comment about the proposed alternatives, but the use of write() and read() appears problematic - it looks inconsistent with the semantics of write() in that the user must not modify the buffer after write() - it looks like aio semantics but without the aio interface.

With regards to mmap preventing zero copy - In many cases you can assemble the things you want to crypt in the mmap buffer upfront. Still in other cases you need to copy the data at least once anyway, so you might as well start off by copying to the mmap area. Taking TLS as an example - you have to build a TLS record before you can encrypt/hash it. The header is poorly aligned and there's the padding at the end - there seems to be no escaping copying it once inside the TLS library. You may place your record buffers in the mmap area. So I guess what I'm saying is that the mmap approach doesn't break interface semantics, and in most important cases can be used to provide a solution that's as good as the zero-copy one.

Just my 2c,
Uri

On 6/10/2010 9:22 PM, Phil Sutter wrote:
> Hello everyone,
> 
> my employer wants to have a lightweight, zero-copy user space interface
> to the Crypto-API and I *think* I'm on the right way to realising this.
> 
> What I've got so far is just a proof-of-concept, tested only with
> cbc(aes), merely as generic as I'd like it to be, but with zero-copy of
> the actual data buffers (which reside in user space) and asynchronous
> operation.
> 
> You can check it out via git://nwl.cc/~n0-1/kcrypto.git or simply have a
> look at http://nwl.cc/cgi-bin/git/gitweb.cgi?p=kcrypto.git;a=summary. If
> you do so, you may in return flame me as much as you like to. :)
> 
> I know that it's far from being ready for use as the sole crypto API
> user space interface, but I'm convinced as well that it will never be
> unless someone with Crypto-API skills points out at least the very basic
> design flaws I've already created.
> 
> As this is not yet being used in production (of course), I'm totally
> open to changes. So taking this as start of a collaborative project
> would be perfect for me, too.
> 
> Greetings (and sorry for yet another interface approach),
> Phil
> --
> To unsubscribe from this list: send the line "unsubscribe linux-crypto" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html