announcing rump kernel TCP/IP stack for DPDK

announcing rump kernel TCP/IP stack for DPDK

[Antti Kantee]

Hi,

I like the opportunities that a technology like DPDK enables, and I felt 
that the availability of an open source TCP/IP stack for DPDK could make 
things even more interesting.  I've been working on a concept called the 
anykernel, where the idea is that an OS kernel should be structured in a 
fashion which allows the driver components to be run independently of a 
monolithic kernel in so-called rump kernels.  Long story short, one of 
the "byproducts" is a run-anywhere standalone version of NetBSD kernel 
TCP/IP stack, so it was a natural progression to integrate that TCP/IP 
stack with the NIC driver layer provided by DPDK.

The published code completes "stage 1: make it work".  I want to thank 
another person who wished to remain anonymous for help with the 
implementation and testing.  Next up is "stage 2: make it fast".  I 
welcome everyone to contribute ideas / use cases / code / etc. towards 
that goal.

You can find the code for plugging DPDK into the rump kernel NIC layer 
along with instructions from:

	https://github.com/anttikantee/dpdk-rumptcpip

In case anyone is interested, I tested the setup with Void Linux running 
in VM using an emulated 82540 NIC (cf. my previous mail to this list) 
and the other party tested on Ubuntu using real hardware and a 82599 NIC.

cheers,
   antti

Re: announcing rump kernel TCP/IP stack for DPDK

[Stephen Hemminger]

On Thu, 23 May 2013 18:40:38 +0200
Antti Kantee <pooka-X3B1VOXEql0@public.gmane.org> wrote:

> Hi,
> 
> I like the opportunities that a technology like DPDK enables, and I felt 
> that the availability of an open source TCP/IP stack for DPDK could make 
> things even more interesting.  I've been working on a concept called the 
> anykernel, where the idea is that an OS kernel should be structured in a 
> fashion which allows the driver components to be run independently of a 
> monolithic kernel in so-called rump kernels.  Long story short, one of 
> the "byproducts" is a run-anywhere standalone version of NetBSD kernel 
> TCP/IP stack, so it was a natural progression to integrate that TCP/IP 
> stack with the NIC driver layer provided by DPDK.
> 
> The published code completes "stage 1: make it work".  I want to thank 
> another person who wished to remain anonymous for help with the 
> implementation and testing.  Next up is "stage 2: make it fast".  I 
> welcome everyone to contribute ideas / use cases / code / etc. towards 
> that goal.
> 
> You can find the code for plugging DPDK into the rump kernel NIC layer 
> along with instructions from:
> 
> 	https://github.com/anttikantee/dpdk-rumptcpip
> 
> In case anyone is interested, I tested the setup with Void Linux running 
> in VM using an emulated 82540 NIC (cf. my previous mail to this list) 
> and the other party tested on Ubuntu using real hardware and a 82599 NIC.
> 
> cheers,
>    antti

Already done several times before.
 http://savannah.nongnu.org/projects/lwip/
 https://code.google.com/p/tcp-ip-in-user-space/
 http://www.ohloh.net/p/tcp-ip-in-user-space
 http://www.ioremap.net/projects/unetstack
 http://www.joerch.org/tcpip/
 http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.167.2757

Re: announcing rump kernel TCP/IP stack for DPDK

[Robert Bays]

On May 23, 2013, at 10:34 AM, "Stephen Hemminger" <stephen@networkplumber.org> wrote:

> On Thu, 23 May 2013 18:40:38 +0200
> Antti Kantee <pooka-X3B1VOXEql0@public.gmane.org> wrote:
> 
>> Hi,
>> 
>> I like the opportunities that a technology like DPDK enables, and I felt 
>> that the availability of an open source TCP/IP stack for DPDK could  using real hardware and a 82599 NIC.
> 
> Already done several times before.
> http://savannah.nongnu.org/projects/lwip/
> https://code.google.com/p/tcp-ip-in-user-space/
> http://www.ohloh.net/p/tcp-ip-in-user-space
> http://www.ioremap.net/projects/unetstack
> http://www.joerch.org/tcpip/
> http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.167.2757
> 

Of course.  And many more times than that.  This is one more option for us and it is BSD licensed.

Ultimately, we will probably need a TCP stack in there.

Re: announcing rump kernel TCP/IP stack for DPDK

[Daniel Cegiełka]

2013/5/23 Robert Bays <robert-ZtmgI6mnKB3QT0dZR+AlfA@public.gmane.org>:

> Of course.  And many more times than that.  This is one more option for us and it is BSD licensed.
>
> Ultimately, we will probably need a TCP stack in there.

indeed, but what do you mean by the TCP stack? I hope not the BSD sockets :)
rx/tx bufs+simple API is all you need.

Daniel

Re: announcing rump kernel TCP/IP stack for DPDK

[Antti Kantee]

On 23.05.2013 20:39, Daniel Cegiełka wrote:
> 2013/5/23 Robert Bays <robert-ZtmgI6mnKB3QT0dZR+AlfA@public.gmane.org>:
>
>> Of course.  And many more times than that.  This is one more option for us and it is BSD licensed.
>>
>> Ultimately, we will probably need a TCP stack in there.
>
> indeed, but what do you mean by the TCP stack? I hope not the BSD sockets :)
> rx/tx bufs+simple API is all you need.

Luckily, ripping out sockets and replacing them with a "high-speed 
aware" API is easier than starting from a simple API and implementing 
IPv{4,6}, TCP, IPsec, ...

Re: announcing rump kernel TCP/IP stack for DPDK

[Robert Bays]

-----Original Message-----
From: Antti Kantee <pooka@iki.fi>
Date: Thursday, May 23, 2013 11:53 AM
To: "dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] announcing rump kernel TCP/IP stack for DPDK

>On 23.05.2013 20:39, Daniel Cegiełka wrote:
>> 2013/5/23 Robert Bays <robert@vyatta.com>:
>>
>>> Of course.  And many more times than that.  This is one more option
>>>for us and it is BSD licensed.
>>>
>>> Ultimately, we will probably need a TCP stack in there.
>>
>> indeed, but what do you mean by the TCP stack? I hope not the BSD
>>sockets :)
>> rx/tx bufs+simple API is all you need.
>
>Luckily, ripping out sockets and replacing them with a "high-speed
>aware" API is easier than starting from a simple API and implementing
>IPv{4,6}, TCP, IPsec, ...
>

Ugh.  We recently switched email to #$@ outlook which goes out if its way
to hide the recipient email address from me.  I meant to forward this
email to the Vyatta dev team which is also dev@.  Sorry about spamming the
list.

The reason for me forwarding this to my team was that it has been on my
list to research TCP reassembly in DPDK to enable things like application
layer classification.

Re: announcing rump kernel TCP/IP stack for DPDK

[Antti Kantee]

On 23.05.2013 19:34, Stephen Hemminger wrote:
> Already done several times before.
>   http://savannah.nongnu.org/projects/lwip/
>   https://code.google.com/p/tcp-ip-in-user-space/
>   http://www.ohloh.net/p/tcp-ip-in-user-space
>   http://www.ioremap.net/projects/unetstack
>   http://www.joerch.org/tcpip/
>   http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.167.2757

I'm aware of the paper you linked given that I wrote it.  I don't agree 
that it supports your claim "already done several times before".

Re: announcing rump kernel TCP/IP stack for DPDK

[Stephen Hemminger]

On Thu, 23 May 2013 20:46:02 +0200
Antti Kantee <pooka-X3B1VOXEql0@public.gmane.org> wrote:

> On 23.05.2013 19:34, Stephen Hemminger wrote:
> > Already done several times before.
> >   http://savannah.nongnu.org/projects/lwip/
> >   https://code.google.com/p/tcp-ip-in-user-space/
> >   http://www.ohloh.net/p/tcp-ip-in-user-space
> >   http://www.ioremap.net/projects/unetstack
> >   http://www.joerch.org/tcpip/
> >   http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.167.2757
> 
> I'm aware of the paper you linked given that I wrote it.  I don't agree 
> that it supports your claim "already done several times before".

I was more worried about multiple different TCP-IP stacks that seem
to be only used by small number of people and not maintained. Doing full
TCP/IP is hard, and there are lots of features inside.
It would be great to have one that is well supported and maintained.

Re: announcing rump kernel TCP/IP stack for DPDK

[Antti Kantee]

On 23.05.2013 21:20, Stephen Hemminger wrote:
>> I'm aware of the paper you linked given that I wrote it.  I don't agree
>> that it supports your claim "already done several times before".
>
> I was more worried about multiple different TCP-IP stacks that seem
> to be only used by small number of people and not maintained. Doing full
> TCP/IP is hard, and there are lots of features inside.
> It would be great to have one that is well supported and maintained.

I'm also well aware that doing TCP/IP right is very very hard.  It's not 
possible sit down with the spec for a weekend, write up some code, and 
hope it will work in the real world.  That's why I didn't implement 
TCP/IP.  The implementation I mentioned is the unmodified NetBSD kernel 
TCP/IP stack running on a very thin hypervisor layer.  It's not 
maintained by a few people, it's literally maintained by the entire 
NetBSD community.  Furthermore, it's not a set of patches available for 
the NetBSD kernel updated every now and then, it's literally [in] the 
kernel.  It's been working this way since 2008.  What I did now was just 
write 200 lines of code to plug the TCP/IP stack onto DPDK.

Hopefully that convinced you that it's not just some random one-shot 
kinda-works-except-in-reality bitrot attractor ;)

   - antti

Re: announcing rump kernel TCP/IP stack for DPDK

[Vincent JARDIN]

Hi Antti,

We had to develop a complete logic from scratch (aka fast path logic) to 
get a TCP/IP stack with good performances for 6WIND (including some 
Openflow switching). Unfortunately, as part of our DPDK opensource 
contribution, we do not plan to publish under open source our 6WINDGate 
stack (yet).

However, you are very welcomed to introduce a TCP/IP stack for the DPDK. 
What kind of support/benchmark did you run? Would you like to get it 
mainstream into the DPDK?

Best regards,
   Vincent



On 23/05/2013 21:38, Antti Kantee wrote:
> On 23.05.2013 21:20, Stephen Hemminger wrote:
>>> I'm aware of the paper you linked given that I wrote it.  I don't agree
>>> that it supports your claim "already done several times before".
>>
>> I was more worried about multiple different TCP-IP stacks that seem
>> to be only used by small number of people and not maintained. Doing full
>> TCP/IP is hard, and there are lots of features inside.
>> It would be great to have one that is well supported and maintained.
>
> I'm also well aware that doing TCP/IP right is very very hard.  It's not
> possible sit down with the spec for a weekend, write up some code, and
> hope it will work in the real world.  That's why I didn't implement
> TCP/IP.  The implementation I mentioned is the unmodified NetBSD kernel
> TCP/IP stack running on a very thin hypervisor layer.  It's not
> maintained by a few people, it's literally maintained by the entire
> NetBSD community.  Furthermore, it's not a set of patches available for
> the NetBSD kernel updated every now and then, it's literally [in] the
> kernel.  It's been working this way since 2008.  What I did now was just
> write 200 lines of code to plug the TCP/IP stack onto DPDK.
>
> Hopefully that convinced you that it's not just some random one-shot
> kinda-works-except-in-reality bitrot attractor ;)
>
>    - antti

Re: announcing rump kernel TCP/IP stack for DPDK

[Antti Kantee]

Vincent,

On 23.05.2013 21:52, Vincent JARDIN wrote:
> We had to develop a complete logic from scratch (aka fast path logic) to
> get a TCP/IP stack with good performances for 6WIND (including some
> Openflow switching). Unfortunately, as part of our DPDK opensource
> contribution, we do not plan to publish under open source our 6WINDGate
> stack (yet).

I'm under no illusion that making things really fast will be trivial 
work.  If it were, the world would be full of fast TCP/IP stacks ;)

> However, you are very welcomed to introduce a TCP/IP stack for the DPDK.
> What kind of support/benchmark did you run? Would you like to get it
> mainstream into the DPDK?

Our test was simple: DHCP and port 80 "GET /".  Since the TCP/IP stack 
and rump kernel themselves are solid, it's been my experience that once 
the above works, everything works even full blast.

I don't have hardware, but the other party I mentioned in my original 
mail wanted to benchmark, so hopefully we'll have some netperf numbers 
soon.  Based on prior experience, I expect some Gbps throughput on a 10G 
without optimizing, but we'll see.

I think currently integration makes sense only as some sort of a 
"reachover", i.e. source is not imported into the DPDK tree, but rather 
fetched on demand.  I'm happy with the current state, but if someone 
wants to work on a reachover solution, I'll be happy to assist, though 
not drive the effort.  Eventually integration may become more pertinent 
as different optimizations demand radically different code, but let's 
think about that if/when we get there.

br,
   antti

Re: announcing rump kernel TCP/IP stack for DPDK

[Thomas Monjalon]

23/05/2013 23:41, Antti Kantee :
> On 23.05.2013 21:52, Vincent JARDIN wrote:
> > Would you like to get it mainstream into the DPDK?
> 
> I think currently integration makes sense only as some sort of a
> "reachover", i.e. source is not imported into the DPDK tree, but rather
> fetched on demand.  I'm happy with the current state, but if someone
> wants to work on a reachover solution, I'll be happy to assist, though
> not drive the effort.  Eventually integration may become more pertinent
> as different optimizations demand radically different code, but let's
> think about that if/when we get there.

dpdk.org may host other projects around DPDK.
So don't hesitate to request hosting when you'll consider it's the right time.

-- 
Thomas