Re: Appropriate DPDK data structures for TCP sockets

[Matthew Hall <mhall-Hv3ogNYU3JfZZajBQzqCxQ@public.gmane.org>]

On Mon, Feb 23, 2015 at 08:48:57AM -0600, Matt Laswell wrote:
> Apologies in advance for likely being a bit long-winded.

Long winded is great, helps me get context.

> First, you really need to take cache performance into account when you're
> choosing a data structure.  Something like a balanced tree can seem awfully
> appealing at first blush

Agreed. I did some amount of DPDK stuff before but without TCP. This is why I 
was figuring a packet-hash is better than a tree.

> Second, rather than synchronizing (perhaps with locks, perhaps with
> lockless data structures), it's often beneficial to create multiple
> threads, each of which holds a fraction of your connection tracking data.

Yes, I REALLY REALLY REALLY wanted to do RSS. But the virtio-net and other 
VM's don't support RSS, unlike the classic PCIe NIC's. In order to get the 
community to use my app I have to give them a "batteries included" 
environment, where the system can still work even with no RSS.

> Third, it's very worthwhile to have a cache for the most recently accessed
> connection.  First, because network traffic is bursty, and you'll
> frequently see multiple packets from the same connection in succession.
> Second, because it can make life easier for your application code.  If you
> have multiple places that need to access connection data, you don't have to
> worry so much about the cost of repeated searches.  Again, this may or may
> not matter for your particular application.  But for ones I've worked on,
> it's been a win.

Yes, this sounds like a really good idea. One advantage in my product, I am 
only doing TCP Syslog, so I don't have an arbitrary zillion connections like 
FW or IPS would want. I could cap it at something like 8192 or 16384 and be 
good enough for some time until a better solution is worked out.

I could make some capped array or linked list of the X most recent ones for 
cheap access. It's just socket pointers so it doesn't hardly cost anything to 
copy a couple pointers into a cache and quickly invalidate when the connection 
closes.

> Anyway, as predicted, this post has gone far too long for a Monday
> morning.  Regardless, I hope you found it useful.

This was great. Thank you!

Matthew.