low overhead packet capturing on linux

[Thomas Fjellstrom <thomas@fjellstrom.ca>]

I'm working on a little tool to monitor and measure bandwidth use on a vm 
host, down to keeping track of all guest and host bandwidth, including, 
eventually per layer7 protocol use.

Right now I have a pretty simple setup, I setup an AF_PACKET socket, select on 
it, and read data as it comes in. Obviously, this has a fatal flaw. It takes up 
a rather large amount of cpu time just to capture the packets. On a GbE 
interface, it uses up easily 60-80% cpu (on a 2.6Ghz amd phenom II cpu core) 
just to capture the packets, trying to do anything fancy with them will likely 
cause the kernel to drop some packets.

So what I'm looking for is a very low overhead way to capture packets. I've 
come up with a few ideas, some of which I have no idea if they'd even work.

One idea that came to mind (that doesn't entirely look possible) is using 
splice or vmsplice to get me as little copying as is necessary from the net 
device to my own chunk of memory. Even better if it can be a circular queue of 
sorts. I'd probably use one thread to just sit on the socket and manage the 
packets, and a second thread to actually do the accounting on the incoming 
packets.

Anyone have any pointers or tips for me?

-- 
Thomas Fjellstrom
thomas@fjellstrom.ca