Kernel hooks just to get rid of copy_[to/from]_user() and syscall overhead?

Kernel hooks just to get rid of copy_[to/from]_user() and syscall overhead?

[Mihnea Balta]

Hi,

I have to implement a system which grabs udp packets off a gigabit connection, 
take some basic action based on what they contain, repack their data with a 
custom protocol header and send them through a gigabit ethernet interface on 
broadcast.

I know how to do this in userspace, but I need to know if doing everyting in 
the kernel would show a considerable speed improvement due to removing 
syscall and memory copy overhead. The system will be quite stressed, having 
to deal with around 15-20000 packets/second.

I didn't want to start this e-mail with an excuse, so I delayed it until here 
:). I appologise if this isn't the right place to ask, it seemed that way to 
me. I wasn't able to find sufficient and coherent information about this 
issue on the internet or on this mailing list's archives, so I decided to ask 
you people directly.

Thanks for your time,
Mihnea Balta

Re: Kernel hooks just to get rid of copy_[to/from]_user() and syscall overhead?

[Richard B. Johnson]

On Fri, 10 Jan 2003, Mihnea Balta wrote:

> Hi,
> 
> I have to implement a system which grabs udp packets off a gigabit connection, 
> take some basic action based on what they contain, repack their data with a 
> custom protocol header and send them through a gigabit ethernet interface on 
> broadcast.
> 
> I know how to do this in userspace, but I need to know if doing everyting in 
> the kernel would show a considerable speed improvement due to removing 
> syscall and memory copy overhead. The system will be quite stressed, having 
> to deal with around 15-20000 packets/second.
> 
> I didn't want to start this e-mail with an excuse, so I delayed it until here 
> :). I appologise if this isn't the right place to ask, it seemed that way to 
> me. I wasn't able to find sufficient and coherent information about this 
> issue on the internet or on this mailing list's archives, so I decided to ask 
> you people directly.
> 
> Thanks for your time,
> Mihnea Balta
> 

I think you should do everything in kernel space, with a user-mode
interface for non-realtime control, i.e., what characteristics
of UDP packets are being "filtered". You just make a module that
contains what you need, with an ioctl() hook to control it.

That way, data from your gigabit interface(s) never has to get to
"user-space" at all.


Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).
Why is the government concerned about the lunatic fringe? Think about it.

Re: Kernel hooks just to get rid of copy_[to/from]_user() and syscall overhead?

[Gianni Tedesco]

[-- Attachment #1: Type: text/plain, Size: 1208 bytes --]

On Fri, 2003-01-10 at 14:45, Mihnea Balta wrote:
> Hi,
> 
> I have to implement a system which grabs udp packets off a gigabit connection, 
> take some basic action based on what they contain, repack their data with a 
> custom protocol header and send them through a gigabit ethernet interface on 
> broadcast.
> 
> I know how to do this in userspace, but I need to know if doing everyting in 
> the kernel would show a considerable speed improvement due to removing 
> syscall and memory copy overhead. The system will be quite stressed, having 
> to deal with around 15-20000 packets/second.

mmap() packet socket interface eliminates the need for system calls when
traffic is coming in at a high rate.  The kernel -> user copy is also
eliminated, but its just replaced with a kernel -> kernel copy :P

You could perhaps also use linux socket filters to minimize the number
of packets you need to evaluate...

Check out this sample code: http://www.scaramanga.co.uk/code-fu/lincap.c

HTH

-- 
// Gianni Tedesco (gianni at scaramanga dot co dot uk)
lynx --source www.scaramanga.co.uk/gianni-at-ecsc.asc | gpg --import
8646BE7D: 6D9F 2287 870E A2C9 8F60 3A3C 91B5 7669 8646 BE7D

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Kernel hooks just to get rid of copy_[to/from]_user() and syscall overhead?

[Mihnea Balta]

On Friday 10 January 2003 16:45, Mihnea Balta wrote:
> Hi,
>
> I have to implement a system which grabs udp packets off a gigabit
> connection, take some basic action based on what they contain, repack their
> data with a custom protocol header and send them through a gigabit ethernet
> interface on broadcast.

Thank you to everybody who answered, I'll try the mmap()ed descriptor 
userspace solution, as it seems good enough.

Re: Kernel hooks just to get rid of copy_[to/from]_user() and syscall overhead?

[Mihnea Balta]

On Friday 10 January 2003 16:45, Mihnea Balta wrote:
> Hi,
>
> I have to implement a system which grabs udp packets off a gigabit
> connection, take some basic action based on what they contain, repack their
> data with a custom protocol header and send them through a gigabit ethernet
> interface on broadcast.

Following the indications I got here, I did the packet recieving stuff using a 
mmaped packet socket. From what I understand, that is a recieve-only 
interface, so it seems that I'm stuck with old sendto() for putting the 
packets back on the wire. I'd like to know if sendto() can do 20000 (small) 
packets/second on a fast x86 MP machine (dual or maybe quad) which doesn't do 
much besides this routing process. If it can't, please tell me if there's any 
feasible way of implementing a kind of buffer, i.e. keeping recieved packets 
in a local buffer and sending them alltoghether when the buffer fills or 
after a timer expires (I'm thinking DMA to the NIC or something simmilar, as 
the buffer will contain the complete packets, with all the required packets).

Thanks,
Mihnea

PS: feasible == not spending 1 month development time for a 1 
microsecond/packet improvement.