Interconnect virtual device?

Interconnect virtual device?

[Ben Greear]

Hello!

I am considering writing a kernel module to provide a very
simple virtual interface.  This interface would be attached
to another interface, and would not be directly associated with
hardware.  It can have IP addresses & routing tables, and in
almost every way appear to be a regular ethernet interface.

The idea is that when you tx on this virtual interface, it
will cause a packet to be received on another interface.  And,
when a packet is received on the interface, it will tx on the
other interface.

I believe I can use this to create some interesting effects with
routing and bridging on a limitted number of physical interfaces.

Comments?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Interconnect virtual device?

[jamal]

On Mon, 2005-02-28 at 00:15, Ben Greear wrote:
> Hello!
> 
> I am considering writing a kernel module to provide a very
> simple virtual interface.  This interface would be attached
> to another interface, and would not be directly associated with
> hardware.  It can have IP addresses & routing tables, and in
> almost every way appear to be a regular ethernet interface.
> 
> The idea is that when you tx on this virtual interface, it
> will cause a packet to be received on another interface.  And,
> when a packet is received on the interface, it will tx on the
> other interface.
> 

In other words, it redirects to another device - I take it based on some
rules. Sounds to me like the mirred action already does this (and in
addition can mirror).

> I believe I can use this to create some interesting effects with
> routing and bridging on a limitted number of physical interfaces.
> 

What "interesting effects" ?

cheers,
jamal

Re: Interconnect virtual device?

[Ben Greear]

jamal wrote:
> On Mon, 2005-02-28 at 00:15, Ben Greear wrote:
> 
>>Hello!
>>
>>I am considering writing a kernel module to provide a very
>>simple virtual interface.  This interface would be attached
>>to another interface, and would not be directly associated with
>>hardware.  It can have IP addresses & routing tables, and in
>>almost every way appear to be a regular ethernet interface.
>>
>>The idea is that when you tx on this virtual interface, it
>>will cause a packet to be received on another interface.  And,
>>when a packet is received on the interface, it will tx on the
>>other interface.
>>
> 
> 
> In other words, it redirects to another device - I take it based on some
> rules. Sounds to me like the mirred action already does this (and in
> addition can mirror).

Actually, I was thinking that either user-space or perhaps a kernel
module could use the standard methods of receiving all pkts in promisc
mode to do the bridging portion.  If I force the real ethernet
interface to have no IP address, and put it into promisc mode, then
I can effectively bridge in user-space.

When I re-insert the pkts into the stack by writing to the virtual
device, the (potentially modified) packet can be
routed and firewalled, etc just like a normal packet received from the
network.  I may need to have two virtual inter-connects, one w/out
an IP that does the bridging portion, and one with an IP that actually
communicates with the rest of the kernel.  In this case, the two virtuals
would be connected to each other.

I think this could provide a way to do very customized actions on
raw packets without having to hack the kernel on an ongoing basis.


Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Interconnect virtual device?

[jamal]

Not sure if i ever responded to you.

On Mon, 2005-02-28 at 12:24, Ben Greear wrote:
> jamal wrote:
[..]
> > In other words, it redirects to another device - I take it based on some
> > rules. Sounds to me like the mirred action already does this (and in
> > addition can mirror).
> 
> Actually, I was thinking that either user-space or perhaps a kernel
> module could use the standard methods of receiving all pkts in promisc
> mode to do the bridging portion.  If I force the real ethernet
> interface to have no IP address, and put it into promisc mode, then
> I can effectively bridge in user-space.
> 
> When I re-insert the pkts into the stack by writing to the virtual
> device, the (potentially modified) packet can be
> routed and firewalled, etc just like a normal packet received from the
> network.  I may need to have two virtual inter-connects, one w/out
> an IP that does the bridging portion, and one with an IP that actually
> communicates with the rest of the kernel.  In this case, the two virtuals
> would be connected to each other.
> 
> I think this could provide a way to do very customized actions on
> raw packets without having to hack the kernel on an ongoing basis.
> 

There are two ways to do this:

a) You could redirect to a packet socket - a small extension needed to
the redirect action (mostly mechanical details involved like keeping
state of which sockets are open etc).

b) My preference is to push this gentleman's PF_RING
(http://www.ntop.org/ntop.html) netdevice into the kernel. He has
replicated unfortunately a lot of the stuff already done by MMAPED
packet socket - but i think we can forgive him since solution a) would
require hacking packet socket.

Reinjection of packets still needs working for that device - just as
much as a few cleanups here and there. The problem is the guy is not
very responsive - I have a lot of notes on his stuff if you are willing
to chase him around.
You can then get redirection to this device for free (for either
incoming or outgoing packets); something like:

tc filter add dev eth0 .... \
match ip src 10.0.0.1/32 \
action mirred egress redirect dev ring0

Assuming you have a program running on user space you should receive all
packets incoming and/or outgoing on eth0.

And no, you dont need the eth device to have a ip address attached.

cheers,
jamal

Re: Interconnect virtual device?

[Ben Greear]

jamal wrote:

> There are two ways to do this:
> 
> a) You could redirect to a packet socket - a small extension needed to
> the redirect action (mostly mechanical details involved like keeping
> state of which sockets are open etc).

I'd rather not take this approach, as I'd like to have this
functionality available in a kernel module as well as user-space.  Netdevices
are easy to work with in both user-space and kernel-space.

> b) My preference is to push this gentleman's PF_RING
> (http://www.ntop.org/ntop.html) netdevice into the kernel. He has
> replicated unfortunately a lot of the stuff already done by MMAPED
> packet socket - but i think we can forgive him since solution a) would
> require hacking packet socket.
> 
> Reinjection of packets still needs working for that device - just as
> much as a few cleanups here and there. The problem is the guy is not
> very responsive - I have a lot of notes on his stuff if you are willing
> to chase him around.
> You can then get redirection to this device for free (for either
> incoming or outgoing packets); something like:
> 
> tc filter add dev eth0 .... \
> match ip src 10.0.0.1/32 \
> action mirred egress redirect dev ring0
> 
> Assuming you have a program running on user space you should receive all
> packets incoming and/or outgoing on eth0.
> 
> And no, you dont need the eth device to have a ip address attached.

Just mirror-ing will not meet my goal.  I may also wish to drop packets
entirely, before they ever reach any of the protocol stacks.

That said, a brief glance at the ntop page leads me to believe that
his packet socket might be interesting for other reasons.  But, I have
enough fun trying to push my own stuff into the kernel... probably
won't bother trying to push his stuff in too :)

Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Interconnect virtual device?

[jamal]

On Wed, 2005-03-02 at 17:34, Ben Greear wrote:
> jamal wrote:
> 
> > There are two ways to do this:
> > 
> > a) You could redirect to a packet socket - a small extension needed to
> > the redirect action (mostly mechanical details involved like keeping
> > state of which sockets are open etc).
> 
> I'd rather not take this approach, as I'd like to have this
> functionality available in a kernel module as well as user-space.  Netdevices
> are easy to work with in both user-space and kernel-space.

sure - you have modules and user space interface. But lets drop 
it here - I dont like it either because it may end up being a lot of
work.

> > tc filter add dev eth0 .... \
> > match ip src 10.0.0.1/32 \
> > action mirred egress redirect dev ring0
> > 
> > Assuming you have a program running on user space you should receive all
> > packets incoming and/or outgoing on eth0.
> > 
> > And no, you dont need the eth device to have a ip address attached.
> 
> Just mirror-ing will not meet my goal. 

The above was a total redirect, not mirroring; to mirror you would
say "action mirred egress mirror dev ring0"
And for fun you could mirror to multiple devices if you wanted.

>  I may also wish to drop packets
> entirely, before they ever reach any of the protocol stacks.

Of course thats what the actions are for. 

tc packets coming on dev XXXX before stack
 match some header ..
   action drop 

Or to add to the fun factor:

match some header ...
// randomly allow every 10th packet
action drop random netrand ok 10
// and redirect the lucky packet to user space
action mirred egress redirect dev ring0

> That said, a brief glance at the ntop page leads me to believe that
> his packet socket might be interesting for other reasons.  But, I have
> enough fun trying to push my own stuff into the kernel... probably
> won't bother trying to push his stuff in too :)
> 

Clearly above you are trying to reinvent whats already available.
And i pointed to that gent because i think hes done a good job already -
theres no point in reinventing what he already has in particular since
hes spent time to debug it and hes got people using it already (he seems
to be selling some product based on it). If he fails to cooperate then
by all means replicating his work should be fine.

cheers,
jamal

Re: Interconnect virtual device?

[Thomas Graf]

> b) My preference is to push this gentleman's PF_RING
> (http://www.ntop.org/ntop.html) netdevice into the kernel. He has
> replicated unfortunately a lot of the stuff already done by MMAPED
> packet socket - but i think we can forgive him since solution a) would
> require hacking packet socket.
> 
> Reinjection of packets still needs working for that device - just as
> much as a few cleanups here and there. The problem is the guy is not
> very responsive - I have a lot of notes on his stuff if you are willing
> to chase him around.
> You can then get redirection to this device for free (for either
> incoming or outgoing packets); something like:
> 
> tc filter add dev eth0 .... \
> match ip src 10.0.0.1/32 \
> action mirred egress redirect dev ring0

I think we talked about this once already and I do like the idea
but the reinjection is at least of the same importance to me. What
I'm thinking of basically gets down to two ring buffers both holding
mmaped buffers. The action enqueues skbs to the first ring buffer
and userspace dequeues it from there. The skb gets completely lost
at this point by having it shot or just cloned if mirroring is requested.
Userspace may reinject the skb again by putting it onto the second
ring buffer for a kernel thread to pick up and reinject it at
netif_receive_skb. Userspace may do whathever it likes as long as
the checksum gets corrected, it may also fragment packets and reinject
more than it originally received. Obviously for the redirect to
userspace the skb must fullfil quite a lot of requirements only
achievable on ingress but it would open up possibilities quite a lot.

Re: Interconnect virtual device?

[jamal]

On Wed, 2005-03-02 at 17:55, Thomas Graf wrote:

> I think we talked about this once already and I do like the idea
> but the reinjection is at least of the same importance to me. What
> I'm thinking of basically gets down to two ring buffers both holding
> mmaped buffers.

The ring device already has the recv side direction ring,
Whats needed is tx side.

>  The action enqueues skbs to the first ring buffer
> and userspace dequeues it from there. The skb gets completely lost
> at this point by having it shot or just cloned if mirroring is requested.

This will happen by default i.e the mirred action will take care of
buffer management and hopefully the ring device will worry about reusing
those skbs.

> Userspace may reinject the skb again by putting it onto the second
> ring buffer for a kernel thread to pick up and reinject it at
> netif_receive_skb. Userspace may do whathever it likes as long as
> the checksum gets corrected, it may also fragment packets and reinject
> more than it originally received. Obviously for the redirect to
> userspace the skb must fullfil quite a lot of requirements only
> achievable on ingress but it would open up possibilities quite a lot.

One thing the PF_RING device needs is some form of metadata that can be
passed to user space. PF_PACKET with MMAP has a few, but we need to do a
lot more such as exposing most if not all of the skb metadata.
Similarly, the direction from user space will contain details of where
this packet is going to go (ingress vs egress) - PF_PACKET assumes
egress only at the moment.

cheers,
jamal

Re: Interconnect virtual device?

[Max Krasnyansky]

jamal wrote:
> On Wed, 2005-03-02 at 17:55, Thomas Graf wrote:
> 
>>I think we talked about this once already and I do like the idea
>>but the reinjection is at least of the same importance to me. What
>>I'm thinking of basically gets down to two ring buffers both holding
>>mmaped buffers.
> 
> 
> The ring device already has the recv side direction ring,
> Whats needed is tx side.
> 
> 
>> The action enqueues skbs to the first ring buffer
>>and userspace dequeues it from there. The skb gets completely lost
>>at this point by having it shot or just cloned if mirroring is requested.
> 
> 
> This will happen by default i.e the mirred action will take care of
> buffer management and hopefully the ring device will worry about reusing
> those skbs.
> 
>>Userspace may reinject the skb again by putting it onto the second
>>ring buffer for a kernel thread to pick up and reinject it at
>>netif_receive_skb. Userspace may do whathever it likes as long as
>>the checksum gets corrected, it may also fragment packets and reinject
>>more than it originally received. Obviously for the redirect to
>>userspace the skb must fullfil quite a lot of requirements only
>>achievable on ingress but it would open up possibilities quite a lot.
> 
> 
> One thing the PF_RING device needs is some form of metadata that can be
> passed to user space. PF_PACKET with MMAP has a few, but we need to do a
> lot more such as exposing most if not all of the skb metadata.
> Similarly, the direction from user space will contain details of where
> this packet is going to go (ingress vs egress) - PF_PACKET assumes
> egress only at the moment.

btw I'm going to add mmap() interface for the TUN driver (I need it for other
stuff myself). In which case it seems that it should do pretty much the same
thing that you guys are talking about. i.e. mmap(/dev/net/tun), setup mirroring
to tunX, copy frames in/out of tunX.

Max