Ethernet-over-UDP virtual network interface

Ethernet-over-UDP virtual network interface

[Mitar]

Hi!

At community wireless network wlan slovenija (http://dev.wlan-si.net/)
we have been using OpenVPN tunnels to connect our WiFi nodes together
over existing Internet infrastructure. After some time of using it we
have discovered throughput problems using a user-land tunneling
solution on those small/consumer Wifi router hardware. Because of the
context switches we cannot get more than 5 Mbit/s even with disabled
encryption. (And as we have a lot of fiber here, it really makes
uplinks unused a lot.)

We found out that we need a really light-weight tunneling solution,
but on a L2 level. The idea is to have a simple encryption-less,
state-less, and session-less L2 tunneling. Some kind of
Ethernet-over-UDP type of virtual interface where we could configure
multiple peer IP address and virtual interface would work as a simple
switch, only instead of cables, it would send packets encapsulated in
UDP (EoIP is not good because many consumer routers do not allow
non-TCP/UDP packets to get through). No session handling, nothing.
Simply, any packets it gets in it is send (based on auto-learned MAC
addresses) to the destination IP address in the UDP packet. On the
other side decapsulated packet is simply output of the virtual
interface there.

I have searched around and have not found anything which would work
like this. Have I missed something? If not, I have decided to write
our own kernel module for this, but first I would like to hear some
feedback from others. Would anybody be interested in developing it
with me or at mentoring me (it would be my first kernel module)?

I have found this code:

https://lwn.net/Articles/199120/

and was planning to use it as a basis, but use UDP for communication.
What is a proper way to do UDP communication from a kernel module?

Best regards and thank you all for the great work you are doing on Linux


Mitar

Re: Ethernet-over-UDP virtual network interface

[Denys Fedoryshchenko]

 From my wiki :)
ip link add eoip1 type gretap remote IP1 local IP2 nopmtudisc
ifconfig eth1 0.0.0.0
ifconfig eoip1 mtu 1500 up
brctl addbr br0
brctl addif br0 eoip1
brctl addif br0 eth1
ifconfig br0 up

But not sure this will work, just a hint for kernel-mode tunnel


On 2012-03-04 13:30, Mitar wrote:
> Hi!
>
> At community wireless network wlan slovenija 
> (http://dev.wlan-si.net/)
> we have been using OpenVPN tunnels to connect our WiFi nodes together
> over existing Internet infrastructure. After some time of using it we
> have discovered throughput problems using a user-land tunneling
> solution on those small/consumer Wifi router hardware. Because of the
> context switches we cannot get more than 5 Mbit/s even with disabled
> encryption. (And as we have a lot of fiber here, it really makes
> uplinks unused a lot.)
>
> We found out that we need a really light-weight tunneling solution,
> but on a L2 level. The idea is to have a simple encryption-less,
> state-less, and session-less L2 tunneling. Some kind of
> Ethernet-over-UDP type of virtual interface where we could configure
> multiple peer IP address and virtual interface would work as a simple
> switch, only instead of cables, it would send packets encapsulated in
> UDP (EoIP is not good because many consumer routers do not allow
> non-TCP/UDP packets to get through). No session handling, nothing.
> Simply, any packets it gets in it is send (based on auto-learned MAC
> addresses) to the destination IP address in the UDP packet. On the
> other side decapsulated packet is simply output of the virtual
> interface there.
>
> I have searched around and have not found anything which would work
> like this. Have I missed something? If not, I have decided to write
> our own kernel module for this, but first I would like to hear some
> feedback from others. Would anybody be interested in developing it
> with me or at mentoring me (it would be my first kernel module)?
>
> I have found this code:
>
> https://lwn.net/Articles/199120/
>
> and was planning to use it as a basis, but use UDP for communication.
> What is a proper way to do UDP communication from a kernel module?
>
> Best regards and thank you all for the great work you are doing on 
> Linux
>
>
> Mitar
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ethernet-over-UDP virtual network interface

[Mitar]

Hi!

The problem of GRE IP packets (IP protocol type 47) is that some
consumer routers filter them by default (or even do not have an option
to pass them through).

Maybe the best way would be just to implement greudptap link type
which would be the same as gretap, only it would be using UDP packets?
Do Linux bridge interfaces have MAC auto-discovery?

Do Linux gretap tunnels have MTU auto-discovery?

BTW, why is necessary to configure local IP address of the tunnel?
Shouldn't only a local MAC address be enough? And IP address only on
bridge interface?


Mitar

On Sun, Mar 4, 2012 at 1:21 PM, Denys Fedoryshchenko <denys@visp.net.lb> wrote:
> From my wiki :)
> ip link add eoip1 type gretap remote IP1 local IP2 nopmtudisc
> ifconfig eth1 0.0.0.0
> ifconfig eoip1 mtu 1500 up
> brctl addbr br0
> brctl addif br0 eoip1
> brctl addif br0 eth1
> ifconfig br0 up
>
> But not sure this will work, just a hint for kernel-mode tunnel
>
>
>
> On 2012-03-04 13:30, Mitar wrote:
>>
>> Hi!
>>
>> At community wireless network wlan slovenija (http://dev.wlan-si.net/)
>> we have been using OpenVPN tunnels to connect our WiFi nodes together
>> over existing Internet infrastructure. After some time of using it we
>> have discovered throughput problems using a user-land tunneling
>> solution on those small/consumer Wifi router hardware. Because of the
>> context switches we cannot get more than 5 Mbit/s even with disabled
>> encryption. (And as we have a lot of fiber here, it really makes
>> uplinks unused a lot.)
>>
>> We found out that we need a really light-weight tunneling solution,
>> but on a L2 level. The idea is to have a simple encryption-less,
>> state-less, and session-less L2 tunneling. Some kind of
>> Ethernet-over-UDP type of virtual interface where we could configure
>> multiple peer IP address and virtual interface would work as a simple
>> switch, only instead of cables, it would send packets encapsulated in
>> UDP (EoIP is not good because many consumer routers do not allow
>> non-TCP/UDP packets to get through). No session handling, nothing.
>> Simply, any packets it gets in it is send (based on auto-learned MAC
>> addresses) to the destination IP address in the UDP packet. On the
>> other side decapsulated packet is simply output of the virtual
>> interface there.
>>
>> I have searched around and have not found anything which would work
>> like this. Have I missed something? If not, I have decided to write
>> our own kernel module for this, but first I would like to hear some
>> feedback from others. Would anybody be interested in developing it
>> with me or at mentoring me (it would be my first kernel module)?
>>
>> I have found this code:
>>
>> https://lwn.net/Articles/199120/
>>
>> and was planning to use it as a basis, but use UDP for communication.
>> What is a proper way to do UDP communication from a kernel module?
>>
>> Best regards and thank you all for the great work you are doing on Linux
>>
>>
>> Mitar
>> --
>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>

Re: Ethernet-over-UDP virtual network interface

[Emanuil Hristov]

Hi
you can try l2tp(v3) :) i think it fit best for your needs

Best regards

On Sun, Mar 4, 2012 at 1:30 PM, Mitar <mmitar@gmail.com> wrote:
> Hi!
>
> At community wireless network wlan slovenija (http://dev.wlan-si.net/)
> we have been using OpenVPN tunnels to connect our WiFi nodes together
> over existing Internet infrastructure. After some time of using it we
> have discovered throughput problems using a user-land tunneling
> solution on those small/consumer Wifi router hardware. Because of the
> context switches we cannot get more than 5 Mbit/s even with disabled
> encryption. (And as we have a lot of fiber here, it really makes
> uplinks unused a lot.)
>
> We found out that we need a really light-weight tunneling solution,
> but on a L2 level. The idea is to have a simple encryption-less,
> state-less, and session-less L2 tunneling. Some kind of
> Ethernet-over-UDP type of virtual interface where we could configure
> multiple peer IP address and virtual interface would work as a simple
> switch, only instead of cables, it would send packets encapsulated in
> UDP (EoIP is not good because many consumer routers do not allow
> non-TCP/UDP packets to get through). No session handling, nothing.
> Simply, any packets it gets in it is send (based on auto-learned MAC
> addresses) to the destination IP address in the UDP packet. On the
> other side decapsulated packet is simply output of the virtual
> interface there.
>
> I have searched around and have not found anything which would work
> like this. Have I missed something? If not, I have decided to write
> our own kernel module for this, but first I would like to hear some
> feedback from others. Would anybody be interested in developing it
> with me or at mentoring me (it would be my first kernel module)?
>
> I have found this code:
>
> https://lwn.net/Articles/199120/
>
> and was planning to use it as a basis, but use UDP for communication.
> What is a proper way to do UDP communication from a kernel module?
>
> Best regards and thank you all for the great work you are doing on Linux
>
>
> Mitar
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ethernet-over-UDP virtual network interface

[Benjamin LaHaise]

On Sun, Mar 04, 2012 at 12:30:01PM +0100, Mitar wrote:
...
> I have found this code:
> 
> https://lwn.net/Articles/199120/
> 
> and was planning to use it as a basis, but use UDP for communication.
> What is a proper way to do UDP communication from a kernel module?
> 
> Best regards and thank you all for the great work you are doing on Linux

Why not use L2TP?  It can operate over UDP (v2 or v3) or directly over IP 
if needed (v3).  L2TPv3 also supports ethernet pseudo-wires.

		-ben
-- 
"Thought is the essence of where you are now."

Re: Ethernet-over-UDP virtual network interface

[Mitar]

Hi!

On Sun, Mar 4, 2012 at 6:01 PM, Benjamin LaHaise <bcrl@kvack.org> wrote:
> Why not use L2TP?  It can operate over UDP (v2 or v3) or directly over IP
> if needed (v3).  L2TPv3 also supports ethernet pseudo-wires.

I would like something state-less. I see L2TPv3 has support for
unmanaged tunnels, but they still require tunnel id and session id
what (if I assume that they have to be unique) makes it useless for
our case. We have (if I simplify) a star-shaped topology with a
central server in the middle. To the central server WiFi nodes
(hundreds of them) connect. But we do not really want to require from
a central server to know each WiFi node and prepare its tunnel
endpoint for each node. We would just like that there would be virtual
interface and UDP port and any packet send to the UDP port would be
decapsulated and output through that virtual interface. The only thing
which we would have to make sure is that each WiFi node has a virtual
interface with unique MAC address.


Mitar

Re: Ethernet-over-UDP virtual network interface

[Benjamin LaHaise]

On Tue, Mar 06, 2012 at 12:58:44AM +0100, Mitar wrote:
> I would like something state-less. I see L2TPv3 has support for

How stateless?

> unmanaged tunnels, but they still require tunnel id and session id
> what (if I assume that they have to be unique) makes it useless for
> our case. We have (if I simplify) a star-shaped topology with a
> central server in the middle. To the central server WiFi nodes
> (hundreds of them) connect. But we do not really want to require from
> a central server to know each WiFi node and prepare its tunnel
> endpoint for each node. We would just like that there would be virtual
> interface and UDP port and any packet send to the UDP port would be
> decapsulated and output through that virtual interface. The only thing
> which we would have to make sure is that each WiFi node has a virtual
> interface with unique MAC address.

L2TP is commonly used for star topology deployments, but with more 
flexibility.  The typical scenario is to use LACs (L2TP Access 
Concentrators) on the edge of the network to tunnel incoming sessions 
from clients to LNSes (L2TP Network servers) that actually perform the 
routing for the network.  There is some state involved, but tunnels can 
be configured without a password, effectively making the addition of new
LACs requiring no manually configured state on the LNS.  The Babylon PPP 
stack that I added L2TP and PPPoE support for is able to perform tunnel 
switching on incoming PPPoE or L2TP sessions.  Sessions can be directed to 
different LNSes based on the username an incoming session authenticates 
as.  It only supports L2TPv2 at present, though, but adding L2TPv3 wouldn't 
be too hard.

		-ben
-- 
"Thought is the essence of where you are now."

Re: Ethernet-over-UDP virtual network interface

[Mitar]

Hi!

On Thu, Mar 8, 2012 at 8:11 AM, Benjamin LaHaise <bcrl@kvack.org> wrote:
> On Tue, Mar 06, 2012 at 12:58:44AM +0100, Mitar wrote:
>> I would like something state-less. I see L2TPv3 has support for
>
> How stateless?

Stateless in the ethernet-like manner: each incoming packet is simply
decapsulated and seen as it came over the (virtual) ethernet wire.

Of course this is because I have one implementation of our needs in my
head. Probably stateless is not necessary if everything is done
automatically. But what I am saying is that it could be even
stateless, something very very simple (and efficient). That we do not
need state/sessions (on a tunnel level, of course on the IP level of
packets send around then network kernel can (and should) keep state).
Not much CPU computation, not much memory copying, no user-space
context switching.

The idea is that we need a very simple solution, robust, no
encryption, no authentication. Just to improve the throughput on
consumer routers (today with N protocol you can easily have 15Mbit/s+
links and with fiber to tunnel them further this means that CPU is
often a bottleneck).

As I see there is no simple (both in implementation and configuration
manner) solution to this problem yet? Would it be then interesting to
develop such kernel module as I described in my initial post? Is there
any chance of it being accepted into the kernel?


Mitar

Re: Ethernet-over-UDP virtual network interface

[Benjamin LaHaise]

On Sat, Mar 10, 2012 at 08:31:28AM -0800, Mitar wrote:
> Hi!
> 
> On Thu, Mar 8, 2012 at 8:11 AM, Benjamin LaHaise <bcrl@kvack.org> wrote:
> > On Tue, Mar 06, 2012 at 12:58:44AM +0100, Mitar wrote:
> >> I would like something state-less. I see L2TPv3 has support for
> >
> > How stateless?
> 
> Stateless in the ethernet-like manner: each incoming packet is simply
> decapsulated and seen as it came over the (virtual) ethernet wire.

To be honest, I don't see any new stateless protocol actually being widely 
supported.  Building a completely proprietary solution (regardless of it 
being open source or not) that isn't supported by any common hardware or 
software networking platforms will paint yourself into a corner when it 
comes time to upgrade the network with additional capacity.

> Of course this is because I have one implementation of our needs in my
> head. Probably stateless is not necessary if everything is done
> automatically. But what I am saying is that it could be even
> stateless, something very very simple (and efficient). That we do not
> need state/sessions (on a tunnel level, of course on the IP level of
> packets send around then network kernel can (and should) keep state).
> Not much CPU computation, not much memory copying, no user-space
> context switching.

MPLS falls under that category, but it isn't commonly used in a completely 
stateless fashion either.  Bridging is relatively inexpensive, and GVRP is 
commonly used to dynamically configure VLANs in aggregation networks.  It 
has the advantage of being well supported by managed switches.  L2TP tunnel 
switching is a low cost operation (change tunnel/session id and ip 
addresses in the header), and, again, is well supported.

If you're using Linux as a router, I'd probably lean towards a VLAN based 
protocol, as anything exotic won't be supported by common nic's hardware 
checksum offloading.  L2TP can at least be carried in UDP packets which 
will can have their checksum calculation offloaded.  I don't think MPLS 
headers are recognized by any commodity NICs.

> The idea is that we need a very simple solution, robust, no
> encryption, no authentication. Just to improve the throughput on
> consumer routers (today with N protocol you can easily have 15Mbit/s+
> links and with fiber to tunnel them further this means that CPU is
> often a bottleneck).

Stateless != robust.  To deal with failed links you need some sort of state 
that is maintained by the system to switch over to backup links.

> As I see there is no simple (both in implementation and configuration
> manner) solution to this problem yet? Would it be then interesting to
> develop such kernel module as I described in my initial post? Is there
> any chance of it being accepted into the kernel?

Again, I don't see the point, as Yet Another Tunneling Protocol is a mess 
of support work for no real gain.  Existing options (MPLS, GVRP, L2TP) can 
all be configured with a minimum amount of state, are widely used for the 
scenario you're describing, and have migration paths when you need to 
upgrade to higher capacity switches/routers.

		-ben
-- 
"Thought is the essence of where you are now."

Re: Ethernet-over-UDP virtual network interface

[Mitar]

Hi!

On Sat, Mar 10, 2012 at 9:06 AM, Benjamin LaHaise <bcrl@kvack.org> wrote:
> To be honest, I don't see any new stateless protocol actually being widely
> supported.

Yes, I agree. This is probably because our use case is quite specific:
using consumer low-price range network equipment without any dedicated
network chips and so on. So simplicity is much more important to use
than, for example, security (which we do not care much on VPN links
because we already have everything open in the mesh so anybody can
capture or inject packets already, so there is no big reason why to do
any encryption or authentication (on the link level) on VPN links.

> Building a completely proprietary solution (regardless of it
> being open source or not) that isn't supported by any common hardware or
> software networking platforms will paint yourself into a corner when it
> comes time to upgrade the network with additional capacity.

That's why I am trying to coordinate our efforts with at least Linux
kernel to be included in the kernel. And also, for example, OpenWrt
distribution. Then we will mostly cover all our target devices.

I am not sure what problems do you have in mind for when upgrading our
network with additional capacity?

> If you're using Linux as a router, I'd probably lean towards a VLAN based
> protocol, as anything exotic won't be supported by common nic's hardware
> checksum offloading.

I do not understand what VLANs have with tunneling over the Internet?
We are building a mesh network, we parts of this network, which do not
see each other wirelessly, to connect through VPN tunnels. Because we
have cheap hardware we want simplest (CPU light) VPN solution to
connect those parts over existing infrastructure (read: ISPs).
Furthermore, because nodes are deployed in chaotic and uncoordinated
fashion by volunteers those tunnels have to be self-configuring (or
zero-configuring). We can provide few VPN servers and all nodes then
connect there, connecting nodes connected to them to the rest of the
network.

> Stateless != robust.  To deal with failed links you need some sort of state
> that is maintained by the system to switch over to backup links.

No. This we do on L3 with dynamic routing protocol (like OLSR) which
we are already using because of the wireless nature of our network
(and because nodes are coming up and down as volunteers' mothers pull
out nodes' power supplies when they are vacuuming). So we just need L2
links. If they work (are not firewalled, VPN servers are online, ...)
or not then determines L3 routing protocol and setup routes
accordingly.

> Existing options (MPLS, GVRP, L2TP) can
> all be configured with a minimum amount of state, are widely used for the
> scenario you're describing, and have migration paths when you need to
> upgrade to higher capacity switches/routers.

Will have to check this a bit more. I admit I do not know much about
MPLS and GVRP. But true, state is not so big problem if tunnels can
still be:

- kernel space only
- no encryption
- zero or auto configuration
- one server can accept multiple connections
- using UDP (and not IP) for transport
- of course, supported in Linux kernel


Mitar