Awkward scenario: 3 interfaces and 3 devices with same ip/subnet.

Awkward scenario: 3 interfaces and 3 devices with same ip/subnet.

[Giacomo Bernardi]

Hi all,
want to solve a very awkward scenario?

There's a linux box with three interfaces (eth0, eth1, eth2) each
directly connected to an embedded device that has configuration:
- IP: 10.0.0.1
- Mask: 255.255.255.0
(in other words: all three devices answer to 10.0.0.1/24 and their
configuration can't be changed)

I need to send and receive snmp packets to an arbitrary given device
among these three ...how?

--- My proposed idea:

I think a solution would be to configure the three interfaces like following:
- eth0: 10.1.0.100/24
- eth1: 10.2.0.100/24
- eth2: 10.3.0.100/24

Then a set of rules like:
iptables -t mangle -A OUTPUT -d 10.3.0.1 -j MARK --set-mark 103     #
mark outgoing packets for 10.3.0.1
ip rule add fwmark 103 table 103                                    #
create an iproute table
ip route add table 103 default dev eth3                             #
send out marked packets on eth3
iptables -t nat -A POSTROUTING -m mark --mark 103 -j SNAT --to
10.0.0.100 # rewrite source ip
iptables -t nat -A OUTPUT -m mark --mark 103 -j DNAT --to 10.0.0.1  #
rewrite dest ip

And then send snmp queries to 'fake' addresses 10.1.0.1, 10.2.0.1 and 10.3.0.1.

Result:
- the device gets the packets and sends back the results (with
srcip=10.0.0.1 and dstip=10.0.0.100)
- but when the linux box gets them back, they are obviously discarded.

How can I do this?

Thanks very much for your help.
mino

Re: Awkward scenario: 3 interfaces and 3 devices with same ip/subnet.

[Jan Engelhardt]

On Tuesday 2010-08-31 13:34, Giacomo Bernardi wrote:

>mark outgoing packets for 10.3.0.1
>ip rule add fwmark 103 table 103                                    #
>create an iproute table
>ip route add table 103 default dev eth3                             #
>send out marked packets on eth3
>iptables -t nat -A POSTROUTING -m mark --mark 103 -j SNAT --to
>10.0.0.100 # rewrite source ip
>iptables -t nat -A OUTPUT -m mark --mark 103 -j DNAT --to 10.0.0.1  #
>rewrite dest ip
>
>And then send snmp queries to 'fake' addresses 10.1.0.1, 10.2.0.1 and 10.3.0.1.
>
>Result:
>- the device gets the packets and sends back the results (with
>srcip=10.0.0.1 and dstip=10.0.0.100)
>- but when the linux box gets them back, they are obviously discarded.
>
>How can I do this?

Check why replies do not get translated back. Use -j TRACE for them.

Re: Awkward scenario: 3 interfaces and 3 devices with same ip/subnet.

[Grant Taylor]

On 08/31/10 06:34, Giacomo Bernardi wrote:
> want to solve a very awkward scenario?
> 
> There's a linux box with three interfaces (eth0, eth1, eth2) each 
> directly connected to an embedded device that has configuration:
> - IP: 10.0.0.1
> - Mask: 255.255.255.0
> (in other words: all three devices answer to 10.0.0.1/24 and their 
> configuration can't be changed)
> 
> I need to send and receive snmp packets to an arbitrary given device 
> among these three ...how?

I don't know if it would work for you or not, but you could try an old / 
odd / if not dirty trick.

Try adding static ARP entries to each device's MAC address using 
different (bogus) IPs.  Then try communicating with the bogus IPs.  You 
might get lucky and be able to communicate.  It really depends on what 
is included in the higher layer protocol.  (I've not dealt with enough 
SNMP to know if this is possible.)

The main thing that the differing IPs are use for is to translate the 
layer 3 IP address to the layer 2 MAC address.  So, if you can side step 
that problem, you may be golden.

Yes, this is a dirty trick / hack, but sometimes that's what you have 
got to do.



Grant. . . .

Re: Awkward scenario: 3 interfaces and 3 devices with same ip/subnet.

[Giacomo Bernardi]

> Try adding static ARP entries to each device's MAC address using different
> (bogus) IPs.  Then try communicating with the bogus IPs.  You might get
> lucky and be able to communicate.  It really depends on what is included in
> the higher layer protocol.  (I've not dealt with enough SNMP to know if this
> is possible.)

Doesn't seem to work, despite I created the arp entry with:
 arp -s 10.2.0.1 00:11:22:33:44:55

To be honest I'm not surprised: how is the receiver of those datagram
supposed to know they are for itself, since the dstip doesn't match
the IP of the local incoming interface?

> Check why replies do not get translated back. Use -j TRACE for them.

Good idea.
I've added a:
iptables -t raw -A PREROUTING -s 10.0.0.1 -j TRACE
which I can see getting matches. However nothing is logging in
kern.log nor anywhere else (even configuring syslog with a "*.*"). Am
I forgetting something about the TRACE target?


-- 
Giacomo "mino" Bernardi

Re: Awkward scenario: 3 interfaces and 3 devices with same ip/subnet.

[Grant Taylor]

On 08/31/10 09:49, Giacomo Bernardi wrote:
> Doesn't seem to work, despite I created the arp entry with:
>  arp -s 10.2.0.1 00:11:22:33:44:55

The bogus IP will need to be in the same subnet to work.  I.e. 10.0.0.11.

> To be honest I'm not surprised: how is the receiver of those datagram
> supposed to know they are for itself, since the dstip doesn't match
> the IP of the local incoming interface?

Sometimes it does, sometimes it doesn't.  It really depends on how 
things are configured on the destination device.

It's one of those "It might fail, but if it works, it's worth the 90 
seconds it took to try it." things.

I often use this when I'm configuring devices (APs) that all share the 
same IP address at initial configuration.  I will set the MAC address in 
the ARP cache and then connect / configure as need.



Grant. . . .

Re: Awkward scenario: 3 interfaces and 3 devices with same ip/subnet.

[Giacomo Bernardi]

>> Doesn't seem to work, despite I created the arp entry with:
>>  arp -s 10.2.0.1 00:11:22:33:44:55
>
> The bogus IP will need to be in the same subnet to work.  I.e. 10.0.0.11.

Doesn't work:
I'm now trying with a single interface:
- eth0: down
- eth1: 10.0.0.100
- eth2: down

There's connected a device with IP 10.0.0.1 and mac 00:11:22:33:44:55.
If I do:
arp -s 10.0.0.11 00:11:22:33:44:55

I can reach 10.0.0.1, but not 10.0.0.11.


-- 
Giacomo "mino" Bernardi

Re: Awkward scenario: 3 interfaces and 3 devices with same ip/subnet.

[Marek Kierdelewicz]

Hi,

Here's the solution for your awkward scenario. It involves using
kvm/qemu guests like in the diagram
http://cat.piasta.pl/awkward_solution.jpeg

After implementing this configuration host system should be able to
access devices on ip addressess 10.(1,2,3).0.3

* Config on v1:

ip addr add 10.0.0.2/24 dev eth1
ip addr add 10.1.0.2/24 dev eth0
ip ro add default via 10.1.0.1
enable ipv4 forwarding

iptables -t nat -A PREROUTING -d 10.1.0.3 -j DNAT --to 10.0.0.1
iptables -t nat -A POSTROUTING -s 10.0.0.1 -j SNAT --to 10.1.0.3

* Config on v2:

ip addr add 10.0.0.2/24 dev eth1
ip addr add 10.2.0.2/24 dev eth0
ip ro add default via 10.2.0.1
enable ipv4 forwarding

iptables -t nat -A PREROUTING -d 10.2.0.3 -j DNAT --to 10.0.0.1
iptables -t nat -A POSTROUTING -s 10.0.0.1 -j SNAT --to 10.2.0.3

* Config on v3:

ip addr add 10.0.0.2/24 dev eth1
ip addr add 10.3.0.2/24 dev eth0
ip ro add default via 10.2.0.1
enable ipv4 forwarding

iptables -t nat -A PREROUTING -d 10.3.0.3 -j DNAT --to 10.0.0.1
iptables -t nat -A POSTROUTING -s 10.0.0.1 -j SNAT --to 10.3.0.3

* Config on host system

start kvms to get tap devices up
ip addr add 10.1.0.1/24 dev tap0
ip addr add 10.2.0.1/24 dev tap2
ip addr add 10.3.0.1/24 dev tap4
brctl addbr br0
ip link set up dev br0
brctl addif br0 tap1
brctl addif br0 eth1
brctl addbr br1
ip link set up dev br1
brctl addif br1 tap3
brctl addif br1 eth2
brctl addbr br2
ip link set up dev br2
brctl addif br1 tap5
brctl addif br1 eth3

echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables


Best regards,
Marek

Re: Awkward scenario: 3 interfaces and 3 devices with same ip/subnet.

[Giacomo Bernardi]

> Here's the solution for your awkward scenario. It involves using
> kvm/qemu guests like in the diagram
> http://cat.piasta.pl/awkward_solution.jpeg

Thanks very much Marek (though I'm still struggling to understand how
it can work).

Using VMs would be a problem as the software application that needs to
address the device is just one, do you see any way of replicating your
solution without VMs (e.g. using bridges and "virtual" interfaces)?


-- 
Giacomo "mino" Bernardi

Re: Awkward scenario: 3 interfaces and 3 devices with same ip/subnet.

[Marek Kierdelewicz]

Hi,

>Using VMs would be a problem as the software application that needs to
>address the device is just one, do you see any way of replicating your
>solution without VMs (e.g. using bridges and "virtual" interfaces)?

The sole purpose of the virtual guests in this solution is limited to
providing three isolated network stacks that can communicate with the
same ip unhindered. It's also achievable (and definitely less
cumbersome) using kernel namespaces - part of lxc [1].

If your software on host system can work with three
different device addresses (10.1.0.3, 10.2.0.3, 10.3.0.3), then you'd
need one instance of the application on host system.

If for some reason application is hardcoded to use 10.0.0.1 for device
access, then my solution won't do you any good :(.

Best regards,
Marek

[1]http://lxc.sourceforge.net/index.php/about/kernel-namespaces/network/

Re: Awkward scenario: 3 interfaces and 3 devices with same ip/subnet.

[Giacomo Bernardi]

Thanks for all your ideas...

> The sole purpose of the virtual guests in this solution is limited to
> providing three isolated network stacks that can communicate with the
> same ip unhindered. It's also achievable (and definitely less
> cumbersome) using kernel namespaces - part of lxc [1].

I'll have a look at lxc.

Anyway, conceptually the whole problem look very simple to me...
Ideally, I'd just need to apply the following pseudo-code that
overwrites the IP fields... isn't there a simple way to do this trick
with netfilter&c.?


// OUTPUT:

if(dst_ip == 10.1.0.1) {
	rewrite dst_ip to 10.0.0.1
	rewrite src_ip to 10.0.0.2
	output on interface eth1
}
if(dst_ip == 10.2.0.1) {
	rewrite dst_ip to 10.0.0.1
	rewrite src_ip to 10.0.0.2
	output on interface eth2
}
if(dst_ip == 10.3.0.1) {
	rewrite dst_ip to 10.0.0.1
	rewrite src_ip to 10.0.0.2
	output on interface eth3
}

// INPUT:

if(input_interface == eth1 && src_ip == 10.0.0.1) {
	rewrite src_ip to 10.1.0.1
	rewrite dst_ip to 10.1.0.2
}
if(input_interface == eth2 && src_ip == 10.0.0.1) {
	rewrite src_ip to 10.2.0.1
	rewrite dst_ip to 10.2.0.2
}
if(input_interface == eth3 && src_ip == 10.0.0.1) {
	rewrite src_ip to 10.3.0.1
	rewrite dst_ip to 10.3.0.2
}



-- 
Giacomo "mino" Bernardi

Re: Awkward scenario: 3 interfaces and 3 devices with same ip/subnet.

[Marek Kierdelewicz]

Hi,

>Anyway, conceptually the whole problem look very simple to me...
>Ideally, I'd just need to apply the following pseudo-code that
>overwrites the IP fields... isn't there a simple way to do this trick
>with netfilter&c.?

I don't think you'll be able to simultanously communicate with all
three devices using just one network stack/namespace. The reason is you
can have just one arpcache entry for 10.0.0.1. Correct me if I'm
wrong. Anyone?

Best regards,
Marek

Re: Awkward scenario: 3 interfaces and 3 devices with same ip/subnet.

[Giacomo Bernardi]

> The reason is you
> can have just one arpcache entry for 10.0.0.1. Correct me if I'm
> wrong. Anyone?

I may be wrong, but isn't arp per-interface?

root@host ~# arp -i eth0 -s 10.0.0.1 11:22:33:44:55:66
root@host ~# arp -i eth1 -s 10.0.0.1 11:22:33:44:55:66
root@host ~# arp -an
...
? (10.0.0.1) at 11:22:33:44:55:66 [ether] PERM on eth0
? (10.0.0.1) at 11:22:33:44:55:66 [ether] PERM on eth1



[sorry I forgot to cc the list]

--
Giacomo "mino" Bernardi

Re: Awkward scenario: 3 interfaces and 3 devices with same ip/subnet.

[Marek Kierdelewicz]

Hi,

>I may be wrong, but isn't arp per-interface?
>root@host ~# arp -i eth0 -s 10.0.0.1 11:22:33:44:55:66
>root@host ~# arp -i eth1 -s 10.0.0.1 11:22:33:44:55:66

Then you're right :). It's doable on one network stack. I'm going to
run some tests.

Best regards,
Marek