Multiple subnets of the same IP space, different VLANs, same box...

Multiple subnets of the same IP space, different VLANs, same box...

[Aaron S. Joyner]

Okay, first let me admit that what I'm trying to do is a bit on the 
weird side, and stems from some historical requirements that I can not 
easily change.  Having said that, here's the scoop.

I have numerous subnets, which are all identical.  Each has devices in 
the 10.1.1.0/24 range that need to talk to a server at 10.1.1.1.  All 
subnets for the sake of argument will be Class C ranges.  (1)  They are 
all connected through Cisco switches, via 802.1q VLANs, back to a single 
Linux box.  Each subnet comes in on a virtual interface (courtesy of 
vconfig), such as eth1.10, eth1.11, etc.  I need to be able to respond 
to each of those interfaces as 10.1.1.1, with (so far) only ICMP and UDP 
traffic.

At first, I'd hoped to be able to simply assign that same IP, 10.1.1.1, 
over and over to each of the VLAN interfaces, and everything would work 
itself out.  Unfortunately, that doesn't work.  It works fine for the 
first box, but not for the second.  The second IP, on the second VLAN, 
is unable to correctly ARP with the second client.  I can beat the ARP 
problem temporarily by forcing a ping out the VLAN interface (ping -I 
eth1.11 10.1.1.2, for example), or by adding a static ARP entry, but 
even after ARP is handled, traffic simply doesn't flow up the IP stack 
(or more realistically I think it's not flowing back *down* the IP stack 
correctly).

In discussing this problem with a friend, he suggests that the problem 
is that when Linux attempts to take a packet from the responding service 
(ICMP from the kernel, or tftp from tftpd, for example), it starts fresh 
to determine how to send out that packet, attempts to look up the 
destination in the ARP cache, regardless of the interface it came in on, 
and sends to the first entry it finds.  I currently believe that this is 
why the simple / ideal solution I describe above is failing.

I've considered that some magic with iptables DNAT, SNAT, or REDIRECT 
targets, perhaps combined with the mangle table's ROUTE target, may 
allow this to be possible, but I can't wrap my head around how to get 
the packets properly mangled, and I'm still not 100% convinced that will 
bypass the ARP problem.  Another possible (very ugly) solution would be 
to run virtual machines attached to each of those VLAN interfaces, which 
would separate out the problem of one "box" having lots of identical IP 
addresses and ARP entries, but it seems like an amazing waste of resources.

Any input anyone can offer about possible solutions, or even providing 
further insight into the problem, would be most appreciated.  :)

Aaron S. Joyner

(1) IP addresses have, of course, been changed to protect the innocent.

Re: [VLAN] Multiple subnets of the same IP space, different VLANs, same box...

[Kevin P. Fleming]

Aaron S. Joyner wrote:

> Any input anyone can offer about possible solutions, or even providing 
> further insight into the problem, would be most appreciated.  :)

I don't see how you can make this work without SNAT'ing all the incoming 
traffic on each VLAN interface into a unique IP address range. In 
addition, as you already mentioned, you'll have to mess with the routing 
tables as well; you can apply a "mark" to the connections in the 
connection table for each VLAN interface, then use that to select an 
outbound routing table.

It's pretty complex; realistically, you might actually be better off 
creating virtual machines to NAT the traffic onto the "internal" network 
in the box, then hide it all from the services on that box.

Alternatively, you could buy a whole boatload of cheap NAT routers, and 
attach each one of them to a non-tagged VLAN port on your switch (one 
for each VLAN), then connect the "upstream" side of them to a 
non-VLAN-enabled switch with your Linux box on it.

Third alternative would be to do the VLAN switching with a device that 
is actually a Layer 3 router as well, that can do the NAT for you.

All of these solutions, though, involve NAT, which will bring along 
other problems if you are running applications/protocols that are not 
NAT friendly.

Re: Multiple subnets of the same IP space, different VLANs, same box...

[Jason Opperisano]

On Tue, Feb 15, 2005 at 12:17:05PM -0500, Aaron S. Joyner wrote:
> Okay, first let me admit that what I'm trying to do is a bit on the 
> weird side,

nah--this request comes up weekly...

> and stems from some historical requirements that I can not 
> easily change.  Having said that, here's the scoop.
> 
> I have numerous subnets, which are all identical.  Each has devices in 
> the 10.1.1.0/24 range that need to talk to a server at 10.1.1.1.  All 
> subnets for the sake of argument will be Class C ranges.  (1)  They are 
> all connected through Cisco switches, via 802.1q VLANs, back to a single 
> Linux box.  Each subnet comes in on a virtual interface (courtesy of 
> vconfig), such as eth1.10, eth1.11, etc.  I need to be able to respond 
> to each of those interfaces as 10.1.1.1, with (so far) only ICMP and UDP 
> traffic.

i can't *believe* i'm actually going to link to this, but the last
time someone asked something similar (k--so it was over a week ago),
i posted this:

  http://marc.theaimsgroup.com/?l=netfilter&m=110737820716813&w=2

it doesn't exactly match your situation, but the concept is the same--as
long as the machine in question only has to respond to requests you can
MARK the connection as it comes in, and route the reply based on the
mark, which is restored using CONNMARK.

as the OP never responded as to whether this suggestion worked, or ate
his dog, i still have no idea as to whether it works or not; as i never
tried it myself...so YMMV.

as someone else already suggested, you can also do this using NETMAP to
nat entire subnets to something unique.

IIRC--there was someone that asked about this, and i came up with:

  http://marc.theaimsgroup.com/?l=netfilter&m=110027573811157&w=2

again--no idea if this works or not...  but you should be able to get
the general idea of what's involved in this sort of chicanery.

-j

--
"Me lose brain? Uh, oh! Ha ha ha! Why I laugh?"
        --The Simpsons

Re: Multiple subnets of the same IP space, different VLANs, same box...

[Tobias DiPasquale]

On Tue, 15 Feb 2005 17:03:56 -0500, Jason Opperisano <opie@817west.com> wrote:
> i can't *believe* i'm actually going to link to this, but the last
> time someone asked something similar (k--so it was over a week ago),
> i posted this:
> 
>   http://marc.theaimsgroup.com/?l=netfilter&m=110737820716813&w=2
[...] 
> as the OP never responded as to whether this suggestion worked, or ate
> his dog, i still have no idea as to whether it works or not; as i never
> tried it myself...so YMMV.

The contents of that post will work. We do iproute2 policy routing of
that nature around my office all the time with configurations that are
similar and even-more whacked out. ;-)

-- 
[ Tobias DiPasquale ]
0x636f6465736c696e67657240676d61696c2e636f6d