Isolating two network processes on same machine

Isolating two network processes on same machine

[Ole Laursen]

Hi,

We need to test a peer-to-peer network application that is supposed to
be scalable. To that end, we have a FreeBSD box with dummynet and a
small cluster of Linux test machines. The box act as the gateway for
the test machines and delay incoming packets for a while before
throwing them back to the cluster to simulate latency on the Internet.

By letting the test machines think they run on separate subnets, we
have been able to fool them into forwarding their packets to the
FreeBSD gateway even though everyone is connected to the same switch.
This is working fine.

The problem is that we need to run several instances of our network
application on the same test machine since we have too few machines.
But when we create two IP addresses on the same machine with

  ifconfig eth0:0 10.0.0.2 netmask 255.255.255.0 broadcast 10.0.0.255
  ifconfig eth0:1 10.0.1.2 netmask 255.255.255.0 broadcast 10.0.1.255

and start two instances on the same machine with the two IP addresses,
then they communicate directly with each other instead of going
through the FreeBSD gateway. Can anyone see a way to solve this
problem?


(I've CC'ed the other guys in my group.)

-- 
Ole Laursen
http://www.cs.aau.dk/~olau/

Re: Isolating two network processes on same machine

[linux-os]

On Wed, 24 Nov 2004, Ole Laursen wrote:

> Hi,
>
> We need to test a peer-to-peer network application that is supposed to
> be scalable. To that end, we have a FreeBSD box with dummynet and a
> small cluster of Linux test machines. The box act as the gateway for
> the test machines and delay incoming packets for a while before
> throwing them back to the cluster to simulate latency on the Internet.
>
> By letting the test machines think they run on separate subnets, we
> have been able to fool them into forwarding their packets to the
> FreeBSD gateway even though everyone is connected to the same switch.
> This is working fine.
>
> The problem is that we need to run several instances of our network
> application on the same test machine since we have too few machines.
> But when we create two IP addresses on the same machine with
>
>  ifconfig eth0:0 10.0.0.2 netmask 255.255.255.0 broadcast 10.0.0.255
>  ifconfig eth0:1 10.0.1.2 netmask 255.255.255.0 broadcast 10.0.1.255
>
> and start two instances on the same machine with the two IP addresses,
> then they communicate directly with each other instead of going
> through the FreeBSD gateway. Can anyone see a way to solve this
> problem?
>


I was going to say, set the netmask small enough so that both
machines are on different networks and set default routes to
your gateway.... But there is a bug somewhere that doesn't
allow a netmask of anything but 0 in the last byte.

So, just add a host route....

route add -host 10.0.1.2 gw server


>
> (I've CC'ed the other guys in my group.)
>
> -- 
> Ole Laursen
> http://www.cs.aau.dk/~olau/
> -

FYI, probably nobody will admit to it being a bug, but it's
another example of policy spreading throughout the kernel.
If I set the netmask to 0.0.0.0 or 255.255.255.255, and
anything in-between, it should let me....

Cheers,
Dick Johnson
Penguin : Linux version 2.6.9 on an i686 machine (5537.79 BogoMips).
  Notice : All mail here is now cached for review by John Ashcroft.
                  98.36% of all statistics are fiction.

Re: Isolating two network processes on same machine

[Lee Revell]

On Wed, 2004-11-24 at 11:23 -0500, linux-os wrote:
> I was going to say, set the netmask small enough so that both
> machines are on different networks and set default routes to
> your gateway.... But there is a bug somewhere that doesn't
> allow a netmask of anything but 0 in the last byte.
> 

Really?  That would be a horrible bug.  How about some references?

Lee

Re: Isolating two network processes on same machine

[linux-os]

<<< No Message Collected >>>

Re: Isolating two network processes on same machine

[linux-os]

On Wed, 24 Nov 2004, linux-os wrote:

> On Wed, 24 Nov 2004, Lee Revell wrote:
>
>> On Wed, 2004-11-24 at 11:23 -0500, linux-os wrote:
>>> I was going to say, set the netmask small enough so that both
>>> machines are on different networks and set default routes to
>>> your gateway.... But there is a bug somewhere that doesn't
>>> allow a netmask of anything but 0 in the last byte.
>>> 
>> 
>> Really?  That would be a horrible bug.  How about some references?
>> 
>> Lee
>> 
>
> Huh? Try it........
>
> Script started on Wed 24 Nov 2004 12:03:43 PM EST
> # ifconfig eth1 1.2.3.4 netmask 255.255.255.7
> SIOCSIFNETMASK: Invalid argument
> # ifconfig eth1 1.2.3.4 netmask 255.255.255.0
> # exit
>
> Script done on Wed 24 Nov 2004 12:04:42 PM EST
>


Well I guess you can set it to:

eth1      Link encap:Ethernet  HWaddr 00:10:5A:27:7B:16
           inet addr:1.3.4.5  Bcast:1.255.255.255  Mask:255.255.255.252
                                                                    ^^^
           inet6 addr: fe80::210:5aff:fe27:7b16/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:68120 errors:0 dropped:0 overruns:0 frame:0
           TX packets:5 errors:0 dropped:0 overruns:0 carrier:3
           collisions:0 txqueuelen:1000
           RX bytes:4383953 (4.1 Mb)  TX bytes:378 (378.0 b)
           Interrupt:22 Base address:0xbc00


That should ne able to isolate the two machines on the same wire.


Cheers,
Dick Johnson
Penguin : Linux version 2.6.9 on an i686 machine (5537.79 BogoMips).
  Notice : All mail here is now cached for review by John Ashcroft.
                  98.36% of all statistics are fiction.

Re: Isolating two network processes on same machine

[Ole Laursen]

linux-os <linux-os@chaos.analogic.com> writes:

> I was going to say, set the netmask small enough so that both
> machines are on different networks and set default routes to
> your gateway....

Yeah, but that part of it is actually working as long as our processes
are running on different machines. The problem is that on the same
machine e.g. with this configuration

> >  ifconfig eth0:0 10.0.0.2 netmask 255.255.255.0 broadcast 10.0.0.255
> >  ifconfig eth0:1 10.0.1.2 netmask 255.255.255.0 broadcast 10.0.1.255

then the kernel somehow shortcircuits the routing table and doesn't
forward the packets to the default gateway, even though the two
addresses are on different subnets. It probably somehow knows that it
possesses both IPs itself, and then skip any further routing.

So basically, our problem is that the kernel is being too clever. If
we could just dumb it down or trick it somehow...


Thanks for your input,

-- 
Ole Laursen
http://www.cs.aau.dk/~olau/

Re: Isolating two network processes on same machine

[Ross Biro]

> The problem is that we need to run several instances of our network
> application on the same test machine since we have too few machines.
> But when we create two IP addresses on the same machine with
> 

The easiest solution is probably to have the FreeBSD box DNAT the
linux boxes so they don't know they are talking to themselves.  Then
you only need to use 1 ip address per linux box.

    Ross

Re: Isolating two network processes on same machine

[Ole Laursen]

Ross Biro <ross.biro@gmail.com> writes:

> > The problem is that we need to run several instances of our network
> > application on the same test machine since we have too few machines.
> > But when we create two IP addresses on the same machine with
> 
> The easiest solution is probably to have the FreeBSD box DNAT the
> linux boxes so they don't know they are talking to themselves.  Then
> you only need to use 1 ip address per linux box.

Thanks, DNAT seems to be a good solution.

I think we will let the Linux boxes use DNAT to send the packets to
the FreeBSD box and then let that use DNAT to send them back again.
This way we won't have to change our test program, which would be a
bit complicated because the addresses of the peers is an integral part
of the design.

Though it would have been simpler if the kernel supported blindly
forwarding a packet to another host without messing with the IP
destination address, but that does not seem to be the case. It could
have saved us from the double DNAT.

Thanks again,

-- 
Ole Laursen
http://www.cs.aau.dk/~olau/