Traffic Mirroring for Debugging

Traffic Mirroring for Debugging

[Volkan YAZICI]

Hi,

In one of our servers, I want to debug a network server daemon. The
problem is I don't have luxury for a downtime or to iterrupt related
server's network traffic. Current routing structure looks like below.

  VPN Switch (192.168.1.1) -> Server Machine (192.168.1.2)

To debug the related server daemon, I planned to redirect the network
traffic to a second sandbox server. Planned routing structure is:

  VPN Switch (192.168.1.1)
  |
  +-> Temporary Linux Machine (192.168.1.10)
      ||
      |+-> Server Machine (192.168.1.2)
      +--> Sandbox Server Machine (192.168.1.20)

For test purposes, I configured a client to connect to 192.168.1.10
(instead of actual server, 192.168.1.2). But I couldn't manage to
redirect incoming traffic of 192.168.1.10 to 192.168.1.2 and
192.168.2.20 machines. How can I get such a mirrored redirection?

Furthermore, I first considered using DNAT of iptables, but learnt that
it doesn't support multiple destinations without load-balancing. Anyway,
I wanted to give it a try for just a single machine. I connected to 1.10
machine and typed below iptables command:

  # iptables -t nat -A PREROUTING -d 192.168.1.10 -j DNAT --to-destination 192.168.1.20
  # iptables -L -n -t nat
  Chain PREROUTING (policy ACCEPT)
  target     prot opt source               destination
  DNAT       0    --  0.0.0.0/0            192.168.1.10 to:192.168.1.20

  Chain POSTROUTING (policy ACCEPT)
  target     prot opt source               destination

  Chain OUTPUT (policy ACCEPT)
  target     prot opt source               destination

And started a netcat listener on 1.20 machine:

  # nc -l -p 2310

Then tried to connect to the started nc listener from 1.10 machine:

  # nc -vvv 192.168.1.10 2310
  192.168.1.10: inverse host lookup failed: Unknown host
  (UNKNOWN) [192.168.1.10] 2310 (?) : Connection refused
   sent 0, rcvd 0

Gosh! Even couldn't manage to make DNAT for a single address work
properly. Any helps will be really really appreciated.


Regards.

Re: Traffic Mirroring for Debugging

[Grant Taylor]

On 8/22/2008 2:57 AM, Volkan YAZICI wrote:
> Gosh! Even couldn't manage to make DNAT for a single address work 
> properly. Any helps will be really really appreciated.

It won't help with the packet duplication, but I think it is related.  I 
think you will need to SNAT the traffic as it leaves 1.10 so that 1.2 / 
1.20 will reply back to 1.10 rather than directly back to the client.  I 
think that 1.2 / 1.20 is replying directly back to the client and the 
client is going WTF???  Why is 1.2 / 1.20 replying to me?  I have not 
started a connection with them.

As far as packet duplication, I'd start with ulogd or something like 
that.  I'm betting you are going to have to pass the packets to user 
space for the duplication.



Grant. . . .

Re: Traffic Mirroring for Debugging

[Volkan YAZICI]

On Fri, 22 Aug 2008, Grant Taylor <gtaylor@riverviewtech.net> writes:
> It won't help with the packet duplication, but I think it is related.  I think
> you will need to SNAT the traffic as it leaves 1.10 so that 1.2 / 1.20 will
> reply back to 1.10 rather than directly back to the client.  I think that 1.2 /
> 1.20 is replying directly back to the client and the client is going WTF???  Why
> is 1.2 / 1.20 replying to me?  I have not started a connection with
> them.

In the actual setup, original packets are received from a mobile device
through a VPN gateway. Therefore, endpoint server replies back to this
client, instead of the gateway packages received from.

> As far as packet duplication, I'd start with ulogd or something like
> that.  I'm betting you are going to have to pass the packets to user
> space for the duplication.

I'll check out ulogd. Thanks.


Regards.

Re: Traffic Mirroring for Debugging

[Eray Aslan]

On 23.08.2008 09:36, Volkan YAZICI wrote:
[...]
>> As far as packet duplication, I'd start with ulogd or something like
>> that.  I'm betting you are going to have to pass the packets to user
>> space for the duplication.
> 
> I'll check out ulogd. Thanks.

Also check the TEE target.  I am not sure what the current status is though:

http://www.gossamer-threads.com/lists/iptables/devel/68781
http://dev.computergmbh.de/files/xtables/

-- 
Eray