Redirect mirrored traffic to userspace app.

Redirect mirrored traffic to userspace app.

[Mateus Caruccio]

Hello everyone,

first of all, I'm not an iptables guru (that is why I'm here asking
you for help:)
Also, sorry for my poor english skills.

Well, at office we have 2 major environments: development and production.
In order to test our software in a more realistic condition, we'd like
to run it in production, but without affectig client application
traffic.

So, what I think is a nice solution is to setup a switch port mirror
to copy all traffic going to production server into our development server.

That's how our network topology looks like (simplified):


                        +-------------------+
+------+   [REQUEST]    |      SWITCH       |
|CLIENT|--------------->|PORT1  PORT2->PORT3|
+------+                +-------------------+
                                 |       |
+-----------------+   [REQUEST]  |       | [MIRRORED REQ]  +------------------+
|PROD_SRV:eth0/IP1|<-------------+       +---------------->|DEVEL_SRV:eth0/IP2|
+-----------------+                                        +------------------+

Thanks to this damn GUI world, probably this ascii-art is ruined. Here
is another one:

CLIENT (send req)
   |
   |
   V
 PORT1
SWITCH PORT2---->PROD_SRV:eth0/IP1 (recv req)
 PORT3
   |
   |
   V
DEVEL_SRV:eth0/IP2 (recv mirrored req)


All traffic is UDP port 2077 only. I can not change any config on PROD_SRV.

Our DEVEL_SRV should receive exactly the same packet PROD_SRV
receives, but with destination address modified
so it can reach our userspace application.

tcpdump shows that all requests are being properly mirrored from PORT2 to PORT3.
ifconfig eth0 on DEVEL_SRV does not present any increment on RX
statistics, but if I "ifconfig eth0 promisc", it starts to increment
RX.

I've tried to "DNAT" it, but without success:

 root@DEVEL_SRV# iptables -t nat -A PREROUTING -p udp --dport 2077 -j
DNAT --to-destination $IP2

From DEVEL_SRV point of view, it receives a legitime request.
I've setup a DROP rule so responses do not interfer on client's requests.

If I'm not clear, please fell free to ask anything.

Thanks for helping,

Mateus Caruccio


-- 
# ################ VOTE NULO ################
# Mateus de Oliveira Caruccio <mateus at caruccio dot com>
# Old programmers never die.  They just branch to another namespace

Re: Redirect mirrored traffic to userspace app.

[Jan Engelhardt]

On Wednesday 2010-10-20 23:58, Mateus Caruccio wrote:
>
>Our DEVEL_SRV should receive exactly the same packet PROD_SRV
>receives, but with destination address modified
>so it can reach our userspace application.
>I've tried to "DNAT" it, but without success:

Because DNAT is factually wrong, as it does not make a copy. Instead,
use -j TEE on the host where the copy is to be made, and use either a
packet socket (tcpdump uses such) or local delivery routing (`ip route
add local dstaddrinpacket` and so).

>tcpdump shows that all requests are being properly mirrored from PORT2 to PORT3.
>ifconfig eth0 on DEVEL_SRV does not present any increment on RX
>statistics, but if I "ifconfig eth0 promisc", it starts to increment
>RX.

Forget about ifconfig. People should use modern tools like ip.

Re: Redirect mirrored traffic to userspace app.

[Mateus Caruccio]

Unfortunately I'm not able to run any command on production servers :(
All I could get was a physical port mirror, using an ethernet cable in
the switch.

Since I do have a copy of the packet on my own box, why can't I change
it, inside my own box, to match my own IP addr and route it to my own
userspace app (no matter interface)?
For me this should be simple:

1 - a pkt reached my interface with prod-server-dst-addr;
2 - before it get routed, rewrite its dst-addr to my interface's addr;
3 - let the kernel delivery it to my interface's local addr:port.

thanx.

On Wed, Oct 20, 2010 at 8:21 PM, Jan Engelhardt <jengelh@medozas.de> wrote:
>
> On Wednesday 2010-10-20 23:58, Mateus Caruccio wrote:
>>
>>Our DEVEL_SRV should receive exactly the same packet PROD_SRV
>>receives, but with destination address modified
>>so it can reach our userspace application.
>>I've tried to "DNAT" it, but without success:
>
> Because DNAT is factually wrong, as it does not make a copy. Instead,
> use -j TEE on the host where the copy is to be made, and use either a
> packet socket (tcpdump uses such) or local delivery routing (`ip route
> add local dstaddrinpacket` and so).
>
>>tcpdump shows that all requests are being properly mirrored from PORT2 to PORT3.
>>ifconfig eth0 on DEVEL_SRV does not present any increment on RX
>>statistics, but if I "ifconfig eth0 promisc", it starts to increment
>>RX.
>
> Forget about ifconfig. People should use modern tools like ip.
>



-- 
# ################ VOTE NULO ################
# Mateus de Oliveira Caruccio <mateus at caruccio dot com>
# Old programmers never die.  They just branch to another namespace

Re: Redirect mirrored traffic to userspace app.

[Jan Engelhardt]

On Thursday 2010-10-21 03:57, Mateus Caruccio wrote:

>Unfortunately I'm not able to run any command on production servers :(
>All I could get was a physical port mirror, using an ethernet cable in
>the switch.
>
>Since I do have a copy of the packet on my own box, why can't I change
>it, inside my own box, to match my own IP addr and route it to my own
>userspace app (no matter interface)?

Because you can route it to the userspace program without having to 
change the address.

>For me this should be simple:
>
>1 - a pkt reached my interface with prod-server-dst-addr;
>2 - before it get routed, rewrite its dst-addr to my interface's addr;
>3 - let the kernel delivery it to my interface's local addr:port.

Re: Redirect mirrored traffic to userspace app.

[Grant Taylor]

On 10/20/10 16:58, Mateus Caruccio wrote:
> Thanks to this damn GUI world, probably this ascii-art is ruined.

Nope.  Using text mode email and a fixed width font (my default) it came 
through just fine.  ;-)

> All traffic is UDP port 2077 only. I can not change any config on 
> PROD_SRV.

Ok...  UDP will take care of the protocol handshake issue that you have 
with TCP.  That means that your dev server can truly receive the request 
and not have borked connections.

> Our DEVEL_SRV should receive exactly the same packet PROD_SRV 
> receives, but with destination address modified so it can reach our 
> userspace application.

On the surface I agree.  Just under the surface I wonder if you really 
need to alter the destination IP or not.  (More in a bit.)

> I've tried to "DNAT" it, but without success:

Rather than DNAT, why not try simple routing.  (More in a bit.)

> From DEVEL_SRV point of view, it receives a legitime request.

I assume that you are looking at the traffic (via TCPDump or something 
like it) and deciding that it's legitimate b/c you haven't gotten 
traffic to make it in to the service yet.

> I've setup a DROP rule so responses do not interfer on client's 
> requests.

I'd wonder if you really want to DROP the traffic or redirect the 
replies so that you could test (analyze) that as well.  (More in a bit.)

> If I'm not clear, please fell free to ask anything.

Na, I think I understand what you are trying to do.  -  Set up something 
so that you can test dev code under (possibly a portion of) live traffic 
load.

For giggles I'd try having the dev server and prod server be in 
different subnets.  That will allow you to bind the prod server's IP to 
a different nic (dummy or something like it) and enable routing.  That 
way when the traffic comes in to the eth0 interface on the dev box, it 
would be routed to the proper nic where your software is bound to the 
real IP.  There by doing away with any hacks needed to get the traffic 
in to your service.

As a bonus, you should have a better emulation of the service between 
the prod and dev servers, there by making it a better test and allowing 
service config migration between the two boxen.

Just something that I'd try.

As far as the DROP rule, I'd rather set the default gateway on the dev 
box so that it would go out a different nic (physical or virtual) such 
that your service could send reply packets that could then be analyzed 
elsewhere.

> Thanks for helping,

n/p

Good luck.

Please provide follow up and let us know what worked for you.



Grant. . . .