Re: Redirect mirrored traffic to userspace app. [RESOLVED]

Re: Redirect mirrored traffic to userspace app. [RESOLVED]

[Mateus Caruccio]

In the end, we came with a very simple solution (no iptables/ip needed):

From DEVEL_SRV we ran:

# ifconfig eth1 PROD_SRV_IP netmask PROD_SRV_NETMASK promisc -arp hw
ether PROD_SRV_HWADDR up

That is it !

All traffic targeted to PROD_SRV_IP is now being accepted by our
mirrored eth1. Since this is an interface aimed to tests only, no
matter what's being accepted.

Thanks Jan and Grant for your suggestions and explanations. That worths a lot!

regards,
Mateus Caruccio


On Thu, Oct 21, 2010 at 1:02 PM, Grant Taylor <gtaylor@riverviewtech.net> wrote:
> On 10/20/10 16:58, Mateus Caruccio wrote:
>>
>> Thanks to this damn GUI world, probably this ascii-art is ruined.
>
> Nope.  Using text mode email and a fixed width font (my default) it came
> through just fine.  ;-)
>
>> All traffic is UDP port 2077 only. I can not change any config on
>> PROD_SRV.
>
> Ok...  UDP will take care of the protocol handshake issue that you have with
> TCP.  That means that your dev server can truly receive the request and not
> have borked connections.
>
>> Our DEVEL_SRV should receive exactly the same packet PROD_SRV receives,
>> but with destination address modified so it can reach our userspace
>> application.
>
> On the surface I agree.  Just under the surface I wonder if you really need
> to alter the destination IP or not.  (More in a bit.)
>
>> I've tried to "DNAT" it, but without success:
>
> Rather than DNAT, why not try simple routing.  (More in a bit.)
>
>> From DEVEL_SRV point of view, it receives a legitime request.
>
> I assume that you are looking at the traffic (via TCPDump or something like
> it) and deciding that it's legitimate b/c you haven't gotten traffic to make
> it in to the service yet.
>
>> I've setup a DROP rule so responses do not interfer on client's requests.
>
> I'd wonder if you really want to DROP the traffic or redirect the replies so
> that you could test (analyze) that as well.  (More in a bit.)
>
>> If I'm not clear, please fell free to ask anything.
>
> Na, I think I understand what you are trying to do.  -  Set up something so
> that you can test dev code under (possibly a portion of) live traffic load.
>
> For giggles I'd try having the dev server and prod server be in different
> subnets.  That will allow you to bind the prod server's IP to a different
> nic (dummy or something like it) and enable routing.  That way when the
> traffic comes in to the eth0 interface on the dev box, it would be routed to
> the proper nic where your software is bound to the real IP.  There by doing
> away with any hacks needed to get the traffic in to your service.
>
> As a bonus, you should have a better emulation of the service between the
> prod and dev servers, there by making it a better test and allowing service
> config migration between the two boxen.
>
> Just something that I'd try.
>
> As far as the DROP rule, I'd rather set the default gateway on the dev box
> so that it would go out a different nic (physical or virtual) such that your
> service could send reply packets that could then be analyzed elsewhere.
>
>> Thanks for helping,
>
> n/p
>
> Good luck.
>
> Please provide follow up and let us know what worked for you.
>
>
>
> Grant. . . .
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
# ################ VOTE NULO ################
# Mateus de Oliveira Caruccio <mateus at caruccio dot com>
# Old programmers never die.  They just branch to another namespace

Re: Redirect mirrored traffic to userspace app. [RESOLVED]

[Grant Taylor]

On 10/21/10 11:42, Mateus Caruccio wrote:
> In the end, we came with a very simple solution (no iptables/ip 
> needed):
> 
>    # ifconfig eth1 PROD_SRV_IP netmask PROD_SRV_NETMASK promisc -arp hw 
> ether PROD_SRV_HWADDR up

That looks like you bound it to eth1 when you were talking about eth0 
before.

Are you using eth0 at all?  Or does the box have two nics (0 & 1) in the 
same subnet?

> That is it !

Simple solutions are nice.

> All traffic targeted to PROD_SRV_IP is now being accepted by our 
> mirrored eth1. Since this is an interface aimed to tests only, no 
> matter what's being accepted.

Ok.

Are you accessing (for management) the dev server across the network at 
all?  Or is everything done on console?

> Thanks Jan and Grant for your suggestions and explanations. That 
> worths a lot!

You are welcome.  I'm glad that you got things working, especially as 
simple as you did.



Grant. . . .

Re: Redirect mirrored traffic to userspace app. [RESOLVED]

[Jan Engelhardt]

On Thursday 2010-10-21 18:42, Mateus Caruccio wrote:

>In the end, we came with a very simple solution (no iptables/ip needed):
>
>>From DEVEL_SRV we ran:
>
># ifconfig eth1 PROD_SRV_IP netmask PROD_SRV_NETMASK promisc -arp hw
>ether PROD_SRV_HWADDR up
>
>That is it !
>
>All traffic targeted to PROD_SRV_IP is now being accepted by our
>mirrored eth1. Since this is an interface aimed to tests only, no
>matter what's being accepted.

It still looks wrong though. When using TEE, no expensive promiscous 
mode is required, nor are static ARP entries.

Re: Redirect mirrored traffic to userspace app. [RESOLVED]

[Mateus Caruccio]

On Thu, Oct 21, 2010 at 3:32 PM, Grant Taylor <gtaylor@riverviewtech.net> wrote:
> On 10/21/10 11:42, Mateus Caruccio wrote:
>>
>> In the end, we came with a very simple solution (no iptables/ip needed):
>>
>>   # ifconfig eth1 PROD_SRV_IP netmask PROD_SRV_NETMASK promisc -arp hw
>> ether PROD_SRV_HWADDR up
>
> That looks like you bound it to eth1 when you were talking about eth0
> before.
>
> Are you using eth0 at all?  Or does the box have two nics (0 & 1) in the
> same subnet?
>

Sorry for that :P

In fact eth0 is attached to port mirror, while eth1 is being used to
management (this is the network we use for everyday work).
So, where you read "ifconfig eth1..." it is actually "ifconfig eth0...".

>> That is it !
>
> Simple solutions are nice.

Agree. Less is more.

>
>> All traffic targeted to PROD_SRV_IP is now being accepted by our mirrored
>> eth1. Since this is an interface aimed to tests only, no matter what's being
>> accepted.
>
> Ok.
>
> Are you accessing (for management) the dev server across the network at all?

Yes, through eth1.

>  Or is everything done on console?
>
>> Thanks Jan and Grant for your suggestions and explanations. That worths a
>> lot!
>
> You are welcome.  I'm glad that you got things working, especially as simple
> as you did.
>
>
>
> Grant. . . .
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
# ################ VOTE NULO ################
# Mateus de Oliveira Caruccio <mateus at caruccio dot com>
# Old programmers never die.  They just branch to another namespace

Re: Redirect mirrored traffic to userspace app. [RESOLVED]

[Mateus Caruccio]

On Thu, Oct 21, 2010 at 3:41 PM, Jan Engelhardt <jengelh@medozas.de> wrote:
> On Thursday 2010-10-21 18:42, Mateus Caruccio wrote:
>
>>In the end, we came with a very simple solution (no iptables/ip needed):
>>
>>>From DEVEL_SRV we ran:
>>
>># ifconfig eth1 PROD_SRV_IP netmask PROD_SRV_NETMASK promisc -arp hw
>>ether PROD_SRV_HWADDR up
>>
>>That is it !
>>
>>All traffic targeted to PROD_SRV_IP is now being accepted by our
>>mirrored eth1. Since this is an interface aimed to tests only, no
>>matter what's being accepted.
>
> It still looks wrong though. When using TEE, no expensive promiscous
> mode is required, nor are static ARP entries.

As I said, I do not have access/permission to run anything in our
production servers (our admins are a "little" paranoid :)
Also, since port mirror is mirroring only those specific proto:port
packets, I dont think that would cause any performance penalty.

Anyway, this is not beautiful as it could, but works for our needs.
Again, thanks for support. I will keep looking for a more
straight/clean solution.
I'm not a protocol/kernel specialist, so this is a good challenge.

Mateus.


>



-- 
# ################ VOTE NULO ################
# Mateus de Oliveira Caruccio <mateus at caruccio dot com>
# Old programmers never die.  They just branch to another namespace

Re: Redirect mirrored traffic to userspace app. [RESOLVED]

[Grant Taylor]

On 10/21/10 12:41, Jan Engelhardt wrote:
> It still looks wrong though. When using TEE, no expensive promiscous 
> mode is required, nor are static ARP entries.

It was my (mis)understanding that the OP couldn't (for what ever reason) 
make any changes to the production box.

So, I agree that TEE would have worked, but it just didn't fit in the 
paradigm that the OP was working with in.  (File it away for later though.)



Grant. . . .

Re: Redirect mirrored traffic to userspace app. [RESOLVED]

[Grant Taylor]

On 10/21/10 12:56, Mateus Caruccio wrote:
> As I said, I do not have access/permission to run anything in our 
> production servers (our admins are a "little" paranoid :) Also, since 
> port mirror is mirroring only those specific proto:port packets, I 
> dont think that would cause any performance penalty.

(I've not used SPAN / port mirroring in a long time.)  Does the SPAN 
truly mirror select protocols (UDP) to a given port (2077)?  Or does 
SPAN mirror all traffic to the switch port?

*chuckle*

I've been on both sides of the paranoia.  Usually it's warranted for 
security / stability.  (Usually)

> Anyway, this is not beautiful as it could, but works for our needs. 
> Again, thanks for support. I will keep looking for a more 
> straight/clean solution.

Honestly, I don't think this solution is that unclean, at least from a 
host point of view.  The only dirty part of this I see is the fact that 
you have an IP / MAC duplication on the network.  However, said 
duplication is isolated by a SPAN configuration in a switch.  So, it's 
not really bad, just something to be mindful of.

I don't know how temporary your dev server is, but I've had 5+ year old 
temp installs break when the prod server was replaced, thus changing the 
MAC address.  Just something else to be mindful of.

> I'm not a protocol/kernel specialist, so this is a good challenge.

Challenges can be fun and frustrating.



Grant. . . .

Re: Redirect mirrored traffic to userspace app. [RESOLVED]

[Mateus Caruccio]

On Thu, Oct 21, 2010 at 4:11 PM, Grant Taylor <gtaylor@riverviewtech.net> wrote:
> On 10/21/10 12:56, Mateus Caruccio wrote:
>>
>> As I said, I do not have access/permission to run anything in our
>> production servers (our admins are a "little" paranoid :) Also, since port
>> mirror is mirroring only those specific proto:port packets, I dont think
>> that would cause any performance penalty.
>
> (I've not used SPAN / port mirroring in a long time.)  Does the SPAN truly
> mirror select protocols (UDP) to a given port (2077)?  Or does SPAN mirror
> all traffic to the switch port?

Is, it does mirror based on proto/port.
In fact it's mirroring only inbound traffic! Amazing, huh?
I do not known what equipment it is, but seems to be very robust.

>
> *chuckle*
>
> I've been on both sides of the paranoia.  Usually it's warranted for
> security / stability.  (Usually)
>
>> Anyway, this is not beautiful as it could, but works for our needs. Again,
>> thanks for support. I will keep looking for a more straight/clean solution.
>
> Honestly, I don't think this solution is that unclean, at least from a host
> point of view.  The only dirty part of this I see is the fact that you have
> an IP / MAC duplication on the network.  However, said duplication is
> isolated by a SPAN configuration in a switch.  So, it's not really bad, just
> something to be mindful of.
>
> I don't know how temporary your dev server is, but I've had 5+ year old temp
> installs break when the prod server was replaced, thus changing the MAC
> address.  Just something else to be mindful of.

That's not a problem now. Our tests will last for 3-6 days only.


Mateus.

>
>> I'm not a protocol/kernel specialist, so this is a good challenge.
>
> Challenges can be fun and frustrating.
>
>
>
> Grant. . . .
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
# ################ VOTE NULO ################
# Mateus de Oliveira Caruccio <mateus at caruccio dot com>
# Old programmers never die.  They just branch to another namespace

Re: Redirect mirrored traffic to userspace app. [RESOLVED]

[Jan Engelhardt]

On Thursday 2010-10-21 19:56, Mateus Caruccio wrote:
>>>>From DEVEL_SRV we ran:
>>>
>>># ifconfig eth1 PROD_SRV_IP netmask PROD_SRV_NETMASK promisc -arp hw
>>>ether PROD_SRV_HWADDR up
>>>
>>>That is it !
>>>
>>>All traffic targeted to PROD_SRV_IP is now being accepted by our
>>>mirrored eth1. Since this is an interface aimed to tests only, no
>>>matter what's being accepted.
>>
>> It still looks wrong though. When using TEE, no expensive promiscous
>> mode is required, nor are static ARP entries.
>
>As I said, I do not have access/permission to run anything in our
>production servers (our admins are a "little" paranoid :)

Right. But what I thought of is that you put the teeing host
_in front of_ the prod server.