Two links and a SMTP sever.

Two links and a SMTP sever.

[Ulysses Almeida]

[-- Attachment #1: Type: text/plain, Size: 773 bytes --]

Hi folks,

  I have an unsolved problem with iptables + iproute2 that I can't
figure out how to solve it.

  I explaint it on the attachment (i sent this attachment, because I
don't thin it will cause any trouble, since it's text/plain, if does,
say and I won't send again). I wrote this as a mini-HowTo, and I
pretend to publish this, but I need to solve the problem before doing
this ;)

  The strange thing. I have almost the same scenario on other host,
What differs is the fact that on the other host I have 4 Internet
links, not only two, and everything works as I expected.  I tried the
same solution on another two hosts, and on both I had this releated
problem.

  If some one can help-me track this problem, I thank in advance.

  Regars.

-- 
 - Ulysses Almeida

[-- Attachment #2: lar_with_smtp.txt --]
[-- Type: text/plain, Size: 4109 bytes --]


  I Have the following scenario:

  - One server with two links:
    * First link has dynamic IP (cheap one)
    * Second links has fixed IP (expensive one)

  - Link(1) is used by my intranet users, they use Internet 
  services by this link. To do this, I just setuped Link(1)
  as my default gw. And made my server a linux router.
    # echo 1 > /proc/sys/net/ipv4/ip_forward
    # iptables -t nat -A POSTROUTING -s <intranet_net> -o <eth_link(1)> -j MASQUERADE 
    # ip route add default via <link(1)_gw> dev <eth_link(1)>

  - Link(2) is used by some incomming requests from internet
  to this server. This services are HTTP and SMTP services. 
  To use second link properly, I setuped iproute2
    # echo 100 link2 >> /etc/iproute2/rt_tables
    # ip route add <intranet_net> dev <eth_intranet> src <server_intranet_ip> table link2
    # ip route add <link(2)_net> dev <eth_link(2)> src <server_link(2)_ip> table link2
    # ip route add default via <link(2)_gw> dev <eth_link(2)> table link2
    # ip rule add from <server_link(2)_ip> lookup link2

 	      ______
             |      |	          __________  
             | S    | Link(1)    /	    \ 
             | E  S |<--------->| 	     |
  ________   | R  M |       	|  INTERNET  |
 |INTRANET|->| V  T | Link(2)   | 	     |
 '--------'  | E  P |<--------->| 	     |
             | R    |	         \__________/
             |______|		       	
  	   			      
  
  
  Almost every thing works at this point. All my intranet users, are 
using Link(1) to navigate on Internet, use MSN/ICQ/Jabber, and etc.
  When someone on Internet want's to access my WEB page, they use 
server_link(2)_ip to get into. The same when they want to accesse their
e-mails. If some MTA on internet, needs to deliver some e-mail to my
smtp server, it also uses the server_link(2)_ip. And it's work fine.

  There's only one problem. If my MTA wants to deliver an e-mail, it
tries it by Link(1), what's reasonable, because that's the server default
gateway. But  it's not good, when the link(1) (dynamic IP) is a black 
listed one

  To solve this problem, let's play with iptables and iproute2 a little more.

  Every outgoing packet, with destination port set to 25, I will mark.
    # iptables -t mangle -A OUTPUT -p tcp --dport 25 -j MARK --set-mark 0x02

  Now I can say to iproute2 use the right routing table.
    # ip rule add fwmark 0x02 lookup link2

  Still not working, if i log outgoing packts, with destination port 25,
I see SMTP packets going through eth_link(2), but with server_link(1)_ip.
Let's handle those packets a little more.
    # iptables -t nat -I POSTROUTING -o <eth_link(2)> -j SNAT --to-source <server_link(2)_ip>

  Yes, now I'm sure every outgoing trafic through eth_link(2) is going
with the correct IP.

  == UNSOLVED PROBLEM ==

  But, argh, it still not working. I tried to track connections:

  # iptables -t mangle -m tcp -I OUTPUT -p tcp --dport 25 -j LOG --log-level 3 --log-prefix 'mangle OUTPUT: '
  # iptables -t nat -m tcp -I POSTROUTING -p tcp --dport 25 -j LOG --log-level 3 --log-prefix 'nat POSTROUTING: 
  # iptables -t mangle -m tcp -I PREROUTING -p tcp --sport 25 -j LOG --log-level 3 --log-prefix 'mangle PREROUTING: '
  # iptables -t nat -m tcp -I PREROUTING -p tcp --sport 25 -j LOG --log-level 3 --log-prefix 'nat PREROUTING: '
  # iptables -t mangle -m tcp -I INPUT -p tcp --sport 25 -j LOG --log-level 3 --log-prefix 'mangle INPUT: '
  # iptables -t filter -m tcp -I INPUT -p tcp --sport 25 -j LOG --log-level 3 --log-prefix 'filter INPUT: '


  To simulate, I started a telnet some_smtp_server 25. I could see:

  - packet on output mangle -> postrouting nat.
  - pachet on prerouting mangle.

  After packet appears on mangle's prerouting, it disappers? I can't
understando why.
  How can I debug this?
  Any one knows what can causinig it?


  I'm using:
# uname -r
2.6.5-63255U10_3cl    (I think is something near 2.6.9, with conectiva patchs)
# iptables -V
iptables v1.2.11
# ip -V  
ip utility, iproute2-ss020116

Re: Two links and a SMTP sever.

[Ulysses Almeida]

Thanks for the answer....


On Tue, 18 Jan 2005 15:54:39 -0500 (EST), Samuel Jean
<sj-netfilter@cookinglinux.org> wrote:
> On Tue, January 18, 2005 8:39 am, Ulysses Almeida said:
>
> (After looking attached file)
> Did you enable NAT of local connection in your kernel ?
> To be honest, I don't know anything about Lartc. And can't tell you
> how it should behave (my guess is it should pickup the right IP, of course)
> 
> However, the (ugly) SNAT work-around should work with
> NAT of local connection enabled.

  Hmm. the NAT/SNAT is working fine, when I recive the SMTP answer
(LOG on PREROUTING and INPUT of filter, nat and mangle) the packet
arrives with the correct ip on dst addr. But I can only see it, on
mangle PREROUTING, but can't on other hooks. I don't know if it is
related with some sanity check or checksum....

  That's another doubt, dos nat PREROUTING called with syn+ack
packets, or only with sym packets?

  I have these modules loaded:
ipt_LOG                 5760  7 
ipt_mark                1792  0 
ipt_MARK                2176  1 
iptable_mangle          2944  1 
ipt_state               2048  8 
iptable_filter          2944  1 
ipt_MASQUERADE          3840  4 
iptable_nat            23980  2 ipt_MASQUERADE
ip_conntrack           33072  3 ipt_state,ipt_MASQUERADE,iptable_nat
ip_tables              17920  8
ipt_LOG,ipt_mark,ipt_MARK,iptable_mangle,ipt_state,iptable_filter,ipt_MASQUERADE,iptable_nat

  
> 
> >
> >   I explaint it on the attachment (i sent this attachment, because I
> > don't thin it will cause any trouble, since it's text/plain, if does,
> > say and I won't send again).
> 
> Please put it in the body so we can comment some part.
> 
Ok, I used to use MUA that shows-me the text/plain attachments as the
e-mail body (like mutt). But now I see it won't happen all the time.
My mistake, sorry.

-- 
 - Ulysses Almeida

Re: Two links and a SMTP sever.

[Samuel Jean]

On Tue, January 18, 2005 8:39 am, Ulysses Almeida said:
> Hi folks,

Hi Ulysses!

>
>   I have an unsolved problem with iptables + iproute2 that I can't
> figure out how to solve it.

(After looking attached file)
Did you enable NAT of local connection in your kernel ?
To be honest, I don't know anything about Lartc. And can't tell you
how it should behave (my guess is it should pickup the right IP, of course)

However, the (ugly) SNAT work-around should work with
NAT of local connection enabled.

>
>   I explaint it on the attachment (i sent this attachment, because I
> don't thin it will cause any trouble, since it's text/plain, if does,
> say and I won't send again).

Please put it in the body so we can comment some part.

>   If some one can help-me track this problem, I thank in advance.
>
>   Regars.
>
> --
>  - Ulysses Almeida
>

HTH,

Samuel

Re: Two links and a SMTP sever.

[Ulysses Almeida]

  How can I debug, what happens after mangle PREROUTING. Can I do it
only for packets with src port set to 25? to get a "clear" output?

On Tue, 18 Jan 2005 16:17:42 -0500 (EST), Samuel Jean
<sj-netfilter@cookinglinux.org> wrote:
> >   That's another doubt, dos nat PREROUTING called with syn+ack
> > packets, or only with sym packets?
> 
> the nat table is special, it only sees the first packet
> of a connection.
> 
> All other subsequent packet of this connection get verdict via
> the conntrack. (bypass NAT chains)
> 
  So this conntrack entry is created while my syn packet is going
out.... Can something wrong happening here, and conntrack gives the
wrong veridict?

> 
> Samuel
> 
> 
-- 
 - Ulysses Almeida

Re: Two links and a SMTP sever.

[Samuel Jean]

On Tue, January 18, 2005 3:52 pm, Ulysses Almeida said:
>   Hmm. the NAT/SNAT is working fine, when I recive the SMTP answer
> (LOG on PREROUTING and INPUT of filter, nat and mangle) the packet
> arrives with the correct ip on dst addr. But I can only see it, on
> mangle PREROUTING, but can't on other hooks. I don't know if it is
> related with some sanity check or checksum....
>
>   That's another doubt, dos nat PREROUTING called with syn+ack
> packets, or only with sym packets?

the nat table is special, it only sees the first packet
of a connection.

All other subsequent packet of this connection get verdict via
the conntrack. (bypass NAT chains)

>
> --
>  - Ulysses Almeida
>
>

Samuel

Re: Two links and a SMTP sever.

[Samuel Jean]

Ulysses Almeida wrote:
>   How can I debug, what happens after mangle PREROUTING. Can I do it
> only for packets with src port set to 25? to get a "clear" output?

If you are willing to debug, use TRACE target from pom. (at least 2 
patches are needed).

> On Tue, 18 Jan 2005 16:17:42 -0500 (EST), Samuel Jean
> <sj-netfilter@cookinglinux.org> wrote:
>>All other subsequent packet of this connection get verdict via
>>the conntrack. (bypass NAT chains)
>>
> 
>   So this conntrack entry is created while my syn packet is going
> out.... Can something wrong happening here, and conntrack gives the
> wrong veridict?

As I said, did you enable this ? :

  CONFIG_IP_NF_NAT_LOCAL

Cheers,

Samuel

Re: Two links and a SMTP sever.

[Jason Opperisano]

On Tue, 2005-01-18 at 15:52, Ulysses Almeida wrote:
>   That's another doubt, dos nat PREROUTING called with syn+ack
> packets, or only with sym packets?

nat PREROUTING sees NEW packets, without regard to TCP flags.

-j
--
"Oh, people can come up with statistics to prove anything, Kent. 14%
 of people know that."
	--The Simpsons

Re: Two links and a SMTP sever.

[Ulysses Almeida]

On Tue, 18 Jan 2005 17:02:31 -0500, Samuel Jean
<sj-netfilter@cookinglinux.org> wrote:
> 
> As I said, did you enable this ? :
> 
>   CONFIG_IP_NF_NAT_LOCAL
> 

Yes CONFIG_IP_NF_NAT_LOCAL=y

  I did put some -j LOG on target host. Tried a telnet to cited host
on port 25. On
target's host everything looks right. I get the packets from the right
ip (link(2)). But it still disapper after mangles PREROUTING. Maybe
i'm missing something on second routing table.... I'll try to find
out.

  It's strange, because I have other host, with this really same
solution, and everything works fine.


-- 
 - Ulysses Almeida

Re: Two links and a SMTP sever.

[Ulysses Almeida]

  Some time ago, I wrote this e-mail below.  A lot of people tried to
help-me (thanks guys), but I couldn't find the problem. A week ago,
with help from someone on IRC I solved my problem (simple one). And I
finished my MiniHowTo.
http://munky.maluco.com.br/lar_with_smtp.html
  Maybe it won't tell you something new. But it's shorter do read
instead the whole LARTC documentation.
  This miniHowTo shows how to use iproute2 with iptables mark rule. I
think it's nice. If someone has something to say about it (in
private), I would thanks.

  PS: The problem was just Reverse Path Filtering enabled.

On Tue, 18 Jan 2005 10:39:04 -0300, Ulysses Almeida
<ulysses.almeida@gmail.com> wrote:
> Hi folks,
> 
>   I have an unsolved problem with iptables + iproute2 that I can't
> figure out how to solve it.
> 
>   I explaint it on the attachment (i sent this attachment, because I
> don't thin it will cause any trouble, since it's text/plain, if does,
> say and I won't send again). I wrote this as a mini-HowTo, and I
> pretend to publish this, but I need to solve the problem before doing
> this ;)
> 
>   The strange thing. I have almost the same scenario on other host,
> What differs is the fact that on the other host I have 4 Internet
> links, not only two, and everything works as I expected.  I tried the
> same solution on another two hosts, and on both I had this releated
> problem.
> 
>   If some one can help-me track this problem, I thank in advance.
> 
>   Regars.
> 
> --
>  - Ulysses Almeida
> 
> 
> 


-- 
 - Ulysses Almeida