Poor upload speed with NAT

Poor upload speed with NAT

[Steve Brown]

I have a linux box acting as a router/gateway between the internet, a
public network and a private LAN. We are using iptables to SNAT our
LAN. The setup for this is quite simple:

IPTABLES=`which iptables`
LAN=eth0
WAN=eth1
PUB=eth2
LAN_BLOCK=192.168.16.0/24
NAT_SOURCE=1.2.3.4

$IPTABLES -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A FORWARD -i $LAN -o $WAN -m state --state NEW -j ACCEPT
$IPTABLES -A FORWARD -i $LAN -o $PUB -m state --state NEW -j ACCEPT
$IPTABLES -A FORWARD -i $PUB -o $LAN -m state --state NEW -j ACCEPT
$IPTABLES -t nat -A POSTROUTING -o $WAN -s $LAN_BLOCK -j SNAT
--to-source $NAT_SOURCE

With these rules, I get great download performance, regardless of
protocol (sftp, ftp, http, etc).

Uploads, however, are a different story. I believe that the source of
this issue is the NATing because, if I upload to the same server from
a box that isn't being NATed (e.g. in the public block), upload speeds
are an order of magnitude greater when not being NATed, regardless of
protocol. For example, if I upload a file from one of our public
servers to a remote location, it will upload at about 500Kb/s. The
same upload to the same location with within the LAN won't see more
than 60Kb/s.

I'm not doing any kind of fancy traffic control or anything like that.
I would expect to see some slowdown because of the NAT (for example,
downloads are 600Kb/s vs 500Kb/s through the NAT), but this kind of
slowdown seems a bit ridiculous.

I'm running a custom 2.6.30.3 kernel.  I've also installed
conntrack-tools thinking that my connection table might be getting
full.  However, according to conntrack -S:

entries                 455
searched                84375
found                   10970723
new                     242106
invalid                 15039
ignore                  53722
delete                  241522
delete_list             225919
insert                  226270
insert_failed           0
drop                    0
early_drop              0
icmp_error              5741
expect_new              28
expect_create           43
expect_delete           34

So I am well below the table max:

# sysctl net.netfilter.nf_conntrack_max
net.netfilter.nf_conntrack_max = 65536

Is this kind of performance drop to be expected w/ netfilter and
iptables?  Any suggestions on things I can do to increase performance?

Re: Poor upload speed with NAT

[Amos Jeffries]

On Mon, 10 Aug 2009 16:43:14 -0500, Steve Brown <sbrown25@gmail.com> wrote:
> I have a linux box acting as a router/gateway between the internet, a
> public network and a private LAN. We are using iptables to SNAT our
> LAN. The setup for this is quite simple:
> 
> IPTABLES=`which iptables`
> LAN=eth0
> WAN=eth1
> PUB=eth2
> LAN_BLOCK=192.168.16.0/24
> NAT_SOURCE=1.2.3.4
> 
> $IPTABLES -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
> $IPTABLES -A FORWARD -i $LAN -o $WAN -m state --state NEW -j ACCEPT
> $IPTABLES -A FORWARD -i $LAN -o $PUB -m state --state NEW -j ACCEPT
> $IPTABLES -A FORWARD -i $PUB -o $LAN -m state --state NEW -j ACCEPT
> $IPTABLES -t nat -A POSTROUTING -o $WAN -s $LAN_BLOCK -j SNAT
> --to-source $NAT_SOURCE
> 
> With these rules, I get great download performance, regardless of
> protocol (sftp, ftp, http, etc).
> 
> Uploads, however, are a different story. I believe that the source of
> this issue is the NATing because, if I upload to the same server from
> a box that isn't being NATed (e.g. in the public block), upload speeds
> are an order of magnitude greater when not being NATed, regardless of
> protocol. For example, if I upload a file from one of our public
> servers to a remote location, it will upload at about 500Kb/s. The
> same upload to the same location with within the LAN won't see more
> than 60Kb/s.
> 
> I'm not doing any kind of fancy traffic control or anything like that.
> I would expect to see some slowdown because of the NAT (for example,
> downloads are 600Kb/s vs 500Kb/s through the NAT), but this kind of
> slowdown seems a bit ridiculous.
> 
> I'm running a custom 2.6.30.3 kernel.  I've also installed
> conntrack-tools thinking that my connection table might be getting
> full.  However, according to conntrack -S:
> 
> entries                 455
> searched                84375
> found                   10970723
> new                     242106
> invalid                 15039
> ignore                  53722
> delete                  241522
> delete_list             225919
> insert                  226270
> insert_failed           0
> drop                    0
> early_drop              0
> icmp_error              5741
> expect_new              28
> expect_create           43
> expect_delete           34
> 
> So I am well below the table max:
> 
> # sysctl net.netfilter.nf_conntrack_max
> net.netfilter.nf_conntrack_max = 65536
> 
> Is this kind of performance drop to be expected w/ netfilter and
> iptables?  Any suggestions on things I can do to increase performance?

I always *NAT in PREROUTING and MASQUERADE in POSTROUTING if needed.
Have never seen any such performance drop on multiple NIC.

AYJ

Re: Poor upload speed with NAT

[Steve Brown]

> I always *NAT in PREROUTING and MASQUERADE in POSTROUTING if needed.

According to the manpage for iptables, SNAT is only valid in the
POSTROUTING chain.  Is this incorrect?

> Have never seen any such performance drop on multiple NIC.

Neither have I, which is why this is bothering me.

Re: Poor upload speed with NAT

[/dev/rob0]

On Tuesday 11 August 2009 09:22:17 Steve Brown wrote:
> > I always *NAT in PREROUTING and MASQUERADE in POSTROUTING if
> > needed. 
>
> According to the manpage for iptables, SNAT is only valid in the
> POSTROUTING chain.  Is this incorrect?

MASQUERADE is a form of SNAT (source NAT). SNAT is done in the
POSTROUTING chain.

> > Have never seen any such performance drop on multiple NIC.
>
> Neither have I, which is why this is bothering me.

I can't really comment on this. You might consider doing some
troubleshooting to test your hypothesis, such as tcpdump(1) of some
poor-performing connections. You haven't posted anything here which
lends credence to it.

On Monday 10 August 2009 16:43:14 you wrote:
> I'm running a custom 2.6.30.3 kernel.

Do the same symptoms manifest when you're running the distro kernel?
The custom kernel would be a possible suspect.
-- 
    Offlist mail to this address is discarded unless
    "/dev/rob0" or "not-spam" is in Subject: header

Re: Poor upload speed with NAT

[Steve Brown]

>> > Have never seen any such performance drop on multiple NIC.
>>
>> Neither have I, which is why this is bothering me.
>
> I can't really comment on this. You might consider doing some
> troubleshooting to test your hypothesis, such as tcpdump(1) of some
> poor-performing connections. You haven't posted anything here which
> lends credence to it.

My setup:

LAN -----> Router -----> Internet

If I don't go through NAT, performance increases dramatically.  As I
stated in the OP, uploading to any location from the router box itself
is about 10x faster than going through NAT.  To me, that indicates the
likely source of the problem, although I'm open to other suggestions.

What would I be looking for in a tcpdump?  Connection resets or
something similar?

>> I'm running a custom 2.6.30.3 kernel.
>
> Do the same symptoms manifest when you're running the distro kernel?
> The custom kernel would be a possible suspect.

Isn't the kernel the "definitive" netfilter source?  How would a
vendor's patches impact performance to this degree?  If I am
understanding you correctly, you are suggesting that Ubuntu's
implementation of netfilter is significantly faster than netfilter's
implementation, is this correct?

I'm willing to allow that the kernel has something to do with this.
For example, when I first started looking at this problem, uploads
wouldn't go faster than 30kb/s on a 2.6.26 kernel.  When I upgraded to
2.6.30, performance doubled to 60kb/s.  So if I could double that
performance 4-5 more times, I would be good to go. ;-)

Unfortunately, these boxes have been in service for a long and the
vendor no longer supports the versions they are running.  They are
maintained by building software and the kernel from source, so going
back to a vendor-supplied kernel isn't really an option ATM.

Re: Poor upload speed with NAT

[Steve Brown]

> Is this kind of performance drop to be expected w/ netfilter and
> iptables?  Any suggestions on things I can do to increase performance?

Am I to assume by the silence of the list that this performance issue
is not going to be resolved?  Are there any other steps I can take to
boost upload speeds?

Re: Poor upload speed with NAT

[Steve Brown]

> At a root shell prompt, type "ip -s link ls" and look at the error
> rates, especially on your LAN interface. I suspect that will point you
> in the right direction.

Ah.... you are correct, that did point me in the right direction.  It
seems my WAN interface actually had the wrong media type specified,
thus creating lots of collisions.  Fixed that and performance
increased to where I was expecting it to be.  Couple of other issues
to work out, but thy aren't netfilter related (as this apparently
wasn't either). ;-)

Thanks!

Re: Poor upload speed with NAT

[Tom Eastep]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Steve Brown wrote:
>> Is this kind of performance drop to be expected w/ netfilter and
>> iptables?  Any suggestions on things I can do to increase performance?
>
> Am I to assume by the silence of the list that this performance issue
> is not going to be resolved?  Are there any other steps I can take to
> boost upload speeds?

At a root shell prompt, type "ip -s link ls" and look at the error
rates, especially on your LAN interface. I suspect that will point you
in the right direction.

- -Tom
- --
Tom Eastep        \ When I die, I want to go like my Grandfather who
Shoreline,         \ died peacefully in his sleep. Not screaming like
Washington, USA     \ all of the passengers in his car
http://shorewall.net \________________________________________________
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkqC58QACgkQO/MAbZfjDLLtlACdFtnrCwS/Gvx+/TlLmCA/jgru
Z60AnA/cPengcIo+L0P6tezkh4Lh0Cm9
=MRew
-----END PGP SIGNATURE-----