Extremely slow upload (and more) from behind NAT

Extremely slow upload (and more) from behind NAT

[Christian Pernegger]

Hi list,

I've been building my own home firewalling/NAT routers from commodity
hardware and Debian stable since the times iptables was called
ipchains and never had a problem, but now I've switched ISPs again and
I just can't seem to get NAT to work properly this time.

The ISP is Chello Austria (UPC), a cable one. For all intents and
purposes it's supposed to be a regular Ethernet connection at my end,
with a nominal bandwidth of 4 Mb/s up, 35 Mb/s down and a de facto
static public IP on my router's external interface. The cable modem
seems to act as a transparent bridge (it doesn't show up on
traceroute).

Everything is fine on the router itself, or any other box I directly
connect to the cable modem, but all NATed clients behind it get
severely degraded service, though only some things seem to be broken:
+ The downstream bandwidth is fine, within a few percent of nominal.
+ The latency (as measured by ping and similar) is excellent.
+ Online games like Team Fortress 2 and World of Warcraft work flawlessly.
- Web browsing only barely works. When trying to access a site,
Firefox will hang for 10+ seconds at the "Waiting for $SERVER" stage,
then the whole site will render instantly. More complex JavaScript or
Flash based sites might hang indefinitely.
I haven't yet stumbled across a site that won't work at all, hitting
refresh will eventually succeed. Also, sites never load partially, no
missing style sheets, "broken" images or the like. It seems to be an
all or nothing deal.
- The upstream bandwidth ... well, there isn't any, really. Test
uploads to an FTP server in the ISP's own network crawl along
erraticly at < 0.1 Mb/s and/or time out. The same test run from the
router gives me 1-2 Mb/s, three connections in parallel net the full 4
Ms/s.
- When testing via speedtest.net, the latency and downstream tests
work but the upload test fails. It just sits there "Connecting ..."
and eventually times out
- When testing via pingtest.net, the packet loss test fails in the
same fashion.
It doesn't matter if a client's running Linux or Windows.

The router is running Debian stable (Squeeze), using kernel
linux-image-2.6.32-5-686 2.6.32-35 and iptables 1.4.8-3 as provided.
DNS is provided by dnsmasq 2.55-2 running on the router with both the
ISP's and Google's servers upstream. IPv6 is turned off via kernel
command line. For testing purposes I've reduced my firewall script to
this one-liner:
iptables -t nat -A POSTROUTING -s 192.168.0.0/24 -o eth1 -j MASQUERADE

The official method for getting one's IP is DHCP, though configuring
it statically reportedly also works. I tried both, no difference.
The DHCP server *does* suggest an MTU of 576 bytes instead of the
ususal 1500 bytes, but that seems to be bogus. Manual PMTU discovery
via don't-fragment pings to various servers is consistent with an MTU
of 1500 and anyway, changing it to 576 doesn't have any appreciable
effect at all, with or without a TCPMSS rule as suggested by the
iptables man page.

Since randomly disabling TCP "offensive" TCP options like syncookies,
ECN and even SACK didn't work either, I'm now officially at my wits
end.

I'd be really grateful if anyone could help me out ...

Thank you,

Christian

Re: Extremely slow upload (and more) from behind NAT

[Tyler J. Wagner]

On 2011-08-12 17:06, Christian Pernegger wrote:
> Since randomly disabling TCP "offensive" TCP options like syncookies,
> ECN and even SACK didn't work either, I'm now officially at my wits
> end.

That's a good one. Do you have any unusual routing loops or other
network oddities? Just eth0 (external) and eth1 (internal)?

Please post your iptables / iptables-restore script.

Regards,
Tyler

-- 
"Never underestimate the bandwidth of a station wagon full of tapes
hurtling down the highway.
   -- Andrew S. Tanenbaum

Re: Extremely slow upload (and more) from behind NAT

[Christian Pernegger]

> Do you have any unusual routing loops or other network oddities? Just eth0 (external)
> and eth1 (internal)?

No, it's a bog standard setup. eth0 is the internal, eth1 the external
interface.

> Please post your iptables / iptables-restore script.

I did, in a way. It's really just that one line, "iptables -t nat -A
POSTROUTING -s 192.168.0.0/24 -o eth1 -j MASQUERADE", at the moment,
executed as a post-up command from /etc/network/interfaces.

Regards,

Christian

Re: Extremely slow upload (and more) from behind NAT

[Tyler J. Wagner]

On 2011-08-12 17:45, Christian Pernegger wrote:
>> Please post your iptables / iptables-restore script.
> 
> I did, in a way. It's really just that one line, "iptables -t nat -A
> POSTROUTING -s 192.168.0.0/24 -o eth1 -j MASQUERADE", at the moment,
> executed as a post-up command from /etc/network/interfaces.

I doubt that's the problem, then. I would write it without the "-s
192.168.0.0/24" statement, as it is not needed if you have no other
networks or interfaces. But I very much doubt it is the cause of the
problem.

You said the problem only happens to NAT'ed traffic. What if it's not
NAT, but eth0, that's the issue? Try swapping eth0 and eth1 and see if
the problem affects traffic to/from the router as well.

I recently had a problem with one server (with an old cheap Via chipset)
that ran fine on Ubuntu 8.04. But when I reformatted it for Ubuntu
10.04, eth0 started having serious issues with throughput that I
eventually concluded was an IRQ issue. Since it had two interfaces, I
renamed them and used the other one.

Regards,
Tyler

-- 
"Offending fundamentalists isn't my goal – but if it is an inevitable
side-effect of defending human rights, so be it."
   -- Johann Hari

Re: Extremely slow upload (and more) from behind NAT

[Christian Pernegger]

> You said the problem only happens to NAT'ed traffic. What if it's not
> NAT, but eth0, that's the issue? Try swapping eth0 and eth1 and see if
> the problem affects traffic to/from the router as well.

Yes! That was it (in a way). Over the long weekend I cobbled together
a replacement router, booted it from an image of the first one and of
course it worked perfectly ... ...

So it's some kind of hardware issue in the router - or one of the
cables / switches is bad intermittently and all the un-, replugging
and wiggling about "fixed" it for now. Strange that the problem only
affected the upstream, though.

Either way, netfilter's not to blame. I'm terribly sorry for having
wasted your time.

Regards,

Christian

Re: Extremely slow upload (and more) from behind NAT

[Jan Engelhardt]

On Friday 2011-08-12 18:06, Christian Pernegger wrote:

>Hi list,
>
>I've been building my own home firewalling/NAT routers from commodity
>hardware and Debian stable since the times iptables was called
>ipchains and never had a problem, but now I've switched ISPs again and
>I just can't seem to get NAT to work properly this time.
>
>The ISP is Chello Austria (UPC), a cable one. For all intents and
>purposes it's supposed to be a regular Ethernet connection at my end,
>with a nominal bandwidth of 4 Mb/s up, 35 Mb/s down and a de facto
>static public IP on my router's external interface. The cable modem
>seems to act as a transparent bridge (it doesn't show up on
>traceroute).

transparent and bridge is a tautology - by definition, bridges, by 
default, like regular switches and hubs, don't show up on traceroutes at 
all, unless you explicitly add a TTL breakpoint.


>Everything is fine on the router itself, or any other box I directly
>connect to the cable modem, but all NATed clients behind it get
>severely degraded service, though only some things seem to be broken:
>+ The downstream bandwidth is fine, within a few percent of nominal.
>+ The latency (as measured by ping and similar) is excellent.
>+ Online games like Team Fortress 2 and World of Warcraft work flawlessly.
>- Web browsing only barely works. When trying to access a site,
>Firefox will hang for 10+ seconds at the "Waiting for $SERVER" stage,
>then the whole site will render instantly.

So what happens (log output!) if you issue

 telnet netfilter.org 80

>- The upstream bandwidth ... well, there isn't any, really. Test
>uploads to an FTP server in the ISP's own network crawl along
>erraticly at < 0.1 Mb/s and/or time out. The same test run from the
>router gives me 1-2 Mb/s, three connections in parallel net the full 4
>Ms/s.

Owing to your mail's subject, does it go away without NAT?

>- When testing via speedtest.net, the latency and downstream tests
>work but the upload test fails. It just sits there "Connecting ..."
>and eventually times out

Those speedtest sites are, most of the times, for the trash anyway, 
since they often only utilize a single connection only, which is not 
going to be a good approximation of utilizable bandwidth. (I used to 
require like 5 concurrent connections to dlc.sun.com to fill 90 Mbit.)

Re: Extremely slow upload (and more) from behind NAT

[Grant Taylor]

On 08/12/11 11:06, Christian Pernegger wrote:
> The official method for getting one's IP is DHCP, though configuring
> it statically reportedly also works. I tried both, no difference.
> The DHCP server *does* suggest an MTU of 576 bytes instead of the
> ususal 1500 bytes, but that seems to be bogus. Manual PMTU discovery
> via don't-fragment pings to various servers is consistent with an MTU
> of 1500 and anyway, changing it to 576 doesn't have any appreciable
> effect at all, with or without a TCPMSS rule as suggested by the
> iptables man page.

I was going to say that this /really/ seems like an MTU / TCPMSS issue 
to me.

For giggles, ssh from one of the clients configuring the ssh client as a 
socks proxy.  Then have your web browser use the ssh / socks proxy for 
testing.  If that does work correctly, I'd still really question MTU / 
TCPMSS.

What happens if you clamp the MTU / TCPMSS really low just to make sure 
you are (way) below any thing interfering.

Have you tried running a network sniffer on any of the traffic to see 
what it's doing?  Do you have any re-transmissions?  Do you see requests 
that don't have associated replies?

Do the sniffs on the inside interface match the outside interface (save 
for the nated IP address)?



Grant. . . .