HTTP slower than SSH on client behind iptables

HTTP slower than SSH on client behind iptables

[Adam Rosi-Kessel]

I have a standard NAT box; the essential configuration for discussion
purposes on the box is just IP masquerading.

On the NAT box, my Internet connection goes up to about 700 kilobytes per
second, regardless of the protocol used (e.g., ssh or http).  

On clients behind the NAT box, however, HTTP connections seem to top out
around 70 kilobytes per second. ssh connections (e.g., rsync) get the
full throughput of the Internet connection.

As far as NAT goes, I don't hvae any special settings.

Can anyone think of an explanation for this behavior? It doesn't make any
sense to me.
-- 
Adam Rosi-Kessel
http://adam.rosi-kessel.org

Re: HTTP slower than SSH on client behind iptables

[Boryan Yotov]

Adam Rosi-Kessel wrote:
> I have a standard NAT box; the essential configuration for discussion
> purposes on the box is just IP masquerading.
> 
> On the NAT box, my Internet connection goes up to about 700 kilobytes per
> second, regardless of the protocol used (e.g., ssh or http).  
> 
> On clients behind the NAT box, however, HTTP connections seem to top out
> around 70 kilobytes per second. ssh connections (e.g., rsync) get the
> full throughput of the Internet connection.
> 
> As far as NAT goes, I don't hvae any special settings.
> 
> Can anyone think of an explanation for this behavior? It doesn't make any
> sense to me.

Are you sure, you don't have some kind of a traffic shaping
active on the NAT gateway's internal interface?

For example: If tc is used, you could check that using:

tc class show dev <nat_box_internal_interface>

and

tc filter show dev <nat_box_internal_interface>

Re: HTTP slower than SSH on client behind iptables

[Adam Rosi-Kessel]

[-- Attachment #1.1: Type: text/plain, Size: 1235 bytes --]

(sorry, should have sent this to the list)

Boryan Yotov wrote:
>> On the NAT box, my Internet connection goes up to about 700 kilobytes per
>> second, regardless of the protocol used (e.g., ssh or http). 
>> On clients behind the NAT box, however, HTTP connections seem to top out
>> around 70 kilobytes per second. ssh connections (e.g., rsync) get the
>> full throughput of the Internet connection.
> Are you sure, you don't have some kind of a traffic shaping
> active on the NAT gateway's internal interface?
> For example: If tc is used, you could check that using:
> tc class show dev <nat_box_internal_interface>
> and
> tc filter show dev <nat_box_internal_interface>

I do have traffic shaping on the *external* interface, but it is not port
dependent.

tc class|filter show dev <internal_interface> gives an empty response.

Also, when I clear the queue (I use wondershaper, so running wondershaper
clear), it seems I still have the same problem.

...But is there some way that outbound traffic shaping on the external
facing interface could somehow impact *inbound* HTTP traffic on an internal
client connected to the internal facing interface?
-- 
Adam Rosi-Kessel
http://adam.rosi-kessel.org

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: HTTP slower than SSH on client behind iptables

[Boryan Yotov]

Adam Rosi-Kessel wrote:
> (sorry, should have sent this to the list)
> 
> Boryan Yotov wrote:
> 
>>>On the NAT box, my Internet connection goes up to about 700 kilobytes per
>>>second, regardless of the protocol used (e.g., ssh or http). 
>>>On clients behind the NAT box, however, HTTP connections seem to top out
>>>around 70 kilobytes per second. ssh connections (e.g., rsync) get the
>>>full throughput of the Internet connection.
>>
>>Are you sure, you don't have some kind of a traffic shaping
>>active on the NAT gateway's internal interface?
>>For example: If tc is used, you could check that using:
>>tc class show dev <nat_box_internal_interface>
>>and
>>tc filter show dev <nat_box_internal_interface>
> 
> 
> I do have traffic shaping on the *external* interface, but it is not port
> dependent.
> 
> tc class|filter show dev <internal_interface> gives an empty response.
> 
> Also, when I clear the queue (I use wondershaper, so running wondershaper
> clear), it seems I still have the same problem.
> 
> ...But is there some way that outbound traffic shaping on the external
> facing interface could somehow impact *inbound* HTTP traffic on an internal
> client connected to the internal facing interface?

I looked shortly at wondershaper v.1.1a content.

1. In this version of the script I see no "clear" option, but only
"stop" and "status". Could you try "./wondershaper stop" and check
if it fix the problem. Or probably you have a newer one.

2. The wondershaper is using kilobits instead of kilobytes. If you
have a 700 kilobytes uplink, then you'll have to set UPLINK equal
to 5600. The same is valid for DOWNLINK.

3. In wondershaper I see a class with ID 1:30, which is shaping
traffic to a rate of "8*UPLINK/10". There is a filter attached to
it, which in the default config will shape traffic with source
port(s) as defined in NOPRIOPORTSRC (in the version I have it
defaults to port 80).

The main idea here is (no matter how impossible it sounds):

  + If the wondershaper's DEV is accidently set to match your
    internall interface

  AND

  + You did wrote 700 instead of 5600 as UPLINK value

  THEN

  = You'll have these 700 kbytes limited to exactly 70 kbytes for
    all HTTP responses going to your LAN clients.

You already mentioned, that you enable wondershaper on the external
interface only, but it is worth to check. It is not unusuall to have
internal iface configured on eth0 and external on eth1.

Hope some of these will help :)

Re: HTTP slower than SSH on client behind iptables

[Adam Rosi-Kessel]

(trying one more time; netfilter archive seems to have scrubbed the whole
message for PGP signature...) ... (sorry, should have sent this to the list)

Boryan Yotov wrote:
>> On the NAT box, my Internet connection goes up to about 700 kilobytes per
>> second, regardless of the protocol used (e.g., ssh or http). 
>> On clients behind the NAT box, however, HTTP connections seem to top out
>> around 70 kilobytes per second. ssh connections (e.g., rsync) get the
>> full throughput of the Internet connection.
> Are you sure, you don't have some kind of a traffic shaping
> active on the NAT gateway's internal interface?
> For example: If tc is used, you could check that using:
> tc class show dev <nat_box_internal_interface>
> and
> tc filter show dev <nat_box_internal_interface>

I do have traffic shaping on the *external* interface, but it is not port
dependent.

tc class|filter show dev <internal_interface> gives an empty response.

Also, when I clear the queue (I use wondershaper, so running wondershaper
clear), it seems I still have the same problem.

...But is there some way that outbound traffic shaping on the external
facing interface could somehow impact *inbound* HTTP traffic on an internal
client connected to the internal facing interface?
-- 
Adam Rosi-Kessel
http://adam.rosi-kessel.org

Re: HTTP slower than SSH on client behind iptables

[Adam Rosi-Kessel]

On Tue, Jan 31, 2006 at 10:10:53AM +0100, Boryan Yotov wrote:
> >On clients behind the NAT box, however, HTTP connections seem to top out
> >around 70 kilobytes per second. ssh connections (e.g., rsync) get the
> >full throughput of the Internet connection.
> >As far as NAT goes, I don't hvae any special settings.
> >Can anyone think of an explanation for this behavior? It doesn't make any
> >sense to me.
> Are you sure, you don't have some kind of a traffic shaping
> active on the NAT gateway's internal interface?
> For example: If tc is used, you could check that using:
> tc class show dev <nat_box_internal_interface>
> and
> tc filter show dev <nat_box_internal_interface>

I figured it out. Apparently I was missing some kernel modules that were
causing wondershaper to behave oddly. I rebuilt the kernel with all QOS
and netfilter configuration options enabled (or built as modules) and now
internal clients can download HTTP at full speed. I suspect there was
some option that was causing tc to not distinguish between interfaces
despite the fact that wondershaper instructed it to only throttle the
external interface. I'm not sure exactly which kernel setting fixed it,
but it is now fixed.
-- 
Adam Rosi-Kessel
http://adam.rosi-kessel.org

Re: HTTP slower than SSH on client behind iptables

[Adam Rosi-Kessel]

On Thu, Feb 02, 2006 at 09:56:13AM -0500, Adam Rosi-Kessel wrote:
> On Tue, Jan 31, 2006 at 10:10:53AM +0100, Boryan Yotov wrote:
> > >On clients behind the NAT box, however, HTTP connections seem to top out
> > >around 70 kilobytes per second. ssh connections (e.g., rsync) get the
> > >full throughput of the Internet connection.
> > >As far as NAT goes, I don't hvae any special settings.
> > >Can anyone think of an explanation for this behavior? It doesn't make any
> > >sense to me.
> > Are you sure, you don't have some kind of a traffic shaping
> > active on the NAT gateway's internal interface?
> > For example: If tc is used, you could check that using:
> > tc class show dev <nat_box_internal_interface>
> > and
> > tc filter show dev <nat_box_internal_interface>
> I figured it out. Apparently I was missing some kernel modules that were
> causing wondershaper to behave oddly. I rebuilt the kernel with all QOS
> and netfilter configuration options enabled (or built as modules) and now
> internal clients can download HTTP at full speed. I suspect there was
> some option that was causing tc to not distinguish between interfaces
> despite the fact that wondershaper instructed it to only throttle the
> external interface. I'm not sure exactly which kernel setting fixed it,
> but it is now fixed.

Actually, I didn't figure it out! Apparently, just rebooting the NAT
system returns everything to full speed. Something happens, either over
time, or as the result of some occasional event, that causes internal
connections to be throttled. Any ideas what this could be?

Re: HTTP slower than SSH on client behind iptables

[Boryan Yotov]

Adam Rosi-Kessel wrote:
> On Thu, Feb 02, 2006 at 09:56:13AM -0500, Adam Rosi-Kessel wrote:
> 
>>On Tue, Jan 31, 2006 at 10:10:53AM +0100, Boryan Yotov wrote:
>>
>>>>On clients behind the NAT box, however, HTTP connections seem to top out
>>>>around 70 kilobytes per second. ssh connections (e.g., rsync) get the
>>>>full throughput of the Internet connection.
>>>>As far as NAT goes, I don't hvae any special settings.
>>>>Can anyone think of an explanation for this behavior? It doesn't make any
>>>>sense to me.
>>>
>>>Are you sure, you don't have some kind of a traffic shaping
>>>active on the NAT gateway's internal interface?
>>>For example: If tc is used, you could check that using:
>>>tc class show dev <nat_box_internal_interface>
>>>and
>>>tc filter show dev <nat_box_internal_interface>
>>
>>I figured it out. Apparently I was missing some kernel modules that were
>>causing wondershaper to behave oddly. I rebuilt the kernel with all QOS
>>and netfilter configuration options enabled (or built as modules) and now
>>internal clients can download HTTP at full speed. I suspect there was
>>some option that was causing tc to not distinguish between interfaces
>>despite the fact that wondershaper instructed it to only throttle the
>>external interface. I'm not sure exactly which kernel setting fixed it,
>>but it is now fixed.
> 
> 
> Actually, I didn't figure it out! Apparently, just rebooting the NAT
> system returns everything to full speed. Something happens, either over
> time, or as the result of some occasional event, that causes internal
> connections to be throttled. Any ideas what this could be?
> 

A good starting point is to retrieve the "tc" status, first
when the NAT gateway is restarted and performs correctly, and
once more when you notice that your client are receiving their
HTTP responses throthled.

To get a complete picture of the "tc" status, first retrieve
the qdisc assigned to each interface:

   # tc qdisc  show

And then retrieve the classes and assigned filters for each
of the interfaces shown in "tc qdisc show":

   # tc class  show dev <NEXT_DEVICE>
   # tc filter show dev <NEXT_DEVICE>

Once you have both states, then check if there is a differense
between them.

The idea here is to make sure your "tc" setup is not becoming
corrupted, when your external link need to be renegotiated (in
case of dynamic IP address assignment found with DSL and Cable
modems).




Another thing worth to try is to check if you use the CBQ based
version of the wondershaper script. If true, then switch to the
HTB version and see if it performs better.