Correctly calculating overheads on unknown connections

Correctly calculating overheads on unknown connections

[Alan Goodman]

Hi,

I am looking to figure out the most fool proof way to calculate stab 
overheads for ADSL/VDSL connections.

ppp0      Link encap:Point-to-Point Protocol
           inet addr:81.149.38.69  P-t-P:81.139.160.1 Mask:255.255.255.255
           UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1492  Metric:1
           RX packets:17368223 errors:0 dropped:0 overruns:0 frame:0
           TX packets:12040295 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:100
           RX bytes:17420109286 (16.2 GiB)  TX bytes:3611007028 (3.3 GiB)

I am setting a longer txqueuelen as I am not currently using any fair 
queuing (buffer bloat issues with sfq)

The connection is a BT Infinity FTTC VDSL connection synced at 
80mbit/20mbit.  The modem is connected directly to the ethernet port on 
a server running a slightly tweaked HFSC setup that you folks helped me 
set up in July - back when I was on ADSL.  I am still running pppoe I 
believe from my server.

The largest ping packet that I can fit out onto the wire is 1464 bytes:

# ping -c 2 -s 1464 -M do google.com
PING google.com (31.55.166.216) 1464(1492) bytes of data.
1472 bytes from 31.55.166.216: icmp_seq=1 ttlX time\x11.7 ms
1472 bytes from 31.55.166.216: icmp_seq=2 ttlX time\x11.9 ms

# ping -c 2 -s 1465 -M do google.com
PING google.com (31.55.166.212) 1465(1493) bytes of data.
 From host81-149-38-69.in-addr.btopenworld.com (81.149.38.69) icmp_seq=1 
Frag needed and DF set (mtu = 1492)
 From host81-149-38-69.in-addr.btopenworld.com (81.149.38.69) icmp_seq=1 
Frag needed and DF set (mtu = 1492)

Based on this I believe overhead should be set to 28, however with 28 
set as my overhead and hfsc ls m2 20000kbit ul m2 20000kbit I seem to be 
loosing about 1.5mbit of upload...

No traffic manager enabled: 
http://www.thinkbroadband.com/speedtest/results.html?id\x141116089424883990118
HFSC traffic manager: 
http://www.thinkbroadband.com/speedtest/results.html?id\x141116216621093133034

Am I calculating overhead incorrectly?

Alan

Re: Correctly calculating overheads on unknown connections

[Andy Furniss]

Alan Goodman wrote:
> Hi,
>
> I am looking to figure out the most fool proof way to calculate stab
>  overheads for ADSL/VDSL connections.
>
> ppp0      Link encap:Point-to-Point Protocol inet addr:81.149.38.69
> P-t-P:81.139.160.1 Mask:255.255.255.255 UP POINTOPOINT RUNNING NOARP
> MULTICAST  MTU:1492  Metric:1 RX packets:17368223 errors:0 dropped:0
> overruns:0 frame:0 TX packets:12040295 errors:0 dropped:0 overruns:0
> carrier:0 collisions:0 txqueuelen:100 RX bytes:17420109286 (16.2 GiB)
> TX bytes:3611007028 (3.3 GiB)
>
> I am setting a longer txqueuelen as I am not currently using any fair
>  queuing (buffer bloat issues with sfq)

Whatever is txqlen is on ppp there is likely some other buffer after it
- the default can hurt with eg, htb as if you don't add qdiscs to
classes it takes (last time I looked) its qlen from that.

Sfq was only ever meant for bulk, so should really be in addition to
some classification to separate interactive - I don't really get the
bufferbloat bit, you could make the default 128 limit lower if you wanted.

> The connection is a BT Infinity FTTC VDSL connection synced at
> 80mbit/20mbit.  The modem is connected directly to the ethernet port
> on a server running a slightly tweaked HFSC setup that you folks
> helped me set up in July - back when I was on ADSL.  I am still
> running pppoe I believe from my server.

I have similar since May 2013 and I still haven't got round to reading
up on everything yet :-)

I have extra geek score for using mini jumbos = running pppoe with mtu
1500 which works for me on plusnet. You need a recent pppd for this and
a nic that works with mtu >= 1508.

As for overheads, initial searching indicated that it's not easy or
maybe even truly possible like adsl.

> The largest ping packet that I can fit out onto the wire is 1464
> bytes:
>
> # ping -c 2 -s 1464 -M do google.com PING google.com (31.55.166.216)
> 1464(1492) bytes of data. 1472 bytes from 31.55.166.216: icmp_seq=1
> ttlX time\x11.7 ms 1472 bytes from 31.55.166.216: icmp_seq=2 ttlX
> time\x11.9 ms
>
> # ping -c 2 -s 1465 -M do google.com PING google.com (31.55.166.212)
> 1465(1493) bytes of data. From
> host81-149-38-69.in-addr.btopenworld.com (81.149.38.69) icmp_seq=1
> Frag needed and DF set (mtu = 1492) From
> host81-149-38-69.in-addr.btopenworld.com (81.149.38.69) icmp_seq=1
> Frag needed and DF set (mtu = 1492)

You can't work out your overheads like this.

On slow uplink adsl it was possible with ping to infer the fixed part
but you needed to send loads of pings increasing in size and plot the
best time for each to make a stepped graph.


> Based on this I believe overhead should be set to 28, however with 28
>  set as my overhead and hfsc ls m2 20000kbit ul m2 20000kbit I seem
> to be loosing about 1.5mbit of upload...

Even if you could do things perfectly I would back off a few kbit just
to be safe. Timers may be different or there may be OAM/Reporting data
going up, albeit rarely.

>
> No traffic manager enabled:
> http://www.thinkbroadband.com/speedtest/results.html?id\x141116089424883990118
>
>
> HFSC traffic manager:
> http://www.thinkbroadband.com/speedtest/results.html?id\x141116216621093133034
>
>
>
> Am I calculating overhead incorrectly?

VDSL doesn't use ATM I think the PTM it uses is 64/65 - so don't specify
atm with stab. Unfortunately stab doesn't do 64/65.

As for the fixed part - I am not sure, but roughly starting with IP as
that's what tc sees on ppp (as opposed to ip + 14 on eth)

IP
+8 for PPPOE
+14 for ethertype and macs
+4 because Openreach modem uses vlan
+2 CRC ??
+ "a few" 64/65

That's it for fixed - of course 64/65 adds another one for every 64 TBH
I didn't get the precice detail from the spec and not having looked
recently I can't remember.

BT Sin 498 does give some of this info and a couple of examples of
throughput for different frame sizes - but it's rounded to kbit which
means I couldn't work out to the byte what the overheads were.

Worse still VDSL can use link layer retransmits and the sin says that
though currently (2013) not enabled, they would be in due course. I have
no clue how these work.

Re: Correctly calculating overheads on unknown connections

[Dave Taht]

We'd had a very long thread on cerowrt-devel and in the end sebastian
(I think) had developed some scripts to exaustively (it took hours)
derive the right encapsulation frame size on a link. I can't find the
relevant link right now, ccing that list...

On Sat, Sep 20, 2014 at 7:17 PM, Andy Furniss <adf.lists@gmail.com> wrote:
> Alan Goodman wrote:
>>
>> Hi,
>>
>> I am looking to figure out the most fool proof way to calculate stab
>>  overheads for ADSL/VDSL connections.
>>
>> ppp0      Link encap:Point-to-Point Protocol inet addr:81.149.38.69
>> P-t-P:81.139.160.1 Mask:255.255.255.255 UP POINTOPOINT RUNNING NOARP
>> MULTICAST  MTU:1492  Metric:1 RX packets:17368223 errors:0 dropped:0
>> overruns:0 frame:0 TX packets:12040295 errors:0 dropped:0 overruns:0
>> carrier:0 collisions:0 txqueuelen:100 RX bytes:17420109286 (16.2 GiB)
>> TX bytes:3611007028 (3.3 GiB)
>>
>> I am setting a longer txqueuelen as I am not currently using any fair
>>  queuing (buffer bloat issues with sfq)
>
>
> Whatever is txqlen is on ppp there is likely some other buffer after it
> - the default can hurt with eg, htb as if you don't add qdiscs to
> classes it takes (last time I looked) its qlen from that.
>
> Sfq was only ever meant for bulk, so should really be in addition to
> some classification to separate interactive - I don't really get the

Hmm? sfq separates bulk from interactive pretty nicely. It tends to do
bad things to bulk as it doesn't manage queue length.

A little bit of prioritization or deprioritization for some traffic is
helpful, but most traffic is hard to classify.

> bufferbloat bit, you could make the default 128 limit lower if you wanted.

htb + fq_codel, if available, is the right thing here....

http://www.bufferbloat.net/projects/cerowrt/wiki/Wondershaper_Must_Die

>> The connection is a BT Infinity FTTC VDSL connection synced at
>> 80mbit/20mbit.  The modem is connected directly to the ethernet port
>> on a server running a slightly tweaked HFSC setup that you folks
>> helped me set up in July - back when I was on ADSL.  I am still
>> running pppoe I believe from my server.
>
>
> I have similar since May 2013 and I still haven't got round to reading
> up on everything yet :-)
>
> I have extra geek score for using mini jumbos = running pppoe with mtu
> 1500 which works for me on plusnet. You need a recent pppd for this and
> a nic that works with mtu >= 1508.
>
> As for overheads, initial searching indicated that it's not easy or
> maybe even truly possible like adsl.
>
>> The largest ping packet that I can fit out onto the wire is 1464
>> bytes:
>>
>> # ping -c 2 -s 1464 -M do google.com PING google.com (31.55.166.216)
>> 1464(1492) bytes of data. 1472 bytes from 31.55.166.216: icmp_seq=1
>> ttlX time\x11.7 ms 1472 bytes from 31.55.166.216: icmp_seq=2 ttlX
>> time\x11.9 ms
>>
>> # ping -c 2 -s 1465 -M do google.com PING google.com (31.55.166.212)
>> 1465(1493) bytes of data. From
>> host81-149-38-69.in-addr.btopenworld.com (81.149.38.69) icmp_seq=1
>> Frag needed and DF set (mtu = 1492) From
>> host81-149-38-69.in-addr.btopenworld.com (81.149.38.69) icmp_seq=1
>> Frag needed and DF set (mtu = 1492)
>
>
> You can't work out your overheads like this.
>
> On slow uplink adsl it was possible with ping to infer the fixed part
> but you needed to send loads of pings increasing in size and plot the
> best time for each to make a stepped graph.
>
>
>> Based on this I believe overhead should be set to 28, however with 28
>>  set as my overhead and hfsc ls m2 20000kbit ul m2 20000kbit I seem
>> to be loosing about 1.5mbit of upload...
>
>
> Even if you could do things perfectly I would back off a few kbit just
> to be safe. Timers may be different or there may be OAM/Reporting data
> going up, albeit rarely.
>
>>
>> No traffic manager enabled:
>>
>> http://www.thinkbroadband.com/speedtest/results.html?id\x141116089424883990118
>>
>>
>> HFSC traffic manager:
>>
>> http://www.thinkbroadband.com/speedtest/results.html?id\x141116216621093133034
>>
>>
>>
>> Am I calculating overhead incorrectly?
>
>
> VDSL doesn't use ATM I think the PTM it uses is 64/65 - so don't specify
> atm with stab. Unfortunately stab doesn't do 64/65.
>
> As for the fixed part - I am not sure, but roughly starting with IP as
> that's what tc sees on ppp (as opposed to ip + 14 on eth)
>
> IP
> +8 for PPPOE
> +14 for ethertype and macs
> +4 because Openreach modem uses vlan
> +2 CRC ??
> + "a few" 64/65
>
> That's it for fixed - of course 64/65 adds another one for every 64 TBH
> I didn't get the precice detail from the spec and not having looked
> recently I can't remember.
>
> BT Sin 498 does give some of this info and a couple of examples of
> throughput for different frame sizes - but it's rounded to kbit which
> means I couldn't work out to the byte what the overheads were.
>
> Worse still VDSL can use link layer retransmits and the sin says that
> though currently (2013) not enabled, they would be in due course. I have
> no clue how these work.
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe lartc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Dave Täht

https://www.bufferbloat.net/projects/make-wifi-fast

Re: Correctly calculating overheads on unknown connections

[Andy Furniss]

Dave Taht wrote:
> We'd had a very long thread on cerowrt-devel and in the end
> sebastian (I think) had developed some scripts to exaustively (it
> took hours) derive the right encapsulation frame size on a link. I
> can't find the relevant link right now, ccing that list...

Thanks, that sounds cool.

>> Sfq was only ever meant for bulk, so should really be in addition
>> to some classification to separate interactive - I don't really get
>> the
>
> Hmm? sfq separates bulk from interactive pretty nicely. It tends to
> do bad things to bulk as it doesn't manage queue length.

Well I come at this from years of qos stuck on 288 then 448 kbit up atm
dsl before the days of fq_codel.

Since it got jhash sfq does at least manage to avoid collisions, but
it's still a total non starter for use alone on a slow link because the
interactive packet may wait for many bulk packets to dequeue before its
turn.

Of course sfq is cleverer now than it used to be - headdrop and red the
latter I've not used.

I agree it's bloaty for bulk, but that's not quite so critical if your
real buffer is not bloaty if you can get classification to work.

> A little bit of prioritization or deprioritization for some traffic
> is helpful, but most traffic is hard to classify.
>
>> bufferbloat bit, you could make the default 128 limit lower if you
>> wanted.
>
> htb + fq_codel, if available, is the right thing here....

Yea, though on a link like my old one I still think classification would
just win. I should test really, but IME a slow link can be hard to
simulate (I have 20mbit up now) - the results tend to look a bit better
because of no real bitrate latency.

Re: Correctly calculating overheads on unknown connections

[Alan Goodman]

Hello all,

On 20/09/14 17:17, Andy Furniss wrote:> Alan Goodman wrote:
 >> Hi,
 >>
 >> I am looking to figure out the most fool proof way to calculate stab
 >>  overheads for ADSL/VDSL connections.
 >>
 >> ppp0      Link encap:Point-to-Point Protocol inet addr:81.149.38.69
 >> P-t-P:81.139.160.1 Mask:255.255.255.255 UP POINTOPOINT RUNNING NOARP
 >> MULTICAST  MTU:1492  Metric:1 RX packets:17368223 errors:0 dropped:0
 >> overruns:0 frame:0 TX packets:12040295 errors:0 dropped:0 overruns:0
 >> carrier:0 collisions:0 txqueuelen:100 RX bytes:17420109286 (16.2 GiB)
 >> TX bytes:3611007028 (3.3 GiB)
 >>
 >> I am setting a longer txqueuelen as I am not currently using any fair
 >>  queuing (buffer bloat issues with sfq)
 >
 > Whatever is txqlen is on ppp there is likely some other buffer after it
 > - the default can hurt with eg, htb as if you don't add qdiscs to
 > classes it takes (last time I looked) its qlen from that.
 >
 > Sfq was only ever meant for bulk, so should really be in addition to
 > some classification to separate interactive - I don't really get the
 > bufferbloat bit, you could make the default 128 limit lower if you 
wanted.

My issue is I am often shaping on connections with low upload speed. 
SFQ limits queue sized based on packets.  Typically I run three queues, 
priority, interactive and bulk.  I want to keep delay in the system 
below around 100ms therefore with SFQ I must keep total 'limit' below 
about 12 packets where the upload is about 1mbit/sec.  In practice a lot 
of traffic hitting the interactive queue is smaller packets though and 
despite attempting to weight HFSC to allow much more of this traffic 
through it gets dropped due to overrunning the SFQ limit.  This results 
in bulk receiving more bandwidth than it should./

I did work around this somewhat by switching to using BFIFO, however I 
found that the most consistent performance for slower uploads where 
there isnt much time for fairness was achieved with no additional levels.

I do still operate SFQ on downstream and am excited to try out 
HFSC+FQ_Codel in Centos 7.

 >> Am I calculating overhead incorrectly?
 >
 > VDSL doesn't use ATM I think the PTM it uses is 64/65 - so don't specify
 > atm with stab. Unfortunately stab doesn't do 64/65.
 >
 > As for the fixed part - I am not sure, but roughly starting with IP as
 > that's what tc sees on ppp (as opposed to ip + 14 on eth)
 >
 > IP
 > +8 for PPPOE
 > +14 for ethertype and macs
 > +4 because Openreach modem uses vlan
 > +2 CRC ??
 > + "a few" 64/65
 >
 > That's it for fixed - of course 64/65 adds another one for every 64 TBH
 > I didn't get the precice detail from the spec and not having looked
 > recently I can't remember.
 >
 > BT Sin 498 does give some of this info and a couple of examples of
 > throughput for different frame sizes - but it's rounded to kbit which
 > means I couldn't work out to the byte what the overheads were.
 >
 > Worse still VDSL can use link layer retransmits and the sin says that
 > though currently (2013) not enabled, they would be in due course. I have
 > no clue how these work.

Both interesting and informative, thank you.

I have done a little bit of experimentation...  stab overhead 28 (taken 
from the results of the pingsweeper) with ls m2 19500kbit ul m2 
19500kbit appears to be providing good results in my basic testing with 
a simultaneous http download and scp upload.  Since BT limit based on IP 
throughput on the downstream in their BRAS system I limit that to 
75000kbit with no stab options.  With 20 http downloads running 
simultaneously in the bulk queue a voip call has 0 packet loss (vs 1% 
ish with no management).

I welcome all comments and criticisms.

Alan