Matching streaming services

Matching streaming services

[Nikolai Lusan]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi,

I have been looking for a way to differentiate traffic from streaming
services (Netflix, Amazon Prime, <insert_locally_available_service>)
from other https traffic, with not much luck. The goal is to add rules
to nftables and tc to ensure quality while allowing the rest of the
link to function normally.

I tried using tcpdump to see if there was something in the packets that
I could use, but they look like any other bit of https traffic.

Does anyone have a method for determining which http/https traffic is
streaming video, and which is not?

Thanks
- -- 
Nikolai Lusan <nikolai@lusan.id.au>
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEVfd4GW6z4nsBxdLo4ZaDRV2VL6QFAl/1/AYACgkQ4ZaDRV2V
L6Qtaw/9EaJHbOuar8d8s4FyKU3JvJxlV0SAPsHSoSGHWFZ3j7eYj+A8SZ8vylni
AGzZ0SbstVcpW7BvnjHfo7RV1rrqdM1dTxXGecXfVuktFWg6PyWcICrUqnwo9Zkd
bQ3LO8hs8/r69mNzhOC9XxRDiHM6y1aPlHpkQgCXNhY5mdPXdqo4vdRH61N6ktg1
mwWZ3vtr0gTqCD8Ir7sVlI0TBbp4Ztl8sqHT0UOhDkwC0aWpDc0MYxg+mMv2+Cmw
wsGcVR52Nm61pryEOTJrdQL5iX7LxBdjuSB+GR+y1nwOevsQlzhrBrad42HiIgE3
WKKPWVKuInq3vOD+ZU/QKP/JPzXbsop4s/cJC6Fj+T8rnO4QPAbqDrMjdAYlmz+R
ntjKg4Nwe8WEVKwV7ftzeeEb8iLBjm+5qbFysMoumMo4D2om9DX3mCmEdzz8HASe
5Xw0vkNJI3AgjNRyU462ybrV+ZtYDiM3e71PiOvf/MKyiORzEZ0OQxtpdd9KHChJ
aAjwtfKgz0H/ZJoQdArtqjhQTTHXEOBR0S6t32MBSsuLbuStdQ5Jbi1eG7KrKvkw
mkRLlupzqShncZArNFLDIatVazG3Vx3Bav//CxXM28A0bm8YgnQkz3p6NIf6SSOY
GTayCQbOZGG7SilV34dA/F8zwN4W5v4baL30CwLkQY1MDD4+xeY=
=kKYX
-----END PGP SIGNATURE-----

Re: Matching streaming services

[Reindl Harald]

Am 06.01.21 um 19:05 schrieb Nikolai Lusan:
> Hi,
> 
> I have been looking for a way to differentiate traffic from streaming
> services (Netflix, Amazon Prime, <insert_locally_available_service>)
> from other https traffic, with not much luck. The goal is to add rules
> to nftables and tc to ensure quality while allowing the rest of the
> link to function normally.
> 
> I tried using tcpdump to see if there was something in the packets that
> I could use, but they look like any other bit of https traffic.
> 
> Does anyone have a method for determining which http/https traffic is
> streaming video, and which is not?

the point of https is to clap on dirty fingers of anyone in the middle 
of the connection, no matter if his intention is good or bad

if you can distinct the content of https traffic we have a problem houston

Re: Matching streaming services

[david]

On Mikrotik routers there is possibility of burst rate setting that 
determines whether it is simple web surfing or continuos data stream.

I think it is possible also in iptables and may be also in 
nftables...unsure, whether you need some additional modules or not.


On 06/01/2021 19:05, Nikolai Lusan wrote:
> Hi,
>
> I have been looking for a way to differentiate traffic from streaming
> services (Netflix, Amazon Prime, <insert_locally_available_service>)
> from other https traffic, with not much luck. The goal is to add rules
> to nftables and tc to ensure quality while allowing the rest of the
> link to function normally.
>
> I tried using tcpdump to see if there was something in the packets that
> I could use, but they look like any other bit of https traffic.
>
> Does anyone have a method for determining which http/https traffic is
> streaming video, and which is not?
>
> Thanks
 >

Re: Matching streaming services

[Reindl Harald]

Am 06.01.21 um 19:44 schrieb david@hajes.org:
> On Mikrotik routers there is possibility of burst rate setting that 
> determines whether it is simple web surfing or continuos data stream.

and how do you imagine distinct between a large download which can 
finished one hour later and nobody cares or streaming?

and whenever you manage it - it's something that need to be fixed and 
changed ASAP sou will have a moving target

> I think it is possible also in iptables and may be also in 
> nftables...unsure, whether you need some additional modules or not.
> 
> 
> On 06/01/2021 19:05, Nikolai Lusan wrote:
>> Hi,
>>
>> I have been looking for a way to differentiate traffic from streaming
>> services (Netflix, Amazon Prime, <insert_locally_available_service>)
>> from other https traffic, with not much luck. The goal is to add rules
>> to nftables and tc to ensure quality while allowing the rest of the
>> link to function normally.
>>
>> I tried using tcpdump to see if there was something in the packets that
>> I could use, but they look like any other bit of https traffic.
>>
>> Does anyone have a method for determining which http/https traffic is
>> streaming video, and which is not?

Re: Matching streaming services

[pauloric]

----- Mensagem original -----
De: "Reindl Harald" <h.reindl@thelounge.net>
Para: david@hajes.org, "netfilter" <netfilter@vger.kernel.org>
Enviadas: Quarta-feira, 6 de janeiro de 2021 16:15:18
Assunto: Re: Matching streaming services

Am 06.01.21 um 19:44 schrieb david@hajes.org:
> On Mikrotik routers there is possibility of burst rate setting that 
> determines whether it is simple web surfing or continuos data stream.

and how do you imagine distinct between a large download which can 
finished one hour later and nobody cares or streaming?

and whenever you manage it - it's something that need to be fixed and 
changed ASAP sou will have a moving target

> I think it is possible also in iptables and may be also in 
> nftables...unsure, whether you need some additional modules or not.
> 
> 
> On 06/01/2021 19:05, Nikolai Lusan wrote:
>> Hi,
>>
>> I have been looking for a way to differentiate traffic from streaming
>> services (Netflix, Amazon Prime, <insert_locally_available_service>)
>> from other https traffic, with not much luck. The goal is to add rules
>> to nftables and tc to ensure quality while allowing the rest of the
>> link to function normally.
>>
>> I tried using tcpdump to see if there was something in the packets that
>> I could use, but they look like any other bit of https traffic.
>>
>> Does anyone have a method for determining which http/https traffic is
>> streaming video, and which is not?


Humm well as https is a application you could use squid + bump + delay pools (MITM), but it is out of nftables...

Re: Matching streaming services

[david]

> and how do you imagine distinct between a large download which can 
> finished one hour later and nobody cares or streaming?
>
> and whenever you manage it - it's something that need to be fixed and 
> changed ASAP sou will have a moving target
>
Streaming service never runs flat out, you have to observe data rates. 
Amazon max. data rate is barely 20Mbps for example.

Whereas data download usually saturates whole line. It is not perfect 
QoS but it worked on 500Mbps cable Internet for me.

Mikrotik also have so called "Layer 7" filtering designed specially for 
filtering per service requests that requires lots of resources to 
inspect packets.

I think netfilter allows such filtering as well.

Re: Matching streaming services

[Nikolai Lusan]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Wed, 2021-01-06 at 19:18 +0100, Reindl Harald wrote:
> the point of https is to clap on dirty fingers of anyone in the
> middle 
> of the connection, no matter if his intention is good or bad

My initial thinking was that the https port was just being used, and
not that it was actually https traffic, although this seems not to be
the case (verifying would require a deeper dive into the packet stream
than I have time for right now).


> if you can distinct the content of https traffic we have a problem
> houston

I agree - but it doesn't stop people using port 443 for other data
transmission, for example ssh on port 443 often allows you to bypass
proxies or overly strict firewalls. Just because it's "reservered" as a
port for secure http transmission doesn't mean that's what it's being
used for. I suspected it might be sctp traffic, but again I can't
verify anything with my current time constraints.

- -- 
Nikolai Lusan <nikolai@lusan.id.au>
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEVfd4GW6z4nsBxdLo4ZaDRV2VL6QFAl/2EXMACgkQ4ZaDRV2V
L6QsWBAAkmb5NgIkNxDqWGM9EAqyd+DvtPADcix4Wq4hdKw1qdW2BJyWq5xgBvCh
ty9MzXkkiqeuOL/vr82srZgAIjZJ3Y7gJ2pozanYLEc6celCGHK1J9ElUFvPijF+
z/W/yObxJPUJ5oFHhHr8VBPSF6fmhgKBAgNQaczErrhUyRu8sMElFXun4mRKTbsX
RKx4mWFF/lQbQ9p2OMngvJAZRl+Zyxxyu7enFjrzX4IUWhBeznhj/3x3AR4dN7om
vhO9gYgQ5uam9/tWURfUhGdBs80DFLrMJMAGoPYfBqLOgzzo6RpBBUyu2A7KtOKx
KKlkc2vPBy8mrAQLB0+JvtyY+xU759wp5ItNPAurdt17i6yh6x/UKNLFOUiVeyZE
zASBUrw5B/1SwBwtHqiSqlEDiqKU1g7zr+oEbBT8tJot3CT+Kzz+QlqGPa8YykDU
Q/mAkTb/kdur4Yil5scpIxBRf+ISCiwZC5gXGo94HkNtHWeLHNWCv/NVLNSbWQb9
5LsGAyXQxA9cmKT78OX4rbGq6iJzZLLKgEi2a0EDAQCy5ZD0y4aGyl91SnHYSmoj
1q9IszexoxC72zqnL7BDpDFSz2RAFoeA7QIrJ5Nq1DxwXw524YOSipqJTrnpeX3/
Trvy610Wf7XA1PtfGfnh2iGC2ETi/Pai/XqWfjlg1ujnROHmwIA=
=GHKy
-----END PGP SIGNATURE-----

RE: Matching streaming services

[Eliezer Croitoru]

Hey Nikolai,

Take a peek at:
* https://github.com/vel21ripn/nDPI
* https://github.com/elico/debian10-dev-ndpi-vel

It works ontop of Debian buster and couple others.
Even if you will not use this you might find in the code how they identify or try to identify specific services.

If you have control on the local DNS service you might be able to identify some of these dynamically.

Eliezer

----
Eliezer Croitoru
Tech Support
Mobile: +972-5-28704261
Email: ngtech1ltd@gmail.com
Zoom: Coming soon


-----Original Message-----
From: Nikolai Lusan <nikolai@lusan.id.au> 
Sent: Wednesday, January 6, 2021 8:06 PM
To: netfilter@vger.kernel.org
Subject: Matching streaming services

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi,

I have been looking for a way to differentiate traffic from streaming
services (Netflix, Amazon Prime, <insert_locally_available_service>)
from other https traffic, with not much luck. The goal is to add rules
to nftables and tc to ensure quality while allowing the rest of the
link to function normally.

I tried using tcpdump to see if there was something in the packets that
I could use, but they look like any other bit of https traffic.

Does anyone have a method for determining which http/https traffic is
streaming video, and which is not?

Thanks
- -- 
Nikolai Lusan <nikolai@lusan.id.au>
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEVfd4GW6z4nsBxdLo4ZaDRV2VL6QFAl/1/AYACgkQ4ZaDRV2V
L6Qtaw/9EaJHbOuar8d8s4FyKU3JvJxlV0SAPsHSoSGHWFZ3j7eYj+A8SZ8vylni
AGzZ0SbstVcpW7BvnjHfo7RV1rrqdM1dTxXGecXfVuktFWg6PyWcICrUqnwo9Zkd
bQ3LO8hs8/r69mNzhOC9XxRDiHM6y1aPlHpkQgCXNhY5mdPXdqo4vdRH61N6ktg1
mwWZ3vtr0gTqCD8Ir7sVlI0TBbp4Ztl8sqHT0UOhDkwC0aWpDc0MYxg+mMv2+Cmw
wsGcVR52Nm61pryEOTJrdQL5iX7LxBdjuSB+GR+y1nwOevsQlzhrBrad42HiIgE3
WKKPWVKuInq3vOD+ZU/QKP/JPzXbsop4s/cJC6Fj+T8rnO4QPAbqDrMjdAYlmz+R
ntjKg4Nwe8WEVKwV7ftzeeEb8iLBjm+5qbFysMoumMo4D2om9DX3mCmEdzz8HASe
5Xw0vkNJI3AgjNRyU462ybrV+ZtYDiM3e71PiOvf/MKyiORzEZ0OQxtpdd9KHChJ
aAjwtfKgz0H/ZJoQdArtqjhQTTHXEOBR0S6t32MBSsuLbuStdQ5Jbi1eG7KrKvkw
mkRLlupzqShncZArNFLDIatVazG3Vx3Bav//CxXM28A0bm8YgnQkz3p6NIf6SSOY
GTayCQbOZGG7SilV34dA/F8zwN4W5v4baL30CwLkQY1MDD4+xeY=
=kKYX
-----END PGP SIGNATURE-----

Re: Matching streaming services

[Reindl Harald]

Am 06.01.21 um 20:37 schrieb Nikolai Lusan:
> On Wed, 2021-01-06 at 19:18 +0100, Reindl Harald wrote:
>> the point of https is to clap on dirty fingers of anyone in the
>> middle
>> of the connection, no matter if his intention is good or bad
> 
> My initial thinking was that the https port was just being used, and
> not that it was actually https traffic, although this seems not to be
> the case (verifying would require a deeper dive into the packet stream
> than I have time for right now).
> 
> 
>> if you can distinct the content of https traffic we have a problem
>> houston
> 
> I agree - but it doesn't stop people using port 443 for other data
> transmission, for example ssh on port 443 often allows you to bypass
> proxies or overly strict firewalls. Just because it's "reservered" as a
> port for secure http transmission doesn't mean that's what it's being
> used for. I suspected it might be sctp traffic, but again I can't
> verify anything with my current time constraints.

don't change the fact that you have no business to mangle around in 
encrypted traffic - that's the whole point of encryption