[LARTC] Intelligent P2P detection

[LARTC] Intelligent P2P detection

[Luman]

Probably, I'm not the first one who needs solve problem with p2p.
Because, large part of my traffic is eaten by p2p software like KazAA,
e-mule, Direct Connect etc, I'm looking for the way of detection of such
traffic and marking it. However simple way with for instance 1214 port
for KazAA doesn't work because this software uses floating port
technology. This traffic can be send via different ports and these ports
can change in the fly. This is rather well known. 
So I'm looking for the stuff working at higher level and analyzing
traffic inside to determine the content and the real protocol. It could
be a patch to the kernel or whatever. It should only be able to mark
packet by a special marker. 

I need this solution not only to prioritizing the traffic (prioritizing
can be achieve in other way) but also to selection the Internet link. I
want to NAT this low quality data for some specific address in order to
send it over cheaper link. 

What do you think is there any solution to do it? Or maybe there is
ongoing project trying to tackle with this global problem with detection
p2p traffic.

Best regards,
Luman




_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Intelligent P2P detection

[Kim Jensen]

Hi Luman,

Sorry, I have a stupid question - why do you wish to mark the traffic, is it 
because you wish to allow it internally or is it because you wish to 
completely drop it?

If you wish to mark it, then I can recommend that you take a look at the 
patch-o-matic module connmark. All packets destined for your internal 
services or external services where the destination port is above 1024 can 
then easily be marked and tracked (at least in theory, I haven't played with 
the connmark module yet).

/Kim

On Tuesday 25 March 2003 09:08, Luman wrote:
> Probably, I'm not the first one who needs solve problem with p2p.
> Because, large part of my traffic is eaten by p2p software like KazAA,
> e-mule, Direct Connect etc, I'm looking for the way of detection of such
> traffic and marking it. However simple way with for instance 1214 port
> for KazAA doesn't work because this software uses floating port
> technology. This traffic can be send via different ports and these ports
> can change in the fly. This is rather well known.
> So I'm looking for the stuff working at higher level and analyzing
> traffic inside to determine the content and the real protocol. It could
> be a patch to the kernel or whatever. It should only be able to mark
> packet by a special marker.
>
> I need this solution not only to prioritizing the traffic (prioritizing
> can be achieve in other way) but also to selection the Internet link. I
> want to NAT this low quality data for some specific address in order to
> send it over cheaper link.
>
> What do you think is there any solution to do it? Or maybe there is
> ongoing project trying to tackle with this global problem with detection
> p2p traffic.
>
> Best regards,
> Luman
>
>
>
>
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

RE: [LARTC] Intelligent P2P detection

[Luman]

Hi Kim,

There are two reasons why I want to detect this sort of traffic. 
1. I want to prioritize traffic
2. I want to push low quality traffic via cheaper links. 

To explain them, I need to say something about my architecture.
In general, my network architecture is divided into two layers. The
first client-routers and the second border-routers. The client-routers
(I have many them) realize connection between my backbone with my
clients. The border-routers are connection between my backbone and the
rest of the world. I have couple them.

Obviously, clients kill my peering connection, by sending low level
quality traffic, generated by p2p applications like KaZaA, DC, e-mule
etc. I would like at the client-routers level to detect the particular
packets generated by p2p soft and mark them in someway. These marked
packets are sent to one of the border-router which checks if the packet
is marked. If it is, the packets have low-priority. (This is the first
problem - prioritizing traffic).

However, I have expensive high quality internet links, and I want to
avoid wasting them for that traffic. In this case if packet is marked,
then it is sent to the other border-router, which operates on cheaper
link. Such traffic is NATed into cheaper link IP address (to avoid
returning via better link - BGP is working) and is sent with the lowest
priority. (the second problem - selection cheaper link)

This is my problem. Actually, I realized everything without one
exception, marking all packets p2p software. I can easy mark packets
using iptables by detecting dst-port 1214 (for instance) but p2p
software are smart. So I need something at higher level, which analysis
content of packets and take the decision not only on the port but basing
on characteristics of sent data inside packets. 


Sorry, for so long text. 
Best regards,
Luman


>-----Original Message-----
>From: lartc-admin@mailman.ds9a.nl [mailto:lartc-admin@mailman.ds9a.nl]
On
>Behalf Of Kim Jensen
>Sent: Tuesday, March 25, 2003 11:22 AM
>To: Luman; lartc@mailman.ds9a.nl
>Subject: Re: [LARTC] Intelligent P2P detection
>
>Hi Luman,
>
>Sorry, I have a stupid question - why do you wish to mark the traffic,
is
>it
>because you wish to allow it internally or is it because you wish to
>completely drop it?
>
>If you wish to mark it, then I can recommend that you take a look at
the
>patch-o-matic module connmark. All packets destined for your internal
>services or external services where the destination port is above 1024
can
>then easily be marked and tracked (at least in theory, I haven't played
>with
>the connmark module yet).
>
>/Kim
>
>On Tuesday 25 March 2003 09:08, Luman wrote:
>> Probably, I'm not the first one who needs solve problem with p2p.
>> Because, large part of my traffic is eaten by p2p software like
KazAA,
>> e-mule, Direct Connect etc, I'm looking for the way of detection of
such
>> traffic and marking it. However simple way with for instance 1214
port
>> for KazAA doesn't work because this software uses floating port
>> technology. This traffic can be send via different ports and these
ports
>> can change in the fly. This is rather well known.
>> So I'm looking for the stuff working at higher level and analyzing
>> traffic inside to determine the content and the real protocol. It
could
>> be a patch to the kernel or whatever. It should only be able to mark
>> packet by a special marker.
>>
>> I need this solution not only to prioritizing the traffic
(prioritizing
>> can be achieve in other way) but also to selection the Internet link.
I
>> want to NAT this low quality data for some specific address in order
to
>> send it over cheaper link.
>>
>> What do you think is there any solution to do it? Or maybe there is
>> ongoing project trying to tackle with this global problem with
detection
>> p2p traffic.
>>
>> Best regards,
>> Luman
>>
>>
>>
>>
>> _______________________________________________
>> LARTC mailing list / LARTC@mailman.ds9a.nl
>> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO:
http://lartc.org/
>
>_______________________________________________
>LARTC mailing list / LARTC@mailman.ds9a.nl
>http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Intelligent P2P detection

[Kim Jensen]

Hi Luman,

I understand that you have a complex setup. But as I lack knowledge about 
Kazaa or other similar P2P services, I don't know if it is possible to say 
that these should only be allowed on certain ports internally. It should be 
possible to configure these so they only use a well known port range - or am 
I completely wron here?

/Kim

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

RE: [LARTC] Intelligent P2P detection

[Luman]

Thank you Thomas for your answer. Your voice is very helpful to me and
uncovers other problems, and makes me realize I'm at the beginning of
the path to solve it. 

Your right, that to solve this problem in general I need an application
level proxy. But first of all I need to detect the traffic, even I would
have a proxy I need to detect traffic. However, analysis of source and
destination ports is not enough in this case. 

Additionally, even if I couldn't solve problem with masquerading of all
protocols, some of them would be useful. Eventually, if I well detect
type of the traffic I can prioritize it on the more expensive link.

Thank you for answer again, and I think that similar problems with p2p
software have many people, who should be able to full control the
traffic and to do anything with it what they want. I don't know if there
is a good solution for this problem now, but this is a good idea to
enhance Linux possibility of routing in the near future, because this
problem is growing.

Regards,
Luman 


>-----Original Message-----
>From: Thomas Lussnig [mailto:lussnig@smcc.net]
>Sent: Tuesday, March 25, 2003 2:29 PM
>To: Luman
>Cc: 'Kim Jensen'; lartc@mailman.ds9a.nl
>Subject: Re: [LARTC] Intelligent P2P detection
>
>Hi,
>i think you will run at least into two problems.
>1. You need and aplication level tool that  will tag the packets
>2. You can run with the masquerading into an problem since some P2P
>software need to setup if they run behind masq.
>3. Even if you fixed 1+2 you have not win
>For eDonkey protocol i know that the back connection is not fixed to
the
>sender ip. Because eDonkey
>support proxy connections it tell in the protocol on witch IP/Port the
>answer (and file data) should be send.
>That mean you have to do it real 100%, you need and transparent
>aplication layer proxy.
>
>Other option is to give the p2p Clients own IP that are routed via the
>cheap geateway.
>Routing should not the problem since the packets are masqed you can use
>source routing.
>And for  prio it depend if you router support TOS based routing. BUT
>than you need to
>take care that the clients do not send TOS with high_troughput on
>unrecognized p2p conections.
>
>Cu Thomas


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

RE: [LARTC] Intelligent P2P detection

[Luman]

Thanks, your proposition is quite good. I must think about it more.
However, p2p soft can transit data over well known ports for instance
smtp but I can't block them. So we are a the beginning point. We need to
analyze the content of the data.

Luman

>-----Original Message-----
>From: Kim Jensen [mailto:kimj@dawn.dk]
>Sent: Tuesday, March 25, 2003 1:21 PM
>To: Luman; lartc@mailman.ds9a.nl
>Subject: Re: [LARTC] Intelligent P2P detection
>
>Hi Luman,
>
>I understand that you have a complex setup. But as I lack knowledge
about
>Kazaa or other similar P2P services, I don't know if it is possible to
say
>that these should only be allowed on certain ports internally. It
should be
>possible to configure these so they only use a well known port range -
or
>am
>I completely wron here?
>
>/Kim


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Intelligent P2P detection

[Matthias Weingart]

On Tue, Mar 25, 2003 at 11:21:34AM +0100, Kim Jensen wrote:

> services or external services where the destination port is above 1024 can 
> then easily be marked and tracked (at least in theory, I haven't played with 

This does not help, because often they use port 80 or 25 or so. But e-mule
is using some fixed ports (4426 or similar?) as control connection (like
passive ftp). A module that is analysing that control traffic (similar to
ftp) and determine the ports that are then used by the transfer connections
should be able to mark all e-mule packets; but I have not seen a
implementation of this yet. However there are sources of P2P servers/clients
available, and it should be possible to implement such a tool or I should
better say different tools for different P2P-software ;-(. (and maybe not
for all kind of that software).  I agree with you, it would be fine to have
such a tool.

        Matthias
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

RE: [LARTC] Intelligent P2P detection

[Robert Kryczało]

Hi,

I think that writing custom conntrack helper module which can be used to
mark packets created by p2p software is the only solution acceptable in my
case. I am also thinking about purchasing cheap E1 link and use it for p2p
trafic.

Two months ago I was thinking about writing such a module. Anyway, I didn't
do anything because of lack of time and knowledge.

I would like to be informed about your progress though. I also offer my help
with testing. I think described module would be very usefull for large
number of netadmins.

RK

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

RE: [LARTC] Intelligent P2P detection

[Luman]

Yes, if there isn't any proper tool already, the conntrack could be a
good template for the beginning.

Luman 

>-----Original Message-----
>From: Robert Krycza³o [mailto:robert.kryczalo@iscnet.pl]
>Sent: Tuesday, March 25, 2003 1:51 PM
>To: Luman; 'Kim Jensen'; lartc@mailman.ds9a.nl
>Subject: RE: [LARTC] Intelligent P2P detection
>
>Hi,
>
>I think that writing custom conntrack helper module which can be used
to
>mark packets created by p2p software is the only solution acceptable in
my
>case. I am also thinking about purchasing cheap E1 link and use it for
p2p
>trafic.
>
>Two months ago I was thinking about writing such a module. Anyway, I
didn't
>do anything because of lack of time and knowledge.
>
>I would like to be informed about your progress though. I also offer my
>help
>with testing. I think described module would be very usefull for large
>number of netadmins.
>
>RK


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

RE: [LARTC] Intelligent P2P detection

[Robert Kryczało]

>Yes, if there isn't any proper tool already, the conntrack could be a
>good template for the beginning.

Some nice features to think about:
- ability to mark different p2p software (kazaa,dc,e-mule,WinMX etc.)
- ability to mark new connections
- ability to limit sessions (trough -m limit).

RK

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

RE: [LARTC] Intelligent P2P detection

[Robert Kryczało]

Hi,
>Yes, if there isn't any proper tool already, the conntrack could be a
>good template for the beginning.

Things worth considering are also helper.patch and CONNMARK.patch for per
connection based marking.

Luman

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Intelligent P2P detection

[Dawid Kuroczko]

On Tue, 25 Mar 2003, Luman wrote:

> I need this solution not only to prioritizing the traffic (prioritizing
> can be achieve in other way) but also to selection the Internet link. I
> want to NAT this low quality data for some specific address in order to
> send it over cheaper link. 

I do not know if it might help you, but here's my "temporary hack quality"
solution.

Assumptions:
  Determine and mark 'good traffic' -- i.e. smtp, ftp, http, ssh, etc.,
  everything which uses well known ports.  Probably most people do it
  anyway, at least to some level.

  All what is left are P2P connections and some other misc connections.
  A bit unfair for other protocol using non-standard ports, like Instant
  Messenger style-software, and lots of other stuff.  So here we introduce
  a trick.  IMs and other low bandwidth traffic will use small packets
  ( < 512 or even < 256), P2P will use maximum MTA available (usually
  1500, but I've seen some using 576 packets, hence i treat > 512 as P2P).

  Probably you've notices that I mention round numbers, as 512 or 1024,
  that's because I use u32 for marking packets.  How I do it, we leave
  as an exercise to the reader. ;-)))

  Regards,
    Dawid
-- 
  ..................        ``The essence of real creativity is a certain
 : *Dawid Kuroczko* `.         playfulness, a flitting from idea to idea
 : q n e x@knm.org.pl :     without getting bogged down by fixated demands.''
 `....................'  Sherkaner Underhill, A Deepness in the Sky, V. Vinge

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

RE: [LARTC] Intelligent P2P detection

[Dawid Kuroczko]

On Tue, 25 Mar 2003, Robert Krycza³o wrote:

> >Yes, if there isn't any proper tool already, the conntrack could be a
> >good template for the beginning.
> Some nice features to think about:
> - ability to mark different p2p software (kazaa,dc,e-mule,WinMX etc.)
> - ability to mark new connections
> - ability to limit sessions (trough -m limit).

A suggestion.  Something which works as more advanced "string" match.
But instead of a string, we use a "pattern".  Say, something like this:

-p tcp -m pattern --pattern "PORT %Sd, %Dd" --set ftpsession

-p tcp -m pattern --get ftpsession -j MARK ...

...first would look for pattern "PORT %d, %d", first being source
port (hence: %S), second destination port (hence: %D) and if such
pattern is found, it is added to a ftpsession list (similar to
ipt_recent).

Second searches the ftpsession list for such and such ports connection
and if found it answers it's OK. :-)

...pattern matching should accept \077 style "binary" strings, and
should not be limited to ascii-decimal "%d" port numbers.  Also
binary forms, in any order.  And even maybe IPs. :-)))  Simple
yet powerful...

...[ so we code it, and some time passes and then we read announcement
that KaZaA released new version which mimicks HTTP and uses strong
cryptography to circumvent our module...  Hopefully it will not come
to pass, but well... :-)

   Regards,
      Dawid
-- 
                 http://www.muppetlabs.com/~breadbox/bf/
+++++[>++++++<-]>++[>+>++>+++>++++<<<<-]>>++++.>+.>---------.<++++++++.--
---.<<.>+++++++.>>--.---.---.<-.>+++++++++++.<++++++++.++++.>>++++++++++.

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

RE: [LARTC] Intelligent P2P detection

[Luman]

>Assumptions:
>  Determine and mark 'good traffic' -- i.e. smtp, ftp, http, ssh, etc.,
>  everything which uses well known ports.  Probably most people do it
>  anyway, at least to some level.

The problem is with that currently P2P soft often use these well known
ports too. So the assumption that port 80 is only for HTTP is bad. 

The intention of the bringing forward my problem is to open wider
discussion aimed to find or if need be to build a "tool" (it might be a
kernel patch, or whatever), which can thoroughly analyze traffic and its
content, and on the base of it can take a decision (likely not with 100%
likelihood) what is the content. For instance it can detect that the
traffic is HTTP even if it is sent to 46723 port, basing on the content
of data. Such tool should based on a modular architecture allowing
adding new testers or new knowledge trying to guess the protocol.
Obviously, it should track connections, session and everything what can
be used to traffic classification. 
As the result packets would be marked by a standardized number
determining type of a protocol, for instance HTTP, KaZaa, MSN, SSH, SCP
etc. This knowledge could be used to traffic shaping and whatever. Can
you imaging the comfort of administrators' work if at the border router,
or at the firewall configuration, can work with this well determined
content. Number or rules would be reduced dramatically. Obviously, the
classification knowledge would be growing day by day.

Whole idea is very similar to Unix 'file' command. For instance I had on
my system "a.gz" file. The type of this file is obvious this is gzip.
However, I changed its name to "a.txt". It should suggest that this is
text file, however, when I run file a.txt I get the fallowing answer:
~# file a.txt 
a.txt: gzip compressed data, deflated, original filename,
`ucspi-tcp-0.88.tar', last modified: Sat Mar 18 16:21:39 2000, max
compression, os: Unix.
This program doesn't care about extensions it tries to guess the type by
analyzing content. Of course many times it gives wrong answer, but that
is related to weak of knowledge. 

Summarizing my pretty long mail, I think our present methods are similar
to determining the content of file basing only on extension of its name.
But I believe we strongly require something more. 

>
>  All what is left are P2P connections and some other misc connections.
>  A bit unfair for other protocol using non-standard ports, like
Instant
>  Messenger style-software, and lots of other stuff.  So here we
introduce
>  a trick.  IMs and other low bandwidth traffic will use small packets
>  ( < 512 or even < 256), P2P will use maximum MTA available (usually
>  1500, but I've seen some using 576 packets, hence i treat > 512 as
P2P).
>
>  Probably you've notices that I mention round numbers, as 512 or 1024,
>  that's because I use u32 for marking packets.  How I do it, we leave
>  as an exercise to the reader. ;-)))


I like your solution very much. I'll try to apply it for my system, as a
temporary solution. 

Thank you for your voice in this discussion.

Best regards,
Luman


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

RE: [LARTC] Intelligent P2P detection

[Luman]

>-----Original Message-----
>From: Dawid Kuroczko [mailto:qnex@atlantis.ssw.krakow.pl]
>Sent: Wednesday, March 26, 2003 10:50 PM
>To: Robert Krycza³o
>Cc: Luman; 'Kim Jensen'; lartc@mailman.ds9a.nl
>Subject: RE: [LARTC] Intelligent P2P detection
>


[...]
>
>A suggestion.  Something which works as more advanced "string" match.
>But instead of a string, we use a "pattern".  Say, something like this:
>
>-p tcp -m pattern --pattern "PORT %Sd, %Dd" --set ftpsession
>
>-p tcp -m pattern --get ftpsession -j MARK ...
>
>...first would look for pattern "PORT %d, %d", first being source
>port (hence: %S), second destination port (hence: %D) and if such
>pattern is found, it is added to a ftpsession list (similar to
>ipt_recent).
>
>Second searches the ftpsession list for such and such ports connection
>and if found it answers it's OK. :-)
>
>...pattern matching should accept \077 style "binary" strings, and
>should not be limited to ascii-decimal "%d" port numbers.  Also
>binary forms, in any order.  And even maybe IPs. :-)))  Simple
>yet powerful..
Yes, it could be. But I think, we need more, something like rule based
expert system, deciding on many factors. As the result, it takes a
decision, what is the content. 

>
>...[ so we code it, and some time passes and then we read announcement
>that KaZaA released new version which mimicks HTTP and uses strong
>cryptography to circumvent our module...  Hopefully it will not come
>to pass, but well... :-)

Even yes, I believe that we can find some pattern in that kind of
traffic, which helps us to determine that this is KaZaa, even it looks
like HTTP. This is what I tried to uncover in my previous mail.

Best regards,
Luman


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Intelligent P2P detection

[Gordan Bobic]

On Thursday 27 Mar 2003 09:24, Luman wrote:
> >Assumptions:
> >  Determine and mark 'good traffic' -- i.e. smtp, ftp, http, ssh, etc.,
> >  everything which uses well known ports.  Probably most people do it
> >  anyway, at least to some level.
>
> The problem is with that currently P2P soft often use these well known
> ports too. So the assumption that port 80 is only for HTTP is bad.

Yes, but you can then analyze the traffic just on those ports for
verification. HTTP patkets have a certain anatomy to them. This can be
detected. Same goes for FTP, SMTP, etc. The problems start with HTTPS, SSH,
IMAPS, POP3S, etc, as they are encrypted, and therefore cannot be analyzed.
With those, you simply have a problem. However, for SSH, IMAPS and POP3S you
don't need lots of bandwidth. You could therefore throttle them to
low-latency low-bandwidth. P2P networks will not like this.

SFTP runs over SSH so you may have a problem with that. HTTPS is also
problematic.

However, you can scan to verify if SSH/HTTPS is being used. You can simply
write a bot that scans the ports when your router detects traffic. It can
send valid SSH/HTTPS connection requests and see if it talks back as
SSH/HTTPS should.

Unfortunately, it gets progressively more difficult when P2P clients learn to
masquerade as the real protocols, and there is at least one P2P application
out there that can operate over SMTP, sending valid requests. :-(

I hope you are prepared to accept that eventually it all comes down to a
battle of wills between the sysadmins writing filters and the P2P developers
finding more ways to outsmart the filters.

> The intention of the bringing forward my problem is to open wider
> discussion aimed to find or if need be to build a "tool" (it might be a
> kernel patch, or whatever), which can thoroughly analyze traffic and its
> content, and on the base of it can take a decision (likely not with 100%
> likelihood) what is the content. For instance it can detect that the
> traffic is HTTP even if it is sent to 46723 port, basing on the content
> of data.

How do you deal with HTTPS/SFTP/SSH/IMAPS/POP3S? Automatically do a man in
 the middle attack on everything, at your router?

> Such tool should based on a modular architecture allowing
> adding new testers or new knowledge trying to guess the protocol.

This can generally be done for unencrypted connections, but once things start
to run over SSL (some already do), the chances of "recognizing" traffic very
soon become adjusted to zero. :-(

> Obviously, it should track connections, session and everything what can
> be used to traffic classification.

In order to write a rule for traffic analysis, you must first know what to
look for. If you can come up with a method to analyze SSL traffic, especially
in real-time (or close to), I am sure a lot of people in the security
industry will want to hear from you.

> As the result packets would be marked by a standardized number
> determining type of a protocol, for instance HTTP, KaZaa, MSN, SSH, SCP
> etc.

If you can get as far as distinguishing packets, then that's great. How to
 get that far is the difficult part.

> This knowledge could be used to traffic shaping and whatever. Can
> you imaging the comfort of administrators' work if at the border router,
> or at the firewall configuration, can work with this well determined
> content. Number or rules would be reduced dramatically. Obviously, the
> classification knowledge would be growing day by day.

Sounds great, in theory.

> Whole idea is very similar to Unix 'file' command. For instance I had on
> my system "a.gz" file. The type of this file is obvious this is gzip.
> However, I changed its name to "a.txt". It should suggest that this is
> text file, however, when I run file a.txt I get the fallowing answer:
> ~# file a.txt
> a.txt: gzip compressed data, deflated, original filename,
> `ucspi-tcp-0.88.tar', last modified: Sat Mar 18 16:21:39 2000, max
> compression, os: Unix.
> This program doesn't care about extensions it tries to guess the type by
> analyzing content. Of course many times it gives wrong answer, but that
> is related to weak of knowledge.

Your knowledge of encrypted traffic is non-existant. That is the whole point
of encryption. How do you disallow valid encrypted traffic while allowing P2P
encrypted traffic? What happens when the method of using SMTP for P2P becomes
more wide spread? You can send perfectly valid looking emails - that are PGP
encrypted, with all the SMTP headers in place to make them indistinguishable
from real PGP encrypted email.

> Summarizing my pretty long mail, I think our present methods are similar
> to determining the content of file basing only on extension of its name.
> But I believe we strongly require something more.

And that will work until encryption is used. As Most P2P networks are now
starting to use encryption on the connectiong streams, this very quickly
becomes unworkable.

> >  All what is left are P2P connections and some other misc connections.
> >  A bit unfair for other protocol using non-standard ports, like
> >  Instant  Messenger style-software, and lots of other stuff.  So here we
> >  introduce
> >  a trick.  IMs and other low bandwidth traffic will use small packets
> >  ( < 512 or even < 256), P2P will use maximum MTA available (usually
> >  1500, but I've seen some using 576 packets, hence i treat > 512 as
> > P2P).
> >
> >  Probably you've notices that I mention round numbers, as 512 or 1024,
> >  that's because I use u32 for marking packets.  How I do it, we leave
> >  as an exercise to the reader. ;-)))
>
> I like your solution very much. I'll try to apply it for my system, as a
> temporary solution.

That sounds like an interesting idea, provided you have some real evidence of
this being the case. And this will only work until P2P network software
starts to randomly change packet sizes to obfuscate itself. :-(

But, I guess we have to work with what we have now, and not worry about the
future advancements before they happen. :-)

I hope you will all forgive me for being... restrained (for want of better
word) in my expectations of the success of such network traffic analysis. It
is a depressing subject to talk about. :-(

I cannot help but think that this is also starting to get slightly off-topic
for this mailing list...

Gordan

-------------------------------------------------------

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

RE: [LARTC] Intelligent P2P detection

[Robert Kryczalo]

Hi,

> -----Original Message-----
> From: lartc-admin@mailman.ds9a.nl
> [mailto:lartc-admin@mailman.ds9a.nl]On Behalf Of Dawid Kuroczko
> Sent: Wednesday, March 26, 2003 10:37 PM
> To: Luman
> Cc: lartc@mailman.ds9a.nl
> Subject: Re: [LARTC] Intelligent P2P detection
>
>
> On Tue, 25 Mar 2003, Luman wrote:
>
> > I need this solution not only to prioritizing the traffic (prioritizing
> > can be achieve in other way) but also to selection the Internet link. I
> > want to NAT this low quality data for some specific address in order to
> > send it over cheaper link.
>
> I do not know if it might help you, but here's my "temporary hack quality"
> solution.
>
> Assumptions:
>   Determine and mark 'good traffic' -- i.e. smtp, ftp, http, ssh, etc.,
>   everything which uses well known ports.  Probably most people do it
>   anyway, at least to some level.
Yes. I do it exactly the same way. Quota patch may be of some help, if we
want to limit more agressive users. Time patch is also of some use.
>
>   All what is left are P2P connections and some other misc connections.
>   A bit unfair for other protocol using non-standard ports, like Instant
>   Messenger style-software, and lots of other stuff.  So here we introduce
>   a trick.  IMs and other low bandwidth traffic will use small packets
>   ( < 512 or even < 256), P2P will use maximum MTA available (usually
>   1500, but I've seen some using 576 packets, hence i treat > 512 as P2P).

Prioitizing small TCP packets carying ACK's, SYN's and small payloads is
also a very good idea. Some other trick include prioritizing UDP and ICMP
packets (sometimes with more strict policy, like limiting packets/s or their
sizes). I didn't dive deeper because I was worried about CPU usage.

I use a mix of methods described above. But in some cases shaping systems
like dyband are better. I am looking for a free alternative.


RK

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

RE: [LARTC] Intelligent P2P detection

[Robert Kryczało]

Hi,

> -----Original Message-----
> From: lartc-admin@mailman.ds9a.nl [mailto:lartc-admin@mailman.ds9a.nl]On
> Behalf Of Gordan Bobic
> Sent: Thursday, March 27, 2003 11:17 AM
> To: lartc@mailman.ds9a.nl
> Subject: Re: [LARTC] Intelligent P2P detection
>

> Unfortunately, it gets progressively more difficult when P2P
> clients learn to
> masquerade as the real protocols, and there is at least one P2P
> application
> out there that can operate over SMTP, sending valid requests. :-(
>
The everlasting battle between creators of swords and shields:) If p2p apps
start to mimick as other protocols and use encription then content based
classificators are of no use.


> I hope you are prepared to accept that eventually it all comes down to a
> battle of wills between the sysadmins writing filters and the P2P
> developers
> finding more ways to outsmart the filters.
Correct. Anyway there are some other solutions.
> That sounds like an interesting idea, provided you have some real
> evidence of
> this being the case. And this will only work until P2P network software
> starts to randomly change packet sizes to obfuscate itself. :-(
I was told that applications doing it exists. I haven't checked it, though.
>
> But, I guess we have to work with what we have now, and not worry
> about the
> future advancements before they happen. :-)
Hehe... yes doing something instead of just talking is a good idea:)
>
> I hope you will all forgive me for being... restrained (for want of better
> word) in my expectations of the success of such network traffic
> analysis. It
> is a depressing subject to talk about. :-(
I think this e-mail is a nice summary. I enjoyed reading it. I could say
that I agree your opinions.

>
> I cannot help but think that this is also starting to get
> slightly off-topic
> for this mailing list...
I think opposite:)

Maybe creating free alternatives to shaping software like those from
www.dyband.com is a way. People using it are very happy actually. They adapt
to network utilization, allow extensive logging, setting different
parameters like max bandwidth, ramps, minimum acceptable rate. The main idea
is to limit aggresive users and give maximum performance and quality
(latency, jitter throughput etc.) to standard users. It looks very well on
paper but I haven't tried dyband yet.... Maybe there is other software like
this I am not aware of.

RK

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Intelligent P2P detection

[Gordan Bobic]

On Thursday 27 Mar 2003 15:32, Robert Krycza³o wrote:

> > Unfortunately, it gets progressively more difficult when P2P
> > clients learn to
> > masquerade as the real protocols, and there is at least one P2P
> > application
> > out there that can operate over SMTP, sending valid requests. :-(
>
> The everlasting battle between creators of swords and shields:) If p2p apps
> start to mimick as other protocols and use encription then content based
> classificators are of no use.

Yup. And this is happening right now...

> > That sounds like an interesting idea, provided you have some real
> > evidence of
> > this being the case. And this will only work until P2P network software
> > starts to randomly change packet sizes to obfuscate itself. :-(
>
> I was told that applications doing it exists. I haven't checked it, though.

I haven't heard of an application that does it, but I have always felt 
reasonably sure that it has either already happened or is about to be 
implemented...

> > But, I guess we have to work with what we have now, and not worry
> > about the
> > future advancements before they happen. :-)
>
> Hehe... yes doing something instead of just talking is a good idea:)

Well, for a little while, anyway, until the new version of the client comes 
out...

> > I hope you will all forgive me for being... restrained (for want of
> > better word) in my expectations of the success of such network traffic
> > analysis. It
> > is a depressing subject to talk about. :-(
>
> I think this e-mail is a nice summary. I enjoyed reading it. I could say
> that I agree your opinions.

Thank you. :-)

> Maybe creating free alternatives to shaping software like those from
> www.dyband.com is a way. People using it are very happy actually. They
> adapt to network utilization, allow extensive logging, setting different
> parameters like max bandwidth, ramps, minimum acceptable rate. The main
> idea is to limit aggresive users and give maximum performance and quality
> (latency, jitter throughput etc.) to standard users. It looks very well on
> paper but I haven't tried dyband yet.... Maybe there is other software like
> this I am not aware of.

I haven't heard about any of them. I am a great believer in "home brewed" 
solutions. :-)

The problem you start getting there is that monitoring and shaping traffic on 
a 100 Mb pipe will take a huge amount of CPU power, and even that will only 
work if the traffic is not encrypted. The only way of attacking the problem I 
can think of is by actually attempting to connect to the client machine on 
the suspiciously used well known ports, and seeing if it works. If it doesn't 
work as expected, you know it's likely to be a P2P application.

I am not sure if you really want to do that, though, as it involves active 
port scanning rather than just monitoring, and some of your customers may 
complain...

Gordan
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

RE: [LARTC] Intelligent P2P detection

[Robert Kryczało]

Hi,
> > > That sounds like an interesting idea, provided you have some real
> > > evidence of
> > > this being the case. And this will only work until P2P
> network software
> > > starts to randomly change packet sizes to obfuscate itself. :-(
> >
> > I was told that applications doing it exists. I haven't checked
> it, though.
>
> I haven't heard of an application that does it, but I have always felt
> reasonably sure that it has either already happened or is about to be
> implemented...
Hehe, I agree:)

>
> > > But, I guess we have to work with what we have now, and not worry
> > > about the
> > > future advancements before they happen. :-)
> >
> > Hehe... yes doing something instead of just talking is a good idea:)
>
> Well, for a little while, anyway, until the new version of the
> client comes
> out...
>
The war without end...

> > I think this e-mail is a nice summary. I enjoyed reading it. I could say
> > that I agree your opinions.
>
> Thank you. :-)
You are welcomed.
>
> > Maybe creating free alternatives to shaping software like those from
> > www.dyband.com is a way. People using it are very happy actually. They
> > adapt to network utilization, allow extensive logging, setting different
> > parameters like max bandwidth, ramps, minimum acceptable rate. The main
> > idea is to limit aggresive users and give maximum performance
> and quality
> > (latency, jitter throughput etc.) to standard users. It looks
> very well on
> > paper but I haven't tried dyband yet.... Maybe there is other
> software like
> > this I am not aware of.
>
> I haven't heard about any of them. I am a great believer in "home brewed"
> solutions. :-)
So am I.
>
> The problem you start getting there is that monitoring and
> shaping traffic on
> a 100 Mb pipe will take a huge amount of CPU power, and even that
> will only
> work if the traffic is not encrypted. The only way of attacking
> the problem I
> can think of is by actually attempting to connect to the client
> machine on
> the suspiciously used well known ports, and seeing if it works.
> If it doesn't
> work as expected, you know it's likely to be a P2P application.
>
> I am not sure if you really want to do that, though, as it
> involves active
> port scanning rather than just monitoring, and some of your customers may
> complain...
Well they will for sure in a scenario described by you. But I think you have
misunderstood me. Dyband don't do any scanning or content analyzing. It
works as a bridge modyfing data rate based on IP addresses. You can set up
complicated scheme of bandwidth sharing. You can even automaticaly limit
some "aggresive users" based on their usage. It happens on the fly and is
very "smooth" from client point of view. If a client doesn't use his
bandwidth for a while the limit raises (recharges). It allows ISP or
enterprise to FULLY (i mean nearly 100%) utilize their uplink. You don't do
provisioning:).

Maybe it is the only reasonable solution....

RK


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Intelligent P2P detection

[Gordan Bobic]

On Thursday 27 Mar 2003 16:38, Robert Krycza³o wrote:

> > The problem you start getting there is that monitoring and
> > shaping traffic on
> > a 100 Mb pipe will take a huge amount of CPU power, and even that
> > will only
> > work if the traffic is not encrypted. The only way of attacking
> > the problem I
> > can think of is by actually attempting to connect to the client
> > machine on
> > the suspiciously used well known ports, and seeing if it works.
> > If it doesn't
> > work as expected, you know it's likely to be a P2P application.
> >
> > I am not sure if you really want to do that, though, as it
> > involves active
> > port scanning rather than just monitoring, and some of your customers may
> > complain...
>
> Well they will for sure in a scenario described by you. But I think you
> have misunderstood me. Dyband don't do any scanning or content analyzing.
> It works as a bridge modyfing data rate based on IP addresses.

Yes, but in order to detect what the traffic is, as the client software starts 
being more clever, you may have to do some pro-active scanning to see whether 
the traffic is legit or not. And even then the client software may fake legit 
server appearance. You would have to mimick the actual P2P connection 
handshake to be sure. And on some of them you have a real problem, e.g. 
FastTrack. They use encrypted connection, and the software is closed-source, 
so it's very difficult to get a handle on cracking the protocol.

> You can set
> up complicated scheme of bandwidth sharing. You can even automaticaly limit
> some "aggresive users" based on their usage. It happens on the fly and is
> very "smooth" from client point of view. If a client doesn't use his
> bandwidth for a while the limit raises (recharges). It allows ISP or
> enterprise to FULLY (i mean nearly 100%) utilize their uplink. You don't do
> provisioning:).
>
> Maybe it is the only reasonable solution....

It sounds like a useful thing to do, but ultimately, you have to detect the 
traffic you want to throttle before you can throttle it. That is where the 
biggest problem is.

Gordan
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Intelligent P2P detection

[Matthias Weingart]

Maybe another way is better. What is the most common of P2P traffic? It
makes much much traffic. So let us catch it there. Monitor the traffic of
one IP and if the traffic within a certain time is high, the bandwidth of
this IP is set down to a lower level automatically (or put in lower priority
queue). After some time - the transfer has finished - it is set back to the
normal level. (I am quite new to qos and HTB and I wonder if this can be done
with linux?). This way you limit all big transfers, nevertheless they are
done by ftp, http or P2P.

I guess it will be _very_ difficult to find and mark all packets of P2P
software (and you will always be behind if new software or new versions are
published).

M.

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Intelligent P2P detection

[Arvid Stüwe]

On Don, 27 Mär 2003, Matthias Weingart wrote:
>Maybe another way is better. What is the most common of P2P traffic? It
>makes much much traffic.

Not really. Well, it depends on your users, if all they do is surfing, you
are right, but not if they are mirroring www.kernel.org.

A better criteria for finding P2P traffic is the number of different IPs
involved. A P2P-Tool usually sends packets to many other hosts (eDonkey and
Overnet in particular). That's how we detect them at our dormitory. Here are
some scripts running here that count the number of IPs a host has sent to
and received from (tcpdump, grep, and some perl). When this number
gets too high too fast, all traffic from that IP gets a special treatment.


>I guess it will be _very_ difficult to find and mark all packets of P2P
>software (and you will always be behind if new software or new versions are
>published).

You don't need *all* packets. You just need to recognize the initial
handshake the programs do to log into the p2p-network. Then you can proceed
by tracking the following packets between the two hosts involved.

cu
	Arvid
-- 
in bunten Bildern wenig Klarheit,
viel Irrtum und ein Fünkchen Wahrheit
	(Johann Wolfgang v. Goethe)
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

RE: [LARTC] Intelligent P2P detection

[Robert Kryczało]

Hi
> >
> > Maybe it is the only reasonable solution....
>
> It sounds like a useful thing to do, but ultimately, you have to
> detect the
> traffic you want to throttle before you can throttle it. That is
> where the
> biggest problem is.
Well in this case w throttle traffic based on IP, IP activity (so aggresive
users are limited on the fly), current uplink utilization etc. In many
sifferent situations this aproach works very well and I am thinking about
shifting to this solution.

RK



_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Intelligent P2P detection

[Szymon Miotk]

Luman wrote:
> Probably, I'm not the first one who needs solve problem with p2p.
> Because, large part of my traffic is eaten by p2p software like KazAA,
> e-mule, Direct Connect etc, I'm looking for the way of detection of such
> traffic and marking it. However simple way with for instance 1214 port
> for KazAA doesn't work because this software uses floating port
> technology. This traffic can be send via different ports and these ports
> can change in the fly. This is rather well known. 
> So I'm looking for the stuff working at higher level and analyzing
> traffic inside to determine the content and the real protocol. It could
> be a patch to the kernel or whatever. It should only be able to mark
> packet by a special marker. 
> 
> I need this solution not only to prioritizing the traffic (prioritizing
> can be achieve in other way) but also to selection the Internet link. I
> want to NAT this low quality data for some specific address in order to
> send it over cheaper link. 
> 
> What do you think is there any solution to do it? Or maybe there is
> ongoing project trying to tackle with this global problem with detection
> p2p traffic.

Snort has set of rules to detect P2P traffic. AFAIK snort is quite fast, 
at least fast enough to cope with 10Mbits on average PC.
Maybe the solution is detecting snort alerts about P2P and automagically 
cutting bandwidth of host playnig with P2P?

Szymon Miotk



_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Intelligent P2P detection

[Patrick McHardy]

Szymon Miotk wrote:

> Luman wrote:
>
>> Probably, I'm not the first one who needs solve problem with p2p.
>> Because, large part of my traffic is eaten by p2p software like KazAA,
>> e-mule, Direct Connect etc, I'm looking for the way of detection of such
>> traffic and marking it. However simple way with for instance 1214 port
>> for KazAA doesn't work because this software uses floating port
>> technology. This traffic can be send via different ports and these ports
>> can change in the fly. This is rather well known. So I'm looking for 
>> the stuff working at higher level and analyzing
>> traffic inside to determine the content and the real protocol. It could
>> be a patch to the kernel or whatever. It should only be able to mark
>> packet by a special marker.
>> I need this solution not only to prioritizing the traffic (prioritizing
>> can be achieve in other way) but also to selection the Internet link. I
>> want to NAT this low quality data for some specific address in order to
>> send it over cheaper link.
>> What do you think is there any solution to do it? Or maybe there is
>> ongoing project trying to tackle with this global problem with detection
>> p2p traffic.
>
>
> Snort has set of rules to detect P2P traffic. AFAIK snort is quite 
> fast, at least fast enough to cope with 10Mbits on average PC.
> Maybe the solution is detecting snort alerts about P2P and 
> automagically cutting bandwidth of host playnig with P2P?
>
> Szymon Miotk


snort signatures are quite poor in some manner. f.e. the X signature 
will not
detect X from big-endian hosts (at least last time i checked). they seem 
to be
extracted from sniffed sessions instead of protocol specifications. 
there is an
interesting projekt called hank (sourceforge), it is missing signatures 
but it is
equipped with almost everything you need for content-based classification,
it can receive packets through netfilter ipq mechanism, with simple 
modifications
you should be able to set skb->priority or skb->nfmark from userspace.
unfortunately there seems to be no active development, but from what
i can judge it looks useable.

Patrick



_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/