Re: Feasability of Protocol Filtering

Re: Feasability of Protocol Filtering

[Eicke Friedrich]

Hi there,

Matt Skidmore wrote:
 > I have gained some interest in adding a new module to the netfilter code
 > for filtering by protocol. However, I do not know how realistic this
 > project would be. I have not found any projects similar to it as of yet.
 > But, I would like to be able to REJECT, DENY, or REDIRECT packets based on
 > the protocol of their connection.

i'm doing a quite similar thing at the moment: i'm developing a match that recognizes p2p 
traffic (kazaa, edonkey finished but more to come) by their protocol and mark them. After 
that i use a tc filter to read the marks and put the packets in QoS classes.
What you need is something characteristic for every protocol. For example: every 
kazaa-download starts with a packet containing the string "GET /.hash=" - i do a 
string-match on each packet and if i find a match i just mark the whole connection with 
CONNMARK.
By doing this i can treat every kazaa-download in the same way regardless of port or 
ip-adress. But i'm a little bit concerned about the required ressources. If you're going 
to use this match for many protocols and in a highly stressed environment you will need 
much ram and lots of cpu power.
I'm going to test the behavior of my match in a couple of weeks for a 10MBit/sec 
environment - if someone is interessted i can send the results to the list or put it on a 
webpage.
I started doing some network sniffin' and if you can find something characteristic for the 
protocols you're going to match it should be easy to create an appropriate module. Hope my 
thoughts will help you a little bit.

Regards,
Eicke.

PS: if you receive this message twice - sorry! first time i used the wrong mailbox.

Re: Feasability of Protocol Filtering

[Matt Skidmore]

> i'm doing a quite similar thing at the moment: i'm developing a match that recognizes p2p
> traffic (kazaa, edonkey finished but more to come) by their protocol and mark them. After
> that i use a tc filter to read the marks and put the packets in QoS classes.
> What you need is something characteristic for every protocol. For example: every
> kazaa-download starts with a packet containing the string "GET /.hash=" - i do a
> string-match on each packet and if i find a match i just mark the whole connection with
> CONNMARK.

thats about what im thinking. im going to use HTTP and SSH as examples
because i know those protocols the best. and to keep it simple ill just
say we wish to REJECT the packet. for HTTP the syn and acks might go
through to establish the handshake, but you can examine the begining few
packet contents to figure out that its trying to GET some url and then
drop that packet and kindly reset the connection to be polite in this
case. same with SSH, when the connection begins it usually says SSH with
the version and some other tidbits that can be easily recognized. i would
say build the basic protocol identification source file and then create
the ability to choose which protocols you want to be able to recognize as
to keep down bloat on a module.

I was heading to p2p programs myself. but this sort of feature would be
nice to identify p2p stuff and perhaps reject or use QoS on them. once the
connection has been established the data in the packets may not match the
pattern for that protocol, but you could use connection tracking with
dest/src ip and port to mark said packets.

the downside to this is, say you want to reject SSH traffic. you issue the
command to reject it based on protocol. if there is a connection already
in progress it probably wouldnt get it unless there are some telltale
signs in the data packets, but since you can at least identify the initial
connection packets you could at least stop additional connections.

I would be interested in seeing what you have so far. as for CPU and RAM
concerns, I thought about this a bit myself. it could also intoduce
latency as well.

-matt

Re: Feasability of Protocol Filtering

[Eicke Friedrich]

Matt Skidmore wrote:
> the downside to this is, say you want to reject SSH traffic. you
> issue the command to reject it based on protocol. if there is a
> connection already in progress it probably wouldnt get it unless
> there are some telltale signs in the data packets, but since you 
> can at least identify the initial connection packets you could at
> least stop additional connections.
Yes, you are absolute right. But i don't see a way to change this. For
p2p the already
established connections are not worth any effort. At some point the
file is completly
transfered (kazaa) or the transfered segment has to be acknowlegded
(edk). My match will
find this ack and treat the following packets as p2p-traffic.

> I would be interested in seeing what you have so far. as for CPU
> and RAM concerns, I thought about this a bit myself. it could also
> intoduce latency as well.
I can send you the source-code if you want to.

Regards,
Eicke.

Re: Feasability of Protocol Filtering

[Martin Josefsson]

On Wed, 2003-04-23 at 23:28, Eicke Friedrich wrote:

> I'm going to test the behavior of my match in a couple of weeks for a 10MBit/sec 
> environment - if someone is interessted i can send the results to the list or put it on a 
> webpage.

I do something similar here, no problem to do two stringmatches on a
100Mbit/s link.

I benchmarked the stringmatch a bit a while ago.
On my pIII it uses ~50k cycles / kB.

If using the Boyer-Moore algorithm instead you get it down to ~3k cycles
/ kB when searching for a 5 byte string (cpu usage decreases with string
length)

The Boyer-Moore patch was sent to this list quite some time ago. I
should update it so it applies.

-- 
/Martin

Re: Feasability of Protocol Filtering

[Jozsef Kadlecsik]

On 24 Apr 2003, Martin Josefsson wrote:

> I benchmarked the stringmatch a bit a while ago.
> On my pIII it uses ~50k cycles / kB.
>
> If using the Boyer-Moore algorithm instead you get it down to ~3k cycles
> / kB when searching for a 5 byte string (cpu usage decreases with string
> length)
>
> The Boyer-Moore patch was sent to this list quite some time ago. I
> should update it so it applies.

Would you do so? Let's try to collect the missed bits :-(

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Feasability of Protocol Filtering

[Martin Josefsson]

On Thu, 2003-04-24 at 14:16, Jozsef Kadlecsik wrote:

> > The Boyer-Moore patch was sent to this list quite some time ago. I
> > should update it so it applies.
> 
> Would you do so? Let's try to collect the missed bits :-(

I'll see what I can do, it needs some cleanups.

There are tons of stuff missed, That reminds me, I once had a plan to go
through my netfilter and netfilter-devel folders and shake out all
missed patches. I'll see what I can do.

-- 
/Martin

pre, pre, premangle?

[Scott MacKay]

Hiyas, 
    Is there any way to hook into incoming packets
such that you get it before any other investigative
code (such as conn tracking), examine it, and mangle
it before letting them work on the packet?

-Scott

__________________________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo
http://search.yahoo.com

Re: pre, pre, premangle?

[Martin Josefsson]

On Thu, 2003-04-24 at 16:16, Scott MacKay wrote:
> Hiyas, 
>     Is there any way to hook into incoming packets
> such that you get it before any other investigative
> code (such as conn tracking), examine it, and mangle
> it before letting them work on the packet?

./runme --batch pending
./runme userspace/raw.patch

iptables -t raw -A PREROUTING yada yada

Many of the targets that mangle packets have checks for which table they
are beeing added too and refuse to be added to other tables than mangle.
So to be able to use them in the raw table you will have to modify them
a bit. The raw table is mostly used for DROP-ing, NOTRACK-ing and
TRACE-ing packets.

userspace/raw.patch tries to patch the userspace iptables as well.

-- 
/Martin

Re: pre, pre, premangle?

[Scott MacKay]

Actually, my goal is to mangle packet contents before
conn track gets its hands on it.  I currently do some
packet mangling as part of a PREROUTING userspace
queue (QUEUE), but I think my mangling is occuring
after conn track has already looked at the contents
(actually the problem is that the packet comes in
'mangled' and my stuff demangles it.  I think conn
track is building its assumptions on the mangled
packet contents).
Using this patch as a base, do you believe I could add
in QUEUE as a target?  If so, is there any way to
identify which table a userspace module is getting the
packet from?  I see ipq_packet_msg has a hook field,
but never saw any apps use it.  Since I still need to
do processing at the mangle PREROUTING (or INPUT &
FORWARD) stages and there can be only 1 QUEUE target,
I would probably need to differentiate between a raw
and mangle table...Thanks for the info so far tho!

-Scott


--- Martin Josefsson <gandalf@wlug.westbo.se> wrote:
> On Thu, 2003-04-24 at 16:16, Scott MacKay wrote:
> > Hiyas, 
> >     Is there any way to hook into incoming packets
> > such that you get it before any other
> investigative
> > code (such as conn tracking), examine it, and
> mangle
> > it before letting them work on the packet?
> 
> ./runme --batch pending
> ./runme userspace/raw.patch
> 
> iptables -t raw -A PREROUTING yada yada
> 
> Many of the targets that mangle packets have checks
> for which table they
> are beeing added too and refuse to be added to other
> tables than mangle.
> So to be able to use them in the raw table you will
> have to modify them
> a bit. The raw table is mostly used for DROP-ing,
> NOTRACK-ing and
> TRACE-ing packets.
> 
> userspace/raw.patch tries to patch the userspace
> iptables as well.
> 
> -- 
> /Martin


__________________________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo
http://search.yahoo.com

ipq_packet_msg, hook?

[Scott MacKay]

Hello,
How exactly do you get anything useful from the
libipq's ipq_packet_msg structure field 'hook'?  Can
anyone point me to which APIs translate that to
something useful?

-Scott

__________________________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo
http://search.yahoo.com

Re: Feasability of Protocol Filtering

[Eicke Friedrich]

Martin Josefsson wrote:
> I do something similar here, no problem to do two stringmatches on a
> 100Mbit/s link.
What about throughput, packets per second and latency? Do you have any experiences? I'm 
going to face a 100MBit/sec link in a couple of months so any hints are appreciated. By 
the way it's a Linux Bridge with netfilter support that has nothing else to do than 
marking p2p-packets and put them in a HTB class to reduce the "wasted" bandwidth.

> If using the Boyer-Moore algorithm instead you get it down to ~3k cycles
> / kB when searching for a 5 byte string (cpu usage decreases with string
> length)
Maybe it's not so important for my project because i know the exact position in the 
payload where the string has to be so i don't search the whole packet but just the 
specific position in the payload.
For this purpose i use memcmp(kazaa) or memchr(edonkey) - maybe there is a way to improve 
this?

int
search_kazaa (char *needle, char *haystack, int needle_len)
{
     char *t = haystack;

     t += 20;		//skippin tcp-header

     if (memcmp(t, needle, needle_len) == 0)
         return 1;
     else
         return 0;

}

This is the one for kazaa where needle is the string "GET /.hash=" and haystack is the 
ip-packet. Next one is similar but i'm just searching for 0xe3 and 0x47 (edonkey get 
segment command).

int
search_edk (char *haystack)
{
     char *t = haystack;

     t += 20;		//skippin tcp-header
     if (memchr(t, 0xe3, 1) == 0)
         return 0;
     else
         t += 5;		//jump to command position
         if (memchr(t, 0x47, 1) != 0) return 1;
         else return 0;

}


Best regards,
Eicke.

Re: Feasability of Protocol Filtering

[Filipe Almeida]

Eicke Friedrich wrote:

> i'm doing a quite similar thing at the moment: i'm developing a match 
> that recognizes p2p traffic (kazaa, edonkey finished but more to come) 
> by their protocol and mark them. After that i use a tc filter to read 
> the marks and put the packets in QoS classes.
> What you need is something characteristic for every protocol. For 
> example: every kazaa-download starts with a packet containing the 
> string "GET /.hash=" - i do a string-match on each packet and if i 
> find a match i just mark the whole connection with CONNMARK.
> By doing this i can treat every kazaa-download in the same way 
> regardless of port or ip-adress. But i'm a little bit concerned about 
> the required ressources. If you're going to use this match for many 
> protocols and in a highly stressed environment you will need much ram 
> and lots of cpu power.
> I'm going to test the behavior of my match in a couple of weeks for a 
> 10MBit/sec environment - if someone is interessted i can send the 
> results to the list or put it on a webpage.
> I started doing some network sniffin' and if you can find something 
> characteristic for the protocols you're going to match it should be 
> easy to create an appropriate module. Hope my thoughts will help you a 
> little bit.
>
> Regards,
> Eicke.
>
> PS: if you receive this message twice - sorry! first time i used the 
> wrong mailbox. 


Hi there,
Hi have something similiar to that, and running for a couple of months 
on a 15MB/s network connection. Almost no performance impact noted.

http://mega.ist.utl.pt/~filipe/ipt_p2p/

It is still very primitive and easily forgeable, but it is efficient.
Fell free to work on top of these code.
It detects kazaa, edonkey, gnutella and direct connect.

Since I'm not doing statefull filtering on the whole connection, it is 
possible to subvert these type of filters, but I am working on a work 
around.

Filipe Almeida

>
>
>

Re: Feasability of Protocol Filtering

[Eicke Friedrich]

Filipe Almeida wrote:
> Hi have something similiar to that, and running for a 
> couple of months on a 15MB/s network connection. Almost 
> no performance impact noted.
What hardware is doing this job? Any tweaks (Kernel patches,
NAPI ...) installed? Is it a router (NAT) or a bridge?

> http://mega.ist.utl.pt/~filipe/ipt_p2p/
Can't reach the url with my DSL-connection but with a
100MBit-connection everything is fine - maybe one of those
internet black holes or whatever they're called?

I'm not sure if the developer list is the right place for
such a discussion but if not please answer directly to my mail
adress (and start flaming me ;-]).

Thanks for your help,
Eicke.

Re: Feasability of Protocol Filtering

[Ian Latter]

Good question;

> I've got a heretic question:
> 
> > Examples:
> > Identify and DENY SSH traffic regardless of port
> 
> How do you cope with an ftp transfer of a tcpdump capture of some
> earlier SSH traffic?

But wouldn't this be resolved by only doing a single pass through of
the detection code, per stream/conntrack?  Ie;  if the ftp data is related
to an existing ftp stream, then it won't have a detection attempt made
on it by the proto_detect module ... surely ...

This whole thread on protocol detection, particularly with protocols like
P2P, is sweet ...  very cool stuff ... very useful.




--
Ian Latter
Internet and Networking Security Officer
Macquarie University

Re: Feasability of Protocol Filtering

[Matt Skidmore]

> But wouldn't this be resolved by only doing a single pass through of
> the detection code, per stream/conntrack?  Ie;  if the ftp data is related
> to an existing ftp stream, then it won't have a detection attempt made
> on it by the proto_detect module ... surely ...

excellent point! once the connection has been completed and it has
"passed" our inspection we no longer need to examine addtional packets
from that connection. with will surely resolve and aleviate our cpu usage
problems greatly. its still not a bad idea to tell it to look for string
matching starting at a certain byte. that is, if it doesnt vary too much.

-matt

Feasability of Protocol Filtering

[Matt Skidmore]

I have gained some interest in adding a new module to the netfilter code
for filtering by protocol. However, I do not know how realistic this
project would be. I have not found any projects similar to it as of yet.
But, I would like to be able to REJECT, DENY, or REDIRECT packets based on
the protocol of their connection.

Examples:
Identify and DENY SSH traffic regardless of port

Identify HTTP traffic regardless of port and REDIRECT/reroute it
transparently through another route or hand it to a transparent proxy.

I believe Cisco has a similar feature in their IOS called protocol
discovery (not 100% sure)
(http://www.cisco.com/en/US/products/sw/iosswrel/ps1839/products_feature_guide09186a0080153efc.html)

If it isn't workable that's fine, I'm just throwing in an idea for
something I might like to contribute to. I've not worked on any parts of
the kernel and I would need to shake the dust off my C and and TCP
knowledge, but I'm willing to jump into the pit and give it a try.

Please be gentle, first post to this list. This is just a probe of a
peliminary idea, if there is some hope for the idea I would continue to
brainstorm and elaborate on other features that may be achievable.

-Matt

Re: Feasability of Protocol Filtering

[Eicke Friedrich]

Hi there,

Matt Skidmore wrote:
 > I have gained some interest in adding a new module to the netfilter code
 > for filtering by protocol. However, I do not know how realistic this
 > project would be. I have not found any projects similar to it as of yet.
 > But, I would like to be able to REJECT, DENY, or REDIRECT packets based on
 > the protocol of their connection.

i'm doing a quite similar thing at the moment: i'm developing a match that recognizes p2p 
traffic (kazaa, edonkey finished but more to come) by their protocol and mark them. After 
that i use a tc filter to read the marks and put the packets in QoS classes.
What you need is something characteristic for every protocol. For example: every 
kazaa-download starts with a packet containing the string "GET /.hash=" - i do a 
string-match on each packet and if i find a match i just mark the whole connection with 
CONNMARK.
By doing this i can treat every kazaa-download in the same way regardless of port or 
ip-adress. But i'm a little bit concerned about the required ressources. If you're going 
to use this match for many protocols and in a highly stressed environment you will need 
much ram and lots of cpu power.
I'm going to test the behavior of my match in a couple of weeks for a 10MBit/sec 
environment - if someone is interessted i can send the results to the list or put it on a 
webpage.
I started doing some network sniffin' and if you can find something characteristic for the 
protocols you're going to match it should be easy to create an appropriate module. Hope my 
thoughts will help you a little bit.

Regards,
Eicke.

Re: Feasability of Protocol Filtering

[Serge Droz]

Eicke Friedrich wrote:
> 
> i'm doing a quite similar thing at the moment: i'm developing a match 
> that recognizes p2p traffic (kazaa, edonkey finished but more to come) 
> by their protocol and mark them. After that i use a tc filter to read 
> the marks and put the packets in QoS classes.
> What you need is something characteristic for every protocol. For 
> example: every kazaa-download starts with a packet containing the string 
> "GET /.hash=" - i do a string-match on each packet and if i find a match 
> i just mark the whole connection with CONNMARK.
So, if by chance this message got split exactly at the right place so 
you'd get a packet witch starts with "GET /.hash=" you couldn't post 
this anymore. Your idea will work 99.9% (say), but you can't aford a 
single missmatch or you kill a legitimate connection (which for sure 
will have been initiated by your boss :-))

You can easily try this out: write a snort filter that captures this 
traffic and see how many false positives you get.

Cheers
Serge


-- 
Serge Droz
Paul Scherrer Institut                mailto:serge.droz@psi.ch
CH-5232 Villigen PSI                   Phone: ++41 56 310 3637
                                          Fax: ++41 56 310 3649

Feasability of Protocol Filtering

[Matt Skidmore]

I have gained some interest in adding a new module to the netfilter code
for filtering by protocol. However, I do not know how realistic this
project would be. I have not found any projects similar to it as of yet.
But, I would like to be able to REJECT, DENY, or REDIRECT packets based on
the protocol of their connection.

Examples:
Identify and DENY SSH traffic regardless of port

Identify HTTP traffic regardless of port and REDIRECT/reroute it
transparently through another route or hand it to a transparent proxy.

I believe Cisco has a similar feature in their IOS called protocol
discovery (not 100% sure)
(http://www.cisco.com/en/US/products/sw/iosswrel/ps1839/products_feature_guide09186a0080153efc.html)

If it isn't workable that's fine, I'm just throwing in an idea for
something I might like to contribute to. I've not worked on any parts of
the kernel and I would need to shake the dust off my C and and TCP
knowledge, but I'm willing to jump into the pit and give it a try.

Please be gentle, first post to this list. This is just a probe of a
peliminary idea, if there is some hope for the idea I would continue to
brainstorm and elaborate on other features that may be achievable.

-Matt

Re: Feasability of Protocol Filtering

[Patrick Schaaf]

Hi,

I've got a heretic question:

> Examples:
> Identify and DENY SSH traffic regardless of port

How do you cope with an ftp transfer of a tcpdump capture of some
earlier SSH traffic?

> If it isn't workable that's fine,

It isn't workable for fundamental reasons, but that doesn't mean
it does not make sense to attempt it.

To see the complexities, the parser of tcpdump or ethereal may be
a good place to look at.

best regards
  Patrick

Re: Feasability of Protocol Filtering

[Eicke Friedrich]

Patrick Schaaf wrote:
>>Identify and DENY SSH traffic regardless of port
> How do you cope with an ftp transfer of a tcpdump capture of some
> earlier SSH traffic?
Hmm, kind of difficult but this depends on the things you know about a 
protocol and of course on luck :-)
If you know the exact position (like the second byte in the payload of 
a packet has to be 0xff) of a characteristic string or pattern of 
bytes you don't need to search the whole packet. If it's not at this 
position it isn't the protocol that you're looking for.
What i mean is that if you transfer a captured ssh-session the chances 
that the characteristic string/pattern/whatever of ssh is at the same 
postion in the ftp transfer as it was in the original ssh connection 
are very low because ftp-data uses bigger packets than ssh.
What you also can do is including packet sizes in your test - for 
example excluding large packets (like ftp-data-packets are) from 
searching for ssh-characteristics.
I know that this was just an example but what i try to say is that if 
you learn much about the protocol an implement everything you know you 
will have a good chance to catch just the things you want to.


Regards,
Eicke.

Re: Feasability of Protocol Filtering

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 966 bytes --]

On Tue, Apr 22, 2003 at 04:25:11PM -0700, Matt Skidmore wrote:
> 
> I have gained some interest in adding a new module to the netfilter code
> for filtering by protocol. However, I do not know how realistic this
> project would be. I have not found any projects similar to it as of yet.
> But, I would like to be able to REJECT, DENY, or REDIRECT packets based on
> the protocol of their connection.
> 
> Examples:
> Identify and DENY SSH traffic regardless of port

You should have a look at libqsearch, which adds a fast pattern matching
library to the linux kernel.

> -Matt

-- 
- Harald Welte <laforge@netfilter.org>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]