real time stateful filtering question

real time stateful filtering question

[peter smith]

Hello,

Sorry if this topic has come up before, but I was unable to find any past
posting on the subject:

If I wish to monitor using netfilter a link that has applications talking on
it that use ephemeral ports (like, for instance, FTP or H323), is there a
way to install filters fast enough /syncronously enough to guarantee that
you won't miss packets that you have already received? For instance, say
that you have some number of packets in your rx ring buffer waiting to be
processed. The first one is processed, and matches some filter you have set.
That packet is queued for delivery to userland.

That packet also has a FTP PORT command inside it's payload that specifies
the new data channel that the two hosts will use to transfer data. (where
channel == the src/dst ip and src/dst ports)

Unfortunately, packets sent on the data channel are also in your receive
buffer, and get processed by netfilter *before* the "PORT COMMAND" packet
gets processed by the userland monitor, or before the monitor can
sucessfully re-program the netfilter filter list. So they are dropped,
silently.

I have exaggerated this example a bit for clarity, but I believe the race
condition exists, particularly wrt the filter reprogramming.

Is this truly a problem? Have others found a way around it? It appears that
for a stateful filtering application, a user must build a faster mechanism
for programming the filters?

Any comments greatly appreciated.

Thanks,
Peter

Re: real time stateful filtering question

[Henrik Nordstrom]

On Mon, 15 Mar 2004, peter smith wrote:

> Unfortunately, packets sent on the data channel are also in your receive
> buffer, and get processed by netfilter *before* the "PORT COMMAND" packet
> gets processed by the userland monitor, or before the monitor can
> sucessfully re-program the netfilter filter list. So they are dropped,
> silently.

This is a somewhat bad example as FTP does not do anything with the data 
channel until a response to the PORT command is seen and thus there can 
not be such race window there.

For most other protocols with "related" channels the same criteria is also
true. But there probably is some protocols not waiting for a response to
their control message before starting to use the "related" channel and
where the race you discuss can occur.

You can minimize the race by running your session tracking in the kernel 
as conntrack protocol helpers, but even then there will be a small race 
window if the appliaction does not wait for some response to the control 
message before starting to use the related connection.

Regards
Henrik

Re: real time stateful filtering question

[Saber zrelli]

On Wed, 17 Mar 2004 08:13:43 +0100 (CET), "Henrik Nordstrom"
<hno@marasystems.com> said:
> On Mon, 15 Mar 2004, peter smith wrote:
> 
> > Unfortunately, packets sent on the data channel are also in your receive
> > buffer, and get processed by netfilter *before* the "PORT COMMAND" packet
> > gets processed by the userland monitor, or before the monitor can
> > sucessfully re-program the netfilter filter list. So they are dropped,
> > silently.
> 
> This is a somewhat bad example as FTP does not do anything with the data 
> channel until a response to the PORT command is seen and thus there can 
> not be such race window there.
> 
> For most other protocols with "related" channels the same criteria is
> also
> true. But there probably is some protocols not waiting for a response to
> their control message before starting to use the "related" channel and
> where the race you discuss can occur.
> 
> You can minimize the race by running your session tracking in the kernel 
> as conntrack protocol helpers, but even then there will be a small race 
> window if the appliaction does not wait for some response to the control 
> message before starting to use the related connection.

if there is a kernel module that does the filtering , why does race still
exist ,
if the filtering is programatically set then no packet will be proceeded
until your return from the hook function .
thats why it is called interruption also .


Regards.
-- 
  Saber Zrelli
  kanazawa university international house 
  kakuma-machi , kanazawa , Ishikawa 
  920-1192 Japan. 

Re: real time stateful filtering question

[Henrik Nordstrom]

On Wed, 17 Mar 2004, Saber zrelli wrote:

> if there is a kernel module that does the filtering , why does race
> still exist

Because if the application does not wait for a response to their control 
message before starting to use the "related" connection there is no 
guarantee the control message is seen before traffic on the related 
connection. This is due to how IP operates, not an artefact of netfilter.

Filtering in userspace only widens this window considerably from tiny to 
significant, but there is numerous other cases where this window gets 
opened

  * SMP, allowing processing of multiple packets in parallell by the 
kernel.
  * Control message lost in transit, before reaching the firewall.
  * Control message delayed in transit, causing the "related" packets to 
reach the firewall before the control message.


In the example of FTP this window does not at all exists as FTP always 
waits for the response to the control message before any "related" 
traffic. Here it does not matter at all if you run the tracking in kernel 
or userspace, in both cases there won't be any related traffic until you 
have processed and forwarded the control message so there is no 
possibility there is related traffic queued at the same time or arriving 
while you are processing the control message.

Regards
Henrik

Re: real time stateful filtering question

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 1826 bytes --]

On Mon, Mar 15, 2004 at 10:54:00AM -0500, peter smith wrote:
> If I wish to monitor using netfilter a link that has applications talking on
> it that use ephemeral ports (like, for instance, FTP or H323), is there a
> way to install filters fast enough /syncronously enough to guarantee that
> you won't miss packets that you have already received? For instance, say
> that you have some number of packets in your rx ring buffer waiting to be
> processed. The first one is processed, and matches some filter you have set.
> That packet is queued for delivery to userland.
> 
> That packet also has a FTP PORT command inside it's payload that specifies
> the new data channel that the two hosts will use to transfer data. (where
> channel == the src/dst ip and src/dst ports)
> 
> Unfortunately, packets sent on the data channel are also in your receive
> buffer, and get processed by netfilter *before* the "PORT COMMAND" packet
> gets processed by the userland monitor, or before the monitor can
> sucessfully re-program the netfilter filter list. So they are dropped,
> silently.

the backlog queue ('receive buffer') is always processed in-order.  so
it is a violation of protocol if an ftp data packet would arrive before
the control connection would show PORT/PASV.

Imagine this being the FTP client/server.  It would return a  TCP RST to
the data packet, since it wouldn't have bound a socket to the respective
address yet (or whatever).

[yes, ftp sucks as an example because this could never happen, but
settinh this aside]

> Thanks,
> Peter

-- 
- Harald Welte <laforge@gnumonks.org>               http://www.gnumonks.org/
============================================================================
Programming is like sex: One mistake and you have to support it your lifetime

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

RE: real time stateful filtering question

[peter smith]

Thanks to all for responding.

My application is *not* participating in the network level session(s). It is
not an end-station. I am building a monitor application - like an IDS -
which simply sniffs the wire by installing a filter at a well known port,
and waits for sessions on that port, and then installs filters for data
sessions that are related to the initial control session. I simply used FTP
as an example of a protocol that behaves this way...

I wanted to use netfilter as-is, and do application level parsing of FTP
commands, etc, in my userland code.

I believe that while the backlog queue is processed in the kernel, in-order,
the fact that the packet containing the FTP "PORT" command isn't processed
by my userland application level decode until a context switch occurs makes
it impossible to guarantee that the userland application level decode will
get to process the PORT command packet will be seen before data channel
packets are discarded ... this seems to be common sense unless there is some
workaround that I am missing (or fundamental tenet I am missing)...

Thanks!
Peter



-----Original Message-----
From: Harald Welte [mailto:laforge@gnumonks.org]
Sent: Wednesday, March 17, 2004 8:58 AM
To: peter smith
Cc: Netfilter Development Mailinglist
Subject: Re: real time stateful filtering question


On Mon, Mar 15, 2004 at 10:54:00AM -0500, peter smith wrote:
> If I wish to monitor using netfilter a link that has applications talking
on
> it that use ephemeral ports (like, for instance, FTP or H323), is there a
> way to install filters fast enough /syncronously enough to guarantee that
> you won't miss packets that you have already received? For instance, say
> that you have some number of packets in your rx ring buffer waiting to be
> processed. The first one is processed, and matches some filter you have
set.
> That packet is queued for delivery to userland.
>
> That packet also has a FTP PORT command inside it's payload that specifies
> the new data channel that the two hosts will use to transfer data. (where
> channel == the src/dst ip and src/dst ports)
>
> Unfortunately, packets sent on the data channel are also in your receive
> buffer, and get processed by netfilter *before* the "PORT COMMAND" packet
> gets processed by the userland monitor, or before the monitor can
> sucessfully re-program the netfilter filter list. So they are dropped,
> silently.

the backlog queue ('receive buffer') is always processed in-order.  so
it is a violation of protocol if an ftp data packet would arrive before
the control connection would show PORT/PASV.

Imagine this being the FTP client/server.  It would return a  TCP RST to
the data packet, since it wouldn't have bound a socket to the respective
address yet (or whatever).

[yes, ftp sucks as an example because this could never happen, but
settinh this aside]

> Thanks,
> Peter

--
- Harald Welte <laforge@gnumonks.org>               http://www.gnumonks.org/
============================================================================
Programming is like sex: One mistake and you have to support it your
lifetime

RE: real time stateful filtering question

[Henrik Nordstrom]

On Wed, 17 Mar 2004, peter smith wrote:

> My application is *not* participating in the network level session(s). It is
> not an end-station. I am building a monitor application - like an IDS -
> which simply sniffs the wire by installing a filter at a well known port,
> and waits for sessions on that port, and then installs filters for data
> sessions that are related to the initial control session. I simply used FTP
> as an example of a protocol that behaves this way...

This important distinction to make is if this sniffing device inline with
the traffic (forwarding), or just passively listening?

If passively listening then netfilter is not the tool for the job, and you 
must make sure packet processing can keep up with the rate needed as you 
can not delay traffic.

If inline, then all of what has been discussed applies, including the 
exaplanation why your expected window is permanently shut close in most 
protocols (certainly for FTP).

> I wanted to use netfilter as-is, and do application level parsing of FTP
> commands, etc, in my userland code.

Which is entirely fine if you are inline. See earlier discussion.

> I believe that while the backlog queue is processed in the kernel, in-order,
> the fact that the packet containing the FTP "PORT" command isn't processed
> by my userland application level decode until a context switch occurs makes
> it impossible to guarantee that the userland application level decode will
> get to process the PORT command packet will be seen before data channel
> packets are discarded

Not at all. The FTP protocol does not progress to use the data channel 
until the response to the PORT command is seen. If you are executing 
inline using netfilter QUEUE then this will not happen until your 
userspace program has processed the PORT command and allowed it to be 
forwarded. End of story.

If you are not executing inline in the traffic stream then it is another 
business, but then netfilter is also not the proper tool for the job.

Regards
Henrik

RE: real time stateful filtering question

[peter smith]

I am a passive tap, not a bump in the wire. So, as you said, apparently
netfilter is not the tool for the job, as for my software, I need to
guarantee that each packet is processed to completion (i.e. by my userland
software) before *any* processing is performed on subsequent packets in the
flow, or subsequent packets in related flows.

Thanks,
Peter


On Wed, 17 Mar 2004, peter smith wrote:

> My application is *not* participating in the network level session(s). It
is
> not an end-station. I am building a monitor application - like an IDS -
> which simply sniffs the wire by installing a filter at a well known port,
> and waits for sessions on that port, and then installs filters for data
> sessions that are related to the initial control session. I simply used
FTP
> as an example of a protocol that behaves this way...

This important distinction to make is if this sniffing device inline with
the traffic (forwarding), or just passively listening?

If passively listening then netfilter is not the tool for the job, and you
must make sure packet processing can keep up with the rate needed as you
can not delay traffic.

If inline, then all of what has been discussed applies, including the
exaplanation why your expected window is permanently shut close in most
protocols (certainly for FTP).

> I wanted to use netfilter as-is, and do application level parsing of FTP
> commands, etc, in my userland code.

Which is entirely fine if you are inline. See earlier discussion.

> I believe that while the backlog queue is processed in the kernel,
in-order,
> the fact that the packet containing the FTP "PORT" command isn't processed
> by my userland application level decode until a context switch occurs
makes
> it impossible to guarantee that the userland application level decode will
> get to process the PORT command packet will be seen before data channel
> packets are discarded

Not at all. The FTP protocol does not progress to use the data channel
until the response to the PORT command is seen. If you are executing
inline using netfilter QUEUE then this will not happen until your
userspace program has processed the PORT command and allowed it to be
forwarded. End of story.

If you are not executing inline in the traffic stream then it is another
business, but then netfilter is also not the proper tool for the job.

Regards
Henrik

Re: real time stateful filtering question

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 1308 bytes --]

On Wed, Mar 17, 2004 at 09:53:34AM -0500, peter smith wrote:
> I believe that while the backlog queue is processed in the kernel, in-order,
> the fact that the packet containing the FTP "PORT" command isn't processed
> by my userland application level decode until a context switch occurs makes
> it impossible to guarantee that the userland application level decode will
> get to process the PORT command packet will be seen before data channel
> packets are discarded ... this seems to be common sense unless there is some
> workaround that I am missing (or fundamental tenet I am missing)...

I thought you were talking about a connection tracking helper module.
Those (like ip_conntrack_ftp) are execute inside the network RX softirq,
so there is no asynchronous behaviour.

If you have to pass the packet up to userspace, you are completely
right.  There would definitely be a race condition.

> Thanks!
> Peter

-- 
- Harald Welte <laforge@netfilter.org>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

RE: real time stateful filtering question

[Henrik Nordstrom]

On Wed, 17 Mar 2004, peter smith wrote:

> I am a passive tap, not a bump in the wire. So, as you said, apparently
> netfilter is not the tool for the job, as for my software, I need to
> guarantee that each packet is processed to completion (i.e. by my userland
> software) before *any* processing is performed on subsequent packets in the
> flow, or subsequent packets in related flows.

And for this you need process packets at "wire speed", not dropping any 
packets until you have finished processing the first..

If you are using an early filter then this filter can only be used to skip 
gaps of uninteresting packets. As soon as you hit a packet where related 
traffic may be involved you must start buffering packets until processing 
of the first has finished.

a packet socket (or the libpcap interface to it) seems most suitable for 
this.

Regards
Henrik

ipt_helper-any error

[Wojciech 'Sas' Cieciwa]

in ipt_helper-any modules is IMHO error
In line 74 id :
	if (info->data[0] == '\0')

but data isn't declared, here is nedded this "data" not "name" ??

					Sas.
-- 
{Wojciech 'Sas' Cieciwa}  {Member of PLD Team                               }
{e-mail: cieciwa@alpha.zarz.agh.edu.pl, http://www2.zarz.agh.edu.pl/~cieciwa}

Re: ipt_helper-any error

[Patrick McHardy]

Wojciech 'Sas' Cieciwa wrote:
> in ipt_helper-any modules is IMHO error
> In line 74 id :
> 	if (info->data[0] == '\0')
> 
> but data isn't declared, here is nedded this "data" not "name" ??

Thanks Wojciech, I'm going to fix this.

Regards
Patrick


> 
> 					Sas.