Forking inside netfilter queue

Forking inside netfilter queue

[Rayed]

Hi,

I am building netfilter queue application, that implement simple HTTP 
filter.

It basically will capture the packet, extract the URL, and issue MySQL 
lookup to see if the address is permitted or blocked.

My only fear that address lookup will block the traffic until I get the 
lookup reply, so I am thinking of using preforking model to fork 10-20 
processes to handle the same queue.

Has any one done multi prcoess handling of the queue, is it even doable?

Rayed

Re: Forking inside netfilter queue

[Jan Engelhardt]

On Apr 30 2007 16:18, Rayed wrote:
>
> I am building netfilter queue application, that implement simple
> HTTP filter.
>
> It basically will capture the packet, extract the URL, and issue
> MySQL lookup to see if the address is permitted or blocked.
>
> My only fear that address lookup will block the traffic until I get
> the lookup reply, so I am thinking of using preforking model to
> fork 10-20 processes to handle the same queue.
>
> Has any one done multi prcoess handling of the queue, is it even doable?

See Apache, it uses preforking. But perhaps threading is much simpler
and equally fast.


Jan
-- 

Re: Forking inside netfilter queue

[Rayed]

Thanks Jan for the reply.

I know how preforking works (almost), what I want to know if multi 
processes can access the same queue API without breaking anything.


Jan Engelhardt wrote:
> On Apr 30 2007 16:18, Rayed wrote:
>   
>> I am building netfilter queue application, that implement simple
>> HTTP filter.
>>
>> It basically will capture the packet, extract the URL, and issue
>> MySQL lookup to see if the address is permitted or blocked.
>>
>> My only fear that address lookup will block the traffic until I get
>> the lookup reply, so I am thinking of using preforking model to
>> fork 10-20 processes to handle the same queue.
>>
>> Has any one done multi prcoess handling of the queue, is it even doable?
>>     
>
> See Apache, it uses preforking. But perhaps threading is much simpler
> and equally fast.
>
>
> Jan
>   

Re: Forking inside netfilter queue

[Jan Engelhardt]

On May 1 2007 09:30, Rayed wrote:
>
> Thanks Jan for the reply.
>
> I know how preforking works (almost), what I want to know if multi processes
> can access the same queue API without breaking anything.

Well, you should be aware that calling ipq_read() [or whatever it's
called now] in a thread may return a packet that does not actually
belong to the connection you created a certain thread for.
Maybe you even need to pthread_mutex_* around calls to ipq_*,
I am not sure if libipq (or whatever we use now) does that by
itself.


Jan
-- 

Can't get --dport to work

[Joel Lindsay]

Hi,

At one time I had this working.  I could match on the ports and deny http 
traffic from certain interfaces.

Now I get an error message and I am not sure what is causing it.  Can anyone 
help?

# iptables -A OUTPUT -i ath0 -p tcp --dport 80 -j ACCEPT
iptables v1.3.6-20061129: Unknown arg `--dport'
Try `iptables -h' or 'iptables --help' for more information.


Thanks,

Joel Lindsay, B.Eng
Project Engineer 

RE: Can't get --dport to work

[Krishnamoorthy (Siva) Sivakumar]

I don't know why it complains about the --dport option. However, I don't
see how -i ath0 option is compatible with the OUTPUT chain. 

Siva

-----Original Message-----
From: netfilter-bounces@lists.netfilter.org
[mailto:netfilter-bounces@lists.netfilter.org] On Behalf Of Joel Lindsay
Sent: Tuesday, May 01, 2007 11:27 AM
To: netfilter@lists.netfilter.org
Subject: Can't get --dport to work

Hi,

At one time I had this working.  I could match on the ports and deny
http 
traffic from certain interfaces.

Now I get an error message and I am not sure what is causing it.  Can
anyone 
help?

# iptables -A OUTPUT -i ath0 -p tcp --dport 80 -j ACCEPT
iptables v1.3.6-20061129: Unknown arg `--dport'
Try `iptables -h' or 'iptables --help' for more information.


Thanks,

Joel Lindsay, B.Eng
Project Engineer 

Re: Forking inside netfilter queue

[Alex]

Unless there are a real lot of these addresses that are permitted, and they 
change rapidly, then I'd expect it would be a better design to get all the 
valid URIs from the MySQL query on init and store them in a linked list.

Also, people already do this type of thing with an HTTP proxy like squid 
before the HTTPd.

----- Original Message ----- 
From: "Rayed" <rayed@saudi.net.sa>
To: <netfilter@lists.netfilter.org>
Sent: Monday, April 30, 2007 2:18 PM
Subject: Forking inside netfilter queue


> Hi,
>
> I am building netfilter queue application, that implement simple HTTP 
> filter.
>
> It basically will capture the packet, extract the URL, and issue MySQL 
> lookup to see if the address is permitted or blocked.
>
> My only fear that address lookup will block the traffic until I get the 
> lookup reply, so I am thinking of using preforking model to fork 10-20 
> processes to handle the same queue.
>
> Has any one done multi prcoess handling of the queue, is it even doable?
>
> Rayed
>
> 

Re: Forking inside netfilter queue

[Rayed Alrashed]

> Unless there are a real lot of these addresses that are permitted, and 
> they change rapidly, then I'd expect it would be a better design to 
> get all the valid URIs from the MySQL query on init and store them in 
> a linked list.
I am afraid this not an option, because my final goal is to integrate my 
application with external vendor database using the vendor own API.


> Also, people already do this type of thing with an HTTP proxy like 
> squid before the HTTPd.

I've tested HTTP proxy before, but the overhead is very big, and I 
couldn't get the proxy to handle high loads.

Another problem with proxy servers is using the proxy own IP for 
outgoing request, unfortunately using the same IP for many requests 
trigger many IDSes to block the proxy IP.

What I am trying to do is a light weight HTTP filter, by searching for 
HTTP requests in packet traveling on the wire, with minimal TCP 
reassembly to find HTTP requests spanning more than one packet.

Re: Forking inside netfilter queue

[Alex]

If you were to do this with the queue lib, then you'd ataualy have to
either let the client establish the connection OR fake that the connection
was established, before you get the HTTP request is sent to rule on.

Also keep in mind that the servers/client might want to do a Keep-alive or
long lived HTTP session, with multiple HTTP requests. Keeping track of
that, from an NFNetlink/Queue based interface would be hecktic.
>
>> Unless there are a real lot of these addresses that are permitted, and
>> they change rapidly, then I'd expect it would be a better design to
>> get all the valid URIs from the MySQL query on init and store them in
>> a linked list.
> I am afraid this not an option, because my final goal is to integrate my
> application with external vendor database using the vendor own API.
>
>
>> Also, people already do this type of thing with an HTTP proxy like
>> squid before the HTTPd.
>
> I've tested HTTP proxy before, but the overhead is very big, and I
> couldn't get the proxy to handle high loads.
>
> Another problem with proxy servers is using the proxy own IP for
> outgoing request, unfortunately using the same IP for many requests
> trigger many IDSes to block the proxy IP.
>
> What I am trying to do is a light weight HTTP filter, by searching for
> HTTP requests in packet traveling on the wire, with minimal TCP
> reassembly to find HTTP requests spanning more than one packet.
>

Re: Forking inside netfilter queue

[Rayed Alrashed]

>
> If you were to do this with the queue lib, then you'd ataualy have to
> either let the client establish the connection OR fake that the connection
> was established, before you get the HTTP request is sent to rule on.
>   

Of course! I want my application to be totally transparent from the 
client. The client will use regular browser without any proxy, and I 
won't inspect TCP handshaking packets, only packets that looks like HTTP 
request.

> Also keep in mind that the servers/client might want to do a Keep-alive or
> long lived HTTP session, with multiple HTTP requests. Keeping track of
> that, from an NFNetlink/Queue based interface would be hecktic.
>   
According to my tests and observations, most HTTP requests reside in a 
single packet, for request that span more than one request, I'll start 
tracking the session when I get a packet that looks like the beginning 
of HTTP request, and stop when I get "Host" header.

For example:
3rd Packet: "GET /ver_long_uri\r\n"
It looks like the start of HTTP request, Start tracking.

4th Packet: "Host: ad-ware.domain\r\n\r\n"
I have URI+HOST, Stop tracking.

In this case I won't have to keep track of the whole TCP session only 
what I need to get URI and HOST.

This will allow me to process more packets with minimal session tracking.