netfilter-devel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [RFC 0/4] nfnetlink_queue bypass queue to userspace X bytes of connection
@ 2010-07-24 15:44 Karl Hiramoto
  2010-07-24 15:44 ` [RFC 1/4] netfilter/Kconfig: NF_QUEUE_CONNBYTES_BYPASS Karl Hiramoto
                   ` (4 more replies)
  0 siblings, 5 replies; 10+ messages in thread
From: Karl Hiramoto @ 2010-07-24 15:44 UTC (permalink / raw)
  To: netfilter-devel; +Cc: Karl Hiramoto


Hi, I'm working on a nf_queue based HTTP filter. Sometimes I want to
NF_ACCEPT X bytes of a connection. In a HTTP connection its part of the
Content-Length.  Being able to directly NF_ACCEPT large parts of a connection
is a 2X to 3X speedup, by not queuing these packets to user-space.

I've tested this patchset only with HTTP on IPv4, and it makes a large
difference in page load times (lower latency), max bandwidth utilization,
and CPU utilization (lower utilization).

This is something like connbytes, but with TCP look at the sequence number
to handle out of order and duplicate packets. I'm thinking about changing
the code to a always queue if the TCP flags SYN, FIN, or RST are set.

Besides HTTP I could imagine other L7 protocol filters using nf_queue
could this feature.

Bad points:

* making nfnetlink_queue depend on conntrack and on on L3 (TCP).


This patchset is also at:
git://github.com/karlhiramoto/linux-2.6.git nfq

libnl patches to send NFQA_ACCEPT_CONNBYTES are at:
git://github.com/karlhiramoto/libnl.git

Karl Hiramoto (4):
  netfilter/Kconfig: NF_QUEUE_CONNBYTES_BYPASS
  nf_conntrack_queue: define struct that will be stored in nf_ct_extend
  nf_conntrack: add nf_queue extension
  nfnetlink_queue: allow part of a connection to bypass the queue

 include/linux/netfilter/nfnetlink_queue.h   |    1 +
 include/net/netfilter/nf_conntrack_extend.h |    2 +
 include/net/netfilter/nf_conntrack_queue.h  |   17 ++++
 net/netfilter/Kconfig                       |   13 +++-
 net/netfilter/nf_conntrack_core.c           |   37 +++++++++-
 net/netfilter/nfnetlink_queue.c             |  112 ++++++++++++++++++++++++++-
 6 files changed, 179 insertions(+), 3 deletions(-)
 create mode 100644 include/net/netfilter/nf_conntrack_queue.h


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [RFC 1/4] netfilter/Kconfig: NF_QUEUE_CONNBYTES_BYPASS
  2010-07-24 15:44 [RFC 0/4] nfnetlink_queue bypass queue to userspace X bytes of connection Karl Hiramoto
@ 2010-07-24 15:44 ` Karl Hiramoto
  2010-07-24 15:44 ` [RFC 2/4] nf_conntrack_queue: define struct that will be stored in nf_ct_extend Karl Hiramoto
                   ` (3 subsequent siblings)
  4 siblings, 0 replies; 10+ messages in thread
From: Karl Hiramoto @ 2010-07-24 15:44 UTC (permalink / raw)
  To: netfilter-devel; +Cc: Karl Hiramoto

Create config option to add nf_queue bypass feature.

Signed-off-by: Karl Hiramoto <karl@hiramoto.org>
---
 net/netfilter/Kconfig |   13 ++++++++++++-
 1 files changed, 12 insertions(+), 1 deletions(-)

diff --git a/net/netfilter/Kconfig b/net/netfilter/Kconfig
index 8593a77..4468365 100644
--- a/net/netfilter/Kconfig
+++ b/net/netfilter/Kconfig
@@ -11,7 +11,18 @@ config NETFILTER_NETLINK_QUEUE
 	help
 	  If this option is enabled, the kernel will include support
 	  for queueing packets via NFNETLINK.
-	  
+
+config NF_QUEUE_CONNBYTES_BYPASS
+	  bool  'NF_QUEUE bypass support'
+	  depends on NF_CONNTRACK
+	  depends on NETFILTER_NETLINK_QUEUE
+	  help
+	  This option allows user-space to tell nfnetlink_queue so allow X bytes
+	  to bypass the queue with the ACCEPT verdict.
+	  In conditions where you know X bytes must be accepted, it avoid the
+	  copy to user-space and can be over a 2X speed improvement.
+
+
 config NETFILTER_NETLINK_LOG
 	tristate "Netfilter LOG over NFNETLINK interface"
 	default m if NETFILTER_ADVANCED=n
-- 
1.7.1


^ permalink raw reply related	[flat|nested] 10+ messages in thread

* [RFC 2/4] nf_conntrack_queue: define struct that will be stored in nf_ct_extend
  2010-07-24 15:44 [RFC 0/4] nfnetlink_queue bypass queue to userspace X bytes of connection Karl Hiramoto
  2010-07-24 15:44 ` [RFC 1/4] netfilter/Kconfig: NF_QUEUE_CONNBYTES_BYPASS Karl Hiramoto
@ 2010-07-24 15:44 ` Karl Hiramoto
  2010-07-24 15:44 ` [RFC 3/4] nf_conntrack: add nf_queue extension Karl Hiramoto
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 10+ messages in thread
From: Karl Hiramoto @ 2010-07-24 15:44 UTC (permalink / raw)
  To: netfilter-devel; +Cc: Karl Hiramoto

Signed-off-by: Karl Hiramoto <karl@hiramoto.org>
---
 include/net/netfilter/nf_conntrack_queue.h |   17 +++++++++++++++++
 1 files changed, 17 insertions(+), 0 deletions(-)
 create mode 100644 include/net/netfilter/nf_conntrack_queue.h

diff --git a/include/net/netfilter/nf_conntrack_queue.h b/include/net/netfilter/nf_conntrack_queue.h
new file mode 100644
index 0000000..417820f
--- /dev/null
+++ b/include/net/netfilter/nf_conntrack_queue.h
@@ -0,0 +1,17 @@
+#ifndef _NF_CONNTRACK_QUEUE_H
+#define _NF_CONNTRACK_QUEUE_H
+
+
+#if defined(CONFIG_NF_QUEUE_CONNBYTES_BYPASS)
+#include <net/netfilter/nf_conntrack_extend.h>
+
+struct nf_conntrack_queue {
+	spinlock_t lock;
+	union {
+		u32     connbytes;
+		u32     tcp_seq; /* TCP sequence number */
+	} dir[IP_CT_DIR_MAX];
+};
+
+#endif
+#endif /* _NF_CONNTRACK_QUEUE_H */
-- 
1.7.1


^ permalink raw reply related	[flat|nested] 10+ messages in thread

* [RFC 3/4] nf_conntrack: add nf_queue extension
  2010-07-24 15:44 [RFC 0/4] nfnetlink_queue bypass queue to userspace X bytes of connection Karl Hiramoto
  2010-07-24 15:44 ` [RFC 1/4] netfilter/Kconfig: NF_QUEUE_CONNBYTES_BYPASS Karl Hiramoto
  2010-07-24 15:44 ` [RFC 2/4] nf_conntrack_queue: define struct that will be stored in nf_ct_extend Karl Hiramoto
@ 2010-07-24 15:44 ` Karl Hiramoto
  2010-07-24 15:44 ` [RFC 4/4] nfnetlink_queue: allow part of a connection to bypass the queue Karl Hiramoto
  2010-07-24 18:26 ` [RFC 0/4] nfnetlink_queue bypass queue to userspace X bytes of connection Pablo Neira Ayuso
  4 siblings, 0 replies; 10+ messages in thread
From: Karl Hiramoto @ 2010-07-24 15:44 UTC (permalink / raw)
  To: netfilter-devel; +Cc: Karl Hiramoto

if CONFIG_NF_QUEUE_CONNBYTES_BYPASS add conntrack extensions to allow bypass nf_queue

Signed-off-by: Karl Hiramoto <karl@hiramoto.org>
---
 include/net/netfilter/nf_conntrack_extend.h |    2 +
 net/netfilter/nf_conntrack_core.c           |   37 ++++++++++++++++++++++++++-
 2 files changed, 38 insertions(+), 1 deletions(-)

diff --git a/include/net/netfilter/nf_conntrack_extend.h b/include/net/netfilter/nf_conntrack_extend.h
index 32d15bd..81f3698 100644
--- a/include/net/netfilter/nf_conntrack_extend.h
+++ b/include/net/netfilter/nf_conntrack_extend.h
@@ -11,6 +11,7 @@ enum nf_ct_ext_id {
 	NF_CT_EXT_ACCT,
 	NF_CT_EXT_ECACHE,
 	NF_CT_EXT_ZONE,
+	NF_CT_EXT_QUEUE,
 	NF_CT_EXT_NUM,
 };
 
@@ -19,6 +20,7 @@ enum nf_ct_ext_id {
 #define NF_CT_EXT_ACCT_TYPE struct nf_conn_counter
 #define NF_CT_EXT_ECACHE_TYPE struct nf_conntrack_ecache
 #define NF_CT_EXT_ZONE_TYPE struct nf_conntrack_zone
+#define NF_CT_EXT_QUEUE_TYPE struct nf_conntrack_queue
 
 /* Extensions: optional stuff which isn't permanently in struct. */
 struct nf_ct_ext {
diff --git a/net/netfilter/nf_conntrack_core.c b/net/netfilter/nf_conntrack_core.c
index eeeb8bc..5b7b105 100644
--- a/net/netfilter/nf_conntrack_core.c
+++ b/net/netfilter/nf_conntrack_core.c
@@ -42,6 +42,7 @@
 #include <net/netfilter/nf_conntrack_extend.h>
 #include <net/netfilter/nf_conntrack_acct.h>
 #include <net/netfilter/nf_conntrack_ecache.h>
+#include <net/netfilter/nf_conntrack_queue.h>
 #include <net/netfilter/nf_conntrack_zones.h>
 #include <net/netfilter/nf_nat.h>
 #include <net/netfilter/nf_nat_core.h>
@@ -632,6 +633,11 @@ struct nf_conn *nf_conntrack_alloc(struct net *net, u16 zone,
 		nf_ct_zone->id = zone;
 	}
 #endif
+
+#if defined(CONFIG_NF_QUEUE_CONNBYTES_BYPASS)
+	if (!nf_ct_ext_add(ct, NF_CT_EXT_QUEUE, gfp))
+		goto out_free;
+#endif
 	/*
 	 * changes to lookup keys must be done before setting refcnt to 1
 	 */
@@ -639,11 +645,14 @@ struct nf_conn *nf_conntrack_alloc(struct net *net, u16 zone,
 	atomic_set(&ct->ct_general.use, 1);
 	return ct;
 
-#ifdef CONFIG_NF_CONNTRACK_ZONES
+#if defined(CONFIG_NF_QUEUE_CONNBYTES_BYPASS) \
+	|| defined(CONFIG_NF_CONNTRACK_ZONES)
+
 out_free:
 	kmem_cache_free(net->ct.nf_conntrack_cachep, ct);
 	return ERR_PTR(-ENOMEM);
 #endif
+
 }
 EXPORT_SYMBOL_GPL(nf_conntrack_alloc);
 
@@ -1010,6 +1019,14 @@ static struct nf_ct_ext_type nf_ct_zone_extend __read_mostly = {
 };
 #endif
 
+#if defined(CONFIG_NF_QUEUE_CONNBYTES_BYPASS)
+static struct nf_ct_ext_type nf_conntrack_queue_extend __read_mostly = {
+	.len    = sizeof(struct nf_conntrack_queue),
+	.align  = __alignof__(struct nf_conntrack_queue),
+	.id     = NF_CT_EXT_QUEUE,
+};
+#endif
+
 #if defined(CONFIG_NF_CT_NETLINK) || defined(CONFIG_NF_CT_NETLINK_MODULE)
 
 #include <linux/netfilter/nfnetlink.h>
@@ -1194,6 +1211,11 @@ static void nf_conntrack_cleanup_init_net(void)
 #ifdef CONFIG_NF_CONNTRACK_ZONES
 	nf_ct_extend_unregister(&nf_ct_zone_extend);
 #endif
+
+#if defined(CONFIG_NF_QUEUE_CONNBYTES_BYPASS)
+	nf_ct_extend_unregister(&nf_conntrack_queue_extend);
+#endif
+
 }
 
 static void nf_conntrack_cleanup_net(struct net *net)
@@ -1362,6 +1384,15 @@ static int nf_conntrack_init_init_net(void)
 	if (ret < 0)
 		goto err_extend;
 #endif
+
+#if defined(CONFIG_NF_QUEUE_CONNBYTES_BYPASS)
+	ret = nf_ct_extend_register(&nf_conntrack_queue_extend);
+	if (ret < 0) {
+		printk(KERN_ERR "nfnetlink_queue: failed to register CT ext.\n");
+		goto err_extend_queue;
+	}
+#endif
+
 	/* Set up fake conntrack: to never be deleted, not in any hashes */
 #ifdef CONFIG_NET_NS
 	nf_conntrack_untracked.ct_net = &init_net;
@@ -1372,7 +1403,11 @@ static int nf_conntrack_init_init_net(void)
 
 	return 0;
 
+#if defined(CONFIG_NF_QUEUE_CONNBYTES_BYPASS)
+err_extend_queue:
+#endif
 #ifdef CONFIG_NF_CONNTRACK_ZONES
+	nf_ct_extend_unregister(&nf_ct_zone_extend);
 err_extend:
 	nf_conntrack_helper_fini();
 #endif
-- 
1.7.1


^ permalink raw reply related	[flat|nested] 10+ messages in thread

* [RFC 4/4] nfnetlink_queue: allow part of a connection to bypass the queue
  2010-07-24 15:44 [RFC 0/4] nfnetlink_queue bypass queue to userspace X bytes of connection Karl Hiramoto
                   ` (2 preceding siblings ...)
  2010-07-24 15:44 ` [RFC 3/4] nf_conntrack: add nf_queue extension Karl Hiramoto
@ 2010-07-24 15:44 ` Karl Hiramoto
  2010-07-24 18:26 ` [RFC 0/4] nfnetlink_queue bypass queue to userspace X bytes of connection Pablo Neira Ayuso
  4 siblings, 0 replies; 10+ messages in thread
From: Karl Hiramoto @ 2010-07-24 15:44 UTC (permalink / raw)
  To: netfilter-devel; +Cc: Karl Hiramoto

If userspace knows that we want to ACCEPT X bytes of the connection,
allow nfnetlink_queue to accept the packets without queuing to userspace.

Signed-off-by: Karl Hiramoto <karl@hiramoto.org>
---
 include/linux/netfilter/nfnetlink_queue.h |    1 +
 net/netfilter/nfnetlink_queue.c           |  112 ++++++++++++++++++++++++++++-
 2 files changed, 112 insertions(+), 1 deletions(-)

diff --git a/include/linux/netfilter/nfnetlink_queue.h b/include/linux/netfilter/nfnetlink_queue.h
index 2455fe5..f3dd99b 100644
--- a/include/linux/netfilter/nfnetlink_queue.h
+++ b/include/linux/netfilter/nfnetlink_queue.h
@@ -41,6 +41,7 @@ enum nfqnl_attr_type {
 	NFQA_IFINDEX_PHYSOUTDEV,	/* __u32 ifindex */
 	NFQA_HWADDR,			/* nfqnl_msg_packet_hw */
 	NFQA_PAYLOAD,			/* opaque data payload */
+	NFQA_ACCEPT_CONNBYTES, /* Accept the next N bytes, bypassing queue  */
 
 	__NFQA_MAX
 };
diff --git a/net/netfilter/nfnetlink_queue.c b/net/netfilter/nfnetlink_queue.c
index 12e1ab3..b5f20c8 100644
--- a/net/netfilter/nfnetlink_queue.c
+++ b/net/netfilter/nfnetlink_queue.c
@@ -30,6 +30,10 @@
 #include <linux/list.h>
 #include <net/sock.h>
 #include <net/netfilter/nf_queue.h>
+#if defined(CONFIG_NF_QUEUE_CONNBYTES_BYPASS)
+#include <net/netfilter/nf_conntrack_queue.h>
+#include <linux/tcp.h>
+#endif
 
 #include <asm/atomic.h>
 
@@ -385,6 +389,67 @@ nla_put_failure:
 	return NULL;
 }
 
+#if defined(CONFIG_NF_QUEUE_CONNBYTES_BYPASS)
+static struct tcphdr *__get_tcp_hdr(const struct sk_buff *skb, uint8_t prot)
+{
+	if (prot == PF_INET)
+		return (struct tcphdr *)(skb->data + (ip_hdr(skb)->ihl * 4));
+	else if (prot == PF_INET6)
+		return (struct tcphdr *)(skb->data + 40);
+	else
+		BUG();
+
+	return NULL;
+}
+
+static int check_entry_reinject(struct nf_queue_entry *entry)
+{
+	struct nf_conn *ct;
+	struct nf_conntrack_queue *ct_queue;
+	enum ip_conntrack_info ctinfo;
+	enum ip_conntrack_dir dir;
+	const struct tcphdr *tcphdr;
+	ct = nf_ct_get(entry->skb, &ctinfo);
+
+	if (!ct || !nf_ct_is_confirmed(ct))
+		return -ENOENT;
+
+	ct_queue = nf_ct_ext_find(ct, NF_CT_EXT_QUEUE);
+	if (!ct_queue)
+		return -ENOENT;
+
+	spin_lock_bh(&ct_queue->lock);
+	pr_debug("%s:%d has ctinfo=%u\n", __func__, __LINE__, ctinfo);
+	dir = CTINFO2DIR(ctinfo);
+	if (ct_queue->dir[dir].connbytes) {
+		if (nf_ct_protonum(ct) == IPPROTO_TCP) {
+			tcphdr = __get_tcp_hdr(entry->skb, nf_ct_l3num(ct));
+
+			if (ntohl(tcphdr->seq) < ct_queue->dir[dir].tcp_seq)
+				goto reinject;
+			else
+				ct_queue->dir[dir].tcp_seq = 0; /* clear */
+		} else {
+			/* other proto */
+			if (ct_queue->dir[dir].connbytes < entry->skb->len) {
+				ct_queue->dir[dir].connbytes -= entry->skb->len;
+				goto reinject;
+			} else
+				ct_queue->dir[dir].connbytes = 0; /* clear */
+		}
+	}
+	spin_unlock_bh(&ct_queue->lock);
+
+
+	return -ENOENT;
+
+reinject:
+	spin_unlock_bh(&ct_queue->lock);
+	nf_reinject(entry, NF_ACCEPT);
+	return 0;
+}
+#endif
+
 static int
 nfqnl_enqueue_packet(struct nf_queue_entry *entry, unsigned int queuenum)
 {
@@ -400,6 +465,11 @@ nfqnl_enqueue_packet(struct nf_queue_entry *entry, unsigned int queuenum)
 	if (queue->copy_mode == NFQNL_COPY_NONE)
 		goto err_out;
 
+#if defined(CONFIG_NF_QUEUE_CONNBYTES_BYPASS)
+	if (!check_entry_reinject(entry))
+		return 0; /* bypass without enqueueing packet */
+#endif
+
 	nskb = nfqnl_build_packet_message(queue, entry);
 	if (nskb == NULL)
 		goto err_out;
@@ -604,6 +674,40 @@ static const struct nla_policy nfqa_verdict_policy[NFQA_MAX+1] = {
 	[NFQA_PAYLOAD]		= { .type = NLA_UNSPEC },
 };
 
+
+#if defined(CONFIG_NF_QUEUE_CONNBYTES_BYPASS)
+static int recv_connbytes(u32 connbytes, struct nfqnl_instance *queue,
+	struct nf_queue_entry *entry)
+{
+	struct nf_conn *ct;
+	struct nf_conntrack_queue *ct_queue;
+	enum ip_conntrack_info ctinfo;
+	enum ip_conntrack_dir dir;
+	const struct tcphdr *tcphdr;
+
+	ct = nf_ct_get(entry->skb, &ctinfo);
+	if (!ct) {
+		pr_err("nf_queue: no conntrack info\n");
+		return -ENOENT;
+	}
+
+	dir = CTINFO2DIR(ctinfo);
+	ct_queue = nf_ct_ext_find(ct, NF_CT_EXT_QUEUE);
+	if (!ct_queue) {
+		pr_err("nf_queue: no conntrack queue ext\n");
+		return -ENOENT;
+	}
+
+	if (nf_ct_protonum(ct) == IPPROTO_TCP) {
+		tcphdr = __get_tcp_hdr(entry->skb, nf_ct_l3num(ct));
+		ct_queue->dir[dir].tcp_seq = connbytes;
+	} else
+		ct_queue->dir[dir].connbytes = connbytes;
+
+	return 0;
+}
+#endif
+
 static int
 nfqnl_recv_verdict(struct sock *ctnl, struct sk_buff *skb,
 		   const struct nlmsghdr *nlh,
@@ -611,7 +715,6 @@ nfqnl_recv_verdict(struct sock *ctnl, struct sk_buff *skb,
 {
 	struct nfgenmsg *nfmsg = NLMSG_DATA(nlh);
 	u_int16_t queue_num = ntohs(nfmsg->res_id);
-
 	struct nfqnl_msg_verdict_hdr *vhdr;
 	struct nfqnl_instance *queue;
 	unsigned int verdict;
@@ -659,6 +762,13 @@ nfqnl_recv_verdict(struct sock *ctnl, struct sk_buff *skb,
 	if (nfqa[NFQA_MARK])
 		entry->skb->mark = ntohl(nla_get_be32(nfqa[NFQA_MARK]));
 
+#if defined(CONFIG_NF_QUEUE_CONNBYTES_BYPASS)
+	if (nfqa[NFQA_ACCEPT_CONNBYTES])
+		recv_connbytes(
+			ntohl(nla_get_be32(nfqa[NFQA_ACCEPT_CONNBYTES])),
+			queue, entry);
+#endif
+
 	nf_reinject(entry, verdict);
 	return 0;
 
-- 
1.7.1


^ permalink raw reply related	[flat|nested] 10+ messages in thread

* Re: [RFC 0/4] nfnetlink_queue bypass queue to userspace X bytes of connection
  2010-07-24 15:44 [RFC 0/4] nfnetlink_queue bypass queue to userspace X bytes of connection Karl Hiramoto
                   ` (3 preceding siblings ...)
  2010-07-24 15:44 ` [RFC 4/4] nfnetlink_queue: allow part of a connection to bypass the queue Karl Hiramoto
@ 2010-07-24 18:26 ` Pablo Neira Ayuso
  2010-07-25  6:55   ` Karl Hiramoto
  4 siblings, 1 reply; 10+ messages in thread
From: Pablo Neira Ayuso @ 2010-07-24 18:26 UTC (permalink / raw)
  To: Karl Hiramoto; +Cc: netfilter-devel

On 24/07/10 17:44, Karl Hiramoto wrote:
> Hi, I'm working on a nf_queue based HTTP filter. Sometimes I want to
> NF_ACCEPT X bytes of a connection. In a HTTP connection its part of the
> Content-Length.  Being able to directly NF_ACCEPT large parts of a connection
> is a 2X to 3X speedup, by not queuing these packets to user-space.

I think that you could avoid this extensions by means of conntrack marks
and the string match. Could you elaborate your application, please?

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [RFC 0/4] nfnetlink_queue bypass queue to userspace X bytes of connection
  2010-07-24 18:26 ` [RFC 0/4] nfnetlink_queue bypass queue to userspace X bytes of connection Pablo Neira Ayuso
@ 2010-07-25  6:55   ` Karl Hiramoto
  2010-07-25 10:42     ` Pablo Neira Ayuso
  0 siblings, 1 reply; 10+ messages in thread
From: Karl Hiramoto @ 2010-07-25  6:55 UTC (permalink / raw)
  To: Pablo Neira Ayuso; +Cc: netfilter-devel

 On 07/24/2010 08:26 PM, Pablo Neira Ayuso wrote:
> On 24/07/10 17:44, Karl Hiramoto wrote:
>> Hi, I'm working on a nf_queue based HTTP filter. Sometimes I want to
>> NF_ACCEPT X bytes of a connection. In a HTTP connection its part of the
>> Content-Length.  Being able to directly NF_ACCEPT large parts of a connection
>> is a 2X to 3X speedup, by not queuing these packets to user-space.
> I think that you could avoid this extensions by means of conntrack marks
> and the string match. Could you elaborate your application, please?
I thought about that before i made the extension but there's no way, to do it reliably and avoid race conditions.  string matches are also really inefficient, with string matches every packet with a certain mark would be scanned, but its not necessary since in HTTP because you know which part of the message you want to scan, but it changes on every request/response.

It's  a HTTP 1.1 filter, so it has to support persistent connections, multiple requests per connection.  There is the possibility that some requests are allowed and others are not.   Using connmark its posible to allow an entire connection to NF_ACCEPT bypassing the queue, however not really possible when there are multiple requests per connection.

Some of the features of my app are:
* Filter Accept/block  by URL, or HTTP "Host:" tag
* Log all request/responses,   URL, content length, time duration of request / response.
* Parts of a HTTP 1.1 persistent connection can be rejected, other parts accepted.


--
Karl Hiramoto


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [RFC 0/4] nfnetlink_queue bypass queue to userspace X bytes of connection
  2010-07-25  6:55   ` Karl Hiramoto
@ 2010-07-25 10:42     ` Pablo Neira Ayuso
  2010-07-26  6:50       ` Karl Hiramoto
       [not found]       ` <4C4D2C33.6050901@hiramoto.org>
  0 siblings, 2 replies; 10+ messages in thread
From: Pablo Neira Ayuso @ 2010-07-25 10:42 UTC (permalink / raw)
  To: Karl Hiramoto; +Cc: netfilter-devel

On 25/07/10 08:55, Karl Hiramoto wrote:
>  On 07/24/2010 08:26 PM, Pablo Neira Ayuso wrote:
>> I think that you could avoid this extensions by means of conntrack marks
>> and the string match. Could you elaborate your application, please?
> I thought about that before i made the extension but there's no way, to do it reliably and avoid race conditions.  string matches are also really inefficient, with string matches every packet with a certain mark would be scanned

You can limit the string matching for only a few bytes in the very
beginning of the packet. If you find a matching, you can pass the packet
to user-space. Using the Boyer-Moore approach provides efficient searching.

> but its not necessary since in HTTP because you know which part of the message you want to scan, but it changes on every request/response.
> 
> It's  a HTTP 1.1 filter, so it has to support persistent connections, multiple requests per connection.
> There is the possibility that some requests are allowed and others are not.   Using connmark its posible to allow an entire connection to NF_ACCEPT bypassing the queue, however not really possible when there are multiple requests per connection.
> 
> Some of the features of my app are:
> * Filter Accept/block  by URL, or HTTP "Host:" tag
> * Log all request/responses,   URL, content length, time duration of request / response.
> * Parts of a HTTP 1.1 persistent connection can be rejected, other parts accepted.

This extension seems to me very specific for HTTP/1.1.

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [RFC 0/4] nfnetlink_queue bypass queue to userspace X bytes of connection
  2010-07-25 10:42     ` Pablo Neira Ayuso
@ 2010-07-26  6:50       ` Karl Hiramoto
       [not found]       ` <4C4D2C33.6050901@hiramoto.org>
  1 sibling, 0 replies; 10+ messages in thread
From: Karl Hiramoto @ 2010-07-26  6:50 UTC (permalink / raw)
  To: Pablo Neira Ayuso; +Cc: netfilter-devel

 On 25/07/2010 12:42, Pablo Neira Ayuso wrote:
>
> You can limit the string matching for only a few bytes in the very
> beginning of the packet. 
That really doesn't help trying to find the "Host:" or path of or URL in HTTP because you don't know variables like cookie length, or other variables. String match also doesn't help me at all if the string is split across multiple packets


> This extension seems to me very specific for HTTP/1.1.
HTTP is the most popular protocol on the internet[1][2][3], optimizing the most common case has merits.

Besides HTTP  I can imagine this extension helping implementing a POP3 or IMAP filter using NF_QUEUE.   For example many network UTM devices that scan attachments for viruses or other blocked content, will skip a compressed file that is over X bytes because there is not enough free memory to decompress and scan it.   In this case you could bypass the queue for X bytes, then continue scanning smaller files.


[1]
http://torrentfreak.com/http-traffic-overtakes-p2p-courtesy-of-youtube/

[2]
http://www.nanog.org/meetings/nanog47/abstracts.php?pt=MTQ1MyZuYW5vZzQ3&nm=nanog47 <http://www.nanog.org/meetings/nanog47/abstracts.php?pt=MTQ1MyZuYW5vZzQ3&nm=nanog47>
[3]
http://www.cisco.com/en/US/netsol/ns827/networking_solutions_sub_solution.html#~forecast <http://www.cisco.com/en/US/netsol/ns827/networking_solutions_sub_solution.html#%7Eforecast>
NOTE talking about video being the most popular, a lot of video is delivered over HTTP. 


--
Karl



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [RFC 0/4] nfnetlink_queue bypass queue to userspace X bytes of connection
       [not found]       ` <4C4D2C33.6050901@hiramoto.org>
@ 2010-07-26 17:35         ` Pablo Neira Ayuso
  0 siblings, 0 replies; 10+ messages in thread
From: Pablo Neira Ayuso @ 2010-07-26 17:35 UTC (permalink / raw)
  To: Karl Hiramoto; +Cc: netfilter-devel

On 26/07/10 08:33, Karl Hiramoto wrote:
>  On 25/07/2010 12:42, Pablo Neira Ayuso wrote:
>>
>> You can limit the string matching for only a few bytes in the very
>> beginning of the packet. 
> That really doesn't help trying to find the "Host:" or path of or URL in
> HTTP because you don't know variables like cookie length, or other
> variables. String match also doesn't help me at all if the string is
> split across multiple packets

We can look for some string (in a limited range of bytes) that
preliminarily identifies some sort of traffic in kernel-space. In case
of matching, we pass the packet (or some short part of it) to user-space
via NFQUEUE. The whole specific packet parsing (such as looking at the
Host tokens of HTTP/1.1 that you refer) is done in user-space.

>> This extension seems to me very specific for HTTP/1.1.
> HTTP is the most popular protocol on the internet[1][2][3], optimizing
> the most common case has merits.
> 
> Besides HTTP  I can imagine this extension helping implementing a POP3
> or IMAP filter using NF_QUEUE.   For example many network UTM devices
> that scan attachments for viruses or other blocked content, will skip a
> compressed file that is over X bytes because there is not enough free
> memory to decompress and scan it.   In this case you could bypass the
> queue for X bytes, then continue scanning smaller files.

Making assumptions on the Content-Length and any other information in
the packet seems wrong to me. Someone could forge a packet to by-pass
filtering.

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2010-07-26 17:35 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-07-24 15:44 [RFC 0/4] nfnetlink_queue bypass queue to userspace X bytes of connection Karl Hiramoto
2010-07-24 15:44 ` [RFC 1/4] netfilter/Kconfig: NF_QUEUE_CONNBYTES_BYPASS Karl Hiramoto
2010-07-24 15:44 ` [RFC 2/4] nf_conntrack_queue: define struct that will be stored in nf_ct_extend Karl Hiramoto
2010-07-24 15:44 ` [RFC 3/4] nf_conntrack: add nf_queue extension Karl Hiramoto
2010-07-24 15:44 ` [RFC 4/4] nfnetlink_queue: allow part of a connection to bypass the queue Karl Hiramoto
2010-07-24 18:26 ` [RFC 0/4] nfnetlink_queue bypass queue to userspace X bytes of connection Pablo Neira Ayuso
2010-07-25  6:55   ` Karl Hiramoto
2010-07-25 10:42     ` Pablo Neira Ayuso
2010-07-26  6:50       ` Karl Hiramoto
     [not found]       ` <4C4D2C33.6050901@hiramoto.org>
2010-07-26 17:35         ` Pablo Neira Ayuso

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).