[PATCH 0/4] [RFC] nfnetlink_acct: Traffic-based and periodic notifications

[PATCH 0/4] [RFC] nfnetlink_acct: Traffic-based and periodic notifications

[valentina.giusti]

From: Valentina Giusti <valentina.giusti@bmw-carit.de>

This patchset introduces notifications of accounting objects to userspace for
nfacct.
Notifications can be sent periodically or based on the amount of accounted bytes
or packets. In case a notification is set by giving a period as well as a given
amount of traffic, the number of notifications triggered during a period can be
rate limited.

The need for notifications stems from monitoring applications (e.g. connman)
which currently need to keep polling accounting objects via nfnl_acct_get() in 
order to get updated statistics. This is far from ideal in scenarios with large
amounts of accounting objects and diverse, unpredictable network traffic.

This implementation introduces a new message type, NFNL_MSG_ACCT_NOTIFY, which
is sent by userspace (typically, libnetfilter_acct) in order to configure
accounting notifications.
The new message parameters are NFACCT_NOTIFY_PACKETS, NFACCT_NOTIFY_BYTES and
NFACCT_NOTIFY_PERIOD, which are sent with the accounting object name. Userspace
applications would typically configure desired notification modes and then get 
appropriate notifications via netlink, on the same socket used for said
configuration.

The patches needed for libnetfilter_acct and nfacct are included in this series.

An example of nfacct usage is:

$ nfacct add http-traffic # Create the accounting object

[create related iptables rules here]

$ nfacct notify http-traffic bytes 1024
[for every 1024 bytes of HTTP traffic:]
{ pkts = 00000000000000039225, bytes = 00000000000002361568 } = http-traffic;
{ pkts = 00000000000000039245, bytes = 00000000000002362608 } = http-traffic;
...

or:

$ nfacct notify-traffic http packets 100
[for every 100 packets of HTTP traffic:]
{ pkts = 00000000000000008303, bytes = 00000000000000486469 } = http-traffic;
{ pkts = 00000000000000008403, bytes = 00000000000000494581 } = http-traffic;
...

or:

$ nfacct notify http-traffic interval 120
[every two minutes:]
{ pkts = 00000000000000008303, bytes = 00000000000000486469 } = http-traffic;
{ pkts = 00000000000000008403, bytes = 00000000000000494581 } = http-traffic;
...

or:

$ nfacct notify http-traffic bytes 1024 bytes-ratelimit 4 period 5
{ pkts = 00000000000000010172, bytes = 00000000000000628714 } = http-traffic;
{ pkts = 00000000000000010192, bytes = 00000000000000629754 } = http-traffic;
{ pkts = 00000000000000010212, bytes = 00000000000000630794 } = http-traffic;
{ pkts = 00000000000000010232, bytes = 00000000000000631834 } = http-traffic;
{ pkts = 00000000000000010894, bytes = 00000000000000672343 } = http-traffic;
...


-- 
1.7.10.4

[PATCH 1/4] nfnetlink_acct: Traffic-based and periodic notifications

[valentina.giusti]

From: Valentina Giusti <valentina.giusti@bmw-carit.de>

This patch introduces notifications of accounting objects to userspace for
nfacct.
Notifications can be sent based on intervals expressed as bytes and packets
or as time periods.
When the notification request specifies both a traffic target (bytes or
packets) and a time target (period), additional parameters are available,
which allow to specify a limit for the amount of notifications based on
accounted packets or bytes to be sent during a single period.

Signed-off-by: Valentina Giusti <valentina.giusti@bmw-carit.de>
Cc: Pablo Neira Ayuso <pablo@netfilter.org>
Cc: Patrick McHardy <kaber@trash.net>
Cc: Jozsef Kadlecsik <kadlec@blackhole.kfki.hu>
Cc: "David S. Miller" <davem@davemloft.net>
---
 include/uapi/linux/netfilter/nfnetlink_acct.h |    6 +
 net/netfilter/nfnetlink_acct.c                |  280 +++++++++++++++++++++++++
 2 files changed, 286 insertions(+)

diff --git a/include/uapi/linux/netfilter/nfnetlink_acct.h b/include/uapi/linux/netfilter/nfnetlink_acct.h
index c7b6269..8125e57 100644
--- a/include/uapi/linux/netfilter/nfnetlink_acct.h
+++ b/include/uapi/linux/netfilter/nfnetlink_acct.h
@@ -10,6 +10,7 @@ enum nfnl_acct_msg_types {
 	NFNL_MSG_ACCT_GET,
 	NFNL_MSG_ACCT_GET_CTRZERO,
 	NFNL_MSG_ACCT_DEL,
+	NFNL_MSG_ACCT_NOTIFY,
 	NFNL_MSG_ACCT_MAX
 };
 
@@ -19,6 +20,11 @@ enum nfnl_acct_type {
 	NFACCT_PKTS,
 	NFACCT_BYTES,
 	NFACCT_USE,
+	NFACCT_NOTIFY_PKTS,
+	NFACCT_NOTIFY_P_RL,	/* packets notification rate limit */
+	NFACCT_NOTIFY_BYTES,
+	NFACCT_NOTIFY_B_RL,	/* bytes notification rate limit */
+	NFACCT_NOTIFY_PERIOD,
 	__NFACCT_MAX
 };
 #define NFACCT_MAX (__NFACCT_MAX - 1)
diff --git a/net/netfilter/nfnetlink_acct.c b/net/netfilter/nfnetlink_acct.c
index c7b6d46..5327c2b 100644
--- a/net/netfilter/nfnetlink_acct.c
+++ b/net/netfilter/nfnetlink_acct.c
@@ -35,9 +35,56 @@ struct nf_acct {
 	struct list_head	head;
 	atomic_t		refcnt;
 	char			name[NFACCT_NAME_MAX];
+	struct list_head	notify_list;
 	struct rcu_head		rcu_head;
 };
 
+/**
+ * struct nf_acct_notify - struct for netfilter accounting notification request
+ *
+ * @notify_rem_pkts:	packets left before the next packet-based notification
+ * @notify_rem_bytes:	bytes left before the next byte-based notification
+ * @notify_limit_pkts:	notifications limit count before the next period-based
+ *			notification
+ * @notify_limit_bytes:	notifications limit count before the next period-based
+ *			notification
+ * @head:		head of the list of notifications active on a single
+ *			nf_acct object
+ * @portid:		ID of the userspace process requesting notifications
+ * @nfnl:		netlink socket descriptor
+ * @nfacct:		pointer to the nf_acct object
+ * @pkts:		traffic target in packets
+ * @pkts_rl:		packets notification rate limit (only in combination
+ *			with period)
+ * @bytes:		traffic target in bytes
+ * @bytes_rl:		bytes notification rate limit (only in combination with
+ *			period)
+ * @period:		time notification target in seconds
+ * @work_t:		traffic-based notifications work
+ * @work_p:		period-based notifications work
+ * @lock:		lock to protect the notify list members upon deletion
+ *
+ */
+
+struct nf_acct_notify {
+	atomic_t		notify_rem_pkts;
+	atomic_t		notify_rem_bytes;
+	atomic_t		notify_limit_pkts;
+	atomic_t		notify_limit_bytes;
+	struct list_head	head;
+	u32			portid;
+	struct sock		*nfnl;
+	struct nf_acct		*nfacct;
+	u32			pkts;
+	u32			pkts_rl;
+	u32			bytes;
+	u32			bytes_rl;
+	u32			period;
+	struct work_struct	work_t;
+	struct delayed_work	work_p;
+	struct mutex		lock;
+};
+
 static int
 nfnl_acct_new(struct sock *nfnl, struct sk_buff *skb,
 	     const struct nlmsghdr *nlh, const struct nlattr * const tb[])
@@ -87,6 +134,7 @@ nfnl_acct_new(struct sock *nfnl, struct sk_buff *skb,
 		atomic64_set(&nfacct->pkts,
 			     be64_to_cpu(nla_get_be64(tb[NFACCT_PKTS])));
 	}
+	INIT_LIST_HEAD(&nfacct->notify_list);
 	atomic_set(&nfacct->refcnt, 1);
 	list_add_tail_rcu(&nfacct->head, &nfnl_acct_list);
 	return 0;
@@ -219,13 +267,204 @@ nfnl_acct_get(struct sock *nfnl, struct sk_buff *skb,
 	return ret;
 }
 
+static int
+nfnl_acct_notify_send(struct nf_acct_notify *n)
+{
+	struct sk_buff *skb;
+	int ret;
+
+	skb = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
+	if (skb == NULL)
+		return -ENOMEM;
+
+	ret = nfnl_acct_fill_info(skb, n->portid, 0, NFNL_MSG_ACCT_NOTIFY,
+				  NFNL_MSG_ACCT_NEW, n->nfacct);
+	if (ret <= 0) {
+		kfree_skb(skb);
+		return ret;
+	}
+
+	return netlink_unicast(n->nfnl, skb, n->portid, MSG_DONTWAIT);
+}
+
+static void
+nfnl_acct_notify_work_traffic(struct work_struct *work)
+{
+	struct nf_acct_notify *n = container_of(work, struct nf_acct_notify,
+						work_t);
+	struct nf_acct_notify *cur, *tmp;
+	int ret;
+
+	ret = nfnl_acct_notify_send(n);
+	if (ret == -ECONNREFUSED) {
+		if (n->period) {
+			ret = mutex_trylock(&n->lock);
+			if (!ret)
+				return;
+		}
+		list_for_each_entry_safe(cur, tmp, &n->nfacct->notify_list,
+					 head) {
+			if (cur->portid != n->portid)
+				continue;
+			if (n->period) {
+				cancel_delayed_work_sync(&cur->work_p);
+				mutex_unlock(&n->lock);
+			}
+			list_del(&cur->head);
+			kfree(cur);
+			return;
+		}
+	}
+}
+
+static void
+nfnl_acct_notify_work_period(struct work_struct *work)
+{
+	struct nf_acct_notify *n = container_of((struct delayed_work *)work,
+						struct nf_acct_notify, work_p);
+	struct nf_acct_notify *cur, *tmp;
+	int ret;
+
+	ret = nfnl_acct_notify_send(n);
+	if (ret == -ECONNREFUSED) {
+		if (n->bytes || n->pkts) {
+			ret = mutex_trylock(&n->lock);
+			if (!ret)
+				return;
+		}
+		list_for_each_entry_safe(cur, tmp, &n->nfacct->notify_list,
+					 head) {
+			if (cur->portid != n->portid)
+				continue;
+			if (n->bytes || n->pkts) {
+				cancel_work_sync(&cur->work_t);
+				mutex_unlock(&n->lock);
+			}
+			list_del(&cur->head);
+			kfree(cur);
+			return;
+		}
+	}
+
+	if (n->pkts_rl) {
+		atomic_set(&n->notify_limit_pkts, n->pkts_rl);
+		atomic_set(&n->notify_rem_pkts, n->pkts);
+	}
+	if (n->bytes_rl) {
+		atomic_set(&n->notify_limit_bytes, n->bytes_rl);
+		atomic_set(&n->notify_rem_pkts, n->pkts);
+	}
+
+	schedule_delayed_work(&n->work_p, n->period * HZ);
+}
+
+static int
+nfnl_acct_notify(struct sock *nfnl, struct sk_buff *skb,
+		 const struct nlmsghdr *nlh, const struct nlattr * const tb[])
+{
+	struct nf_acct *cur;
+	char *name;
+	u32 pkts = 0, pkts_rl = 0, bytes = 0, bytes_rl = 0, period = 0;
+	struct nf_acct_notify *new;
+	int ret = -ENOENT;
+
+	if (!tb[NFACCT_NAME])
+		return -EINVAL;
+	name = nla_data(tb[NFACCT_NAME]);
+
+	if (tb[NFACCT_NOTIFY_PKTS])
+		pkts = be32_to_cpu(nla_get_be32(tb[NFACCT_NOTIFY_PKTS]));
+
+	if (tb[NFACCT_NOTIFY_BYTES])
+		bytes = be32_to_cpu(nla_get_be32(tb[NFACCT_NOTIFY_BYTES]));
+
+	if (tb[NFACCT_NOTIFY_PERIOD])
+		period = be32_to_cpu(nla_get_be32(tb[NFACCT_NOTIFY_PERIOD]));
+
+	if (tb[NFACCT_NOTIFY_P_RL] && period)
+		pkts_rl = be32_to_cpu(nla_get_be32(tb[NFACCT_NOTIFY_P_RL]));
+	if (tb[NFACCT_NOTIFY_P_RL] && !period)
+		return -EINVAL;
+
+	if (tb[NFACCT_NOTIFY_B_RL] && period)
+		bytes_rl = be32_to_cpu(nla_get_be32(tb[NFACCT_NOTIFY_B_RL]));
+	if (tb[NFACCT_NOTIFY_B_RL] && !period)
+		return -EINVAL;
+
+	list_for_each_entry(cur, &nfnl_acct_list, head) {
+		if (strncmp(cur->name, name, NFACCT_NAME_MAX))
+			continue;
+
+		ret = 0;
+		new = kzalloc(sizeof(struct nf_acct_notify), GFP_KERNEL);
+		if (!new)
+			return -ENOMEM;
+
+		if (pkts) {
+			atomic_set(&new->notify_rem_pkts, pkts);
+			new->pkts = pkts;
+			if (pkts_rl) {
+				atomic_set(&new->notify_limit_pkts, pkts_rl);
+				new->pkts_rl = pkts_rl;
+			} else {
+				atomic_set(&new->notify_limit_pkts, -1);
+				new->pkts_rl = 0;
+			}
+		}
+
+		if (bytes) {
+			atomic_set(&new->notify_rem_bytes, bytes);
+			new->bytes = bytes;
+			if (bytes_rl) {
+				atomic_set(&new->notify_limit_bytes, bytes_rl);
+				new->bytes_rl = bytes_rl;
+			} else {
+				atomic_set(&new->notify_limit_bytes, -1);
+				new->bytes_rl = 0;
+			}
+		}
+
+		if (period)
+			new->period = period;
+
+		new->portid = NETLINK_CB(skb).portid;
+		new->nfnl = nfnl;
+		new->nfacct = cur;
+
+		list_add(&new->head, &cur->notify_list);
+
+		if ((new->bytes || new->pkts) && new->period)
+			mutex_init(&new->lock);
+
+		if (new->bytes || new->pkts)
+			INIT_WORK(&new->work_t, nfnl_acct_notify_work_traffic);
+		if (new->period) {
+			INIT_DELAYED_WORK(&new->work_p,
+					  nfnl_acct_notify_work_period);
+			schedule_delayed_work(&new->work_p, new->period * HZ);
+		}
+	}
+
+	return ret;
+}
+
 /* try to delete object, fail if it is still in use. */
 static int nfnl_acct_try_del(struct nf_acct *cur)
 {
 	int ret = 0;
+	struct nf_acct_notify *n_cur, *n_tmp;
 
 	/* we want to avoid races with nfnl_acct_find_get. */
 	if (atomic_dec_and_test(&cur->refcnt)) {
+		list_for_each_entry_safe(n_cur, n_tmp, &cur->notify_list,
+					 head) {
+			if (n_cur->pkts || n_cur->bytes)
+				cancel_work_sync(&n_cur->work_t);
+			if (n_cur->period)
+				cancel_delayed_work_sync(&n_cur->work_p);
+			list_del(&n_cur->head);
+			kfree(n_cur);
+		}
 		/* We are protected by nfnl mutex. */
 		list_del_rcu(&cur->head);
 		kfree_rcu(cur, rcu_head);
@@ -270,6 +509,11 @@ static const struct nla_policy nfnl_acct_policy[NFACCT_MAX+1] = {
 	[NFACCT_NAME] = { .type = NLA_NUL_STRING, .len = NFACCT_NAME_MAX-1 },
 	[NFACCT_BYTES] = { .type = NLA_U64 },
 	[NFACCT_PKTS] = { .type = NLA_U64 },
+	[NFACCT_NOTIFY_BYTES] = { .type = NLA_U32 },
+	[NFACCT_NOTIFY_B_RL] = { .type = NLA_U32 },
+	[NFACCT_NOTIFY_PKTS] = { .type = NLA_U32 },
+	[NFACCT_NOTIFY_P_RL] = { .type = NLA_U32 },
+	[NFACCT_NOTIFY_PERIOD] = { .type = NLA_U32 },
 };
 
 static const struct nfnl_callback nfnl_acct_cb[NFNL_MSG_ACCT_MAX] = {
@@ -285,6 +529,9 @@ static const struct nfnl_callback nfnl_acct_cb[NFNL_MSG_ACCT_MAX] = {
 	[NFNL_MSG_ACCT_DEL]		= { .call = nfnl_acct_del,
 					    .attr_count = NFACCT_MAX,
 					    .policy = nfnl_acct_policy },
+	[NFNL_MSG_ACCT_NOTIFY]		= { .call = nfnl_acct_notify,
+					    .attr_count = NFACCT_MAX,
+					    .policy = nfnl_acct_policy },
 };
 
 static const struct nfnetlink_subsystem nfnl_acct_subsys = {
@@ -331,8 +578,31 @@ EXPORT_SYMBOL_GPL(nfnl_acct_put);
 
 void nfnl_acct_update(const struct sk_buff *skb, struct nf_acct *nfacct)
 {
+	int tmp, limit;
+	struct nf_acct_notify *cur;
+
 	atomic64_inc(&nfacct->pkts);
 	atomic64_add(skb->len, &nfacct->bytes);
+
+	list_for_each_entry(cur, &nfacct->notify_list, head) {
+		limit = atomic_read(&cur->notify_limit_pkts);
+		if (cur->pkts &&
+		    atomic_dec_and_test(&cur->notify_rem_pkts) && limit) {
+			schedule_work(&cur->work_t);
+			atomic_set(&cur->notify_rem_pkts, cur->pkts);
+			if (limit > 0)
+				atomic_dec(&cur->notify_limit_pkts);
+		}
+
+		tmp = atomic_sub_return(skb->len, &cur->notify_rem_bytes);
+		limit = atomic_read(&cur->notify_limit_bytes);
+		if (cur->bytes && (tmp <= 0) && limit) {
+			schedule_work(&cur->work_t);
+			atomic_set(&cur->notify_rem_bytes, cur->bytes - tmp);
+			if (limit > 0)
+				atomic_dec(&cur->notify_limit_bytes);
+		}
+	}
 }
 EXPORT_SYMBOL_GPL(nfnl_acct_update);
 
@@ -354,11 +624,21 @@ err_out:
 static void __exit nfnl_acct_exit(void)
 {
 	struct nf_acct *cur, *tmp;
+	struct nf_acct_notify *n_cur, *n_tmp;
 
 	pr_info("nfnl_acct: unregistering from nfnetlink.\n");
 	nfnetlink_subsys_unregister(&nfnl_acct_subsys);
 
 	list_for_each_entry_safe(cur, tmp, &nfnl_acct_list, head) {
+		list_for_each_entry_safe(n_cur, n_tmp, &cur->notify_list,
+					 head) {
+			if (n_cur->pkts || n_cur->bytes)
+				cancel_work_sync(&n_cur->work_t);
+			if (n_cur->period)
+				cancel_delayed_work_sync(&n_cur->work_p);
+			list_del(&n_cur->head);
+			kfree(n_cur);
+		}
 		list_del_rcu(&cur->head);
 		/* We are sure that our objects have no clients at this point,
 		 * it's safe to release them all without checking refcnt. */
-- 
1.7.10.4

[PATCH 2/4] libnetfilter-acct: Introduce support for notifications

[valentina.giusti]

From: Valentina Giusti <valentina.giusti@bmw-carit.de>

Introduce support for notifications. Intervals can be configured as bytes
and packets or as time periods. When the intervals are configured in both
ways, it's also possible to specify a limit for the amount of
notifications based on accounted packets or bytes during a single period.

Signed-off-by: Valentina Giusti <valentina.giusti@bmw-carit.de>
Cc: Pablo Neira Ayuso <pablo@netfilter.org>
Cc: Patrick McHardy <kaber@trash.net>
Cc: Jozsef Kadlecsik <kadlec@blackhole.kfki.hu>
Cc: "David S. Miller" <davem@davemloft.net>
---
 include/libnetfilter_acct/libnetfilter_acct.h |    5 +++
 include/linux/netfilter/nfnetlink_acct.h      |    6 ++++
 src/libnetfilter_acct.c                       |   45 +++++++++++++++++++++++++
 3 files changed, 56 insertions(+)

diff --git a/include/libnetfilter_acct/libnetfilter_acct.h b/include/libnetfilter_acct/libnetfilter_acct.h
index b00e366..b0a5bf6 100644
--- a/include/libnetfilter_acct/libnetfilter_acct.h
+++ b/include/libnetfilter_acct/libnetfilter_acct.h
@@ -14,6 +14,11 @@ enum nfacct_attr_type {
 	NFACCT_ATTR_NAME = 0,
 	NFACCT_ATTR_PKTS,
 	NFACCT_ATTR_BYTES,
+	NFACCT_ATTR_NOTIFY_PKTS,
+	NFACCT_ATTR_NOTIFY_P_RL, /* packets notification rate limit */
+	NFACCT_ATTR_NOTIFY_BYTES,
+	NFACCT_ATTR_NOTIFY_B_RL, /* bytes notification rate limit */
+	NFACCT_ATTR_NOTIFY_PERIOD,
 };
 
 struct nfacct *nfacct_alloc(void);
diff --git a/include/linux/netfilter/nfnetlink_acct.h b/include/linux/netfilter/nfnetlink_acct.h
index c7b6269..8125e57 100644
--- a/include/linux/netfilter/nfnetlink_acct.h
+++ b/include/linux/netfilter/nfnetlink_acct.h
@@ -10,6 +10,7 @@ enum nfnl_acct_msg_types {
 	NFNL_MSG_ACCT_GET,
 	NFNL_MSG_ACCT_GET_CTRZERO,
 	NFNL_MSG_ACCT_DEL,
+	NFNL_MSG_ACCT_NOTIFY,
 	NFNL_MSG_ACCT_MAX
 };
 
@@ -19,6 +20,11 @@ enum nfnl_acct_type {
 	NFACCT_PKTS,
 	NFACCT_BYTES,
 	NFACCT_USE,
+	NFACCT_NOTIFY_PKTS,
+	NFACCT_NOTIFY_P_RL,	/* packets notification rate limit */
+	NFACCT_NOTIFY_BYTES,
+	NFACCT_NOTIFY_B_RL,	/* bytes notification rate limit */
+	NFACCT_NOTIFY_PERIOD,
 	__NFACCT_MAX
 };
 #define NFACCT_MAX (__NFACCT_MAX - 1)
diff --git a/src/libnetfilter_acct.c b/src/libnetfilter_acct.c
index ba89e2d..4fb0d26 100644
--- a/src/libnetfilter_acct.c
+++ b/src/libnetfilter_acct.c
@@ -60,6 +60,11 @@ struct nfacct {
 	char		name[NFACCT_NAME_MAX];
 	uint64_t	pkts;
 	uint64_t	bytes;
+	uint32_t	notify_pkts;
+	uint32_t	notify_pkts_rl;
+	uint32_t	notify_bytes;
+	uint32_t	notify_bytes_rl;
+	uint32_t	notify_period;
 	uint32_t	bitset;
 };
 
@@ -114,6 +119,26 @@ nfacct_attr_set(struct nfacct *nfacct, enum nfacct_attr_type type,
 		nfacct->bytes = *((uint64_t *) data);
 		nfacct->bitset |= (1 << NFACCT_ATTR_BYTES);
 		break;
+	case NFACCT_ATTR_NOTIFY_PKTS:
+		nfacct->notify_pkts = *((uint32_t *) data);
+		nfacct->bitset |= (1 << NFACCT_ATTR_NOTIFY_PKTS);
+		break;
+	case NFACCT_ATTR_NOTIFY_P_RL:
+		nfacct->notify_pkts_rl = *((uint32_t *) data);
+		nfacct->bitset |= (1 << NFACCT_ATTR_NOTIFY_P_RL);
+		break;
+	case NFACCT_ATTR_NOTIFY_BYTES:
+		nfacct->notify_bytes = *((uint32_t *) data);
+		nfacct->bitset |= (1 << NFACCT_ATTR_NOTIFY_BYTES);
+		break;
+	case NFACCT_ATTR_NOTIFY_B_RL:
+		nfacct->notify_bytes_rl = *((uint32_t *) data);
+		nfacct->bitset |= (1 << NFACCT_ATTR_NOTIFY_B_RL);
+		break;
+	case NFACCT_ATTR_NOTIFY_PERIOD:
+		nfacct->notify_period = *((uint32_t *) data);
+		nfacct->bitset |= (1 << NFACCT_ATTR_NOTIFY_PERIOD);
+		break;
 	}
 }
 EXPORT_SYMBOL(nfacct_attr_set);
@@ -424,6 +449,26 @@ void nfacct_nlmsg_build_payload(struct nlmsghdr *nlh, struct nfacct *nfacct)
 
 	if (nfacct->bitset & (1 << NFACCT_ATTR_BYTES))
 		mnl_attr_put_u64(nlh, NFACCT_BYTES, htobe64(nfacct->bytes));
+
+	if (nfacct->bitset & (1 << NFACCT_ATTR_NOTIFY_PKTS))
+		mnl_attr_put_u32(nlh, NFACCT_NOTIFY_PKTS,
+				 htobe32(nfacct->notify_pkts));
+
+	if (nfacct->bitset & (1 << NFACCT_ATTR_NOTIFY_P_RL))
+		mnl_attr_put_u32(nlh, NFACCT_NOTIFY_P_RL,
+				 htobe32(nfacct->notify_pkts_rl));
+
+	if (nfacct->bitset & (1 << NFACCT_ATTR_NOTIFY_BYTES))
+		mnl_attr_put_u32(nlh, NFACCT_NOTIFY_BYTES,
+				 htobe32(nfacct->notify_bytes));
+
+	if (nfacct->bitset & (1 << NFACCT_ATTR_NOTIFY_B_RL))
+		mnl_attr_put_u32(nlh, NFACCT_NOTIFY_B_RL,
+				 htobe32(nfacct->notify_bytes_rl));
+
+	if (nfacct->bitset & (1 << NFACCT_ATTR_NOTIFY_PERIOD))
+		mnl_attr_put_u32(nlh, NFACCT_NOTIFY_PERIOD,
+				 htobe32(nfacct->notify_period));
 }
 EXPORT_SYMBOL(nfacct_nlmsg_build_payload);
 
-- 
1.7.10.4

Re: [PATCH 2/4] libnetfilter-acct: Introduce support for notifications

[Pablo Neira Ayuso]

On Wed, Sep 04, 2013 at 02:05:31PM +0200, valentina.giusti@bmw-carit.de wrote:
> From: Valentina Giusti <valentina.giusti@bmw-carit.de>
> 
> Introduce support for notifications. Intervals can be configured as bytes
> and packets or as time periods. When the intervals are configured in both
> ways, it's also possible to specify a limit for the amount of
> notifications based on accounted packets or bytes during a single period.
> 
> Signed-off-by: Valentina Giusti <valentina.giusti@bmw-carit.de>
> Cc: Pablo Neira Ayuso <pablo@netfilter.org>
> Cc: Patrick McHardy <kaber@trash.net>
> Cc: Jozsef Kadlecsik <kadlec@blackhole.kfki.hu>
> Cc: "David S. Miller" <davem@davemloft.net>
> ---
>  include/libnetfilter_acct/libnetfilter_acct.h |    5 +++
>  include/linux/netfilter/nfnetlink_acct.h      |    6 ++++
>  src/libnetfilter_acct.c                       |   45 +++++++++++++++++++++++++
>  3 files changed, 56 insertions(+)
> 
> diff --git a/include/libnetfilter_acct/libnetfilter_acct.h b/include/libnetfilter_acct/libnetfilter_acct.h
> index b00e366..b0a5bf6 100644
> --- a/include/libnetfilter_acct/libnetfilter_acct.h
> +++ b/include/libnetfilter_acct/libnetfilter_acct.h
> @@ -14,6 +14,11 @@ enum nfacct_attr_type {
>  	NFACCT_ATTR_NAME = 0,
>  	NFACCT_ATTR_PKTS,
>  	NFACCT_ATTR_BYTES,
> +	NFACCT_ATTR_NOTIFY_PKTS,
> +	NFACCT_ATTR_NOTIFY_P_RL, /* packets notification rate limit */
> +	NFACCT_ATTR_NOTIFY_BYTES,
> +	NFACCT_ATTR_NOTIFY_B_RL, /* bytes notification rate limit */
> +	NFACCT_ATTR_NOTIFY_PERIOD,
>  };
>  
>  struct nfacct *nfacct_alloc(void);
> diff --git a/include/linux/netfilter/nfnetlink_acct.h b/include/linux/netfilter/nfnetlink_acct.h
> index c7b6269..8125e57 100644
> --- a/include/linux/netfilter/nfnetlink_acct.h
> +++ b/include/linux/netfilter/nfnetlink_acct.h
> @@ -10,6 +10,7 @@ enum nfnl_acct_msg_types {
>  	NFNL_MSG_ACCT_GET,
>  	NFNL_MSG_ACCT_GET_CTRZERO,
>  	NFNL_MSG_ACCT_DEL,
> +	NFNL_MSG_ACCT_NOTIFY,
>  	NFNL_MSG_ACCT_MAX
>  };
>  
> @@ -19,6 +20,11 @@ enum nfnl_acct_type {
>  	NFACCT_PKTS,
>  	NFACCT_BYTES,
>  	NFACCT_USE,
> +	NFACCT_NOTIFY_PKTS,
> +	NFACCT_NOTIFY_P_RL,	/* packets notification rate limit */
> +	NFACCT_NOTIFY_BYTES,
> +	NFACCT_NOTIFY_B_RL,	/* bytes notification rate limit */
> +	NFACCT_NOTIFY_PERIOD,
>  	__NFACCT_MAX
>  };
>  #define NFACCT_MAX (__NFACCT_MAX - 1)
> diff --git a/src/libnetfilter_acct.c b/src/libnetfilter_acct.c
> index ba89e2d..4fb0d26 100644
> --- a/src/libnetfilter_acct.c
> +++ b/src/libnetfilter_acct.c
> @@ -60,6 +60,11 @@ struct nfacct {
>  	char		name[NFACCT_NAME_MAX];
>  	uint64_t	pkts;
>  	uint64_t	bytes;
> +	uint32_t	notify_pkts;
> +	uint32_t	notify_pkts_rl;
> +	uint32_t	notify_bytes;
> +	uint32_t	notify_bytes_rl;
> +	uint32_t	notify_period;

This is bloating the nfacct object for everyone in the earth not using
this notification infrastructure.

Re: [PATCH 2/4] libnetfilter-acct: Introduce support for notifications

[Valentina Giusti]

Hi Pablo,

On 09/04/2013 02:44 PM, Pablo Neira Ayuso wrote:
> On Wed, Sep 04, 2013 at 02:05:31PM +0200,valentina.giusti@bmw-carit.de  wrote:

<snip>

>>   #define NFACCT_MAX (__NFACCT_MAX - 1)
>> diff --git a/src/libnetfilter_acct.c b/src/libnetfilter_acct.c
>> index ba89e2d..4fb0d26 100644
>> --- a/src/libnetfilter_acct.c
>> +++ b/src/libnetfilter_acct.c
>> @@ -60,6 +60,11 @@ struct nfacct {
>>   	char		name[NFACCT_NAME_MAX];
>>   	uint64_t	pkts;
>>   	uint64_t	bytes;
>> +	uint32_t	notify_pkts;
>> +	uint32_t	notify_pkts_rl;
>> +	uint32_t	notify_bytes;
>> +	uint32_t	notify_bytes_rl;
>> +	uint32_t	notify_period;
> This is bloating the nfacct object for everyone in the earth not using
> this notification infrastructure.

Right... I'll replace this with a pointer to another structure. Do you 
suggest that I also #ifdef everything?

Thank you.

-V.

[PATCH 3/4] nfacct: Add 'notify' commands for notifications

[valentina.giusti]

From: Valentina Giusti <valentina.giusti@bmw-carit.de>

The notify command allows to receive notifications based on intervals
expressed as bytes and packets or as time periods.

Examples of usage:
nfacct notify http-traffic bytes 1024	# notification every 1024 bytes
nfacct notify http-traffic packets 100	# notification every 100 packets
nfacct notify http-traffic period 120	# notification every 2 minutes

When the notification request specifies both a traffic target (bytes or
packets) and a time target (period), additional parameters are available,
which allow to specify a limit for the amount of notifications based on
accounted packets or bytes to be sent during a single period:

nfacct notify http-traffic bytes 1024 bytes-ratelimit 10 period 5 # sends a
notification at least every 5s and at most 10 more if 1024 bytes are
accounted

Signed-off-by: Valentina Giusti <valentina.giusti@bmw-carit.de>
Cc: Pablo Neira Ayuso <pablo@netfilter.org>
Cc: Patrick McHardy <kaber@trash.net>
Cc: Jozsef Kadlecsik <kadlec@blackhole.kfki.hu>
Cc: "David S. Miller" <davem@davemloft.net>
---
 include/linux/netfilter/nfnetlink_acct.h |    1 +
 src/nfacct.c                             |  139 ++++++++++++++++++++++++++++++
 2 files changed, 140 insertions(+)

diff --git a/include/linux/netfilter/nfnetlink_acct.h b/include/linux/netfilter/nfnetlink_acct.h
index 7c4279b..28c01db 100644
--- a/include/linux/netfilter/nfnetlink_acct.h
+++ b/include/linux/netfilter/nfnetlink_acct.h
@@ -10,6 +10,7 @@ enum nfnl_acct_msg_types {
 	NFNL_MSG_ACCT_GET,
 	NFNL_MSG_ACCT_GET_CTRZERO,
 	NFNL_MSG_ACCT_DEL,
+	NFNL_MSG_ACCT_NOTIFY,
 	NFNL_MSG_ACCT_MAX
 };
 
diff --git a/src/nfacct.c b/src/nfacct.c
index 2ef93c3..eab06ea 100644
--- a/src/nfacct.c
+++ b/src/nfacct.c
@@ -29,6 +29,7 @@ enum {
 	NFACCT_CMD_ADD,
 	NFACCT_CMD_DELETE,
 	NFACCT_CMD_GET,
+	NFACCT_CMD_NOTIFY,
 	NFACCT_CMD_FLUSH,
 	NFACCT_CMD_VERSION,
 	NFACCT_CMD_HELP,
@@ -39,6 +40,7 @@ static int nfacct_cmd_list(int argc, char *argv[]);
 static int nfacct_cmd_add(int argc, char *argv[]);
 static int nfacct_cmd_delete(int argc, char *argv[]);
 static int nfacct_cmd_get(int argc, char *argv[]);
+static int nfacct_cmd_notify(int argc, char *argv[]);
 static int nfacct_cmd_flush(int argc, char *argv[]);
 static int nfacct_cmd_version(int argc, char *argv[]);
 static int nfacct_cmd_help(int argc, char *argv[]);
@@ -76,6 +78,8 @@ int main(int argc, char *argv[])
 		cmd = NFACCT_CMD_DELETE;
 	else if (strncmp(argv[1], "get", strlen(argv[1])) == 0)
 		cmd = NFACCT_CMD_GET;
+	else if (strncmp(argv[1], "notify", strlen(argv[1])) == 0)
+		cmd = NFACCT_CMD_NOTIFY;
 	else if (strncmp(argv[1], "flush", strlen(argv[1])) == 0)
 		cmd = NFACCT_CMD_FLUSH;
 	else if (strncmp(argv[1], "version", strlen(argv[1])) == 0)
@@ -104,6 +108,9 @@ int main(int argc, char *argv[])
 	case NFACCT_CMD_GET:
 		ret = nfacct_cmd_get(argc, argv);
 		break;
+	case NFACCT_CMD_NOTIFY:
+		ret = nfacct_cmd_notify(argc, argv);
+		break;
 	case NFACCT_CMD_FLUSH:
 		ret = nfacct_cmd_flush(argc, argv);
 		break;
@@ -438,6 +445,138 @@ static int nfacct_cmd_get(int argc, char *argv[])
 	return 0;
 }
 
+static int nfacct_cmd_notify(int argc, char *argv[])
+{
+	bool xml = false;
+	struct mnl_socket *nl;
+	char buf[MNL_SOCKET_BUFFER_SIZE];
+	struct nlmsghdr *nlh;
+	uint32_t portid, seq;
+	struct nfacct *nfacct;
+	int ret, i;
+	uint32_t *bytes = NULL, *bytes_rl = NULL;
+	uint32_t *pkts = NULL, *pkts_rl = NULL;
+	uint32_t *period = NULL;
+
+	if (argc < 5) {
+		nfacct_perror("missing object name, bytes, packets or period");
+		return -1;
+	}
+
+	nfacct = nfacct_alloc();
+	if (nfacct == NULL) {
+		nfacct_perror("OOM");
+		return -1;
+	}
+	nfacct_attr_set(nfacct, NFACCT_ATTR_NAME, argv[2]);
+
+	for (i = 3; i < argc; i += 2) {
+		if (strncmp(argv[i], "xml", strlen(argv[i])) == 0) {
+			xml = true;
+		} else if (strncmp(argv[i], "bytes", strlen(argv[i])) == 0) {
+			bytes = malloc(sizeof(uint32_t));
+			if (!bytes)
+				return -ENOMEM;
+			ret = sscanf(argv[i + 1], "%u", bytes);
+			if (ret < 1)
+				goto input_error;
+			nfacct_attr_set(nfacct, NFACCT_ATTR_NOTIFY_BYTES,
+					bytes);
+		} else if (strncmp(argv[i], "bytes-ratelimit",
+			   strlen(argv[i])) == 0) {
+			if (!bytes || !*bytes)
+				goto input_error;
+			bytes_rl = malloc(sizeof(uint32_t));
+			if (!bytes_rl)
+				return -ENOMEM;
+			ret = sscanf(argv[i + 1], "%u", bytes_rl);
+			if (ret < 1 || !*bytes_rl)
+				goto input_error;
+			nfacct_attr_set(nfacct, NFACCT_ATTR_NOTIFY_B_RL,
+					bytes_rl);
+		} else if (strncmp(argv[i], "packets", strlen(argv[i])) == 0) {
+			pkts = malloc(sizeof(uint32_t));
+			if (!pkts)
+				return -ENOMEM;
+			ret = sscanf(argv[i + 1], "%u", pkts);
+			if (ret < 1)
+				goto input_error;
+			nfacct_attr_set(nfacct, NFACCT_ATTR_NOTIFY_PKTS, pkts);
+		} else if (strncmp(argv[i], "packets-ratelimit",
+			   strlen(argv[i])) == 0) {
+			if (!pkts || !*pkts)
+				goto input_error;
+			pkts_rl = malloc(sizeof(uint32_t));
+			if (!pkts_rl)
+				return -ENOMEM;
+			ret = sscanf(argv[i + 1], "%u", pkts_rl);
+			if (ret < 1 || !*pkts_rl)
+				goto input_error;
+			nfacct_attr_set(nfacct, NFACCT_ATTR_NOTIFY_P_RL,
+					pkts_rl);
+		} else if (strncmp(argv[i], "period", strlen(argv[i])) == 0) {
+			period = malloc(sizeof(uint32_t));
+			if (!period)
+				return -ENOMEM;
+			ret = sscanf(argv[i + 1], "%u", period);
+			if (ret < 1)
+				goto input_error;
+			nfacct_attr_set(nfacct, NFACCT_ATTR_NOTIFY_PERIOD,
+					period);
+		} else {
+			nfacct_perror("unknown argument");
+			return -1;
+		}
+	}
+
+	seq = time(NULL);
+	nlh = nfacct_nlmsg_build_hdr(buf, NFNL_MSG_ACCT_NOTIFY,
+				     NLM_F_ACK, seq);
+
+	nfacct_nlmsg_build_payload(nlh, nfacct);
+	nfacct_free(nfacct);
+
+	nl = mnl_socket_open(NETLINK_NETFILTER);
+	if (nl == NULL) {
+		nfacct_perror("mnl_socket_open");
+		return -1;
+	}
+
+	if (mnl_socket_bind(nl, 0, MNL_SOCKET_AUTOPID) < 0) {
+		nfacct_perror("mnl_socket_bind");
+		return -1;
+	}
+	portid = mnl_socket_get_portid(nl);
+
+	if (mnl_socket_sendto(nl, nlh, nlh->nlmsg_len) < 0) {
+		nfacct_perror("mnl_socket_send");
+		return -1;
+	}
+
+	ret = mnl_socket_recvfrom(nl, buf, sizeof(buf));
+	while (ret > 0) {
+		ret = mnl_cb_run(buf, ret, seq, portid, nfacct_cb, &xml);
+		if (ret <= 0 && (errno == EINVAL || errno == ENOENT))
+			break;
+		ret = mnl_socket_recvfrom(nl, buf, sizeof(buf));
+	}
+	if (ret == -1) {
+		nfacct_perror("error");
+		return -1;
+	}
+
+	mnl_socket_close(nl);
+
+	if (xml_header)
+		printf("</nfacct>\n");
+
+	return 0;
+
+input_error:
+	nfacct_perror("input error");
+	return -1;
+}
+
 static int nfacct_cmd_flush(int argc, char *argv[])
 {
 	struct mnl_socket *nl;
-- 
1.7.10.4

[PATCH 4/4] nfacct: Add man page section for the 'notify' commands

[valentina.giusti]

From: Valentina Giusti <valentina.giusti@bmw-carit.de>

Signed-off-by: Valentina Giusti <valentina.giusti@bmw-carit.de>
Cc: Pablo Neira Ayuso <pablo@netfilter.org>
Cc: Patrick McHardy <kaber@trash.net>
Cc: Jozsef Kadlecsik <kadlec@blackhole.kfki.hu>
Cc: "David S. Miller" <davem@davemloft.net>
---
 nfacct.8 |   59 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 59 insertions(+)

diff --git a/nfacct.8 b/nfacct.8
index 0c3249c..8cc79b7 100644
--- a/nfacct.8
+++ b/nfacct.8
@@ -30,6 +30,65 @@ Delete accounting object.
 .BI "get "
 Get existing accounting object.
 .TP
+.BI "notify "
+Get notifications for existing accounting object.
+Notifications can be sent either periodically, with the option "period", or
+based on the amount of accounted bytes or packets, with the options "bytes"
+and "packets", respectively.
+
+For example, a notification every 1024 bytes can be requested for the accounting
+object "http-traffic":
+.in +4n
+.nf
+
+.RB "$" "nfacct notify http-traffic bytes 1024"
+{ pkts = 00000000000000039225, bytes = 00000000000002361568 } = http-traffic;
+{ pkts = 00000000000000039245, bytes = 00000000000002362608 } = http-traffic;
+ ...
+.fi
+.in
+
+or every 100 packets:
+
+.in +4n
+.nf
+.RB "$" "nfacct notify http-traffic packets 100"
+{ pkts = 00000000000000008303, bytes = 00000000000000486469 } = http-traffic;
+{ pkts = 00000000000000008403, bytes = 00000000000000494581 } = http-traffic;
+ ...
+.fi
+.in
+
+or every 2 minutes:
+
+.in +4n
+.nf
+.RB "$" "nfacct notify http-traffic period 120"
+{ pkts = 00000000000000013133, bytes = 00000000000000796388 } = http-traffic;
+{ pkts = 00000000000000015459, bytes = 00000000000000952884 } = http-traffic;
+ ...
+.fi
+.in
+
+Notification requests can specify at the same time a traffic target (bytes or
+packets) and a time target (period). In this case, additional parameters are
+available, which allow to set a limit for the amount of notifications based on
+accounted packets or bytes to be sent during a single period:
+
+.in +4n
+.nf
+.RB "$" "nfacct notify http-traffic bytes 1024 bytes-ratelimit 4 period 5"
+{ pkts = 00000000000000010172, bytes = 00000000000000628714 } = http-traffic;
+{ pkts = 00000000000000010192, bytes = 00000000000000629754 } = http-traffic;
+{ pkts = 00000000000000010212, bytes = 00000000000000630794 } = http-traffic;
+{ pkts = 00000000000000010232, bytes = 00000000000000631834 } = http-traffic;
+{ pkts = 00000000000000010894, bytes = 00000000000000672343 } = http-traffic;
+ ...
+.fi
+.in
+
+.PP
+.TP
 .BI "flush "
 Delete all unused accounting objects.
 .TP
-- 
1.7.10.4

Re: [PATCH 0/4] [RFC] nfnetlink_acct: Traffic-based and periodic notifications

[Pablo Neira Ayuso]

On Wed, Sep 04, 2013 at 02:05:29PM +0200, valentina.giusti@bmw-carit.de wrote:
> From: Valentina Giusti <valentina.giusti@bmw-carit.de>
> 
> This patchset introduces notifications of accounting objects to userspace for
> nfacct.
> Notifications can be sent periodically or based on the amount of accounted bytes
> or packets. In case a notification is set by giving a period as well as a given
> amount of traffic, the number of notifications triggered during a period can be
> rate limited.
> 
> The need for notifications stems from monitoring applications (e.g. connman)
> which currently need to keep polling accounting objects via nfnl_acct_get() in 
> order to get updated statistics. This is far from ideal in scenarios with large
> amounts of accounting objects and diverse, unpredictable network traffic.
> 
> This implementation introduces a new message type, NFNL_MSG_ACCT_NOTIFY, which
> is sent by userspace (typically, libnetfilter_acct) in order to configure
> accounting notifications.
> The new message parameters are NFACCT_NOTIFY_PACKETS, NFACCT_NOTIFY_BYTES and
> NFACCT_NOTIFY_PERIOD, which are sent with the accounting object name. Userspace
> applications would typically configure desired notification modes and then get 
> appropriate notifications via netlink, on the same socket used for said
> configuration.
> 
> The patches needed for libnetfilter_acct and nfacct are included in this series.
> 
> An example of nfacct usage is:
> 
> $ nfacct add http-traffic # Create the accounting object
> 
> [create related iptables rules here]
> 
> $ nfacct notify http-traffic bytes 1024
> [for every 1024 bytes of HTTP traffic:]
> { pkts = 00000000000000039225, bytes = 00000000000002361568 } = http-traffic;
> { pkts = 00000000000000039245, bytes = 00000000000002362608 } = http-traffic;
> ...
> 
> or:
> 
> $ nfacct notify-traffic http packets 100
> [for every 100 packets of HTTP traffic:]
> { pkts = 00000000000000008303, bytes = 00000000000000486469 } = http-traffic;
> { pkts = 00000000000000008403, bytes = 00000000000000494581 } = http-traffic;
> ...
> 
> or:
> 
> $ nfacct notify http-traffic interval 120
> [every two minutes:]
> { pkts = 00000000000000008303, bytes = 00000000000000486469 } = http-traffic;
> { pkts = 00000000000000008403, bytes = 00000000000000494581 } = http-traffic;
> ...
> 
> or:
> 
> $ nfacct notify http-traffic bytes 1024 bytes-ratelimit 4 period 5
> { pkts = 00000000000000010172, bytes = 00000000000000628714 } = http-traffic;
> { pkts = 00000000000000010192, bytes = 00000000000000629754 } = http-traffic;
> { pkts = 00000000000000010212, bytes = 00000000000000630794 } = http-traffic;
> { pkts = 00000000000000010232, bytes = 00000000000000631834 } = http-traffic;
> { pkts = 00000000000000010894, bytes = 00000000000000672343 } = http-traffic;
> ...

You can implement this from userspace by polling to obtain the
counters.

Re: [PATCH 0/4] [RFC] nfnetlink_acct: Traffic-based and periodic notifications

[Valentina Giusti]

Hi Pablo,

On 09/04/2013 02:39 PM, Pablo Neira Ayuso wrote:
> On Wed, Sep 04, 2013 at  02:05:29PM +0200,
 > valentina.giusti@bmw-carit.de wrote:
 >> From: Valentina Giusti <valentina.giusti@bmw-carit.de>
 >>
 >> This patchset introduces notifications of accounting objects to
 >> userspace for nfacct. Notifications can be sent periodically or
 >> based on the amount of accounted bytes or packets. In case a
 >> notification is set by giving a period as well as a given amount of
 >> traffic, the number of notifications triggered during a period can
 >> be rate limited.
 >>
 >> The need for notifications stems from monitoring applications (e.g.
 >> connman) which currently need to keep polling accounting objects
 >> via nfnl_acct_get() in order to get updated statistics. This is far
 >> from ideal in scenarios with large amounts of accounting objects
 >> and diverse, unpredictable network traffic.
 >>
 >> This implementation introduces a new message type,
 >> NFNL_MSG_ACCT_NOTIFY, which is sent by userspace (typically,
 >> libnetfilter_acct) in order to configure accounting notifications.
 >> The new message parameters are NFACCT_NOTIFY_PACKETS,
 >> NFACCT_NOTIFY_BYTES and NFACCT_NOTIFY_PERIOD, which are sent with
 >> the accounting object name. Userspace applications would typically
 >> configure desired notification modes and then get appropriate
 >> notifications via netlink, on the same socket used for said
 >> configuration.
 >>
 >> The patches needed for libnetfilter_acct and nfacct are included in
 >> this series.
 >>
 >> An example of nfacct usage is:
 >>
 >> $ nfacct add http-traffic # Create the accounting object
 >>
 >> [create related iptables rules here]
 >>
 >> $ nfacct notify http-traffic bytes 1024 [for every 1024 bytes of
 >> HTTP traffic:] { pkts = 00000000000000039225, bytes =
 >> 00000000000002361568 } = http-traffic; { pkts =
 >> 00000000000000039245, bytes = 00000000000002362608 } =
 >> http-traffic; ...
 >>
 >> or:
 >>
 >> $ nfacct notify-traffic http packets 100 [for every 100 packets of
 >> HTTP traffic:] { pkts = 00000000000000008303, bytes =
 >> 00000000000000486469 } = http-traffic; { pkts =
 >> 00000000000000008403, bytes = 00000000000000494581 } =
 >> http-traffic; ...
 >>
 >> or:
 >>
 >> $ nfacct notify http-traffic interval 120 [every two minutes:] {
 >> pkts = 00000000000000008303, bytes = 00000000000000486469 } =
 >> http-traffic; { pkts = 00000000000000008403, bytes =
 >> 00000000000000494581 } = http-traffic; ...
 >>
 >> or:
 >>
 >> $ nfacct notify http-traffic bytes 1024 bytes-ratelimit 4 period 5
 >> { pkts = 00000000000000010172, bytes = 00000000000000628714 } =
 >> http-traffic; { pkts = 00000000000000010192, bytes =
 >> 00000000000000629754 } = http-traffic; { pkts =
 >> 00000000000000010212, bytes = 00000000000000630794 } =
 >> http-traffic; { pkts = 00000000000000010232, bytes =
 >> 00000000000000631834 } = http-traffic; { pkts =
 >> 00000000000000010894, bytes = 00000000000000672343 } =
 >> http-traffic; ...
 > You can implement this from userspace by polling to obtain the
 > counters.

As I mentioned, there are

"applications (e.g. connman) which currently need to keep polling 
accounting objects via nfnl_acct_get() in order to get updated 
statistics. This is far from ideal in scenarios with large amounts of 
accounting objects and diverse, unpredictable network traffic."

- V.

Re: [PATCH 0/4] [RFC] nfnetlink_acct: Traffic-based and periodic notifications

[Pablo Neira Ayuso]

On Thu, Sep 05, 2013 at 09:58:52AM +0200, Valentina Giusti wrote:
[...]
> As I mentioned, there are
> 
> "applications (e.g. connman) which currently need to keep polling
> accounting objects via nfnl_acct_get() in order to get updated
> statistics. This is far from ideal in scenarios with large amounts
> of accounting objects and diverse, unpredictable network traffic."

You get around 100 accounting objects with one single recv syscall on
x86_64 when polling from userspace. If you're noticing performance
issues with this approach, please report them more precisely.

Regards.

Re: [PATCH 0/4] [RFC] nfnetlink_acct: Traffic-based and periodic notifications

[Valentina Giusti]

Hi,

On 09/05/2013 10:27 AM, Pablo Neira Ayuso wrote:
> On Thu, Sep 05, 2013 at 09:58:52AM +0200, Valentina Giusti wrote:
> [...]
>> As I mentioned, there are
>>
>> "applications (e.g. connman) which currently need to keep polling
>> accounting objects via nfnl_acct_get() in order to get updated
>> statistics. This is far from ideal in scenarios with large amounts
>> of accounting objects and diverse, unpredictable network traffic."
> You get around 100 accounting objects with one single recv syscall on
> x86_64 when polling from userspace. If you're noticing performance
> issues with this approach, please report them more precisely.

It's actually rather a conceptual matter here.
Requesting notifications from userspace via nfnl_acct_get() implies that 
for each and every one of them, userspace has to make a new query. Is 
polling for stats  better than receiving notifications from the kernel?

Thanks,
- V.

Re: [PATCH 0/4] [RFC] nfnetlink_acct: Traffic-based and periodic notifications

[Pablo Neira Ayuso]

On Thu, Sep 05, 2013 at 04:58:40PM +0200, Valentina Giusti wrote:
> Hi,
> 
> On 09/05/2013 10:27 AM, Pablo Neira Ayuso wrote:
> >On Thu, Sep 05, 2013 at 09:58:52AM +0200, Valentina Giusti wrote:
> >[...]
> >>As I mentioned, there are
> >>
> >>"applications (e.g. connman) which currently need to keep polling
> >>accounting objects via nfnl_acct_get() in order to get updated
> >>statistics. This is far from ideal in scenarios with large amounts
> >>of accounting objects and diverse, unpredictable network traffic."
> >
> >You get around 100 accounting objects with one single recv syscall on
> >x86_64 when polling from userspace. If you're noticing performance
> >issues with this approach, please report them more precisely.
> 
> It's actually rather a conceptual matter here.
> Requesting notifications from userspace via nfnl_acct_get() implies
> that for each and every one of them, userspace has to make a new
> query. Is polling for stats  better than receiving notifications
> from the kernel?

The question is that you're proposing to move the complexity from user
to kernel-space to resolve this problem, which results in more kernel
LOC for a very specific task. I think we can make it in a more generic
way.

My suggestion is to extend the iptables nfacct match to match if the
bytes/packets have passed some user-define threshold, then use the
NFLOG target to report such situation.

The hypothetical iptables command line would look like:

-m nfacct --nfacct-name test --above-bytes 1000 --match-once \
        -j NFLOG --nflog-prefix "test"

You can add a new nfacct match revision to implement this new feature.

Regards.