[ULOGD RFC 00/30] [ULOGD RFC] ulogd V2 improvements, round #1

[ULOGD RFC 00/30] [ULOGD RFC] ulogd V2 improvements, round #1

[heitzenberger]

Hi,

this is a patchset for you to comment on, which tries to turn ulogd V2
into something usefull.

Some of the patches are are quite intrusive, others might be to be
discussed.  Please take a look at all of them and give some comments
back, I'll appreciate it.

In the end they turn ulogd v2 into some powerfull network accounting
solution being capable to handle 400000 or more concurrent flows.
That's where your post-processing starts to get important of course
:).

Some points which might be of interest to you:

 * ulogd-ifi.diff: replacement for the IFINDEX plugin, as that
 functionality IMO is better homed in the ulogd core and not in a
 plugin (remove IFINDEX?).

 * some of the patches are quite large and contain many individual 
 changes in the end.  Should I split them?

 * ulogd-NFCT-new-lib.diff: makes ulogd work with a recent
 libnetfilter-conntrack.  This one is fairly untested, I'll test that
 before sending the final patchset.  Note also that I'd like to use
 libnl with ulogd rather soon, so this is only a short-term solution.

 * the libnetfilter-conntrack patch I've send before is required
 to build ulogd.

Some of those patches still contain trailing whitespace.  Please
ignore that as of now, I'll send additional patches removing
whitespace later.

Thanks in advance for your feedback.

 /holger

[ULOGD RFC 01/30] Add NACCT output plugin

[heitzenberger]

[-- Attachment #1: ulogd-NACCT-plugin.diff --]
[-- Type: text/plain, Size: 6383 bytes --]

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/output/Makefile.am
===================================================================
--- ulogd-netfilter.orig/output/Makefile.am
+++ ulogd-netfilter/output/Makefile.am
@@ -4,7 +4,8 @@ LIBS=""
 SUBDIRS= pcap mysql pgsql sqlite3
 
 pkglib_LTLIBRARIES = ulogd_output_LOGEMU.la ulogd_output_SYSLOG.la \
-		     ulogd_output_OPRINT.la ulogd_output_IPFIX.la
+		     ulogd_output_OPRINT.la ulogd_output_IPFIX.la \
+			 ulogd_output_NACCT.la
 
 ulogd_output_LOGEMU_la_SOURCES = ulogd_output_LOGEMU.c
 ulogd_output_LOGEMU_la_LDFLAGS = -module
@@ -18,3 +19,5 @@ ulogd_output_OPRINT_la_LDFLAGS = -module
 ulogd_output_IPFIX_la_SOURCES = ulogd_output_IPFIX.c
 ulogd_output_IPFIX_la_LDFLAGS = -module
 
+ulogd_output_NACCT_la_SOURCES = ulogd_output_NACCT.c
+ulogd_output_NACCT_la_LDFLAGS = -module
Index: ulogd-netfilter/output/ulogd_output_NACCT.c
===================================================================
--- /dev/null
+++ ulogd-netfilter/output/ulogd_output_NACCT.c
@@ -0,0 +1,206 @@
+/*
+ * ulogd_outpout_NACCT.c
+ *
+ * ulogd output plugin for accounting which tries to stay mostly
+ * compatible with nacct output.
+ *
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 
+ * as published by the Free Software Foundation
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+ *
+ * Holger Eitzenberger <holger@eitzenberger.org>  Astaro AG 2008
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <arpa/inet.h>
+#include <ulogd/ulogd.h>
+#include <ulogd/conffile.h>
+
+#define NACCT_FILE_DEFAULT	"/var/log/nacctdata.log"
+
+#define HIPQUAD(addr) \
+        ((unsigned char *)&addr)[3], \
+        ((unsigned char *)&addr)[2], \
+        ((unsigned char *)&addr)[1], \
+        ((unsigned char *)&addr)[0]
+
+/* config accessors (lazy me...) */
+#define NACCT_CFG_FILE(pi)	((pi)->config_kset->ces[0].u.string)
+#define NACCT_CFG_SYNC(pi)	((pi)->config_kset->ces[1].u.value)
+
+#define KEY(pi,idx)		((pi)->input.keys[(idx)].u.source)
+
+/* input keys */
+#define KEY_IP_SADDR(pi)		KEY(pi, 0)
+#define KEY_IP_DADDR(pi)		KEY(pi, 1)
+#define KEY_IP_PROTO(pi)		KEY(pi, 2)
+#define KEY_L4_SPORT(pi)		KEY(pi, 3)
+#define KEY_L4_DPORT(pi)		KEY(pi, 4)
+#define KEY_RAW_PKTLEN(pi)		KEY(pi, 5)
+#define KEY_RAW_PKTCNT(pi)		KEY(pi, 6)
+#define KEY_ICMP_CODE(pi)		KEY(pi, 7)
+#define KEY_ICMP_TYPE(pi)		KEY(pi, 8)
+#define KEY_FLOW_START(pi)		KEY(pi, 11)
+#define KEY_FLOW_END(pi)		KEY(pi, 13)
+
+struct nacct_priv {
+	FILE *of;
+};
+
+
+static int
+nacct_interp(struct ulogd_pluginstance *pi)
+{
+	struct nacct_priv *priv = (struct nacct_priv *)&pi->private;
+	static char buf[80];
+
+	/* try to be as close to nacct as possible.  Instead of nacct's
+	   'timestamp' value use 'flow.end.sec' */
+	if (KEY_IP_PROTO(pi)->u.value.ui8 == IPPROTO_ICMP) {
+		snprintf(buf, sizeof(buf),
+				 "%u\t%u\t%u.%u.%u.%u\t%u\t%u.%u.%u.%u\t%u\t%u\t%u",
+				 KEY_FLOW_END(pi)->u.value.ui32,
+				 KEY_IP_PROTO(pi)->u.value.ui8,
+				 HIPQUAD(KEY_IP_SADDR(pi)->u.value.ui32),
+				 KEY_ICMP_TYPE(pi)->u.value.ui8,
+				 HIPQUAD(KEY_IP_DADDR(pi)->u.value.ui32),
+				 KEY_ICMP_CODE(pi)->u.value.ui8,
+				 KEY_RAW_PKTCNT(pi)->u.value.ui32,
+				 KEY_RAW_PKTLEN(pi)->u.value.ui32);
+	} else {
+		snprintf(buf, sizeof(buf),
+				 "%u\t%u\t%u.%u.%u.%u\t%u\t%u.%u.%u.%u\t%u\t%u\t%u",
+				 KEY_FLOW_END(pi)->u.value.ui32,
+				 KEY_IP_PROTO(pi)->u.value.ui8,
+				 HIPQUAD(KEY_IP_SADDR(pi)->u.value.ui32),
+				 KEY_L4_SPORT(pi)->u.value.ui8,
+				 HIPQUAD(KEY_IP_DADDR(pi)->u.value.ui32),
+				 KEY_L4_DPORT(pi)->u.value.ui8,
+				 KEY_RAW_PKTCNT(pi)->u.value.ui32,
+				 KEY_RAW_PKTLEN(pi)->u.value.ui32);
+	}
+
+	fprintf(priv->of, "%s\n", buf);
+
+	if (NACCT_CFG_SYNC(pi) != 0)
+		fflush(priv->of);
+
+	return 0;
+}
+
+static struct config_keyset nacct_kset = {
+	.num_ces = 2,
+	.ces = {
+		{
+			.key = "file", 
+			.type = CONFIG_TYPE_STRING, 
+			.options = CONFIG_OPT_NONE,
+			.u = {.string = NACCT_FILE_DEFAULT },
+		},
+		{
+			.key = "sync",
+			.type = CONFIG_TYPE_INT,
+			.options = CONFIG_OPT_NONE,
+			.u = { .value = 0 },
+		},
+	},
+};
+
+static void
+sighup_handler_print(struct ulogd_pluginstance *pi, int signal)
+{
+	struct nacct_priv *oi = (struct nacct_priv *)&pi->private;
+
+	switch (signal) {
+	case SIGHUP:
+	{
+		ulogd_log(ULOGD_NOTICE, "NACCT: reopening logfile\n");
+		fclose(oi->of);
+		oi->of = fopen(NACCT_CFG_FILE(pi), "a");
+		if (!oi->of)
+			ulogd_log(ULOGD_ERROR, "%s: %s\n", NACCT_CFG_FILE(pi),
+					  strerror(errno));
+		break;
+	}
+
+	default:
+		break;
+	}
+}
+
+static int
+nacct_conf(struct ulogd_pluginstance *pi,
+		   struct ulogd_pluginstance_stack *stack)
+{
+	int ret;
+
+	if ((ret = ulogd_wildcard_inputkeys(pi)) < 0)
+		return ret;
+
+	if ((ret = config_parse_file(pi->id, pi->config_kset)) < 0)
+		return ret;
+
+	return 0;
+}
+
+static int
+nacct_init(struct ulogd_pluginstance *pi)
+{
+	struct nacct_priv *op = (struct nacct_priv *)&pi->private;
+
+	if ((op->of = fopen(NACCT_CFG_FILE(pi), "a")) == NULL) {
+		ulogd_log(ULOGD_FATAL, "%s: %s\n", 
+				  NACCT_CFG_FILE(pi), strerror(errno));
+		return -1;
+	}		
+	return 0;
+}
+
+static int
+nacct_fini(struct ulogd_pluginstance *pi)
+{
+	struct nacct_priv *op = (struct nacct_priv *)&pi->private;
+
+	if (op->of != stdout)
+		fclose(op->of);
+
+	return 0;
+}
+
+static struct ulogd_plugin nacct_plugin = {
+	.name = "NACCT", 
+	.input = {
+		.type = ULOGD_DTYPE_PACKET | ULOGD_DTYPE_FLOW,
+	},
+	.output = {
+		.type = ULOGD_DTYPE_SINK,
+	},
+	.configure = &nacct_conf,
+	.interp	= &nacct_interp,
+	.start 	= &nacct_init,
+	.stop	= &nacct_fini,
+	.signal = &sighup_handler_print,
+	.config_kset = &nacct_kset,
+	.version = ULOGD_VERSION,
+};
+
+void __attribute__ ((constructor)) init(void);
+
+void
+init(void)
+{
+	ulogd_register_plugin(&nacct_plugin);
+}

-- 

[ULOGD RFC 02/30] common.h: added

[heitzenberger]

[-- Attachment #1: ulogd-add-common-code.diff --]
[-- Type: text/plain, Size: 1770 bytes --]

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/include/ulogd/common.h
===================================================================
--- /dev/null
+++ ulogd-netfilter/include/ulogd/common.h
@@ -0,0 +1,50 @@
+/*
+ * common.h
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2
+ * as published by the Free Software Foundation
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+ *
+ * Holger Eitzenberger <holger@eitzenberger.org>  Astaro AG 2008
+ */
+#ifndef COMMON_H
+#define COMMON_H
+
+#include <stdlib.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <string.h>
+#include <errno.h>
+#include <assert.h>
+
+#define min(x, y) ({ \
+        typeof(x) _x = (x);  typeof(y) _y = (y); \
+        _x < _y ? _x : _y; })
+#define max(x, y) ({ \
+        typeof(x) _x = (x);  typeof(y) _y = (y); \
+        _x > _y ? _x : _y; })
+
+#define SEC		* 1
+#define MIN		* 60 SEC
+#define HOUR	* 60 MIN
+#define DAY		* 24 HOUR
+
+#define __unused				__attribute__((unused))
+#define __fmt_printf(i, first)	__attribute__((format (printtf,(i),(f))))
+
+#ifdef DEBUG
+#define pr_debug(fmt, ...)		ulogd_log(ULOGD_DEBUG, fmt, ## __VA_ARGS__)
+#else
+#define pr_debug(fmt, ...)
+#endif /* DEBUG */
+
+#endif /* COMMON_H */

-- 

[ULOGD RFC 03/30] Replace timer code by working version

[heitzenberger]

[-- Attachment #1: ulogd-timer-handling.diff --]
[-- Type: text/plain, Size: 6987 bytes --]

Replace existing timer code by simple and more importantly working
version.  Current resolution is one second, which may be easily
extended if need be.

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/include/ulogd/ulogd.h
===================================================================
--- ulogd-netfilter.orig/include/ulogd/ulogd.h
+++ ulogd-netfilter/include/ulogd/ulogd.h
@@ -240,17 +240,31 @@ void ulogd_unregister_fd(struct ulogd_fd
 int ulogd_select_main();
 
 /***********************************************************************
- * timer handling
+ * timer handling (timer.c)
  ***********************************************************************/
+#define TIMER_F_USED			0x0001
+#define TIMER_F_PERIODIC		0x0002
 
 struct ulogd_timer {
 	struct llist_head list;
-	struct timeval expires;
-	void (*cb)(void *data);
-	void *data;
+	unsigned expires;			/* seconds */
+	unsigned ival;				/* seconds */
+	unsigned flags;
+	void (* cb)(struct ulogd_timer *);
+	void *data;					/* usually (ulogd_pluginstance *) */
 };
 
+extern struct timeval tv_now;
+extern struct timeval tv_now_local;
+
+#define t_now			tv_now.tv_sec
+#define t_now_local		tv_now_local.tv_sec
+
+int ulogd_timer_init(void);
+int ulogd_timer_run(void);
 int ulogd_register_timer(struct ulogd_timer *timer);
 void ulogd_unregister_timer(struct ulogd_timer *timer);
+void ulogd_timer_schedule(void);
+int ulogd_timer_handle(void);
 
 #endif /* _ULOGD_H */
Index: ulogd-netfilter/src/timer.c
===================================================================
--- ulogd-netfilter.orig/src/timer.c
+++ ulogd-netfilter/src/timer.c
@@ -18,6 +18,9 @@
  *  You should have received a copy of the GNU General Public License
  *  along with this program; if not, write to the Free Software
  *  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+ *
+ *
+ * H. Eitzenberger <holger@eitzenberger.org>  Astaro AG, 2007
  */
 
 #include <unistd.h>
@@ -25,143 +28,115 @@
 #include <string.h>
 #include <sys/time.h>
 #include <time.h>
+#include <errno.h>
 
 #include <ulogd/ulogd.h>
+#include <ulogd/common.h>
 #include <ulogd/linuxlist.h>
 
 static LLIST_HEAD(ulogd_timers);
+static struct tm tm_local;
+struct timeval tv_now, tv_now_local;
 
-static void tv_normalize(struct timeval *out)
-{
-	out->tv_sec += (out->tv_usec / 1000000);
-	out->tv_usec = (out->tv_usec % 1000000);
-}
 
-/* subtract two struct timevals */
-static int tv_sub(struct timeval *res, const struct timeval *from,
-		  const struct timeval *sub)
+int
+ulogd_register_timer(struct ulogd_timer *timer)
 {
-	/* FIXME: this stinks.  Deal with wraps, carry, ... */
-	res->tv_sec = from->tv_sec - sub->tv_sec;
-	res->tv_usec = from->tv_usec - sub->tv_usec;
+	pr_debug("%s: timer=%p\n", __func__, timer);
+
+	if (timer->flags & TIMER_F_USED) {
+		ulogd_log(ULOGD_ERROR, "timer already registered\n");
+		return -1;
+	}
+
+	if (timer->flags & TIMER_F_PERIODIC) {
+		timer->expires = t_now + timer->ival;
+	} else {
+		if (timer->expires == 0) {
+			errno = EINVAL;
+			return -1;
+		}
+	}
+
+	timer->flags |= TIMER_F_USED;
+
+	llist_add_tail(&timer->list, &ulogd_timers);
 
 	return 0;
 }
 
-static int tv_add(struct timeval *res, const struct timeval *a1,
-		  const struct timeval *a2)
+
+void
+ulogd_unregister_timer(struct ulogd_timer *timer)
 {
-	unsigned int carry;
+	pr_debug("%s: timer=%p\n", __func__, timer);
 
-	res->tv_sec = a1->tv_sec + a2->tv_sec;
-	res->tv_usec = a1->tv_usec + a2->tv_usec;
+	/* TODO check for race conditions on unregister */
 
-	tv_normalize(res);
-}
+	if ((timer->flags & TIMER_F_USED) == 0)
+		return;					/* not registered */
 
-static int tv_later(const struct timeval *expires, const struct timeval *now)
-{
-	if (expires->tv_sec < now->tv_sec)
-		return 0;
-	else if (expires->tv_sec > now->tv_sec)
-		return 1;
-	else /* if (expires->tv_sec == now->tv_sec */ {
-		if (expires->tv_usec >= now->tv_usec)
-			return 1;
-	}
+	timer->flags &= ~TIMER_F_USED;
 
-	return 0;
+	llist_del(&timer->list);
 }
 
-static int tv_smaller(const struct timeval *t1, const struct timeval *t2)
-{
-	return tv_later(t2, t1);
-}
 
-static int calc_next_expiration(void)
+int
+ulogd_timer_handle(void)
 {
-	struct ulogd_timer *cur;
-	struct timeval min, now, diff;
-	struct itimerval iti;
-	int ret;
-
-retry:
-	if (llist_empty(&ulogd_timers))
-		return 0;
-
-	llist_for_each_entry(cur, &ulogd_timers, list) {
-		if (ulogd_timers.next == &cur->list)
-			min = cur->expires;
-
-		if (tv_smaller(&cur->expires, &min))
-			min = cur->expires;
-	}
+	struct ulogd_timer *t;
 
-	if (tv_sub(&diff, &min, &now) < 0) {
-		/* FIXME: run expired timer callbacks */
-		/* we cannot run timers from here since we might be
-		 * called from register_timer() within check_n_run() */
+	t_now = time(NULL);			/* UTC */
 
-		/* FIXME: restart with next minimum timer */
-		goto retry;
-	}
+	pr_debug("%s: t_now=%ld\n", __func__, t_now);
 
-	/* re-set kernel timer */
-	memset(&iti, 0, sizeof(iti));
-	memcpy(&iti.it_value, &diff, sizeof(iti.it_value));
-	ret = setitimer(ITIMER_REAL, &iti, NULL);
-	if (ret < 0)
-		return ret;
+	/* get offset to local time every hour */
+	if ((t_now % (1 HOUR)) == 0)
+		localtime_r(&t_now, &tm_local);
 
-	return 0;
-}
+	t_now_local = t_now + tm_local.tm_gmtoff;
 
-void ulogd_timer_check_n_run(void)
-{
-	struct ulogd_timer *cur, *cur2;
-	struct timeval now;
+	llist_for_each_entry(t, &ulogd_timers, list) {
+		assert(t->flags & TIMER_F_USED);
 
-	if (gettimeofday(&now, NULL) < 0)
-		return;
+		if (t->expires <= t_now) {
+			(t->cb)(t);
 
-	llist_for_each_entry_safe(cur, cur2, &ulogd_timers, list) {
-		if (tv_later(&cur->expires, &now)) {
-			/* fist delete it from the list of timers */
-			llist_del(&cur->list);
-			/* then call.  called function can re-add it */
-			(cur->cb)(cur->data);
+			if (t->flags & TIMER_F_PERIODIC)
+				t->expires = t_now + t->ival;
+			else
+				llist_del(&t->list);
 		}
 	}
 
-	calc_next_expiration();
+	return 1;
 }
 
 
-int ulogd_register_timer(struct ulogd_timer *timer)
+int
+ulogd_timer_init(void)
 {
-	int ret;
-	struct timeval tv;
-
-	ret = gettimeofday(&tv, NULL);
-	if (ret < 0)
-		return ret;
-
-	/* convert expiration time into absoulte time */
-	timer->expires.tv_sec += tv.tv_sec;
-	timer->expires.tv_usec += tv.tv_usec;
-
-	llist_add_tail(&timer->list, &ulogd_timers);
-
-	/* re-calculate next expiration */
-	calc_next_expiration();
+	t_now = time(NULL);
+	localtime_r(&t_now, &tm_local);
 
 	return 0;
 }
 
-void ulogd_unregister_timer(struct ulogd_timer *timer)
-{
-	llist_del(&timer->list);
 
-	/* re-calculate next expiration */
-	calc_next_expiration();
+/* start periodic timer */
+int
+ulogd_timer_run(void)
+{
+	struct itimerval itv = {	/* run timer every second */
+		.it_interval = { .tv_sec = 1, },
+		.it_value = { .tv_sec = 1, },
+	};
+
+	if (setitimer(ITIMER_REAL, &itv, NULL) < 0) {
+		ulogd_log(ULOGD_ERROR, "setitimer: %s\n", strerror(errno));
+		return -1;
+	}
+
+	return 0;
 }

-- 

Re: [ULOGD RFC 03/30] Replace timer code by working version

[Pablo Neira Ayuso]

heitzenberger@astaro.com wrote:
> Replace existing timer code by simple and more importantly working
> version.  Current resolution is one second, which may be easily
> extended if need be.

You lose precision here using time(), I think the way to go are the
timeradd(), timercmp(), timersub(), etc. functions that lives in
sys/time.h. I have a patch for those here that I want to test.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: [ULOGD RFC 03/30] Replace timer code by working version

[Holger Eitzenberger]

Pablo Neira Ayuso <pablo@netfilter.org> writes:

>> Replace existing timer code by simple and more importantly working
>> version.  Current resolution is one second, which may be easily
>> extended if need be.
>
> You lose precision here using time(), I think the way to go are the
> timeradd(), timercmp(), timersub(), etc. functions that lives in
> sys/time.h. I have a patch for those here that I want to test.

The inititial timer code was incomplete and IMO too complicated for what
it did.  AFAIRK at the time I started hacking on that there wasn't even
a single user of that API.

This patch turns the timers into simple and more importantly working
timers which with 1-sec resolution.  They might easily be adjusted or
improved later on.

  /holger

Re: [ULOGD RFC 03/30] Replace timer code by working version

[Patrick McHardy]

Holger Eitzenberger wrote:
> Pablo Neira Ayuso <pablo@netfilter.org> writes:
> 
>>> Replace existing timer code by simple and more importantly working
>>> version.  Current resolution is one second, which may be easily
>>> extended if need be.
>> You lose precision here using time(), I think the way to go are the
>> timeradd(), timercmp(), timersub(), etc. functions that lives in
>> sys/time.h. I have a patch for those here that I want to test.

Patches welcome :)

> The inititial timer code was incomplete and IMO too complicated for what
> it did.  AFAIRK at the time I started hacking on that there wasn't even
> a single user of that API.
> 
> This patch turns the timers into simple and more importantly working
> timers which with 1-sec resolution.  They might easily be adjusted or
> improved later on.
+

Agreed, from what I remeber about ulogd2 and our discussions there
doesn't seem to be any need for higher precision. I'll look into
the details later.

Re: [ULOGD RFC 03/30] Replace timer code by working version

[Holger Eitzenberger]

Patrick McHardy <kaber@trash.net> writes:

>> The inititial timer code was incomplete and IMO too complicated for what
>> it did.  AFAIRK at the time I started hacking on that there wasn't even
>> a single user of that API.
>> This patch turns the timers into simple and more importantly working
>> timers which with 1-sec resolution.  They might easily be adjusted or
>> improved later on.

> Agreed, from what I remeber about ulogd2 and our discussions there
> doesn't seem to be any need for higher precision. I'll look into
> the details later.

Yes, there currently is no need for higher-precision timers in ulogd.
Looking at the code I think that Haralds intention was to have those
timers because IPFIX supports (at least principially) higher-precision
timerstamps.  IMO ideally there should be added a config switch for
that.

My next round of patches will clean up the existing code base further,
with focus on the ulogd core.

 /holger

Re: [ULOGD RFC 03/30] Replace timer code by working version

[Patrick McHardy]

Holger Eitzenberger wrote:
> Patrick McHardy <kaber@trash.net> writes:
> 
>>> The inititial timer code was incomplete and IMO too complicated for what
>>> it did.  AFAIRK at the time I started hacking on that there wasn't even
>>> a single user of that API.
>>> This patch turns the timers into simple and more importantly working
>>> timers which with 1-sec resolution.  They might easily be adjusted or
>>> improved later on.
> 
>> Agreed, from what I remeber about ulogd2 and our discussions there
>> doesn't seem to be any need for higher precision. I'll look into
>> the details later.
> 
> Yes, there currently is no need for higher-precision timers in ulogd.
> Looking at the code I think that Haralds intention was to have those
> timers because IPFIX supports (at least principially) higher-precision
> timerstamps.  IMO ideally there should be added a config switch for
> that.
> 
> My next round of patches will clean up the existing code base further,
> with focus on the ulogd core.


Thanks! I'll wait whether Pablo has further objections, but so far I
don't see any show-stoppers that wouldn't be worth the improvements.

[ULOGD RFC 04/30] Add IFI list

[heitzenberger]

[-- Attachment #1: ulogd-ifi.diff --]
[-- Type: text/plain, Size: 10294 bytes --]

IFI implements a list of up-to-date network interfaces with MAC
addresses, mostly used for logging and accounting.  It is therefore
quite redudant to the IFINDEX plugin, but IMO this should go into
ulogd core and not into a plugin.

A future step might be to drop IFINDEX plugin.

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/include/ulogd/ifi.h
===================================================================
--- /dev/null
+++ ulogd-netfilter/include/ulogd/ifi.h
@@ -0,0 +1,43 @@
+/*
+ * ifi.h
+ *
+ * Maintain a list of network interfaces.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2
+ * as published by the Free Software Foundation
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+ *
+ *
+ * Holger Eitzenberger <holger@eitzenberger.org>  Astaro AG, 2007.
+ */
+#ifndef IFI_H
+#define IFI_H
+
+#include <ulogd/linuxlist.h>
+#include <net/if.h>
+
+
+struct ifi {
+	struct llist_head link;
+	unsigned idx;			/* interface index */
+	unsigned used : 1;
+	unsigned flags;
+	char name[IFNAMSIZ];
+	unsigned char lladdr[6];
+};
+
+int ifi_init(void);
+void ifi_fini(void);
+
+struct ifi *ifi_find_by_idx(unsigned);
+
+#endif /* IFI_H */
Index: ulogd-netfilter/src/ifi.c
===================================================================
--- /dev/null
+++ ulogd-netfilter/src/ifi.c
@@ -0,0 +1,379 @@
+/*
+ * ifi.c
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2
+ * as published by the Free Software Foundation
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+ *
+ * Holger Eitzenberger <holger@eitzenberger.org>  Astaro AG 2007.
+ */
+
+#include <ulogd/ulogd.h>
+#include <ulogd/common.h>
+#include <ulogd/ifi.h>
+#include <unistd.h>
+#include <fcntl.h>
+#include <sys/types.h>
+#include <sys/socket.h>
+#include <linux/rtnetlink.h>
+
+#define IFI_STATIC_MAX			64
+
+
+/* the first IFI_STATIC_MAX entries are kept in ifi_static[] for performance
+   reasons, whereas all entries with an interface index larger than
+   IFI_STATIX_MAX-1 are kept in the linked ifi_list. */
+static LLIST_HEAD(ifi_list);
+static struct ifi ifi_static[IFI_STATIC_MAX];
+
+static unsigned nl_seq;			/* last seq# */
+
+
+static struct ifi *
+ifi_alloc(void)
+{
+	struct ifi *ifi;
+
+	if ((ifi = calloc(1, sizeof(struct ifi))) == NULL)
+		return NULL;
+
+	return ifi;
+}
+
+
+struct ifi *
+ifi_find_by_idx(unsigned idx)
+{
+	struct ifi *ifi;
+
+	if (idx < IFI_STATIC_MAX)
+		ifi = &ifi_static[idx];
+	else {
+		llist_for_each_entry(ifi, &ifi_list, link) {
+			if (ifi->idx == idx)
+				break;
+		}
+
+		if (ifi == NULL)
+			return NULL;
+	}
+
+	return ifi->used ? ifi : NULL;
+}
+
+
+static struct ifi *
+ifi_find_or_add(unsigned idx)
+{
+	struct ifi *ifi = ifi_find_by_idx(idx);
+
+	if (ifi != NULL) {
+		pr_debug("%s: found ifi '%s' (index %u)\n", __func__,
+				 ifi->name, ifi->idx);
+		return ifi;
+	}
+
+	/* add */
+	if (idx < IFI_STATIC_MAX)
+		ifi = &ifi_static[idx];
+	else {
+		if ((ifi = ifi_alloc()) == NULL) {
+			ulogd_log(ULOGD_ERROR, "ifi: out of memory\n");
+			return NULL;
+		}
+
+		llist_add_tail(&ifi->link, &ifi_list);
+	}
+
+	ifi->idx = idx;
+	ifi->used = 1;
+
+	pr_debug("%s: added interface '%s' (index %u)\n", __func__,
+			 ifi->name, ifi->idx);
+
+	return ifi;
+}
+
+
+static bool
+ifi_del(unsigned idx)
+{
+	struct ifi *ifi;
+
+	if (idx < IFI_STATIC_MAX) {
+		ifi = &ifi_static[idx];
+
+		if (ifi->used) {
+			ifi->used = 0;
+
+			pr_debug("%s: deleted interface '%s' (index %u)\n", __func__,
+					 ifi->name, ifi->idx);
+
+			return true;
+		}
+	} else {
+		struct ifi *tmp;
+
+		llist_for_each_entry_safe(ifi, tmp, &ifi_list, link) {
+			if (ifi->idx == idx) {
+				llist_del(&ifi->link);
+				free(ifi);
+
+				return true;
+			}
+		}
+	}
+
+	return false;
+}
+
+
+static void dump_bytes(const char *, unsigned char *, size_t) __unused;
+
+static void
+dump_bytes(const char *prefix, unsigned char *data, size_t len)
+{
+	int i;
+	static char buf[1024];
+	char *pch = buf;
+
+	if (prefix)
+		pch += sprintf(pch, "%s: ", prefix);
+
+	for (i = 0; i < len; i++)
+		pch += sprintf(pch, "0x%.2x ", data[i]);
+
+	fprintf(stdout, "%s\n", buf);
+}
+
+
+static void dump_nlmsg(FILE *, struct nlmsghdr *) __unused;
+
+static void
+dump_nlmsg(FILE *fp, struct nlmsghdr *nlh)
+{
+	fprintf(fp, "rtmsg: len=%04x type=%08x flags=%08x seq=%08x\n",
+			nlh->nlmsg_len,	nlh->nlmsg_type, nlh->nlmsg_flags,
+			nlh->nlmsg_seq);
+}
+
+
+static ssize_t sprint_lladdr(char *, size_t, const unsigned char *) __unused;
+
+static ssize_t
+sprint_lladdr(char *buf, size_t len, const unsigned char *addr)
+{
+	char *pch = buf;
+
+	pch += sprintf(pch, "%02x:%02x:%02x:%02x:%02x:%02x",
+				   addr[0], addr[1], addr[2], addr[3], addr[4], addr[5]);
+
+	return pch - buf;
+}
+
+static int
+rtnl_parse_attrs(struct rtattr *attr, size_t attr_len,
+				 struct rtattr **rta, size_t rta_len)
+{
+	memset(rta, 0, rta_len * sizeof(struct rtattr *));
+
+	while (RTA_OK(attr, attr_len)) {
+		if (attr->rta_type < rta_len)
+			rta[attr->rta_type] = attr;
+
+		attr = RTA_NEXT(attr, attr_len);
+	}
+
+	return 0;
+}
+
+
+static int
+nl_send(int fd, struct nlmsghdr *nlh)
+{
+	struct sockaddr_nl sa;
+
+	memset(&sa, 0, sizeof(sa));
+
+	sa.nl_family = AF_NETLINK;
+
+	nlh->nlmsg_pid = getpid();
+	nlh->nlmsg_seq = ++nl_seq;
+
+	return sendto(fd, nlh, nlh->nlmsg_len, 0, (struct sockaddr *)&sa,
+				  sizeof(sa));
+}
+
+
+static int
+nl_dump_request(int fd, int type)
+{
+	struct {
+		struct nlmsghdr nlh;
+		struct rtgenmsg gen;
+	} req = {
+		.nlh.nlmsg_flags = NLM_F_REQUEST | NLM_F_DUMP,
+		.gen.rtgen_family = AF_UNSPEC,
+	};
+
+	req.nlh.nlmsg_type = type;
+	req.nlh.nlmsg_len = sizeof(req);
+
+	return nl_send(fd, &req.nlh);
+}
+
+
+static int
+nl_listen(int fd, char *buf, size_t len)
+{
+	return read(fd, buf, len);
+}
+
+
+static int
+rtnl_handle_link(struct nlmsghdr *nlh)
+{
+	struct ifinfomsg *m = NLMSG_DATA(nlh);
+	struct rtattr *ifla[IFLA_MAX];
+	struct ifi *ifi;
+	size_t len = nlh->nlmsg_len - NLMSG_LENGTH(sizeof(struct ifinfomsg));
+
+	rtnl_parse_attrs(IFLA_RTA(m), len, ifla, IFLA_MAX);
+
+	switch (nlh->nlmsg_type) {
+	case RTM_NEWLINK:
+		if (m->ifi_flags & IFF_UP) {
+			if ((ifi = ifi_find_or_add(m->ifi_index)) == NULL)
+				return -1;
+
+			ifi->flags = m->ifi_flags;
+
+			if (ifla[IFLA_ADDRESS])
+				memcpy(ifi->lladdr, RTA_DATA(ifla[IFLA_ADDRESS]), 6);
+
+			if (ifla[IFLA_IFNAME])
+				strcpy(ifi->name, RTA_DATA(ifla[IFLA_IFNAME]));
+		} else
+			ifi_del(m->ifi_index);
+		break;
+
+	case RTM_DELLINK:
+		break;
+
+	default:
+		break;
+	}
+
+	return 0;
+}
+
+
+static int
+rtnl_handle_msg(struct nlmsghdr *nlh, size_t len)
+{
+	if (nlh == NULL)
+		return -1;
+
+	while (NLMSG_OK(nlh, len)) {
+		if (nlh->nlmsg_type & NLMSG_DONE)
+			return 0;
+
+#if 0
+		dump_nlmsg(stdout, nlh);
+#endif /* 0 */
+
+		switch (nlh->nlmsg_type) {
+		case RTM_NEWLINK:
+		case RTM_DELLINK:
+			rtnl_handle_link(nlh);
+			break;
+
+		case NLMSG_ERROR:
+			break;
+
+		default:
+			break;
+		}
+
+		nlh = NLMSG_NEXT(nlh, len);
+	}
+
+
+	return 0;
+}
+
+
+static int
+rtnl_read_cb(int fd, unsigned what, void *data)
+{
+	static char buf[4096];
+
+	for (;;) {
+		int nbytes;
+
+		if ((nbytes = nl_listen(fd, buf, sizeof(buf))) < 0) {
+			if (errno == EWOULDBLOCK)
+				return 0;
+
+			ulogd_log(ULOGD_ERROR, "nl_listen: %s\n", strerror(errno));
+
+			return -1;
+		}
+
+		rtnl_handle_msg((struct nlmsghdr *)buf, nbytes);
+	}
+
+	return 0;
+}
+
+
+static struct ulogd_fd nl_fd = {
+	.fd = -1,
+	.cb = rtnl_read_cb,
+	.when = ULOGD_FD_READ,
+};
+
+
+int
+ifi_init(void)
+{
+	struct sockaddr_nl sa = {
+		.nl_family = AF_NETLINK,
+		.nl_groups = RTNLGRP_LINK,
+	};
+
+	pr_debug("Initializing ifi\n");
+
+	sa.nl_pid = getpid();
+
+	if ((nl_fd.fd = socket(PF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE)) < 0) {
+		ulogd_log(ULOGD_ERROR, "ifi: socket: %s\n", strerror(errno));
+		return -1;
+	}
+
+	if (bind(nl_fd.fd, (struct sockaddr *)&sa, sizeof(sa)) < 0) {
+		ulogd_log(ULOGD_ERROR, "ifi: bind: %s\n", strerror(errno));
+		return -1;
+	}
+
+	if (connect(nl_fd.fd, (struct sockaddr *)&sa, sizeof(sa)) < 0) {
+		ulogd_log(ULOGD_ERROR, "ifi: connect: %s\n", strerror(errno));
+		return -1;
+	}
+
+	if (ulogd_register_fd(&nl_fd) < 0)
+		return -1;
+
+	nl_dump_request(nl_fd.fd, RTM_GETLINK);
+
+	return 0;
+}
Index: ulogd-netfilter/src/Makefile.am
===================================================================
--- ulogd-netfilter.orig/src/Makefile.am
+++ ulogd-netfilter/src/Makefile.am
@@ -5,5 +5,5 @@ AM_CPPFLAGS = $(all_includes) -I$(top_sr
 
 sbin_PROGRAMS = ulogd
 
-ulogd_SOURCES = ulogd.c select.c timer.c conffile.c
+ulogd_SOURCES = ulogd.c ifi.c select.c timer.c conffile.c
 ulogd_LDFLAGS = -export-dynamic
Index: ulogd-netfilter/src/ulogd.c
===================================================================
--- ulogd-netfilter.orig/src/ulogd.c
+++ ulogd-netfilter/src/ulogd.c
@@ -63,6 +63,9 @@
 #include <syslog.h>
 #include <ulogd/conffile.h>
 #include <ulogd/ulogd.h>
+#include <ulogd/ifi.h>
+
+
 #ifdef DEBUG
 #define DEBUGP(format, args...) fprintf(stderr, format, ## args)
 #else
@@ -983,6 +986,9 @@ int main(int argc, char* argv[])
 	ulogd_log(ULOGD_INFO, 
 		  "initialization finished, entering main loop\n");
 
+	if (ifi_init() < 0)
+		exit(EXIT_FAILURE);
+
 	ulogd_main_loop();
 
 	/* hackish, but result is the same */

-- 

[ULOGD RFC 05/30] Add signalling subsystem

[heitzenberger]

[-- Attachment #1: ulogd-fix-signal-handling.diff --]
[-- Type: text/plain, Size: 13248 bytes --]

This patch adds the concept of synchronous and asynchronous signal
handlers to ulogd, where 'synchronous' just means to be synchronous to
the underlying IO multiplexer.

Will later be used by plugins like SQLITE3 and NFCT.

One of the changes herein is the usage of the pthread library.  This
is strictly necessary because some plugins might (and SQLITE3 *does*)
use pthreads.

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/src/Makefile.am
===================================================================
--- ulogd-netfilter.orig/src/Makefile.am
+++ ulogd-netfilter/src/Makefile.am
@@ -5,5 +5,5 @@ AM_CPPFLAGS = $(all_includes) -I$(top_sr
 
 sbin_PROGRAMS = ulogd
 
-ulogd_SOURCES = ulogd.c ifi.c select.c timer.c conffile.c
-ulogd_LDFLAGS = -export-dynamic
+ulogd_SOURCES = ulogd.c ifi.c select.c timer.c signal.c conffile.c
+ulogd_LDFLAGS = -export-dynamic -lpthread
Index: ulogd-netfilter/src/select.c
===================================================================
--- ulogd-netfilter.orig/src/select.c
+++ ulogd-netfilter/src/select.c
@@ -23,6 +23,7 @@
 
 #include <fcntl.h>
 #include <ulogd/ulogd.h>
+#include <ulogd/common.h>
 #include <ulogd/linuxlist.h>
 
 static int maxfd = 0;
@@ -59,6 +60,7 @@ int ulogd_select_main()
 {
 	struct ulogd_fd *ufd;
 	fd_set readset, writeset, exceptset;
+	struct timeval tv = { .tv_sec = 1, };
 	int i;
 
 	FD_ZERO(&readset);
@@ -77,7 +79,13 @@ int ulogd_select_main()
 			FD_SET(ufd->fd, &exceptset);
 	}
 
-	i = select(maxfd+1, &readset, &writeset, &exceptset, NULL);
+ again:
+	i = select(maxfd+1, &readset, &writeset, &exceptset, &tv);
+	if (i < 0) {
+		if (errno == EINTR)
+			goto again;
+	}
+
 	if (i > 0) {
 		/* call registered callback functions */
 		llist_for_each_entry(ufd, &ulogd_fds, list) {
@@ -96,5 +104,6 @@ int ulogd_select_main()
 				ufd->cb(ufd->fd, flags, ufd->data);
 		}
 	}
+
 	return i;
 }
Index: ulogd-netfilter/src/ulogd.c
===================================================================
--- ulogd-netfilter.orig/src/ulogd.c
+++ ulogd-netfilter/src/ulogd.c
@@ -53,16 +53,19 @@
 #include <errno.h>
 #include <time.h>
 #include <ctype.h>
-#include <signal.h>
 #include <dlfcn.h>
 #include <sys/types.h>
 #include <dirent.h>
 #include <getopt.h>
 #include <pwd.h>
 #include <grp.h>
+#include <pthread.h>
 #include <syslog.h>
-#include <ulogd/conffile.h>
+
 #include <ulogd/ulogd.h>
+#include <ulogd/common.h>
+#include <ulogd/conffile.h>
+#include <ulogd/signal.h>
 #include <ulogd/ifi.h>
 
 
@@ -204,7 +207,6 @@ int ulogd_wildcard_inputkeys(struct ulog
 
 	/* second pass: copy key names */
 	llist_for_each_entry(pi_cur, &stack->list, list) {
-		struct ulogd_key *cur;
 		int i;
 
 		for (i = 0; i < pi_cur->plugin->output.num_keys; i++)
@@ -298,7 +300,7 @@ void __ulogd_log(int level, char *file, 
 		vsyslog(ulogd2syslog_level(level), format, ap);
 		va_end(ap);
 	} else {
-		if (logfile)
+  		if (logfile)
 			outfd = logfile;
 		else
 			outfd = stderr;
@@ -711,21 +713,22 @@ out_buf:
 
 static int ulogd_main_loop(void)
 {
+	sigset_t curr;
 	int ret = 0;
 
-	while (1) {
+	ulogd_get_sigset(&curr);
+
+	pthread_sigmask(SIG_UNBLOCK, &curr, NULL);
+
+	for (;;) {
 		ret = ulogd_select_main();
 		if (ret == 0) 
 			continue;
 
 		if (ret < 0) {
-			if (errno == -EINTR)
-				continue;
-			else {
-				ulogd_log(ULOGD_ERROR, "select returned %s\n",
+			ulogd_log(ULOGD_ERROR, "select returned %s\n",
 					  strerror(errno));
-				break;
-			}
+			break;
 		}
 	}
 
@@ -736,7 +739,7 @@ static int ulogd_main_loop(void)
 static int logfile_open(const char *name)
 {
 	if (name)
-		ulogd_logfile = name;
+		ulogd_logfile = (char *)name;
 
 	if (!strcmp(name, "stdout")) {
 		logfile = stdout;
@@ -794,54 +797,83 @@ static int parse_conffile(const char *se
 	return 1;
 }
 
-static void deliver_signal_pluginstances(int signal)
+static int
+for_each_pluginstance(int (* cb)(struct ulogd_pluginstance *,
+								 struct ulogd_pluginstance_stack *,
+								 void *), void *arg)
 {
 	struct ulogd_pluginstance_stack *stack;
-	struct ulogd_pluginstance *pi;
+	int sum = 0;
+
+	pr_debug("%s: cb=%p\n", __func__, cb);
 
 	llist_for_each_entry(stack, &ulogd_pi_stacks, stack_list) {
+		struct ulogd_pluginstance *pi;
+
 		llist_for_each_entry(pi, &stack->list, list) {
-			if (pi->plugin->signal)
-				(*pi->plugin->signal)(pi, signal);
+			int ret;
+
+			if ((ret = cb(pi, stack, arg)) < 0)
+				return -1;
+
+			sum += ret;
 		}
 	}
+
+	return sum;
 }
 
-static void sigterm_handler(int signal)
+static int
+_do_signal(struct ulogd_pluginstance *pi,
+		   struct ulogd_pluginstance_stack *stack, void *arg)
 {
-	
-	ulogd_log(ULOGD_NOTICE, "sigterm received, exiting\n");
+	int signo = (int) arg;
 
-	deliver_signal_pluginstances(signal);
+	if (pi->plugin->signal) {
+		pi->plugin->signal(pi, signo);
 
-	if (logfile != stdout)
-		fclose(logfile);
+		return 1;
+	}
 
-	exit(0);
+	return 0;
 }
 
-static void signal_handler(int signal)
+static void
+sync_sig_handler(int signo)
 {
-	ulogd_log(ULOGD_NOTICE, "signal received, calling pluginstances\n");
-	
-	switch (signal) {
+	pr_debug("%s: signal '%d' received\n", __func__, signo);
+
+	switch (signo) {
 	case SIGHUP:
-		/* reopen logfile */
-		if (logfile != stdout && logfile != &syslog_dummy) {
-			fclose(logfile);
-			logfile = fopen(ulogd_logfile, "a");
-			if (!logfile)
-				sigterm_handler(signal);
-		}
 		break;
+
 	case SIGALRM:
-		ulogd_timer_check_n_run();
+		ulogd_timer_handle();
+		break;
+
+	case SIGTERM:
 		break;
+
 	default:
 		break;
 	}
 
-	deliver_signal_pluginstances(signal);
+	for_each_pluginstance(_do_signal, (void *) signo);
+}
+
+static void
+sig_handler(int signo)
+{
+	pr_debug("%s: signal '%d' received\n", __func__, signo);
+
+	switch (signo) {
+	case SIGINT:
+		exit(0);
+		break;
+
+	default:
+		break;
+	}
 }
 
 static void print_usage(void)
@@ -867,7 +899,8 @@ static struct option opts[] = {
 	{ 0 }
 };
 
-int main(int argc, char* argv[])
+int
+main(int argc, char* argv[])
 {
 	int argch;
 	int daemonize = 0;
@@ -922,6 +955,11 @@ int main(int argc, char* argv[])
 		}
 	}
 
+	if (ulogd_signal_init() < 0)
+		exit(EXIT_FAILURE);
+
+	ulogd_timer_init();
+
 	if (config_register_file(ulogd_configfile)) {
 		ulogd_log(ULOGD_FATAL, "error registering configfile \"%s\"\n",
 			  ulogd_configfile);
@@ -977,21 +1015,21 @@ int main(int argc, char* argv[])
 		setsid();
 	}
 
-	signal(SIGTERM, &sigterm_handler);
-	signal(SIGHUP, &signal_handler);
-	signal(SIGALRM, &signal_handler);
-	signal(SIGUSR1, &signal_handler);
-	signal(SIGUSR2, &signal_handler);
-
-	ulogd_log(ULOGD_INFO, 
-		  "initialization finished, entering main loop\n");
+	ulogd_register_signal(SIGTERM, sync_sig_handler, ULOGD_SIGF_SYNC);
+	ulogd_register_signal(SIGINT, sig_handler, 0);
+	ulogd_register_signal(SIGHUP, sync_sig_handler, ULOGD_SIGF_SYNC);
+	ulogd_register_signal(SIGALRM, sync_sig_handler, ULOGD_SIGF_SYNC);
+	ulogd_register_signal(SIGUSR1, sync_sig_handler, ULOGD_SIGF_SYNC);
+	ulogd_register_signal(SIGUSR2, sync_sig_handler, ULOGD_SIGF_SYNC);
 
 	if (ifi_init() < 0)
 		exit(EXIT_FAILURE);
 
+	ulogd_timer_run();
+
+	ulogd_log(ULOGD_INFO, "entering main loop\n");
+
 	ulogd_main_loop();
 
-	/* hackish, but result is the same */
-	sigterm_handler(SIGTERM);	
-	return(0);
+	return 0;
 }
Index: ulogd-netfilter/include/ulogd/signal.h
===================================================================
--- /dev/null
+++ ulogd-netfilter/include/ulogd/signal.h
@@ -0,0 +1,46 @@
+/*
+ * signal.h
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2
+ * as published by the Free Software Foundation
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+ *
+ *
+ * Holger Eitzenberger <holger@eitzenberger.org>  Astaro AG, 2007.
+ */
+#ifndef SIGNAL_H
+#define SIGNAL_H
+
+#include <signal.h>
+#include <ulogd/linuxlist.h>
+
+
+/* signal flags */
+#define ULOGD_SIGF_SYNC		0x00000001 /* signal is synchronous */
+
+
+struct ulogd_signal {
+	struct llist_head link;
+	int signo;
+	unsigned flags;
+	void (* handler)(int);
+};
+
+
+struct ulogd_signal *ulogd_register_signal(int, void (*)(int), unsigned);
+int ulogd_unregister_signal(struct ulogd_signal *);
+int ulogd_sigaddset(int);
+int ulogd_get_sigset(sigset_t *);
+int ulogd_deliver_signal(int signo);
+int ulogd_signal_init(void);
+
+#endif /* SIGNAL_H */
Index: ulogd-netfilter/src/signal.c
===================================================================
--- /dev/null
+++ ulogd-netfilter/src/signal.c
@@ -0,0 +1,193 @@
+/*
+ * signal.c
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2
+ * as published by the Free Software Foundation
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+ *
+ * Holger Eitzenberger <holger@eitzenberger.org  Astaro AG, 2007.
+ */
+#include <ulogd/ulogd.h>
+#include <ulogd/common.h>
+#include <ulogd/signal.h>
+#include <unistd.h>
+
+#define SIG_F_USED		0x00000001
+
+
+static struct sig_state {
+	struct llist_head head;
+	struct llist_head async_head;
+	unsigned flags;
+	unsigned cnt;
+} sig_state[NSIG];
+static sigset_t currset;
+static int sig_pipe[2] = { -1, -1 };
+static struct ulogd_fd sig_pipe_fd;
+
+
+static void
+sig_handler(int signo)
+{
+	struct ulogd_signal *sig;
+	sigset_t sigset;
+
+	assert(sig_pipe[1] >= 0);
+
+	pr_debug("%s: received signal '%d'\n", __func__, signo);
+
+	sigemptyset(&sigset);
+	sigaddset(&sigset, signo);
+
+	pthread_sigmask(SIG_BLOCK, &sigset, NULL);
+
+	llist_for_each_entry(sig, &sig_state[signo].async_head, link)
+		sig->handler(signo);
+
+	pthread_sigmask(SIG_UNBLOCK, &sigset, NULL);
+
+	if (!llist_empty(&sig_state[signo].head))
+		write(sig_pipe[1], &signo, sizeof(signo));
+}
+
+struct ulogd_signal *
+ulogd_register_signal(int signo, void (* sigh)(int), unsigned flags)
+{
+	struct ulogd_signal *sig;
+
+	if (signo < 0 || signo > NSIG || sigh == NULL)
+		return NULL;
+
+	/* TODO check signo for values which may not be used synchronous */
+
+	pr_debug("%s: registering handler %p for signal '%d'\n", __func__,
+			 sigh, signo);
+
+	if ((sig = calloc(1, sizeof(struct ulogd_signal))) == NULL)
+		return NULL;
+
+	sig->signo = signo;
+	sig->flags = flags;
+	sig->handler = sigh;
+
+	sig_state[signo].cnt++;
+
+	/* add real signal handler */
+	if ((sig_state[signo].flags & SIG_F_USED) == 0) {
+		signal(signo, sig_handler);
+
+		sig_state[signo].flags |= SIG_F_USED;
+	}
+
+	if (flags & ULOGD_SIGF_SYNC)
+		llist_add_tail(&sig->link, &sig_state[signo].head);
+	else
+		llist_add_tail(&sig->link, &sig_state[signo].async_head);
+
+	ulogd_sigaddset(signo);
+
+	return sig;
+}
+
+int
+ulogd_unregister_signal(struct ulogd_signal *sig)
+{
+	if (sig == NULL || sig->signo < 0 || sig->signo >= NSIG)
+		return -1;
+
+	pr_debug("%s: unregistering handler %p\n", __func__, sig);
+
+	if (--sig_state[sig->signo].cnt == 0)
+		signal(sig->signo, SIG_DFL);
+
+	llist_del(&sig->link);
+
+	free(sig);
+
+	return 0;
+}
+
+int
+ulogd_sigaddset(int signo)
+{
+	return sigaddset(&currset, signo);
+}
+
+int
+ulogd_get_sigset(sigset_t *sigset)
+{
+	assert(sigset != NULL);
+
+	memcpy(sigset, &currset, sizeof(sigset_t));
+
+	return 0;
+}
+
+static int
+sig_pipe_cb(int fd, unsigned what, void *arg)
+{
+	struct ulogd_signal *sig;
+	sigset_t sigset;
+	int signo, nbytes;
+
+	assert(what == ULOGD_FD_READ);
+
+	nbytes = read(fd, &signo, sizeof(signo));
+
+	pr_debug("%s: signo=%d\n", __func__, signo);
+
+	assert(nbytes == sizeof(signo));
+
+	if (signo < 0 || signo > NSIG)
+		abort();
+
+	sigemptyset(&sigset);
+	sigaddset(&sigset, signo);
+
+	pthread_sigmask(SIG_BLOCK, &sigset, NULL);
+
+	llist_for_each_entry(sig, &sig_state[signo].head, link)
+		sig->handler(signo);
+
+	pthread_sigmask(SIG_UNBLOCK, &sigset, NULL);
+
+	return 0;
+}
+
+int
+ulogd_signal_init(void)
+{
+	int i;
+	sigset_t fullset;
+
+	sigfillset(&fullset);
+	pthread_sigmask(SIG_SETMASK, &fullset, NULL);
+
+	sigemptyset(&currset);
+
+	for (i = 0; i < NSIG; i++) {
+		INIT_LLIST_HEAD(&sig_state[i].head);
+		INIT_LLIST_HEAD(&sig_state[i].async_head);
+	}
+
+	/* init signal pipe */
+	if (pipe(sig_pipe) < 0) {
+		ulogd_log(ULOGD_FATAL, "unable to initialize signal pipe\n");
+		return -1;
+	}
+
+	sig_pipe_fd.fd = sig_pipe[0];
+	sig_pipe_fd.cb = sig_pipe_cb;
+	sig_pipe_fd.when = ULOGD_FD_READ;
+
+	return ulogd_register_fd(&sig_pipe_fd);
+}

-- 

Re: [ULOGD RFC 05/30] Add signalling subsystem

[Pablo Neira Ayuso]

heitzenberger@astaro.com wrote:
> This patch adds the concept of synchronous and asynchronous signal
> handlers to ulogd, where 'synchronous' just means to be synchronous to
> the underlying IO multiplexer.

Interesting.

> Will later be used by plugins like SQLITE3 and NFCT.
> 
> One of the changes herein is the usage of the pthread library.  This
> is strictly necessary because some plugins might (and SQLITE3 *does*)
> use pthreads.
> Index: ulogd-netfilter/src/select.c
> ===================================================================
> --- ulogd-netfilter.orig/src/select.c
> +++ ulogd-netfilter/src/select.c
> @@ -23,6 +23,7 @@
>  
>  #include <fcntl.h>
>  #include <ulogd/ulogd.h>
> +#include <ulogd/common.h>
>  #include <ulogd/linuxlist.h>
>  
>  static int maxfd = 0;
> @@ -59,6 +60,7 @@ int ulogd_select_main()
>  {
>  	struct ulogd_fd *ufd;
>  	fd_set readset, writeset, exceptset;
> +	struct timeval tv = { .tv_sec = 1, };
>  	int i;
>  
>  	FD_ZERO(&readset);
> @@ -77,7 +79,13 @@ int ulogd_select_main()
>  			FD_SET(ufd->fd, &exceptset);
>  	}
>  
> -	i = select(maxfd+1, &readset, &writeset, &exceptset, NULL);
> + again:
> +	i = select(maxfd+1, &readset, &writeset, &exceptset, &tv);

Hm, why do we poll now every 1 second event even if there's nothing to do?

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: [ULOGD RFC 05/30] Add signalling subsystem

[Holger Eitzenberger]

Pablo Neira Ayuso <pablo@netfilter.org> writes:

> Hm, why do we poll now every 1 second event even if there's nothing to do?

AFAIRK I initially planned to use that for timers.  But yes, this is
worth to be removed later on.  Thanks.

  /holger

Re: [ULOGD RFC 05/30] Add signalling subsystem

[Patrick McHardy]

Holger Eitzenberger wrote:
> Pablo Neira Ayuso <pablo@netfilter.org> writes:
> 
>> Hm, why do we poll now every 1 second event even if there's nothing to do?
> 
> AFAIRK I initially planned to use that for timers.  But yes, this is
> worth to be removed later on.  Thanks.


Thanks for noticing this. Since its a probably a major effort to
respin these patches I tend to favour the "fix later" solution.
Holger, I assume this would be a simple fix on top?

Re: [ULOGD RFC 05/30] Add signalling subsystem

[Holger Eitzenberger]

Patrick McHardy <kaber@trash.net> writes:

>>> Hm, why do we poll now every 1 second event even if there's nothing to do?
>> AFAIRK I initially planned to use that for timers.  But yes, this is
>> worth to be removed later on.  Thanks.

> Thanks for noticing this. Since its a probably a major effort to
> respin these patches I tend to favour the "fix later" solution.
> Holger, I assume this would be a simple fix on top?

Yes, I'll provide one.

Thanks Patrick!

 /holger

Re: [ULOGD RFC 05/30] Add signalling subsystem

[Holger Eitzenberger]

Pablo Neira Ayuso <pablo@netfilter.org> writes:

>> +	i = select(maxfd+1, &readset, &writeset, &exceptset, &tv);
>
> Hm, why do we poll now every 1 second event even if there's nothing to do?

I just noticed that patch

 [ULOGD RFC 05/30] Add signalling subsystem

already does what you request.  So, in retrospect 1) I'll check my code
first before commenting and 2) you definitely should check those patches
in reverse order :).

 /holger

 

[ULOGD RFC 06/30] Conffile cleanup, use common pr_debug()

[heitzenberger]

[-- Attachment #1: ulogd-config-changes.diff --]
[-- Type: text/plain, Size: 3412 bytes --]

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/src/conffile.c
===================================================================
--- ulogd-netfilter.orig/src/conffile.c
+++ ulogd-netfilter/src/conffile.c
@@ -17,17 +17,10 @@
  *  along with this program; if not, write to the Free Software
  *  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
  */
-
-#include <stdio.h>
-#include <string.h>
-#include <stdlib.h>
+#include <ulogd/ulogd.h>
+#include <ulogd/common.h>
 #include <ulogd/conffile.h>
 
-#ifdef DEBUG_CONF
-#define DEBUGC(format, args...) fprintf(stderr, format, ## args)
-#else
-#define DEBUGC(format, args...)
-#endif
 
 /* points to config entry with error */
 struct config_entry *config_errce = NULL;
@@ -101,6 +94,8 @@ int config_register_file(const char *fil
 	if (fname)
 		return 1;
 
+	pr_debug("%s: registered config file '%s'\n", __func__, file);
+
 	fname = (char *) malloc(strlen(file)+1);
 	if (!fname)
 		return -ERROOM;
@@ -121,12 +116,12 @@ int config_parse_file(const char *sectio
 	char linebuf[LINE_LEN+1];
 	char *line = linebuf;
 
+	pr_debug("%s: section='%s' file='%s'\n", __func__, section, fname);
+
 	cfile = fopen(fname, "r");
 	if (!cfile)
 		return -ERROPEN;
 
-	DEBUGC("parsing section [%s]\n", section);
-
 	/* Search for correct section */
 	while (fgets(line, LINE_LEN, cfile)) {
 		char wordbuf[LINE_LEN];
@@ -137,7 +132,7 @@ int config_parse_file(const char *sectio
 
 		if (!(wordend = get_word(line, " \t\n[]", (char *) wordbuf)))
 			continue;
-		DEBUGC("word: \"%s\"\n", wordbuf);
+		pr_debug("word: \"%s\"\n", wordbuf);
 		if (!strcmp(wordbuf, section)) {
 			found = 1;
 			break;
@@ -156,7 +151,7 @@ int config_parse_file(const char *sectio
 		char wordbuf[LINE_LEN];
 		char *wordend;
 		
-		DEBUGC("line read: %s\n", line);
+		pr_debug("line read: %s\n", line);
 		if (*line == '#')
 			continue;
 
@@ -164,14 +159,14 @@ int config_parse_file(const char *sectio
 			continue;
 
 		if (wordbuf[0] == '[' ) {
-			DEBUGC("Next section '%s' encountered\n", wordbuf);
+			pr_debug("Next section '%s' encountered\n", wordbuf);
 			break;
 		}
 
-		DEBUGC("parse_file: entering main loop\n");
+		pr_debug("parse_file: entering main loop\n");
 		for (i = 0; i < kset->num_ces; i++) {
 			struct config_entry *ce = &kset->ces[i];
-			DEBUGC("parse main loop, key: %s\n", ce->key);
+			pr_debug("parse main loop, key: %s\n", ce->key);
 			if (strcmp(ce->key, (char *) &wordbuf)) {
 				continue;
 			}
@@ -181,7 +176,7 @@ int config_parse_file(const char *sectio
 
 			if (ce->hit && !(ce->options & CONFIG_OPT_MULTI))
 			{
-				DEBUGC("->ce-hit and option not multi!\n");
+				pr_debug("->ce-hit and option not multi!\n");
 				config_errce = ce;
 				err = -ERRMULT;
 				goto cpf_error;
@@ -205,15 +200,15 @@ int config_parse_file(const char *sectio
 			}
 			break;
 		}
-		DEBUGC("parse_file: exiting main loop\n");
+		pr_debug("parse_file: exiting main loop\n");
 	}
 
 
 	for (i = 0; i < kset->num_ces; i++) {
 		struct config_entry *ce = &kset->ces[i];
-		DEBUGC("ce post loop, ce=%s\n", ce->key);
+		pr_debug("ce post loop, ce=%s\n", ce->key);
 		if ((ce->options & CONFIG_OPT_MANDATORY) && (ce->hit == 0)) {
-			DEBUGC("Mandatory config directive \"%s\" not found\n",
+			pr_debug("Mandatory config directive \"%s\" not found\n",
 				ce->key);
 			config_errce = ce;
 			err = -ERRMAND;

-- 

[ULOGD RFC 07/30] Renice to -1 on startup

[heitzenberger]

[-- Attachment #1: ulogd-renice-on-startup.diff --]
[-- Type: text/plain, Size: 462 bytes --]

Thus possibly preventing e.g. ctnetlink from overruns on busy sites.

TODO: make this conditional

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/src/ulogd.c
===================================================================
--- ulogd-netfilter.orig/src/ulogd.c
+++ ulogd-netfilter/src/ulogd.c
@@ -1004,6 +1004,8 @@ main(int argc, char* argv[])
 		}
 	}
 
+	nice(-1);
+
 	if (daemonize){
 		if (fork()) {
 			exit(0);

-- 

Re: [ULOGD RFC 07/30] Renice to -1 on startup

[Pablo Neira Ayuso]

heitzenberger@astaro.com wrote:
> Thus possibly preventing e.g. ctnetlink from overruns on busy sites.

Also interesting, do you really observe a real improvement? I'll try
this with conntrackd tomorrow in my testbed.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: [ULOGD RFC 07/30] Renice to -1 on startup

[Patrick McHardy]

Pablo Neira Ayuso wrote:
> heitzenberger@astaro.com wrote:
>> Thus possibly preventing e.g. ctnetlink from overruns on busy sites.
> 
> Also interesting, do you really observe a real improvement? I'll try
> this with conntrackd tomorrow in my testbed.


I noticed huge differences in nfnetlink_queue performance by renicing,
which is a clear indication of insufficient buffer space in the
socker receive queue (in case of nfnetlink_queue actually receive
queue size *or* kernel queue size). The buffers have to be dimensioned
large enough to catch userspace latency fluctuations, so the proper
fix is most likely to simply increase the receive queue size.

Which reminds me, for nfnetlink_queue we should think about providing
a mechanism to automatically size both kernel queue and socket receive
buffers properly or at least measure latencies.

Re: [ULOGD RFC 07/30] Renice to -1 on startup

[Holger Eitzenberger]

Pablo Neira Ayuso <pablo@netfilter.org> writes:

> Also interesting, do you really observe a real improvement? I'll try
> this with conntrackd tomorrow in my testbed.

Yes, there is a real improvement by that.  Especially on systems
whith many contending processes this is a real improvement.

 /holger

[ULOGD RFC 08/30] NFCT: rework

[heitzenberger]

[-- Attachment #1: ulogd-NFCT-plugin.diff --]
[-- Type: text/plain, Size: 28013 bytes --]

Also implement garbage collection to account for the fact that netlink
messages are sometimes lost (ENOBUFS) on busy sites.

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/input/flow/linux_jhash.h
===================================================================
--- /dev/null
+++ ulogd-netfilter/input/flow/linux_jhash.h
@@ -0,0 +1,146 @@
+#ifndef _LINUX_JHASH_H
+#define _LINUX_JHASH_H
+
+/* jhash.h: Jenkins hash support.
+ *
+ * Copyright (C) 1996 Bob Jenkins (bob_jenkins@burtleburtle.net)
+ *
+ * http://burtleburtle.net/bob/hash/
+ *
+ * These are the credits from Bob's sources:
+ *
+ * lookup2.c, by Bob Jenkins, December 1996, Public Domain.
+ * hash(), hash2(), hash3, and mix() are externally useful functions.
+ * Routines to test the hash are included if SELF_TEST is defined.
+ * You can use this free for any purpose.  It has no warranty.
+ *
+ * Copyright (C) 2003 David S. Miller (davem@redhat.com)
+ *
+ * I've modified Bob's hash to be useful in the Linux kernel, and
+ * any bugs present are surely my fault.  -DaveM
+ */
+
+#define u32		uint32_t
+#define u8		uint8_t
+
+/* NOTE: Arguments are modified. */
+#define __jhash_mix(a, b, c) \
+{ \
+  a -= b; a -= c; a ^= (c>>13); \
+  b -= c; b -= a; b ^= (a<<8); \
+  c -= a; c -= b; c ^= (b>>13); \
+  a -= b; a -= c; a ^= (c>>12);  \
+  b -= c; b -= a; b ^= (a<<16); \
+  c -= a; c -= b; c ^= (b>>5); \
+  a -= b; a -= c; a ^= (c>>3);  \
+  b -= c; b -= a; b ^= (a<<10); \
+  c -= a; c -= b; c ^= (b>>15); \
+}
+
+/* The golden ration: an arbitrary value */
+#define JHASH_GOLDEN_RATIO	0x9e3779b9
+
+/* The most generic version, hashes an arbitrary sequence
+ * of bytes.  No alignment or length assumptions are made about
+ * the input key.
+ */
+static inline u32 jhash(const void *key, u32 length, u32 initval)
+{
+	u32 a, b, c, len;
+	const u8 *k = key;
+
+	len = length;
+	a = b = JHASH_GOLDEN_RATIO;
+	c = initval;
+
+	while (len >= 12) {
+		a += (k[0] +((u32)k[1]<<8) +((u32)k[2]<<16) +((u32)k[3]<<24));
+		b += (k[4] +((u32)k[5]<<8) +((u32)k[6]<<16) +((u32)k[7]<<24));
+		c += (k[8] +((u32)k[9]<<8) +((u32)k[10]<<16)+((u32)k[11]<<24));
+
+		__jhash_mix(a,b,c);
+
+		k += 12;
+		len -= 12;
+	}
+
+	c += length;
+	switch (len) {
+	case 11: c += ((u32)k[10]<<24);
+	case 10: c += ((u32)k[9]<<16);
+	case 9 : c += ((u32)k[8]<<8);
+	case 8 : b += ((u32)k[7]<<24);
+	case 7 : b += ((u32)k[6]<<16);
+	case 6 : b += ((u32)k[5]<<8);
+	case 5 : b += k[4];
+	case 4 : a += ((u32)k[3]<<24);
+	case 3 : a += ((u32)k[2]<<16);
+	case 2 : a += ((u32)k[1]<<8);
+	case 1 : a += k[0];
+	};
+
+	__jhash_mix(a,b,c);
+
+	return c;
+}
+
+/* A special optimized version that handles 1 or more of u32s.
+ * The length parameter here is the number of u32s in the key.
+ */
+static inline u32 jhash2(u32 *k, u32 length, u32 initval)
+{
+	u32 a, b, c, len;
+
+	a = b = JHASH_GOLDEN_RATIO;
+	c = initval;
+	len = length;
+
+	while (len >= 3) {
+		a += k[0];
+		b += k[1];
+		c += k[2];
+		__jhash_mix(a, b, c);
+		k += 3; len -= 3;
+	}
+
+	c += length * 4;
+
+	switch (len) {
+	case 2 : b += k[1];
+	case 1 : a += k[0];
+	};
+
+	__jhash_mix(a,b,c);
+
+	return c;
+}
+
+
+/* A special ultra-optimized versions that knows they are hashing exactly
+ * 3, 2 or 1 word(s).
+ *
+ * NOTE: In partilar the "c += length; __jhash_mix(a,b,c);" normally
+ *       done at the end is not done here.
+ */
+static inline u32 jhash_3words(u32 a, u32 b, u32 c, u32 initval)
+{
+	a += JHASH_GOLDEN_RATIO;
+	b += JHASH_GOLDEN_RATIO;
+	c += initval;
+
+	__jhash_mix(a, b, c);
+
+	return c;
+}
+
+static inline u32 jhash_2words(u32 a, u32 b, u32 initval)
+{
+	return jhash_3words(a, b, 0, initval);
+}
+
+static inline u32 jhash_1word(u32 a, u32 initval)
+{
+	return jhash_3words(a, 0, 0, initval);
+}
+
+#endif /* _LINUX_JHASH_H */
Index: ulogd-netfilter/input/flow/ulogd_inpflow_NFCT.c
===================================================================
--- ulogd-netfilter.orig/input/flow/ulogd_inpflow_NFCT.c
+++ ulogd-netfilter/input/flow/ulogd_inpflow_NFCT.c
@@ -27,6 +27,7 @@
  */
 
 #include <stdlib.h>
+#include <stdbool.h>
 #include <string.h>
 #include <errno.h>
 
@@ -35,38 +36,53 @@
 #include <ulogd/linuxlist.h>
 
 #include <ulogd/ulogd.h>
+#include <ulogd/common.h>
 #include <ulogd/ipfix_protocol.h>
 
 #include <libnetfilter_conntrack/libnetfilter_conntrack.h>
+#include <linux/netfilter/nf_conntrack_tcp.h>
+#include "linux_jhash.h"
+
+#define CT_EVENTS		(NF_NETLINK_CONNTRACK_NEW \
+						 | NF_NETLINK_CONNTRACK_UPDATE \
+						 | NF_NETLINK_CONNTRACK_DESTROY)
+
+#undef INADDR_LOOPBACK
+#define INADDR_LOOPBACK		0x0100007f /* 127.0.0.1 */
+
 
 typedef enum TIMES_ { START, STOP, __TIME_MAX } TIMES;
- 
+typedef unsigned conntrack_hash_t;
+
 struct ct_timestamp {
 	struct llist_head list;
+	struct nfct_tuple tuple;
+	unsigned last_seq;
 	struct timeval time[__TIME_MAX];
-	int id;
 };
 
 struct ct_htable {
 	struct llist_head *buckets;
-	int num_buckets;
-	int prealloc;
-	struct llist_head idle;
-	struct ct_timestamp *ts;
+	unsigned num_buckets;
+	unsigned curr_bucket;
+	unsigned used;
 };
 
 struct nfct_pluginstance {
 	struct nfct_handle *cth;
 	struct ulogd_fd nfct_fd;
+	struct ct_htable *htable;
 	struct ulogd_timer timer;
-	struct ct_htable *ct_active;
+	struct {
+		unsigned nl_err;
+		unsigned nl_ovr;
+	} stats;
 };
 
-#define HTABLE_SIZE	(8192)
-#define MAX_ENTRIES	(4 * HTABLE_SIZE)
+#define HTABLE_SIZE	(512)
 
 static struct config_keyset nfct_kset = {
-	.num_ces = 5,
+	.num_ces = 3,
 	.ces = {
 		{
 			.key	 = "pollinterval",
@@ -75,18 +91,6 @@ static struct config_keyset nfct_kset = 
 			.u.value = 0,
 		},
 		{
-			.key	 = "hash_enable",
-			.type	 = CONFIG_TYPE_INT,
-			.options = CONFIG_OPT_NONE,
-			.u.value = 1,
-		},
-		{
-			.key	 = "hash_prealloc",
-			.type	 = CONFIG_TYPE_INT,
-			.options = CONFIG_OPT_NONE,
-			.u.value = 1,
-		},
-		{
 			.key	 = "hash_buckets",
 			.type	 = CONFIG_TYPE_INT,
 			.options = CONFIG_OPT_NONE,
@@ -96,17 +100,37 @@ static struct config_keyset nfct_kset = 
 			.key	 = "hash_max_entries",
 			.type	 = CONFIG_TYPE_INT,
 			.options = CONFIG_OPT_NONE,
-			.u.value = MAX_ENTRIES,
+			.u.value = 0,
 		},
 	},
 };
 #define pollint_ce(x)	(x->ces[0])
-#define usehash_ce(x)	(x->ces[1])
-#define prealloc_ce(x)	(x->ces[2])
-#define buckets_ce(x)	(x->ces[3])
-#define maxentries_ce(x) (x->ces[4])
+#define buckets_ce(x)	(x->ces[1])
 
-static struct ulogd_key nfct_okeys[] = {
+enum {
+	O_IP_SADDR = 0,
+	O_IP_DADDR,
+	O_IP_PROTO,
+	O_L4_SPORT,
+	O_L4_DPORT,
+	O_RAW_IN_PKTLEN,
+	O_RAW_IN_PKTCOUNT,
+	O_RAW_OUT_PKTLEN,
+	O_RAW_OUT_PKTCOUNT,
+	O_ICMP_CODE,
+	O_ICMP_TYPE,
+	O_CT_MARK,
+	O_CT_ID,
+	O_FLOW_START_SEC,
+	O_FLOW_START_USEC,
+	O_FLOW_START_DAY,
+	O_FLOW_END_SEC,
+	O_FLOW_END_USEC,
+	O_FLOW_DURATION,
+	__O_MAX
+};
+
+static struct ulogd_key nfct_okeys[__O_MAX] = {
 	{
 		.type 	= ULOGD_RET_IPADDR,
 		.flags 	= ULOGD_RETF_NONE,
@@ -155,7 +179,7 @@ static struct ulogd_key nfct_okeys[] = {
 	{
 		.type	= ULOGD_RET_UINT32,
 		.flags	= ULOGD_RETF_NONE,
-		.name	= "raw.pktlen",
+		.name	= "raw.in.pktlen",
 		.ipfix	= { 
 			.vendor 	= IPFIX_VENDOR_IETF,
 			.field_id 	= IPFIX_octetTotalCount,
@@ -165,7 +189,27 @@ static struct ulogd_key nfct_okeys[] = {
 	{
 		.type	= ULOGD_RET_UINT32,
 		.flags	= ULOGD_RETF_NONE,
-		.name	= "raw.pktcount",
+		.name	= "raw.in.pktcount",
+		.ipfix	= { 
+			.vendor 	= IPFIX_VENDOR_IETF,
+			.field_id 	= IPFIX_packetTotalCount,
+			/* FIXME: this could also be packetDeltaCount */
+		},
+	},
+	{
+		.type	= ULOGD_RET_UINT32,
+		.flags	= ULOGD_RETF_NONE,
+		.name	= "raw.out.pktlen",
+		.ipfix	= { 
+			.vendor 	= IPFIX_VENDOR_IETF,
+			.field_id 	= IPFIX_octetTotalCount,
+			/* FIXME: this could also be octetDeltaCount */
+		},
+	},
+	{
+		.type	= ULOGD_RET_UINT32,
+		.flags	= ULOGD_RETF_NONE,
+		.name	= "raw.out.pktcount",
 		.ipfix	= { 
 			.vendor 	= IPFIX_VENDOR_IETF,
 			.field_id 	= IPFIX_packetTotalCount,
@@ -227,6 +271,11 @@ static struct ulogd_key nfct_okeys[] = {
 		},
 	},
 	{
+		.type 	= ULOGD_RET_UINT16,
+		.flags 	= ULOGD_RETF_NONE,
+		.name	= "flow.start.day",
+	},
+	{
 		.type	= ULOGD_RET_UINT32,
 		.flags	= ULOGD_RETF_NONE,
 		.name	= "flow.end.sec",
@@ -245,326 +294,520 @@ static struct ulogd_key nfct_okeys[] = {
 		},
 	},
 	{
-		.type = ULOGD_RET_BOOL,
+		.type = ULOGD_RET_UINT32,
 		.flags = ULOGD_RETF_NONE,
-		.name = "dir",
+		.name = "flow.duration",
 	},
 };
 
-static struct ct_htable *htable_alloc(int htable_size, int prealloc)
+
+/* forward declarations */
+static struct ct_timestamp * ct_hash_find_seq(const struct ct_htable *,
+											  unsigned);
+static void ct_hash_free(struct ct_htable *, struct ct_timestamp *);
+
+
+static int
+nl_error(struct ulogd_pluginstance *pi, struct nlmsghdr *nlh, int *err)
+{
+	struct nfct_pluginstance *priv = (void *)pi->private;
+	struct nlmsgerr *e = NLMSG_DATA(nlh);
+	struct ct_timestamp *ts;
+
+	ts = ct_hash_find_seq(priv->htable, e->msg.nlmsg_seq);
+	if (ts == NULL)
+		return 0;						/* already gone */
+
+	switch (-e->error) {
+	case ENOENT:
+		/* destroy message was lost (FIXME log all what we got) */
+		ct_hash_free(priv->htable, ts);
+		break;
+
+	case 0:								/* "Success" */
+		break;
+
+	default:
+		ulogd_log(ULOGD_ERROR, "netlink error: %s (seq %u)\n",
+				  strerror(-e->error), e->msg.nlmsg_seq);
+		break;
+	}
+
+	*err = -e->error;
+
+	return 0;
+}
+
+
+/* this should go into its own file */
+static int
+nfnl_recv_msgs(struct nfnl_handle *nfnlh,
+			   int (* cb)(struct nlmsghdr *, void *arg), void *arg)
+{
+	static unsigned char buf[NFNL_BUFFSIZE];
+	struct ulogd_pluginstance *pi = arg;
+	struct nfct_pluginstance *priv = (void *)pi->private;
+
+	for (;;) {
+		struct nlmsghdr *nlh = (void *)buf;
+		ssize_t nread;
+
+		nread = nfnl_recv(nfct_nfnlh(priv->cth), buf, sizeof(buf));
+		if (nread < 0) {
+			if (errno == EWOULDBLOCK)
+				break;
+
+			return -1;
+		}
+
+		while (NLMSG_OK(nlh, nread)) {
+			int err = 0;
+
+			if (nlh->nlmsg_type == NLMSG_ERROR) {
+				if (nl_error(pi, nlh, &err) == 0 && err != 0)
+					priv->stats.nl_err++;
+
+				break;
+			}
+
+			if (nlh->nlmsg_type == NLMSG_OVERRUN)
+				priv->stats.nl_ovr++;	/* continue?  payload? */
+				
+			(cb)(nlh, pi);
+
+			nlh = NLMSG_NEXT(nlh, nread);
+		}
+	}
+
+	return 0;
+}
+
+
+static int
+nfct_msg_type(const struct nlmsghdr *nlh)
+{
+	uint16_t type = NFNL_MSG_TYPE(nlh->nlmsg_type);
+	int nfct_type;
+
+	if (type == IPCTNL_MSG_CT_NEW) {
+		if (nlh->nlmsg_flags & (NLM_F_CREATE | NLM_F_EXCL))
+			nfct_type = NFCT_MSG_NEW;
+		else
+			nfct_type = NFCT_MSG_UPDATE;
+	} else if (type == IPCTNL_MSG_CT_DELETE)
+		nfct_type = NFCT_MSG_DESTROY;
+	else
+		nfct_type = NFCT_MSG_UNKNOWN;
+
+	return nfct_type;
+}
+
+
+/* seq: sequence number used for the request */
+static int
+nfct_get_conntrack_x(struct nfct_handle *cth, struct nfct_tuple *t,
+					 int dir, uint32_t *seq)
+{
+	static char buf[NFNL_BUFFSIZE];
+	struct nfnlhdr *req = (void *)buf;
+	int cta_dir, nbytes;
+
+	memset(buf, 0, sizeof(buf));
+
+	/* intendedly do not set NLM_F_ACK in order to skip the
+	   ACK message (but NACKs are still send) */
+	nfnl_fill_hdr(nfct_subsys_ct(cth), &req->nlh, 0, t->l3protonum,
+				  0, IPCTNL_MSG_CT_GET, NLM_F_REQUEST);
+
+	if (seq != NULL)
+		*seq = req->nlh.nlmsg_seq;
+
+	cta_dir = (dir == NFCT_DIR_ORIGINAL) ? CTA_TUPLE_ORIG : CTA_TUPLE_REPLY;
+
+	nfct_build_tuple(req, sizeof(buf), t, cta_dir);
+
+	nbytes = nfnl_send(nfct_nfnlh(cth), &req->nlh);
+
+	return nbytes;
+}
+
+
+static conntrack_hash_t
+hash_conntrack(const struct nfct_tuple *t, size_t hash_sz)
+{
+	static unsigned rnd;
+
+	if (rnd == 0U)
+		rnd = rand();
+
+	return jhash_3words(t->src.v4, t->dst.v4 ^ t->protonum,
+						t->l4src.all | (t->l4dst.all << 16), rnd) % hash_sz;
+}
+
+static inline bool
+ct_cmp(const struct nfct_tuple *t1, const struct nfct_tuple *t2)
+{
+	return memcmp(t1, t2, sizeof(struct nfct_tuple)) == 0;
+}
+
+
+static struct ct_htable *
+htable_alloc(int htable_size)
 {
 	struct ct_htable *htable;
-	struct ct_timestamp *ct;
 	int i;
 
 	htable = malloc(sizeof(*htable)
-			+ sizeof(struct llist_head)*htable_size);
+			+ sizeof(struct llist_head) * htable_size);
 	if (!htable)
 		return NULL;
 
 	htable->buckets = (void *)htable + sizeof(*htable);
 	htable->num_buckets = htable_size;
-	htable->prealloc = prealloc;
-	INIT_LLIST_HEAD(&htable->idle);
+	htable->used = 0;
 
 	for (i = 0; i < htable->num_buckets; i++)
 		INIT_LLIST_HEAD(&htable->buckets[i]);
 	
-	if (!htable->prealloc)
-		return htable;
-
-	ct = malloc(sizeof(struct ct_timestamp)
-		    * htable->num_buckets * htable->prealloc);
-	if (!ct) {
-		free(htable);
-		return NULL;
-	}
-
-	/* save the pointer for later free()ing */
-	htable->ts = ct;
-
-	for (i = 0; i < htable->num_buckets * htable->prealloc; i++)
-		llist_add(&ct[i].list, &htable->idle);
-
 	return htable;
 }
 
-static void htable_free(struct ct_htable *htable)
+static void
+htable_free(struct ct_htable *htable)
 {
 	struct llist_head *ptr, *ptr2;
 	int i;
 
-	if (htable->prealloc) {
-		/* the easy case */
-		free(htable->ts);
-		free(htable);
-
-		return;
-	}
-
-	/* non-prealloc case */
-
 	for (i = 0; i < htable->num_buckets; i++) {
 		llist_for_each_safe(ptr, ptr2, &htable->buckets[i])
 			free(container_of(ptr, struct ct_timestamp, list));
 	}
 
-	/* don't need to check for 'idle' list, since it is only used in
-	 * the preallocated case */
+	free(htable);
 }
 
-static int ct_hash_add(struct ct_htable *htable, unsigned int id)
+static struct ct_timestamp *
+ct_hash_add(struct ct_htable *htable, const struct nfct_tuple *t)
 {
-	struct ct_timestamp *ct;
+	struct ct_timestamp *ts;
+	conntrack_hash_t h;
 
-	if (htable->prealloc) {
-		if (llist_empty(&htable->idle)) {
-			ulogd_log(ULOGD_ERROR, "Not enough ct_timestamp entries\n");
-			return -1;
-		}
+	h = hash_conntrack(t, htable->num_buckets);
 
-		ct = container_of(htable->idle.next, struct ct_timestamp, list);
+	if ((ts = calloc(1, sizeof(struct ct_timestamp))) == NULL) {
+		ulogd_log(ULOGD_ERROR, "Out of memory\n");
+		return NULL;
+	}
 
-		ct->id = id;
-		gettimeofday(&ct->time[START], NULL);
+	memcpy(&ts->tuple, t, sizeof(struct nfct_tuple));
 
-		llist_move(&ct->list, &htable->buckets[id % htable->num_buckets]);
-	} else {
-		ct = malloc(sizeof *ct);
-		if (!ct) {
-			ulogd_log(ULOGD_ERROR, "Not enough memory\n");
-			return -1;
-		}
+	llist_add(&ts->list, &htable->buckets[h]);
+	htable->used++;
 
-		ct->id = id;
-		gettimeofday(&ct->time[START], NULL);
+	return ts;
+}
+
+static struct ct_timestamp *
+ct_hash_find(struct ct_htable *htable, const struct nfct_tuple *t)
+{  
+	struct llist_head *ptr;
+	conntrack_hash_t h = hash_conntrack(t, htable->num_buckets);
 
-		llist_add(&ct->list, &htable->buckets[id % htable->num_buckets]);
+	llist_for_each(ptr, &htable->buckets[h]) {
+		struct ct_timestamp *ts = container_of(ptr, struct ct_timestamp, list);
+
+		if (ct_cmp(t, &ts->tuple))
+			return ts;
 	}
 
-	return 0;
+	return NULL;
 }
 
-static struct ct_timestamp *ct_hash_get(struct ct_htable *htable, uint32_t id)
+
+static struct ct_timestamp *
+ct_hash_find_seq(const struct ct_htable *htable, unsigned seq)
 {
-	struct ct_timestamp *ct = NULL;
-	struct llist_head *ptr;
+	int i;
 
-	llist_for_each(ptr, &htable->buckets[id % htable->num_buckets]) {
-		ct = container_of(ptr, struct ct_timestamp, list);
-		if (ct->id == id) {
-			gettimeofday(&ct->time[STOP], NULL);
-			if (htable->prealloc)
-				llist_move(&ct->list, &htable->idle);
-			else
-				free(ct);
-			break;
+	for (i = 0; i < htable->num_buckets; i++) {
+		struct ct_timestamp *ts;
+
+		llist_for_each_entry(ts, &htable->buckets[i], list) {
+			if (ts->last_seq == seq)
+				return ts;
 		}
 	}
-	return ct;
+
+	return NULL;
+}
+
+
+/* time diff with second resolution */
+static inline unsigned
+tv_diff_sec(const struct ct_timestamp *ts)
+{
+	if (ts->time[STOP].tv_sec >= ts->time[START].tv_sec)
+		return max(ts->time[STOP].tv_sec - ts->time[START].tv_sec, 1);
+
+	return ts->time[START].tv_sec - ts->time[STOP].tv_sec;
 }
 
-static int propagate_ct_flow(struct ulogd_pluginstance *upi, 
-		             struct nfct_conntrack *ct,
-			     unsigned int flags,
-			     int dir,
-			     struct ct_timestamp *ts)
+static void
+ct_hash_free(struct ct_htable *htable, struct ct_timestamp *ts)
+{
+	llist_del(&ts->list);
+
+	free(ts);
+	htable->used--;
+}
+
+static int
+propagate_ct_flow(struct ulogd_pluginstance *upi, 
+				  struct nfct_conntrack *ct, unsigned int flags,
+				  int dir, struct ct_timestamp *ts)
 {
 	struct ulogd_key *ret = upi->output.keys;
 
-	ret[0].u.value.ui32 = htonl(ct->tuple[dir].src.v4);
-	ret[0].flags |= ULOGD_RETF_VALID;
+	ret[O_IP_SADDR].u.value.ui32 = htonl(ct->tuple[dir].src.v4);
+	ret[O_IP_SADDR].flags |= ULOGD_RETF_VALID;
 
-	ret[1].u.value.ui32 = htonl(ct->tuple[dir].dst.v4);
-	ret[1].flags |= ULOGD_RETF_VALID;
+	ret[O_IP_DADDR].u.value.ui32 = htonl(ct->tuple[dir].dst.v4);
+	ret[O_IP_DADDR].flags |= ULOGD_RETF_VALID;
 
-	ret[2].u.value.ui8 = ct->tuple[dir].protonum;
-	ret[2].flags |= ULOGD_RETF_VALID;
+	ret[O_IP_PROTO].u.value.ui8 = ct->tuple[dir].protonum;
+	ret[O_IP_PROTO].flags |= ULOGD_RETF_VALID;
 
-	switch (ct->tuple[1].protonum) {
+	switch (ct->tuple[dir].protonum) {
 	case IPPROTO_TCP:
 	case IPPROTO_UDP:
 	case IPPROTO_SCTP:
 		/* FIXME: DCCP */
-		ret[3].u.value.ui16 = htons(ct->tuple[dir].l4src.tcp.port);
-		ret[3].flags |= ULOGD_RETF_VALID;
-		ret[4].u.value.ui16 = htons(ct->tuple[dir].l4dst.tcp.port);
-		ret[4].flags |= ULOGD_RETF_VALID;
+		ret[O_L4_SPORT].u.value.ui16 = htons(ct->tuple[dir].l4src.tcp.port);
+		ret[O_L4_SPORT].flags |= ULOGD_RETF_VALID;
+		ret[O_L4_DPORT].u.value.ui16 = htons(ct->tuple[dir].l4dst.tcp.port);
+		ret[O_L4_DPORT].flags |= ULOGD_RETF_VALID;
 		break;
 	case IPPROTO_ICMP:
-		ret[7].u.value.ui8 = ct->tuple[dir].l4src.icmp.code;
-		ret[7].flags |= ULOGD_RETF_VALID;
-		ret[8].u.value.ui8 = ct->tuple[dir].l4src.icmp.type;
-		ret[8].flags |= ULOGD_RETF_VALID;
+		ret[O_ICMP_CODE].u.value.ui8 = ct->tuple[dir].l4src.icmp.code;
+		ret[O_ICMP_CODE].flags |= ULOGD_RETF_VALID;
+		ret[O_ICMP_TYPE].u.value.ui8 = ct->tuple[dir].l4src.icmp.type;
+		ret[O_ICMP_TYPE].flags |= ULOGD_RETF_VALID;
 		break;
 	}
 
-	if ((dir == NFCT_DIR_ORIGINAL && flags & NFCT_COUNTERS_ORIG) ||
-	    (dir == NFCT_DIR_REPLY && flags & NFCT_COUNTERS_RPLY)) {
-		ret[5].u.value.ui64 = ct->counters[dir].bytes;
-		ret[5].flags |= ULOGD_RETF_VALID;
-
-		ret[6].u.value.ui64 = ct->counters[dir].packets;
-		ret[6].flags |= ULOGD_RETF_VALID;
+	if (flags & NFCT_COUNTERS_ORIG) {
+		ret[O_RAW_IN_PKTLEN].u.value.ui32 = ct->counters[0].bytes;
+		ret[O_RAW_IN_PKTLEN].flags |= ULOGD_RETF_VALID;
+		ret[O_RAW_IN_PKTCOUNT].u.value.ui32 = ct->counters[0].packets;
+		ret[O_RAW_IN_PKTCOUNT].flags |= ULOGD_RETF_VALID;
+
+		ret[O_RAW_OUT_PKTLEN].u.value.ui32 = ct->counters[1].bytes;
+		ret[O_RAW_OUT_PKTLEN].flags |= ULOGD_RETF_VALID;
+		ret[O_RAW_OUT_PKTCOUNT].u.value.ui32 = ct->counters[1].packets;
+		ret[O_RAW_OUT_PKTCOUNT].flags |= ULOGD_RETF_VALID;
 	}
 
 	if (flags & NFCT_MARK) {
-		ret[9].u.value.ui32 = ct->mark;
-		ret[9].flags |= ULOGD_RETF_VALID;
+		ret[O_CT_MARK].u.value.ui32 = ct->mark;
+		ret[O_CT_MARK].flags |= ULOGD_RETF_VALID;
 	}
 
 	if (flags & NFCT_ID) {
-		ret[10].u.value.ui32 = ct->id;
-		ret[10].flags |= ULOGD_RETF_VALID;
+		ret[O_CT_ID].u.value.ui32 = ct->id;
+		ret[O_CT_ID].flags |= ULOGD_RETF_VALID;
 	}
 
-	if (ts) {
-		ret[11].u.value.ui32 = ts->time[START].tv_sec;
-		ret[11].flags |= ULOGD_RETF_VALID;
-		ret[12].u.value.ui32 = ts->time[START].tv_usec;
-		ret[12].flags |= ULOGD_RETF_VALID;
-		ret[13].u.value.ui32 = ts->time[STOP].tv_sec;
-		ret[13].flags |= ULOGD_RETF_VALID;
-		ret[14].u.value.ui32 = ts->time[STOP].tv_usec;
-		ret[14].flags |= ULOGD_RETF_VALID;
-	}
+	ret[O_FLOW_START_SEC].u.value.ui32 = ts->time[START].tv_sec;
+	ret[O_FLOW_START_SEC].flags |= ULOGD_RETF_VALID;
+	ret[O_FLOW_START_USEC].u.value.ui32 = ts->time[START].tv_usec;
+	ret[O_FLOW_START_USEC].flags |= ULOGD_RETF_VALID;
+	ret[O_FLOW_START_DAY].u.value.ui16 = ts->time[START].tv_sec / (1 DAY);
+	ret[O_FLOW_START_DAY].flags |= ULOGD_RETF_VALID;
+
+	ret[O_FLOW_END_SEC].u.value.ui32 = ts->time[STOP].tv_sec;
+	ret[O_FLOW_END_SEC].flags |= ULOGD_RETF_VALID;
+	ret[O_FLOW_END_USEC].u.value.ui32 = ts->time[STOP].tv_usec;
+	ret[O_FLOW_END_USEC].flags |= ULOGD_RETF_VALID;
 
-	ret[15].u.value.b = (dir == NFCT_DIR_ORIGINAL) ? 0 : 1;
-	ret[15].flags |= ULOGD_RETF_VALID;
+	ret[O_FLOW_DURATION].u.value.ui32 = tv_diff_sec(ts);
+	ret[O_FLOW_DURATION].flags |= ULOGD_RETF_VALID;
 
 	ulogd_propagate_results(upi);
 
 	return 0;
 }
 
-static int propagate_ct(struct ulogd_pluginstance *upi,
-			struct nfct_conntrack *ct,
-			unsigned int flags,
-			struct ct_timestamp *ctstamp)
-{
-	int rc;
+static int
+propagate_ct(struct ulogd_pluginstance *upi, struct nfct_conntrack *ct,
+			 struct ct_timestamp *ts, unsigned int flags)
+{
+	struct nfct_pluginstance *priv = (void *)upi->private;
+
+	do {
+		if (ct->tuple[NFCT_DIR_ORIGINAL].src.v4 == INADDR_LOOPBACK
+			|| ct->tuple[NFCT_DIR_ORIGINAL].dst.v4 == INADDR_LOOPBACK)
+			break;
 
-	rc = propagate_ct_flow(upi, ct, flags, NFCT_DIR_ORIGINAL, ctstamp);
-	if (rc < 0)
-		return rc;
+		gettimeofday(&ts->time[STOP], NULL);
+		
+		propagate_ct_flow(upi, ct, flags, NFCT_DIR_ORIGINAL, ts);
+	} while (0);
 
-	return propagate_ct_flow(upi, ct, flags, NFCT_DIR_REPLY, ctstamp);
+	ct_hash_free(priv->htable, ts);
+
+	return 0;
 }
 
-static int event_handler(void *arg, unsigned int flags, int type,
-			 void *data)
+
+static int
+do_nfct_msg(struct nlmsghdr *nlh, void *arg)
 {
-	struct nfct_conntrack *ct = arg;
-	struct ulogd_pluginstance *upi = data;
-	struct nfct_pluginstance *cpi = 
-				(struct nfct_pluginstance *) upi->private;
+	struct ulogd_pluginstance *pi = arg;
+	struct nfct_pluginstance *priv = (void *)pi->private;
+	struct nfgenmsg *nfh = NLMSG_DATA(nlh);
+	struct nfct_conntrack ct;
+	struct ct_timestamp *ts;
+	int flags, type = nfct_msg_type(nlh);
+
+	if (type == NFCT_MSG_UNKNOWN)
+		return 0;
+
+	bzero(&ct, sizeof(ct));
+
+	ct.tuple[NFCT_DIR_ORIGINAL].l3protonum = 
+		ct.tuple[NFCT_DIR_REPLY].l3protonum = nfh->nfgen_family;
 
-	if (type == NFCT_MSG_NEW) {
-		if (usehash_ce(upi->config_kset).u.value != 0)
-			ct_hash_add(cpi->ct_active, ct->id);
-	} else if (type == NFCT_MSG_DESTROY) {
-		struct ct_timestamp *ts = NULL;
+	if (nfct_netlink_to_conntrack(nlh, &ct, &flags) < 0)
+		return -1;
+
+	switch (type) { 
+	case NFCT_MSG_NEW:
+		ts = ct_hash_add(priv->htable, &ct.tuple[NFCT_DIR_ORIGINAL]);
+		if (ts != NULL) {
+			gettimeofday(&ts->time[START], NULL);
+		}
+		break;
 
-		if (usehash_ce(upi->config_kset).u.value != 0)
-			ts = ct_hash_get(cpi->ct_active, ct->id);
+	case NFCT_MSG_UPDATE:
+		ts = ct_hash_find(priv->htable, &ct.tuple[NFCT_DIR_ORIGINAL]);
+		if (ts == NULL) {
+			/* do not add CT to cache, as there would be no start
+			   information */
+			break;
+		}
 
-		return propagate_ct(upi, ct, flags, ts);
+		/* handle TCP connections differently in order not to bloat CT
+		   hash with many TIME_WAIT connections */
+		if (ct.tuple[NFCT_DIR_ORIGINAL].protonum == IPPROTO_TCP) {
+			if (ct.protoinfo.tcp.state == TCP_CONNTRACK_TIME_WAIT)
+				return propagate_ct(pi, &ct, ts, flags);
+		}
+		break;
+		
+	case NFCT_MSG_DESTROY:
+		ts = ct_hash_find(priv->htable, &ct.tuple[NFCT_DIR_ORIGINAL]);
+		if (ts != NULL)
+			return propagate_ct(pi, &ct, ts, flags);
+		break;
+		
+	default:
+		break;
 	}
+
 	return 0;
 }
 
-static int read_cb_nfct(int fd, unsigned int what, void *param)
+
+static int
+read_cb_nfct(int fd, unsigned what, void *param)
 {
-	struct nfct_pluginstance *cpi = (struct nfct_pluginstance *) param;
+	struct ulogd_pluginstance *pi = param;
+	struct nfct_pluginstance *priv = (void *)pi->private;
 
 	if (!(what & ULOGD_FD_READ))
 		return 0;
 
-	/* FIXME: implement this */
-	nfct_event_conntrack(cpi->cth);
-	return 0;
+	return nfnl_recv_msgs(nfct_nfnlh(priv->cth), do_nfct_msg, pi);
 }
 
-static int get_ctr_zero(struct ulogd_pluginstance *upi)
-{
-	struct nfct_pluginstance *cpi = 
-			(struct nfct_pluginstance *)upi->private;
 
-	return nfct_dump_conntrack_table_reset_counters(cpi->cth, AF_INET);
-}
+/* choosing powers of two for all values helps here */
+#define STOP_HERE(h)	(((h)->curr_bucket + 16) % (h)->num_buckets)
 
-static void getctr_timer_cb(void *data)
+static void
+timer_cb(struct ulogd_timer *t)
 {
-	struct ulogd_pluginstance *upi = data;
+	struct ulogd_pluginstance *pi = t->data;
+	struct nfct_pluginstance *priv = (void *)pi->private;
+	struct ct_htable *ht = priv->htable;
+	struct ct_timestamp *ts;
+	int end = STOP_HERE(ht);
+
+	do {
+		assert(ht->curr_bucket < ht->num_buckets);
 
-	get_ctr_zero(upi);
+		llist_for_each_entry(ts, &ht->buckets[ht->curr_bucket], list) {
+			/* check if its still there */
+			nfct_get_conntrack_x(priv->cth, &ts->tuple, NFCT_DIR_ORIGINAL,
+								 &ts->last_seq);
+		}
+
+		ht->curr_bucket = (ht->curr_bucket + 1) % ht->num_buckets;
+	} while (ht->curr_bucket != end);
 }
 
 static int configure_nfct(struct ulogd_pluginstance *upi,
 			  struct ulogd_pluginstance_stack *stack)
 {
-	struct nfct_pluginstance *cpi = 
-			(struct nfct_pluginstance *)upi->private;
+	struct nfct_pluginstance *priv = (void *)upi->private;
 	int ret;
+
+	memset(priv, 0, sizeof(struct nfct_pluginstance));
 	
 	ret = config_parse_file(upi->id, upi->config_kset);
 	if (ret < 0)
 		return ret;
-	
-	/* initialize getctrzero timer structure */
-	memset(&cpi->timer, 0, sizeof(cpi->timer));
-	cpi->timer.cb = &getctr_timer_cb;
-	cpi->timer.data = cpi;
-
-	if (pollint_ce(upi->config_kset).u.value != 0) {
-		cpi->timer.expires.tv_sec = 
-			pollint_ce(upi->config_kset).u.value;
-		ulogd_register_timer(&cpi->timer);
-	}
 
 	return 0;
 }
 
+
 static int constructor_nfct(struct ulogd_pluginstance *upi)
 {
-	struct nfct_pluginstance *cpi = 
-			(struct nfct_pluginstance *)upi->private;
-	int prealloc;
-
-	memset(cpi, 0, sizeof(*cpi));
-
-	/* FIXME: make eventmask configurable */
-	cpi->cth = nfct_open(NFNL_SUBSYS_CTNETLINK, NF_NETLINK_CONNTRACK_NEW|
-			     NF_NETLINK_CONNTRACK_DESTROY);
+	struct nfct_pluginstance *cpi = (void *)upi->private;
+
+	cpi->cth = nfct_open(NFNL_SUBSYS_CTNETLINK, CT_EVENTS);
 	if (!cpi->cth) {
 		ulogd_log(ULOGD_FATAL, "error opening ctnetlink\n");
 		return -1;
 	}
 
-	nfct_register_callback(cpi->cth, &event_handler, upi);
-
 	cpi->nfct_fd.fd = nfct_fd(cpi->cth);
 	cpi->nfct_fd.cb = &read_cb_nfct;
-	cpi->nfct_fd.data = cpi;
+	cpi->nfct_fd.data = upi;
 	cpi->nfct_fd.when = ULOGD_FD_READ;
 
 	ulogd_register_fd(&cpi->nfct_fd);
 
-	if (prealloc_ce(upi->config_kset).u.value != 0)
-		prealloc = maxentries_ce(upi->config_kset).u.value / 
-				buckets_ce(upi->config_kset).u.value;
-	else
-		prealloc = 0;
+	cpi->htable = htable_alloc(buckets_ce(upi->config_kset).u.value);
+	if (cpi->htable == NULL) {
+		ulogd_log(ULOGD_FATAL, "htable_alloc: out of memory\n");
 
-	if (usehash_ce(upi->config_kset).u.value != 0) {
-		cpi->ct_active = htable_alloc(buckets_ce(upi->config_kset).u.value,
-					      prealloc);
-		if (!cpi->ct_active) {
-			ulogd_log(ULOGD_FATAL, "error allocating hash\n");
-			nfct_close(cpi->cth);
-			return -1;
-		}
+		nfct_close(cpi->cth);
+		cpi->cth = NULL;
+
+		return -1;
 	}
+
+	cpi->timer.cb = timer_cb;
+	cpi->timer.ival = 1 SEC;
+	cpi->timer.flags = TIMER_F_PERIODIC;
+	cpi->timer.data = upi;
+
+	ulogd_register_timer(&cpi->timer);
+
+	ulogd_log(ULOGD_INFO, "%s: hashsize %u\n", upi->id,
+			  cpi->htable->num_buckets);
 	
 	return 0;
 }
@@ -572,13 +815,11 @@ static int constructor_nfct(struct ulogd
 static int destructor_nfct(struct ulogd_pluginstance *pi)
 {
 	struct nfct_pluginstance *cpi = (void *) pi;
-	int rc;
 	
-	htable_free(cpi->ct_active);
+	nfct_close(cpi->cth);
+	cpi->cth = NULL;
 
-	rc = nfct_close(cpi->cth);
-	if (rc < 0)
-		return rc;
+	htable_free(cpi->htable);
 
 	return 0;
 }
@@ -587,7 +828,6 @@ static void signal_nfct(struct ulogd_plu
 {
 	switch (signal) {
 	case SIGUSR2:
-		get_ctr_zero(pi);
 		break;
 	}
 }

-- 

Re: [ULOGD RFC 08/30] NFCT: rework

[Pablo Neira Ayuso]

heitzenberger@astaro.com wrote:
> Also implement garbage collection to account for the fact that netlink
> messages are sometimes lost (ENOBUFS) on busy sites.

Well, this NFCT stuff is a big hack. I'm sure that you have solved
several issues with those patches but sorry, I don't like them at all.
There's several parts of conntrackd that we could use to rework this eg.
an implementation of a generic hash table in conntrackd that we can
reuse and nfnl_recv_msgs looks very similar to nfnl_catch, etc. Also,
the ENOBUFS handling is still a mess. I'm better keeping back these NFCT
patches. I'll try to give some spins to it.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: [ULOGD RFC 08/30] NFCT: rework

[Holger Eitzenberger]

Pablo Neira Ayuso <pablo@netfilter.org> writes:

> Well, this NFCT stuff is a big hack. I'm sure that you have solved
> several issues with those patches but sorry, I don't like them at all.
> There's several parts of conntrackd that we could use to rework this eg.
> an implementation of a generic hash table in conntrackd that we can
> reuse and nfnl_recv_msgs looks very similar to nfnl_catch, etc. Also,
> the ENOBUFS handling is still a mess. I'm better keeping back these NFCT
> patches. I'll try to give some spins to it.

Hi Pablo,

are you talking about this particular patch or the end result after
applying all of the NFCT patches?

Also note that IMO this particular patch turned a non-working NFCT
solution into some better-working solution.  I just suggest to
comment on the final NFCT code and not on this one.  Thanks.

  /holger

Re: [ULOGD RFC 08/30] NFCT: rework

[Patrick McHardy]

Holger Eitzenberger wrote:
> Pablo Neira Ayuso <pablo@netfilter.org> writes:
> 
>> Well, this NFCT stuff is a big hack. I'm sure that you have solved
>> several issues with those patches but sorry, I don't like them at all.
>> There's several parts of conntrackd that we could use to rework this eg.
>> an implementation of a generic hash table in conntrackd that we can
>> reuse and nfnl_recv_msgs looks very similar to nfnl_catch, etc. Also,
>> the ENOBUFS handling is still a mess. I'm better keeping back these NFCT
>> patches. I'll try to give some spins to it.
> 
> Hi Pablo,
> 
> are you talking about this particular patch or the end result after
> applying all of the NFCT patches?
> 
> Also note that IMO this particular patch turned a non-working NFCT
> solution into some better-working solution.  I just suggest to
> comment on the final NFCT code and not on this one.  Thanks.


I agree, this code has bitrotten for too long, I'm happy about
any progress. Any objections should be clearly stated please.

Re: [ULOGD RFC 08/30] NFCT: rework

[Pablo Neira Ayuso]

Patrick McHardy wrote:
> Holger Eitzenberger wrote:
>> are you talking about this particular patch or the end result after
>> applying all of the NFCT patches?
>>
>> Also note that IMO this particular patch turned a non-working NFCT
>> solution into some better-working solution.  I just suggest to
>> comment on the final NFCT code and not on this one.  Thanks.
> 
> I agree, this code has bitrotten for too long, I'm happy about
> any progress. Any objections should be clearly stated please.

I know, this code need some spins but there are several stuff that it's
better if we fix it now than later:

* CT_EVENTS is a duplicated flags, already exists NFCT_ALL_CT_GROUPS
* This patch arbitrarily disables loopback logging, this must be an option
* Default hashtable size reduced to 512, why?
* nfnl_recv_msgs looks like a duplicated of nfnl_catch (or nfct_catch).
nfct_msg_type is not required if you use the libnetfilter_conntrack API
properly. do_nfct_msg should use the libnetfilter_conntrack framework
which would simplify it.
* This patch checks if every conntrack exists in the kernel every N
seconds to handle overruns. Instead, why doesn't it wait for ENOBUFS in
the recv buffer and, then try to resync to kernel?
* Where is the NLMSG_OVERRUN flag used in the netlink code?
* ct_hash_find_seq is O(n). Overruns sometimes happen because the CPU
reaches 100% consumption, so if it can't backoff, this function won't
help that much in those cases.

I like that this patch kills the preallocation code which made this
thing more complex. Unfortunately it is mixed with many other things.
Still the netlink handling need some spins IMO.

An observation, the asynchronous nature of the ulogd timers keep this
hard since the timer callback can be called while accessing whatever
section of code. I think that the way to go is to use select and
implement time-slicing.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: [ULOGD RFC 08/30] NFCT: rework

[Holger Eitzenberger]

Pablo Neira Ayuso wrote:

> * CT_EVENTS is a duplicated flags, already exists NFCT_ALL_CT_GROUPS

I've already queued a patch for that, I'll send that later.

> * This patch arbitrarily disables loopback logging, this must be an option

I totally agree on that.  My plan is to provide a filtering capability 
as generic as possible later on.

> * Default hashtable size reduced to 512, why?

You are still talking about the ulogd-NFCT-plugin.diff, right?  Please 
comment on the version as it is at the end of the patchset.

> * This patch checks if every conntrack exists in the kernel every N
> seconds to handle overruns. Instead, why doesn't it wait for ENOBUFS in
> the recv buffer and, then try to resync to kernel?

This is one of the future improvements I've only queued locally.  As 
this isn't critical I suggest to wait for that.

> * Where is the NLMSG_OVERRUN flag used in the netlink code?

When the point above is implemented.

> * ct_hash_find_seq is O(n). Overruns sometimes happen because the CPU
> reaches 100% consumption, so if it can't backoff, this function won't
> help that much in those cases.

  [ULOGD RFC 15/30] NFCT: add sequence cache

That patch was provided exactly to solve that issue.

> An observation, the asynchronous nature of the ulogd timers keep this
> hard since the timer callback can be called while accessing whatever
> section of code. I think that the way to go is to use select and
> implement time-slicing.

Either I didn't understand your point or you totally missed what the 
'synchronous signal handlers' are all about.

  /holger

Re: [ULOGD RFC 08/30] NFCT: rework

[Pablo Neira Ayuso]

Holger Eitzenberger wrote:
> Pablo Neira Ayuso wrote:
>> * Default hashtable size reduced to 512, why?
> 
> You are still talking about the ulogd-NFCT-plugin.diff, right?  Please
> comment on the version as it is at the end of the patchset.

Sorry, I don't understand your patchset logic since I have to apply them
all to understand what you want to do, this is confusing.

>> * This patch checks if every conntrack exists in the kernel every N
>> seconds to handle overruns. Instead, why doesn't it wait for ENOBUFS in
>> the recv buffer and, then try to resync to kernel?
> 
> This is one of the future improvements I've only queued locally.  As
> this isn't critical I suggest to wait for that.

The point is that I don't understand why we have to apply these NFCT
patches which IMO do a sloppy netlink handling and then wait until this
is completely rewritten again properly... (continue below)

>> * ct_hash_find_seq is O(n). Overruns sometimes happen because the CPU
>> reaches 100% consumption, so if it can't backoff, this function won't
>> help that much in those cases.
> 
>  [ULOGD RFC 15/30] NFCT: add sequence cache
> 
> That patch was provided exactly to solve that issue.

... because AFAICS if we check for ENOBUFS and then resync against the
kernel table using GET_CONNTRACK we won't need the sequence cache later,
will we?

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: [ULOGD RFC 08/30] NFCT: rework

[Holger Eitzenberger]

Pablo Neira Ayuso wrote:

> The point is that I don't understand why we have to apply these NFCT
> patches which IMO do a sloppy netlink handling and then wait until this
> is completely rewritten again properly... (continue below)

Well, those patches were done over time of about a year or so.  The 
initial NFCT code wasn't working for me when I initially tried it, so I 
started to hack on that myself.  Only the later patches turn it into the 
  well-working solution you should use.

Don't forget that this is a patchset with an RFC in mind.  With that 
series of patches you clearly see how NFCT evolved over time, with the 
possibility to get the rational of all those patches.

Another option would be to just provide the newest code, without a 
history at all.  Is that what you like more?

>> That patch was provided exactly to solve that issue.
> 
> ... because AFAICS if we check for ENOBUFS and then resync against the
> kernel table using GET_CONNTRACK we won't need the sequence cache later,
> will we?

You will on a busy site.  The sole purpose of checking ENOBUFS or 
NETLINK_OVERRUN is to check whether you need that logic during normal 
operation.  So, on a not-so-busy site you won't have those GET_CONNTRACK 
requests at all.

Of course the GET_CONNTRACK requests should be disabled if there wasn't 
any overrun for a certain period of time (e. g. one turnaround).

But still that's only an optimisation.

  /holger

Re: [ULOGD RFC 08/30] NFCT: rework

[Pablo Neira Ayuso]

Holger Eitzenberger wrote:
> Another option would be to just provide the newest code, without a
> history at all.  Is that what you like more?

No, you don't get my point. I don't doubt on the worthiness of your
contributions, I'm only stating that you don't do it right. Please,
rework your contribution with my suggestions and try again.

>>> That patch was provided exactly to solve that issue.
>>
>> ... because AFAICS if we check for ENOBUFS and then resync against the
>> kernel table using GET_CONNTRACK we won't need the sequence cache later,
>> will we?
> 
> You will on a busy site.  The sole purpose of checking ENOBUFS or
> NETLINK_OVERRUN is to check whether you need that logic during normal
> operation.  So, on a not-so-busy site you won't have those GET_CONNTRACK
> requests at all.
> 
> Of course the GET_CONNTRACK requests should be disabled if there wasn't
> any overrun for a certain period of time (e. g. one turnaround).
> 
> But still that's only an optimisation.

Sorry, I don't understand what you mean here.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: [ULOGD RFC 08/30] NFCT: rework

[Patrick McHardy]

Pablo Neira Ayuso wrote:
> heitzenberger@astaro.com wrote:
>> Also implement garbage collection to account for the fact that netlink
>> messages are sometimes lost (ENOBUFS) on busy sites.
> 
> Well, this NFCT stuff is a big hack. I'm sure that you have solved
> several issues with those patches but sorry, I don't like them at all.
> There's several parts of conntrackd that we could use to rework this eg.
> an implementation of a generic hash table in conntrackd that we can
> reuse and nfnl_recv_msgs looks very similar to nfnl_catch, etc. Also,
> the ENOBUFS handling is still a mess. I'm better keeping back these NFCT
> patches. I'll try to give some spins to it.
> 

What exactly are your objections? From what I can see it simply
improves performance of the existing code and cleans it up a bit.

Re: [ULOGD RFC 08/30] NFCT: rework

[Patrick McHardy]

Patrick McHardy wrote:
> Pablo Neira Ayuso wrote:
>> heitzenberger@astaro.com wrote:
>>> Also implement garbage collection to account for the fact that netlink
>>> messages are sometimes lost (ENOBUFS) on busy sites.
>>
>> Well, this NFCT stuff is a big hack. I'm sure that you have solved
>> several issues with those patches but sorry, I don't like them at all.
>> There's several parts of conntrackd that we could use to rework this eg.
>> an implementation of a generic hash table in conntrackd that we can
>> reuse and nfnl_recv_msgs looks very similar to nfnl_catch, etc. Also,
>> the ENOBUFS handling is still a mess. I'm better keeping back these NFCT
>> patches. I'll try to give some spins to it.
>>
> 
> What exactly are your objections? From what I can see it simply
> improves performance of the existing code and cleans it up a bit.


Generally speaking, conntrackd doesn't matter here in my opinion,
features are either present in the libraries or have to be implemented
in users of the same. Demanding that code uses foreign non-library
code in a execve-fashion is just not reasonable.

[PATCH] Re: [ULOGD RFC 08/30] NFCT: rework

[Eric Leblond]

[-- Attachment #1.1: Type: text/plain, Size: 1733 bytes --]

Hello,

You changelog is not complete as it did not mention important
modification of the output keys.

On Wednesday, 2008 January 30 at 19:58:55 +0100, heitzenberger@astaro.com wrote:
  
> -static struct ulogd_key nfct_okeys[] = {
> +enum {
> +	O_IP_SADDR = 0,
> +	O_IP_DADDR,
> +	O_IP_PROTO,
> +	O_L4_SPORT,
> +	O_L4_DPORT,
> +	O_RAW_IN_PKTLEN,
> +	O_RAW_IN_PKTCOUNT,
> +	O_RAW_OUT_PKTLEN,
> +	O_RAW_OUT_PKTCOUNT,

Accounting is done here for both side of connection ...

> -	rc = propagate_ct_flow(upi, ct, flags, NFCT_DIR_ORIGINAL, ctstamp);
> -	if (rc < 0)
> -		return rc;
> +		gettimeofday(&ts->time[STOP], NULL);
> +		
> +		propagate_ct_flow(upi, ct, flags, NFCT_DIR_ORIGINAL, ts);
> +	} while (0);
>  
> -	return propagate_ct_flow(upi, ct, flags, NFCT_DIR_REPLY, ctstamp);

and you are now just sending one message per connection (suppression of
call to propagate_ct_flow in NFCT_DIR_REPLY way).

I'm totally agree with last part: Sending two messages per connection
(one per tuple) is IMO not correct but Harald may have some good explanation to
provide about this.

But your patch loose really important information as it keep only one
side of the connection. I understand that NFCT_DIR_ORIGINAL is the human
way of seing a connection but loosing information contained in NFCT_DIR_REPLY 
is IMO an error.

I attached to this mail a patch to the NFCT plugin which do not loose
these informations.

I think you could get to your actual set of keys by using a filter in the stack
which could suppress NFCT_DIR_REPLY information but IMO it can not be
considered as the general case.

Anyway, Holger thanks for this patchset.

BR,
-- 
Eric Leblond
INL: http://www.inl.fr/

[-- Attachment #1.2: 0004-Do-not-propagate-one-conntrack-event-via-2-messages.patch --]
[-- Type: text/x-diff, Size: 11416 bytes --]

From 3ffd8ab5f0f2b50764e591d4edef95d9a8a88c84 Mon Sep 17 00:00:00 2001
From: Eric leblond <eric@inl.fr>
Date: Fri, 11 Jan 2008 23:39:46 +0100
Subject: [PATCH] Do not propagate one conntrack event via 2 messages:
 * ulogd2 was propagating through a stack 2 message for one single
 conntrack event
 * Fall back to on message per event
 * Use an enum to improve code readability instead of direct access to
 array via numerical index

Signed-off-by: Eric leblond <eric@inl.fr>
---
 input/flow/ulogd_inpflow_NFCT.c |  236 ++++++++++++++++++++++++++++-----------
 1 files changed, 168 insertions(+), 68 deletions(-)

diff --git a/input/flow/ulogd_inpflow_NFCT.c b/input/flow/ulogd_inpflow_NFCT.c
index d3cd20c..bf6587d 100644
--- a/input/flow/ulogd_inpflow_NFCT.c
+++ b/input/flow/ulogd_inpflow_NFCT.c
@@ -106,11 +106,101 @@ static struct config_keyset nfct_kset = {
 #define buckets_ce(x)	(x->ces[3])
 #define maxentries_ce(x) (x->ces[4])
 
+enum nfct_keys {
+	NFCT_ORIG_IP_SADDR = 0,
+	NFCT_ORIG_IP_DADDR,
+	NFCT_ORIG_IP_PROTOCOL,
+	NFCT_ORIG_L4_SPORT,
+	NFCT_ORIG_L4_DPORT,
+	NFCT_ORIG_RAW_PKTLEN,
+	NFCT_ORIG_RAW_PKTCOUNT,
+	NFCT_REPLY_IP_SADDR,
+	NFCT_REPLY_IP_DADDR,
+	NFCT_REPLY_IP_PROTOCOL,
+	NFCT_REPLY_L4_SPORT,
+	NFCT_REPLY_L4_DPORT,
+	NFCT_REPLY_RAW_PKTLEN,
+	NFCT_REPLY_RAW_PKTCOUNT,
+	NFCT_ICMP_CODE,
+	NFCT_ICMP_TYPE,
+	NFCT_CT_MARK,
+	NFCT_CT_ID,
+	NFCT_FLOW_START_SEC,
+	NFCT_FLOW_START_USEC,
+	NFCT_FLOW_END_SEC,
+	NFCT_FLOW_END_USEC,
+};
+
 static struct ulogd_key nfct_okeys[] = {
 	{
 		.type 	= ULOGD_RET_IPADDR,
 		.flags 	= ULOGD_RETF_NONE,
-		.name	= "ip.saddr",
+		.name	= "orig.ip.saddr",
+		.ipfix	= { 
+			.vendor = IPFIX_VENDOR_IETF,
+			.field_id = IPFIX_sourceIPv4Address,
+		},
+	},
+	{
+		.type	= ULOGD_RET_IPADDR,
+		.flags	= ULOGD_RETF_NONE,
+		.name	= "orig.ip.daddr",
+		.ipfix	= {
+			.vendor = IPFIX_VENDOR_IETF,
+			.field_id = IPFIX_destinationIPv4Address,
+		},
+	},
+	{
+		.type	= ULOGD_RET_UINT8,
+		.flags	= ULOGD_RETF_NONE,
+		.name	= "orig.ip.protocol",
+		.ipfix	= { 
+			.vendor = IPFIX_VENDOR_IETF,
+			.field_id = IPFIX_protocolIdentifier,
+		},
+	},
+	{
+		.type	= ULOGD_RET_UINT16,
+		.flags 	= ULOGD_RETF_NONE,
+		.name	= "orig.l4.sport",
+		.ipfix	= {
+			.vendor 	= IPFIX_VENDOR_IETF,
+			.field_id 	= IPFIX_sourceTransportPort,
+		},
+	},
+	{
+		.type	= ULOGD_RET_UINT16,
+		.flags 	= ULOGD_RETF_NONE,
+		.name	= "orig.l4.dport",
+		.ipfix	= {
+			.vendor 	= IPFIX_VENDOR_IETF,
+			.field_id 	= IPFIX_destinationTransportPort,
+		},
+	},
+	{
+		.type	= ULOGD_RET_UINT32,
+		.flags	= ULOGD_RETF_NONE,
+		.name	= "orig.raw.pktlen",
+		.ipfix	= { 
+			.vendor 	= IPFIX_VENDOR_IETF,
+			.field_id 	= IPFIX_octetTotalCount,
+			/* FIXME: this could also be octetDeltaCount */
+		},
+	},
+	{
+		.type	= ULOGD_RET_UINT32,
+		.flags	= ULOGD_RETF_NONE,
+		.name	= "orig.raw.pktcount",
+		.ipfix	= { 
+			.vendor 	= IPFIX_VENDOR_IETF,
+			.field_id 	= IPFIX_packetTotalCount,
+			/* FIXME: this could also be packetDeltaCount */
+		},
+	},
+	{
+		.type 	= ULOGD_RET_IPADDR,
+		.flags 	= ULOGD_RETF_NONE,
+		.name	= "reply.ip.saddr",
 		.ipfix	= { 
 			.vendor = IPFIX_VENDOR_IETF,
 			.field_id = IPFIX_sourceIPv4Address,
@@ -119,7 +209,7 @@ static struct ulogd_key nfct_okeys[] = {
 	{
 		.type	= ULOGD_RET_IPADDR,
 		.flags	= ULOGD_RETF_NONE,
-		.name	= "ip.daddr",
+		.name	= "reply.ip.daddr",
 		.ipfix	= {
 			.vendor = IPFIX_VENDOR_IETF,
 			.field_id = IPFIX_destinationIPv4Address,
@@ -128,7 +218,7 @@ static struct ulogd_key nfct_okeys[] = {
 	{
 		.type	= ULOGD_RET_UINT8,
 		.flags	= ULOGD_RETF_NONE,
-		.name	= "ip.protocol",
+		.name	= "reply.ip.protocol",
 		.ipfix	= { 
 			.vendor = IPFIX_VENDOR_IETF,
 			.field_id = IPFIX_protocolIdentifier,
@@ -137,7 +227,7 @@ static struct ulogd_key nfct_okeys[] = {
 	{
 		.type	= ULOGD_RET_UINT16,
 		.flags 	= ULOGD_RETF_NONE,
-		.name	= "l4.sport",
+		.name	= "reply.l4.sport",
 		.ipfix	= {
 			.vendor 	= IPFIX_VENDOR_IETF,
 			.field_id 	= IPFIX_sourceTransportPort,
@@ -146,7 +236,7 @@ static struct ulogd_key nfct_okeys[] = {
 	{
 		.type	= ULOGD_RET_UINT16,
 		.flags 	= ULOGD_RETF_NONE,
-		.name	= "l4.dport",
+		.name	= "reply.l4.dport",
 		.ipfix	= {
 			.vendor 	= IPFIX_VENDOR_IETF,
 			.field_id 	= IPFIX_destinationTransportPort,
@@ -155,7 +245,7 @@ static struct ulogd_key nfct_okeys[] = {
 	{
 		.type	= ULOGD_RET_UINT32,
 		.flags	= ULOGD_RETF_NONE,
-		.name	= "raw.pktlen",
+		.name	= "reply.raw.pktlen",
 		.ipfix	= { 
 			.vendor 	= IPFIX_VENDOR_IETF,
 			.field_id 	= IPFIX_octetTotalCount,
@@ -165,7 +255,7 @@ static struct ulogd_key nfct_okeys[] = {
 	{
 		.type	= ULOGD_RET_UINT32,
 		.flags	= ULOGD_RETF_NONE,
-		.name	= "raw.pktcount",
+		.name	= "reply.raw.pktcount",
 		.ipfix	= { 
 			.vendor 	= IPFIX_VENDOR_IETF,
 			.field_id 	= IPFIX_packetTotalCount,
@@ -244,11 +334,6 @@ static struct ulogd_key nfct_okeys[] = {
 			.field_id	= IPFIX_flowEndSeconds,
 		},
 	},
-	{
-		.type = ULOGD_RET_BOOL,
-		.flags = ULOGD_RETF_NONE,
-		.name = "dir",
-	},
 };
 
 static struct ct_htable *htable_alloc(int htable_size, int prealloc)
@@ -364,93 +449,108 @@ static struct ct_timestamp *ct_hash_get(struct ct_htable *htable, uint32_t id)
 	return ct;
 }
 
-static int propagate_ct_flow(struct ulogd_pluginstance *upi, 
-		             struct nfct_conntrack *ct,
-			     unsigned int flags,
-			     int dir,
-			     struct ct_timestamp *ts)
+static int propagate_ct(struct ulogd_pluginstance *upi,
+			struct nfct_conntrack *ct,
+			unsigned int flags,
+			struct ct_timestamp *ts)
 {
 	struct ulogd_key *ret = upi->output.keys;
+	int dir;
+	
+	dir = NFCT_DIR_ORIGINAL;
+	ret[NFCT_ORIG_IP_SADDR].u.value.ui32 = htonl(ct->tuple[dir].src.v4);
+	ret[NFCT_ORIG_IP_SADDR].flags |= ULOGD_RETF_VALID;
 
-	ret[0].u.value.ui32 = htonl(ct->tuple[dir].src.v4);
-	ret[0].flags |= ULOGD_RETF_VALID;
-
-	ret[1].u.value.ui32 = htonl(ct->tuple[dir].dst.v4);
-	ret[1].flags |= ULOGD_RETF_VALID;
+	ret[NFCT_ORIG_IP_DADDR].u.value.ui32 = htonl(ct->tuple[dir].dst.v4);
+	ret[NFCT_ORIG_IP_DADDR].flags |= ULOGD_RETF_VALID;
 
-	ret[2].u.value.ui8 = ct->tuple[dir].protonum;
-	ret[2].flags |= ULOGD_RETF_VALID;
+	ret[NFCT_ORIG_IP_PROTOCOL].u.value.ui8 = ct->tuple[dir].protonum;
+	ret[NFCT_ORIG_IP_PROTOCOL].flags |= ULOGD_RETF_VALID;
 
-	switch (ct->tuple[1].protonum) {
+	switch (ct->tuple[dir].protonum) {
 	case IPPROTO_TCP:
 	case IPPROTO_UDP:
 	case IPPROTO_SCTP:
 		/* FIXME: DCCP */
-		ret[3].u.value.ui16 = htons(ct->tuple[dir].l4src.tcp.port);
-		ret[3].flags |= ULOGD_RETF_VALID;
-		ret[4].u.value.ui16 = htons(ct->tuple[dir].l4dst.tcp.port);
-		ret[4].flags |= ULOGD_RETF_VALID;
+		ret[NFCT_ORIG_L4_SPORT].u.value.ui16 = htons(ct->tuple[dir].l4src.tcp.port);
+		ret[NFCT_ORIG_L4_SPORT].flags |= ULOGD_RETF_VALID;
+		ret[NFCT_ORIG_L4_DPORT].u.value.ui16 = htons(ct->tuple[dir].l4dst.tcp.port);
+		ret[NFCT_ORIG_L4_DPORT].flags |= ULOGD_RETF_VALID;
 		break;
 	case IPPROTO_ICMP:
-		ret[7].u.value.ui8 = ct->tuple[dir].l4src.icmp.code;
-		ret[7].flags |= ULOGD_RETF_VALID;
-		ret[8].u.value.ui8 = ct->tuple[dir].l4src.icmp.type;
-		ret[8].flags |= ULOGD_RETF_VALID;
+		ret[NFCT_ICMP_CODE].u.value.ui8 = ct->tuple[dir].l4src.icmp.code;
+		ret[NFCT_ICMP_CODE].flags |= ULOGD_RETF_VALID;
+		ret[NFCT_ICMP_TYPE].u.value.ui8 = ct->tuple[dir].l4src.icmp.type;
+		ret[NFCT_ICMP_TYPE].flags |= ULOGD_RETF_VALID;
 		break;
 	}
 
-	if ((dir == NFCT_DIR_ORIGINAL && flags & NFCT_COUNTERS_ORIG) ||
-	    (dir == NFCT_DIR_REPLY && flags & NFCT_COUNTERS_RPLY)) {
-		ret[5].u.value.ui64 = ct->counters[dir].bytes;
-		ret[5].flags |= ULOGD_RETF_VALID;
+	ret[NFCT_ORIG_RAW_PKTLEN].u.value.ui64 = ct->counters[dir].bytes;
+	ret[NFCT_ORIG_RAW_PKTLEN].flags |= ULOGD_RETF_VALID;
+
+	ret[NFCT_ORIG_RAW_PKTCOUNT].u.value.ui64 = ct->counters[dir].packets;
+	ret[NFCT_ORIG_RAW_PKTCOUNT].flags |= ULOGD_RETF_VALID;
+
+	dir = NFCT_DIR_REPLY;
+	ret[NFCT_REPLY_IP_SADDR].u.value.ui32 = htonl(ct->tuple[dir].src.v4);
+	ret[NFCT_REPLY_IP_SADDR].flags |= ULOGD_RETF_VALID;
 
-		ret[6].u.value.ui64 = ct->counters[dir].packets;
-		ret[6].flags |= ULOGD_RETF_VALID;
+	ret[NFCT_REPLY_IP_DADDR].u.value.ui32 = htonl(ct->tuple[dir].dst.v4);
+	ret[NFCT_REPLY_IP_DADDR].flags |= ULOGD_RETF_VALID;
+
+	ret[NFCT_REPLY_IP_PROTOCOL].u.value.ui8 = ct->tuple[dir].protonum;
+	ret[NFCT_REPLY_IP_PROTOCOL].flags |= ULOGD_RETF_VALID;
+
+	switch (ct->tuple[dir].protonum) {
+	case IPPROTO_TCP:
+	case IPPROTO_UDP:
+	case IPPROTO_SCTP:
+		/* FIXME: DCCP */
+		ret[NFCT_REPLY_L4_SPORT].u.value.ui16 = htons(ct->tuple[dir].l4src.tcp.port);
+		ret[NFCT_REPLY_L4_SPORT].flags |= ULOGD_RETF_VALID;
+		ret[NFCT_REPLY_L4_DPORT].u.value.ui16 = htons(ct->tuple[dir].l4dst.tcp.port);
+		ret[NFCT_REPLY_L4_DPORT].flags |= ULOGD_RETF_VALID;
+		break;
+	case IPPROTO_ICMP:
+		ret[NFCT_ICMP_CODE].u.value.ui8 = ct->tuple[dir].l4src.icmp.code;
+		ret[NFCT_ICMP_CODE].flags |= ULOGD_RETF_VALID;
+		ret[NFCT_ICMP_TYPE].u.value.ui8 = ct->tuple[dir].l4src.icmp.type;
+		ret[NFCT_ICMP_TYPE].flags |= ULOGD_RETF_VALID;
+		break;
 	}
 
+	ret[NFCT_REPLY_RAW_PKTLEN].u.value.ui64 = ct->counters[dir].bytes;
+	ret[NFCT_REPLY_RAW_PKTLEN].flags |= ULOGD_RETF_VALID;
+
+	ret[NFCT_REPLY_RAW_PKTCOUNT].u.value.ui64 = ct->counters[dir].packets;
+	ret[NFCT_REPLY_RAW_PKTCOUNT].flags |= ULOGD_RETF_VALID;
+
 	if (flags & NFCT_MARK) {
-		ret[9].u.value.ui32 = ct->mark;
-		ret[9].flags |= ULOGD_RETF_VALID;
+		ret[NFCT_CT_MARK].u.value.ui32 = ct->mark;
+		ret[NFCT_CT_MARK].flags |= ULOGD_RETF_VALID;
 	}
 
 	if (flags & NFCT_ID) {
-		ret[10].u.value.ui32 = ct->id;
-		ret[10].flags |= ULOGD_RETF_VALID;
+		ret[NFCT_CT_ID].u.value.ui32 = ct->id;
+		ret[NFCT_CT_ID].flags |= ULOGD_RETF_VALID;
 	}
 
 	if (ts) {
-		ret[11].u.value.ui32 = ts->time[START].tv_sec;
-		ret[11].flags |= ULOGD_RETF_VALID;
-		ret[12].u.value.ui32 = ts->time[START].tv_usec;
-		ret[12].flags |= ULOGD_RETF_VALID;
-		ret[13].u.value.ui32 = ts->time[STOP].tv_sec;
-		ret[13].flags |= ULOGD_RETF_VALID;
-		ret[14].u.value.ui32 = ts->time[STOP].tv_usec;
-		ret[14].flags |= ULOGD_RETF_VALID;
+		ret[NFCT_FLOW_START_SEC].u.value.ui32 = ts->time[START].tv_sec;
+		ret[NFCT_FLOW_START_SEC].flags |= ULOGD_RETF_VALID;
+		ret[NFCT_FLOW_START_USEC].u.value.ui32 = ts->time[START].tv_usec;
+		ret[NFCT_FLOW_START_USEC].flags |= ULOGD_RETF_VALID;
+		ret[NFCT_FLOW_END_SEC].u.value.ui32 = ts->time[STOP].tv_sec;
+		ret[NFCT_FLOW_END_SEC].flags |= ULOGD_RETF_VALID;
+		ret[NFCT_FLOW_END_USEC].u.value.ui32 = ts->time[STOP].tv_usec;
+		ret[NFCT_FLOW_END_USEC].flags |= ULOGD_RETF_VALID;
 	}
 
-	ret[15].u.value.b = (dir == NFCT_DIR_ORIGINAL) ? 0 : 1;
-	ret[15].flags |= ULOGD_RETF_VALID;
-
 	ulogd_propagate_results(upi);
 
 	return 0;
 }
 
-static int propagate_ct(struct ulogd_pluginstance *upi,
-			struct nfct_conntrack *ct,
-			unsigned int flags,
-			struct ct_timestamp *ctstamp)
-{
-	int rc;
-
-	rc = propagate_ct_flow(upi, ct, flags, NFCT_DIR_ORIGINAL, ctstamp);
-	if (rc < 0)
-		return rc;
-
-	return propagate_ct_flow(upi, ct, flags, NFCT_DIR_REPLY, ctstamp);
-}
-
 static int event_handler(void *arg, unsigned int flags, int type,
 			 void *data)
 {
-- 
1.5.2.5


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH] Re: [ULOGD RFC 08/30] NFCT: rework

[Holger Eitzenberger]

Eric Leblond <eric@inl.fr> writes:

Hi Eric,

> You changelog is not complete as it did not mention important
> modification of the output keys.

Sorry, my way of presenting all those changes in very few patches was
intentional, I just tried to keep the number of patches small.

> Accounting is done here for both side of connection ...

Yes, principally this should be a configuration option sooner or later,
but this is what I needed at the time of writing that patch.  The former
way might not be what someone would expect though.  But sure, that change
should have been documented somewhere.

I'll provide a patchset which stuffs all NFCT-related patches into one.
Therein contained are two small changes which basically root from
Pablo Neiras suggestions.

As my NFCT changes currently still depend on one small patch to
libnetfilter-conntrack and Pablo seems quite reluctant to apply that
I'll suggest in my next announcement to apply all but the NFCT changes.
I would like to solve that issue by switching to libnl soon.

However, that switch only makes sense if the other netlink users (NFLOG,
ifi) are switched as well.

If Patrick decides to apply that big patch (retaining compatibility of
output keys of course) I'm fine.  If not I would have to do a gradual
switch to the new code base, but then preferably switching to libnl on
the go.

Thanks for reviewing!

 /holger

[ULOGD RFC 09/30] Port to ulogd 2.00, mostly a rewrite

[heitzenberger]

[-- Attachment #1: ulogd-SQLITE3-plugin.diff --]
[-- Type: text/plain, Size: 25446 bytes --]

As described in

 http://www.sqlite.org/cvstrac/wiki?p=CorruptionFollowingBusyError

some versions of SQLITE contain an error when transactions other than
EXCLUSIVE are used.  If a SQL commands to create a transaction or
inside a transaction ever gets SQLITE_BUSY a ROLLBACK has to be done,
as otherwise a corrupt database might be the result.

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/output/sqlite3/ulogd_output_SQLITE3.c
===================================================================
--- ulogd-netfilter.orig/output/sqlite3/ulogd_output_SQLITE3.c
+++ ulogd-netfilter/output/sqlite3/ulogd_output_SQLITE3.c
@@ -1,4 +1,3 @@
-#if 0
 /*
  * ulogd output plugin for logging to a SQLITE database
  *
@@ -26,389 +25,736 @@
  *
  *  2005-02-09 Harald Welte <laforge@gnumonks.org>:
  *  	- port to ulogd-1.20 
+ *
+ *  Holger Eitzenberger <holger@eitzenberger.org>  Astaro AG, 2007
+ *  	- port to ulogd-2.00
  */
 
-#include <stdlib.h>
-#include <string.h>
-#include <arpa/inet.h>
 #include <ulogd/ulogd.h>
 #include <ulogd/conffile.h>
+#include <ulogd/common.h>
+#include <arpa/inet.h>
 #include <sqlite3.h>
+#include <sys/queue.h>
+
+#define PFX		"SQLITE3: "
+
+/* config defaults */
+#define CFG_BUFFER_DEFAULT		100
+#define CFG_TIMER_DEFAULT		1 SEC
+#define CFG_MAX_BACKLOG_DEFAULT	0		/* unlimited */
+
 
-#ifdef DEBUG_SQLITE3
+#define SQLITE3_BUSY_TIMEOUT 300
+
+/* number of colums we have (really should be configurable) */
+#define DB_NUM_COLS	10
+
+#if 0
 #define DEBUGP(x, args...)	fprintf(stderr, x, ## args)
 #else
 #define DEBUGP(x, args...)
 #endif
 
-struct _field {
+/* map DB column to ulogd key */
+struct col {
 	char name[ULOGD_MAX_KEYLEN];
-	unsigned int id;
-	struct _field *next;
+	struct ulogd_key *key;
 };
 
-/* the database handle we are using */
-static sqlite3 *dbh;
+struct row {
+	TAILQ_ENTRY(row) link;
+	uint32_t ip_saddr;
+	uint32_t ip_daddr;
+	unsigned char ip_proto;
+	unsigned l4_dport;
+	unsigned raw_in_pktlen;
+	unsigned raw_in_pktcount;
+	unsigned raw_out_pktlen;
+	unsigned raw_out_pktcount;
+	unsigned flow_start_sec;
+	unsigned flow_duration;
+};
 
-/* a linked list of the fields the table has */
-static struct _field *fields;
+TAILQ_HEAD(row_lh, row);
 
-/* buffer for our insert statement */
-static char *stmt;
+#define TAILQ_FOR_EACH(pos, head, link) \
+        for (pos = (head).tqh_first; pos != NULL; pos = pos->link.tqe_next)
 
-/* pointer to the final prepared statement */
-static sqlite3_stmt *p_stmt;
+#define RKEY(key)	((key)->u.source)
 
-/* number of statements to buffer before we commit */
-static int buffer_size;
 
-/* number of statements currently in the buffer */
-static int buffer_ctr;
+struct sqlite3_priv {
+	sqlite3 *dbh;				/* database handle we are using */
+	char *stmt;
+	sqlite3_stmt *p_stmt;
+	int buffer_size;
 
-/* our configuration directives */
-static config_entry_t db_ce = { 
-	.key = "db", 
-	.type = CONFIG_TYPE_STRING,
-	.options = CONFIG_OPT_MANDATORY,
-};
+	struct ulogd_timer timer;
+
+	struct col cols[DB_NUM_COLS];
 
-static config_entry_t table_ce = { 
-	.next = &db_ce, 
-	.key = "table",
-	.type = CONFIG_TYPE_STRING,
-	.options = CONFIG_OPT_MANDATORY,
+	/* our backlog buffer */
+	struct row_lh rows;
+	int num_rows;
+	int max_rows;				/* number of rows actually seen */
+	int max_rows_allowed;
+
+	unsigned overlimit_msg : 1;
 };
 
-static config_entry_t buffer_ce = { 
-	.next = &table_ce,
-	.key = "buffer",
-	.type = CONFIG_TYPE_INT,
-	.options = CONFIG_OPT_MANDATORY,
+
+static int do_reinit;
+static struct config_keyset sqlite3_kset = {
+	.num_ces = 5,
+	.ces = {
+		{
+			.key = "db",
+			.type = CONFIG_TYPE_STRING,
+			.options = CONFIG_OPT_MANDATORY,
+		},
+		{
+			.key = "table",
+			.type = CONFIG_TYPE_STRING,
+			.options = CONFIG_OPT_MANDATORY,
+		},
+		{
+			.key = "buffer",
+			.type = CONFIG_TYPE_INT,
+			.options = CONFIG_OPT_NONE,
+			.u.value = CFG_BUFFER_DEFAULT,
+		},
+		{
+			.key = "timer",
+			.type = CONFIG_TYPE_INT,
+			.options = CONFIG_OPT_NONE,
+			.u.value = CFG_TIMER_DEFAULT,
+		},
+		{
+			.key = "max-backlog",
+			.type = CONFIG_TYPE_INT,
+			.options = CONFIG_OPT_NONE,
+			.u.value = CFG_MAX_BACKLOG_DEFAULT,
+		},
+	},
 };
 
-/* our main output function, called by ulogd */
-static int _sqlite3_output(ulog_iret_t *result)
+#define db_ce(pi)		(pi)->config_kset->ces[0].u.string
+#define table_ce(pi)	(pi)->config_kset->ces[1].u.string
+#define buffer_ce(pi)	(pi)->config_kset->ces[2].u.value
+#define timer_ce(pi)	(pi)->config_kset->ces[3].u.value
+#define max_backlog_ce(pi)	(pi)->config_kset->ces[4].u.value
+
+
+#define SQL_CREATE_STR \
+		"create table daily(ip_saddr integer, ip_daddr integer, " \
+		"ip_protocol integer, l4_dport integer, raw_in_pktlen integer, " \
+		"raw_in_pktcount integer, raw_out_pktlen integer, " \
+		"raw_out_pktcount integer, flow_start_sec integer, " \
+		"flow_duration integer)"
+
+
+static struct row *
+row_new(void)
 {
-	struct _field *f;
-	ulog_iret_t *res;
-	int col_counter;
-#ifdef IP_AS_STRING
-	char *ipaddr;
-	struct in_addr addr;
-#endif
+	struct row *row;
+
+	if ((row = calloc(1, sizeof(struct row))) == NULL)
+		ulogd_error("%s: out of memory\n", __func__);
+
+	return row;
+}
 
-	col_counter = 1;
-	for (f = fields; f; f = f->next) {
-		res = keyh_getres(f->id);
-
-		if (!res) {
-			ulogd_log(ULOGD_NOTICE,
-				"no result for %s ?!?\n", f->name);
-		}
-			
-		if (!res || !IS_VALID((*res))) {
-			/* no result, pass a null */
-			sqlite3_bind_null(p_stmt, col_counter);
-			col_counter++;
-			continue;
-		}
-		
-		switch (res->type) {
-			case ULOGD_RET_INT8:
-				sqlite3_bind_int(p_stmt,col_counter,res->value.i8);
-				break;
-			case ULOGD_RET_INT16:
-				sqlite3_bind_int(p_stmt,col_counter,res->value.i16);
-				break;
-			case ULOGD_RET_INT32:
-				sqlite3_bind_int(p_stmt,col_counter,res->value.i32);
-				break;
-			case ULOGD_RET_INT64:
-				sqlite3_bind_int64(p_stmt,col_counter,res->value.i64);
-				break;
-			case ULOGD_RET_UINT8:
-				sqlite3_bind_int(p_stmt,col_counter,res->value.ui8);
-				break;
-			case ULOGD_RET_UINT16:
-				sqlite3_bind_int(p_stmt,col_counter,res->value.ui16);
-				break;
-			case ULOGD_RET_IPADDR:
-#ifdef IP_AS_STRING
-				memset(&addr, 0, sizeof(addr));
-				addr.s_addr = ntohl(res->value.ui32);
-				ipaddr = inet_ntoa(addr);
-				sqlite3_bind_text(p_stmt,col_counter,ipaddr,strlen(ipaddr),SQLITE_STATIC);
-                                break;
-#endif /* IP_AS_STRING */
-			/* EVIL: fallthrough when logging IP as u_int32_t */
-			case ULOGD_RET_UINT32:
-				sqlite3_bind_int(p_stmt,col_counter,res->value.ui32);
-				break;
-			case ULOGD_RET_UINT64:
-				sqlite3_bind_int64(p_stmt,col_counter,res->value.ui64);
-				break;
-			case ULOGD_RET_BOOL:
-				sqlite3_bind_int(p_stmt,col_counter,res->value.b);
-				break;
-			case ULOGD_RET_STRING:
-				sqlite3_bind_text(p_stmt,col_counter,res->value.ptr,strlen(res->value.ptr),SQLITE_STATIC);
-				break;
-			default:
-				ulogd_log(ULOGD_NOTICE,
-					"unknown type %d for %s\n",
-					res->type, res->key);
-				break;
-		} 
-
-		col_counter++;
-	}
-
-	/* now we have created our statement, insert it */
-
-	if (sqlite3_step(p_stmt) == SQLITE_DONE) {
-		sqlite3_reset(p_stmt);
-		buffer_ctr++;
-	} else {
-		ulogd_log(ULOGD_ERROR, "sql error during insert: %s\n",
-				sqlite3_errmsg(dbh));
-		return 1;
-	}
 
-	/* commit all of the inserts to the database, ie flush buffer */
-	if (buffer_ctr >= buffer_size) {
-		if (sqlite3_exec(dbh,"commit",NULL,NULL,NULL) != SQLITE_OK)
-			ulogd_log(ULOGD_ERROR,"unable to commit records to db.");
+static void
+row_del(struct sqlite3_priv *priv, struct row *row)
+{
+	TAILQ_REMOVE(&priv->rows, row, link);
+
+	free(row);
+
+	priv->num_rows--;
+}
 
-		if (sqlite3_exec(dbh,"begin deferred",NULL,NULL,NULL) != SQLITE_OK)
-			ulogd_log(ULOGD_ERROR,"unable to begin a new transaction.");
 
-		buffer_ctr = 0;
-		DEBUGP("committing.\n");
+static void
+row_add(struct sqlite3_priv *priv, struct row *row)
+{
+	if (priv->max_rows_allowed && priv->num_rows > priv->max_rows_allowed) {
+		if (!priv->overlimit_msg) {
+			ulogd_error(PFX "over max-backlog limit, dropping row\n");
+
+			priv->overlimit_msg = 1;
+		}
+
+		return;
 	}
 
-	return 0;
+	TAILQ_INSERT_TAIL(&priv->rows, row, link);
+
+	priv->num_rows++;
 }
 
+
 #define _SQLITE3_INSERTTEMPL   "insert into X (Y) values (Z)"
 
-/* create the static part of our insert statement */
-static int _sqlite3_createstmt(void)
+/* create static part of our insert statement */
+static int
+db_createstmt(struct ulogd_pluginstance *pi)
 {
-	struct _field *f;
-	unsigned int size;
+	struct sqlite3_priv *priv = (void *)pi->private;
 	char buf[ULOGD_MAX_KEYLEN];
 	char *underscore;
 	char *stmt_pos;
-	int col_count;
 	int i;
 
-	if (stmt) {
-		ulogd_log(ULOGD_NOTICE, "createstmt called, but stmt"
-			" already existing\n");	
-		return 1;
-	}
-
-	/* caclulate the size for the insert statement */
-	size = strlen(_SQLITE3_INSERTTEMPL) + strlen(table_ce.u.string);
-
-	DEBUGP("initial size: %u\n", size);
+	if (priv->stmt != NULL)
+		free(priv->stmt);
 
-	col_count = 0;
-	for (f = fields; f; f = f->next) {
-		/* we need space for the key and a comma, and a ? */
-		size += strlen(f->name) + 3;
-		DEBUGP("size is now %u since adding %s\n",size,f->name);
-		col_count++;
+	if ((priv->stmt = calloc(1, 1024)) == NULL) {
+		ulogd_error(PFX "out of memory\n");
+		return -1;
 	}
 
-	DEBUGP("there were %d columns\n",col_count);
-	DEBUGP("after calc name length: %u\n",size);
+	sprintf(priv->stmt, "insert into %s (", table_ce(pi));
+	stmt_pos = priv->stmt + strlen(priv->stmt);
 
-	ulogd_log(ULOGD_DEBUG, "allocating %u bytes for statement\n", size);
+	for (i = 0; i < DB_NUM_COLS; i++) {
+		struct col *col = &priv->cols[i];
 
-	stmt = (char *) malloc(size);
+		/* convert name */
+		strncpy(buf, col->name, ULOGD_MAX_KEYLEN);
 
-	if (!stmt) {
-		ulogd_log(ULOGD_ERROR, "OOM!\n");
-		return 1;
-	}
-
-	sprintf(stmt, "insert into %s (", table_ce.u.string);
-	stmt_pos = stmt + strlen(stmt);
-
-	for (f = fields; f; f = f->next) {
-		strncpy(buf, f->name, ULOGD_MAX_KEYLEN);	
 		while ((underscore = strchr(buf, '.')))
 			*underscore = '_';
+
 		sprintf(stmt_pos, "%s,", buf);
-		stmt_pos = stmt + strlen(stmt);
+		stmt_pos = priv->stmt + strlen(priv->stmt);
 	}
 
 	*(stmt_pos - 1) = ')';
 
 	sprintf(stmt_pos, " values (");
-	stmt_pos = stmt + strlen(stmt);
+	stmt_pos = priv->stmt + strlen(priv->stmt);
 
-	for (i = 0; i < col_count - 1; i++) {
+	for (i = 0; i < DB_NUM_COLS - 1; i++) {
 		sprintf(stmt_pos,"?,");
 		stmt_pos += 2;
 	}
 
 	sprintf(stmt_pos, "?)");
-	ulogd_log(ULOGD_DEBUG, "stmt='%s'\n", stmt);
+	ulogd_log(ULOGD_DEBUG, "%s: stmt='%s'\n", pi->id, priv->stmt);
 
 	DEBUGP("about to prepare statement.\n");
 
-	sqlite3_prepare(dbh,stmt,-1,&p_stmt,0);
+	sqlite3_prepare(priv->dbh, priv->stmt, -1, &priv->p_stmt, 0);
+	if (priv->p_stmt == NULL) {
+		ulogd_error(PFX "prepare: %s\n", sqlite3_errmsg(priv->dbh));
+		return 1;
+	}
 
 	DEBUGP("statement prepared.\n");
 
-	if (!p_stmt) {
-		ulogd_log(ULOGD_ERROR,"unable to prepare statement");
-		return 1;
+	return 0;
+}
+
+
+static struct ulogd_key *
+ulogd_find_key(struct ulogd_pluginstance *pi, const char *name)
+{
+	int i;
+
+	for (i = 0; i < pi->input.num_keys; i++) {
+		if (strcmp(pi->input.keys[i].name, name) == 0)
+			return &pi->input.keys[i];
 	}
 
-	return 0;
+	return NULL;
+}
+
+#define SELECT_ALL_STR			"select * from "
+#define SELECT_ALL_LEN			sizeof(SELECT_ALL_STR)
+
+static int
+db_count_cols(struct ulogd_pluginstance *pi, sqlite3_stmt **stmt)
+{
+	struct sqlite3_priv *priv = (void *)pi->private;
+	char query[SELECT_ALL_LEN + CONFIG_VAL_STRING_LEN] = SELECT_ALL_STR;
+
+	strncat(query, table_ce(pi), LINE_LEN);
+
+	if (sqlite3_prepare(priv->dbh, query, -1, stmt, 0) != SQLITE_OK) {
+		return -1;
+	}
+
+	return sqlite3_column_count(*stmt);
 }
 
 
-/* length of "select * from \0" */
-#define SQLITE_SELECT_LEN 15
+static int
+db_create_tbl(struct ulogd_pluginstance *pi)
+{
+	struct sqlite3_priv *priv = (void *)pi->private;
+	char *errmsg;
+	int ret;
+
+	sqlite3_exec(priv->dbh, "drop table daily", NULL, NULL, NULL);
+
+	ret = sqlite3_exec(priv->dbh, SQL_CREATE_STR, NULL, NULL, &errmsg);
+	if (ret != SQLITE_OK) {
+		ulogd_error(PFX "create table: %s\n", errmsg);
+		sqlite3_free(errmsg);
+
+		return -1;
+	}
+
+	return 0;
+}
+
 
-/* find out which columns the table has */
-static int _sqlite3_get_columns(const char *table)
+/* initialize DB, possibly creating it */
+static int
+db_init(struct ulogd_pluginstance *pi)
 {
+	struct sqlite3_priv *priv = (void *)pi->private;
 	char buf[ULOGD_MAX_KEYLEN];
-	char query[SQLITE_SELECT_LEN + CONFIG_VAL_STRING_LEN] = "select * from \0";
 	char *underscore;
-	struct _field *f;
 	sqlite3_stmt *schema_stmt;
-	int column;
-	int result;
-	int id;
+	int num_cols, i;
 
-	if (!dbh)
-		return 1;
+	if (priv->dbh == NULL)
+		return -1;
 
-	strncat(query,table,LINE_LEN);
-	
-	result = sqlite3_prepare(dbh,query,-1,&schema_stmt,0);
-	
-	if (result != SQLITE_OK)
-		return 1;
+	num_cols = db_count_cols(pi, &schema_stmt);
+	if (num_cols != DB_NUM_COLS) {
+		ulogd_log(ULOGD_INFO, PFX "(re)creating database\n");
+
+		if (db_create_tbl(pi) < 0)
+			return -1;
+
+		num_cols = db_count_cols(pi, &schema_stmt);
+	}
+
+	assert(num_cols == DB_NUM_COLS);
+
+	for (i = 0; i < DB_NUM_COLS; i++) {
+		struct col *col = &priv->cols[i];
+
+		strncpy(buf, sqlite3_column_name(schema_stmt, i), ULOGD_MAX_KEYLEN);
 
-	for (column = 0; column < sqlite3_column_count(schema_stmt); column++) {
 		/* replace all underscores with dots */
-		strncpy(buf, sqlite3_column_name(schema_stmt,column), ULOGD_MAX_KEYLEN);
-		while ((underscore = strchr(buf, '_')))
+		while ((underscore = strchr(buf, '_')) != NULL)
 			*underscore = '.';
 
-		DEBUGP("field '%s' found: ", buf);
+		DEBUGP("column '%s' found\n", buf);
 
-		if (!(id = keyh_getid(buf))) {
-			DEBUGP(" no keyid!\n");
-			continue;
+		strncpy(col->name, buf, ULOGD_MAX_KEYLEN);
+
+		if ((col->key = ulogd_find_key(pi, buf)) == NULL) {
+			printf(PFX "%s: key not found\n", buf);
+			return -1;
 		}
+	}
 
-		DEBUGP("keyid %u\n", id);
+	ulogd_log(ULOGD_INFO, PFX "database successfully opened\n");
 
-		/* prepend it to the linked list */
-		f = (struct _field *) malloc(sizeof *f);
-		if (!f) {
-			ulogd_log(ULOGD_ERROR, "OOM!\n");
-			return 1;
-		}
-		strncpy(f->name, buf, ULOGD_MAX_KEYLEN);
-		f->id = id;
-		f->next = fields;
-		fields = f;	
+	if (sqlite3_finalize(schema_stmt) != SQLITE_OK) {
+		ulogd_error(PFX "sqlite_finalize: %s\n",
+					sqlite3_errmsg(priv->dbh));
+		return -1;
 	}
 
-	sqlite3_finalize(schema_stmt);
 	return 0;
 }
 
-/** 
- * make connection and select database 
- * returns 0 if database failed to open.
- */
-static int _sqlite3_open_db(char *db_file)
+
+static void
+db_reset(struct ulogd_pluginstance *pi)
 {
-	DEBUGP("opening database.\n");
-	return sqlite3_open(db_file,&dbh);
+	struct sqlite3_priv *priv = (void *)pi->private;
+
+	sqlite3_finalize(priv->p_stmt);
+
+	sqlite3_close(priv->dbh);
+	priv->dbh = NULL;
+
+	free(priv->stmt);
+	priv->stmt = NULL;
 }
 
-/* give us an opportunity to close the database down properly */
-static void _sqlite3_fini(void)
+
+static int
+db_start(struct ulogd_pluginstance *pi)
 {
-	DEBUGP("cleaning up db connection\n");
+	struct sqlite3_priv *priv = (void *)pi->private;
 
-	/* free up our prepared statements so we can close the db */
-	if (p_stmt) {
-		sqlite3_finalize(p_stmt);
-		DEBUGP("prepared statement finalized\n");
+	ulogd_log(ULOGD_DEBUG, PFX "opening database connection\n");
+
+	if (sqlite3_open(db_ce(pi), &priv->dbh) != SQLITE_OK) {
+		ulogd_error(PFX "%s\n", sqlite3_errmsg(priv->dbh));
+		return -1;
+	}
+
+	/* set the timeout so that we don't automatically fail
+	   if the table is busy */
+	sqlite3_busy_timeout(priv->dbh, SQLITE3_BUSY_TIMEOUT);
+
+	/* read the fieldnames to know which values to insert */
+	if (db_init(pi) < 0)
+		return -1;
+
+	/* initialize our buffer size and counter */
+	priv->buffer_size = buffer_ce(pi);
+
+	priv->max_rows_allowed = max_backlog_ce(pi);
+
+	/* create and prepare the actual insert statement */
+	db_createstmt(pi);
+
+	return 0;
+}
+
+
+static int
+db_add_row(struct ulogd_pluginstance *pi, const struct row *row)
+{
+	struct sqlite3_priv *priv = (void *)pi->private;
+	int db_col = 1, ret = 0, db_ret;
+
+	db_ret = sqlite3_bind_int64(priv->p_stmt, db_col++, row->ip_saddr);
+	if (db_ret != SQLITE_OK)
+		goto err_bind;
+
+	db_ret = sqlite3_bind_int64(priv->p_stmt, db_col++, row->ip_daddr);
+	if (db_ret != SQLITE_OK)
+		goto err_bind;
+
+	db_ret = sqlite3_bind_int(priv->p_stmt, db_col++, row->ip_proto);
+	if (db_ret != SQLITE_OK)
+		goto err_bind;
+
+	db_ret = sqlite3_bind_int(priv->p_stmt, db_col++, row->l4_dport);
+	if (db_ret != SQLITE_OK)
+		goto err_bind;
+
+	db_ret = sqlite3_bind_int(priv->p_stmt, db_col++, row->raw_in_pktlen);
+	if (db_ret != SQLITE_OK)
+		goto err_bind;
+
+	db_ret = sqlite3_bind_int64(priv->p_stmt, db_col++, row->raw_in_pktcount);
+	if (db_ret != SQLITE_OK)
+		goto err_bind;
+
+	db_ret = sqlite3_bind_int(priv->p_stmt, db_col++, row->raw_out_pktlen);
+	if (db_ret != SQLITE_OK)
+		goto err_bind;
+
+	db_ret = sqlite3_bind_int64(priv->p_stmt, db_col++, row->raw_out_pktcount);
+	if (db_ret != SQLITE_OK)
+		goto err_bind;
+
+	db_ret = sqlite3_bind_int(priv->p_stmt, db_col++, row->flow_start_sec);
+	if (db_ret != SQLITE_OK)
+		goto err_bind;
+
+	db_ret = sqlite3_bind_int(priv->p_stmt, db_col++, row->flow_duration);
+	if (db_ret != SQLITE_OK)
+		goto err_bind;
+
+	db_ret = sqlite3_step(priv->p_stmt);
+
+	if (db_ret == SQLITE_DONE) {
+		/* the SQLITE book doesn't say that expclicitely _but_ between
+		   two sqlite_bind_*() calls to the same variable you need to
+		   call sqlite3_reset(). */
+		sqlite3_reset(priv->p_stmt);
+
+		return 0;
+	}
+
+	/* Ok, this is a bit confusing: some errors are reported as return
+	   values, most others are reported through sqlite3_errcode() instead.
+	   I think the only authorative source of information is the sqlite
+	   source code.	*/
+	switch (sqlite3_errcode(priv->dbh)) {
+	case SQLITE_LOCKED:
+	case SQLITE_BUSY:
+		break;
+
+	case SQLITE_SCHEMA:
+		if (priv->stmt) {
+			sqlite3_finalize(priv->p_stmt);
+
+			db_createstmt(pi);
+		}
+		return -1;
+
+	case SQLITE_ERROR: /* e.g. constraint violation */
+	case SQLITE_MISUSE:
+		ulogd_error(PFX "step: %s\n", sqlite3_errmsg(priv->dbh));
+		ret = -1;
+		break;
+
+	default:
+		break;
 	}
 
-	if (dbh) {
-		int result;
-		/* flush the remaining insert statements to the database. */
-		result = sqlite3_exec(dbh,"commit",NULL,NULL,NULL);
+	sqlite3_reset(priv->p_stmt);
+
+	return ret;
 
-		if (result != SQLITE_OK)
-			ulogd_log(ULOGD_ERROR,"unable to commit remaining records to db.");
+ err_bind:
+	ulogd_error(PFX "bind: %s\n", sqlite3_errmsg(priv->dbh));
 
-		sqlite3_close(dbh);
-		DEBUGP("database file closed\n");
+	sqlite3_reset(priv->p_stmt);
+
+	return -1;
+}
+
+
+static int
+delete_all_rows(struct ulogd_pluginstance *pi)
+{
+	struct sqlite3_priv *priv = (void *)pi->private;
+
+	while (priv->rows.tqh_first != NULL) {
+		struct row *row = priv->rows.tqh_first;
+
+		row_del(priv, row);
 	}
+
+	return 0;
 }
 
-#define _SQLITE3_BUSY_TIMEOUT 300
 
-static int _sqlite3_init(void)
+static int
+db_commit_rows(struct ulogd_pluginstance *pi)
 {
-	/* have the opts parsed */
-	config_parse_file("SQLITE3", &buffer_ce);
+	struct sqlite3_priv *priv = (void *)pi->private;
+	struct row *row;
+	int ret, rows = 0;
+
+	ret = sqlite3_exec(priv->dbh, "begin immediate transaction", NULL,
+					   NULL, NULL);
+	if (ret != SQLITE_OK) {
+		if (ret == SQLITE_BUSY)
+			goto err_rollback;
 
-	if (_sqlite3_open_db(db_ce.u.string)) {
-		ulogd_log(ULOGD_ERROR, "can't open the database file\n");
-		return 1;
+		if (sqlite3_errcode(priv->dbh) == SQLITE_LOCKED)
+			return 0;			/* perform commit later */
+	
+		ulogd_error(PFX "begin transaction: %s\n", sqlite3_errmsg(priv->dbh));
+
+		return -1;
 	}
 
-	/* set the timeout so that we don't automatically fail
-         * if the table is busy. */
-	sqlite3_busy_timeout(dbh, _SQLITE3_BUSY_TIMEOUT);
+	TAILQ_FOR_EACH(row, priv->rows, link) {
+		if (db_add_row(pi, row) < 0)
+			goto err_rollback;
 
-	/* read the fieldnames to know which values to insert */
-	if (_sqlite3_get_columns(table_ce.u.string)) {
-		ulogd_log(ULOGD_ERROR, "unable to get sqlite columns\n");
-		return 1;
+		rows++;
 	}
 
-	/* initialize our buffer size and counter */
-	buffer_size = buffer_ce.u.value;
-	buffer_ctr = 0;
+	ret = sqlite3_exec(priv->dbh, "commit", NULL, NULL, NULL);
+	if (ret == SQLITE_OK) {
+		sqlite3_reset(priv->p_stmt);
 
-	DEBUGP("Have a buffer size of : %d\n", buffer_size);
+		if (priv->num_rows > priv->buffer_size)
+			ulogd_log(ULOGD_INFO, PFX "commited backlog buffer (%d rows)\n",
+					  priv->num_rows);
 
-	if (sqlite3_exec(dbh,"begin deferred",NULL,NULL,NULL) != SQLITE_OK)
-		ulogd_log(ULOGD_ERROR,"can't create a new transaction\n");
+		delete_all_rows(pi);
 
-	/* create and prepare the actual insert statement */
-	_sqlite3_createstmt();
+		if (priv->overlimit_msg)
+			priv->overlimit_msg = 0;
+
+		return 0;
+	}
+
+ err_rollback:
+	if (sqlite3_errcode(priv->dbh) == SQLITE_LOCKED)
+		return 0;
+
+	sqlite3_exec(priv->dbh, "rollback", NULL, NULL, NULL);
+
+	return -1;
+}
+
+
+/* our main output function, called by ulogd */
+static int
+sqlite3_interp(struct ulogd_pluginstance *pi)
+{
+	struct sqlite3_priv *priv = (void *)pi->private;
+	struct col *cols = priv->cols;
+	struct row *row;
+
+	if (do_reinit) {
+		db_reset(pi);
+
+		if (db_start(pi) < 0)
+			return ULOGD_IRET_STOP;
+
+		do_reinit = 0;
+	}
+
+	if ((row = row_new()) == NULL)
+		return ULOGD_IRET_ERR;
+
+	row->ip_saddr = RKEY(cols[0].key)->u.value.ui32;
+	row->ip_daddr = RKEY(cols[1].key)->u.value.ui32;
+	row->ip_proto = RKEY(cols[2].key)->u.value.ui8;
+	row->l4_dport = RKEY(cols[3].key)->u.value.ui16;
+	row->raw_in_pktlen = RKEY(cols[4].key)->u.value.ui32;
+	row->raw_in_pktcount = RKEY(cols[5].key)->u.value.ui32;
+	row->raw_out_pktlen = RKEY(cols[6].key)->u.value.ui32;
+	row->raw_out_pktcount = RKEY(cols[7].key)->u.value.ui32;
+	row->flow_start_sec = RKEY(cols[9].key)->u.value.ui32;
+	row->flow_duration = RKEY(cols[10].key)->u.value.ui32;
+
+	row_add(priv, row);
+
+	if (priv->num_rows >= priv->buffer_size)
+		db_commit_rows(pi);
+
+	return ULOGD_IRET_OK;
+}
+
+
+static void
+timer_cb(struct ulogd_timer *t)
+{
+	struct ulogd_pluginstance *pi = t->data;
+	struct sqlite3_priv *priv = (void *)pi->private;
+
+	priv->max_rows = max(priv->max_rows, priv->num_rows);
+
+	if (priv->num_rows > 0)
+		db_commit_rows(pi);
+}
+
+
+static int
+sqlite3_configure(struct ulogd_pluginstance *pi,
+				  struct ulogd_pluginstance_stack *stack)
+{
+	struct sqlite3_priv *priv = (void *)pi->private;
+
+	config_parse_file(pi->id, pi->config_kset);
+
+	if (ulogd_wildcard_inputkeys(pi) < 0)
+		return -1;
+
+	if (db_ce(pi) == NULL) {
+		ulogd_error(PFX "configure: no database specified\n");
+		return -1;
+	}
+
+	if (table_ce(pi) == NULL) {
+		ulogd_error(PFX "configure: no table specified\n");
+		return -1;
+	}
+
+	if (timer_ce(pi) <= 0) {
+		ulogd_error(PFX "configure: invalid timer value\n");
+		return -1;
+	}
+
+	if (max_backlog_ce(pi)) {
+		if (max_backlog_ce(pi) <= buffer_ce(pi)) {
+			ulogd_error(PFX "configure: invalid max-backlog value\n");
+			return -1;
+		}
+	}
+
+	priv->max_rows_allowed = max_backlog_ce(pi);
+
+	DEBUGP("%s: db='%s' table='%s' timer=%d max-backlog=%d\n", pi->id,
+		   db_ce(pi), table_ce(pi), timer_ce(pi), max_backlog_ce(pi));
+
+	/* init timer */
+	priv->timer.cb = timer_cb;
+	priv->timer.ival = timer_ce(pi);
+	priv->timer.flags = TIMER_F_PERIODIC;
+	priv->timer.data = pi;
+
+	ulogd_register_timer(&priv->timer);
 
 	return 0;
 }
 
-static ulog_output_t _sqlite3_plugin = { 
-	.name = "sqlite3", 
-	.output = &_sqlite3_output, 
-	.init = &_sqlite3_init,
-	.fini = &_sqlite3_fini,
-};
 
-void _init(void) 
+static int
+sqlite3_start(struct ulogd_pluginstance *pi)
 {
-	register_output(&_sqlite3_plugin);
+	struct sqlite3_priv *priv = (void *)pi->private;
+
+	priv->num_rows = priv->max_rows = 0;
+	TAILQ_INIT(&priv->rows);
+
+	return db_start(pi);
 }
 
-#endif
+
+/* give us an opportunity to close the database down properly */
+static int
+sqlite3_stop(struct ulogd_pluginstance *pi)
+{
+	struct sqlite3_priv *priv = (void *)pi->private;
+
+	/* free up our prepared statements so we can close the db */
+	if (priv->p_stmt) {
+		sqlite3_finalize(priv->p_stmt);
+		DEBUGP("prepared statement finalized\n");
+	}
+
+	if (priv->dbh == NULL)
+		return -1;
+
+	sqlite3_close(priv->dbh);
+
+	return 0;
+}
+
+
+static void
+sqlite3_signal(struct ulogd_pluginstance *pi, int sig)
+{
+	switch (sig) {
+	case SIGUSR1:
+		do_reinit++;
+		break;
+
+	default:
+		break;
+	}
+}
+
+
+static struct ulogd_plugin sqlite3_plugin = { 
+	.name = "SQLITE3", 
+	.input = {
+		.type = ULOGD_DTYPE_PACKET | ULOGD_DTYPE_FLOW,
+	},
+	.output = {
+		.type = ULOGD_DTYPE_SINK,
+	},
+	.config_kset = &sqlite3_kset,
+	.priv_size = sizeof(struct sqlite3_priv),
+	.configure = sqlite3_configure,
+	.start = sqlite3_start,
+	.stop = sqlite3_stop,
+	.signal = sqlite3_signal,
+	.interp = sqlite3_interp,
+	.version = ULOGD_VERSION,
+};
+
+static void init(void) __attribute__((constructor));
+
+static void
+init(void) 
+{
+	ulogd_register_plugin(&sqlite3_plugin);
+
+	ulogd_log(ULOGD_INFO, "using Sqlite version %s\n", sqlite3_libversion());
+}

-- 

[ULOGD RFC 10/30] Initial round to make plugins reconfigurable

[heitzenberger]

[-- Attachment #1: ulogd-allow-plugins-to-be-reconfigurable.diff --]
[-- Type: text/plain, Size: 2982 bytes --]

In order to make ulogd fully reconfigurable without restarting this
patch does some introductory work letting plugins be able to be
flagged as reconfigurable.

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/include/ulogd/ulogd.h
===================================================================
--- ulogd-netfilter.orig/include/ulogd/ulogd.h
+++ ulogd-netfilter/include/ulogd/ulogd.h
@@ -125,6 +125,10 @@ struct ulogd_keyset {
 
 struct ulogd_pluginstance_stack;
 struct ulogd_pluginstance;
+
+/* plugin flags */
+#define ULOGD_PF_RECONF			0x00000001
+
 struct ulogd_plugin {
 	/* global list of plugins */
 	struct llist_head list;
@@ -135,6 +139,8 @@ struct ulogd_plugin {
 	/* ID for this plugin (dynamically assigned) */
 	unsigned int id;
 
+	unsigned flags;
+
 	struct ulogd_keyset input;
 	struct ulogd_keyset output;
 
Index: ulogd-netfilter/src/ulogd.c
===================================================================
--- ulogd-netfilter.orig/src/ulogd.c
+++ ulogd-netfilter/src/ulogd.c
@@ -218,6 +218,23 @@ int ulogd_wildcard_inputkeys(struct ulog
 	return 0;
 }
 
+/*
+  ulogd_pluginstance_reset_cfg() - reset to default config
+*/
+int
+ulogd_pluginstance_reset_cfg(const struct ulogd_pluginstance *pi)
+{
+	size_t size;
+
+	assert(pi->plugin != NULL);
+
+	size = sizeof(struct config_keyset)
+		+ pi->plugin->config_kset->num_ces * sizeof(struct config_entry);
+
+	memcpy(pi->config_kset, pi->plugin->config_kset, size);
+
+	return 0;
+}
 
 /***********************************************************************
  * PLUGIN MANAGEMENT 
@@ -823,6 +840,69 @@ for_each_pluginstance(int (* cb)(struct 
 	return sum;
 }
 
+enum ReconfOp
+{
+	INVAL = 0,
+	STOP,
+	CONFIGURE,
+	START,
+};
+
+static int
+_do_reconf(struct ulogd_pluginstance *pi,
+		   struct ulogd_pluginstance_stack *stack, void *arg)
+{
+	enum ReconfOp op = (unsigned)arg;
+	int ret = 0;
+
+	assert(pi != NULL);
+
+	if ((pi->plugin->flags & ULOGD_PF_RECONF) == 0)
+		return 0;
+
+	switch (op) {
+	case STOP:
+		ret = pi->plugin->stop(pi);
+		break;
+
+	case CONFIGURE:
+		ulogd_pluginstance_reset_cfg(pi);
+		ret = pi->plugin->configure(pi, stack);
+		break;
+
+	case START:
+		ret = pi->plugin->start(pi);
+		break;
+
+	default:
+		return -1;
+	}
+
+	if (ret < 0) {
+		ulogd_log(ULOGD_FATAL, "reconfiguring '%s' failed\n", pi->id);
+		return -1;
+	}
+
+	return 1;
+}
+
+static int
+reconfigure_plugins(void)
+{
+	ulogd_log(ULOGD_INFO, "reconfiguring plugins\n");
+
+	if (for_each_pluginstance(_do_reconf, (void *)STOP) < 0)
+		abort();
+
+	if (for_each_pluginstance(_do_reconf, (void *)CONFIGURE) < 0)
+		abort();
+
+	if (for_each_pluginstance(_do_reconf, (void *)START) < 0)
+		abort();
+
+	return 0;
+}
+
 static int
 _do_signal(struct ulogd_pluginstance *pi,
 		   struct ulogd_pluginstance_stack *stack, void *arg)
@@ -845,6 +925,7 @@ sync_sig_handler(int signo)
 
 	switch (signo) {
 	case SIGHUP:
+		reconfigure_plugins();
 		break;
 
 	case SIGALRM:

-- 

[ULOGD RFC 11/30] NFCT: make reconfigurable

[heitzenberger]

[-- Attachment #1: ulogd-NFCT-make-reconfigurable.diff --]
[-- Type: text/plain, Size: 5401 bytes --]

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/input/flow/ulogd_inpflow_NFCT.c
===================================================================
--- ulogd-netfilter.orig/input/flow/ulogd_inpflow_NFCT.c
+++ ulogd-netfilter/input/flow/ulogd_inpflow_NFCT.c
@@ -33,10 +33,10 @@
 
 #include <sys/time.h>
 #include <time.h>
-#include <ulogd/linuxlist.h>
 
 #include <ulogd/ulogd.h>
 #include <ulogd/common.h>
+#include <ulogd/linuxlist.h>
 #include <ulogd/ipfix_protocol.h>
 
 #include <libnetfilter_conntrack/libnetfilter_conntrack.h>
@@ -464,6 +464,7 @@ htable_alloc(int htable_size)
 
 	htable->buckets = (void *)htable + sizeof(*htable);
 	htable->num_buckets = htable_size;
+	htable->curr_bucket = 0;
 	htable->used = 0;
 
 	for (i = 0; i < htable->num_buckets; i++)
@@ -735,7 +736,7 @@ read_cb_nfct(int fd, unsigned what, void
 #define STOP_HERE(h)	(((h)->curr_bucket + 16) % (h)->num_buckets)
 
 static void
-timer_cb(struct ulogd_timer *t)
+nfct_timer_cb(struct ulogd_timer *t)
 {
 	struct ulogd_pluginstance *pi = t->data;
 	struct nfct_pluginstance *priv = (void *)pi->private;
@@ -754,14 +755,19 @@ timer_cb(struct ulogd_timer *t)
 
 		ht->curr_bucket = (ht->curr_bucket + 1) % ht->num_buckets;
 	} while (ht->curr_bucket != end);
+
+	pr_debug("%s: now=%ld\n", __func__, t_now);
 }
 
-static int configure_nfct(struct ulogd_pluginstance *upi,
-			  struct ulogd_pluginstance_stack *stack)
+static int
+nfct_configure(struct ulogd_pluginstance *upi,
+			   struct ulogd_pluginstance_stack *stack)
 {
 	struct nfct_pluginstance *priv = (void *)upi->private;
 	int ret;
 
+	pr_debug("%s: pi=%p\n", __func__, upi);
+
 	memset(priv, 0, sizeof(struct nfct_pluginstance));
 	
 	ret = config_parse_file(upi->id, upi->config_kset);
@@ -772,68 +778,87 @@ static int configure_nfct(struct ulogd_p
 }
 
 
-static int constructor_nfct(struct ulogd_pluginstance *upi)
+static int
+nfct_start(struct ulogd_pluginstance *upi)
 {
-	struct nfct_pluginstance *cpi = (void *)upi->private;
+	struct nfct_pluginstance *priv = (void *)upi->private;
 
-	cpi->cth = nfct_open(NFNL_SUBSYS_CTNETLINK, CT_EVENTS);
-	if (!cpi->cth) {
-		ulogd_log(ULOGD_FATAL, "error opening ctnetlink\n");
+	pr_debug("%s: pi=%p\n", __func__, upi);
+
+	priv->htable = htable_alloc(buckets_ce(upi->config_kset).u.value);
+	if (priv->htable == NULL) {
+		ulogd_log(ULOGD_FATAL, "%s: out of memory\n", upi->id);
 		return -1;
 	}
 
-	cpi->nfct_fd.fd = nfct_fd(cpi->cth);
-	cpi->nfct_fd.cb = &read_cb_nfct;
-	cpi->nfct_fd.data = upi;
-	cpi->nfct_fd.when = ULOGD_FD_READ;
-
-	ulogd_register_fd(&cpi->nfct_fd);
-
-	cpi->htable = htable_alloc(buckets_ce(upi->config_kset).u.value);
-	if (cpi->htable == NULL) {
-		ulogd_log(ULOGD_FATAL, "htable_alloc: out of memory\n");
+	priv->cth = nfct_open(NFNL_SUBSYS_CTNETLINK, CT_EVENTS);
+	if (priv->cth == NULL) {
+		ulogd_log(ULOGD_FATAL, "error opening ctnetlink\n");
+		goto err_free;
+	}
 
-		nfct_close(cpi->cth);
-		cpi->cth = NULL;
+	priv->nfct_fd.fd = nfct_fd(priv->cth);
+	priv->nfct_fd.cb = &read_cb_nfct;
+	priv->nfct_fd.data = upi;
+	priv->nfct_fd.when = ULOGD_FD_READ;
 
-		return -1;
-	}
+	if (ulogd_register_fd(&priv->nfct_fd) < 0)
+		goto err_nfct_close;
 
-	cpi->timer.cb = timer_cb;
-	cpi->timer.ival = 1 SEC;
-	cpi->timer.flags = TIMER_F_PERIODIC;
-	cpi->timer.data = upi;
+	priv->timer.cb = nfct_timer_cb;
+	priv->timer.ival = 1 SEC;
+	priv->timer.flags = TIMER_F_PERIODIC;
+	priv->timer.data = upi;
 
-	ulogd_register_timer(&cpi->timer);
+	if (ulogd_register_timer(&priv->timer) < 0)
+		goto err_unreg_fd;
+
+	ulogd_log(ULOGD_INFO, "%s: started\n", upi->id);
 
-	ulogd_log(ULOGD_INFO, "%s: hashsize %u\n", upi->id,
-			  cpi->htable->num_buckets);
-	
 	return 0;
+
+ err_unreg_fd:
+	ulogd_unregister_fd(&priv->nfct_fd);
+ err_nfct_close:
+	nfct_close(priv->cth);
+	priv->cth = NULL;
+ err_free:
+	htable_free(priv->htable);
+	priv->htable = NULL;
+
+	return -1;
 }
 
-static int destructor_nfct(struct ulogd_pluginstance *pi)
+static int
+nfct_stop(struct ulogd_pluginstance *pi)
 {
-	struct nfct_pluginstance *cpi = (void *) pi;
-	
-	nfct_close(cpi->cth);
-	cpi->cth = NULL;
+	struct nfct_pluginstance *priv = (void *)pi->private;
 
-	htable_free(cpi->htable);
+	pr_debug("%s: pi=%p\n", __func__, pi);
 
-	return 0;
-}
+	if (priv->htable == NULL)
+		return 0;				/* already stopped */
 
-static void signal_nfct(struct ulogd_pluginstance *pi, int signal)
-{
-	switch (signal) {
-	case SIGUSR2:
-		break;
+	ulogd_unregister_timer(&priv->timer);
+
+	ulogd_unregister_fd(&priv->nfct_fd);
+
+	if (priv->cth != NULL) {
+		nfct_close(priv->cth);
+		priv->cth = NULL;
 	}
+
+	if (priv->htable != NULL) {
+		htable_free(priv->htable);
+		priv->htable = NULL;
+	}
+
+	return 0;
 }
 
 static struct ulogd_plugin nfct_plugin = {
 	.name = "NFCT",
+	.flags = ULOGD_PF_RECONF,
 	.input = {
 		.type = ULOGD_DTYPE_SOURCE,
 	},
@@ -844,17 +869,17 @@ static struct ulogd_plugin nfct_plugin =
 	},
 	.config_kset 	= &nfct_kset,
 	.interp 	= NULL,
-	.configure	= &configure_nfct,
-	.start		= &constructor_nfct,
-	.stop		= &destructor_nfct,
-	.signal		= &signal_nfct,
+	.configure	= nfct_configure,
+	.start		= nfct_start,
+	.stop		= nfct_stop,
 	.priv_size	= sizeof(struct nfct_pluginstance),
 	.version	= ULOGD_VERSION,
 };
 
 void __attribute__ ((constructor)) init(void);
 
-void init(void)
+void
+init(void)
 {
 	ulogd_register_plugin(&nfct_plugin);
 }

-- 

NFCT ulogd pluging [was Re: [ULOGD RFC 11/30] NFCT: make reconfigurable]

[Pablo Neira Ayuso]

Hi,

JFYI: conntrackd also does connection tracking accounting so I'm not
sure if ulogd should do this as well via the NFCT plugin. This looks
like duplicated effort IMO and, as for now, conntrackd also provides
userspace conntrack table dumping without locking the kernel via
`conntrackd -i' which is a feature that ulogd does not provide.

I'm not sure if the effort is worth, Harald?

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: NFCT ulogd pluging [was Re: [ULOGD RFC 11/30] NFCT: make reconfigurable]

[Patrick McHardy]

Pablo Neira Ayuso wrote:
> Hi,
>
> JFYI: conntrackd also does connection tracking accounting so I'm not
> sure if ulogd should do this as well via the NFCT plugin. This looks
> like duplicated effort IMO and, as for now, conntrackd also provides
> userspace conntrack table dumping without locking the kernel via
> `conntrackd -i' which is a feature that ulogd does not provide.
>   

Mhh .. I'm not sure how that is duplicated effort, we're talking about
two different programs that do completely different things.

Locking is not an issue BTW, I have patches queued which finally
move conntrack to RCU.

Re: NFCT ulogd pluging [was Re: [ULOGD RFC 11/30] NFCT: make reconfigurable]

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 1362 bytes --]

On Wed, Jan 30, 2008 at 08:30:43PM +0100, Pablo Neira Ayuso wrote:
> Hi,
> 
> JFYI: conntrackd also does connection tracking accounting so I'm not
> sure if ulogd should do this as well via the NFCT plugin. This looks
> like duplicated effort IMO and, as for now, conntrackd also provides
> userspace conntrack table dumping without locking the kernel via
> `conntrackd -i' which is a feature that ulogd does not provide.
> 
> I'm not sure if the effort is worth, Harald?

I think it is still worth having this in ulogd2, since ulogd2 is an
entire framework for all kinds of logging, including
input/filter/aggregator/output plugins where you can build IPFIX
compliant network-wide accounting/logging systems that are interoperable
with different vendors.

So conntrackd might be fine for the special case of locally logging into
a file or piping the output through some script.  But for all the more
sophisticated scenarios, ulogd2 is the solution of choice, I'd say.

-- 
- Harald Welte <laforge@netfilter.org>                 http://netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: NFCT ulogd pluging [was Re: [ULOGD RFC 11/30] NFCT: make reconfigurable]

[Holger Eitzenberger]

Pablo Neira Ayuso wrote:

> JFYI: conntrackd also does connection tracking accounting so I'm not
> sure if ulogd should do this as well via the NFCT plugin. This looks
> like duplicated effort IMO and, as for now, conntrackd also provides
> userspace conntrack table dumping without locking the kernel via
> `conntrackd -i' which is a feature that ulogd does not provide.

Hi Pablo,

yes, I know that conntrackd does that, but for a different audience. 
Also, can conntrackd write a SQLITE database too?

IMO the strong point in favour of ulogd is its flexibility.  Want 
Postgres as a backend?  Or a plain text file?  Not a problem, just 
change the config file.

   /holger

Re: NFCT ulogd pluging [was Re: [ULOGD RFC 11/30] NFCT: make reconfigurable]

[Pablo Neira Ayuso]

Holger Eitzenberger wrote:
> Pablo Neira Ayuso wrote:
> 
>> JFYI: conntrackd also does connection tracking accounting so I'm not
>> sure if ulogd should do this as well via the NFCT plugin. This looks
>> like duplicated effort IMO and, as for now, conntrackd also provides
>> userspace conntrack table dumping without locking the kernel via
>> `conntrackd -i' which is a feature that ulogd does not provide.
> 
> yes, I know that conntrackd does that, but for a different audience.
> Also, can conntrackd write a SQLITE database too?
> 
> IMO the strong point in favour of ulogd is its flexibility.  Want
> Postgres as a backend?  Or a plain text file?  Not a problem, just
> change the config file.

Indeed. This makes sense.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

[ULOGD RFC 12/30] SQLITE3: make reconfigurable

[heitzenberger]

[-- Attachment #1: ulogd-SQLITE3-make-reconfigurable.diff --]
[-- Type: text/plain, Size: 3994 bytes --]

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/output/sqlite3/ulogd_output_SQLITE3.c
===================================================================
--- ulogd-netfilter.orig/output/sqlite3/ulogd_output_SQLITE3.c
+++ ulogd-netfilter/output/sqlite3/ulogd_output_SQLITE3.c
@@ -50,11 +50,6 @@
 /* number of colums we have (really should be configurable) */
 #define DB_NUM_COLS	10
 
-#if 0
-#define DEBUGP(x, args...)	fprintf(stderr, x, ## args)
-#else
-#define DEBUGP(x, args...)
-#endif
 
 /* map DB column to ulogd key */
 struct col {
@@ -245,7 +240,7 @@ db_createstmt(struct ulogd_pluginstance 
 	sprintf(stmt_pos, "?)");
 	ulogd_log(ULOGD_DEBUG, "%s: stmt='%s'\n", pi->id, priv->stmt);
 
-	DEBUGP("about to prepare statement.\n");
+	pr_debug("about to prepare statement.\n");
 
 	sqlite3_prepare(priv->dbh, priv->stmt, -1, &priv->p_stmt, 0);
 	if (priv->p_stmt == NULL) {
@@ -253,7 +248,7 @@ db_createstmt(struct ulogd_pluginstance 
 		return 1;
 	}
 
-	DEBUGP("statement prepared.\n");
+	pr_debug("statement prepared.\n");
 
 	return 0;
 }
@@ -346,7 +341,7 @@ db_init(struct ulogd_pluginstance *pi)
 		while ((underscore = strchr(buf, '_')) != NULL)
 			*underscore = '.';
 
-		DEBUGP("column '%s' found\n", buf);
+		pr_debug("column '%s' found\n", buf);
 
 		strncpy(col->name, buf, ULOGD_MAX_KEYLEN);
 
@@ -589,15 +584,6 @@ sqlite3_interp(struct ulogd_pluginstance
 	struct col *cols = priv->cols;
 	struct row *row;
 
-	if (do_reinit) {
-		db_reset(pi);
-
-		if (db_start(pi) < 0)
-			return ULOGD_IRET_STOP;
-
-		do_reinit = 0;
-	}
-
 	if ((row = row_new()) == NULL)
 		return ULOGD_IRET_ERR;
 
@@ -622,7 +608,7 @@ sqlite3_interp(struct ulogd_pluginstance
 
 
 static void
-timer_cb(struct ulogd_timer *t)
+sqlite_timer_cb(struct ulogd_timer *t)
 {
 	struct ulogd_pluginstance *pi = t->data;
 	struct sqlite3_priv *priv = (void *)pi->private;
@@ -669,11 +655,11 @@ sqlite3_configure(struct ulogd_pluginsta
 
 	priv->max_rows_allowed = max_backlog_ce(pi);
 
-	DEBUGP("%s: db='%s' table='%s' timer=%d max-backlog=%d\n", pi->id,
-		   db_ce(pi), table_ce(pi), timer_ce(pi), max_backlog_ce(pi));
+	pr_debug("%s: db='%s' table='%s' timer=%d max-backlog=%d\n", pi->id,
+			 db_ce(pi), table_ce(pi), timer_ce(pi), max_backlog_ce(pi));
 
 	/* init timer */
-	priv->timer.cb = timer_cb;
+	priv->timer.cb = sqlite_timer_cb;
 	priv->timer.ival = timer_ce(pi);
 	priv->timer.flags = TIMER_F_PERIODIC;
 	priv->timer.data = pi;
@@ -689,10 +675,17 @@ sqlite3_start(struct ulogd_pluginstance 
 {
 	struct sqlite3_priv *priv = (void *)pi->private;
 
+	pr_debug("%s: pi=%p\n", __func__, pi);
+
 	priv->num_rows = priv->max_rows = 0;
 	TAILQ_INIT(&priv->rows);
 
-	return db_start(pi);
+	if (db_start(pi) < 0)
+		return -1;
+
+	ulogd_log(ULOGD_INFO, "%s: started\n", pi->id);
+
+	return 0;
 }
 
 
@@ -702,16 +695,12 @@ sqlite3_stop(struct ulogd_pluginstance *
 {
 	struct sqlite3_priv *priv = (void *)pi->private;
 
-	/* free up our prepared statements so we can close the db */
-	if (priv->p_stmt) {
-		sqlite3_finalize(priv->p_stmt);
-		DEBUGP("prepared statement finalized\n");
-	}
+	pr_debug("%s: pi=%p\n", __func__, pi);
 
 	if (priv->dbh == NULL)
-		return -1;
+		return 0;				/* already stopped */
 
-	sqlite3_close(priv->dbh);
+	db_reset(pi);
 
 	return 0;
 }
@@ -720,9 +709,18 @@ sqlite3_stop(struct ulogd_pluginstance *
 static void
 sqlite3_signal(struct ulogd_pluginstance *pi, int sig)
 {
+	struct sqlite3_priv *priv = (void *)pi->private;
+
 	switch (sig) {
 	case SIGUSR1:
-		do_reinit++;
+		if (priv->dbh != NULL) {
+			db_reset(pi);
+
+			if (db_start(pi) < 0) {
+				ulogd_log(ULOGD_FATAL, "%s: database reset failed\n", pi->id);
+				exit(EXIT_FAILURE);
+			}
+		}
 		break;
 
 	default:
@@ -732,7 +730,8 @@ sqlite3_signal(struct ulogd_pluginstance
 
 
 static struct ulogd_plugin sqlite3_plugin = { 
-	.name = "SQLITE3", 
+	.name = "SQLITE3",
+	.flags = ULOGD_PF_RECONF,
 	.input = {
 		.type = ULOGD_DTYPE_PACKET | ULOGD_DTYPE_FLOW,
 	},

-- 

[ULOGD RFC 13/30] NFCT: add disable switch

[heitzenberger]

[-- Attachment #1: ulogd-NFCT-add-disable-switch.diff --]
[-- Type: text/plain, Size: 2386 bytes --]

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/input/flow/ulogd_inpflow_NFCT.c
===================================================================
--- ulogd-netfilter.orig/input/flow/ulogd_inpflow_NFCT.c
+++ ulogd-netfilter/input/flow/ulogd_inpflow_NFCT.c
@@ -73,6 +73,7 @@ struct nfct_pluginstance {
 	struct ulogd_fd nfct_fd;
 	struct ct_htable *htable;
 	struct ulogd_timer timer;
+	unsigned disable : 1;
 	struct {
 		unsigned nl_err;
 		unsigned nl_ovr;
@@ -82,7 +83,7 @@ struct nfct_pluginstance {
 #define HTABLE_SIZE	(512)
 
 static struct config_keyset nfct_kset = {
-	.num_ces = 3,
+	.num_ces = 4,
 	.ces = {
 		{
 			.key	 = "pollinterval",
@@ -102,10 +103,18 @@ static struct config_keyset nfct_kset = 
 			.options = CONFIG_OPT_NONE,
 			.u.value = 0,
 		},
+		{
+			.key	 = "disable",
+			.type	 = CONFIG_TYPE_INT,
+			.options = CONFIG_OPT_NONE,
+			.u.value = 0,
+		},
 	},
 };
 #define pollint_ce(x)	(x->ces[0])
 #define buckets_ce(x)	(x->ces[1])
+#define hash_max_entries(x)		((x)->ces[2])
+#define disable_ce(x)	(x->ces[3])
 
 enum {
 	O_IP_SADDR = 0,
@@ -774,6 +783,8 @@ nfct_configure(struct ulogd_pluginstance
 	if (ret < 0)
 		return ret;
 
+	priv->disable = disable_ce(upi->config_kset).u.value;
+
 	return 0;
 }
 
@@ -785,6 +796,11 @@ nfct_start(struct ulogd_pluginstance *up
 
 	pr_debug("%s: pi=%p\n", __func__, upi);
 
+	if (priv->disable) {
+		ulogd_log(ULOGD_INFO, "%s: disabled\n", upi->id);
+		return 0;
+	}
+
 	priv->htable = htable_alloc(buckets_ce(upi->config_kset).u.value);
 	if (priv->htable == NULL) {
 		ulogd_log(ULOGD_FATAL, "%s: out of memory\n", upi->id);
@@ -797,6 +813,8 @@ nfct_start(struct ulogd_pluginstance *up
 		goto err_free;
 	}
 
+	ulogd_log(ULOGD_DEBUG, "%s: ctnetlink connection opened\n", upi->id);
+
 	priv->nfct_fd.fd = nfct_fd(priv->cth);
 	priv->nfct_fd.cb = &read_cb_nfct;
 	priv->nfct_fd.data = upi;
@@ -836,6 +854,9 @@ nfct_stop(struct ulogd_pluginstance *pi)
 
 	pr_debug("%s: pi=%p\n", __func__, pi);
 
+	if (priv->disable)
+		return 0;				/* wasn't started */
+
 	if (priv->htable == NULL)
 		return 0;				/* already stopped */
 
@@ -848,6 +869,8 @@ nfct_stop(struct ulogd_pluginstance *pi)
 		priv->cth = NULL;
 	}
 
+	ulogd_log(ULOGD_DEBUG, "%s: ctnetlink connection closed\n", pi->id);
+
 	if (priv->htable != NULL) {
 		htable_free(priv->htable);
 		priv->htable = NULL;

-- 

[ULOGD RFC 14/30] SQLITLE3: add disable switch

[heitzenberger]

[-- Attachment #1: ulogd-SQLITE3-add-disable-switch.diff --]
[-- Type: text/plain, Size: 4813 bytes --]

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/output/sqlite3/ulogd_output_SQLITE3.c
===================================================================
--- ulogd-netfilter.orig/output/sqlite3/ulogd_output_SQLITE3.c
+++ ulogd-netfilter/output/sqlite3/ulogd_output_SQLITE3.c
@@ -95,13 +95,13 @@ struct sqlite3_priv {
 	int max_rows;				/* number of rows actually seen */
 	int max_rows_allowed;
 
+	unsigned disable : 1;
 	unsigned overlimit_msg : 1;
 };
 
 
-static int do_reinit;
 static struct config_keyset sqlite3_kset = {
-	.num_ces = 5,
+	.num_ces = 6,
 	.ces = {
 		{
 			.key = "db",
@@ -131,6 +131,12 @@ static struct config_keyset sqlite3_kset
 			.options = CONFIG_OPT_NONE,
 			.u.value = CFG_MAX_BACKLOG_DEFAULT,
 		},
+		{
+			.key = "disable",
+			.type = CONFIG_TYPE_INT,
+			.options = CONFIG_OPT_NONE,
+			.u.value = 0,
+		},
 	},
 };
 
@@ -139,6 +145,7 @@ static struct config_keyset sqlite3_kset
 #define buffer_ce(pi)	(pi)->config_kset->ces[2].u.value
 #define timer_ce(pi)	(pi)->config_kset->ces[3].u.value
 #define max_backlog_ce(pi)	(pi)->config_kset->ces[4].u.value
+#define disable_ce(pi)	(pi)->config_kset->ces[5].u.value
 
 
 #define SQL_CREATE_STR \
@@ -248,7 +255,7 @@ db_createstmt(struct ulogd_pluginstance 
 		return 1;
 	}
 
-	pr_debug("statement prepared.\n");
+	pr_debug("%s: statement prepared.\n", pi->id);
 
 	return 0;
 }
@@ -322,7 +329,7 @@ db_init(struct ulogd_pluginstance *pi)
 
 	num_cols = db_count_cols(pi, &schema_stmt);
 	if (num_cols != DB_NUM_COLS) {
-		ulogd_log(ULOGD_INFO, PFX "(re)creating database\n");
+		ulogd_log(ULOGD_INFO, "%s: (re)creating database\n", pi->id);
 
 		if (db_create_tbl(pi) < 0)
 			return -1;
@@ -351,7 +358,7 @@ db_init(struct ulogd_pluginstance *pi)
 		}
 	}
 
-	ulogd_log(ULOGD_INFO, PFX "database successfully opened\n");
+	ulogd_log(ULOGD_INFO, "%s: database opened\n", pi->id);
 
 	if (sqlite3_finalize(schema_stmt) != SQLITE_OK) {
 		ulogd_error(PFX "sqlite_finalize: %s\n",
@@ -383,7 +390,7 @@ db_start(struct ulogd_pluginstance *pi)
 {
 	struct sqlite3_priv *priv = (void *)pi->private;
 
-	ulogd_log(ULOGD_DEBUG, PFX "opening database connection\n");
+	ulogd_log(ULOGD_DEBUG, "%s: opening database connection\n", pi->id);
 
 	if (sqlite3_open(db_ce(pi), &priv->dbh) != SQLITE_OK) {
 		ulogd_error(PFX "%s\n", sqlite3_errmsg(priv->dbh));
@@ -538,7 +545,8 @@ db_commit_rows(struct ulogd_pluginstance
 		if (sqlite3_errcode(priv->dbh) == SQLITE_LOCKED)
 			return 0;			/* perform commit later */
 	
-		ulogd_error(PFX "begin transaction: %s\n", sqlite3_errmsg(priv->dbh));
+		ulogd_error("%s: begin transaction: %s\n", pi->id,
+					sqlite3_errmsg(priv->dbh));
 
 		return -1;
 	}
@@ -555,8 +563,8 @@ db_commit_rows(struct ulogd_pluginstance
 		sqlite3_reset(priv->p_stmt);
 
 		if (priv->num_rows > priv->buffer_size)
-			ulogd_log(ULOGD_INFO, PFX "commited backlog buffer (%d rows)\n",
-					  priv->num_rows);
+			ulogd_log(ULOGD_INFO, "%s: commited backlog buffer (%d rows)\n",
+					  pi->id, priv->num_rows);
 
 		delete_all_rows(pi);
 
@@ -626,34 +634,38 @@ sqlite3_configure(struct ulogd_pluginsta
 {
 	struct sqlite3_priv *priv = (void *)pi->private;
 
+	memset(priv, 0, sizeof(struct sqlite3_priv));
+	
 	config_parse_file(pi->id, pi->config_kset);
 
 	if (ulogd_wildcard_inputkeys(pi) < 0)
 		return -1;
 
 	if (db_ce(pi) == NULL) {
-		ulogd_error(PFX "configure: no database specified\n");
+		ulogd_error("%s: configure: no database specified\n", pi->id);
 		return -1;
 	}
 
 	if (table_ce(pi) == NULL) {
-		ulogd_error(PFX "configure: no table specified\n");
+		ulogd_error("%s: configure: no table specified\n", pi->id);
 		return -1;
 	}
 
 	if (timer_ce(pi) <= 0) {
-		ulogd_error(PFX "configure: invalid timer value\n");
+		ulogd_error("%s: configure: invalid timer value\n", pi->id);
 		return -1;
 	}
 
 	if (max_backlog_ce(pi)) {
 		if (max_backlog_ce(pi) <= buffer_ce(pi)) {
-			ulogd_error(PFX "configure: invalid max-backlog value\n");
+			ulogd_error("%s: configure: invalid max-backlog value\n",
+						pi->id);
 			return -1;
 		}
 	}
 
 	priv->max_rows_allowed = max_backlog_ce(pi);
+	priv->disable = disable_ce(pi);
 
 	pr_debug("%s: db='%s' table='%s' timer=%d max-backlog=%d\n", pi->id,
 			 db_ce(pi), table_ce(pi), timer_ce(pi), max_backlog_ce(pi));
@@ -677,6 +689,11 @@ sqlite3_start(struct ulogd_pluginstance 
 
 	pr_debug("%s: pi=%p\n", __func__, pi);
 
+	if (priv->disable) {
+		ulogd_log(ULOGD_NOTICE, "%s: disabled\n", pi->id);
+		return 0;
+	}
+
 	priv->num_rows = priv->max_rows = 0;
 	TAILQ_INIT(&priv->rows);
 
@@ -697,6 +714,9 @@ sqlite3_stop(struct ulogd_pluginstance *
 
 	pr_debug("%s: pi=%p\n", __func__, pi);
 
+	if (priv->disable)
+		return 0;				/* wasn't started */
+
 	if (priv->dbh == NULL)
 		return 0;				/* already stopped */
 

-- 

[ULOGD RFC 15/30] NFCT: add sequence cache

[heitzenberger]

[-- Attachment #1: ulogd-NFCT-add-seq-cache.diff --]
[-- Type: text/plain, Size: 25555 bytes --]

On large sites it turns out that quite some time is spent in
ct_hash_find_seq() sequentially going over conntrack cache, which
sadly is indexed by tuple and not by netlink seq#.

I solve that by replacing the generic cache by a tuple cache and
a sequence cache.  The sequence cache is then used during garbage
collection to easily find the conntrack in question by seq#.

To keep the code maintainable I generalize out the cache and use it
then later for use by both cache implementations.

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/input/flow/ulogd_inpflow_NFCT.c
===================================================================
--- ulogd-netfilter.orig/input/flow/ulogd_inpflow_NFCT.c
+++ ulogd-netfilter/input/flow/ulogd_inpflow_NFCT.c
@@ -25,19 +25,13 @@
  * 	  the messages via IPFX to one aggregator who then runs ulogd with a 
  * 	  network wide connection hash table.
  */
-
-#include <stdlib.h>
-#include <stdbool.h>
-#include <string.h>
-#include <errno.h>
-
-#include <sys/time.h>
-#include <time.h>
-
 #include <ulogd/ulogd.h>
 #include <ulogd/common.h>
 #include <ulogd/linuxlist.h>
 #include <ulogd/ipfix_protocol.h>
+#include <time.h>
+#include <sys/time.h>
+#include <netinet/in.h>
 
 #include <libnetfilter_conntrack/libnetfilter_conntrack.h>
 #include <linux/netfilter/nf_conntrack_tcp.h>
@@ -47,74 +41,79 @@
 						 | NF_NETLINK_CONNTRACK_UPDATE \
 						 | NF_NETLINK_CONNTRACK_DESTROY)
 
-#undef INADDR_LOOPBACK
-#define INADDR_LOOPBACK		0x0100007f /* 127.0.0.1 */
-
+/* configuration defaults */
+#define TCACHE_SIZE       8192
+#define SCACHE_SIZE       64
+#define SCACHE_REQ_MAX    100
+#define TIMEOUT           30 SEC
 
-typedef enum TIMES_ { START, STOP, __TIME_MAX } TIMES;
+typedef enum TIMES_ { START, UPDATE, STOP, __TIME_MAX } TIMES;
 typedef unsigned conntrack_hash_t;
 
-struct ct_timestamp {
+struct conntrack {
 	struct llist_head list;
+	struct llist_head seq_link;
 	struct nfct_tuple tuple;
 	unsigned last_seq;
 	struct timeval time[__TIME_MAX];
+	time_t t_req;
+	unsigned used;
 };
 
-struct ct_htable {
-	struct llist_head *buckets;
-	unsigned num_buckets;
-	unsigned curr_bucket;
-	unsigned used;
+struct cache_head {
+	struct llist_head link;
+	unsigned cnt;
+};
+ 
+struct cache {
+	struct cache_head *c_head;
+	unsigned c_num_heads;
+	unsigned c_curr_head;
+	unsigned c_cnt;
+	conntrack_hash_t (* c_hash)(struct cache *, struct conntrack *);
+	int (* c_add)(struct cache *, struct conntrack *);
+	int (* c_del)(struct cache *, struct conntrack *);
 };
 
 struct nfct_pluginstance {
 	struct nfct_handle *cth;
 	struct ulogd_fd nfct_fd;
-	struct ct_htable *htable;
+	struct cache *tcache;       /* tuple cache */
+	struct cache *scache;       /* sequence cache */
 	struct ulogd_timer timer;
-	unsigned disable : 1;
 	struct {
 		unsigned nl_err;
 		unsigned nl_ovr;
 	} stats;
 };
 
-#define HTABLE_SIZE	(512)
-
+static unsigned num_conntrack;
 static struct config_keyset nfct_kset = {
-	.num_ces = 4,
+	.num_ces = 3,
 	.ces = {
 		{
-			.key	 = "pollinterval",
-			.type	 = CONFIG_TYPE_INT,
-			.options = CONFIG_OPT_NONE,
-			.u.value = 0,
-		},
-		{
 			.key	 = "hash_buckets",
 			.type	 = CONFIG_TYPE_INT,
 			.options = CONFIG_OPT_NONE,
-			.u.value = HTABLE_SIZE,
+			.u.value = TCACHE_SIZE,
 		},
 		{
-			.key	 = "hash_max_entries",
+			.key	 = "disable",
 			.type	 = CONFIG_TYPE_INT,
 			.options = CONFIG_OPT_NONE,
 			.u.value = 0,
 		},
 		{
-			.key	 = "disable",
+			.key	 = "timeout",
 			.type	 = CONFIG_TYPE_INT,
 			.options = CONFIG_OPT_NONE,
-			.u.value = 0,
+			.u.value = TIMEOUT,
 		},
 	},
 };
-#define pollint_ce(x)	(x->ces[0])
-#define buckets_ce(x)	(x->ces[1])
-#define hash_max_entries(x)		((x)->ces[2])
-#define disable_ce(x)	(x->ces[3])
+#define buckets_ce(pi)    ((pi)->config_kset->ces[0].u.value)
+#define disable_ce(pi)    ((pi)->config_kset->ces[1].u.value)
+#define timeout_ce(pi)    ((pi)->config_kset->ces[2].u.value)
 
 enum {
 	O_IP_SADDR = 0,
@@ -311,9 +310,11 @@ static struct ulogd_key nfct_okeys[__O_M
 
 
 /* forward declarations */
-static struct ct_timestamp * ct_hash_find_seq(const struct ct_htable *,
-											  unsigned);
-static void ct_hash_free(struct ct_htable *, struct ct_timestamp *);
+static int cache_del(struct cache *, struct conntrack *);
+static struct conntrack *tcache_find(const struct ulogd_pluginstance *,
+									 const struct nfct_tuple *);
+static struct conntrack *scache_find(const struct ulogd_pluginstance *,
+									 unsigned);
 
 
 static int
@@ -321,16 +322,25 @@ nl_error(struct ulogd_pluginstance *pi, 
 {
 	struct nfct_pluginstance *priv = (void *)pi->private;
 	struct nlmsgerr *e = NLMSG_DATA(nlh);
-	struct ct_timestamp *ts;
+	struct conntrack *ct;
 
-	ts = ct_hash_find_seq(priv->htable, e->msg.nlmsg_seq);
-	if (ts == NULL)
+	if (e->msg.nlmsg_seq == 0)
+		return 0;
+
+	ct = scache_find(pi, e->msg.nlmsg_seq);
+	if (ct == NULL)
 		return 0;						/* already gone */
 
 	switch (-e->error) {
 	case ENOENT:
 		/* destroy message was lost (FIXME log all what we got) */
-		ct_hash_free(priv->htable, ts);
+		if (ct->used > 1) {
+			struct conntrack *ct_tmp = tcache_find(pi, &ct->tuple);
+
+			if (ct == ct_tmp)
+				cache_del(priv->tcache, ct);
+		}
+		cache_del(priv->scache, ct);
 		break;
 
 	case 0:								/* "Success" */
@@ -381,7 +391,7 @@ nfnl_recv_msgs(struct nfnl_handle *nfnlh
 
 			if (nlh->nlmsg_type == NLMSG_OVERRUN)
 				priv->stats.nl_ovr++;	/* continue?  payload? */
-				
+
 			(cb)(nlh, pi);
 
 			nlh = NLMSG_NEXT(nlh, nread);
@@ -440,206 +450,315 @@ nfct_get_conntrack_x(struct nfct_handle 
 	return nbytes;
 }
 
+struct conntrack *
+ct_alloc(const struct nfct_tuple *tuple)
+{
+	struct conntrack *ct;
+
+	if ((ct = calloc(1, sizeof(struct conntrack))) == NULL)
+		return NULL;
+
+	memcpy(&ct->tuple, tuple, sizeof(struct nfct_tuple));
+
+	num_conntrack++;
+
+	return ct;
+}
+
+static inline void
+ct_get(struct conntrack *ct)
+{
+	ct->used++;
+}
+
+static inline void
+ct_put(struct conntrack *ct)
+{
+	if (--ct->used == 0) {
+		assert(num_conntrack > 0);
+
+		free(ct);
+
+		num_conntrack--;
+	}
+}
+
+/* tuple cache */
+static struct conntrack ct_search;		/* used by scache too */
 
 static conntrack_hash_t
-hash_conntrack(const struct nfct_tuple *t, size_t hash_sz)
+tcache_hash(struct cache *c, struct conntrack *ct)
 {
 	static unsigned rnd;
+	struct nfct_tuple *t = &ct->tuple;
 
 	if (rnd == 0U)
 		rnd = rand();
 
-	return jhash_3words(t->src.v4, t->dst.v4 ^ t->protonum,
-						t->l4src.all | (t->l4dst.all << 16), rnd) % hash_sz;
+	return jhash_3words(t->src.v4, t->dst.v4 ^ t->protonum,	t->l4src.all
+						| (t->l4dst.all << 16), rnd) % c->c_num_heads;
 }
 
-static inline bool
-ct_cmp(const struct nfct_tuple *t1, const struct nfct_tuple *t2)
+static int
+tcache_add(struct cache *c, struct conntrack *ct)
 {
-	return memcmp(t1, t2, sizeof(struct nfct_tuple)) == 0;
-}
+	conntrack_hash_t h = c->c_hash(c, ct);
+
+	llist_add(&ct->list, &c->c_head[h].link);
+	c->c_head[h].cnt++;
 
+	pr_debug("%s: ct=%p (h %u, %u/%u)\n", __func__, ct, h,
+			 c->c_head[h].cnt, c->c_cnt);
 
-static struct ct_htable *
-htable_alloc(int htable_size)
+	return 0;
+}
+
+static int
+tcache_del(struct cache *c, struct conntrack *ct)
 {
-	struct ct_htable *htable;
-	int i;
+	conntrack_hash_t h = c->c_hash(c, ct);
 
-	htable = malloc(sizeof(*htable)
-			+ sizeof(struct llist_head) * htable_size);
-	if (!htable)
-		return NULL;
+	assert(c->c_head[h].cnt > 0);
 
-	htable->buckets = (void *)htable + sizeof(*htable);
-	htable->num_buckets = htable_size;
-	htable->curr_bucket = 0;
-	htable->used = 0;
+	pr_debug("%s: ct=%p (h %u, %u/%u)\n", __func__, ct, h,
+			 c->c_head[h].cnt, c->c_cnt);
 
-	for (i = 0; i < htable->num_buckets; i++)
-		INIT_LLIST_HEAD(&htable->buckets[i]);
-	
-	return htable;
+	llist_del(&ct->list);
+	c->c_head[h].cnt--;
+
+	return 0;
 }
 
-static void
-htable_free(struct ct_htable *htable)
+static struct conntrack *
+tcache_find(const struct ulogd_pluginstance *pi,
+			const struct nfct_tuple *tuple)
 {
-	struct llist_head *ptr, *ptr2;
-	int i;
+	struct nfct_pluginstance *priv = (void *)pi->private;
+	struct cache *c = priv->tcache;
+	struct conntrack *ct;
+	conntrack_hash_t h;
 
-	for (i = 0; i < htable->num_buckets; i++) {
-		llist_for_each_safe(ptr, ptr2, &htable->buckets[i])
-			free(container_of(ptr, struct ct_timestamp, list));
+	memcpy(&ct_search.tuple, tuple, sizeof(struct nfct_tuple));
+	h = c->c_hash(c, &ct_search);
+
+	llist_for_each_entry(ct, &c->c_head[h].link, list) {
+		if (memcmp(&ct->tuple, tuple, sizeof(*tuple)) == 0)
+			return ct;
 	}
 
-	free(htable);
+	return NULL;
 }
 
-static struct ct_timestamp *
-ct_hash_add(struct ct_htable *htable, const struct nfct_tuple *t)
+/* sequence cache */
+static conntrack_hash_t
+scache_hash(struct cache *c, struct conntrack *ct)
 {
-	struct ct_timestamp *ts;
-	conntrack_hash_t h;
+	static unsigned rnd;
 
-	h = hash_conntrack(t, htable->num_buckets);
+	if (rnd == 0U)
+		rnd = rand();
 
-	if ((ts = calloc(1, sizeof(struct ct_timestamp))) == NULL) {
-		ulogd_log(ULOGD_ERROR, "Out of memory\n");
-		return NULL;
-	}
+	return (ct->last_seq ^ rnd) % c->c_num_heads;
+}
 
-	memcpy(&ts->tuple, t, sizeof(struct nfct_tuple));
+static int
+scache_add(struct cache *c, struct conntrack *ct)
+{
+	conntrack_hash_t h = c->c_hash(c, ct);
 
-	llist_add(&ts->list, &htable->buckets[h]);
-	htable->used++;
+	llist_add(&ct->seq_link, &c->c_head[h].link);
+	c->c_head[h].cnt++;
 
-	return ts;
+	pr_debug("%s: ct=%p (h %u, %u/%u)\n", __func__, ct, h,
+			 c->c_head[h].cnt, c->c_cnt);
+
+	return 0;
 }
 
-static struct ct_timestamp *
-ct_hash_find(struct ct_htable *htable, const struct nfct_tuple *t)
-{  
-	struct llist_head *ptr;
-	conntrack_hash_t h = hash_conntrack(t, htable->num_buckets);
+static int
+scache_del(struct cache *c, struct conntrack *ct)
+{
+	conntrack_hash_t h = c->c_hash(c, ct);
+
+	assert(c->c_head[h].cnt > 0);
 
-	llist_for_each(ptr, &htable->buckets[h]) {
-		struct ct_timestamp *ts = container_of(ptr, struct ct_timestamp, list);
+	pr_debug("%s: ct=%p (h %u, %u/%u)\n", __func__, ct, h,
+			 c->c_head[h].cnt, c->c_cnt);
 
-		if (ct_cmp(t, &ts->tuple))
-			return ts;
+	llist_del(&ct->seq_link);
+	ct->last_seq = 0;
+
+	c->c_head[h].cnt--;
+
+	return 0;
+}
+
+static struct conntrack *
+scache_find(const struct ulogd_pluginstance *pi, unsigned seq)
+{
+	struct nfct_pluginstance *priv = (void *)pi->private;
+	struct cache *c = priv->scache;
+	struct conntrack *ct;
+	conntrack_hash_t h;
+
+	ct_search.last_seq = seq;
+	h = c->c_hash(c, &ct_search);
+
+	llist_for_each_entry(ct, &c->c_head[h].link, seq_link) {
+		if (ct->last_seq == ct_search.last_seq)
+			return ct;
 	}
 
 	return NULL;
 }
 
+static struct cache *
+cache_alloc(int cache_size)
+{
+	struct cache *c;
+	int i;
+
+	c = malloc(sizeof(*c) + sizeof(struct cache_head) * cache_size);
+	if (c == NULL)
+		return NULL;
+
+	c->c_head = (void *)c + sizeof(*c);
+	c->c_num_heads = cache_size;
+	c->c_curr_head = 0;
+	c->c_cnt = 0;
+
+	for (i = 0; i < c->c_num_heads; i++) {
+		INIT_LLIST_HEAD(&c->c_head[i].link);
+		c->c_head[i].cnt = 0;
+	}
+	
+	return c;
+}
 
-static struct ct_timestamp *
-ct_hash_find_seq(const struct ct_htable *htable, unsigned seq)
+static void
+cache_free(struct cache *c)
 {
 	int i;
 
-	for (i = 0; i < htable->num_buckets; i++) {
-		struct ct_timestamp *ts;
+	for (i = 0; i < c->c_num_heads; i++) {
+		struct llist_head *ptr, *ptr2;
 
-		llist_for_each_entry(ts, &htable->buckets[i], list) {
-			if (ts->last_seq == seq)
-				return ts;
-		}
+		llist_for_each_safe(ptr, ptr2, &c->c_head[i].link)
+			free(container_of(ptr, struct conntrack, list));
 	}
 
-	return NULL;
+	free(c);
 }
 
+int
+cache_add(struct cache *c, struct conntrack *ct)
+{
+	ct_get(ct);
 
-/* time diff with second resolution */
-static inline unsigned
-tv_diff_sec(const struct ct_timestamp *ts)
+	ct->time[UPDATE].tv_sec = ct->time[START].tv_sec = t_now;
+
+	/* order of these two is important for debugging purposes */
+	c->c_cnt++;
+	c->c_add(c, ct);
+
+	return 0;
+}
+
+int
+cache_del(struct cache *c, struct conntrack *ct)
 {
-	if (ts->time[STOP].tv_sec >= ts->time[START].tv_sec)
-		return max(ts->time[STOP].tv_sec - ts->time[START].tv_sec, 1);
+	assert(c->c_cnt > 0);
+	assert(ct->used > 0);
 
-	return ts->time[START].tv_sec - ts->time[STOP].tv_sec;
+	/* order of these two is important for debugging purposes */
+	c->c_del(c, ct);
+	c->c_cnt--;
+
+	ct_put(ct);
+
+	return 0;
 }
 
-static void
-ct_hash_free(struct ct_htable *htable, struct ct_timestamp *ts)
+/* time diff with second resolution */
+static inline unsigned
+tv_diff_sec(const struct timeval *tv1, const struct timeval *tv2)
 {
-	llist_del(&ts->list);
+	if (tv2->tv_sec >= tv1->tv_sec)
+		return max(tv2->tv_sec - tv1->tv_sec, 1);
 
-	free(ts);
-	htable->used--;
+	return tv1->tv_sec - tv2->tv_sec;
 }
 
 static int
 propagate_ct_flow(struct ulogd_pluginstance *upi, 
-				  struct nfct_conntrack *ct, unsigned int flags,
-				  int dir, struct ct_timestamp *ts)
+				  struct nfct_conntrack *nfct, unsigned int flags,
+				  int dir, struct conntrack *ct)
 {
 	struct ulogd_key *ret = upi->output.keys;
 
-	ret[O_IP_SADDR].u.value.ui32 = htonl(ct->tuple[dir].src.v4);
+	ret[O_IP_SADDR].u.value.ui32 = htonl(nfct->tuple[dir].src.v4);
 	ret[O_IP_SADDR].flags |= ULOGD_RETF_VALID;
 
-	ret[O_IP_DADDR].u.value.ui32 = htonl(ct->tuple[dir].dst.v4);
+	ret[O_IP_DADDR].u.value.ui32 = htonl(nfct->tuple[dir].dst.v4);
 	ret[O_IP_DADDR].flags |= ULOGD_RETF_VALID;
 
-	ret[O_IP_PROTO].u.value.ui8 = ct->tuple[dir].protonum;
+	ret[O_IP_PROTO].u.value.ui8 = nfct->tuple[dir].protonum;
 	ret[O_IP_PROTO].flags |= ULOGD_RETF_VALID;
 
-	switch (ct->tuple[dir].protonum) {
+	switch (nfct->tuple[dir].protonum) {
 	case IPPROTO_TCP:
 	case IPPROTO_UDP:
 	case IPPROTO_SCTP:
 		/* FIXME: DCCP */
-		ret[O_L4_SPORT].u.value.ui16 = htons(ct->tuple[dir].l4src.tcp.port);
+		ret[O_L4_SPORT].u.value.ui16 = htons(nfct->tuple[dir].l4src.tcp.port);
 		ret[O_L4_SPORT].flags |= ULOGD_RETF_VALID;
-		ret[O_L4_DPORT].u.value.ui16 = htons(ct->tuple[dir].l4dst.tcp.port);
+		ret[O_L4_DPORT].u.value.ui16 = htons(nfct->tuple[dir].l4dst.tcp.port);
 		ret[O_L4_DPORT].flags |= ULOGD_RETF_VALID;
 		break;
 	case IPPROTO_ICMP:
-		ret[O_ICMP_CODE].u.value.ui8 = ct->tuple[dir].l4src.icmp.code;
+		ret[O_ICMP_CODE].u.value.ui8 = nfct->tuple[dir].l4src.icmp.code;
 		ret[O_ICMP_CODE].flags |= ULOGD_RETF_VALID;
-		ret[O_ICMP_TYPE].u.value.ui8 = ct->tuple[dir].l4src.icmp.type;
+		ret[O_ICMP_TYPE].u.value.ui8 = nfct->tuple[dir].l4src.icmp.type;
 		ret[O_ICMP_TYPE].flags |= ULOGD_RETF_VALID;
 		break;
 	}
 
 	if (flags & NFCT_COUNTERS_ORIG) {
-		ret[O_RAW_IN_PKTLEN].u.value.ui32 = ct->counters[0].bytes;
+		ret[O_RAW_IN_PKTLEN].u.value.ui32 = nfct->counters[0].bytes;
 		ret[O_RAW_IN_PKTLEN].flags |= ULOGD_RETF_VALID;
-		ret[O_RAW_IN_PKTCOUNT].u.value.ui32 = ct->counters[0].packets;
+		ret[O_RAW_IN_PKTCOUNT].u.value.ui32 = nfct->counters[0].packets;
 		ret[O_RAW_IN_PKTCOUNT].flags |= ULOGD_RETF_VALID;
 
-		ret[O_RAW_OUT_PKTLEN].u.value.ui32 = ct->counters[1].bytes;
+		ret[O_RAW_OUT_PKTLEN].u.value.ui32 = nfct->counters[1].bytes;
 		ret[O_RAW_OUT_PKTLEN].flags |= ULOGD_RETF_VALID;
-		ret[O_RAW_OUT_PKTCOUNT].u.value.ui32 = ct->counters[1].packets;
+		ret[O_RAW_OUT_PKTCOUNT].u.value.ui32 = nfct->counters[1].packets;
 		ret[O_RAW_OUT_PKTCOUNT].flags |= ULOGD_RETF_VALID;
 	}
 
 	if (flags & NFCT_MARK) {
-		ret[O_CT_MARK].u.value.ui32 = ct->mark;
+		ret[O_CT_MARK].u.value.ui32 = nfct->mark;
 		ret[O_CT_MARK].flags |= ULOGD_RETF_VALID;
 	}
 
 	if (flags & NFCT_ID) {
-		ret[O_CT_ID].u.value.ui32 = ct->id;
+		ret[O_CT_ID].u.value.ui32 = nfct->id;
 		ret[O_CT_ID].flags |= ULOGD_RETF_VALID;
 	}
 
-	ret[O_FLOW_START_SEC].u.value.ui32 = ts->time[START].tv_sec;
+	ret[O_FLOW_START_SEC].u.value.ui32 = ct->time[START].tv_sec;
 	ret[O_FLOW_START_SEC].flags |= ULOGD_RETF_VALID;
-	ret[O_FLOW_START_USEC].u.value.ui32 = ts->time[START].tv_usec;
+	ret[O_FLOW_START_USEC].u.value.ui32 = ct->time[START].tv_usec;
 	ret[O_FLOW_START_USEC].flags |= ULOGD_RETF_VALID;
-	ret[O_FLOW_START_DAY].u.value.ui16 = ts->time[START].tv_sec / (1 DAY);
+	ret[O_FLOW_START_DAY].u.value.ui16 = ct->time[START].tv_sec / (1 DAY);
 	ret[O_FLOW_START_DAY].flags |= ULOGD_RETF_VALID;
 
-	ret[O_FLOW_END_SEC].u.value.ui32 = ts->time[STOP].tv_sec;
+	ret[O_FLOW_END_SEC].u.value.ui32 = ct->time[STOP].tv_sec;
 	ret[O_FLOW_END_SEC].flags |= ULOGD_RETF_VALID;
-	ret[O_FLOW_END_USEC].u.value.ui32 = ts->time[STOP].tv_usec;
+	ret[O_FLOW_END_USEC].u.value.ui32 = ct->time[STOP].tv_usec;
 	ret[O_FLOW_END_USEC].flags |= ULOGD_RETF_VALID;
 
-	ret[O_FLOW_DURATION].u.value.ui32 = tv_diff_sec(ts);
+	ret[O_FLOW_DURATION].u.value.ui32 = tv_diff_sec(&ct->time[START],
+													&ct->time[STOP]);
 	ret[O_FLOW_DURATION].flags |= ULOGD_RETF_VALID;
 
 	ulogd_propagate_results(upi);
@@ -648,22 +767,22 @@ propagate_ct_flow(struct ulogd_pluginsta
 }
 
 static int
-propagate_ct(struct ulogd_pluginstance *upi, struct nfct_conntrack *ct,
-			 struct ct_timestamp *ts, unsigned int flags)
+propagate_ct(struct ulogd_pluginstance *upi, struct nfct_conntrack *nfct,
+			 struct conntrack *ct, unsigned int flags)
 {
 	struct nfct_pluginstance *priv = (void *)upi->private;
 
 	do {
-		if (ct->tuple[NFCT_DIR_ORIGINAL].src.v4 == INADDR_LOOPBACK
-			|| ct->tuple[NFCT_DIR_ORIGINAL].dst.v4 == INADDR_LOOPBACK)
+		if (nfct->tuple[NFCT_DIR_ORIGINAL].src.v4 == INADDR_LOOPBACK
+			|| nfct->tuple[NFCT_DIR_ORIGINAL].dst.v4 == INADDR_LOOPBACK)
 			break;
 
-		gettimeofday(&ts->time[STOP], NULL);
-		
-		propagate_ct_flow(upi, ct, flags, NFCT_DIR_ORIGINAL, ts);
+		ct->time[STOP].tv_sec = t_now;
+
+		propagate_ct_flow(upi, nfct, flags, NFCT_DIR_ORIGINAL, ct);
 	} while (0);
 
-	ct_hash_free(priv->htable, ts);
+	cache_del(priv->tcache, ct);
 
 	return 0;
 }
@@ -675,49 +794,64 @@ do_nfct_msg(struct nlmsghdr *nlh, void *
 	struct ulogd_pluginstance *pi = arg;
 	struct nfct_pluginstance *priv = (void *)pi->private;
 	struct nfgenmsg *nfh = NLMSG_DATA(nlh);
-	struct nfct_conntrack ct;
-	struct ct_timestamp *ts;
+	struct nfct_conntrack nfct;
+	struct conntrack *ct;
 	int flags, type = nfct_msg_type(nlh);
 
 	if (type == NFCT_MSG_UNKNOWN)
 		return 0;
 
-	bzero(&ct, sizeof(ct));
+	bzero(&nfct, sizeof(nfct));
 
-	ct.tuple[NFCT_DIR_ORIGINAL].l3protonum = 
-		ct.tuple[NFCT_DIR_REPLY].l3protonum = nfh->nfgen_family;
+	nfct.tuple[NFCT_DIR_ORIGINAL].l3protonum = 
+		nfct.tuple[NFCT_DIR_REPLY].l3protonum = nfh->nfgen_family;
 
-	if (nfct_netlink_to_conntrack(nlh, &ct, &flags) < 0)
+	if (nfct_netlink_to_conntrack(nlh, &nfct, &flags) < 0)
 		return -1;
 
+	/* TODO handle NFCT_COUNTER_FILLING */
+
 	switch (type) { 
 	case NFCT_MSG_NEW:
-		ts = ct_hash_add(priv->htable, &ct.tuple[NFCT_DIR_ORIGINAL]);
-		if (ts != NULL) {
-			gettimeofday(&ts->time[START], NULL);
-		}
+		if ((ct = ct_alloc(&nfct.tuple[NFCT_DIR_ORIGINAL])) == NULL)
+			return -1;
+
+		if (cache_add(priv->tcache, ct) < 0)
+			return -1;
 		break;
 
 	case NFCT_MSG_UPDATE:
-		ts = ct_hash_find(priv->htable, &ct.tuple[NFCT_DIR_ORIGINAL]);
-		if (ts == NULL) {
+		ct = tcache_find(pi, &nfct.tuple[NFCT_DIR_ORIGINAL]);
+		if (ct == NULL) {
 			/* do not add CT to cache, as there would be no start
 			   information */
 			break;
 		}
 
+		ct->time[UPDATE].tv_sec = t_now;
+
+		if (ct->used > 1) {
+			struct conntrack *ct_tmp = scache_find(pi, nlh->nlmsg_seq);
+
+			if (ct_tmp != NULL) {
+				assert(ct_tmp == ct);
+
+				cache_del(priv->scache, ct);
+			}
+		}
+
 		/* handle TCP connections differently in order not to bloat CT
 		   hash with many TIME_WAIT connections */
-		if (ct.tuple[NFCT_DIR_ORIGINAL].protonum == IPPROTO_TCP) {
-			if (ct.protoinfo.tcp.state == TCP_CONNTRACK_TIME_WAIT)
-				return propagate_ct(pi, &ct, ts, flags);
+		if (nfct.tuple[NFCT_DIR_ORIGINAL].protonum == IPPROTO_TCP) {
+			if (nfct.protoinfo.tcp.state == TCP_CONNTRACK_TIME_WAIT)
+				return propagate_ct(pi, &nfct, ct, flags);
 		}
 		break;
 		
 	case NFCT_MSG_DESTROY:
-		ts = ct_hash_find(priv->htable, &ct.tuple[NFCT_DIR_ORIGINAL]);
-		if (ts != NULL)
-			return propagate_ct(pi, &ct, ts, flags);
+		ct = tcache_find(pi, &nfct.tuple[NFCT_DIR_ORIGINAL]);
+		if (ct != NULL)
+			return propagate_ct(pi, &nfct, ct, flags);
 		break;
 		
 	default:
@@ -741,31 +875,68 @@ read_cb_nfct(int fd, unsigned what, void
 }
 
 
-/* choosing powers of two for all values helps here */
-#define STOP_HERE(h)	(((h)->curr_bucket + 16) % (h)->num_buckets)
+#define STOP_HERE(h)	(((h)->c_curr_head + 32) % (h)->c_num_heads)
 
+/*
+  nfct_timer_cb()
+
+  This is a synchronous timer, do whatever you want.
+*/
 static void
 nfct_timer_cb(struct ulogd_timer *t)
 {
 	struct ulogd_pluginstance *pi = t->data;
 	struct nfct_pluginstance *priv = (void *)pi->private;
-	struct ct_htable *ht = priv->htable;
-	struct ct_timestamp *ts;
-	int end = STOP_HERE(ht);
+	struct cache *ht = priv->tcache;
+	int i, req = 0, end = STOP_HERE(ht);
+	struct conntrack *ct;
+
+	/* cleanup stale entries from sequence cache */
+	for (i = 0; i < priv->scache->c_num_heads; i++) {
+		if (llist_empty(&priv->scache->c_head[i].link))
+			continue;
+
+		ct = container_of(priv->scache->c_head[i].link.prev,
+						  struct conntrack, seq_link);
+
+		assert(ct->t_req != 0);
+
+		if (ct->t_req > 0 && (t_now - ct->t_req) > 5 SEC)
+			cache_del(priv->scache, ct);
+	}
 
+	/* check entries in tuple cache */
 	do {
-		assert(ht->curr_bucket < ht->num_buckets);
+		assert(ht->c_curr_head < ht->c_num_heads);
+
+		llist_for_each_entry(ct, &ht->c_head[ht->c_curr_head].link, list) {
+			if (tv_diff_sec(&ct->time[UPDATE], &tv_now) < timeout_ce(pi))
+				continue;
 
-		llist_for_each_entry(ts, &ht->buckets[ht->curr_bucket], list) {
 			/* check if its still there */
-			nfct_get_conntrack_x(priv->cth, &ts->tuple, NFCT_DIR_ORIGINAL,
-								 &ts->last_seq);
+			nfct_get_conntrack_x(priv->cth, &ct->tuple, NFCT_DIR_ORIGINAL,
+								 &ct->last_seq);
+
+			if (&ct->last_seq != 0) {
+				ct->t_req = t_now;
+
+				assert(scache_find(pi, ct->last_seq) == NULL);
+
+				cache_add(priv->scache, ct);
+			}
+
+			if (++req > SCACHE_REQ_MAX)
+				break;
 		}
 
-		ht->curr_bucket = (ht->curr_bucket + 1) % ht->num_buckets;
-	} while (ht->curr_bucket != end);
+		ht->c_curr_head = (ht->c_curr_head + 1) % ht->c_num_heads;
 
-	pr_debug("%s: now=%ld\n", __func__, t_now);
+		if (req > SCACHE_REQ_MAX)
+			break;
+	} while (ht->c_curr_head != end);
+
+	ulogd_log(ULOGD_DEBUG, "%s: ct:%u t:%u s:%u\n", pi->id,
+			  num_conntrack, priv->tcache->c_cnt, priv->scache->c_cnt);
 }
 
 static int
@@ -775,19 +946,51 @@ nfct_configure(struct ulogd_pluginstance
 	struct nfct_pluginstance *priv = (void *)upi->private;
 	int ret;
 
-	pr_debug("%s: pi=%p\n", __func__, upi);
-
 	memset(priv, 0, sizeof(struct nfct_pluginstance));
 	
 	ret = config_parse_file(upi->id, upi->config_kset);
 	if (ret < 0)
 		return ret;
 
-	priv->disable = disable_ce(upi->config_kset).u.value;
-
 	return 0;
 }
 
+static int
+init_caches(struct ulogd_pluginstance *pi)
+{
+	struct nfct_pluginstance *priv = (void *)pi->private;
+	struct cache *c;
+
+	assert(priv->tcache == NULL && priv->scache == NULL);
+
+	/* tuple cache */
+	c = priv->tcache = cache_alloc(buckets_ce(pi));
+	if (priv->tcache == NULL) {
+		ulogd_log(ULOGD_FATAL, "%s: out of memory\n", pi->id);
+		return -1;
+	}
+
+	c->c_hash = tcache_hash;
+	c->c_add = tcache_add;
+	c->c_del = tcache_del;
+
+	/* sequence cache */
+	c = priv->scache = cache_alloc(SCACHE_SIZE);
+	if (priv->scache == NULL) {
+		ulogd_log(ULOGD_FATAL, "%s: out of memory\n", pi->id);
+
+		cache_free(priv->tcache);
+		priv->tcache = NULL;
+
+		return -1;
+	}
+
+	c->c_hash = scache_hash;
+	c->c_add = scache_add;
+	c->c_del = scache_del;
+
+	return 0;
+}
 
 static int
 nfct_start(struct ulogd_pluginstance *upi)
@@ -796,16 +999,13 @@ nfct_start(struct ulogd_pluginstance *up
 
 	pr_debug("%s: pi=%p\n", __func__, upi);
 
-	if (priv->disable) {
+	if (disable_ce(upi) != 0) {
 		ulogd_log(ULOGD_INFO, "%s: disabled\n", upi->id);
 		return 0;
 	}
 
-	priv->htable = htable_alloc(buckets_ce(upi->config_kset).u.value);
-	if (priv->htable == NULL) {
-		ulogd_log(ULOGD_FATAL, "%s: out of memory\n", upi->id);
+	if (init_caches(upi) < 0)
 		return -1;
-	}
 
 	priv->cth = nfct_open(NFNL_SUBSYS_CTNETLINK, CT_EVENTS);
 	if (priv->cth == NULL) {
@@ -831,7 +1031,8 @@ nfct_start(struct ulogd_pluginstance *up
 	if (ulogd_register_timer(&priv->timer) < 0)
 		goto err_unreg_fd;
 
-	ulogd_log(ULOGD_INFO, "%s: started\n", upi->id);
+	ulogd_log(ULOGD_INFO, "%s: started (tcache %u, scache %u)\n", upi->id,
+			  priv->tcache->c_num_heads, priv->scache->c_num_heads);
 
 	return 0;
 
@@ -841,8 +1042,8 @@ nfct_start(struct ulogd_pluginstance *up
 	nfct_close(priv->cth);
 	priv->cth = NULL;
  err_free:
-	htable_free(priv->htable);
-	priv->htable = NULL;
+	cache_free(priv->tcache);
+	priv->tcache = NULL;
 
 	return -1;
 }
@@ -854,10 +1055,10 @@ nfct_stop(struct ulogd_pluginstance *pi)
 
 	pr_debug("%s: pi=%p\n", __func__, pi);
 
-	if (priv->disable)
+	if (disable_ce(pi) != 0)
 		return 0;				/* wasn't started */
 
-	if (priv->htable == NULL)
+	if (priv->tcache == NULL)
 		return 0;				/* already stopped */
 
 	ulogd_unregister_timer(&priv->timer);
@@ -871,9 +1072,9 @@ nfct_stop(struct ulogd_pluginstance *pi)
 
 	ulogd_log(ULOGD_DEBUG, "%s: ctnetlink connection closed\n", pi->id);
 
-	if (priv->htable != NULL) {
-		htable_free(priv->htable);
-		priv->htable = NULL;
+	if (priv->tcache != NULL) {
+		cache_free(priv->tcache);
+		priv->tcache = NULL;
 	}
 
 	return 0;

-- 

[ULOGD RFC 16/30] SQLITE3: handle locked DB smarter

[heitzenberger]

[-- Attachment #1: ulogd-SQLITE3-handle-locked-db-smarter.diff --]
[-- Type: text/plain, Size: 7683 bytes --]

If the DB was locked for some longer time it turned out that ulogd
consumed was hammering the Sqlite DB too often, which resulted in high
CPU usage.

Also replace BSD list (sys/queue.h) by Linux list implementation, as
used by other parts of ulogd.

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/output/sqlite3/ulogd_output_SQLITE3.c
===================================================================
--- ulogd-netfilter.orig/output/sqlite3/ulogd_output_SQLITE3.c
+++ ulogd-netfilter/output/sqlite3/ulogd_output_SQLITE3.c
@@ -33,9 +33,9 @@
 #include <ulogd/ulogd.h>
 #include <ulogd/conffile.h>
 #include <ulogd/common.h>
+#include <ulogd/linuxlist.h>
 #include <arpa/inet.h>
 #include <sqlite3.h>
-#include <sys/queue.h>
 
 #define PFX		"SQLITE3: "
 
@@ -58,7 +58,7 @@ struct col {
 };
 
 struct row {
-	TAILQ_ENTRY(row) link;
+	struct llist_head link;
 	uint32_t ip_saddr;
 	uint32_t ip_daddr;
 	unsigned char ip_proto;
@@ -71,11 +71,6 @@ struct row {
 	unsigned flow_duration;
 };
 
-TAILQ_HEAD(row_lh, row);
-
-#define TAILQ_FOR_EACH(pos, head, link) \
-        for (pos = (head).tqh_first; pos != NULL; pos = pos->link.tqe_next)
-
 #define RKEY(key)	((key)->u.source)
 
 
@@ -90,10 +85,11 @@ struct sqlite3_priv {
 	struct col cols[DB_NUM_COLS];
 
 	/* our backlog buffer */
-	struct row_lh rows;
+	struct llist_head rows;
 	int num_rows;
-	int max_rows;				/* number of rows actually seen */
-	int max_rows_allowed;
+	int max_backlog;
+
+	time_t commit_time;
 
 	unsigned disable : 1;
 	unsigned overlimit_msg : 1;
@@ -168,11 +164,9 @@ row_new(void)
 }
 
 
-static void
-row_del(struct sqlite3_priv *priv, struct row *row)
+static inline void
+__row_del(struct sqlite3_priv *priv, struct row *row)
 {
-	TAILQ_REMOVE(&priv->rows, row, link);
-
 	free(row);
 
 	priv->num_rows--;
@@ -180,23 +174,49 @@ row_del(struct sqlite3_priv *priv, struc
 
 
 static void
+row_del(struct sqlite3_priv *priv, struct row *row)
+{
+	llist_del(&row->link);
+
+	__row_del(priv, row);
+}
+
+
+static int
 row_add(struct sqlite3_priv *priv, struct row *row)
 {
-	if (priv->max_rows_allowed && priv->num_rows > priv->max_rows_allowed) {
+	if (priv->max_backlog && priv->num_rows > priv->max_backlog) {
 		if (!priv->overlimit_msg) {
 			ulogd_error(PFX "over max-backlog limit, dropping row\n");
 
 			priv->overlimit_msg = 1;
 		}
 
-		return;
+		__row_del(priv, row);
+
+		return -1;
 	}
 
-	TAILQ_INSERT_TAIL(&priv->rows, row, link);
+	llist_add_tail(&row->link, &priv->rows);
 
 	priv->num_rows++;
+
+	return 0;
 }
 
+/* set_commit_time() - set time for next try on locked database
+ *
+ * The database is effectively locked in between.
+ */
+static void
+set_commit_time(const struct ulogd_pluginstance *pi)
+{
+	struct sqlite3_priv *priv = (void *)pi->private;
+
+	priv->commit_time = t_now + 1;
+
+	pr_debug("%s: commit time %d\n", __func__, priv->commit_time);
+}
 
 #define _SQLITE3_INSERTTEMPL   "insert into X (Y) values (Z)"
 
@@ -408,7 +428,7 @@ db_start(struct ulogd_pluginstance *pi)
 	/* initialize our buffer size and counter */
 	priv->buffer_size = buffer_ce(pi);
 
-	priv->max_rows_allowed = max_backlog_ce(pi);
+	priv->max_backlog = max_backlog_ce(pi);
 
 	/* create and prepare the actual insert statement */
 	db_createstmt(pi);
@@ -481,6 +501,7 @@ db_add_row(struct ulogd_pluginstance *pi
 	switch (sqlite3_errcode(priv->dbh)) {
 	case SQLITE_LOCKED:
 	case SQLITE_BUSY:
+		set_commit_time(pi);
 		break;
 
 	case SQLITE_SCHEMA:
@@ -513,14 +534,22 @@ db_add_row(struct ulogd_pluginstance *pi
 	return -1;
 }
 
+#define llist_for_each_prev_safe(pos, n, head) \
+    for (pos = (head)->prev, n = pos->prev; pos != (head); \
+        pos = n, n = pos->prev)
 
+/* delete_rows() - delete rows from the tail of the list */
 static int
-delete_all_rows(struct ulogd_pluginstance *pi)
+delete_rows(struct ulogd_pluginstance *pi, int rows)
 {
 	struct sqlite3_priv *priv = (void *)pi->private;
+	struct llist_head *curr, *tmp;
 
-	while (priv->rows.tqh_first != NULL) {
-		struct row *row = priv->rows.tqh_first;
+    llist_for_each_prev_safe(curr, tmp, &priv->rows) {
+		struct row *row = container_of(curr, struct row, link);
+
+		if (rows-- == 0)
+			break;
 
 		row_del(priv, row);
 	}
@@ -528,22 +557,33 @@ delete_all_rows(struct ulogd_pluginstanc
 	return 0;
 }
 
+/*
+  db_commit_rows()
 
+  RETURN
+    >0	rows commited
+    0	locked
+   -1	error
+*/
 static int
 db_commit_rows(struct ulogd_pluginstance *pi)
 {
 	struct sqlite3_priv *priv = (void *)pi->private;
 	struct row *row;
-	int ret, rows = 0;
+	int ret, rows = 0, max_commit;
 
 	ret = sqlite3_exec(priv->dbh, "begin immediate transaction", NULL,
 					   NULL, NULL);
 	if (ret != SQLITE_OK) {
-		if (ret == SQLITE_BUSY)
+		if (ret == SQLITE_BUSY) {
+			set_commit_time(pi);
 			goto err_rollback;
+		}
 
-		if (sqlite3_errcode(priv->dbh) == SQLITE_LOCKED)
+		if (sqlite3_errcode(priv->dbh) == SQLITE_LOCKED) {
+			set_commit_time(pi);
 			return 0;			/* perform commit later */
+		}
 	
 		ulogd_error("%s: begin transaction: %s\n", pi->id,
 					sqlite3_errmsg(priv->dbh));
@@ -551,27 +591,35 @@ db_commit_rows(struct ulogd_pluginstance
 		return -1;
 	}
 
-	TAILQ_FOR_EACH(row, priv->rows, link) {
+
+	/* Limit number of rows to commit.  Note that currently three times
+	   buffer_size is a bit arbitrary and therefore might be adjusted in
+	   the future. */
+	max_commit = max(3 * priv->buffer_size, 1024);
+
+	llist_for_each_entry_reverse(row, &priv->rows, link) {
+		if (++rows > max_commit)
+			break;
+
 		if (db_add_row(pi, row) < 0)
 			goto err_rollback;
-
-		rows++;
 	}
 
 	ret = sqlite3_exec(priv->dbh, "commit", NULL, NULL, NULL);
 	if (ret == SQLITE_OK) {
 		sqlite3_reset(priv->p_stmt);
 
-		if (priv->num_rows > priv->buffer_size)
-			ulogd_log(ULOGD_INFO, "%s: commited backlog buffer (%d rows)\n",
-					  pi->id, priv->num_rows);
+		pr_debug("%s: commited %d/%d rows\n", pi->id, rows, priv->num_rows);
+
+		delete_rows(pi, rows);
 
-		delete_all_rows(pi);
+		if (priv->commit_time >= t_now)
+			priv->commit_time = 0;		/* release commit lock */
 
 		if (priv->overlimit_msg)
 			priv->overlimit_msg = 0;
 
-		return 0;
+		return rows;
 	}
 
  err_rollback:
@@ -606,9 +654,10 @@ sqlite3_interp(struct ulogd_pluginstance
 	row->flow_start_sec = RKEY(cols[9].key)->u.value.ui32;
 	row->flow_duration = RKEY(cols[10].key)->u.value.ui32;
 
-	row_add(priv, row);
+	if (row_add(priv, row) < 0)
+		return ULOGD_IRET_OK;
 
-	if (priv->num_rows >= priv->buffer_size)
+	if (priv->num_rows >= priv->buffer_size && priv->commit_time == 0)
 		db_commit_rows(pi);
 
 	return ULOGD_IRET_OK;
@@ -620,11 +669,20 @@ sqlite_timer_cb(struct ulogd_timer *t)
 {
 	struct ulogd_pluginstance *pi = t->data;
 	struct sqlite3_priv *priv = (void *)pi->private;
+	int rows;
 
-	priv->max_rows = max(priv->max_rows, priv->num_rows);
+	pr_debug("%s: timer=%p\n", __func__, t);
 
-	if (priv->num_rows > 0)
-		db_commit_rows(pi);
+	if (priv->commit_time != 0 && priv->commit_time > t_now)
+		return;
+
+	if (priv->num_rows == 0)
+		return;
+
+	rows = db_commit_rows(pi);
+
+	ulogd_log(ULOGD_DEBUG, "%s: rows=%d commited=%d\n", pi->id,
+			  priv->num_rows, rows);
 }
 
 
@@ -664,7 +722,7 @@ sqlite3_configure(struct ulogd_pluginsta
 		}
 	}
 
-	priv->max_rows_allowed = max_backlog_ce(pi);
+	priv->max_backlog = max_backlog_ce(pi);
 	priv->disable = disable_ce(pi);
 
 	pr_debug("%s: db='%s' table='%s' timer=%d max-backlog=%d\n", pi->id,
@@ -694,8 +752,8 @@ sqlite3_start(struct ulogd_pluginstance 
 		return 0;
 	}
 
-	priv->num_rows = priv->max_rows = 0;
-	TAILQ_INIT(&priv->rows);
+	priv->num_rows = 0;
+	INIT_LLIST_HEAD(&priv->rows);
 
 	if (db_start(pi) < 0)
 		return -1;

-- 

[ULOGD RFC 17/30] NFCT: use localtime for timestamps

[heitzenberger]

[-- Attachment #1: ulogd-NFCT-use-localtime.diff --]
[-- Type: text/plain, Size: 1123 bytes --]

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/input/flow/ulogd_inpflow_NFCT.c
===================================================================
--- ulogd-netfilter.orig/input/flow/ulogd_inpflow_NFCT.c
+++ ulogd-netfilter/input/flow/ulogd_inpflow_NFCT.c
@@ -655,7 +655,7 @@ cache_add(struct cache *c, struct conntr
 {
 	ct_get(ct);
 
-	ct->time[UPDATE].tv_sec = ct->time[START].tv_sec = t_now;
+	ct->time[UPDATE].tv_sec = ct->time[START].tv_sec = t_now_local;
 
 	/* order of these two is important for debugging purposes */
 	c->c_cnt++;
@@ -777,7 +777,7 @@ propagate_ct(struct ulogd_pluginstance *
 			|| nfct->tuple[NFCT_DIR_ORIGINAL].dst.v4 == INADDR_LOOPBACK)
 			break;
 
-		ct->time[STOP].tv_sec = t_now;
+		ct->time[STOP].tv_sec = t_now_local;
 
 		propagate_ct_flow(upi, nfct, flags, NFCT_DIR_ORIGINAL, ct);
 	} while (0);
@@ -828,7 +828,7 @@ do_nfct_msg(struct nlmsghdr *nlh, void *
 			break;
 		}
 
-		ct->time[UPDATE].tv_sec = t_now;
+		ct->time[UPDATE].tv_sec = t_now_local;
 
 		if (ct->used > 1) {
 			struct conntrack *ct_tmp = scache_find(pi, nlh->nlmsg_seq);

-- 

[ULOGD RFC 18/30] NFCT: properly account both directions in all cases

[heitzenberger]

[-- Attachment #1: ulogd-NFCT-accounting-fix.diff --]
[-- Type: text/plain, Size: 1358 bytes --]

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/input/flow/ulogd_inpflow_NFCT.c
===================================================================
--- ulogd-netfilter.orig/input/flow/ulogd_inpflow_NFCT.c
+++ ulogd-netfilter/input/flow/ulogd_inpflow_NFCT.c
@@ -696,10 +696,10 @@ propagate_ct_flow(struct ulogd_pluginsta
 {
 	struct ulogd_key *ret = upi->output.keys;
 
-	ret[O_IP_SADDR].u.value.ui32 = htonl(nfct->tuple[dir].src.v4);
+	ret[O_IP_SADDR].u.value.ui32 = htonl(nfct->tuple[0].src.v4);
 	ret[O_IP_SADDR].flags |= ULOGD_RETF_VALID;
 
-	ret[O_IP_DADDR].u.value.ui32 = htonl(nfct->tuple[dir].dst.v4);
+	ret[O_IP_DADDR].u.value.ui32 = htonl(nfct->tuple[1].src.v4);
 	ret[O_IP_DADDR].flags |= ULOGD_RETF_VALID;
 
 	ret[O_IP_PROTO].u.value.ui8 = nfct->tuple[dir].protonum;
@@ -710,9 +710,9 @@ propagate_ct_flow(struct ulogd_pluginsta
 	case IPPROTO_UDP:
 	case IPPROTO_SCTP:
 		/* FIXME: DCCP */
-		ret[O_L4_SPORT].u.value.ui16 = htons(nfct->tuple[dir].l4src.tcp.port);
+		ret[O_L4_SPORT].u.value.ui16 = htons(nfct->tuple[0].l4src.tcp.port);
 		ret[O_L4_SPORT].flags |= ULOGD_RETF_VALID;
-		ret[O_L4_DPORT].u.value.ui16 = htons(nfct->tuple[dir].l4dst.tcp.port);
+		ret[O_L4_DPORT].u.value.ui16 = htons(nfct->tuple[1].l4src.tcp.port);
 		ret[O_L4_DPORT].flags |= ULOGD_RETF_VALID;
 		break;
 	case IPPROTO_ICMP:

-- 

[ULOGD RFC 19/30] llist: add llist_for_each_prev_safe()

[heitzenberger]

[-- Attachment #1: ulogd-common-add-llist_for_each_prev_safe.diff --]
[-- Type: text/plain, Size: 986 bytes --]

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/include/ulogd/linuxlist.h
===================================================================
--- ulogd-netfilter.orig/include/ulogd/linuxlist.h
+++ ulogd-netfilter/include/ulogd/linuxlist.h
@@ -245,6 +245,16 @@ static inline void llist_splice_init(str
         	pos = pos->prev, prefetch(pos->prev))
         	
 /**
+ * llist_for_each_prev_safe - iterate over llist backwards safe against removal
+ * @pos:	the type * to use as a loop counter.
+ * @n:		another type * to use as temporary storage
+ * @head:	the head for your llist.
+ */
+#define llist_for_each_prev_safe(pos, n, head) \
+    for (pos = (head)->prev, n = pos->prev; pos != (head); \
+        pos = n, n = pos->prev)
+
+/**
  * llist_for_each_safe	-	iterate over a llist safe against removal of llist entry
  * @pos:	the &struct llist_head to use as a loop counter.
  * @n:		another &struct llist_head to use as temporary storage

-- 

[ULOGD RFC 20/30] SQLITE3: generalize error handling

[heitzenberger]

[-- Attachment #1: ulogd-SQLITE3-handle-errors-correctly.diff --]
[-- Type: text/plain, Size: 8452 bytes --]

Add db_err() error handling function and place it where necessary.

The patch looks large than necessary because of some code
reorganization.

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/output/sqlite3/ulogd_output_SQLITE3.c
===================================================================
--- ulogd-netfilter.orig/output/sqlite3/ulogd_output_SQLITE3.c
+++ ulogd-netfilter/output/sqlite3/ulogd_output_SQLITE3.c
@@ -167,9 +167,9 @@ row_new(void)
 static inline void
 __row_del(struct sqlite3_priv *priv, struct row *row)
 {
-	free(row);
+	assert(row != NULL);
 
-	priv->num_rows--;
+	free(row);
 }
 
 
@@ -179,15 +179,17 @@ row_del(struct sqlite3_priv *priv, struc
 	llist_del(&row->link);
 
 	__row_del(priv, row);
+
+	priv->num_rows--;
 }
 
 
 static int
 row_add(struct sqlite3_priv *priv, struct row *row)
 {
-	if (priv->max_backlog && priv->num_rows > priv->max_backlog) {
+	if (priv->max_backlog && priv->num_rows >= priv->max_backlog) {
 		if (!priv->overlimit_msg) {
-			ulogd_error(PFX "over max-backlog limit, dropping row\n");
+			ulogd_error(PFX "over max-backlog limit, dropping rows\n");
 
 			priv->overlimit_msg = 1;
 		}
@@ -436,107 +438,118 @@ db_start(struct ulogd_pluginstance *pi)
 	return 0;
 }
 
-
+/* db_err() - handle database errors */
 static int
-db_add_row(struct ulogd_pluginstance *pi, const struct row *row)
+db_err(struct ulogd_pluginstance *pi, int ret)
 {
 	struct sqlite3_priv *priv = (void *)pi->private;
-	int db_col = 1, ret = 0, db_ret;
 
-	db_ret = sqlite3_bind_int64(priv->p_stmt, db_col++, row->ip_saddr);
-	if (db_ret != SQLITE_OK)
-		goto err_bind;
-
-	db_ret = sqlite3_bind_int64(priv->p_stmt, db_col++, row->ip_daddr);
-	if (db_ret != SQLITE_OK)
-		goto err_bind;
-
-	db_ret = sqlite3_bind_int(priv->p_stmt, db_col++, row->ip_proto);
-	if (db_ret != SQLITE_OK)
-		goto err_bind;
-
-	db_ret = sqlite3_bind_int(priv->p_stmt, db_col++, row->l4_dport);
-	if (db_ret != SQLITE_OK)
-		goto err_bind;
-
-	db_ret = sqlite3_bind_int(priv->p_stmt, db_col++, row->raw_in_pktlen);
-	if (db_ret != SQLITE_OK)
-		goto err_bind;
-
-	db_ret = sqlite3_bind_int64(priv->p_stmt, db_col++, row->raw_in_pktcount);
-	if (db_ret != SQLITE_OK)
-		goto err_bind;
-
-	db_ret = sqlite3_bind_int(priv->p_stmt, db_col++, row->raw_out_pktlen);
-	if (db_ret != SQLITE_OK)
-		goto err_bind;
-
-	db_ret = sqlite3_bind_int64(priv->p_stmt, db_col++, row->raw_out_pktcount);
-	if (db_ret != SQLITE_OK)
-		goto err_bind;
-
-	db_ret = sqlite3_bind_int(priv->p_stmt, db_col++, row->flow_start_sec);
-	if (db_ret != SQLITE_OK)
-		goto err_bind;
-
-	db_ret = sqlite3_bind_int(priv->p_stmt, db_col++, row->flow_duration);
-	if (db_ret != SQLITE_OK)
-		goto err_bind;
-
-	db_ret = sqlite3_step(priv->p_stmt);
-
-	if (db_ret == SQLITE_DONE) {
-		/* the SQLITE book doesn't say that expclicitely _but_ between
-		   two sqlite_bind_*() calls to the same variable you need to
-		   call sqlite3_reset(). */
-		sqlite3_reset(priv->p_stmt);
+	pr_debug("%s: ret=%d (errcode %d)\n", __func__, ret,
+			 sqlite3_errcode(priv->dbh));
 
-		return 0;
-	}
+	assert(ret != SQLITE_OK && ret != SQLITE_DONE);
 
-	/* Ok, this is a bit confusing: some errors are reported as return
-	   values, most others are reported through sqlite3_errcode() instead.
-	   I think the only authorative source of information is the sqlite
-	   source code.	*/
-	switch (sqlite3_errcode(priv->dbh)) {
-	case SQLITE_LOCKED:
-	case SQLITE_BUSY:
+	if (ret == SQLITE_BUSY || ret == SQLITE_LOCKED)
 		set_commit_time(pi);
-		break;
+	else {
+		switch (sqlite3_errcode(priv->dbh)) {
+		case SQLITE_LOCKED:
+		case SQLITE_BUSY:
+			set_commit_time(pi);
+			break;
 
-	case SQLITE_SCHEMA:
-		if (priv->stmt) {
-			sqlite3_finalize(priv->p_stmt);
+		case SQLITE_SCHEMA:
+			if (priv->stmt) {
+				sqlite3_finalize(priv->p_stmt);
 
-			db_createstmt(pi);
-		}
-		return -1;
+				db_createstmt(pi);
 
-	case SQLITE_ERROR: /* e.g. constraint violation */
-	case SQLITE_MISUSE:
-		ulogd_error(PFX "step: %s\n", sqlite3_errmsg(priv->dbh));
-		ret = -1;
-		break;
+				ulogd_log(ULOGD_INFO, "%s: database schema changed\n",
+						  pi->id);
+			}
+			break;
 
-	default:
-		break;
+		default:
+			ulogd_error("%s: transaction: %s\n", pi->id,
+						sqlite3_errmsg(priv->dbh));
+			break;
+		}
 	}
 
-	sqlite3_reset(priv->p_stmt);
+	sqlite3_exec(priv->dbh, "rollback", NULL, NULL, NULL);
 
-	return ret;
+	/* no sqlit3_clear_bindings(), as an unbind will be done implicitely
+	   on next bind. */
+	if (priv->p_stmt != NULL)
+		sqlite3_reset(priv->p_stmt);
 
- err_bind:
-	ulogd_error(PFX "bind: %s\n", sqlite3_errmsg(priv->dbh));
+	return 0;
+}
 
-	sqlite3_reset(priv->p_stmt);
+static int
+db_add_row(struct ulogd_pluginstance *pi, const struct row *row)
+{
+	struct sqlite3_priv *priv = (void *)pi->private;
+	int db_col = 1, ret;
 
-	return -1;
-}
+	do {
+		ret = sqlite3_bind_int64(priv->p_stmt, db_col++, row->ip_saddr);
+		if (ret != SQLITE_OK)
+			break;
+
+		ret = sqlite3_bind_int64(priv->p_stmt, db_col++, row->ip_daddr);
+		if (ret != SQLITE_OK)
+			break;
+
+		ret = sqlite3_bind_int(priv->p_stmt, db_col++, row->ip_proto);
+		if (ret != SQLITE_OK)
+			break;
+
+		ret = sqlite3_bind_int(priv->p_stmt, db_col++, row->l4_dport);
+		if (ret != SQLITE_OK)
+			break;
+
+		ret = sqlite3_bind_int(priv->p_stmt, db_col++, row->raw_in_pktlen);
+		if (ret != SQLITE_OK)
+			break;
+
+		ret = sqlite3_bind_int64(priv->p_stmt, db_col++, row->raw_in_pktcount);
+		if (ret != SQLITE_OK)
+			break;
+
+		ret = sqlite3_bind_int(priv->p_stmt, db_col++, row->raw_out_pktlen);
+		if (ret != SQLITE_OK)
+			break;
+
+		ret = sqlite3_bind_int64(priv->p_stmt, db_col++,
+								 row->raw_out_pktcount);
+		if (ret != SQLITE_OK)
+			break;
+
+		ret = sqlite3_bind_int(priv->p_stmt, db_col++, row->flow_start_sec);
+		if (ret != SQLITE_OK)
+			break;
+
+		ret = sqlite3_bind_int(priv->p_stmt, db_col++, row->flow_duration);
+		if (ret != SQLITE_OK)
+			break;
 
-#define llist_for_each_prev_safe(pos, n, head) \
-    for (pos = (head)->prev, n = pos->prev; pos != (head); \
-        pos = n, n = pos->prev)
+		if ((ret = sqlite3_step(priv->p_stmt)) == SQLITE_DONE) {
+			/* no sqlite3_clear_bindings(), as an unbind will be
+			   implicetely done before next bind. */
+			sqlite3_reset(priv->p_stmt);
+
+			return 0;
+		}
+
+		/* according to the documentation sqlite3_step() always returns a
+		   generic SQLITE_ERROR.  In order to find out the cause of the
+		   error you have to call sqlite3_reset() or sqlite3_finalize(). */
+		ret = sqlite3_reset(priv->p_stmt);
+	} while (0);
+
+	return db_err(pi, ret);
+}
 
 /* delete_rows() - delete rows from the tail of the list */
 static int
@@ -574,23 +587,8 @@ db_commit_rows(struct ulogd_pluginstance
 
 	ret = sqlite3_exec(priv->dbh, "begin immediate transaction", NULL,
 					   NULL, NULL);
-	if (ret != SQLITE_OK) {
-		if (ret == SQLITE_BUSY) {
-			set_commit_time(pi);
-			goto err_rollback;
-		}
-
-		if (sqlite3_errcode(priv->dbh) == SQLITE_LOCKED) {
-			set_commit_time(pi);
-			return 0;			/* perform commit later */
-		}
-	
-		ulogd_error("%s: begin transaction: %s\n", pi->id,
-					sqlite3_errmsg(priv->dbh));
-
-		return -1;
-	}
-
+	if (ret != SQLITE_OK)
+		return db_err(pi, ret);
 
 	/* Limit number of rows to commit.  Note that currently three times
 	   buffer_size is a bit arbitrary and therefore might be adjusted in
@@ -602,33 +600,26 @@ db_commit_rows(struct ulogd_pluginstance
 			break;
 
 		if (db_add_row(pi, row) < 0)
-			goto err_rollback;
+			return db_err(pi, ret);
 	}
 
 	ret = sqlite3_exec(priv->dbh, "commit", NULL, NULL, NULL);
-	if (ret == SQLITE_OK) {
-		sqlite3_reset(priv->p_stmt);
-
-		pr_debug("%s: commited %d/%d rows\n", pi->id, rows, priv->num_rows);
-
-		delete_rows(pi, rows);
+	if (ret != SQLITE_OK)
+		return db_err(pi, ret);
 
-		if (priv->commit_time >= t_now)
-			priv->commit_time = 0;		/* release commit lock */
-
-		if (priv->overlimit_msg)
-			priv->overlimit_msg = 0;
-
-		return rows;
-	}
-
- err_rollback:
-	if (sqlite3_errcode(priv->dbh) == SQLITE_LOCKED)
-		return 0;
+	sqlite3_reset(priv->p_stmt);
+	
+	pr_debug("%s: commited %d/%d rows\n", pi->id, rows, priv->num_rows);
 
-	sqlite3_exec(priv->dbh, "rollback", NULL, NULL, NULL);
+	delete_rows(pi, rows);
+	
+	if (priv->commit_time >= t_now)
+		priv->commit_time = 0;		/* release commit lock */
+	
+	if (priv->overlimit_msg)
+		priv->overlimit_msg = 0;
 
-	return -1;
+	return rows;
 }
 
 

-- 

[ULOGD RFC 21/30] Improve select performance

[heitzenberger]

[-- Attachment #1: ulogd-select-performance-improvement.diff --]
[-- Type: text/plain, Size: 5354 bytes --]

The previous code consumed quite lots of CPU cycles because of 
inefficiently handling fd_sets.

This patch corrects that.

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/src/select.c
===================================================================
--- ulogd-netfilter.orig/src/select.c
+++ ulogd-netfilter/src/select.c
@@ -24,86 +24,125 @@
 #include <fcntl.h>
 #include <ulogd/ulogd.h>
 #include <ulogd/common.h>
+#include <ulogd/signal.h>
 #include <ulogd/linuxlist.h>
 
-static int maxfd = 0;
+static fd_set readset, writeset, exceptset;
+static int maxfd = -1;
 static LLIST_HEAD(ulogd_fds);
 
-int ulogd_register_fd(struct ulogd_fd *fd)
+
+static int
+set_nonblock(int fd)
 {
 	int flags;
 
-	/* make FD nonblocking */
-	flags = fcntl(fd->fd, F_GETFL);
-	if (flags < 0)
+	if ((flags = fcntl(fd, F_GETFL)) < 0)
 		return -1;
+
 	flags |= O_NONBLOCK;
-	flags = fcntl(fd->fd, F_SETFL, flags);
-	if (flags < 0)
+
+	if ((flags = fcntl(fd, F_SETFL, flags)) < 0)
+		return -1;
+
+	return 0;
+}
+
+int
+ulogd_register_fd(struct ulogd_fd *ufd)
+{
+	if (set_nonblock(ufd->fd) < 0)
 		return -1;
 
-	/* Register FD */
-	if (fd->fd > maxfd)
-		maxfd = fd->fd;
+	if (ufd->when & ULOGD_FD_READ)
+		FD_SET(ufd->fd, &readset);
+	
+	if (ufd->when & ULOGD_FD_WRITE)
+		FD_SET(ufd->fd, &writeset);
+	
+	if (ufd->when & ULOGD_FD_EXCEPT)
+		FD_SET(ufd->fd, &exceptset);
+
+	if (ufd->fd > maxfd)
+		maxfd = ufd->fd;
 
-	llist_add_tail(&fd->list, &ulogd_fds);
+	llist_add_tail(&ufd->list, &ulogd_fds);
 
 	return 0;
 }
 
-void ulogd_unregister_fd(struct ulogd_fd *fd)
+void
+ulogd_unregister_fd(struct ulogd_fd *ufd)
 {
-	llist_del(&fd->list);
+	if (ufd->when & ULOGD_FD_READ)
+		FD_CLR(ufd->fd, &readset);
+	
+	if (ufd->when & ULOGD_FD_WRITE)
+		FD_CLR(ufd->fd, &writeset);
+	
+	if (ufd->when & ULOGD_FD_EXCEPT)
+		FD_CLR(ufd->fd, &exceptset);
+
+	llist_del(&ufd->list);
+
+	maxfd = -1;
+	llist_for_each_entry(ufd, &ulogd_fds, list) {
+		assert(ufd->fd >= 0);
+
+		if (ufd->fd > maxfd)
+			maxfd = ufd->fd;
+	}
 }
 
-int ulogd_select_main()
+/* ulogd_dispatch() - dispatch events */
+int
+ulogd_dispatch(void)
 {
-	struct ulogd_fd *ufd;
-	fd_set readset, writeset, exceptset;
-	struct timeval tv = { .tv_sec = 1, };
-	int i;
-
-	FD_ZERO(&readset);
-	FD_ZERO(&writeset);
-	FD_ZERO(&exceptset);
+	fd_set rds_tmp, wrs_tmp, exs_tmp;
+	sigset_t curr_ss;
 
-	/* prepare read and write fdsets */
-	llist_for_each_entry(ufd, &ulogd_fds, list) {
-		if (ufd->when & ULOGD_FD_READ)
-			FD_SET(ufd->fd, &readset);
+	ulogd_get_sigset(&curr_ss);
 
-		if (ufd->when & ULOGD_FD_WRITE)
-			FD_SET(ufd->fd, &writeset);
+	pthread_sigmask(SIG_UNBLOCK, &curr_ss, NULL);
 
-		if (ufd->when & ULOGD_FD_EXCEPT)
-			FD_SET(ufd->fd, &exceptset);
-	}
+	for (;;) {
+		struct ulogd_fd *ufd;
+		int n;
 
- again:
-	i = select(maxfd+1, &readset, &writeset, &exceptset, &tv);
-	if (i < 0) {
-		if (errno == EINTR)
-			goto again;
-	}
+	again:
+		rds_tmp = readset;
+		wrs_tmp = writeset;
+		exs_tmp = exceptset;
 
-	if (i > 0) {
-		/* call registered callback functions */
-		llist_for_each_entry(ufd, &ulogd_fds, list) {
-			int flags = 0;
-
-			if (FD_ISSET(ufd->fd, &readset))
-				flags |= ULOGD_FD_READ;
+		n = select(maxfd+1, &rds_tmp, &wrs_tmp, &exs_tmp, NULL);
+		if (n < 0) {
+			if (errno == EINTR)
+				goto again;
 
-			if (FD_ISSET(ufd->fd, &writeset))
-				flags |= ULOGD_FD_WRITE;
+			ulogd_log(ULOGD_FATAL, "select: %m\n");
 
-			if (FD_ISSET(ufd->fd, &exceptset))
-				flags |= ULOGD_FD_EXCEPT;
+			break;
+		}
 
-			if (flags)
-				ufd->cb(ufd->fd, flags, ufd->data);
+		if (n > 0) {
+			/* call registered callback functions */
+			llist_for_each_entry(ufd, &ulogd_fds, list) {
+				int flags = 0;
+
+				if (FD_ISSET(ufd->fd, &rds_tmp))
+					flags |= ULOGD_FD_READ;
+
+				if (FD_ISSET(ufd->fd, &wrs_tmp))
+					flags |= ULOGD_FD_WRITE;
+
+				if (FD_ISSET(ufd->fd, &exs_tmp))
+					flags |= ULOGD_FD_EXCEPT;
+
+				if (flags)
+					ufd->cb(ufd->fd, flags, ufd->data);
+			}
 		}
 	}
 
-	return i;
+	return 0;
 }
Index: ulogd-netfilter/src/ulogd.c
===================================================================
--- ulogd-netfilter.orig/src/ulogd.c
+++ ulogd-netfilter/src/ulogd.c
@@ -727,32 +727,6 @@ out_buf:
 	return ret;
 }
 	
-
-static int ulogd_main_loop(void)
-{
-	sigset_t curr;
-	int ret = 0;
-
-	ulogd_get_sigset(&curr);
-
-	pthread_sigmask(SIG_UNBLOCK, &curr, NULL);
-
-	for (;;) {
-		ret = ulogd_select_main();
-		if (ret == 0) 
-			continue;
-
-		if (ret < 0) {
-			ulogd_log(ULOGD_ERROR, "select returned %s\n",
-					  strerror(errno));
-			break;
-		}
-	}
-
-	return ret;
-}
-
-/* open the logfile */
 static int logfile_open(const char *name)
 {
 	if (name)
@@ -1112,7 +1086,7 @@ main(int argc, char* argv[])
 
 	ulogd_log(ULOGD_INFO, "entering main loop\n");
 
-	ulogd_main_loop();
+	ulogd_dispatch();
 
 	return 0;
 }
Index: ulogd-netfilter/include/ulogd/ulogd.h
===================================================================
--- ulogd-netfilter.orig/include/ulogd/ulogd.h
+++ ulogd-netfilter/include/ulogd/ulogd.h
@@ -243,7 +243,7 @@ struct ulogd_fd {
 
 int ulogd_register_fd(struct ulogd_fd *ufd);
 void ulogd_unregister_fd(struct ulogd_fd *ufd);
-int ulogd_select_main();
+int ulogd_dispatch(void);
 
 /***********************************************************************
  * timer handling (timer.c)

-- 

[ULOGD RFC 22/30] Add set_sockbuf_len()

[heitzenberger]

[-- Attachment #1: ulogd-add-set_sockbuf_len.diff --]
[-- Type: text/plain, Size: 2483 bytes --]

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/include/ulogd/common.h
===================================================================
--- ulogd-netfilter.orig/include/ulogd/common.h
+++ ulogd-netfilter/include/ulogd/common.h
@@ -47,4 +47,6 @@
 #define pr_debug(fmt, ...)
 #endif /* DEBUG */
 
+int set_sockbuf_len(int fd, int rcv_len, int snd_len);
+
 #endif /* COMMON_H */
Index: ulogd-netfilter/src/common.c
===================================================================
--- /dev/null
+++ ulogd-netfilter/src/common.c
@@ -0,0 +1,52 @@
+/*
+ * common.c
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2
+ * as published by the Free Software Foundation
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+ *
+ * Holger Eitzenberger <holger@eitzenberger.org>, 2007.
+ */
+#include <ulogd/ulogd.h>
+#include <ulogd/common.h>
+#include <sys/types.h>
+#include <sys/socket.h>
+
+
+int
+set_sockbuf_len(int fd, int rcv_len, int snd_len)
+{
+	int ret;
+
+	pr_debug("%s: fd=%d rcv-len=%d snd-len\n", __func__, fd, rcv_len,
+			 snd_len);
+
+	if (snd_len > 0) {
+		ret = setsockopt(fd, SOL_SOCKET, SO_SNDBUF, &snd_len,
+						 sizeof(snd_len));
+		if (ret < 0) {
+			ulogd_log(ULOGD_ERROR, "setsockopt: SO_SNDBUF: %m\n");
+			return -1;
+		}
+	}
+
+	if (rcv_len > 0) {
+		ret = setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &rcv_len,
+						 sizeof(rcv_len));
+		if (ret < 0) {
+			ulogd_log(ULOGD_ERROR, "setsockopt: SO_RCVBUF: %m\n");
+			return -1;
+		}
+	}
+
+	return 0;
+}
Index: ulogd-netfilter/src/Makefile.am
===================================================================
--- ulogd-netfilter.orig/src/Makefile.am
+++ ulogd-netfilter/src/Makefile.am
@@ -5,5 +5,5 @@ AM_CPPFLAGS = $(all_includes) -I$(top_sr
 
 sbin_PROGRAMS = ulogd
 
-ulogd_SOURCES = ulogd.c ifi.c select.c timer.c signal.c conffile.c
+ulogd_SOURCES = ulogd.c ifi.c select.c timer.c signal.c common.c conffile.c
 ulogd_LDFLAGS = -export-dynamic -lpthread

-- 

Re: [ULOGD RFC 22/30] Add set_sockbuf_len()

[Pablo Neira Ayuso]

heitzenberger@astaro.com wrote:
> +int
> +set_sockbuf_len(int fd, int rcv_len, int snd_len)

AFAIK, there's a similar function in libnfnetlink which uses
SO_RCVBUF_FORCE instead.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: [ULOGD RFC 22/30] Add set_sockbuf_len()

[Holger Eitzenberger]

Pablo Neira Ayuso <pablo@netfilter.org> writes:

> AFAIK, there's a similar function in libnfnetlink which uses
> SO_RCVBUF_FORCE instead.

Thanks for the info.  libnl also implements a similar kind of
function.  At the time I completely switch that I'll of course
use that one.

 /holger

Re: [ULOGD RFC 22/30] Add set_sockbuf_len()

[Patrick McHardy]

Holger Eitzenberger wrote:
> Pablo Neira Ayuso <pablo@netfilter.org> writes:
> 
>> AFAIK, there's a similar function in libnfnetlink which uses
>> SO_RCVBUF_FORCE instead.
> 
> Thanks for the info.  libnl also implements a similar kind of
> function.  At the time I completely switch that I'll of course
> use that one.


I guess thats also a simple patch on top. I'll see whether
I can apply there patches while switching to the libnfnetlink
function. I guess it shouldn't cause any conflicts since its
only a single caller from what I can see.

[ULOGD RFC 23/30] NFCT: make sequence cache bigger, make handling smarter

[heitzenberger]

[-- Attachment #1: ulogd-NFCT-make-scache-scale-better.diff --]
[-- Type: text/plain, Size: 8870 bytes --]

Increase sequence cache and therefore make cache handling smarter for
it too.

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/input/flow/ulogd_inpflow_NFCT.c
===================================================================
--- ulogd-netfilter.orig/input/flow/ulogd_inpflow_NFCT.c
+++ ulogd-netfilter/input/flow/ulogd_inpflow_NFCT.c
@@ -42,10 +42,13 @@
 						 | NF_NETLINK_CONNTRACK_DESTROY)
 
 /* configuration defaults */
-#define TCACHE_SIZE       8192
-#define SCACHE_SIZE       64
-#define SCACHE_REQ_MAX    100
-#define TIMEOUT           30 SEC
+#define TCACHE_SIZE		8192
+#define SCACHE_SIZE		512
+#define TCACHE_REQ_MAX	100
+#define TIMEOUT			30 SEC
+
+#define RCVBUF_LEN		(1 << 18)
+#define SNDBUF_LEN		RCVBUF_LEN
 
 typedef enum TIMES_ { START, UPDATE, STOP, __TIME_MAX } TIMES;
 typedef unsigned conntrack_hash_t;
@@ -429,7 +432,7 @@ nfct_get_conntrack_x(struct nfct_handle 
 {
 	static char buf[NFNL_BUFFSIZE];
 	struct nfnlhdr *req = (void *)buf;
-	int cta_dir, nbytes;
+	int cta_dir;
 
 	memset(buf, 0, sizeof(buf));
 
@@ -445,9 +448,17 @@ nfct_get_conntrack_x(struct nfct_handle 
 
 	nfct_build_tuple(req, sizeof(buf), t, cta_dir);
 
-	nbytes = nfnl_send(nfct_nfnlh(cth), &req->nlh);
+	return nfnl_send(nfct_nfnlh(cth), &req->nlh);
+}
+
+/* time diff with second resolution */
+static inline unsigned
+tv_diff_sec(const struct timeval *tv1, const struct timeval *tv2)
+{
+	if (tv2->tv_sec >= tv1->tv_sec)
+		return max(tv2->tv_sec - tv1->tv_sec, 1);
 
-	return nbytes;
+	return tv1->tv_sec - tv2->tv_sec;
 }
 
 struct conntrack *
@@ -483,6 +494,85 @@ ct_put(struct conntrack *ct)
 	}
 }
 
+static struct cache *
+cache_alloc(int cache_size)
+{
+	struct cache *c;
+	int i;
+
+	c = malloc(sizeof(*c) + sizeof(struct cache_head) * cache_size);
+	if (c == NULL)
+		return NULL;
+
+	c->c_head = (void *)c + sizeof(*c);
+	c->c_num_heads = cache_size;
+	c->c_curr_head = 0;
+	c->c_cnt = 0;
+
+	for (i = 0; i < c->c_num_heads; i++) {
+		INIT_LLIST_HEAD(&c->c_head[i].link);
+		c->c_head[i].cnt = 0;
+	}
+
+	return c;
+}
+
+static void
+cache_free(struct cache *c)
+{
+	int i;
+
+	for (i = 0; i < c->c_num_heads; i++) {
+		struct llist_head *ptr, *ptr2;
+
+		llist_for_each_safe(ptr, ptr2, &c->c_head[i].link)
+			free(container_of(ptr, struct conntrack, list));
+	}
+
+	free(c);
+}
+
+int
+cache_add(struct cache *c, struct conntrack *ct)
+{
+	ct_get(ct);
+
+	ct->time[UPDATE].tv_sec = ct->time[START].tv_sec = t_now_local;
+
+	/* order of these two is important for debugging purposes */
+	c->c_cnt++;
+	c->c_add(c, ct);
+
+	return 0;
+}
+
+int
+cache_del(struct cache *c, struct conntrack *ct)
+{
+	assert(c->c_cnt > 0);
+	assert(ct->used > 0);
+
+	/* order of these two is important for debugging purposes */
+	c->c_del(c, ct);
+	c->c_cnt--;
+
+	ct_put(ct);
+
+	return 0;
+}
+
+static inline conntrack_hash_t
+cache_head_next(const struct cache *c)
+{
+	return (c->c_curr_head + 1) % c->c_num_heads;
+}
+
+static inline conntrack_hash_t
+cache_slice_end(const struct cache *c, unsigned n)
+{
+	return (c->c_curr_head + n) % c->c_num_heads;
+}
+
 /* tuple cache */
 static struct conntrack ct_search;		/* used by scache too */
 
@@ -549,6 +639,55 @@ tcache_find(const struct ulogd_pluginsta
 	return NULL;
 }
 
+/* check entries in tuple cache */
+static int
+tcache_cleanup(struct ulogd_pluginstance *pi)
+{
+	struct nfct_pluginstance *priv = (void *)pi->private;
+	struct cache *c = priv->tcache;
+	conntrack_hash_t end = cache_slice_end(c, 32);
+	struct conntrack *ct;
+	int ret, req = 0;
+
+	do {
+		llist_for_each_entry_reverse(ct, &c->c_head[c->c_curr_head].link,
+									 list) {
+			if (tv_diff_sec(&ct->time[UPDATE], &tv_now) < timeout_ce(pi))
+				continue;
+
+			/* check if its still there */
+			ret = nfct_get_conntrack_x(priv->cth, &ct->tuple,
+									   NFCT_DIR_ORIGINAL, &ct->last_seq);
+			if (ret < 0) {
+				if (errno == EWOULDBLOCK)
+					break;
+
+				ulogd_log(ULOGD_ERROR, "nfct_get_conntrack: ct=%p: %m\n",
+						  ct);
+				break;
+			}
+
+			if (&ct->last_seq != 0) {
+				ct->t_req = t_now;
+
+				assert(scache_find(pi, ct->last_seq) == NULL);
+
+				cache_add(priv->scache, ct);
+			}
+
+			if (++req > TCACHE_REQ_MAX)
+				break;
+		}
+
+		c->c_curr_head = cache_head_next(c);
+
+		if (req > TCACHE_REQ_MAX)
+			break;
+	} while (c->c_curr_head != end);
+
+	return req;
+}
+
 /* sequence cache */
 static conntrack_hash_t
 scache_hash(struct cache *c, struct conntrack *ct)
@@ -612,81 +751,39 @@ scache_find(const struct ulogd_pluginsta
 	return NULL;
 }
 
-static struct cache *
-cache_alloc(int cache_size)
-{
-	struct cache *c;
-	int i;
-
-	c = malloc(sizeof(*c) + sizeof(struct cache_head) * cache_size);
-	if (c == NULL)
-		return NULL;
-
-	c->c_head = (void *)c + sizeof(*c);
-	c->c_num_heads = cache_size;
-	c->c_curr_head = 0;
-	c->c_cnt = 0;
-
-	for (i = 0; i < c->c_num_heads; i++) {
-		INIT_LLIST_HEAD(&c->c_head[i].link);
-		c->c_head[i].cnt = 0;
-	}
-	
-	return c;
-}
-
-static void
-cache_free(struct cache *c)
-{
-	int i;
-
-	for (i = 0; i < c->c_num_heads; i++) {
-		struct llist_head *ptr, *ptr2;
-
-		llist_for_each_safe(ptr, ptr2, &c->c_head[i].link)
-			free(container_of(ptr, struct conntrack, list));
-	}
-
-	free(c);
-}
-
-int
-cache_add(struct cache *c, struct conntrack *ct)
+static int
+scache_cleanup(struct ulogd_pluginstance *pi)
 {
-	ct_get(ct);
+	struct nfct_pluginstance *priv = (void *)pi->private;
+	struct cache *c = priv->scache;
+	conntrack_hash_t end = cache_slice_end(c, 16);
+	struct conntrack *ct;
+	int del = 0;
 
-	ct->time[UPDATE].tv_sec = ct->time[START].tv_sec = t_now_local;
+	if (c->c_cnt == 0)
+		return 0;
 
-	/* order of these two is important for debugging purposes */
-	c->c_cnt++;
-	c->c_add(c, ct);
+	do {
+		struct llist_head *curr, *tmp;
 
-	return 0;
-}
+		assert(c->c_curr_head < c->c_num_heads);
 
-int
-cache_del(struct cache *c, struct conntrack *ct)
-{
-	assert(c->c_cnt > 0);
-	assert(ct->used > 0);
+		llist_for_each_prev_safe(curr, tmp, &c->c_head[c->c_curr_head].link) {
+			ct = container_of(curr, struct conntrack, seq_link);
 
-	/* order of these two is important for debugging purposes */
-	c->c_del(c, ct);
-	c->c_cnt--;
+			assert(ct->t_req != 0);
 
-	ct_put(ct);
+			if ((t_now - ct->t_req) < 5 SEC)
+				break;
 
-	return 0;
-}
+			cache_del(priv->scache, ct);
+			del++;
+		}
 
-/* time diff with second resolution */
-static inline unsigned
-tv_diff_sec(const struct timeval *tv1, const struct timeval *tv2)
-{
-	if (tv2->tv_sec >= tv1->tv_sec)
-		return max(tv2->tv_sec - tv1->tv_sec, 1);
+		c->c_curr_head = cache_head_next(c);
+	} while (c->c_curr_head != end);
 
-	return tv1->tv_sec - tv2->tv_sec;
+	return del;
 }
 
 static int
@@ -874,9 +971,6 @@ read_cb_nfct(int fd, unsigned what, void
 	return nfnl_recv_msgs(nfct_nfnlh(priv->cth), do_nfct_msg, pi);
 }
 
-
-#define STOP_HERE(h)	(((h)->c_curr_head + 32) % (h)->c_num_heads)
-
 /*
   nfct_timer_cb()
 
@@ -887,56 +981,21 @@ nfct_timer_cb(struct ulogd_timer *t)
 {
 	struct ulogd_pluginstance *pi = t->data;
 	struct nfct_pluginstance *priv = (void *)pi->private;
-	struct cache *ht = priv->tcache;
-	int i, req = 0, end = STOP_HERE(ht);
-	struct conntrack *ct;
-
-	/* cleanup stale entries from sequence cache */
-	for (i = 0; i < priv->scache->c_num_heads; i++) {
-		if (llist_empty(&priv->scache->c_head[i].link))
-			continue;
-
-		ct = container_of(priv->scache->c_head[i].link.prev,
-						  struct conntrack, seq_link);
-
-		assert(ct->t_req != 0);
-
-		if (ct->t_req > 0 && (t_now - ct->t_req) > 5 SEC)
-			cache_del(priv->scache, ct);
-	}
-
-	/* check entries in tuple cache */
-	do {
-		assert(ht->c_curr_head < ht->c_num_heads);
-
-		llist_for_each_entry(ct, &ht->c_head[ht->c_curr_head].link, list) {
-			if (tv_diff_sec(&ct->time[UPDATE], &tv_now) < timeout_ce(pi))
-				continue;
+	unsigned sc_start, sc_end, tc_start, tc_end;
 
-			/* check if its still there */
-			nfct_get_conntrack_x(priv->cth, &ct->tuple, NFCT_DIR_ORIGINAL,
-								 &ct->last_seq);
+	sc_start = priv->scache->c_curr_head;
+	tc_start = priv->tcache->c_curr_head;
 
-			if (&ct->last_seq != 0) {
-				ct->t_req = t_now;
+	scache_cleanup(pi);
+	tcache_cleanup(pi);
 
-				assert(scache_find(pi, ct->last_seq) == NULL);
-
-				cache_add(priv->scache, ct);
-			}
-
-			if (++req > SCACHE_REQ_MAX)
-				break;
-		}
-
-		ht->c_curr_head = (ht->c_curr_head + 1) % ht->c_num_heads;
-
-		if (req > SCACHE_REQ_MAX)
-			break;
-	} while (ht->c_curr_head != end);
+	sc_end = priv->scache->c_curr_head;
+	tc_end = priv->tcache->c_curr_head;
 
-	ulogd_log(ULOGD_DEBUG, "%s: ct:%u t:%u s:%u\n", pi->id,
-			  num_conntrack, priv->tcache->c_cnt, priv->scache->c_cnt);
+	ulogd_log(ULOGD_DEBUG, "%s: ct=%u t=%u [%u,%u[ s=%u [%u,%u[\n",
+			  pi->id, num_conntrack,
+			  priv->tcache->c_cnt, tc_start, tc_end,
+			  priv->scache->c_cnt, sc_start, sc_end);
 }
 
 static int

-- 

[ULOGD RFC 24/30] NFCT: increase socket buffers

[heitzenberger]

[-- Attachment #1: ulogd-NFCT-inc-sockbuf.diff --]
[-- Type: text/plain, Size: 677 bytes --]

Might reduce number of ENOBUFS.

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/input/flow/ulogd_inpflow_NFCT.c
===================================================================
--- ulogd-netfilter.orig/input/flow/ulogd_inpflow_NFCT.c
+++ ulogd-netfilter/input/flow/ulogd_inpflow_NFCT.c
@@ -1074,6 +1074,9 @@ nfct_start(struct ulogd_pluginstance *up
 
 	ulogd_log(ULOGD_DEBUG, "%s: ctnetlink connection opened\n", upi->id);
 
+    if (set_sockbuf_len(nfct_fd(priv->cth), RCVBUF_LEN, SNDBUF_LEN) < 0)
+	        goto err_free;
+
 	priv->nfct_fd.fd = nfct_fd(priv->cth);
 	priv->nfct_fd.cb = &read_cb_nfct;
 	priv->nfct_fd.data = upi;

-- 

[ULOGD RFC 25/30] Introduce global state, skip some stacks during reconfiguration

[heitzenberger]

[-- Attachment #1: ulogd-fix-races-during-reconf.diff --]
[-- Type: text/plain, Size: 7678 bytes --]

Some plugins are not reconfigurable, therefore skips entire stacks if
there is at least one plugin which is not reconfigurable.

Also introduce a global state to counter some pathological
reconfiguration issues.

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/src/ulogd.c
===================================================================
--- ulogd-netfilter.orig/src/ulogd.c
+++ ulogd-netfilter/src/ulogd.c
@@ -83,6 +83,7 @@ static FILE *logfile = NULL;		/* logfile
 static char *ulogd_configfile = ULOGD_CONFIGFILE;
 static char *ulogd_logfile = ULOGD_LOGFILE_DEFAULT;
 static FILE syslog_dummy;
+static enum GlobalState state;
 
 /* linked list for all registered plugins */
 static LLIST_HEAD(ulogd_plugins);
@@ -129,6 +130,19 @@ static struct config_keyset ulogd_kset =
 #define loglevel_ce	ulogd_kset.ces[2]
 #define stack_ce	ulogd_kset.ces[3]
 
+
+void
+ulogd_set_state(enum GlobalState s)
+{
+	state = s;
+}
+
+enum GlobalState
+ulogd_get_state(void)
+{
+	return state;
+}
+
 /***********************************************************************
  * UTILITY FUNCTIONS FOR PLUGINS
  ***********************************************************************/
@@ -195,8 +209,8 @@ int ulogd_wildcard_inputkeys(struct ulog
 
 	/* first pass: count keys */
 	llist_for_each_entry(pi_cur, &stack->list, list) {
-		ulogd_log(ULOGD_DEBUG, "iterating over pluginstance '%s'\n",
-			  pi_cur->id);
+		ulogd_log(ULOGD_DEBUG, "%s: iterating over pluginstance '%s'\n",
+				  upi->id, pi_cur->id);
 		num_keys += pi_cur->plugin->output.num_keys;
 	}
 
@@ -209,8 +223,12 @@ int ulogd_wildcard_inputkeys(struct ulog
 	llist_for_each_entry(pi_cur, &stack->list, list) {
 		int i;
 
-		for (i = 0; i < pi_cur->plugin->output.num_keys; i++)
+		for (i = 0; i < pi_cur->plugin->output.num_keys; i++) {
+			pr_debug("%s: copy key '%s' from plugin '%s'\n", upi->id,
+					 pi_cur->plugin->output.keys[i].name, pi_cur->id);
+
 			upi->input.keys[index++] = pi_cur->output.keys[i];
+		}
 	}
 
 	upi->input.num_keys = num_keys;
@@ -650,6 +668,10 @@ static int create_stack(const char *opti
 	}
 	INIT_LLIST_HEAD(&stack->list);
 
+	/* By default a stack is reconfigurable unless there is a plugin
+	   which is not reconfigurable */
+	stack->flags = ULOGD_SF_RECONF;
+
 	ulogd_log(ULOGD_DEBUG, "building new pluginstance stack (%s):\n",
 		  option);
 
@@ -698,6 +720,9 @@ static int create_stack(const char *opti
 			
 		ulogd_log(ULOGD_DEBUG, "pushing `%s' on stack\n", pl->name);
 		llist_add_tail(&pi->list, &stack->list);
+
+		if ((pi->plugin->flags & ULOGD_PF_RECONF) == 0)
+			stack->flags &= ~ULOGD_SF_RECONF;
 	}
 
 	/* PASS 2: resolve key connections from bottom to top of stack */
@@ -788,10 +813,20 @@ static int parse_conffile(const char *se
 	return 1;
 }
 
+/**
+ * Call callback on each ulogd_pluginstance.
+ * @arg cb      Callback to call for each ulogd_pluginstance.
+ * @arg arg     Argument to pass to each call to argument.
+ *
+ * Call callback on each ulogd_pluginstance, skipping stacks which are
+ * not reconfigurable.
+ *
+ * @return Number of calls.
+ */
 static int
 for_each_pluginstance(int (* cb)(struct ulogd_pluginstance *,
 								 struct ulogd_pluginstance_stack *,
-								 void *), void *arg)
+								 unsigned long), unsigned long arg)
 {
 	struct ulogd_pluginstance_stack *stack;
 	int sum = 0;
@@ -801,6 +836,9 @@ for_each_pluginstance(int (* cb)(struct 
 	llist_for_each_entry(stack, &ulogd_pi_stacks, stack_list) {
 		struct ulogd_pluginstance *pi;
 
+		if ((stack->flags & ULOGD_SF_RECONF) == 0)
+			continue;
+
 		llist_for_each_entry(pi, &stack->list, list) {
 			int ret;
 
@@ -824,14 +862,15 @@ enum ReconfOp
 
 static int
 _do_reconf(struct ulogd_pluginstance *pi,
-		   struct ulogd_pluginstance_stack *stack, void *arg)
+		   struct ulogd_pluginstance_stack *stack, unsigned long arg)
 {
-	enum ReconfOp op = (unsigned)arg;
+	enum ReconfOp op = arg;
 	int ret = 0;
 
 	assert(pi != NULL);
 
-	if ((pi->plugin->flags & ULOGD_PF_RECONF) == 0)
+	/* check whether stack is reconfigurable */
+	if ((stack->flags & ULOGD_SF_RECONF) == 0)
 		return 0;
 
 	switch (op) {
@@ -839,16 +878,12 @@ _do_reconf(struct ulogd_pluginstance *pi
 		ret = pi->plugin->stop(pi);
 		break;
 
-	case CONFIGURE:
-		ulogd_pluginstance_reset_cfg(pi);
-		ret = pi->plugin->configure(pi, stack);
-		break;
-
 	case START:
 		ret = pi->plugin->start(pi);
 		break;
 
 	default:
+		ulogd_log(ULOGD_ERROR, "%s: invalid op '%d'\n", __func__, op);
 		return -1;
 	}
 
@@ -863,25 +898,43 @@ _do_reconf(struct ulogd_pluginstance *pi
 static int
 reconfigure_plugins(void)
 {
+	struct ulogd_pluginstance_stack *stack;
+
 	ulogd_log(ULOGD_INFO, "reconfiguring plugins\n");
 
-	if (for_each_pluginstance(_do_reconf, (void *)STOP) < 0)
-		abort();
+	ulogd_set_state(GS_INITIALIZING);
 
-	if (for_each_pluginstance(_do_reconf, (void *)CONFIGURE) < 0)
-		abort();
+	/* Skip stacks which contain at least one plugin which is not
+	   reconfigurable.  This a workaround until all plugins are
+	   reconfigurable. */
+	if (for_each_pluginstance(_do_reconf, STOP) < 0)
+		goto err;
 
-	if (for_each_pluginstance(_do_reconf, (void *)START) < 0)
-		abort();
+	llist_for_each_entry(stack, &ulogd_pi_stacks, stack_list) {
+		if ((stack->flags & ULOGD_SF_RECONF) == 0)
+			continue;
+
+		if (create_stack_resolve_keys(stack) < 0)
+			goto err;
+	}
+
+	if (for_each_pluginstance(_do_reconf, START) < 0)
+		goto err;
+
+	ulogd_set_state(GS_RUNNING);
 
 	return 0;
+
+ err:
+ 	ulogd_log(ULOGD_FATAL, "failed to reconfigure\n");
+ 	return -1;
 }
 
 static int
 _do_signal(struct ulogd_pluginstance *pi,
-		   struct ulogd_pluginstance_stack *stack, void *arg)
+		   struct ulogd_pluginstance_stack *stack, unsigned long arg)
 {
-	int signo = (int) arg;
+	int signo = (int)arg;
 
 	if (pi->plugin->signal) {
 		pi->plugin->signal(pi, signo);
@@ -897,6 +950,9 @@ sync_sig_handler(int signo)
 {
 	pr_debug("%s: signal '%d' received\n", __func__, signo);
 
+	if (ulogd_get_state() != GS_RUNNING)
+		return;
+
 	switch (signo) {
 	case SIGHUP:
 		reconfigure_plugins();
@@ -913,7 +969,7 @@ sync_sig_handler(int signo)
 		break;
 	}
 
-	for_each_pluginstance(_do_signal, (void *) signo);
+	for_each_pluginstance(_do_signal, (unsigned long)signo);
 }
 
 static void
@@ -965,6 +1021,7 @@ main(int argc, char* argv[])
 	uid_t uid = 0;
 	gid_t gid = 0;
 
+	ulogd_set_state(GS_INITIALIZING);
 
 	while ((argch = getopt_long(argc, argv, "c:dh::Vu:", opts, NULL)) != -1) {
 		switch (argch) {
@@ -1086,6 +1143,8 @@ main(int argc, char* argv[])
 
 	ulogd_log(ULOGD_INFO, "entering main loop\n");
 
+	ulogd_set_state(GS_RUNNING);
+
 	ulogd_dispatch();
 
 	return 0;
Index: ulogd-netfilter/include/ulogd/ulogd.h
===================================================================
--- ulogd-netfilter.orig/include/ulogd/ulogd.h
+++ ulogd-netfilter/include/ulogd/ulogd.h
@@ -189,17 +189,29 @@ struct ulogd_pluginstance {
 	char private[0];
 };
 
+/* stack flags */
+#define ULOGD_SF_RECONF			0x00000001 /* stack is reconfigurable */
+
 struct ulogd_pluginstance_stack {
 	/* global list of pluginstance stacks */
 	struct llist_head stack_list;
 	/* list of plugins in this stack */
 	struct llist_head list;
 	char *name;
+	unsigned flags;
 };
 
 /***********************************************************************
  * PUBLIC INTERFACE 
  ***********************************************************************/
+enum GlobalState {
+	GS_INVAL = 0,
+	GS_INITIALIZING,			/* also reconfiguration */
+	GS_RUNNING,
+};
+
+void ulogd_set_state(enum GlobalState);
+enum GlobalState ulogd_get_state(void);
 
 void ulogd_propagate_results(struct ulogd_pluginstance *pi);
 

-- 

[ULOGD RFC 26/30] llist: turn poisoning off by default

[heitzenberger]

[-- Attachment #1: ulogd-llist-poison-debug-only.diff --]
[-- Type: text/plain, Size: 573 bytes --]

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/include/ulogd/linuxlist.h
===================================================================
--- ulogd-netfilter.orig/include/ulogd/linuxlist.h
+++ ulogd-netfilter/include/ulogd/linuxlist.h
@@ -117,8 +117,10 @@ static inline void __llist_del(struct ll
 static inline void llist_del(struct llist_head *entry)
 {
 	__llist_del(entry->prev, entry->next);
+#ifdef ULOGD_LLIST_DEBUG
 	entry->next = LLIST_POISON1;
 	entry->prev = LLIST_POISON2;
+#endif /* ULOGD_LLIST_DEBUG */
 }
 
 /**

-- 

[ULOGD RFC 27/30] NFCT: cleanup direction handling

[heitzenberger]

[-- Attachment #1: ulogd-NFCT-tuple-indexing-cleanup.diff --]
[-- Type: text/plain, Size: 6545 bytes --]

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/input/flow/ulogd_inpflow_NFCT.c
===================================================================
--- ulogd-netfilter.orig/input/flow/ulogd_inpflow_NFCT.c
+++ ulogd-netfilter/input/flow/ulogd_inpflow_NFCT.c
@@ -41,6 +41,9 @@
 						 | NF_NETLINK_CONNTRACK_UPDATE \
 						 | NF_NETLINK_CONNTRACK_DESTROY)
 
+#define ORIG	NFCT_DIR_ORIGINAL
+#define REPL	NFCT_DIR_REPLY
+
 /* configuration defaults */
 #define TCACHE_SIZE		8192
 #define SCACHE_SIZE		512
@@ -425,14 +428,17 @@ nfct_msg_type(const struct nlmsghdr *nlh
 }
 
 
-/* seq: sequence number used for the request */
+/*
+ * nfct_get_conntrack_seq()
+ *
+ * Do GET_CONNTRACK, return seq# used.
+ */
 static int
-nfct_get_conntrack_x(struct nfct_handle *cth, struct nfct_tuple *t,
-					 int dir, uint32_t *seq)
+nfct_get_conntrack_seq(struct nfct_handle *cth, struct nfct_tuple *t,
+					  uint32_t *seq)
 {
 	static char buf[NFNL_BUFFSIZE];
 	struct nfnlhdr *req = (void *)buf;
-	int cta_dir;
 
 	memset(buf, 0, sizeof(buf));
 
@@ -444,9 +450,7 @@ nfct_get_conntrack_x(struct nfct_handle 
 	if (seq != NULL)
 		*seq = req->nlh.nlmsg_seq;
 
-	cta_dir = (dir == NFCT_DIR_ORIGINAL) ? CTA_TUPLE_ORIG : CTA_TUPLE_REPLY;
-
-	nfct_build_tuple(req, sizeof(buf), t, cta_dir);
+	nfct_build_tuple(req, sizeof(buf), t, CTA_TUPLE_ORIG);
 
 	return nfnl_send(nfct_nfnlh(cth), &req->nlh);
 }
@@ -656,8 +660,8 @@ tcache_cleanup(struct ulogd_pluginstance
 				continue;
 
 			/* check if its still there */
-			ret = nfct_get_conntrack_x(priv->cth, &ct->tuple,
-									   NFCT_DIR_ORIGINAL, &ct->last_seq);
+			ret = nfct_get_conntrack_seq(priv->cth, &ct->tuple,
+									   &ct->last_seq);
 			if (ret < 0) {
 				if (errno == EWOULDBLOCK)
 					break;
@@ -789,46 +793,48 @@ scache_cleanup(struct ulogd_pluginstance
 static int
 propagate_ct_flow(struct ulogd_pluginstance *upi, 
 				  struct nfct_conntrack *nfct, unsigned int flags,
-				  int dir, struct conntrack *ct)
+				  struct conntrack *ct)
 {
 	struct ulogd_key *ret = upi->output.keys;
 
-	ret[O_IP_SADDR].u.value.ui32 = htonl(nfct->tuple[0].src.v4);
+	ret[O_IP_SADDR].u.value.ui32 = htonl(nfct->tuple[ORIG].src.v4);
 	ret[O_IP_SADDR].flags |= ULOGD_RETF_VALID;
 
-	ret[O_IP_DADDR].u.value.ui32 = htonl(nfct->tuple[1].src.v4);
+	ret[O_IP_DADDR].u.value.ui32 = htonl(nfct->tuple[REPL].src.v4);
 	ret[O_IP_DADDR].flags |= ULOGD_RETF_VALID;
 
-	ret[O_IP_PROTO].u.value.ui8 = nfct->tuple[dir].protonum;
+	ret[O_IP_PROTO].u.value.ui8 = nfct->tuple[ORIG].protonum;
 	ret[O_IP_PROTO].flags |= ULOGD_RETF_VALID;
 
-	switch (nfct->tuple[dir].protonum) {
+	switch (nfct->tuple[ORIG].protonum) {
 	case IPPROTO_TCP:
 	case IPPROTO_UDP:
 	case IPPROTO_SCTP:
 		/* FIXME: DCCP */
-		ret[O_L4_SPORT].u.value.ui16 = htons(nfct->tuple[0].l4src.tcp.port);
+		ret[O_L4_SPORT].u.value.ui16
+			= htons(nfct->tuple[ORIG].l4src.tcp.port);
 		ret[O_L4_SPORT].flags |= ULOGD_RETF_VALID;
-		ret[O_L4_DPORT].u.value.ui16 = htons(nfct->tuple[1].l4src.tcp.port);
+		ret[O_L4_DPORT].u.value.ui16
+			= htons(nfct->tuple[REPL].l4src.tcp.port);
 		ret[O_L4_DPORT].flags |= ULOGD_RETF_VALID;
 		break;
 	case IPPROTO_ICMP:
-		ret[O_ICMP_CODE].u.value.ui8 = nfct->tuple[dir].l4src.icmp.code;
+		ret[O_ICMP_CODE].u.value.ui8 = nfct->tuple[ORIG].l4src.icmp.code;
 		ret[O_ICMP_CODE].flags |= ULOGD_RETF_VALID;
-		ret[O_ICMP_TYPE].u.value.ui8 = nfct->tuple[dir].l4src.icmp.type;
+		ret[O_ICMP_TYPE].u.value.ui8 = nfct->tuple[ORIG].l4src.icmp.type;
 		ret[O_ICMP_TYPE].flags |= ULOGD_RETF_VALID;
 		break;
 	}
 
 	if (flags & NFCT_COUNTERS_ORIG) {
-		ret[O_RAW_IN_PKTLEN].u.value.ui32 = nfct->counters[0].bytes;
+		ret[O_RAW_IN_PKTLEN].u.value.ui32 = nfct->counters[ORIG].bytes;
 		ret[O_RAW_IN_PKTLEN].flags |= ULOGD_RETF_VALID;
-		ret[O_RAW_IN_PKTCOUNT].u.value.ui32 = nfct->counters[0].packets;
+		ret[O_RAW_IN_PKTCOUNT].u.value.ui32 = nfct->counters[REPL].packets;
 		ret[O_RAW_IN_PKTCOUNT].flags |= ULOGD_RETF_VALID;
 
-		ret[O_RAW_OUT_PKTLEN].u.value.ui32 = nfct->counters[1].bytes;
+		ret[O_RAW_OUT_PKTLEN].u.value.ui32 = nfct->counters[REPL].bytes;
 		ret[O_RAW_OUT_PKTLEN].flags |= ULOGD_RETF_VALID;
-		ret[O_RAW_OUT_PKTCOUNT].u.value.ui32 = nfct->counters[1].packets;
+		ret[O_RAW_OUT_PKTCOUNT].u.value.ui32 = nfct->counters[REPL].packets;
 		ret[O_RAW_OUT_PKTCOUNT].flags |= ULOGD_RETF_VALID;
 	}
 
@@ -870,13 +876,13 @@ propagate_ct(struct ulogd_pluginstance *
 	struct nfct_pluginstance *priv = (void *)upi->private;
 
 	do {
-		if (nfct->tuple[NFCT_DIR_ORIGINAL].src.v4 == INADDR_LOOPBACK
-			|| nfct->tuple[NFCT_DIR_ORIGINAL].dst.v4 == INADDR_LOOPBACK)
+		if (nfct->tuple[ORIG].src.v4 == INADDR_LOOPBACK
+			|| nfct->tuple[ORIG].dst.v4 == INADDR_LOOPBACK)
 			break;
 
 		ct->time[STOP].tv_sec = t_now_local;
 
-		propagate_ct_flow(upi, nfct, flags, NFCT_DIR_ORIGINAL, ct);
+		propagate_ct_flow(upi, nfct, flags, ct);
 	} while (0);
 
 	cache_del(priv->tcache, ct);
@@ -900,8 +906,8 @@ do_nfct_msg(struct nlmsghdr *nlh, void *
 
 	bzero(&nfct, sizeof(nfct));
 
-	nfct.tuple[NFCT_DIR_ORIGINAL].l3protonum = 
-		nfct.tuple[NFCT_DIR_REPLY].l3protonum = nfh->nfgen_family;
+	nfct.tuple[ORIG].l3protonum =
+		nfct.tuple[REPL].l3protonum = nfh->nfgen_family;
 
 	if (nfct_netlink_to_conntrack(nlh, &nfct, &flags) < 0)
 		return -1;
@@ -910,7 +916,7 @@ do_nfct_msg(struct nlmsghdr *nlh, void *
 
 	switch (type) { 
 	case NFCT_MSG_NEW:
-		if ((ct = ct_alloc(&nfct.tuple[NFCT_DIR_ORIGINAL])) == NULL)
+		if ((ct = ct_alloc(&nfct.tuple[ORIG])) == NULL)
 			return -1;
 
 		if (cache_add(priv->tcache, ct) < 0)
@@ -918,8 +924,7 @@ do_nfct_msg(struct nlmsghdr *nlh, void *
 		break;
 
 	case NFCT_MSG_UPDATE:
-		ct = tcache_find(pi, &nfct.tuple[NFCT_DIR_ORIGINAL]);
-		if (ct == NULL) {
+		if ((ct = tcache_find(pi, &nfct.tuple[ORIG])) == NULL) {
 			/* do not add CT to cache, as there would be no start
 			   information */
 			break;
@@ -939,15 +944,14 @@ do_nfct_msg(struct nlmsghdr *nlh, void *
 
 		/* handle TCP connections differently in order not to bloat CT
 		   hash with many TIME_WAIT connections */
-		if (nfct.tuple[NFCT_DIR_ORIGINAL].protonum == IPPROTO_TCP) {
+		if (nfct.tuple[ORIG].protonum == IPPROTO_TCP) {
 			if (nfct.protoinfo.tcp.state == TCP_CONNTRACK_TIME_WAIT)
 				return propagate_ct(pi, &nfct, ct, flags);
 		}
 		break;
 		
 	case NFCT_MSG_DESTROY:
-		ct = tcache_find(pi, &nfct.tuple[NFCT_DIR_ORIGINAL]);
-		if (ct != NULL)
+		if ((ct = tcache_find(pi, &nfct.tuple[ORIG])) != NULL)
 			return propagate_ct(pi, &nfct, ct, flags);
 		break;
 		

-- 

[ULOGD RFC 28/30] NFCT: fix start/stop handling

[heitzenberger]

[-- Attachment #1: ulogd-NFCT-start-stop-cleanup.diff --]
[-- Type: text/plain, Size: 1381 bytes --]

Free sequence cache too.

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/input/flow/ulogd_inpflow_NFCT.c
===================================================================
--- ulogd-netfilter.orig/input/flow/ulogd_inpflow_NFCT.c
+++ ulogd-netfilter/input/flow/ulogd_inpflow_NFCT.c
@@ -526,6 +526,9 @@ cache_free(struct cache *c)
 {
 	int i;
 
+	if (c == NULL)
+		return;
+
 	for (i = 0; i < c->c_num_heads; i++) {
 		struct llist_head *ptr, *ptr2;
 
@@ -1079,7 +1082,7 @@ nfct_start(struct ulogd_pluginstance *up
 	ulogd_log(ULOGD_DEBUG, "%s: ctnetlink connection opened\n", upi->id);
 
     if (set_sockbuf_len(nfct_fd(priv->cth), RCVBUF_LEN, SNDBUF_LEN) < 0)
-	        goto err_free;
+	        goto err_nfct_close;
 
 	priv->nfct_fd.fd = nfct_fd(priv->cth);
 	priv->nfct_fd.cb = &read_cb_nfct;
@@ -1110,6 +1113,8 @@ nfct_start(struct ulogd_pluginstance *up
  err_free:
 	cache_free(priv->tcache);
 	priv->tcache = NULL;
+	cache_free(priv->scache);
+	priv->tcache = NULL;
 
 	return -1;
 }
@@ -1138,10 +1143,10 @@ nfct_stop(struct ulogd_pluginstance *pi)
 
 	ulogd_log(ULOGD_DEBUG, "%s: ctnetlink connection closed\n", pi->id);
 
-	if (priv->tcache != NULL) {
-		cache_free(priv->tcache);
-		priv->tcache = NULL;
-	}
+	cache_free(priv->tcache);
+	priv->tcache = NULL;
+	cache_free(priv->scache);
+	priv->scache = NULL;
 
 	return 0;
 }

-- 

[ULOGD RFC 29/30] NFCT: adapt to new libnetfilter_conntrack

[heitzenberger]

[-- Attachment #1: ulogd-NFCT-new-lib.diff --]
[-- Type: text/plain, Size: 8166 bytes --]

Short-term solution until using libnl for that.

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/input/flow/ulogd_inpflow_NFCT.c
===================================================================
--- ulogd-netfilter.orig/input/flow/ulogd_inpflow_NFCT.c
+++ ulogd-netfilter/input/flow/ulogd_inpflow_NFCT.c
@@ -84,6 +84,7 @@ struct cache {
 struct nfct_pluginstance {
 	struct nfct_handle *cth;
 	struct ulogd_fd nfct_fd;
+	struct nf_conntrack *nfct_opaque;
 	struct cache *tcache;       /* tuple cache */
 	struct cache *scache;       /* sequence cache */
 	struct ulogd_timer timer;
@@ -795,8 +796,8 @@ scache_cleanup(struct ulogd_pluginstance
 
 static int
 propagate_ct_flow(struct ulogd_pluginstance *upi, 
-				  struct nfct_conntrack *nfct, unsigned int flags,
-				  struct conntrack *ct)
+				  const struct nfct_conntrack *nfct,
+				  const struct conntrack *ct)
 {
 	struct ulogd_key *ret = upi->output.keys;
 
@@ -829,27 +830,21 @@ propagate_ct_flow(struct ulogd_pluginsta
 		break;
 	}
 
-	if (flags & NFCT_COUNTERS_ORIG) {
-		ret[O_RAW_IN_PKTLEN].u.value.ui32 = nfct->counters[ORIG].bytes;
-		ret[O_RAW_IN_PKTLEN].flags |= ULOGD_RETF_VALID;
-		ret[O_RAW_IN_PKTCOUNT].u.value.ui32 = nfct->counters[REPL].packets;
-		ret[O_RAW_IN_PKTCOUNT].flags |= ULOGD_RETF_VALID;
-
-		ret[O_RAW_OUT_PKTLEN].u.value.ui32 = nfct->counters[REPL].bytes;
-		ret[O_RAW_OUT_PKTLEN].flags |= ULOGD_RETF_VALID;
-		ret[O_RAW_OUT_PKTCOUNT].u.value.ui32 = nfct->counters[REPL].packets;
-		ret[O_RAW_OUT_PKTCOUNT].flags |= ULOGD_RETF_VALID;
-	}
+	ret[O_RAW_IN_PKTLEN].u.value.ui32 = nfct->counters[ORIG].bytes;
+	ret[O_RAW_IN_PKTLEN].flags |= ULOGD_RETF_VALID;
+	ret[O_RAW_IN_PKTCOUNT].u.value.ui32 = nfct->counters[ORIG].packets;
+	ret[O_RAW_IN_PKTCOUNT].flags |= ULOGD_RETF_VALID;
+
+	ret[O_RAW_OUT_PKTLEN].u.value.ui32 = nfct->counters[REPL].bytes;
+	ret[O_RAW_OUT_PKTLEN].flags |= ULOGD_RETF_VALID;
+	ret[O_RAW_OUT_PKTCOUNT].u.value.ui32 = nfct->counters[REPL].packets;
+	ret[O_RAW_OUT_PKTCOUNT].flags |= ULOGD_RETF_VALID;
 
-	if (flags & NFCT_MARK) {
-		ret[O_CT_MARK].u.value.ui32 = nfct->mark;
-		ret[O_CT_MARK].flags |= ULOGD_RETF_VALID;
-	}
+	ret[O_CT_MARK].u.value.ui32 = nfct->mark;
+	ret[O_CT_MARK].flags |= ULOGD_RETF_VALID;
 
-	if (flags & NFCT_ID) {
-		ret[O_CT_ID].u.value.ui32 = nfct->id;
-		ret[O_CT_ID].flags |= ULOGD_RETF_VALID;
-	}
+	ret[O_CT_ID].u.value.ui32 = nfct->id;
+	ret[O_CT_ID].flags |= ULOGD_RETF_VALID;
 
 	ret[O_FLOW_START_SEC].u.value.ui32 = ct->time[START].tv_sec;
 	ret[O_FLOW_START_SEC].flags |= ULOGD_RETF_VALID;
@@ -873,8 +868,8 @@ propagate_ct_flow(struct ulogd_pluginsta
 }
 
 static int
-propagate_ct(struct ulogd_pluginstance *upi, struct nfct_conntrack *nfct,
-			 struct conntrack *ct, unsigned int flags)
+propagate_ct(struct ulogd_pluginstance *upi,
+			 struct nfct_conntrack *nfct, struct conntrack *ct)
 {
 	struct nfct_pluginstance *priv = (void *)upi->private;
 
@@ -885,7 +880,7 @@ propagate_ct(struct ulogd_pluginstance *
 
 		ct->time[STOP].tv_sec = t_now_local;
 
-		propagate_ct_flow(upi, nfct, flags, ct);
+		propagate_ct_flow(upi, nfct, ct);
 	} while (0);
 
 	cache_del(priv->tcache, ct);
@@ -893,26 +888,97 @@ propagate_ct(struct ulogd_pluginstance *
 	return 0;
 }
 
+/* nfct_to_conntrack() - translate from opaque type to nfct_conntrack */
+static int
+nfct_to_conntrack(const struct ulogd_pluginstance *pi,
+				  const struct nf_conntrack *ct, struct nfct_conntrack *out)
+{
+	bzero(out, sizeof(struct nfct_conntrack));
+
+	assert(nfct_attr_is_set(ct, ATTR_L3PROTO));
+	out->tuple[ORIG].l3protonum = nfct_get_attr_u8(ct, ATTR_L3PROTO);
+
+	assert(nfct_attr_is_set(ct, ATTR_L4PROTO));
+	out->tuple[ORIG].protonum = nfct_get_attr_u8(ct, ATTR_L4PROTO);
+
+	out->tuple[ORIG].src.v4 = nfct_get_attr_u32(ct, ATTR_IPV4_SRC);
+	out->tuple[REPL].src.v4 = nfct_get_attr_u32(ct, ATTR_REPL_IPV4_SRC);
+	out->tuple[ORIG].dst.v4 = nfct_get_attr_u32(ct, ATTR_IPV4_DST);
+	out->tuple[REPL].dst.v4 = nfct_get_attr_u32(ct, ATTR_REPL_IPV4_DST);
+
+	if (out->tuple[ORIG].l3protonum == IPPROTO_ICMP) {
+		out->tuple[ORIG].l4src.icmp.type
+			= nfct_get_attr_u8(ct, ATTR_ICMP_TYPE);
+		out->tuple[ORIG].l4src.icmp.code
+			= nfct_get_attr_u8(ct, ATTR_ICMP_CODE);
+		out->tuple[ORIG].l4src.icmp.id
+			= nfct_get_attr_u16(ct, ATTR_ICMP_ID);
+	}
+
+	if (out->tuple[ORIG].protonum == IPPROTO_TCP
+		|| out->tuple[ORIG].protonum == IPPROTO_UDP) {
+		assert(nfct_attr_is_set(ct, ATTR_ORIG_PORT_SRC));
+		out->tuple[ORIG].l4src.tcp.port
+			= nfct_get_attr_u16(ct, ATTR_ORIG_PORT_SRC);
+
+		assert(nfct_attr_is_set(ct, ATTR_ORIG_PORT_DST));
+		out->tuple[ORIG].l4dst.tcp.port
+			= nfct_get_attr_u16(ct, ATTR_ORIG_PORT_DST);
+
+		assert(nfct_attr_is_set(ct, ATTR_REPL_PORT_SRC));
+		out->tuple[REPL].l4src.tcp.port
+			= nfct_get_attr_u16(ct, ATTR_REPL_PORT_SRC);
+
+		assert(nfct_attr_is_set(ct, ATTR_REPL_PORT_DST));
+		out->tuple[REPL].l4dst.tcp.port
+			= nfct_get_attr_u16(ct, ATTR_REPL_PORT_DST);
+
+		if (nfct_attr_is_set(ct, ATTR_TCP_STATE))
+			out->protoinfo.tcp.state = nfct_get_attr_u8(ct, ATTR_TCP_STATE);
+	}
+
+	if (nfct_attr_is_set(ct, ATTR_STATUS))
+		out->status = nfct_get_attr_u32(ct, ATTR_STATUS);
+	if (nfct_attr_is_set(ct, ATTR_TIMEOUT))
+		out->timeout = nfct_get_attr_u32(ct, ATTR_TIMEOUT);
+	if (nfct_attr_is_set(ct, ATTR_MARK))
+		out->mark = nfct_get_attr_u32(ct, ATTR_MARK);
+	if (nfct_attr_is_set(ct, ATTR_USE))
+		out->use = nfct_get_attr_u32(ct, ATTR_USE);
+
+	/* counter */
+	if (nfct_attr_is_set(ct, ATTR_ORIG_COUNTER_BYTES))
+		out->counters[ORIG].bytes
+			= nfct_get_attr_u32(ct, ATTR_ORIG_COUNTER_BYTES);
+	if (nfct_attr_is_set(ct, ATTR_ORIG_COUNTER_PACKETS))
+		out->counters[ORIG].packets
+			= nfct_get_attr_u32(ct, ATTR_ORIG_COUNTER_PACKETS);
+	if (nfct_attr_is_set(ct, ATTR_REPL_COUNTER_BYTES))
+		out->counters[ORIG].bytes
+			= nfct_get_attr_u32(ct, ATTR_REPL_COUNTER_BYTES);
+	if (nfct_attr_is_set(ct, ATTR_REPL_COUNTER_PACKETS))
+		out->counters[REPL].packets
+			= nfct_get_attr_u32(ct, ATTR_REPL_COUNTER_PACKETS);
+
+	return 0;
+}
 
 static int
 do_nfct_msg(struct nlmsghdr *nlh, void *arg)
 {
 	struct ulogd_pluginstance *pi = arg;
 	struct nfct_pluginstance *priv = (void *)pi->private;
-	struct nfgenmsg *nfh = NLMSG_DATA(nlh);
 	struct nfct_conntrack nfct;
 	struct conntrack *ct;
-	int flags, type = nfct_msg_type(nlh);
+	int type = nfct_msg_type(nlh);
 
 	if (type == NFCT_MSG_UNKNOWN)
 		return 0;
 
-	bzero(&nfct, sizeof(nfct));
-
-	nfct.tuple[ORIG].l3protonum =
-		nfct.tuple[REPL].l3protonum = nfh->nfgen_family;
+	if (nfct_parse_conntrack(NFCT_T_ALL, nlh, priv->nfct_opaque) < 0)
+		return -1;
 
-	if (nfct_netlink_to_conntrack(nlh, &nfct, &flags) < 0)
+	if (nfct_to_conntrack(pi, priv->nfct_opaque, &nfct) < 0)
 		return -1;
 
 	/* TODO handle NFCT_COUNTER_FILLING */
@@ -949,13 +1015,13 @@ do_nfct_msg(struct nlmsghdr *nlh, void *
 		   hash with many TIME_WAIT connections */
 		if (nfct.tuple[ORIG].protonum == IPPROTO_TCP) {
 			if (nfct.protoinfo.tcp.state == TCP_CONNTRACK_TIME_WAIT)
-				return propagate_ct(pi, &nfct, ct, flags);
+				return propagate_ct(pi, &nfct, ct);
 		}
 		break;
 		
 	case NFCT_MSG_DESTROY:
 		if ((ct = tcache_find(pi, &nfct.tuple[ORIG])) != NULL)
-			return propagate_ct(pi, &nfct, ct, flags);
+			return propagate_ct(pi, &nfct, ct);
 		break;
 		
 	default:
@@ -1070,8 +1136,15 @@ nfct_start(struct ulogd_pluginstance *up
 		return 0;
 	}
 
+	if (priv->nfct_opaque == NULL) {
+		if ((priv->nfct_opaque = nfct_new()) == NULL) {
+			ulogd_log(ULOGD_FATAL, "%s: out of memory\n", upi->id);
+			return -1;
+		}
+	}
+
 	if (init_caches(upi) < 0)
-		return -1;
+		goto err_free;
 
 	priv->cth = nfct_open(NFNL_SUBSYS_CTNETLINK, CT_EVENTS);
 	if (priv->cth == NULL) {
@@ -1111,6 +1184,8 @@ nfct_start(struct ulogd_pluginstance *up
 	nfct_close(priv->cth);
 	priv->cth = NULL;
  err_free:
+	free(priv->nfct_opaque);
+	priv->nfct_opaque = NULL;
 	cache_free(priv->tcache);
 	priv->tcache = NULL;
 	cache_free(priv->scache);
@@ -1148,6 +1223,9 @@ nfct_stop(struct ulogd_pluginstance *pi)
 	cache_free(priv->scache);
 	priv->scache = NULL;
 
+	free(priv->nfct_opaque);
+	priv->nfct_opaque = NULL;
+
 	return 0;
 }
 

-- 

Re: [ULOGD RFC 29/30] NFCT: adapt to new libnetfilter_conntrack

[Pablo Neira Ayuso]

heitzenberger@astaro.com wrote:
> +static int
> +nfct_to_conntrack(const struct ulogd_pluginstance *pi,
> +				  const struct nf_conntrack *ct, struct nfct_conntrack *out)

I frankly prefer if you don't this "translator" and use the new API
directly.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: [ULOGD RFC 29/30] NFCT: adapt to new libnetfilter_conntrack

[Holger Eitzenberger]

Pablo Neira Ayuso <pablo@netfilter.org> writes:

> I frankly prefer if you don't this "translator" and use the new API
> directly.

Generally speaking I don't like this further indirection introduced by
this patch too.  As I said in my introductory email I consider this
solution only a short-term solution.  Later patches will change that by
using libnl instead or at least get rid of that further indirection.

Thanks.

 /holger

[ULOGD RFC 30/30] SQLITE3: move timer initialization to start handler

[heitzenberger]

[-- Attachment #1: ulogd-SQLITE3-move-timer-initialization.diff --]
[-- Type: text/plain, Size: 1116 bytes --]

Signed-off-by: Holger Eitzenberger <holger@eitzenberger.org>

Index: ulogd-netfilter/output/sqlite3/ulogd_output_SQLITE3.c
===================================================================
--- ulogd-netfilter.orig/output/sqlite3/ulogd_output_SQLITE3.c
+++ ulogd-netfilter/output/sqlite3/ulogd_output_SQLITE3.c
@@ -719,14 +719,6 @@ sqlite3_configure(struct ulogd_pluginsta
 	pr_debug("%s: db='%s' table='%s' timer=%d max-backlog=%d\n", pi->id,
 			 db_ce(pi), table_ce(pi), timer_ce(pi), max_backlog_ce(pi));
 
-	/* init timer */
-	priv->timer.cb = sqlite_timer_cb;
-	priv->timer.ival = timer_ce(pi);
-	priv->timer.flags = TIMER_F_PERIODIC;
-	priv->timer.data = pi;
-
-	ulogd_register_timer(&priv->timer);
-
 	return 0;
 }
 
@@ -749,6 +741,15 @@ sqlite3_start(struct ulogd_pluginstance 
 	if (db_start(pi) < 0)
 		return -1;
 
+	/* init timer */
+	priv->timer.cb = sqlite_timer_cb;
+	priv->timer.ival = timer_ce(pi);
+	priv->timer.flags = TIMER_F_PERIODIC;
+	priv->timer.data = pi;
+
+	if (ulogd_register_timer(&priv->timer) < 0)
+		return -1;
+
 	ulogd_log(ULOGD_INFO, "%s: started\n", pi->id);
 
 	return 0;

-- 

Re: [ULOGD RFC 00/30] [ULOGD RFC] ulogd V2 improvements, round #1

[Patrick McHardy]

heitzenberger@astaro.com wrote:
> Hi,
>
> this is a patchset for you to comment on, which tries to turn ulogd V2
> into something usefull.
>
> Some of the patches are are quite intrusive, others might be to be
> discussed.  Please take a look at all of them and give some comments
> back, I'll appreciate it.
>
> In the end they turn ulogd v2 into some powerfull network accounting
> solution being capable to handle 400000 or more concurrent flows.
> That's where your post-processing starts to get important of course
> :).
>
> Some points which might be of interest to you:
>
>  * ulogd-ifi.diff: replacement for the IFINDEX plugin, as that
>  functionality IMO is better homed in the ulogd core and not in a
>  plugin (remove IFINDEX?).
>
>  * some of the patches are quite large and contain many individual 
>  changes in the end.  Should I split them?
>
>  * ulogd-NFCT-new-lib.diff: makes ulogd work with a recent
>  libnetfilter-conntrack.  This one is fairly untested, I'll test that
>  before sending the final patchset.  Note also that I'd like to use
>  libnl with ulogd rather soon, so this is only a short-term solution.
>
>  * the libnetfilter-conntrack patch I've send before is required
>  to build ulogd.
>
> Some of those patches still contain trailing whitespace.  Please
> ignore that as of now, I'll send additional patches removing
> whitespace later.
>   

Thanks for sending these patches Holger, I'll try to review all of them
tommorrow. Just for my understanding since the ones I already looked at
look like I could merge them immediately, do you want me to already
commit them or is this really just meant to gather comments?

Re: [ULOGD RFC 00/30] [ULOGD RFC] ulogd V2 improvements, round #1

[Holger Eitzenberger]

Patrick McHardy wrote:

> Thanks for sending these patches Holger, I'll try to review all of them
> tommorrow. Just for my understanding since the ones I already looked at
> look like I could merge them immediately, do you want me to already
> commit them or is this really just meant to gather comments?

Hi Patrick,

most of the code in those patches is quite an improvement, so there is 
IMO not much concern integrating those changes, as they change code 
which was sometimes not even working.  I was more thinking about 
presentation of those patches, which would eventually make it easier for 
others to understand all the motivation behind those changes.

My initial round of patches was more than 60 or so, what you see here is 
just a distillation of all those.  Some of those changes were quite 
small, which maybe let me think that this distillation would be helpfull 
in the end to just reduce the number of patches.

Of course, if you are fine even with the larger patches feel free to 
commit them.  Just remember that most of these patches are strongly 
in-order, you have to commit them in the same order.

Thanks.  /holger