[PATCH REPOST 0/2][RFC] Network Event Notifier Mechanism

[PATCH REPOST 0/2][RFC] Network Event Notifier Mechanism

[Steve Wise]

This patch implements a mechanism that allows interested clients to
register for notification of certain network events. The intended use
is to allow RDMA devices (linux/drivers/infiniband) to be notified of
neighbour updates, ICMP redirects, path MTU changes, and route changes.

The reason these devices need update events is because they typically
cache this information in hardware and need to be notified when this
information has been updated.

The key events of interest are:

- neighbour mac address change 
- routing redirect (the next hop neighbour changes for a dst_entry)
- path mtu change (the patch mtu for a dst_entry changes).
- route add/deletes

This approach is one of many possibilities and may be preferred because it
uses an existing notification mechanism that has precedent in the stack.
An alternative would be to add a netdev method to notify affect devices
of these events.

This code does not yet implement path MTU change because the number of
places in which this value is updated is large and if this mechanism
seems reasonable, it would be probably be best to funnel these updates
through a single function.

We would like to get this or similar functionality included in 2.6.19
and request comments.

This patchset consists of 2 patches:

1) New files implementing the Network Event Notifier
2) Core network changes to generate network event notifications

Signed-off-by: Tom Tucker <tom@opengridcomputing.com>
Signed-off-by: Steve Wise <swise@opengridcomputing.com>

[PATCH REPOST 1/2] Network Event Notifier Mechanism.

[Steve Wise]

This patch uses notifier blocks to implement a network event
notifier mechanism.

Clients register their callback function by calling
register_netevent_notifier() like this:

static struct notifier_block nb = {
        .notifier_call = my_callback_func
};

...

register_netevent_notifier(&nb);
---

 include/net/netevent.h |   41 +++++++++++++++++++++++++++++
 net/core/netevent.c    |   67 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 108 insertions(+), 0 deletions(-)

diff --git a/include/net/netevent.h b/include/net/netevent.h
new file mode 100644
index 0000000..9ceab27
--- /dev/null
+++ b/include/net/netevent.h
@@ -0,0 +1,41 @@
+#ifndef _NET_EVENT_H
+#define _NET_EVENT_H
+
+/*
+ *	Generic netevent notifiers
+ *
+ *	Authors:
+ *      Tom Tucker              <tom@opengridcomputing.com>
+ *
+ * 	Changes:
+ */
+
+#ifdef __KERNEL__
+
+#include <net/dst.h>
+
+struct netevent_redirect {
+	struct dst_entry *old;
+	struct dst_entry *new;
+};
+
+struct netevent_route_change {
+        int event;
+        struct fib_info *fib_info;
+};
+
+enum netevent_notif_type {
+	NETEVENT_NEIGH_UPDATE = 1, /* arg is * struct neighbour */
+	NETEVENT_ROUTE_UPDATE,	   /* arg is * netevent_route_change */
+	NETEVENT_PMTU_UPDATE,
+	NETEVENT_REDIRECT,	   /* arg is * struct netevent_redirect */
+};
+
+extern int register_netevent_notifier(struct notifier_block *nb);
+extern int unregister_netevent_notifier(struct notifier_block *nb);
+extern int call_netevent_notifiers(unsigned long val, void *v);
+
+#endif
+#endif
+
+
diff --git a/net/core/netevent.c b/net/core/netevent.c
new file mode 100644
index 0000000..8f3e0a6
--- /dev/null
+++ b/net/core/netevent.c
@@ -0,0 +1,67 @@
+/*
+ *	Network event notifiers
+ *
+ *	Authors:
+ *      Tom Tucker             <tom@opengridcomputing.com>
+ *
+ *	This program is free software; you can redistribute it and/or
+ *      modify it under the terms of the GNU General Public License
+ *      as published by the Free Software Foundation; either version
+ *      2 of the License, or (at your option) any later version.
+ *
+ *	Fixes:
+ */
+
+#include <linux/rtnetlink.h>
+#include <linux/notifier.h>
+
+static ATOMIC_NOTIFIER_HEAD(netevent_notif_chain);
+
+/**
+ *	register_netevent_notifier - register a netevent notifier block
+ *	@nb: notifier
+ *
+ *	Register a notifier to be called when a netevent occurs.
+ *	The notifier passed is linked into the kernel structures and must
+ *	not be reused until it has been unregistered. A negative errno code
+ *	is returned on a failure.
+ */
+int register_netevent_notifier(struct notifier_block *nb)
+{
+	int err;
+
+	err = atomic_notifier_chain_register(&netevent_notif_chain, nb);
+	return err;
+}
+
+/**
+ *	netevent_unregister_notifier - unregister a netevent notifier block
+ *	@nb: notifier
+ *
+ *	Unregister a notifier previously registered by
+ *	register_neigh_notifier(). The notifier is unlinked into the
+ *	kernel structures and may then be reused. A negative errno code
+ *	is returned on a failure.
+ */
+
+int unregister_netevent_notifier(struct notifier_block *nb)
+{
+	return atomic_notifier_chain_unregister(&netevent_notif_chain, nb);
+}
+
+/**
+ *	call_netevent_notifiers - call all netevent notifier blocks
+ *      @val: value passed unmodified to notifier function
+ *      @v:   pointer passed unmodified to notifier function
+ *
+ *	Call all neighbour notifier blocks.  Parameters and return value
+ *	are as for notifier_call_chain().
+ */
+
+int call_netevent_notifiers(unsigned long val, void *v)
+{
+	return atomic_notifier_call_chain(&netevent_notif_chain, val, v);
+}
+
+EXPORT_SYMBOL_GPL(register_netevent_notifier);
+EXPORT_SYMBOL_GPL(unregister_netevent_notifier);

Re: [PATCH REPOST 1/2] Network Event Notifier Mechanism.

[David Miller]

From: Steve Wise <swise@opengridcomputing.com>
Date: Fri, 23 Jun 2006 15:19:28 -0500

> +struct netevent_route_change {
> +        int event;
> +        struct fib_info *fib_info;
> +};

It's not generic if you're putting ipv4 FIB route objects
in the datastructure.

This is so much unnecessary syntactic sugar for what you're
trying to accomplish.

Just pass a "void *", and have the type of the object defined by which
notifier it is.  This is exactly how all notifiers work today.

Then you don't need any new infrastructure at all.  The existing
notifier bits handle this kind of scheme already, no need to
invent new stuff.

Re: [PATCH REPOST 1/2] Network Event Notifier Mechanism.

[Steve Wise]

On Fri, 2006-06-23 at 13:26 -0700, David Miller wrote:
> From: Steve Wise <swise@opengridcomputing.com>
> Date: Fri, 23 Jun 2006 15:19:28 -0500
> 
> > +struct netevent_route_change {
> > +        int event;
> > +        struct fib_info *fib_info;
> > +};
> 
> It's not generic if you're putting ipv4 FIB route objects
> in the datastructure.
> 
> This is so much unnecessary syntactic sugar for what you're
> trying to accomplish.
> 
> Just pass a "void *", and have the type of the object defined by which
> notifier it is.  This is exactly how all notifiers work today.
> 
> Then you don't need any new infrastructure at all.  The existing
> notifier bits handle this kind of scheme already, no need to
> invent new stuff.

Ok.  

But what about the redirect event?  It really needs to pass two bits of
info:  The old dst_entry ptr and the new dst_entry ptr.  This is needed
so the rdma driver can figure out which connections now need to use the
new dst_entry...

Re: [PATCH REPOST 1/2] Network Event Notifier Mechanism.

[David Miller]

From: Steve Wise <swise@opengridcomputing.com>
Date: Fri, 23 Jun 2006 15:34:55 -0500

> But what about the redirect event?  It really needs to pass two bits of
> info:  The old dst_entry ptr and the new dst_entry ptr.  This is needed
> so the rdma driver can figure out which connections now need to use the
> new dst_entry...

struct foo_event_data {
	struct whatever a;
	struct thistoo b;
};

	struct foo_event_data x;

	x->a = xxx;
	x->b = yyy;

	event->data = (void *) &x;
	send_event(&event);

Why do you need typed data to do this?

Re: [PATCH REPOST 1/2] Network Event Notifier Mechanism.

[Steve Wise]

----- Original Message ----- 
From: "David Miller" <davem@davemloft.net>
To: <swise@opengridcomputing.com>
Cc: <netdev@vger.kernel.org>
Sent: Friday, June 23, 2006 3:56 PM
Subject: Re: [PATCH REPOST 1/2] Network Event Notifier Mechanism.


| From: Steve Wise <swise@opengridcomputing.com>
| Date: Fri, 23 Jun 2006 15:34:55 -0500
|
| > But what about the redirect event?  It really needs to pass two bits 
of
| > info:  The old dst_entry ptr and the new dst_entry ptr.  This is 
needed
| > so the rdma driver can figure out which connections now need to use 
the
| > new dst_entry...
|
| struct foo_event_data {
| struct whatever a;
| struct thistoo b;
| };
|
| struct foo_event_data x;
|
| x->a = xxx;
| x->b = yyy;
|
| event->data = (void *) &x;
| send_event(&event);
|
| Why do you need typed data to do this?
|

You don't.  This is fine with me...

Re: [PATCH REPOST 1/2] Network Event Notifier Mechanism.

[Steve Wise]

On Fri, 2006-06-23 at 13:26 -0700, David Miller wrote:
> From: Steve Wise <swise@opengridcomputing.com>
> Date: Fri, 23 Jun 2006 15:19:28 -0500
> 
> > +struct netevent_route_change {
> > +        int event;
> > +        struct fib_info *fib_info;
> > +};
> 
> It's not generic if you're putting ipv4 FIB route objects
> in the datastructure.
> 

True. 

I guess what I think we should do is pass the fib_info * when its a IPv4
route add/del, and a rt6_info * when its a IPv6 add/del.  This avoids
having to create some new family independent struct.  What I'll have to
do, however, is have specific notifier event enums for each: 

NETEVENT_IPV4_ROUTE_ADD
NETEVENT_IPV4_ROUTE_DEL
NETEVENT_IPV6_ROUTE_ADD
NETEVENT_IPV6_ROUTE_DEL

This keeps it simple IMO...

Does that sound reasonable to you?

Steve.

Re: [PATCH REPOST 1/2] Network Event Notifier Mechanism.

[David Miller]

From: Steve Wise <swise@opengridcomputing.com>
Date: Mon, 26 Jun 2006 10:26:11 -0500

> I guess what I think we should do is pass the fib_info * when its a IPv4
> route add/del, and a rt6_info * when its a IPv6 add/del.  This avoids
> having to create some new family independent struct.  What I'll have to
> do, however, is have specific notifier event enums for each: 
> 
> NETEVENT_IPV4_ROUTE_ADD
> NETEVENT_IPV4_ROUTE_DEL
> NETEVENT_IPV6_ROUTE_ADD
> NETEVENT_IPV6_ROUTE_DEL
> 
> This keeps it simple IMO...
> 
> Does that sound reasonable to you?

You can avoid creating an event per address family by defining your
data as:

	struct netevent_route_info {
		u16 family;
		void *data;
	};

and passing that through the notifier.

Then you just need NETEVENT_ROUTE_{ADD,DEL}

Just an idea...

Re: [PATCH REPOST 1/2] Network Event Notifier Mechanism.

[Steve Wise]

On Mon, 2006-06-26 at 10:43 -0700, David Miller wrote:
> From: Steve Wise <swise@opengridcomputing.com>
> Date: Mon, 26 Jun 2006 10:26:11 -0500
> 
> > I guess what I think we should do is pass the fib_info * when its a IPv4
> > route add/del, and a rt6_info * when its a IPv6 add/del.  This avoids
> > having to create some new family independent struct.  What I'll have to
> > do, however, is have specific notifier event enums for each: 
> > 
> > NETEVENT_IPV4_ROUTE_ADD
> > NETEVENT_IPV4_ROUTE_DEL
> > NETEVENT_IPV6_ROUTE_ADD
> > NETEVENT_IPV6_ROUTE_DEL
> > 
> > This keeps it simple IMO...
> > 
> > Does that sound reasonable to you?
> 
> You can avoid creating an event per address family by defining your
> data as:
> 
> 	struct netevent_route_info {
> 		u16 family;
> 		void *data;
> 	};
> 
> and passing that through the notifier.
> 
> Then you just need NETEVENT_ROUTE_{ADD,DEL}
> 
> Just an idea...

Seems reasonable.   

Thanks,

Steve.

[PATCH REPOST 2/2] Core network changes to support network event notification.

[Steve Wise]

This patch adds event calls for neighbour change, route update, and
routing redirect events.

TODO: PMTU change events.
---

 net/core/Makefile        |    2 +-
 net/core/neighbour.c     |    8 ++++++++
 net/ipv4/fib_semantics.c |    7 +++++++
 net/ipv4/route.c         |    6 ++++++
 4 files changed, 22 insertions(+), 1 deletions(-)

diff --git a/net/core/Makefile b/net/core/Makefile
index e9bd246..2645ba4 100644
--- a/net/core/Makefile
+++ b/net/core/Makefile
@@ -7,7 +7,7 @@ obj-y := sock.o request_sock.o skbuff.o 
 
 obj-$(CONFIG_SYSCTL) += sysctl_net_core.o
 
-obj-y		     += dev.o ethtool.o dev_mcast.o dst.o \
+obj-y		     += dev.o ethtool.o dev_mcast.o dst.o netevent.o \
 			neighbour.o rtnetlink.o utils.o link_watch.o filter.o
 
 obj-$(CONFIG_XFRM) += flow.o
diff --git a/net/core/neighbour.c b/net/core/neighbour.c
index 50a8c73..c637897 100644
--- a/net/core/neighbour.c
+++ b/net/core/neighbour.c
@@ -30,9 +30,11 @@ #include <linux/times.h>
 #include <net/neighbour.h>
 #include <net/dst.h>
 #include <net/sock.h>
+#include <net/netevent.h>
 #include <linux/rtnetlink.h>
 #include <linux/random.h>
 #include <linux/string.h>
+#include <linux/notifier.h>
 
 #define NEIGH_DEBUG 1
 
@@ -755,6 +757,7 @@ #endif
 			neigh->nud_state = NUD_STALE;
 			neigh->updated = jiffies;
 			neigh_suspect(neigh);
+			call_netevent_notifiers(NETEVENT_NEIGH_UPDATE, neigh);
 		}
 	} else if (state & NUD_DELAY) {
 		if (time_before_eq(now, 
@@ -763,6 +766,7 @@ #endif
 			neigh->nud_state = NUD_REACHABLE;
 			neigh->updated = jiffies;
 			neigh_connect(neigh);
+			call_netevent_notifiers(NETEVENT_NEIGH_UPDATE, neigh);
 			next = neigh->confirmed + neigh->parms->reachable_time;
 		} else {
 			NEIGH_PRINTK2("neigh %p is probed.\n", neigh);
@@ -783,6 +787,7 @@ #endif
 		neigh->nud_state = NUD_FAILED;
 		neigh->updated = jiffies;
 		notify = 1;
+		call_netevent_notifiers(NETEVENT_NEIGH_UPDATE, neigh);
 		NEIGH_CACHE_STAT_INC(neigh->tbl, res_failed);
 		NEIGH_PRINTK2("neigh %p is failed.\n", neigh);
 
@@ -1056,6 +1061,9 @@ out:
 			(neigh->flags | NTF_ROUTER) :
 			(neigh->flags & ~NTF_ROUTER);
 	}
+
+	call_netevent_notifiers(NETEVENT_NEIGH_UPDATE, neigh);
+
 	write_unlock_bh(&neigh->lock);
 #ifdef CONFIG_ARPD
 	if (notify && neigh->parms->app_probes)
diff --git a/net/ipv4/fib_semantics.c b/net/ipv4/fib_semantics.c
index 0f4145b..67a30af 100644
--- a/net/ipv4/fib_semantics.c
+++ b/net/ipv4/fib_semantics.c
@@ -45,6 +45,7 @@ #include <net/tcp.h>
 #include <net/sock.h>
 #include <net/ip_fib.h>
 #include <net/ip_mp_alg.h>
+#include <net/netevent.h>
 
 #include "fib_lookup.h"
 
@@ -278,9 +279,15 @@ void rtmsg_fib(int event, u32 key, struc
 	       struct nlmsghdr *n, struct netlink_skb_parms *req)
 {
 	struct sk_buff *skb;
+	struct netevent_route_change rev;
+
 	u32 pid = req ? req->pid : n->nlmsg_pid;
 	int size = NLMSG_SPACE(sizeof(struct rtmsg)+256);
 
+	rev.event = event;
+	rev.fib_info = fa->fa_info;
+	call_netevent_notifiers(NETEVENT_ROUTE_UPDATE, &rev);
+
 	skb = alloc_skb(size, GFP_KERNEL);
 	if (!skb)
 		return;
diff --git a/net/ipv4/route.c b/net/ipv4/route.c
index cc9423d..e9ba831 100644
--- a/net/ipv4/route.c
+++ b/net/ipv4/route.c
@@ -105,6 +105,7 @@ #include <net/tcp.h>
 #include <net/icmp.h>
 #include <net/xfrm.h>
 #include <net/ip_mp_alg.h>
+#include <net/netevent.h>
 #ifdef CONFIG_SYSCTL
 #include <linux/sysctl.h>
 #endif
@@ -1120,6 +1121,7 @@ void ip_rt_redirect(u32 old_gw, u32 dadd
 	struct rtable *rth, **rthp;
 	u32  skeys[2] = { saddr, 0 };
 	int  ikeys[2] = { dev->ifindex, 0 };
+	struct netevent_redirect netevent;
 
 	if (!in_dev)
 		return;
@@ -1211,6 +1213,10 @@ void ip_rt_redirect(u32 old_gw, u32 dadd
 					rt_drop(rt);
 					goto do_next;
 				}
+				
+				netevent.old = &rth->u.dst;
+				netevent.new = &rt->u.dst;
+				call_netevent_notifiers(NETEVENT_REDIRECT, &netevent);
 
 				rt_del(hash, rth);
 				if (!rt_intern_hash(hash, rt, &rt))

Re: [PATCH REPOST 0/2][RFC] Network Event Notifier Mechanism

[Herbert Xu]

Steve Wise <swise@opengridcomputing.com> wrote:
> 
> This patch implements a mechanism that allows interested clients to
> register for notification of certain network events. The intended use
> is to allow RDMA devices (linux/drivers/infiniband) to be notified of
> neighbour updates, ICMP redirects, path MTU changes, and route changes.
> 
> The reason these devices need update events is because they typically
> cache this information in hardware and need to be notified when this
> information has been updated.
> 
> The key events of interest are:
> 
> - neighbour mac address change 
> - routing redirect (the next hop neighbour changes for a dst_entry)
> - path mtu change (the patch mtu for a dst_entry changes).
> - route add/deletes

I'd like to know more about what the RDMA device is going to do with this
information.  I thought RDMA was for receiving packets? Most of the info
here pertains to transmission.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH REPOST 0/2][RFC] Network Event Notifier Mechanism

[Steve Wise]

> I'd like to know more about what the RDMA device is going to do with this
> information.  I thought RDMA was for receiving packets? Most of the info
> here pertains to transmission.
> 

RDMA Ethernet devices adhere to a set of protocols defined by the IETF.
See the RDDP WG (http://www.ietf.org/html.charters/rddp-charter.html)
for the Internet Drafts that define the protocols.

Steve.

Re: [PATCH REPOST 0/2][RFC] Network Event Notifier Mechanism

[Herbert Xu]

On Tue, Jun 27, 2006 at 09:31:57AM -0500, Steve Wise wrote:
> 
> > I'd like to know more about what the RDMA device is going to do with this
> > information.  I thought RDMA was for receiving packets? Most of the info
> > here pertains to transmission.
> 
> RDMA Ethernet devices adhere to a set of protocols defined by the IETF.
> See the RDDP WG (http://www.ietf.org/html.charters/rddp-charter.html)
> for the Internet Drafts that define the protocols.

Would it be possible for you to give us a quick summary of the relevant
points?

Thanks,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt