[PATCH Round 2 0/2][RFC] Network Event Notifier Mechanism

[PATCH Round 2 0/2][RFC] Network Event Notifier Mechanism

[Steve Wise]

Round 2 Changes:

- cleaned up event structures per review feedback.
- began integration with netlink (see neighbour changes in patch 2).
- added IPv6 support.

STILL TODO:

- PMTU events/netlink
- Redirect netlink (need to define a new netlink message for this).

Questions:

- this patch is enabling some of the functionality that was under
  CONFIG_ARPD.  Should I remove the CONFIG_ARPD stuff, leave it as-is
  or what?

- what about support for other afs?  Should I hook AF_DECnet?  IPX? Or is
  AF_INET and AF_INET6 sufficient?

- I added a new RTM type in rtnetlink.h for neighbour change netlink
  messages.  I suspect I did it wrong...


------

This patch implements a mechanism that allows interested clients to
register for notification of certain network events. The intended use
is to allow RDMA devices (linux/drivers/infiniband) to be notified of
neighbour updates, ICMP redirects, path MTU changes, and route changes.

The reason these devices need update events is because they typically
cache this information in hardware and need to be notified when this
information has been updated.

The key events of interest are:

- neighbour mac address change 
- routing redirect (the next hop neighbour changes for a dst_entry)
- path mtu change (the patch mtu for a dst_entry changes).
- route add/deletes

NOTE: Network events are also passed up to user space via netlink.

We would like to get this or similar functionality included in 2.6.19
and request comments.

This patchset consists of 2 patches:

1) New files implementing the Network Event Notifier
2) Core network changes to generate network event notifications

Signed-off-by: Tom Tucker <tom@opengridcomputing.com>
Signed-off-by: Steve Wise <swise@opengridcomputing.com>

[PATCH Round 2 1/2] Network Event Notifier Mechanism.

[Steve Wise]

This patch uses notifier blocks to implement a network event
notifier mechanism.

Clients register their callback function by calling
register_netevent_notifier() like this:

static struct notifier_block nb = {
        .notifier_call = my_callback_func
};

...

register_netevent_notifier(&nb);
---

 include/net/netevent.h |   49 +++++++++++++++++++++++++++++++++++
 net/core/netevent.c    |   67 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 116 insertions(+), 0 deletions(-)

diff --git a/include/net/netevent.h b/include/net/netevent.h
new file mode 100644
index 0000000..22214c8
--- /dev/null
+++ b/include/net/netevent.h
@@ -0,0 +1,49 @@
+#ifndef _NET_EVENT_H
+#define _NET_EVENT_H
+
+/*
+ *	Generic netevent notifiers
+ *
+ *	Authors:
+ *      Tom Tucker              <tom@opengridcomputing.com>
+ *
+ * 	Changes:
+ */
+
+#ifdef __KERNEL__
+
+#include <net/dst.h>
+
+/* 
+ * Generic route info structure.
+ *
+ * Family	  Data ptr type
+ * --------------------------------
+ * AF_INET 	- struct fib_info *
+ * AF_INET6	- struct rt6_info *
+ * AF_DECnet	- struct dn_route *
+ */
+struct netevent_route_info {
+	u16 family;
+	void *data;	
+};
+
+struct netevent_redirect {
+	struct dst_entry *old;
+	struct dst_entry *new;
+};
+
+enum netevent_notif_type {
+	NETEVENT_NEIGH_UPDATE = 1, /* arg is struct neighbour ptr */
+	NETEVENT_ROUTE_ADD,   	   /* arg is struct netevent_route_info ptr */
+	NETEVENT_ROUTE_DEL,   	   /* arg is struct netevent_route_info ptr */
+	NETEVENT_PMTU_UPDATE,	   /* arg is struct dst_entry ptr */
+	NETEVENT_REDIRECT,	   /* arg is struct netevent_redirect ptr */
+};
+
+extern int register_netevent_notifier(struct notifier_block *nb);
+extern int unregister_netevent_notifier(struct notifier_block *nb);
+extern int call_netevent_notifiers(unsigned long val, void *v);
+
+#endif
+#endif
diff --git a/net/core/netevent.c b/net/core/netevent.c
new file mode 100644
index 0000000..8f3e0a6
--- /dev/null
+++ b/net/core/netevent.c
@@ -0,0 +1,67 @@
+/*
+ *	Network event notifiers
+ *
+ *	Authors:
+ *      Tom Tucker             <tom@opengridcomputing.com>
+ *
+ *	This program is free software; you can redistribute it and/or
+ *      modify it under the terms of the GNU General Public License
+ *      as published by the Free Software Foundation; either version
+ *      2 of the License, or (at your option) any later version.
+ *
+ *	Fixes:
+ */
+
+#include <linux/rtnetlink.h>
+#include <linux/notifier.h>
+
+static ATOMIC_NOTIFIER_HEAD(netevent_notif_chain);
+
+/**
+ *	register_netevent_notifier - register a netevent notifier block
+ *	@nb: notifier
+ *
+ *	Register a notifier to be called when a netevent occurs.
+ *	The notifier passed is linked into the kernel structures and must
+ *	not be reused until it has been unregistered. A negative errno code
+ *	is returned on a failure.
+ */
+int register_netevent_notifier(struct notifier_block *nb)
+{
+	int err;
+
+	err = atomic_notifier_chain_register(&netevent_notif_chain, nb);
+	return err;
+}
+
+/**
+ *	netevent_unregister_notifier - unregister a netevent notifier block
+ *	@nb: notifier
+ *
+ *	Unregister a notifier previously registered by
+ *	register_neigh_notifier(). The notifier is unlinked into the
+ *	kernel structures and may then be reused. A negative errno code
+ *	is returned on a failure.
+ */
+
+int unregister_netevent_notifier(struct notifier_block *nb)
+{
+	return atomic_notifier_chain_unregister(&netevent_notif_chain, nb);
+}
+
+/**
+ *	call_netevent_notifiers - call all netevent notifier blocks
+ *      @val: value passed unmodified to notifier function
+ *      @v:   pointer passed unmodified to notifier function
+ *
+ *	Call all neighbour notifier blocks.  Parameters and return value
+ *	are as for notifier_call_chain().
+ */
+
+int call_netevent_notifiers(unsigned long val, void *v)
+{
+	return atomic_notifier_call_chain(&netevent_notif_chain, val, v);
+}
+
+EXPORT_SYMBOL_GPL(register_netevent_notifier);
+EXPORT_SYMBOL_GPL(unregister_netevent_notifier);

[PATCH Round 2 2/2] Core network changes to support network event notification.

[Steve Wise]

This patch adds netevent and netlink calls for neighbour change, route
add/del, and routing redirect events.

TBD: 
	PMTU change netevent and netlink calls.
	netlink call for redirect events.
---

 include/linux/rtnetlink.h |    2 ++
 net/core/Makefile         |    2 +-
 net/core/neighbour.c      |   37 ++++++++++++++++++++++++++++++-------
 net/ipv4/fib_semantics.c  |    9 +++++++++
 net/ipv4/route.c          |   16 ++++++++++++++++
 net/ipv6/route.c          |   22 ++++++++++++++++++++++
 6 files changed, 80 insertions(+), 8 deletions(-)

diff --git a/include/linux/rtnetlink.h b/include/linux/rtnetlink.h
index df0cdd4..14f5622 100644
--- a/include/linux/rtnetlink.h
+++ b/include/linux/rtnetlink.h
@@ -42,6 +42,8 @@ #define RTM_NEWNEIGH	RTM_NEWNEIGH
 #define RTM_DELNEIGH	RTM_DELNEIGH
 	RTM_GETNEIGH,
 #define RTM_GETNEIGH	RTM_GETNEIGH
+	RTM_CHGNEIGH,
+#define RTM_CHGNEIGH	RTM_CHGNEIGH
 
 	RTM_NEWRULE	= 32,
 #define RTM_NEWRULE	RTM_NEWRULE
diff --git a/net/core/Makefile b/net/core/Makefile
index e9bd246..2645ba4 100644
--- a/net/core/Makefile
+++ b/net/core/Makefile
@@ -7,7 +7,7 @@ obj-y := sock.o request_sock.o skbuff.o 
 
 obj-$(CONFIG_SYSCTL) += sysctl_net_core.o
 
-obj-y		     += dev.o ethtool.o dev_mcast.o dst.o \
+obj-y		     += dev.o ethtool.o dev_mcast.o dst.o netevent.o \
 			neighbour.o rtnetlink.o utils.o link_watch.o filter.o
 
 obj-$(CONFIG_XFRM) += flow.o
diff --git a/net/core/neighbour.c b/net/core/neighbour.c
index 50a8c73..7b67da8 100644
--- a/net/core/neighbour.c
+++ b/net/core/neighbour.c
@@ -30,9 +30,11 @@ #include <linux/times.h>
 #include <net/neighbour.h>
 #include <net/dst.h>
 #include <net/sock.h>
+#include <net/netevent.h>
 #include <linux/rtnetlink.h>
 #include <linux/random.h>
 #include <linux/string.h>
+#include <linux/notifier.h>
 
 #define NEIGH_DEBUG 1
 
@@ -59,6 +61,7 @@ static void neigh_app_notify(struct neig
 #endif
 static int pneigh_ifdown(struct neigh_table *tbl, struct net_device *dev);
 void neigh_changeaddr(struct neigh_table *tbl, struct net_device *dev);
+static void rtm_neigh_change(struct neighbour *n);
 
 static struct neigh_table *neigh_tables;
 #ifdef CONFIG_PROC_FS
@@ -755,6 +758,7 @@ #endif
 			neigh->nud_state = NUD_STALE;
 			neigh->updated = jiffies;
 			neigh_suspect(neigh);
+			notify = 1;
 		}
 	} else if (state & NUD_DELAY) {
 		if (time_before_eq(now, 
@@ -763,6 +767,7 @@ #endif
 			neigh->nud_state = NUD_REACHABLE;
 			neigh->updated = jiffies;
 			neigh_connect(neigh);
+			notify = 1;
 			next = neigh->confirmed + neigh->parms->reachable_time;
 		} else {
 			NEIGH_PRINTK2("neigh %p is probed.\n", neigh);
@@ -820,6 +825,8 @@ #endif
 out:
 		write_unlock(&neigh->lock);
 	}
+	if (notify)
+		rtm_neigh_change(neigh);
 
 #ifdef CONFIG_ARPD
 	if (notify && neigh->parms->app_probes)
@@ -927,9 +934,7 @@ int neigh_update(struct neighbour *neigh
 {
 	u8 old;
 	int err;
-#ifdef CONFIG_ARPD
 	int notify = 0;
-#endif
 	struct net_device *dev;
 	int update_isrouter = 0;
 
@@ -949,9 +954,7 @@ #endif
 			neigh_suspect(neigh);
 		neigh->nud_state = new;
 		err = 0;
-#ifdef CONFIG_ARPD
 		notify = old & NUD_VALID;
-#endif
 		goto out;
 	}
 
@@ -1023,9 +1026,7 @@ #endif
 		if (!(new & NUD_CONNECTED))
 			neigh->confirmed = jiffies -
 				      (neigh->parms->base_reachable_time << 1);
-#ifdef CONFIG_ARPD
 		notify = 1;
-#endif
 	}
 	if (new == old)
 		goto out;
@@ -1056,7 +1057,11 @@ out:
 			(neigh->flags | NTF_ROUTER) :
 			(neigh->flags & ~NTF_ROUTER);
 	}
+
 	write_unlock_bh(&neigh->lock);
+
+	if (notify)
+		rtm_neigh_change(neigh);
 #ifdef CONFIG_ARPD
 	if (notify && neigh->parms->app_probes)
 		neigh_app_notify(neigh);
@@ -2370,9 +2375,27 @@ static void neigh_app_notify(struct neig
 	NETLINK_CB(skb).dst_group  = RTNLGRP_NEIGH;
 	netlink_broadcast(rtnl, skb, 0, RTNLGRP_NEIGH, GFP_ATOMIC);
 }
-
 #endif /* CONFIG_ARPD */
 
+static void rtm_neigh_change(struct neighbour *n)
+{
+	struct nlmsghdr *nlh;
+	int size = NLMSG_SPACE(sizeof(struct ndmsg) + 256);
+	struct sk_buff *skb = alloc_skb(size, GFP_ATOMIC);
+
+	call_netevent_notifiers(NETEVENT_NEIGH_UPDATE, n);
+	if (!skb)
+		return;
+
+	if (neigh_fill_info(skb, n, 0, 0, RTM_CHGNEIGH, 0) < 0) {
+		kfree_skb(skb);
+		return;
+	}
+	nlh			   = (struct nlmsghdr *)skb->data;
+	NETLINK_CB(skb).dst_group  = RTNLGRP_NEIGH;
+	netlink_broadcast(rtnl, skb, 0, RTNLGRP_NEIGH, GFP_ATOMIC);
+}
+
 #ifdef CONFIG_SYSCTL
 
 static struct neigh_sysctl_table {
diff --git a/net/ipv4/fib_semantics.c b/net/ipv4/fib_semantics.c
index 0f4145b..197c365 100644
--- a/net/ipv4/fib_semantics.c
+++ b/net/ipv4/fib_semantics.c
@@ -45,6 +45,7 @@ #include <net/tcp.h>
 #include <net/sock.h>
 #include <net/ip_fib.h>
 #include <net/ip_mp_alg.h>
+#include <net/netevent.h>
 
 #include "fib_lookup.h"
 
@@ -280,6 +281,14 @@ void rtmsg_fib(int event, u32 key, struc
 	struct sk_buff *skb;
 	u32 pid = req ? req->pid : n->nlmsg_pid;
 	int size = NLMSG_SPACE(sizeof(struct rtmsg)+256);
+	struct netevent_route_info nri;
+	int netevent;
+
+	nri.family = AF_INET;
+	nri.data = &fa->fa_info;
+	netevent = event == RTM_NEWROUTE ? NETEVENT_ROUTE_ADD 
+					 : NETEVENT_ROUTE_DEL;
+	call_netevent_notifiers(netevent, &nri);
 
 	skb = alloc_skb(size, GFP_KERNEL);
 	if (!skb)
diff --git a/net/ipv4/route.c b/net/ipv4/route.c
index cc9423d..a20c320 100644
--- a/net/ipv4/route.c
+++ b/net/ipv4/route.c
@@ -105,6 +105,7 @@ #include <net/tcp.h>
 #include <net/icmp.h>
 #include <net/xfrm.h>
 #include <net/ip_mp_alg.h>
+#include <net/netevent.h>
 #ifdef CONFIG_SYSCTL
 #include <linux/sysctl.h>
 #endif
@@ -1112,6 +1113,19 @@ static void rt_del(unsigned hash, struct
 	spin_unlock_bh(rt_hash_lock_addr(hash));
 }
 
+static void rtm_redirect(struct rtable *old, struct rtable *new)
+{
+	struct netevent_redirect netevent;
+	netevent.old = &old->u.dst;
+	netevent.new = &new->u.dst;
+	struct sk_buff *skb;
+
+	/* notify netevent subscribers */
+	call_netevent_notifiers(NETEVENT_REDIRECT, &netevent);
+
+	/* TBD...post NETLINK msg */
+}
+
 void ip_rt_redirect(u32 old_gw, u32 daddr, u32 new_gw,
 		    u32 saddr, struct net_device *dev)
 {
@@ -1211,6 +1225,8 @@ void ip_rt_redirect(u32 old_gw, u32 dadd
 					rt_drop(rt);
 					goto do_next;
 				}
+				
+				rtm_redirect(rth, rt);
 
 				rt_del(hash, rth);
 				if (!rt_intern_hash(hash, rt, &rt))
diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index 8a77793..ff523fb 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -908,6 +908,7 @@ int ip6_route_add(struct in6_rtmsg *rtms
 	struct net_device *dev = NULL;
 	struct inet6_dev *idev = NULL;
 	int addr_type;
+	struct netevent_route_info nri;
 
 	rta = (struct rtattr **) _rtattr;
 
@@ -1086,6 +1087,9 @@ install_route:
 		rt->u.dst.metrics[RTAX_ADVMSS-1] = ipv6_advmss(dst_mtu(&rt->u.dst));
 	rt->u.dst.dev = dev;
 	rt->rt6i_idev = idev;
+	nri.family = AF_INET6;
+	nri.data = rt;
+	call_netevent_notifiers(NETEVENT_ROUTE_ADD, &nri);
 	return ip6_ins_rt(rt, nlh, _rtattr, req);
 
 out:
@@ -1117,6 +1121,7 @@ static int ip6_route_del(struct in6_rtms
 	struct fib6_node *fn;
 	struct rt6_info *rt;
 	int err = -ESRCH;
+	struct netevent_route_info nri;
 
 	read_lock_bh(&rt6_lock);
 
@@ -1138,6 +1143,10 @@ static int ip6_route_del(struct in6_rtms
 				continue;
 			dst_hold(&rt->u.dst);
 			read_unlock_bh(&rt6_lock);
+			
+			nri.family = AF_INET6;
+			nri.data = rt;
+			call_netevent_notifiers(NETEVENT_ROUTE_DEL, &nri);
 
 			return ip6_del_rt(rt, nlh, _rtattr, req);
 		}
@@ -1147,6 +1156,17 @@ static int ip6_route_del(struct in6_rtms
 	return err;
 }
 
+static void rtm_redirect(struct rt6_info *old, struct rt6_info *new)
+{
+	struct netevent_redirect netevent;
+
+	netevent.old = &old->u.dst;
+	netevent.new = &new->u.dst;
+	call_netevent_notifiers(NETEVENT_REDIRECT, &netevent);
+
+	/* TBD...post NETLINK msg */
+}
+
 /*
  *	Handle redirects
  */
@@ -1253,6 +1273,8 @@ restart:
 	if (ip6_ins_rt(nrt, NULL, NULL, NULL))
 		goto out;
 
+	rtm_redirect(rt, nrt);
+
 	if (rt->rt6i_flags&RTF_CACHE) {
 		ip6_del_rt(rt, NULL, NULL, NULL);
 		return;

Re: [PATCH Round 2 0/2][RFC] Network Event Notifier Mechanism

[David Miller]

From: Steve Wise <swise@opengridcomputing.com>
Date: Mon, 26 Jun 2006 15:36:38 -0500

> Round 2 Changes:
> 
> - cleaned up event structures per review feedback.
> - began integration with netlink (see neighbour changes in patch 2).
> - added IPv6 support.

Yes, ipv6 support is the main think I saw lacking.
Especially the neighbour stuff.

How will you interpret the neighbour object properly, by
looking at neigh->ops->family?

> Questions:
> 
> - this patch is enabling some of the functionality that was under
>   CONFIG_ARPD.  Should I remove the CONFIG_ARPD stuff, leave it as-is
>   or what?

I think how you changed it is fine.

> - what about support for other afs?  Should I hook AF_DECnet?  IPX? Or is
>   AF_INET and AF_INET6 sufficient?

Probably this should be done on an as-needed basis.

> - I added a new RTM type in rtnetlink.h for neighbour change netlink
>   messages.  I suspect I did it wrong...

The low 3-bits of rtnetlink commands indicate the "kind".
So you have to be careful.  See rtnetlink_rcv_msg() for details.

You also have to update the tables in net/core/rtnetlink.c when
you add new values.  In particular rt_min[].

Re: [PATCH Round 2 0/2][RFC] Network Event Notifier Mechanism

[Steve Wise]

On Tue, 2006-06-27 at 03:09 -0700, David Miller wrote:
> From: Steve Wise <swise@opengridcomputing.com>
> Date: Mon, 26 Jun 2006 15:36:38 -0500
> 
> > Round 2 Changes:
> > 
> > - cleaned up event structures per review feedback.
> > - began integration with netlink (see neighbour changes in patch 2).
> > - added IPv6 support.
> 
> Yes, ipv6 support is the main think I saw lacking.
> Especially the neighbour stuff.
> 
> How will you interpret the neighbour object properly, by
> looking at neigh->ops->family?
> 

Yea.

Also:  WRT a netlink message for REDIRECT events.  After thinking more
about this, it seems like users will already get notification of a
REDIRECT in the form of 2 messages, a RTM_DELROUTE for the old route and
a RTM_NEWROUTE message with the RTPROT_REDIRECT protocol type.  I think
this is perhaps sufficient for user mode notifications.  What do you
think?

For the RDMA kernel subsystem, however, we still need a specific event.
We need both the old and new dst_entry struct ptrs to figure out which
active connections were using the old dst_entry and should be updated to
use the new dst_entry.

Steve.

Re: [PATCH Round 2 0/2][RFC] Network Event Notifier Mechanism

[David Miller]

From: Steve Wise <swise@opengridcomputing.com>
Date: Tue, 27 Jun 2006 10:02:19 -0500

> For the RDMA kernel subsystem, however, we still need a specific event.
> We need both the old and new dst_entry struct ptrs to figure out which
> active connections were using the old dst_entry and should be updated to
> use the new dst_entry.

This change isn't truly atomic from a kernel standpoint either.

The new dst won't be selected by the socket until later,
when the socket tries to send something, notices the old dst
is obsolete, and looks up a new one.

Your code could do the same thing.

Re: [PATCH Round 2 0/2][RFC] Network Event Notifier Mechanism

[Steve Wise]

On Tue, 2006-06-27 at 13:14 -0700, David Miller wrote:
> This change isn't truly atomic from a kernel standpoint either.
> 
> The new dst won't be selected by the socket until later,
> when the socket tries to send something, notices the old dst
> is obsolete, and looks up a new one.
> 
> Your code could do the same thing.
> 

For an RDMA NIC, all this logic is in HW, which is why we need the event
notification; to tell the HW to change its next hop information.

Re: [PATCH Round 2 0/2][RFC] Network Event Notifier Mechanism

[David Miller]

From: Steve Wise <swise@opengridcomputing.com>
Date: Tue, 27 Jun 2006 15:19:08 -0500

> For an RDMA NIC, all this logic is in HW, which is why we need the event
> notification; to tell the HW to change its next hop information.

Back to the route change notification, I still think you can
get what you need by just looking for the route delete.

You can match if any RDMA connection is using the deleted
route, mark it "update pending" or something like that,
and when the you get the "new route" event you can walk the
"pending" list and try to relookup the route for those
connections.

Re: [PATCH Round 2 0/2][RFC] Network Event Notifier Mechanism

[Steve Wise]

On Tue, 2006-06-27 at 13:21 -0700, David Miller wrote:
> From: Steve Wise <swise@opengridcomputing.com>
> Date: Tue, 27 Jun 2006 15:19:08 -0500
> 
> > For an RDMA NIC, all this logic is in HW, which is why we need the event
> > notification; to tell the HW to change its next hop information.
> 
> Back to the route change notification, I still think you can
> get what you need by just looking for the route delete.
> 
> You can match if any RDMA connection is using the deleted
> route, mark it "update pending" or something like that,
> and when the you get the "new route" event you can walk the
> "pending" list and try to relookup the route for those
> connections.

>From my experimentation with netlink, RTM_NEWROUTE and RTM_DELROUTE
messages do not get sent up for redirect events.  I have, in fact, added
this with the new patch I'll send out soon.  So either way I need to
change the IPv[46] code to generate a notification for redirects.  With
the single NETEVENT_REDIRECT call, the RDMA driver can, in one sweep,
update all the connections.  It seems more efficient.  At the place
where I've hooked redirect, both the old route and the new route are
already created.


Steve.

Re: [PATCH Round 2 0/2][RFC] Network Event Notifier Mechanism

[David Miller]

From: Steve Wise <swise@opengridcomputing.com>
Date: Tue, 27 Jun 2006 15:33:19 -0500

> From my experimentation with netlink, RTM_NEWROUTE and RTM_DELROUTE
> messages do not get sent up for redirect events.  I have, in fact, added
> this with the new patch I'll send out soon.  So either way I need to
> change the IPv[46] code to generate a notification for redirects.  With
> the single NETEVENT_REDIRECT call, the RDMA driver can, in one sweep,
> update all the connections.  It seems more efficient.  At the place
> where I've hooked redirect, both the old route and the new route are
> already created.

Ok, let's see what it looks like.