* Re: send(), sendmsg(), sendto() not thread-safe
From: Stephen Hemminger @ 2006-05-15 23:35 UTC (permalink / raw)
To: Rick Jones; +Cc: mark1smi, David S. Miller, netdev
In-Reply-To: <44690C1C.8030101@hp.com>
On Mon, 15 May 2006 16:17:48 -0700
Rick Jones <rick.jones2@hp.com> wrote:
> David S. Miller wrote:
> > From: Mark A Smith <mark1smi@us.ibm.com>
> > Date: Mon, 15 May 2006 14:39:06 -0700
> >
> >
> >>I discovered that in some cases, send(), sendmsg(), and sendto() are not
> >>thread-safe. Although the man page for these functions does not specify
> >>whether these functions are supposed to be thread-safe, my reading of the
> >>POSIX/SUSv3 specification tells me that they should be. I traced the
> >>problem to tcp_sendmsg(). I was very curious about this issue, so I wrote
> >>up a small page to describe in more detail my findings. You can find it at:
> >>http://www.almaden.ibm.com/cs/people/marksmith/sendmsg.html .
>
>
> # ./sendmsgclient localhost
> ERROR! We should have all 0! We don't!
> buff[16384]=1
> buff[16385]=1
> buff[16386]=1
> buff[16387]=1
> buff[16388]=1
> buff[16389]=1
> buff[16390]=1
> buff[16391]=1
> buff[16392]=1
> buff[16393]=1
> That's 10/32768 bad bytes
> # uname -a
> HP-UX tarry B.11.23 U ia64 2397028692 unlimited-user license
>
> Given that the URL above asserts that HP-UX claims atomicity, either
> there is a bug in the UX stack, or perhaps the test? I took a quick
> look at the HP-UX 11iv2 (aka 11.23) manpage for sendmsg and didn't see
> anything about atomicity there - on which manpage(s) or docs was the
> assertion of HP-UX atomicity made?
>
> I presume this is only for "blocking" sockets? I cannot at least off
> the top of my head see how a stack could offer it on non-blocking sockets.
The test seems to be based on sending a big message. In this case,
on non-blocking sockets, the send call will return partial status. The
return from the system call will be less than the number of bytes requested.
>
> > And frankly, BSD defines BSD socket semantics here not some wording in
> > the POSIX standards.
>
> Have BSD socket semantics ever been updated/clarified any any
> quasi-official manner since the popular presence of threads? Or
> are/were Posix/Xopen filling a gap?
>
> rick jones
> -
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] tcpdump may trace some outbound packets twice.
From: Stephen Hemminger @ 2006-05-15 23:41 UTC (permalink / raw)
To: Ranjit Manomohan; +Cc: David S. Miller, ranjitm, akpm, linux-kernel, netdev
In-Reply-To: <Pine.LNX.4.56.0605151602330.29636@ranjit.corp.google.com>
On Mon, 15 May 2006 16:11:05 -0700 (PDT)
Ranjit Manomohan <ranjitm@google.com> wrote:
> On Mon, 15 May 2006, David S. Miller wrote:
>
> > From: Ranjit Manomohan <ranjitm@google.com>
> > Date: Mon, 15 May 2006 14:19:06 -0700 (PDT)
> >
> > > Heres a new version which does a copy instead of the clone to avoid
> > > the double cloning issue.
> >
> > I still very much dislike this patch because it is creating
> > 1 more clone per packet than is actually necessary and that
> > is very expensive.
> >
> > dev_queue_xmit_nit() is going to clone whatever SKB you send into
> > there, so better to just bump the reference count (with skb_get())
> > instead of cloning or copying.
> >
>
> I was a bit apprehensive about just incrementing the refcnt but that works
> too. Attached is the modified version.
>
> -Thanks,
> Ranjit
>
> --- linux-2.6/net/sched/sch_generic.c 2006-05-10 12:34:52.000000000 -0700
> +++ linux/net/sched/sch_generic.c 2006-05-15 15:48:03.000000000 -0700
> @@ -136,8 +136,12 @@
>
> if (!netif_queue_stopped(dev)) {
> int ret;
> + struct sk_buff *skbc = NULL;
> + /* Increment the reference count on the skb so
> + * that we can use it after a successful xmit.
> + */
> if (netdev_nit)
> - dev_queue_xmit_nit(skb, dev);
> + skbc = skb_get(skb);
skbc = netdev_nit ? skb_get(skb) : NULL;
>
> ret = dev->hard_start_xmit(skb, dev);
> if (ret == NETDEV_TX_OK) {
> @@ -145,9 +149,20 @@
> dev->xmit_lock_owner = -1;
> spin_unlock(&dev->xmit_lock);
> }
> + if (skbc) {
> + /* transmit succeeded,
> + * trace the buffer. */
> + dev_queue_xmit_nit(skbc,dev);
> + kfree_skb(skbc);
> + }
> spin_lock(&dev->queue_lock);
> return -1;
> }
> +
> + /* Call free in case we incremented refcnt */
> + if (skbc)
> + kfree_skb(skbc);
kfree_skb(NULL) is legal so the conditional here is unneeded.
But the increased calls to kfree_skb(NULL) would probably bring the
"unlikely()" hordes descending on kfree_skb, so maybe:
if (unlikely(netdev_nit))
kfree_skb(skbc);
^ permalink raw reply
* Re: send(), sendmsg(), sendto() not thread-safe
From: Rick Jones @ 2006-05-16 0:02 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: mark1smi, David S. Miller, netdev
In-Reply-To: <20060515163545.4e3d755a@localhost.localdomain>
>>I presume this is only for "blocking" sockets? I cannot at least off
>>the top of my head see how a stack could offer it on non-blocking sockets.
>
>
> The test seems to be based on sending a big message. In this case,
> on non-blocking sockets, the send call will return partial status. The
> return from the system call will be less than the number of bytes requested.
Right, and at that point it is already too late to do anything about
keeping some other thread of execution from shoving _its_ bytes into the
socket before the rest of the first sends' can be put there.
(Perhaps I'm preaching to the choir)
rick jones
^ permalink raw reply
* Re: [PATCH] tcpdump may trace some outbound packets twice.
From: David S. Miller @ 2006-05-16 0:08 UTC (permalink / raw)
To: shemminger; +Cc: ranjitm, akpm, linux-kernel, netdev
In-Reply-To: <20060515164101.054afa29@localhost.localdomain>
From: Stephen Hemminger <shemminger@osdl.org>
Date: Mon, 15 May 2006 16:41:01 -0700
> kfree_skb(NULL) is legal so the conditional here is unneeded.
>
> But the increased calls to kfree_skb(NULL) would probably bring the
> "unlikely()" hordes descending on kfree_skb, so maybe:
And unfortunately as Patrick McHardy states, we can't use
this trick here because things like tc actions can do stuff
like pskb_expand_head() which cannot handle shared SKBs.
We need another solution to this problem, because cloning an
extra SKB is just rediculious overhead so isn't something we
can seriously consider to solve this problem.
Another option is to say this anomaly doesn't matter enough
to justify the complexity we're looking at here just to fix
this glitch.
Other implementation possibility suggestions welcome :-)
^ permalink raw reply
* Re: [PATCH] tcpdump may trace some outbound packets twice.
From: Patrick McHardy @ 2006-05-16 0:21 UTC (permalink / raw)
To: David S. Miller; +Cc: shemminger, ranjitm, akpm, linux-kernel, netdev
In-Reply-To: <20060515.170835.126804002.davem@davemloft.net>
David S. Miller wrote:
> From: Stephen Hemminger <shemminger@osdl.org>
> Date: Mon, 15 May 2006 16:41:01 -0700
>
>
>>kfree_skb(NULL) is legal so the conditional here is unneeded.
>>
>>But the increased calls to kfree_skb(NULL) would probably bring the
>>"unlikely()" hordes descending on kfree_skb, so maybe:
>
>
> And unfortunately as Patrick McHardy states, we can't use
> this trick here because things like tc actions can do stuff
> like pskb_expand_head() which cannot handle shared SKBs.
>
> We need another solution to this problem, because cloning an
> extra SKB is just rediculious overhead so isn't something we
> can seriously consider to solve this problem.
>
> Another option is to say this anomaly doesn't matter enough
> to justify the complexity we're looking at here just to fix
> this glitch.
>
> Other implementation possibility suggestions welcome :-)
We could just mark the skb to make sure its only passed to taps
on the first transmission attempt. Since we have the timestamp
optimization there shouldn't be any visible change besides
the desired effect.
^ permalink raw reply
* Re: [PATCH] tcpdump may trace some outbound packets twice.
From: Herbert Xu @ 2006-05-16 0:37 UTC (permalink / raw)
To: David S. Miller; +Cc: shemminger, ranjitm, akpm, linux-kernel, netdev
In-Reply-To: <20060515.170835.126804002.davem@davemloft.net>
David S. Miller <davem@davemloft.net> wrote:
>
> Other implementation possibility suggestions welcome :-)
I see two possibilities:
1) Move the af_packet hook into the NIC driver.
2) Rethink the lockless tx setup. If all NICs followed the tg3 and
replaced spin_lock_irqsave with spin_lock then we should be able
to go back to just using the xmit_lock.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: send(), sendmsg(), sendto() not thread-safe
From: Rick Jones @ 2006-05-16 0:43 UTC (permalink / raw)
To: Mark A Smith; +Cc: shemminger, Linux Network Development list
In-Reply-To: <OFA7F8723C.2DDF9383-ON85257170.00014BE5-88257170.00019BEA@us.ibm.com>
Mark A Smith wrote:
> Hi Rick, Stephen,
>
> The thread-safe claim is at:
>
> http://devrsrc1.external.hp.com/STKS/cgi-bin/man2html?manpage=/usr/share/man/man2.Z/send.2
>
> Specifically,
>
> "
> MULTITHREAD USAGE
> The send(), sendmsg(), and sendto() system calls are thread-safe.
> They each have a cancellation point; and they are async-cancel safe,
> async-signal safe, and fork-safe.
> "
That looks to be the 11iv1 manpage (aka 11.11). I wonder if perhaps
there is a distinction made between "thread-safety" and an atomicity
semantic?
Also, _strictly_ speaking, since your test is calling fork() rather than
pthread_create(), it isn't really testing "thread safty" but multiple
process atomicity right?
> I noticed that you were thinking that the problem may be with my test and
> that the send call is returning partial status.
Either the test, or the stack.
> Actually, that's exactly
> the issue. On the systems I tested on, and I assume HP-UX, send is _not_
> returning partial status, it is returning that the entire buffer has been
> written, and yet is interleaving data from packets in the other thread.
Ostensibly, I should see some ten byte send messages in the output of
sendmsgserver yes? I just ran a test where it was all 32768's, no 10's
but the client still reported an error. Is that supposed to be possible?
# sendmsgserver > sendmsgserver.log
# wc sendmsgserver.log
165888 663552 3981312 sendmsgserver.log
# grep -v 32768 sendmsgserver.log
#
# ./sendmsgclient localhost
ERROR! We should have all 0! We don't!
buff[16384]=1
buff[16385]=1
buff[16386]=1
buff[16387]=1
buff[16388]=1
buff[16389]=1
buff[16390]=1
buff[16391]=1
buff[16392]=1
buff[16393]=1
That's 10/32768 bad bytes
I've also seen it fail at 12288 rather than 16384. I wonder if perhaps
there are unstated limits to the size of the write that can be atomic?
Looking at the 11.11 manpage for write(2) in the discussion of writes to
a pipe or FIFO it says:
+ Write requests of {PIPE_BUF} bytes or less will not be
interleaved with data from other processes doing writes on the
same pipe. Writes of greater than {PIPE_BUF} bytes may have
data interleaved, on arbitrary boundaries, with writes by
other processes, whether or not the O_NONBLOCK flag of the
file status flags is set.
from limits.h:
# define _POSIX_PIPE_BUF 512 /* The number of bytes that can
be written atomically when
writing to a pipe. */
later
# define PIPE_BUF 8192 /* max number bytes that is guaranteed
to be atomic when writing to a pipe */
Under various #ifdef checks and such. It would not surprise me if there
was a limit to the size of a buffer in a send/sendto/sendmsg call
similar to that for write against a pipe.
I wonder if similar limits exist for the other stacks in the "yes" column.
rick jones
^ permalink raw reply
* Re: [PATCH] tcpdump may trace some outbound packets twice.
From: Tom Young @ 2006-05-16 0:48 UTC (permalink / raw)
To: Patrick McHardy
Cc: David S. Miller, shemminger, ranjitm, akpm, linux-kernel, netdev
In-Reply-To: <44691B07.6070003@trash.net>
On Tue, May 16, 2006 at 02:21:27AM +0200, Patrick McHardy wrote:
> David S. Miller wrote:
> > From: Stephen Hemminger <shemminger@osdl.org>
> > Date: Mon, 15 May 2006 16:41:01 -0700
> >
> >
> >>kfree_skb(NULL) is legal so the conditional here is unneeded.
> >>
> >>But the increased calls to kfree_skb(NULL) would probably bring the
> >>"unlikely()" hordes descending on kfree_skb, so maybe:
> >
> >
> > And unfortunately as Patrick McHardy states, we can't use
> > this trick here because things like tc actions can do stuff
> > like pskb_expand_head() which cannot handle shared SKBs.
> >
> > We need another solution to this problem, because cloning an
> > extra SKB is just rediculious overhead so isn't something we
> > can seriously consider to solve this problem.
> >
> > Another option is to say this anomaly doesn't matter enough
> > to justify the complexity we're looking at here just to fix
> > this glitch.
> >
> > Other implementation possibility suggestions welcome :-)
>
>
> We could just mark the skb to make sure its only passed to taps
> on the first transmission attempt. Since we have the timestamp
> optimization there shouldn't be any visible change besides
> the desired effect.
It would be nice to have a solution that taps after its been sent to the
driver. I clearly need to investigate the tc problems, but I've been using a
similar patch now for a few weeks to allow more accurate timestamps to be
performed in the driver without cloning problems getting in the way. (the tap
skb only gets cloned after the driver has time stamped it). This is more
accuracy than most need but is required for the clock synchronisation i'm
working on. It also seems less intrusive to let the driver have the skb first
before pushing it to the taps.
Tom
--
Thomas Young
http://cubinlab.ee.unimelb.edu.au/~tyo/
Research Assistant
CUBIN Research Centre - University of Melbourne
^ permalink raw reply
* Re: [PATCH 1/3] Rough VJ Channel Implementation - vj_core.patch
From: Kelly Daly @ 2006-05-16 1:02 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, rusty
In-Reply-To: <200605051248.42662.kelly@au.ibm.com>
On Friday 05 May 2006 12:48, Kelly Daly wrote:
> done! I will continue with implementation of default netchannel for now.
___________________
diff -urp davem_orig/include/net/inet_hashtables.h kelly/include/net/inet_hashtables.h
--- davem_orig/include/net/inet_hashtables.h 2006-04-27 00:08:32.000000000 +1000
+++ kelly/include/net/inet_hashtables.h 2006-05-05 12:45:44.000000000 +1000
@@ -418,4 +418,7 @@ static inline struct sock *inet_lookup(s
extern int inet_hash_connect(struct inet_timewait_death_row *death_row,
struct sock *sk);
+extern void inet_hash_register(u8 proto, struct inet_hashinfo *hashinfo);
+extern struct sock *inet_lookup_proto(u8 protocol, u32 saddr, u16 sport, u32 daddr, u16 dport, int ifindex);
+
#endif /* _INET_HASHTABLES_H */
diff -urp davem_orig/include/net/sock.h kelly/include/net/sock.h
--- davem_orig/include/net/sock.h 2006-05-02 13:42:10.000000000 +1000
+++ kelly/include/net/sock.h 2006-05-04 14:28:59.000000000 +1000
@@ -196,6 +196,7 @@ struct sock {
unsigned short sk_type;
int sk_rcvbuf;
socket_lock_t sk_lock;
+ struct netchannel *sk_channel;
wait_queue_head_t *sk_sleep;
struct dst_entry *sk_dst_cache;
struct xfrm_policy *sk_policy[2];
diff -urp davem_orig/net/core/dev.c kelly/net/core/dev.c
--- davem_orig/net/core/dev.c 2006-04-27 15:49:27.000000000 +1000
+++ kelly/net/core/dev.c 2006-05-15 12:21:41.000000000 +1000
@@ -113,9 +113,11 @@
#include <linux/delay.h>
#include <linux/wireless.h>
#include <linux/netchannel.h>
+#include <linux/kthread.h>
#include <net/iw_handler.h>
#include <asm/current.h>
#include <linux/audit.h>
+#include <net/inet_hashtables.h>
/*
* The list of packet types we will receive (as opposed to discard)
@@ -190,6 +192,10 @@ static inline struct hlist_head *dev_ind
return &dev_index_head[ifindex & ((1<<NETDEV_HASHBITS)-1)];
}
+/* default netchannel shtuff */
+static struct netchannel default_netchannel;
+static wait_queue_head_t default_netchannel_wq;
+
/*
* Our notifier list
*/
@@ -1854,6 +1860,35 @@ softnet_break:
goto out;
}
+static void default_netchannel_wake(struct netchannel *np)
+{
+ wake_up(&default_netchannel_wq);
+}
+
+/* handles default chan buffers that nobody else wants */
+static int default_netchannel_thread(void *unused)
+{
+ wait_queue_t wait;
+ struct netchannel_buftrailer *bp;
+ struct sk_buff *skbp;
+
+ wait.private = current;
+ wait.func = default_wake_function;;
+ INIT_LIST_HEAD(&wait.task_list);
+
+ add_wait_queue(&default_netchannel_wq, &wait);
+ set_current_state(TASK_UNINTERRUPTIBLE);
+ while (!kthread_should_stop()) {
+ bp = __netchannel_dequeue(&default_netchannel);
+ skbp = skb_netchan_graft(bp, GFP_ATOMIC);
+ netif_receive_skb(skbp);
+ }
+ remove_wait_queue(&default_netchannel_wq, &wait);
+ __set_current_state(TASK_RUNNING);
+ return 0;
+}
+
void netchannel_init(struct netchannel *np,
void (*callb)(struct netchannel *), void *callb_data)
{
@@ -1907,6 +1942,34 @@ struct netchannel_buftrailer *__netchann
}
EXPORT_SYMBOL_GPL(__netchannel_dequeue);
+
+/* Find the channel for a packet, or return default channel. */
+struct netchannel *find_netchannel(const struct netchannel_buftrailer *np)
+{
+ struct sock *sk = NULL;
+ unsigned long dlen = np->netchan_buf_len - np->netchan_buf_offset;
+ void *data = (void *)np - dlen;
+
+ switch (np->netchan_buf_proto) {
+ case __constant_htons(ETH_P_IP): {
+ struct iphdr *ip = data;
+ int iphl = ip->ihl * 4;
+
+ if (dlen >= (iphl + 4) && iphl == sizeof(struct iphdr)) {
+ u16 *ports = (u16 *)(ip + 1);
+ sk = inet_lookup_proto(ip->protocol,
+ ip->saddr, ports[0],
+ ip->daddr, ports[1],
+ np->netchan_buf_dev->ifindex);
+ break;
+ }
+ }
+ }
+ if (sk && sk->sk_channel)
+ return sk->sk_channel;
+ return &default_netchannel;
+}
+
static gifconf_func_t * gifconf_list [NPROTO];
/**
@@ -3375,6 +3438,7 @@ static int dev_cpu_callback(struct notif
static int __init net_dev_init(void)
{
int i, rc = -ENOMEM;
+ struct task_struct *netchan_thread;
BUG_ON(!dev_boot_phase);
@@ -3421,7 +3485,12 @@ static int __init net_dev_init(void)
hotcpu_notifier(dev_cpu_callback, 0);
dst_init();
dev_mcast_init();
- rc = 0;
+
+ netchannel_init(&default_netchannel, default_netchannel_wake, NULL);
+ netchan_thread = kthread_run(default_netchannel_thread, NULL, "kvj_def");
+
+ if (!IS_ERR(netchan_thread)) /* kthread_run returned thread */
+ rc = 0;
out:
return rc;
}
diff -urp davem_orig/net/ipv4/inet_hashtables.c kelly/net/ipv4/inet_hashtables.c
--- davem_orig/net/ipv4/inet_hashtables.c 2006-04-27 00:08:33.000000000 +1000
+++ kelly/net/ipv4/inet_hashtables.c 2006-05-05 12:45:44.000000000 +1000
@@ -337,3 +337,25 @@ out:
}
EXPORT_SYMBOL_GPL(inet_hash_connect);
+
+static struct inet_hashinfo *inet_hashes[256];
+
+void inet_hash_register(u8 proto, struct inet_hashinfo *hashinfo)
+{
+ BUG_ON(inet_hashes[proto]);
+ inet_hashes[proto] = hashinfo;
+}
+EXPORT_SYMBOL(inet_hash_register);
+
+struct sock *inet_lookup_proto(u8 protocol, u32 saddr, u16 sport, u32 daddr, u16 dport, int ifindex)
+{
+ struct sock *sk = NULL;
+ if (inet_hashes[protocol]) {
+ sk = inet_lookup(inet_hashes[protocol],
+ saddr, sport,
+ daddr, dport,
+ ifindex);
+ }
+ return sk;
+}
+EXPORT_SYMBOL(inet_lookup_proto);
diff -urp davem_orig/net/ipv4/tcp.c kelly/net/ipv4/tcp.c
--- davem_orig/net/ipv4/tcp.c 2006-04-27 00:08:33.000000000 +1000
+++ kelly/net/ipv4/tcp.c 2006-05-05 11:29:18.000000000 +1000
@@ -2173,6 +2173,7 @@ void __init tcp_init(void)
tcp_hashinfo.ehash_size << 1, tcp_hashinfo.bhash_size);
tcp_register_congestion_control(&tcp_reno);
+ inet_hash_register(IPPROTO_TCP, &tcp_hashinfo);
}
EXPORT_SYMBOL(tcp_close);
^ permalink raw reply
* Re: [PATCH 1/3] Rough VJ Channel Implementation - vj_core.patch
From: David S. Miller @ 2006-05-16 1:05 UTC (permalink / raw)
To: kelly; +Cc: netdev, rusty
In-Reply-To: <200605161102.29472.kelly@au.ibm.com>
From: Kelly Daly <kelly@au1.ibm.com>
Date: Tue, 16 May 2006 11:02:29 +1000
> On Friday 05 May 2006 12:48, Kelly Daly wrote:
> > done! I will continue with implementation of default netchannel for now.
Some context? It's been a week since we were discussing this,
so I'd like to know what we're looking at here in this patch :)
^ permalink raw reply
* Re: [PATCH 1/3] Rough VJ Channel Implementation - vj_core.patch
From: Kelly Daly @ 2006-05-16 1:15 UTC (permalink / raw)
To: David S. Miller; +Cc: kelly, netdev, rusty
In-Reply-To: <20060515.180535.90170603.davem@davemloft.net>
On Tuesday 16 May 2006 11:05, David S. Miller wrote:
> From: Kelly Daly <kelly@au1.ibm.com>
> Date: Tue, 16 May 2006 11:02:29 +1000
>
> > On Friday 05 May 2006 12:48, Kelly Daly wrote:
> > > done! I will continue with implementation of default netchannel for
> > > now.
>
> Some context? It's been a week since we were discussing this,
> so I'd like to know what we're looking at here in this patch :)
the implementation of the default netchannel =)
> -
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] tcpdump may trace some outbound packets twice.
From: Patrick McHardy @ 2006-05-16 1:17 UTC (permalink / raw)
To: Herbert Xu
Cc: David S. Miller, shemminger, ranjitm, akpm, linux-kernel, netdev
In-Reply-To: <E1FfnZP-0003St-00@gondolin.me.apana.org.au>
Herbert Xu wrote:
> David S. Miller <davem@davemloft.net> wrote:
>
>>Other implementation possibility suggestions welcome :-)
>
>
> I see two possibilities:
>
> 1) Move the af_packet hook into the NIC driver.
> 2) Rethink the lockless tx setup. If all NICs followed the tg3 and
> replaced spin_lock_irqsave with spin_lock then we should be able
> to go back to just using the xmit_lock.
3) Clone the skb and have dev_queue_xmit_nit() consume it.
That should actually be pretty easy.
^ permalink raw reply
* Re: [PATCH] tcpdump may trace some outbound packets twice.
From: Herbert Xu @ 2006-05-16 1:20 UTC (permalink / raw)
To: Patrick McHardy
Cc: David S. Miller, shemminger, ranjitm, akpm, linux-kernel, netdev
In-Reply-To: <44692847.4080100@trash.net>
On Tue, May 16, 2006 at 03:17:59AM +0200, Patrick McHardy wrote:
>
> 3) Clone the skb and have dev_queue_xmit_nit() consume it.
>
> That should actually be pretty easy.
Unfortunately that would mean an unconditional copy for all TSO packets
on NICs such as tg3/e1000. These drivers have to modify the data area
of the skb.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH] tcpdump may trace some outbound packets twice.
From: Patrick McHardy @ 2006-05-16 1:22 UTC (permalink / raw)
To: Herbert Xu
Cc: David S. Miller, shemminger, ranjitm, akpm, linux-kernel, netdev
In-Reply-To: <44692847.4080100@trash.net>
Patrick McHardy wrote:
> 3) Clone the skb and have dev_queue_xmit_nit() consume it.
>
> That should actually be pretty easy.
On second thought, thats not so great either. netdev_nit
just globally signals that there are some taps, but we
don't know if they're interested in a specific packet.
^ permalink raw reply
* Re: send(), sendmsg(), sendto() not thread-safe
From: Mark A Smith @ 2006-05-16 1:50 UTC (permalink / raw)
To: Rick Jones; +Cc: Linux Network Development list, shemminger
In-Reply-To: <4469201F.4030207@hp.com>
I cannot think of another possible definition for thread-safe in the
context of these functions. Certainly they should not cause a "crash" when
called from multiple threads, but that's a requirement independent of
thread-safety. The POSIX specification doesn't mention atomicity with
respect to these functions. In the context of read(), it defines atomicity
as follows: "Atomic means that all the bytes from a single operation that
started out together end up together, without interleaving from other I/O
operations." This seems to be the semantics I'm expecting from a
"thread-safe" sendmsg(). What else could thread-safe mean for send(),
sendmsg()?
As for fork() vs. pthreads, my original post indicated that this occurs
with or without pthreads, the same problem occurs if you use pthreads. I
tried to indicate that I was using the term "thread" to be independent of
whether the address space was shared or not.
If you do not see any "send() sent 10 bytes" messages in your server
output, it means that the problem occured the very first time the 10-byte
send was attempted. If you watch the data going over the wire, you'll see
those 10 bytes of 1's.
That write() document is excellent, and reads exactly like the POSIX
specification.
As for limits, I modified my test to use very small buffers, below any
reasonable limit. By removing the printf's, which slow the server
considerably, I can still get the problem to produce.
I think the question really boils down to: "What does thread-safe mean with
respect to send()?" It might be more easily answered by asking, "How would
a non-thread-safe send() behave?" I think it would behave the way we're
seeing it behave.
-Mark
That's excellent documentation, and reads exactly to the write()
specification in POSIX.
|---------+---------------------------->
| | Rick Jones |
| | <rick.jones2@hp.c|
| | om> |
| | |
| | 05/15/2006 05:43 |
| | PM |
|---------+---------------------------->
>-----------------------------------------------------------------------------------------------------------------|
| |
| To: Mark A Smith/Almaden/IBM@IBMUS |
| cc: shemminger@osdl.org, Linux Network Development list <netdev@vger.kernel.org> |
| Subject: Re: send(), sendmsg(), sendto() not thread-safe |
>-----------------------------------------------------------------------------------------------------------------|
Mark A Smith wrote:
> Hi Rick, Stephen,
>
> The thread-safe claim is at:
>
>
http://devrsrc1.external.hp.com/STKS/cgi-bin/man2html?manpage=/usr/share/man/man2.Z/send.2
>
> Specifically,
>
> "
> MULTITHREAD USAGE
> The send(), sendmsg(), and sendto() system calls are thread-safe.
> They each have a cancellation point; and they are async-cancel
safe,
> async-signal safe, and fork-safe.
> "
That looks to be the 11iv1 manpage (aka 11.11). I wonder if perhaps
there is a distinction made between "thread-safety" and an atomicity
semantic?
Also, _strictly_ speaking, since your test is calling fork() rather than
pthread_create(), it isn't really testing "thread safty" but multiple
process atomicity right?
> I noticed that you were thinking that the problem may be with my test and
> that the send call is returning partial status.
Either the test, or the stack.
> Actually, that's exactly
> the issue. On the systems I tested on, and I assume HP-UX, send is _not_
> returning partial status, it is returning that the entire buffer has been
> written, and yet is interleaving data from packets in the other thread.
Ostensibly, I should see some ten byte send messages in the output of
sendmsgserver yes? I just ran a test where it was all 32768's, no 10's
but the client still reported an error. Is that supposed to be possible?
# sendmsgserver > sendmsgserver.log
# wc sendmsgserver.log
165888 663552 3981312 sendmsgserver.log
# grep -v 32768 sendmsgserver.log
#
# ./sendmsgclient localhost
ERROR! We should have all 0! We don't!
buff[16384]=1
buff[16385]=1
buff[16386]=1
buff[16387]=1
buff[16388]=1
buff[16389]=1
buff[16390]=1
buff[16391]=1
buff[16392]=1
buff[16393]=1
That's 10/32768 bad bytes
I've also seen it fail at 12288 rather than 16384. I wonder if perhaps
there are unstated limits to the size of the write that can be atomic?
Looking at the 11.11 manpage for write(2) in the discussion of writes to
a pipe or FIFO it says:
+ Write requests of {PIPE_BUF} bytes or less will not be
interleaved with data from other processes doing writes on the
same pipe. Writes of greater than {PIPE_BUF} bytes may have
data interleaved, on arbitrary boundaries, with writes by
other processes, whether or not the O_NONBLOCK flag of the
file status flags is set.
from limits.h:
# define _POSIX_PIPE_BUF 512 /* The number of bytes that can
be written atomically when
writing to a pipe. */
later
# define PIPE_BUF 8192 /* max number bytes that is guaranteed
to be atomic when writing to a pipe */
Under various #ifdef checks and such. It would not surprise me if there
was a limit to the size of a buffer in a send/sendto/sendmsg call
similar to that for write against a pipe.
I wonder if similar limits exist for the other stacks in the "yes" column.
rick jones
^ permalink raw reply
* Re: too many iterations (6) in nv_nic_irq
From: Jeff Garzik @ 2006-05-16 2:11 UTC (permalink / raw)
To: Carl-Daniel Hailfinger; +Cc: Ayaz Abdulla, netdev
In-Reply-To: <445B22CA.4050405@gmx.net>
Carl-Daniel Hailfinger wrote:
> Hi Jeff,
>
> IIRC you had "too many iterations (6) in nv_nic_irq" appear regularly
> in your dmesg with kernel 2.6.16. Did this disappear in more recent
> kernels? If not, can you try the disable_msi and disable_msix module
> parameters if they help in your case?
Yes, it disappeared in recent kernels.
Jeff
^ permalink raw reply
* Re: [PATCH] pcnet32.c: modify RX ring size through module parameter
From: Wen Hsin Chang @ 2006-05-16 3:05 UTC (permalink / raw)
To: Jon Mason; +Cc: davem, netdev, tsbogend, brazilnut
In-Reply-To: <20060515155856.GA17646@us.ibm.com>
There are cases where rx errors were found due to rx ring size being too
small
during remote installation using pcnet32. It is found that ethtool
functionality
may not be available under such circumenstance. This is the purpose behind
this patch.
However, as Don pointed out, there are several points that need to be
considered. I'll further revise this patch and see how to come up with
a better solution.
Thanks for your comment~
Wen Hsin Chang
Jon Mason wrote:
>Why is this necessary? There is already an ethtool function to set
>the rx ring size (pcnet32_set_ringparam). Since module parameters
>are being phased out in favor of the ethtool functions, why not use
>the existing ethtool infrastructure for this?
>
>Thanks,
>Jon
>
>On Mon, May 15, 2006 at 11:32:14AM +0800, Wen Hsin Chang wrote:
>
>
>>This patch is created from pcnet32.c v1.32. it will allow users to
>>specify RX ring size upon module
>>insertion via module parameter 'rx_log_size'. This is needed in some
>>cases where too small the rx ring
>>size will cause RX errors upon remote installation via pcnet32 NIC card.
>>
>>Signed-off-by: Wen Hsin Chang <whchang@tw.ibm.com>
>>----------------------------------------------------------------------------------------------------
>>
>>--- a/drivers/net/pcnet32.c 2006-03-30 09:49:10.000000000 +0800
>>+++ b/drivers/net/pcnet32.c 2006-05-15 11:14:45.000000000 +0800
>>@@ -93,6 +93,9 @@ static struct net_device *pcnet32_dev;
>>static int max_interrupt_work = 2;
>>static int rx_copybreak = 200;
>>
>>+/* Module parameter to specify RX ring size at module insertion */
>>+static int rx_log_size = 0;
>>+
>>#define PCNET32_PORT_AUI 0x00
>>#define PCNET32_PORT_10BT 0x01
>>#define PCNET32_PORT_GPSI 0x02
>>@@ -1264,7 +1267,10 @@ pcnet32_probe1(unsigned long ioaddr, int
>> lp->name = chipname;
>> lp->shared_irq = shared;
>> lp->tx_ring_size = TX_RING_SIZE; /* default tx ring size */
>>- lp->rx_ring_size = RX_RING_SIZE; /* default rx ring size */
>>+ if (!rx_log_size)
>>+ lp->rx_ring_size = RX_RING_SIZE; /* default rx ring size */
>>+ else
>>+ lp->rx_ring_size = (1 << (rx_log_size));
>> lp->tx_mod_mask = lp->tx_ring_size - 1;
>> lp->rx_mod_mask = lp->rx_ring_size - 1;
>> lp->tx_len_bits = (PCNET32_LOG_TX_BUFFERS << 12);
>>@@ -2707,6 +2713,11 @@ module_param(tx_start_pt, int, 0);
>>MODULE_PARM_DESC(tx_start_pt, DRV_NAME " transmit start point (0-3)");
>>module_param(pcnet32vlb, int, 0);
>>MODULE_PARM_DESC(pcnet32vlb, DRV_NAME " Vesa local bus (VLB) support
>>(0/1)");
>>+
>>+/* Module parameter to specify RX ring size at module insertion */
>>+module_param(rx_log_size, int, 0);
>>+MODULE_PARM_DESC(rx_log_size, DRV_NAME " RX Ring Buffer Size (log_2
>>#BUF) ");
>>+
>>module_param_array(options, int, NULL, 0);
>>MODULE_PARM_DESC(options, DRV_NAME " initial option setting(s) (0-15)");
>>module_param_array(full_duplex, int, NULL, 0);
>>@@ -2731,6 +2742,12 @@ static int __init pcnet32_init_module(vo
>>
>> if ((tx_start_pt >= 0) && (tx_start_pt <= 3))
>> tx_start = tx_start_pt;
>>+
>>+ /* validating rx_log_size */
>>+ if ((rx_log_size <= 0) ||
>>+ (rx_log_size > PCNET32_LOG_MAX_RX_BUFFERS))
>>+ rx_log_size = 0;
>>+
>>
>> /* find the PCI devices */
>> if (!pci_module_init(&pcnet32_driver))
>>
>>-
>>To unsubscribe from this list: send the line "unsubscribe netdev" in
>>the body of a message to majordomo@vger.kernel.org
>>More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
>-
>To unsubscribe from this list: send the line "unsubscribe netdev" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
^ permalink raw reply
* RE: Hardware button support for Wireless cards
From: Mark Wallis @ 2006-05-16 3:10 UTC (permalink / raw)
To: 'Jouni Malinen', 'Ivo van Doorn'
Cc: 'Dan Williams', netdev, 'David Zeuthen'
In-Reply-To: <20060515162045.GA10717@instant802.com>
On Tue, May 16, 2006, Jouni Malinen wrote:
> Some hardware designs disable the radio in hardware, some
> don't.. In other words, the behavior would not be consistent
> between hardware designs unless the radio would be disabled
> unconditionally in software-processed case. I would like to
> be able to trust on the rfkill to really kill the radio
> regardless of whether the user space mechanism is in place or
> not, i.e., I see this as a function that can be used to
> disable radio in environments were use of wireless devices is
> not allowed for some reason.
>
> Sending an event to user space would be a good additional
> indication that the radio was disabled. If user space wants
> to do something first, I would think that doing this with
> full software control (i.e., not depending on any particular
> hardware button and just having a UI mechanism for triggering
> this) would allow more consistent behavior.
Sounds like this is the popular option - I agree that immediate
radio-off is the "safe" way to go here as at least the driver
can guarantee that the radio is disabled no matter the userspace
config.
Does someone want to put their hand up and specify the userspace
interface here and now so we all keep to a standard ? Working
as an input device as David suggested seems to give us the most
flexibility.
Regards,
Mark Wallis
E: mwallis@serialmonkey.com
^ permalink raw reply
* Re: [PATCH] tcpdump may trace some outbound packets twice.
From: David S. Miller @ 2006-05-16 4:18 UTC (permalink / raw)
To: kaber; +Cc: herbert, shemminger, ranjitm, akpm, linux-kernel, netdev
In-Reply-To: <4469294D.6010509@trash.net>
From: Patrick McHardy <kaber@trash.net>
Date: Tue, 16 May 2006 03:22:21 +0200
> Patrick McHardy wrote:
> > 3) Clone the skb and have dev_queue_xmit_nit() consume it.
> >
> > That should actually be pretty easy.
>
> On second thought, thats not so great either. netdev_nit
> just globally signals that there are some taps, but we
> don't know if they're interested in a specific packet.
Yes, and all of these issues coming up are why we do the
dev_queue_xmit_nit() before not after we try to give it to the device.
Even the "NIT done" bit doesn't work, for cases like ICMP replies
which can reuse the SKB received, over loopback. Sure we can try to
forcefully clear the bit everywhere we reuse buffers for replies like
that, but I wish anyone trying to implement that the best of luck
finding all spots successfully.
To be honest, I don't think this bug is worth all the energy we are
trying to put into fixing it.
We get a double packet when the spinlock is hit, big deal.
^ permalink raw reply
* Re: [PATCH] net_sched: potential jiffy wrap bug in dev_watchdog
From: David S. Miller @ 2006-05-16 5:12 UTC (permalink / raw)
To: shemminger; +Cc: netdev
In-Reply-To: <20060515162858.35eee987@localhost.localdomain>
From: Stephen Hemminger <shemminger@osdl.org>
Date: Mon, 15 May 2006 16:28:58 -0700
> There is a potential jiffy wraparound bug in the transmit watchdog
> that is easily avoided by using time_after().
>
> Signed-off-by: Stephen Hemminger <shemminger@osdl.org>
Applied, thanks Stephen.
^ permalink raw reply
* Re: [PATCH 1/3] Rough VJ Channel Implementation - vj_core.patch
From: David S. Miller @ 2006-05-16 5:16 UTC (permalink / raw)
To: kelly; +Cc: netdev, rusty
In-Reply-To: <200605161102.29472.kelly@au.ibm.com>
From: Kelly Daly <kelly@au1.ibm.com>
Date: Tue, 16 May 2006 11:02:29 +1000
> +/* handles default chan buffers that nobody else wants */
> +static int default_netchannel_thread(void *unused)
> +{
> + wait_queue_t wait;
> + struct netchannel_buftrailer *bp;
> + struct sk_buff *skbp;
> +
> + wait.private = current;
> + wait.func = default_wake_function;;
> + INIT_LIST_HEAD(&wait.task_list);
> +
> + add_wait_queue(&default_netchannel_wq, &wait);
> + set_current_state(TASK_UNINTERRUPTIBLE);
> + while (!kthread_should_stop()) {
> + bp = __netchannel_dequeue(&default_netchannel);
> + skbp = skb_netchan_graft(bp, GFP_ATOMIC);
> + netif_receive_skb(skbp);
> + }
> + remove_wait_queue(&default_netchannel_wq, &wait);
> + __set_current_state(TASK_RUNNING);
> + return 0;
> +}
> +
When does this thread ever go to sleep? Seems like it will loop
forever and not block when the default_netchannel queue is empty.
:-)
> + unsigned long dlen = np->netchan_buf_len - np->netchan_buf_offset;
Probably deserves a "netchan_buf_len(bp)" inline in linux/netchannel.h
> diff -urp davem_orig/net/ipv4/inet_hashtables.c kelly/net/ipv4/inet_hashtables.c
> --- davem_orig/net/ipv4/inet_hashtables.c 2006-04-27 00:08:33.000000000 +1000
> +++ kelly/net/ipv4/inet_hashtables.c 2006-05-05 12:45:44.000000000 +1000
The hash table bits look good, just as they did last time :-)
So I'll put this part into my vj-2.6 tree now, thanks.
^ permalink raw reply
* [1/1] netchannel subsystem.
From: Evgeniy Polyakov @ 2006-05-16 6:19 UTC (permalink / raw)
To: netdev; +Cc: David S. Miller, netdev, Kelly Daly, rusty
In-Reply-To: <200605161102.29472.kelly@au.ibm.com>
Let me also bring attention to another netchannel implementation.
Some design notes [blog copypasts, sorry if it is out of sync sometimes].
First of all, do not use sockets. Just forget that such interface
exists.
New receiving channel abstraction will be created by special syscall,
which allows to specify in creation time source IP range, or it can be
wildcarded to IPADDR_ANY.
Interrupt handler (or netif_receive_skb()) will check skb and try to get
netchannel, then skb will be linked into netchannel receive queue if
memory limits allow it and that is all. No protocol processing.
If hardware allows to split header from data it will be possible to
create zero-copy technique: when netchannel is selected in
netif_receive_skb() it will have ->alloc_data() callback which would
allow, for example, to allocate data from userspace mapped area.
When receiver is awakened after data has been added to netchannel, it
will call netchannel's ->get_data() callback which would allow to remap
data pages from skb into process' VMA.
Unified network cache will hold information about source and destination
IP addresses, or it's hashes for IPv6, information about ports and
well-defined IP protocols.
It is built as two dimensional array of default 8 order (i.e. 64k
entries) with XOR hash. Comparison of Jenkins hash with XOR hash used in
TCP socket selection code can be found at [1].
Main problem here is routing cache. Linux is great in routing, it
supports several cache algorithms, which are extremely fast in
appropriate environments, but main question is "do we need routing for
netchannels"?. So it looks like netchannel either must include that
route interface or, if it will be designed as userspace-only
communication channel, implement own trivial pseudo-route algorithm.
Most of the drivers do not set skb->pkt_type, so it stays default
PACKET_HOST, which means that every packet, received by interface must
be checked to be valid for the system, which is a point to use standard
Linux routing, but from the other point of view, it is quite easy to
create own callback for IP address setup, which would update netchannel
routing table.
No one need an excuse to rewrite anything, so I will create own cache
for netchannels, which will be updated when IP addresses are changed.
Ok, basic interfaces have been created:
* unified cache to store netchannels (IPv4 and stub for IPv6 hashes,
TCP and UDP)
* skb queueing mechanism
* netchannel creation/removing commands
* netchannel's callback to allocate/free pages (for
example to get data from mapped area) not only from SLAB cache
There are only couple of things left (today maybe):
* create netchannel's callback to move/copy data to userspace
* create read data command
* test new netchannel interface :) (it boots and works ok
with netchannel interface turned on for usual sockets)
If you have read all above, here is link to hash comparison [1].
[1]. Compared Jenkins hash with XOR hash used in TCP socket selection code.
http://tservice.net.ru/~s0mbre/blog/2006/05/14#2006_05_14
And proof-of-concept patch itself.
Signed-off-by: Evgeniy Polyakov <johnpol@2ka.mipt.ru>
diff --git a/arch/i386/kernel/syscall_table.S b/arch/i386/kernel/syscall_table.S
index f48bef1..b69d7d3 100644
--- a/arch/i386/kernel/syscall_table.S
+++ b/arch/i386/kernel/syscall_table.S
@@ -315,3 +315,4 @@ ENTRY(sys_call_table)
.long sys_splice
.long sys_sync_file_range
.long sys_tee /* 315 */
+ .long sys_netchannel_control
diff --git a/arch/x86_64/ia32/ia32entry.S b/arch/x86_64/ia32/ia32entry.S
index 5a92fed..fdfb997 100644
--- a/arch/x86_64/ia32/ia32entry.S
+++ b/arch/x86_64/ia32/ia32entry.S
@@ -696,4 +696,5 @@ ia32_sys_call_table:
.quad sys_sync_file_range
.quad sys_tee
.quad compat_sys_vmsplice
+ .quad sys_netchannel_control
ia32_syscall_end:
diff --git a/include/asm-x86_64/unistd.h b/include/asm-x86_64/unistd.h
index feb77cb..08c230e 100644
--- a/include/asm-x86_64/unistd.h
+++ b/include/asm-x86_64/unistd.h
@@ -617,8 +617,10 @@ __SYSCALL(__NR_tee, sys_tee)
__SYSCALL(__NR_sync_file_range, sys_sync_file_range)
#define __NR_vmsplice 278
__SYSCALL(__NR_vmsplice, sys_vmsplice)
+#define __NR_netchannel_control 279
+__SYSCALL(__NR_vmsplice, sys_netchannel_control)
-#define __NR_syscall_max __NR_vmsplice
+#define __NR_syscall_max __NR_netchannel_control
#ifndef __NO_STUBS
diff --git a/include/linux/netchannel.h b/include/linux/netchannel.h
new file mode 100644
index 0000000..4e65c9b
--- /dev/null
+++ b/include/linux/netchannel.h
@@ -0,0 +1,73 @@
+/*
+ * netchannel.h
+ *
+ * 2006 Copyright (c) Evgeniy Polyakov <johnpol@2ka.mipt.ru>
+ * All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ */
+
+#ifndef __NETCHANNEL_H
+#define __NETCHANNEL_H
+
+#include <linux/types.h>
+
+enum netchannel_commands {
+ NETCHANNEL_CREATE = 0,
+ NETCHANNEL_REMOVE,
+ NETCHANNEL_BIND,
+ NETCHANNEL_READ,
+};
+
+struct unetchannel
+{
+ __u32 src, dst; /* source/destination hashes */
+ __u16 sport, dport; /* source/destination ports */
+ __u8 proto; /* IP protocol number */
+ __u8 listen;
+ __u8 reserved[2];
+};
+
+struct unetchannel_control
+{
+ struct unetchannel unc;
+ __u32 cmd;
+ __u32 len;
+};
+
+#ifdef __KERNEL__
+
+struct netchannel
+{
+ struct hlist_node node;
+ atomic_t refcnt;
+ struct rcu_head rcu_head;
+ struct unetchannel unc;
+ unsigned long hit;
+
+ struct page * (*nc_alloc_page)(unsigned int size);
+ void (*nc_free_page)(struct page *page);
+
+ struct sk_buff_head list;
+};
+
+struct netchannel_cache_head
+{
+ struct hlist_head head;
+ struct mutex mutex;
+};
+
+#endif /* __KERNEL__ */
+#endif /* __NETCHANNEL_H */
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index a461b51..9924911 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -684,6 +684,15 @@ extern void dev_queue_xmit_nit(struct s
extern void dev_init(void);
+#ifdef CONFIG_NETCHANNEL
+extern int netchannel_recv(struct sk_buff *skb);
+#else
+static int netchannel_recv(struct sk_buff *skb)
+{
+ return -1;
+}
+#endif
+
extern int netdev_nit;
extern int netdev_budget;
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index f8f2347..accd00b 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -301,6 +301,7 @@ struct sk_buff {
* Handling routines are only of interest to the kernel
*/
#include <linux/slab.h>
+#include <linux/netchannel.h>
#include <asm/system.h>
@@ -314,6 +315,17 @@ static inline struct sk_buff *alloc_skb(
return __alloc_skb(size, priority, 0);
}
+#ifdef CONFIG_NETCHANNEL
+extern struct sk_buff *netchannel_alloc(struct unetchannel *unc, unsigned int header_size,
+ unsigned int total_size, gfp_t gfp_mask);
+#else
+static struct sk_buff *netchannel_alloc(struct unetchannel *unc, unsigned int header_size,
+ unsigned int total_size, gfp_t gfp_mask)
+{
+ return NULL;
+}
+#endif
+
static inline struct sk_buff *alloc_skb_fclone(unsigned int size,
gfp_t priority)
{
diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h
index 3996960..296929d 100644
--- a/include/linux/syscalls.h
+++ b/include/linux/syscalls.h
@@ -582,4 +582,6 @@ asmlinkage long sys_tee(int fdin, int fd
asmlinkage long sys_sync_file_range(int fd, loff_t offset, loff_t nbytes,
unsigned int flags);
+asmlinkage int sys_netchannel_control(void __user *arg);
+
#endif
diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c
index 5433195..1747fc3 100644
--- a/kernel/sys_ni.c
+++ b/kernel/sys_ni.c
@@ -132,3 +132,5 @@ cond_syscall(sys_mincore);
cond_syscall(sys_madvise);
cond_syscall(sys_mremap);
cond_syscall(sys_remap_file_pages);
+
+cond_syscall(sys_netchannel_control);
diff --git a/net/Kconfig b/net/Kconfig
index 4193cdc..465e37b 100644
--- a/net/Kconfig
+++ b/net/Kconfig
@@ -66,6 +66,14 @@ source "net/ipv6/Kconfig"
endif # if INET
+config NETCHANNEL
+ bool "Network channels"
+ ---help---
+ Network channels are peer-to-peer abstraction, which allows to create
+ high performance communications.
+ Main advantages are unified address cache, protocol processing moved
+ to userspace, receiving zero-copy support and other interesting features.
+
menuconfig NETFILTER
bool "Network packet filtering (replaces ipchains)"
---help---
diff --git a/net/core/Makefile b/net/core/Makefile
index 79fe12c..7119812 100644
--- a/net/core/Makefile
+++ b/net/core/Makefile
@@ -16,3 +16,4 @@ obj-$(CONFIG_NET_DIVERT) += dv.o
obj-$(CONFIG_NET_PKTGEN) += pktgen.o
obj-$(CONFIG_WIRELESS_EXT) += wireless.o
obj-$(CONFIG_NETPOLL) += netpoll.o
+obj-$(CONFIG_NETCHANNEL) += netchannel.o
diff --git a/net/core/dev.c b/net/core/dev.c
index 9ab3cfa..2721111 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -1712,6 +1712,10 @@ int netif_receive_skb(struct sk_buff *sk
}
}
+ ret = netchannel_recv(skb);
+ if (!ret)
+ goto out;
+
#ifdef CONFIG_NET_CLS_ACT
if (pt_prev) {
ret = deliver_skb(skb, pt_prev, orig_dev);
diff --git a/net/core/netchannel.c b/net/core/netchannel.c
new file mode 100644
index 0000000..5b3b2bb
--- /dev/null
+++ b/net/core/netchannel.c
@@ -0,0 +1,583 @@
+/*
+ * netchannel.c
+ *
+ * 2006 Copyright (c) Evgeniy Polyakov <johnpol@2ka.mipt.ru>
+ * All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ */
+
+#include <linux/types.h>
+#include <linux/unistd.h>
+#include <linux/linkage.h>
+#include <linux/notifier.h>
+#include <linux/list.h>
+#include <linux/slab.h>
+#include <linux/skbuff.h>
+#include <linux/errno.h>
+
+#include <linux/in.h>
+#include <linux/ip.h>
+#include <linux/tcp.h>
+#include <linux/udp.h>
+
+#include <linux/netdevice.h>
+#include <linux/inetdevice.h>
+#include <net/addrconf.h>
+
+#include <asm/uaccess.h>
+
+static unsigned int netchannel_hash_order = 8;
+static struct netchannel_cache_head ***netchannel_hash_table;
+static kmem_cache_t *netchannel_cache;
+
+static int netchannel_inetaddr_notifier_call(struct notifier_block *, unsigned long, void *);
+static struct notifier_block netchannel_inetaddr_notifier = {
+ .notifier_call = &netchannel_inetaddr_notifier_call
+};
+
+#ifdef CONFIG_IPV6
+static int netchannel_inet6addr_notifier_call(struct notifier_block *, unsigned long, void *);
+static struct notifier_block netchannel_inet6addr_notifier = {
+ .notifier_call = &netchannel_inet6addr_notifier_call
+};
+#endif
+
+static inline unsigned int netchannel_hash(struct unetchannel *unc)
+{
+ unsigned int h = (unc->dst ^ unc->dport) ^ (unc->src ^ unc->sport);
+ h ^= h >> 16;
+ h ^= h >> 8;
+ h ^= unc->proto;
+ return h & ((1 << 2*netchannel_hash_order) - 1);
+}
+
+static inline void netchannel_convert_hash(unsigned int hash, unsigned int *col, unsigned int *row)
+{
+ *row = hash & ((1 << netchannel_hash_order) - 1);
+ *col = (hash >> netchannel_hash_order) & ((1 << netchannel_hash_order) - 1);
+}
+
+static struct netchannel_cache_head *netchannel_bucket(struct unetchannel *unc)
+{
+ unsigned int hash = netchannel_hash(unc);
+ unsigned int col, row;
+
+ netchannel_convert_hash(hash, &col, &row);
+ return netchannel_hash_table[col][row];
+}
+
+static inline int netchannel_hash_equal_full(struct unetchannel *unc1, struct unetchannel *unc2)
+{
+ return (unc1->dport == unc2->dport) && (unc1->dst == unc2->dst) &&
+ (unc1->sport == unc2->sport) && (unc1->src == unc2->src) &&
+ (unc1->proto == unc2->proto);
+}
+
+static inline int netchannel_hash_equal_dest(struct unetchannel *unc1, struct unetchannel *unc2)
+{
+ return ((unc1->dport == unc2->dport) && (unc1->dst == unc2->dst) && (unc1->proto == unc2->proto));
+}
+
+static struct netchannel *netchannel_check_dest(struct unetchannel *unc, struct netchannel_cache_head *bucket)
+{
+ struct netchannel *nc;
+ struct hlist_node *node;
+ int found = 0;
+
+ hlist_for_each_entry_rcu(nc, node, &bucket->head, node) {
+ if (netchannel_hash_equal_dest(&nc->unc, unc)) {
+ found = 1;
+ break;
+ }
+ }
+
+ return (found)?nc:NULL;
+}
+
+static struct netchannel *netchannel_check_full(struct unetchannel *unc, struct netchannel_cache_head *bucket)
+{
+ struct netchannel *nc;
+ struct hlist_node *node;
+ int found = 0;
+
+ hlist_for_each_entry_rcu(nc, node, &bucket->head, node) {
+ if (netchannel_hash_equal_full(&nc->unc, unc)) {
+ found = 1;
+ break;
+ }
+ }
+
+ return (found)?nc:NULL;
+}
+
+static void netchannel_free_rcu(struct rcu_head *rcu)
+{
+ struct netchannel *nc = container_of(rcu, struct netchannel, rcu_head);
+
+ kmem_cache_free(netchannel_cache, nc);
+}
+
+static inline void netchannel_get(struct netchannel *nc)
+{
+ atomic_inc(&nc->refcnt);
+}
+
+static inline void netchannel_put(struct netchannel *nc)
+{
+ if (atomic_dec_and_test(&nc->refcnt))
+ call_rcu(&nc->rcu_head, &netchannel_free_rcu);
+}
+
+
+static int netchannel_convert_skb_ipv6(struct sk_buff *skb, struct unetchannel *unc)
+{
+ /*
+ * Hash IP addresses into src/dst. Setup TCP/UDP ports.
+ * Not supported yet.
+ */
+ return -1;
+}
+
+static int netchannel_convert_skb_ipv4(struct sk_buff *skb, struct unetchannel *unc)
+{
+ struct iphdr *iph;
+ u32 len;
+ struct tcphdr *th;
+ struct udphdr *uh;
+
+ if (!pskb_may_pull(skb, sizeof(struct iphdr)))
+ goto inhdr_error;
+
+ iph = skb->nh.iph;
+
+ if (iph->ihl < 5 || iph->version != 4)
+ goto inhdr_error;
+
+ if (!pskb_may_pull(skb, iph->ihl*4))
+ goto inhdr_error;
+
+ iph = skb->nh.iph;
+
+ if (unlikely(ip_fast_csum((u8 *)iph, iph->ihl)))
+ goto inhdr_error;
+
+ len = ntohs(iph->tot_len);
+ if (skb->len < len || len < (iph->ihl*4))
+ goto inhdr_error;
+
+ unc->dst = iph->daddr;
+ unc->src = iph->saddr;
+ unc->proto = iph->protocol;
+
+ len = skb->len;
+
+ switch (unc->proto) {
+ case IPPROTO_TCP:
+ if (!pskb_may_pull(skb, sizeof(struct tcphdr)))
+ goto inhdr_error;
+ th = skb->h.th;
+
+ if (th->doff < sizeof(struct tcphdr) / 4)
+ goto inhdr_error;
+
+ unc->dport = th->dest;
+ unc->sport = th->source;
+ break;
+ case IPPROTO_UDP:
+ if (!pskb_may_pull(skb, sizeof(struct udphdr)))
+ goto inhdr_error;
+ uh = skb->h.uh;
+
+ if (ntohs(uh->len) > len || ntohs(uh->len) < sizeof(*uh))
+ goto inhdr_error;
+
+ unc->dport = uh->dest;
+ unc->sport = uh->source;
+ break;
+ default:
+ goto inhdr_error;
+ }
+
+ return 0;
+
+inhdr_error:
+ return -1;
+}
+
+static int netchannel_convert_skb(struct sk_buff *skb, struct unetchannel *unc)
+{
+ if (skb->pkt_type == PACKET_OTHERHOST)
+ return -1;
+
+ switch (skb->protocol) {
+ case ETH_P_IP:
+ return netchannel_convert_skb_ipv4(skb, unc);
+ case ETH_P_IPV6:
+ return netchannel_convert_skb_ipv6(skb, unc);
+ default:
+ return -1;
+ }
+}
+
+/*
+ * By design netchannels allow to "allocate" data
+ * not only from SLAB cache, but get it from mapped area
+ * or from VFS cache (requires process' context or preallocation).
+ */
+struct sk_buff *netchannel_alloc(struct unetchannel *unc, unsigned int header_size,
+ unsigned int total_size, gfp_t gfp_mask)
+{
+ struct netchannel *nc;
+ struct netchannel_cache_head *bucket;
+ int err;
+ struct sk_buff *skb = NULL;
+ unsigned int size, pnum, i;
+
+ skb = alloc_skb(header_size, gfp_mask);
+ if (!skb)
+ return NULL;
+
+ rcu_read_lock();
+ bucket = netchannel_bucket(unc);
+ nc = netchannel_check_full(unc, bucket);
+ if (!nc) {
+ err = -ENODEV;
+ goto err_out_free_skb;
+ }
+
+ if (!nc->nc_alloc_page || !nc->nc_free_page) {
+ err = -EINVAL;
+ goto err_out_free_skb;
+ }
+
+ netchannel_get(nc);
+
+ size = total_size - header_size;
+ pnum = PAGE_ALIGN(size) >> PAGE_SHIFT;
+
+ for (i=0; i<pnum; ++i) {
+ unsigned int cs = min_t(unsigned int, PAGE_SIZE, size);
+ struct page *page;
+
+ page = nc->nc_alloc_page(cs);
+ if (!page)
+ break;
+
+ skb_fill_page_desc(skb, skb_shinfo(skb)->nr_frags, page, 0, cs);
+
+ skb->len += cs;
+ skb->data_len += cs;
+ skb->truesize += cs;
+
+ size -= cs;
+ }
+
+ if (i < pnum) {
+ pnum = i;
+ err = -ENOMEM;
+ goto err_out_free_frags;
+ }
+
+ rcu_read_unlock();
+
+ return skb;
+
+err_out_free_frags:
+ for (i=0; i<pnum; ++i) {
+ unsigned int cs = skb_shinfo(skb)->frags[i].size;
+ struct page *page = skb_shinfo(skb)->frags[i].page;
+
+ nc->nc_free_page(page);
+
+ skb->len -= cs;
+ skb->data_len -= cs;
+ skb->truesize -= cs;
+ }
+
+err_out_free_skb:
+ kfree_skb(skb);
+ return NULL;
+}
+
+int netchannel_recv(struct sk_buff *skb)
+{
+ struct netchannel *nc;
+ struct unetchannel unc;
+ struct netchannel_cache_head *bucket;
+ int err;
+
+ if (!netchannel_hash_table)
+ return -ENODEV;
+
+ rcu_read_lock();
+
+ err = netchannel_convert_skb(skb, &unc);
+ if (err)
+ goto unlock;
+
+ bucket = netchannel_bucket(&unc);
+ nc = netchannel_check_full(&unc, bucket);
+ if (!nc) {
+ err = -ENODEV;
+ goto unlock;
+ }
+
+ nc->hit++;
+
+ skb_queue_tail(&nc->list, skb);
+
+unlock:
+ rcu_read_unlock();
+ return err;
+}
+
+static int netchannel_create(struct unetchannel *unc)
+{
+ struct netchannel *nc;
+ int err = -ENOMEM;
+ struct netchannel_cache_head *bucket;
+
+ if (!netchannel_hash_table)
+ return -ENODEV;
+
+ bucket = netchannel_bucket(unc);
+
+ mutex_lock(&bucket->mutex);
+
+ if (netchannel_check_full(unc, bucket)) {
+ err = -EEXIST;
+ goto out_unlock;
+ }
+
+ if (unc->listen && netchannel_check_dest(unc, bucket)) {
+ err = -EEXIST;
+ goto out_unlock;
+ }
+
+ nc = kmem_cache_alloc(netchannel_cache, GFP_KERNEL);
+ if (!nc)
+ goto out_exit;
+
+ memset(nc, 0, sizeof(struct netchannel));
+
+ nc->hit = 0;
+ skb_queue_head_init(&nc->list);
+ atomic_set(&nc->refcnt, 1);
+ memcpy(&nc->unc, unc, sizeof(struct unetchannel));
+
+ hlist_add_head_rcu(&nc->node, &bucket->head);
+
+out_unlock:
+ mutex_unlock(&bucket->mutex);
+out_exit:
+ printk("netchannel: create %u.%u.%u.%u:%d -> %u.%u.%u.%u:%d, proto: %d, err: %d.\n",
+ NIPQUAD(unc->src), unc->sport, NIPQUAD(unc->dst), unc->dport, unc->proto, err);
+
+ return err;
+}
+
+static int netchannel_remove(struct unetchannel *unc)
+{
+ struct netchannel *nc;
+ int err = -ENODEV;
+ struct netchannel_cache_head *bucket;
+ unsigned long hit = 0;
+
+ if (!netchannel_hash_table)
+ return -ENODEV;
+
+ bucket = netchannel_bucket(unc);
+
+ mutex_lock(&bucket->mutex);
+
+ nc = netchannel_check_full(unc, bucket);
+ if (!nc)
+ nc = netchannel_check_dest(unc, bucket);
+
+ if (!nc)
+ goto out_unlock;
+
+ hlist_del_rcu(&nc->node);
+ hit = nc->hit;
+
+ netchannel_put(nc);
+ err = 0;
+
+out_unlock:
+ mutex_unlock(&bucket->mutex);
+ printk("netchannel: remove %u.%u.%u.%u:%d -> %u.%u.%u.%u:%d, proto: %d, err: %d, hit: %lu.\n",
+ NIPQUAD(unc->src), unc->sport, NIPQUAD(unc->dst), unc->dport, unc->proto, err,
+ hit);
+ return 0;
+}
+
+asmlinkage int sys_netchannel_control(void __user *arg)
+{
+ struct unetchannel_control ctl;
+ int ret;
+
+ if (!netchannel_hash_table)
+ return -ENODEV;
+
+ if (copy_from_user(&ctl, arg, sizeof(struct unetchannel_control)))
+ return -ERESTARTSYS;
+
+ switch (ctl.cmd) {
+ case NETCHANNEL_CREATE:
+ case NETCHANNEL_BIND:
+ ret = netchannel_create(&ctl.unc);
+ break;
+ case NETCHANNEL_REMOVE:
+ ret = netchannel_remove(&ctl.unc);
+ break;
+ default:
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+static inline void netchannel_dump_addr(struct in_ifaddr *ifa, char *str)
+{
+ printk("netchannel: %s %u.%u.%u.%u/%u.%u.%u.%u\n", str, NIPQUAD(ifa->ifa_local), NIPQUAD(ifa->ifa_mask));
+}
+
+static int netchannel_inetaddr_notifier_call(struct notifier_block *this, unsigned long event, void *ptr)
+{
+ struct in_ifaddr *ifa = ptr;
+
+ printk("netchannel: inet event=%lx, ifa=%p.\n", event, ifa);
+
+ switch (event) {
+ case NETDEV_UP:
+ netchannel_dump_addr(ifa, "add");
+ break;
+ case NETDEV_DOWN:
+ netchannel_dump_addr(ifa, "del");
+ break;
+ default:
+ netchannel_dump_addr(ifa, "unk");
+ break;
+ }
+
+ return NOTIFY_DONE;
+}
+
+#ifdef CONFIG_IPV6
+static int netchannel_inet6addr_notifier_call(struct notifier_block *this, unsigned long event, void *ptr)
+{
+ struct inet6_ifaddr *ifa = ptr;
+
+ printk("netchannel: inet6 event=%lx, ifa=%p.\n", event, ifa);
+ return NOTIFY_DONE;
+}
+#endif
+
+static int __init netchannel_init(void)
+{
+ unsigned int i, j, size;
+ int err = -ENOMEM;
+
+ size = (1 << netchannel_hash_order);
+
+ netchannel_hash_table = kzalloc(size * sizeof(void *), GFP_KERNEL);
+ if (!netchannel_hash_table)
+ goto err_out_exit;
+
+ for (i=0; i<size; ++i) {
+ struct netchannel_cache_head **col;
+
+ col = kzalloc(size * sizeof(void *), GFP_KERNEL);
+ if (!col)
+ break;
+
+ for (j=0; j<size; ++j) {
+ struct netchannel_cache_head *head;
+
+ head = kzalloc(sizeof(struct netchannel_cache_head), GFP_KERNEL);
+ if (!head)
+ break;
+
+ INIT_HLIST_HEAD(&head->head);
+ mutex_init(&head->mutex);
+
+ col[j] = head;
+ }
+
+ if (j<size && j>0) {
+ while (j >= 0)
+ kfree(col[j--]);
+ kfree(col);
+ break;
+ }
+
+ netchannel_hash_table[i] = col;
+ }
+
+ if (i<size) {
+ size = i;
+ goto err_out_free;
+ }
+
+ netchannel_cache = kmem_cache_create("netchannel", sizeof(struct netchannel), 0, 0,
+ NULL, NULL);
+ if (!netchannel_cache)
+ goto err_out_free;
+
+ register_inetaddr_notifier(&netchannel_inetaddr_notifier);
+#ifdef CONFIG_IPV6
+ register_inet6addr_notifier(&netchannel_inet6addr_notifier);
+#endif
+
+ printk("netchannel: Created %u order two-dimensional hash table.\n",
+ netchannel_hash_order);
+
+ return 0;
+
+err_out_free:
+ for (i=0; i<size; ++i) {
+ for (j=0; j<(1 << netchannel_hash_order); ++j)
+ kfree(netchannel_hash_table[i][j]);
+ kfree(netchannel_hash_table[i]);
+ }
+ kfree(netchannel_hash_table);
+err_out_exit:
+
+ printk("netchannel: Failed to create %u order two-dimensional hash table.\n",
+ netchannel_hash_order);
+ return err;
+}
+
+static void __exit netchannel_exit(void)
+{
+ unsigned int i, j;
+
+ unregister_inetaddr_notifier(&netchannel_inetaddr_notifier);
+#ifdef CONFIG_IPV6
+ unregister_inet6addr_notifier(&netchannel_inet6addr_notifier);
+#endif
+ kmem_cache_destroy(netchannel_cache);
+
+ for (i=0; i<(1 << netchannel_hash_order); ++i) {
+ for (j=0; j<(1 << netchannel_hash_order); ++j)
+ kfree(netchannel_hash_table[i][j]);
+ kfree(netchannel_hash_table[i]);
+ }
+ kfree(netchannel_hash_table);
+}
+
+late_initcall(netchannel_init);
--
Evgeniy Polyakov
^ permalink raw reply related
* Re: [1/1] netchannel subsystem.
From: David S. Miller @ 2006-05-16 6:57 UTC (permalink / raw)
To: johnpol; +Cc: netdev, kelly, rusty
In-Reply-To: <20060516061909.GA28670@2ka.mipt.ru>
From: Evgeniy Polyakov <johnpol@2ka.mipt.ru>
Date: Tue, 16 May 2006 10:19:09 +0400
> +static int netchannel_convert_skb_ipv4(struct sk_buff *skb, struct unetchannel *unc)
> +{
...
> + switch (unc->proto) {
> + case IPPROTO_TCP:
...
> + case IPPROTO_UDP:
...
Why do people write code like this?
Port location is protocol agnostic, there are always 2
16-bit ports at beginning of header without exception.
Without this, ICMP would be useless :-)
^ permalink raw reply
* Re: [1/1] netchannel subsystem.
From: Evgeniy Polyakov @ 2006-05-16 6:59 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, kelly, rusty
In-Reply-To: <20060515.235712.26771832.davem@davemloft.net>
On Mon, May 15, 2006 at 11:57:12PM -0700, David S. Miller (davem@davemloft.net) wrote:
> From: Evgeniy Polyakov <johnpol@2ka.mipt.ru>
> Date: Tue, 16 May 2006 10:19:09 +0400
>
> > +static int netchannel_convert_skb_ipv4(struct sk_buff *skb, struct unetchannel *unc)
> > +{
> ...
> > + switch (unc->proto) {
> > + case IPPROTO_TCP:
> ...
> > + case IPPROTO_UDP:
> ...
>
> Why do people write code like this?
>
> Port location is protocol agnostic, there are always 2
> 16-bit ports at beginning of header without exception.
>
> Without this, ICMP would be useless :-)
And what if we use ESP which would place it's hashed sequence number as
port?
--
Evgeniy Polyakov
^ permalink raw reply
* YOU WANT SOME OUTRIGHT SEX - DON'T YOU?
From: Pat Conrad @ 2006-05-16 6:59 UTC (permalink / raw)
To: netdev
The Best Place To Find The Cheapest Medicine.
http://juwv.pxvxugcuv4cf77p0c77iuppp.tullianhf.com/?esuhn
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox