* Re: [PATCH net-next ] net: doc: fix many typos in scaling.txt
From: David Miller @ 2011-12-20 19:12 UTC (permalink / raw)
To: shanwei88; +Cc: netdev
In-Reply-To: <4EEFF427.1090801@gmail.com>
From: Shan Wei <shanwei88@gmail.com>
Date: Tue, 20 Dec 2011 10:34:15 +0800
> Fix some trivial typos.
>
> Signed-off-by: Shan Wei <shanwei88@gmail.com>
Applied.
^ permalink raw reply
* Re: [PATCH] batman-adv: checkpatch cleanup - remove lines longer than 80 chars
From: Marek Lindner @ 2011-12-20 19:15 UTC (permalink / raw)
To: b.a.t.m.a.n-ZwoEplunGu2X36UT3dwllkB+6BGkLq7r
Cc: netdev-u79uwXL29TY76Z2rM5mHXA, David Miller
In-Reply-To: <20111220.140209.618109781658024408.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
On Wednesday, December 21, 2011 03:02:09 David Miller wrote:
> From: Marek Lindner <lindner_marek-LWAfsSFWpa4@public.gmane.org>
> Date: Wed, 21 Dec 2011 02:59:41 +0800
>
> > On Wednesday, December 21, 2011 02:32:30 David Miller wrote:
> >> From: Marek Lindner <lindner_marek-LWAfsSFWpa4@public.gmane.org>
> >> Date: Tue, 20 Dec 2011 18:38:24 +0800
> >>
> >> > The long line was introduced with b26e478f8f.
> >> >
> >> > Signed-off-by: Marek Lindner <lindner_marek-LWAfsSFWpa4@public.gmane.org>
> >>
> >> I'm not applying this.
> >>
> >> I tell people to make sure arguments line up correctly to the
> >> openning parenthesis on the previous line, and that is what is
> >> happening here.
> >
> > I am not against lining up the arguments but what about checkpatch ? Are
> > we going to ignore the complaints or is this line limit about to be
> > changed ?
>
> Refactor the code so that both constraints can be satisfied.
>
> Is this so hard to understand?
Well, my crystal ball did not unveil that a checkpatch complaint you silently
introduced would mean somebody else has to refactor the code. Guess I have to
get a replacement unit.
Cheers,
Marek
^ permalink raw reply
* Re: [PATCH RFC] virtio_net: fix refill related races
From: Michael S. Tsirkin @ 2011-12-20 19:30 UTC (permalink / raw)
To: Tejun Heo; +Cc: Amit Shah, netdev, linux-kernel, virtualization
In-Reply-To: <20111220190946.GD10752@google.com>
On Tue, Dec 20, 2011 at 11:09:46AM -0800, Tejun Heo wrote:
> Hello, Michael.
>
> On Tue, Dec 20, 2011 at 09:09:08PM +0200, Michael S. Tsirkin wrote:
> > Another question, wanted to make sure:
> > virtnet_poll does schedule_delayed_work(&vi->refill, 0);
> > separately refill work itself also does
> > schedule_delayed_work(&vi->refill, HZ/2);
> > If two such events happen twice, on different CPUs, we are still guaranteed
> > the work will only run once, right?
>
> No, it's not. Normal workqueues only guarantee non-reentrance on
> local CPU. If you want to guarantee that only one instance of a given
> item is executing across all CPUs, you need to use the nrt workqueue.
>
> Thanks.
Hmm, in that case it looks like a nasty race could get
triggered, with try_fill_recv run on multiple CPUs in parallel,
corrupting the linked list within the vq.
Using the mutex as my patch did will fix that naturally, as well.
Rusty, am I missing something?
> --
> tejun
^ permalink raw reply
* Re: [PATCH RFC] virtio_net: fix refill related races
From: Tejun Heo @ 2011-12-20 19:31 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: Amit Shah, netdev, linux-kernel, virtualization
In-Reply-To: <20111220193055.GA26392@redhat.com>
On Tue, Dec 20, 2011 at 09:30:55PM +0200, Michael S. Tsirkin wrote:
> Hmm, in that case it looks like a nasty race could get
> triggered, with try_fill_recv run on multiple CPUs in parallel,
> corrupting the linked list within the vq.
>
> Using the mutex as my patch did will fix that naturally, as well.
Don't know the code but just use nrt wq. There's even a system one
called system_nrq_wq.
Thanks.
--
tejun
^ permalink raw reply
* [RFC][PATCH] net-tcp: TCP/IP stack bypass for loopback connections.
From: Bruce Brutus Curtis @ 2011-11-23 0:09 UTC (permalink / raw)
To: davem; +Cc: netdev
TCP/IP loopback socket pair stack bypass, based on an idea by, and
rough upstream patch from, David Miller <davem@davemloft.net> called
"friends", the data structure modifcations and connection scheme are
reused with new dedicated code for the data path.
A new sysctl, net.ipv4.tcp_friends, is added:
0: disable friends and use the stock data path.
1: enable friends and bypass the stack data path, the default.
Note, when friends is enabled any loopback interpose, e.g. tcpdump,
will only see the TCP/IP packets during connection establishment and
finish, all data bypasses the stack and instead is delivered to the
destination socket directly.
Testing on a Westmere 3.2 GHz CPU based system, netperf results for
a single connection show increased TCP_STREAM throughput, increased
TCP_RR and TCP_CRR transaction rate for most message sizes vs baseline
and comparable to AF_UNIX.
TCP_RR:
netperf Baseline AF_UNIX Friends
-r N,N Trans./S Trans./S Trans./S
64 120415 255529 212% 279107 232% 109%
1K 112217 242684 216% 268292 239% 111%
8K 79352 184050 232% 196160 247% 107%
32K 40156 66678 166% 65389 163% 98%
64K 24876 44071 177% 36450 147% 83%
128K 13805 22745 165% 17408 126% 77%
256K 8325 11811 142% 10190 122% 86%
512K 4859 6268 129% 5683 117% 91%
1M 2610 3234 124% 3152 121% 97%
16M 88 128 145% 128 145% 100%
TCP_CRR:
netperf Baseline AF_UNIX Friends
-r N Trans./S Trans./S Trans./S
64 32943 - 44720 136%
1K 32172 - 43759 136%
8K 27991 - 39313 140%
32K 19316 - 25297 131%
64K 12801 - 17915 140%
128K 3710* - 6996 *
256K 4* - 6166 *
512K 4* - 4186 *
1M 2* - 2838 *
16M 49* - 131 *
TCP_STREAM:
netperf Baseline AF_UNIX Friends
-m/-M N Mbits/S Mbits/S Mbits/S
64 2399 1064 44% 1646 69% 155%
1K 14412 15310 106% 15554 108% 102%
8K 27468 58198 212% 52320 190% 90%
32K 37382 67252 180% 64611 173% 96%
64K 40949 64505 158% 66874 163% 104%
128K 38149 54670 143% 59852 157% 109%
256K 39660 53474 135% 57464 145% 107%
512K 40975 53506 131% 58050 142% 108%
1M 40541 54017 133% 57193 141% 106%
16M 27539 38515 140% 35270 128% 92%
Note, "-" denotes test not supported for transport.
Note, "*" denotes test results reported without statistical confidence.
Testing with multiple netperf instances:
N copies of: netperf -l 100 -t TCP_STREAM
netperf -l 100 -t STREAM_STREAM -- -s 51882 -m 16384 -M 87380
netperf -l 100 -t TCP_STREAM
Baseline AF_UNIX Friends
N Mbits/S %CPU Mbits/S %CPU Mbits/S %CPU
1 27799 167 52715 196 52958 202
2 59777 291 111137 388 111116 402
10 102822 1674 149151 1997 154970 1896
20 79894 2224 146425 2392 152906 2388
100 75611 2225 80926 2399 92491 2399
200 79623 2230 125498 2400 110742 2400
1000 86717 2236 108066 2400 111225 2400
Signed-off-by: Bruce Curtis <brutus@google.com>
---
include/linux/skbuff.h | 2 +
include/net/request_sock.h | 1 +
include/net/sock.h | 2 +
include/net/tcp.h | 48 +++
net/core/skbuff.c | 1 +
net/ipv4/Makefile | 2 +-
net/ipv4/inet_connection_sock.c | 7 +-
net/ipv4/sysctl_net_ipv4.c | 10 +
net/ipv4/tcp.c | 27 +-
net/ipv4/tcp_friend.c | 841 +++++++++++++++++++++++++++++++++++++++
net/ipv4/tcp_input.c | 9 +-
net/ipv4/tcp_ipv4.c | 1 +
net/ipv4/tcp_minisocks.c | 5 +
net/ipv4/tcp_output.c | 17 +-
net/ipv6/tcp_ipv6.c | 1 +
15 files changed, 962 insertions(+), 12 deletions(-)
create mode 100644 net/ipv4/tcp_friend.c
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 6a6b352..2777e0d 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -319,6 +319,7 @@ typedef unsigned char *sk_buff_data_t;
* @cb: Control buffer. Free for use by every layer. Put private vars here
* @_skb_refdst: destination entry (with norefcount bit)
* @sp: the security path, used for xfrm
+ * @friend: loopback friend socket
* @len: Length of actual data
* @data_len: Data length
* @mac_len: Length of link layer header
@@ -391,6 +392,7 @@ struct sk_buff {
#ifdef CONFIG_XFRM
struct sec_path *sp;
#endif
+ struct sock *friend;
unsigned int len,
data_len;
__u16 mac_len,
diff --git a/include/net/request_sock.h b/include/net/request_sock.h
index 4c0766e..2c74420 100644
--- a/include/net/request_sock.h
+++ b/include/net/request_sock.h
@@ -63,6 +63,7 @@ struct request_sock {
unsigned long expires;
const struct request_sock_ops *rsk_ops;
struct sock *sk;
+ struct sock *friend;
u32 secid;
u32 peer_secid;
};
diff --git a/include/net/sock.h b/include/net/sock.h
index 5ac682f..2dd0179 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -218,6 +218,7 @@ struct sock_common {
* @sk_rxhash: flow hash received from netif layer
* @sk_filter: socket filtering instructions
* @sk_protinfo: private area, net family specific, when not using slab
+ * @sk_friend: private area, net family specific, when have a friend
* @sk_timer: sock cleanup timer
* @sk_stamp: time stamp of last packet received
* @sk_socket: Identd and reporting IO signals
@@ -326,6 +327,7 @@ struct sock {
long sk_rcvtimeo;
long sk_sndtimeo;
void *sk_protinfo;
+ void *sk_friend;
struct timer_list sk_timer;
ktime_t sk_stamp;
struct socket *sk_socket;
diff --git a/include/net/tcp.h b/include/net/tcp.h
index e147f42..2549025 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -1558,6 +1558,54 @@ static inline struct tcp_extend_values *tcp_xv(struct request_values *rvp)
return (struct tcp_extend_values *)rvp;
}
+/*
+ * For TCP a struct sock sk_friend member has 1 of 5 values:
+ *
+ * 1) NULL on initialization, no friend
+ * 2) dummy address &tcp_friend_CONNECTING on connect() return before accept()
+ * 3) a valid struct tcp_friend address once a friend has been made.
+ * 4) dummy address &tcp_friend_EARLYCLOSE on close() of connect()ed before
+ * accept()
+ * 5) dummy address &tcp_friend_CLOSED on close() to denote no longer a friend,
+ * this is used during connection teardown to skip TCP_TIME_WAIT
+ */
+extern unsigned tcp_friend_connecting;
+extern unsigned tcp_friend_earlyclose;
+extern unsigned tcp_friend_closed;
+
+#define tcp_friend_CONNECTING ((void *)&tcp_friend_connecting)
+#define tcp_friend_EARLYCLOSE ((void *)&tcp_friend_earlyclose)
+#define tcp_friend_CLOSED ((void *)&tcp_friend_closed)
+
+static inline int tcp_had_friend(struct sock *sk)
+{
+ if (sk->sk_friend == tcp_friend_CLOSED ||
+ sk->sk_friend == tcp_friend_EARLYCLOSE)
+ return 1;
+ return 0;
+}
+
+static inline int tcp_has_friend(struct sock *sk)
+{
+ if (sk->sk_friend && !tcp_had_friend(sk))
+ return 1;
+ return 0;
+}
+
+#define tcp_sk_friend(__sk) ((struct tcp_friend *)(__sk)->sk_friend)
+
+extern int tcp_friend_sendmsg(struct kiocb *iocb, struct sock *sk,
+ struct msghdr *msg, size_t size, long *timeop);
+extern int tcp_friend_recvmsg(struct kiocb *iocb, struct sock *sk,
+ struct msghdr *msg, size_t len, int nonblock,
+ int flags);
+extern void tcp_friend_connect(struct sock *sk, struct sock *other);
+extern void tcp_friend_shutdown(struct sock *sk, int how);
+extern void tcp_friend_close(struct sock *sk);
+
+extern void tcp_v4_init(void);
+extern void tcp_init(void);
+
extern void tcp_v4_init(void);
extern void tcp_init(void);
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index ca4db40..2fc779d 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -545,6 +545,7 @@ static void __copy_skb_header(struct sk_buff *new, const struct sk_buff *old)
#ifdef CONFIG_XFRM
new->sp = secpath_get(old->sp);
#endif
+ new->friend = old->friend;
memcpy(new->cb, old->cb, sizeof(old->cb));
new->csum = old->csum;
new->local_df = old->local_df;
diff --git a/net/ipv4/Makefile b/net/ipv4/Makefile
index f2dc69c..919264d 100644
--- a/net/ipv4/Makefile
+++ b/net/ipv4/Makefile
@@ -7,7 +7,7 @@ obj-y := route.o inetpeer.o protocol.o \
ip_output.o ip_sockglue.o inet_hashtables.o \
inet_timewait_sock.o inet_connection_sock.o \
tcp.o tcp_input.o tcp_output.o tcp_timer.o tcp_ipv4.o \
- tcp_minisocks.o tcp_cong.o \
+ tcp_minisocks.o tcp_cong.o tcp_friend.o \
datagram.o raw.o udp.o udplite.o \
arp.o icmp.o devinet.o af_inet.o igmp.o \
fib_frontend.o fib_semantics.o fib_trie.o \
diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c
index c14d88a..e65e905 100644
--- a/net/ipv4/inet_connection_sock.c
+++ b/net/ipv4/inet_connection_sock.c
@@ -466,9 +466,9 @@ void inet_csk_reqsk_queue_hash_add(struct sock *sk, struct request_sock *req,
}
EXPORT_SYMBOL_GPL(inet_csk_reqsk_queue_hash_add);
-/* Only thing we need from tcp.h */
+/* Only things we need from tcp.h */
extern int sysctl_tcp_synack_retries;
-
+extern void tcp_friend_connect(struct sock *sk, struct sock *other);
/* Decide when to expire the request and when to resend SYN-ACK */
static inline void syn_ack_recalc(struct request_sock *req, const int thresh,
@@ -596,6 +596,9 @@ struct sock *inet_csk_clone(struct sock *sk, const struct request_sock *req,
if (newsk != NULL) {
struct inet_connection_sock *newicsk = inet_csk(newsk);
+ if (req->friend)
+ tcp_friend_connect(newsk, req->friend);
+
newsk->sk_state = TCP_SYN_RECV;
newicsk->icsk_bind_hash = NULL;
diff --git a/net/ipv4/sysctl_net_ipv4.c b/net/ipv4/sysctl_net_ipv4.c
index 69fd720..c90cbce 100644
--- a/net/ipv4/sysctl_net_ipv4.c
+++ b/net/ipv4/sysctl_net_ipv4.c
@@ -35,6 +35,9 @@ static int ip_ttl_max = 255;
static int ip_ping_group_range_min[] = { 0, 0 };
static int ip_ping_group_range_max[] = { GID_T_MAX, GID_T_MAX };
+/* Loopback bypass */
+int sysctl_tcp_friends = 1;
+
/* Update system visible IP port range */
static void set_local_port_range(int range[2])
{
@@ -721,6 +724,13 @@ static struct ctl_table ipv4_net_table[] = {
.mode = 0644,
.proc_handler = ipv4_ping_group_range,
},
+ {
+ .procname = "tcp_friends",
+ .data = &sysctl_tcp_friends,
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec
+ },
{ }
};
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 34f5db1..9caa2dd 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -935,6 +935,16 @@ int tcp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg,
/* This should be in poll */
clear_bit(SOCK_ASYNC_NOSPACE, &sk->sk_socket->flags);
+ err = -EPIPE;
+ if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN))
+ goto out_err;
+
+ if (tcp_has_friend(sk)) {
+ err = tcp_friend_sendmsg(iocb, sk, msg, size, &timeo);
+ release_sock(sk);
+ return err;
+ }
+
mss_now = tcp_send_mss(sk, &size_goal, flags);
/* Ok commence sending. */
@@ -942,10 +952,6 @@ int tcp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg,
iov = msg->msg_iov;
copied = 0;
- err = -EPIPE;
- if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN))
- goto out_err;
-
sg = sk->sk_route_caps & NETIF_F_SG;
while (--iovlen >= 0) {
@@ -1427,6 +1433,12 @@ int tcp_recvmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg,
if (flags & MSG_OOB)
goto recv_urg;
+ if (tcp_has_friend(sk)) {
+ err = tcp_friend_recvmsg(iocb, sk, msg, len, nonblock, flags);
+ release_sock(sk);
+ return err;
+ }
+
seq = &tp->copied_seq;
if (flags & MSG_PEEK) {
peek_seq = tp->copied_seq;
@@ -1855,6 +1867,9 @@ static int tcp_close_state(struct sock *sk)
void tcp_shutdown(struct sock *sk, int how)
{
+ if (tcp_has_friend(sk))
+ tcp_friend_shutdown(sk, how);
+
/* We need to grab some memory, and put together a FIN,
* and then put it into the queue to be sent.
* Tim MacKenzie(tym@dibbler.cs.monash.edu.au) 4 Dec '92.
@@ -1880,8 +1895,12 @@ void tcp_close(struct sock *sk, long timeout)
int state;
lock_sock(sk);
+
sk->sk_shutdown = SHUTDOWN_MASK;
+ if (tcp_has_friend(sk))
+ tcp_friend_close(sk);
+
if (sk->sk_state == TCP_LISTEN) {
tcp_set_state(sk, TCP_CLOSE);
diff --git a/net/ipv4/tcp_friend.c b/net/ipv4/tcp_friend.c
new file mode 100644
index 0000000..617cc59
--- /dev/null
+++ b/net/ipv4/tcp_friend.c
@@ -0,0 +1,841 @@
+/* net/ipv4/tcp_friend.c
+ *
+ * TCP/IP loopback socket pair stack bypass, based on an idea by, and
+ * rough patch from, David Miller <davem@davemloft.net> called "friends"
+ * but with code for a dedicated data path for maximum performance.
+ *
+ * Authors: Bruce "Brutus" Curtis, <brutus@google.com>
+ *
+ * Copyright (C) 2011 Google Incorporated
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/types.h>
+#include <linux/fcntl.h>
+#include <linux/poll.h>
+#include <linux/init.h>
+#include <linux/fs.h>
+#include <linux/skbuff.h>
+#include <linux/scatterlist.h>
+#include <linux/splice.h>
+#include <linux/net.h>
+#include <linux/socket.h>
+#include <linux/random.h>
+#include <linux/bootmem.h>
+#include <linux/highmem.h>
+#include <linux/swap.h>
+#include <linux/cache.h>
+#include <linux/err.h>
+#include <linux/crypto.h>
+#include <linux/time.h>
+#include <linux/slab.h>
+#include <linux/uaccess.h>
+
+#include <net/icmp.h>
+#include <net/tcp.h>
+#include <net/xfrm.h>
+#include <net/ip.h>
+#include <net/netdma.h>
+#include <net/sock.h>
+
+#include <asm/ioctls.h>
+
+/*
+ * Dummy struct tcp_friend stubs, see "include/net/tcp.h" for details.
+ */
+unsigned tcp_friend_connecting;
+unsigned tcp_friend_earlyclose;
+unsigned tcp_friend_closed;
+
+/**
+ * enum tcp_friend_mode - friend sendmsg() -> recvmsg() mode.
+ * @DATA_BYPASS: in stack data bypass
+ * @DATA_HIWAT: filled sk_buff, is waiting / will wait
+ * @SHUTDOWN: other_sk SEND_SHUTDOWN
+ */
+enum tcp_friend_mode {
+ DATA_BYPASS,
+ DATA_HIWAT,
+ SHUTDOWN
+};
+
+/**
+ * struct tcp_friend - sendmsg() -> recvmsg() state, one for each friend.
+ * @other_sk: other sock bypassed to
+ * @other_tf: other sock's struct tcp_friend
+ * @mode: mode of sendmsg() -> recvmsg()
+ * @send_tail: last sendmsg() tail fill message size
+ * @send_pend: have sendmsg() -> recvmsg() data pending
+ * @have_rspace: have recv space for sendmsg() -> recvmsg()
+ * @using_seq: one shared by both friends *use_seq value
+ * @use_seq: use full TCP sequence state
+ * @ref: count of pointers to
+ * @lock: spinlock for exclusive access to
+ */
+struct tcp_friend {
+ struct sock *other_sk;
+ struct tcp_friend *other_tf;
+ enum tcp_friend_mode mode;
+ int send_tail;
+ int send_pend;
+ int have_rspace;
+ atomic_t using_seq;
+ atomic_t *use_seq;
+ int ref;
+ spinlock_t lock;
+};
+
+/*
+ * Called when sk_friend == CONNECTING to handle connect()/{send,recv}msg()
+ * race with accept(), wait for accept() to finish.
+ */
+static int tcp_friend_wait_connect(struct sock *sk, long *timeo_p)
+{
+ int done;
+
+ DEFINE_WAIT(wait);
+
+ /* Wait for friends to be made */
+ do {
+ if (!*timeo_p)
+ return -EAGAIN;
+ if (signal_pending(current))
+ return sock_intr_errno(*timeo_p);
+
+ prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
+ done = sk_wait_event(sk, timeo_p,
+ (tcp_sk_friend(sk) != tcp_friend_CONNECTING));
+ finish_wait(sk_sleep(sk), &wait);
+ } while (!done);
+
+ if (tcp_had_friend(sk)) {
+ /* While waiting, closed */
+ return -EPIPE;
+ }
+
+ return 0;
+}
+
+static inline void tcp_friend_have_rspace(struct tcp_friend *tf, int true)
+{
+ struct sock *osk = tf->other_sk;
+
+ if (true) {
+ if (!tf->have_rspace) {
+ tf->have_rspace = 1;
+ /* Ready for send(), rm back-pressure */
+ osk->sk_wmem_queued -= osk->sk_sndbuf;
+ }
+ } else {
+ if (tf->have_rspace) {
+ tf->have_rspace = 0;
+ /* No more send() please, back-pressure */
+ osk->sk_wmem_queued += osk->sk_sndbuf;
+ }
+ }
+}
+
+static void tcp_friend_space_wait(struct sock *sk, spinlock_t *lock,
+ long *timeo_p)
+{
+ DEFINE_WAIT(wait);
+
+ prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
+ set_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
+
+ spin_unlock(lock);
+ release_sock(sk);
+ *timeo_p = schedule_timeout(*timeo_p);
+ lock_sock(sk);
+ spin_lock(lock);
+ /* sk_write_space() clears SOCK_NOSPACE */
+
+ finish_wait(sk_sleep(sk), &wait);
+}
+
+static inline void tcp_friend_send_seq(struct sock *osk,
+ struct tcp_friend *otf,
+ int len)
+{
+ struct tcp_sock *otp = tcp_sk(osk);
+
+ if (!atomic_read(otf->use_seq)) {
+ otp->rcv_nxt += len;
+ } else {
+ local_bh_disable();
+
+ bh_lock_sock(osk);
+ otp->rcv_nxt += len;
+ otp->rcv_wup += len;
+ bh_unlock_sock(osk);
+
+ osk = otf->other_sk;
+ otp = tcp_sk(osk);
+ bh_lock_sock(osk);
+ otp->snd_nxt += len;
+ otp->write_seq += len;
+ otp->pushed_seq += len;
+ otp->snd_una += len;
+ otp->snd_up += len;
+ bh_unlock_sock(osk);
+
+ local_bh_enable();
+ }
+}
+
+/*
+ * tcp_friend_sendmsg() - friends interpose on tcp_sendmsg().
+ */
+int tcp_friend_sendmsg(struct kiocb *iocb, struct sock *sk,
+ struct msghdr *msg, size_t size, long *timeo_p)
+{
+ struct tcp_friend *tf = tcp_sk_friend(sk);
+ struct tcp_friend *otf;
+ struct sock *osk;
+ int len;
+ int chunk;
+ int istail;
+ int usetail;
+ int sk_buff;
+ struct sk_buff *skb = NULL;
+ int sent = 0;
+ int err = 0;
+
+ if (tf == tcp_friend_CONNECTING) {
+ err = tcp_friend_wait_connect(sk, timeo_p);
+ if (err)
+ goto ret_err;
+ tf = tcp_sk_friend(sk);
+ }
+ otf = tf->other_tf;
+ osk = tf->other_sk;
+ sk_buff = sk->sk_sndbuf + osk->sk_rcvbuf;
+
+ /* Fit at least 2 (truesize) chunks in an empty sk_buff */
+ chunk = sk_buff >> 1;
+ len = SKB_DATA_ALIGN(chunk);
+ chunk -= len - chunk;
+ chunk -= sizeof(struct skb_shared_info);
+ len = SKB_MAX_ORDER(sizeof(struct skb_shared_info), 2);
+ if (chunk > len)
+ chunk = len;
+ chunk -= sizeof(struct sk_buff);
+
+ /* For message sizes < 1/2 of a chunk use tail fill */
+ if (size < (chunk >> 1))
+ usetail = 1;
+ else
+ usetail = 0;
+
+ spin_lock(&otf->lock);
+ otf->send_pend = size;
+ while (size) {
+ if (osk->sk_shutdown & RCV_SHUTDOWN) {
+ sk->sk_err = ECONNRESET;
+ break;
+ }
+
+ if (usetail) {
+ /*
+ * Do tail fill, if last skb has enough tailroom use
+ * it, else set alloc len to chunk then as long as a
+ * a recvmsg() is pending subsequent sendmsg() calls
+ * can simply tail fill it.
+ */
+ skb = skb_peek_tail(&osk->sk_receive_queue);
+ if (skb) {
+ if (skb_tailroom(skb) >= size) {
+ otf->send_tail = size;
+ istail = 1;
+ len = size;
+ } else {
+ skb = NULL;
+ istail = 0;
+ len = chunk;
+ }
+ } else {
+ istail = 0;
+ len = chunk;
+ }
+ } else {
+ /* Allocate at most one chunk at a time */
+ otf->send_tail = 0;
+ skb = NULL;
+ istail = 0;
+ len = min_t(int, size, chunk);
+ }
+
+ if (!skb) {
+ if (otf->mode == DATA_HIWAT) {
+ if ((sk->sk_shutdown & SEND_SHUTDOWN) ||
+ sk->sk_err) {
+ err = -EPIPE;
+ goto out;
+ }
+ if (!(*timeo_p)) {
+ err = -EAGAIN;
+ goto out;
+ }
+
+ if (signal_pending(current))
+ goto out_sig;
+
+ tcp_friend_space_wait(sk, &otf->lock, timeo_p);
+ continue;
+ }
+ spin_unlock(&otf->lock);
+
+ skb = alloc_skb(len, sk->sk_allocation);
+ if (!skb) {
+ err = -ENOBUFS;
+ spin_lock(&otf->lock);
+ goto out;
+ }
+ skb->friend = sk;
+
+ if (usetail && len > size) {
+ /* For tail fill, alloc len > messages size */
+ len = size;
+ }
+ }
+
+ err = memcpy_fromiovec(skb_put(skb, len), msg->msg_iov, len);
+
+ if (!istail)
+ spin_lock(&otf->lock);
+
+ if (err) {
+ if (istail)
+ skb_trim(skb, skb->len - len);
+ else
+ __kfree_skb(skb);
+ goto out;
+ }
+
+ if (osk->sk_shutdown & RCV_SHUTDOWN) {
+ if (!istail)
+ __kfree_skb(skb);
+ err = -EPIPE;
+ goto out;
+ }
+
+ if (!istail) {
+ int used;
+
+ if (!sk_rmem_schedule(osk, skb->truesize)) {
+ __kfree_skb(skb);
+ atomic_inc(&osk->sk_drops);
+ err = -ENOBUFS;
+ goto out;
+ }
+ skb_set_owner_r(skb, osk);
+ __skb_queue_tail(&osk->sk_receive_queue, skb);
+
+ /* Data ready if used > 75% of sk_buff */
+ used = atomic_read(&osk->sk_rmem_alloc);
+ if (used > ((sk_buff >> 1) + (sk_buff >> 2))) {
+ if (used >= sk_buff)
+ otf->mode = DATA_HIWAT;
+
+ tcp_friend_have_rspace(otf, 0);
+ if (size)
+ osk->sk_data_ready(osk, 0);
+ }
+ }
+
+ tcp_friend_send_seq(osk, otf, len);
+ sent += len;
+ size -= len;
+ }
+
+ if (skb && (msg->msg_flags & MSG_OOB)) {
+ /*
+ * Out-of-Order-Byte message so move last byte of last skb
+ * to TCP's urgent data. Note, in the case of SOCK_URGINLINE
+ * our recvmsg() handles reading of, else tcp_recvmsg() will.
+ */
+ struct tcp_sock *otp = tcp_sk(osk);
+ u8 tmp;
+
+ otp->urg_seq = otp->rcv_nxt - 1;
+ if (skb_copy_bits(skb, skb->len - 1, &tmp, 1))
+ BUG();
+ __skb_trim(skb, skb->len - 1);
+ otp->urg_data = TCP_URG_VALID | tmp;
+
+ sk_send_sigurg(osk);
+ }
+out:
+ otf->send_pend = 0;
+ osk->sk_data_ready(osk, 0);
+ spin_unlock(&otf->lock);
+ if (sent || !err)
+ return sent;
+ret_err:
+ err = sk_stream_error(sk, msg->msg_flags, err);
+ return err;
+
+out_sig:
+ err = sock_intr_errno(*timeo_p);
+ goto out;
+}
+
+static void tcp_friend_data_wait(struct sock *sk, spinlock_t *lock,
+ long *timeo_p)
+{
+ DEFINE_WAIT(wait);
+
+ prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
+ set_bit(SOCK_ASYNC_WAITDATA, &sk->sk_socket->flags);
+
+ spin_unlock(lock);
+ release_sock(sk);
+ *timeo_p = schedule_timeout(*timeo_p);
+ lock_sock(sk);
+ spin_lock(lock);
+
+ clear_bit(SOCK_ASYNC_WAITDATA, &sk->sk_socket->flags);
+ finish_wait(sk_sleep(sk), &wait);
+}
+
+static int tcp_friend_urg_out(struct sock *sk, struct msghdr *msg, int flags)
+{
+ struct tcp_friend *tf = tcp_sk_friend(sk);
+ struct tcp_sock *tp = tcp_sk(sk);
+ int copied;
+
+ if (sock_flag(sk, SOCK_URGINLINE)) {
+ if (!(flags & MSG_TRUNC)) {
+ u8 urg_c = tp->urg_data;
+
+ spin_unlock(&tf->lock);
+ if (memcpy_toiovec(msg->msg_iov, &urg_c, 1))
+ return 0;
+ spin_lock(&tf->lock);
+ }
+ copied = 1;
+ } else
+ copied = -1;
+
+ if (!(flags & MSG_PEEK))
+ tp->urg_data = 0;
+
+ return copied;
+}
+
+/*
+ * tcp_friend_recvmsg() - friends interpose on tcp_recvmsg().
+ */
+int tcp_friend_recvmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg,
+ size_t size, int nonblock, int flags)
+{
+ struct tcp_friend *tf = tcp_sk_friend(sk);
+ struct tcp_sock *tp = tcp_sk(sk);
+ struct sock *osk;
+ struct sk_buff *skb;
+ int len;
+ int target;
+ int urg_offset = -1;
+ int copied = 0;
+ int err = 0;
+ long timeo = sock_rcvtimeo(sk, nonblock);
+ u32 *seq;
+ u32 peek_seq;
+ int peek_copied;
+
+ if (tf == tcp_friend_CONNECTING) {
+ err = tcp_friend_wait_connect(sk, &timeo);
+ if (err)
+ return sk_stream_error(sk, msg->msg_flags, err);
+ tf = tcp_sk_friend(sk);
+ }
+ osk = tf->other_sk;
+ target = sock_rcvlowat(sk, flags & MSG_WAITALL, size);
+
+ seq = &tp->copied_seq;
+ if (flags & MSG_PEEK) {
+ peek_seq = *seq;
+ seq = &peek_seq;
+ peek_copied = 0;
+ }
+
+ spin_lock(&tf->lock);
+ skb = skb_peek(&sk->sk_receive_queue);
+ while (size && urg_offset != 0) {
+ if (skb && !skb->friend) {
+ /* Got a FIN via the stack from the other */
+ BUG_ON(skb->len);
+ BUG_ON(!tcp_hdr(skb)->fin);
+ atomic_dec(tf->use_seq);
+ tp->copied_seq++;
+ __skb_unlink(skb, &sk->sk_receive_queue);
+ kfree_skb(skb);
+ break;
+ }
+
+ /* If urgent data calc urgent data offset */
+ if (tp->urg_data)
+ urg_offset = tp->urg_seq - *seq;
+
+ /* No skb or empty tail skb (for sender tail fill)? */
+ if (!skb || (skb_queue_is_last(&sk->sk_receive_queue, skb) &&
+ !skb->len)) {
+ /* No sender active and have enough data? */
+ if (!tf->send_pend && copied >= target)
+ break;
+ if (sock_flag(sk, SOCK_DONE))
+ break;
+
+ err = sock_error(sk);
+ if (err || (osk->sk_shutdown & SEND_SHUTDOWN) ||
+ (sk->sk_shutdown & RCV_SHUTDOWN))
+ break;
+
+ if (!timeo) {
+ err = -EAGAIN;
+ break;
+ }
+ tcp_friend_data_wait(sk, &tf->lock, &timeo);
+
+ if (signal_pending(current)) {
+ err = sock_intr_errno(timeo);
+ break;
+ }
+ skb = skb_peek(&sk->sk_receive_queue);
+ continue;
+ }
+
+ len = min_t(unsigned int, skb->len, size);
+
+ if (!len)
+ goto skip;
+
+ if (urg_offset == 0) {
+ /* At urgent byte, consume and optionally copyout */
+ len = tcp_friend_urg_out(sk, msg, flags);
+ if (len == 0) {
+ /* On error, returns with spin_unlock() !!! */
+ err = -EFAULT;
+ goto out;
+ }
+ if (len > 0) {
+ copied++;
+ size--;
+ }
+ (*seq)++;
+ urg_offset = -1;
+ continue;
+ } else if (urg_offset != -1 && urg_offset < len) {
+ /* Have an urgent byte in skb, copyout up-to */
+ len = urg_offset;
+ }
+
+ if (!(flags & MSG_TRUNC)) {
+ spin_unlock(&tf->lock);
+ if (memcpy_toiovec(msg->msg_iov, skb->data, len)) {
+ err = -EFAULT;
+ goto out;
+ }
+ spin_lock(&tf->lock);
+ }
+ *seq += len;
+ copied += len;
+ size -= len;
+ if (urg_offset != -1)
+ urg_offset -= len;
+
+ if (!(flags & MSG_PEEK)) {
+ skb_pull(skb, len);
+ /*
+ * If skb is empty and, no more to recv or last send
+ * message not tail filled or not last skb on queue
+ * or not likely enough tail room in skb for next
+ * send message tail fill, then unlink and free, if
+ * more to recv get next skb (if any), and last if
+ * queued data size <= 1/2 of sk_buff have send space.
+ *
+ * Else, more to copyout or leave the empty skb on
+ * queue for the next sendmsg() to use for tail fill.
+ */
+ if (!skb->len && (!size || !tf->send_tail ||
+ !skb_queue_is_last(&sk->sk_receive_queue, skb) ||
+ skb_tailroom(skb) < tf->send_tail)) {
+skip:
+ __skb_unlink(skb, &sk->sk_receive_queue);
+ __kfree_skb(skb);
+
+ if (size)
+ skb = skb_peek(&sk->sk_receive_queue);
+
+ /* Write space if used <= 25% of sk_buff */
+ if (!(osk->sk_shutdown & SEND_SHUTDOWN) &&
+ atomic_read(&sk->sk_rmem_alloc) <=
+ ((osk->sk_sndbuf + sk->sk_rcvbuf) >> 2)) {
+
+ if (tf->mode == DATA_HIWAT)
+ tf->mode = DATA_BYPASS;
+
+ tcp_friend_have_rspace(tf, 1);
+ osk->sk_write_space(osk);
+ }
+ }
+ } else {
+ if ((copied - peek_copied) < skb->len)
+ continue;
+ if (skb_queue_is_last(&sk->sk_receive_queue, skb))
+ break;
+ peek_copied = copied;
+ skb = skb_queue_next(&sk->sk_receive_queue, skb);
+ }
+ }
+ /*
+ * If empty skb on tail of queue (see tail fill comment above) then
+ * need to clean it up before returning so unlink and free it.
+ */
+ skb = skb_peek_tail(&sk->sk_receive_queue);
+ if (skb && !skb->len) {
+ __skb_unlink(skb, &sk->sk_receive_queue);
+ __kfree_skb(skb);
+ }
+ spin_unlock(&tf->lock);
+
+out:
+ return copied ? : err;
+}
+
+static inline void tcp_friend_release(struct tcp_friend *tf)
+{
+ spin_lock(&tf->lock);
+ if (tf->ref == 1) {
+ sock_put(tf->other_sk);
+ kfree(tf);
+ } else {
+ tf->ref--;
+ spin_unlock(&tf->lock);
+ }
+}
+
+static inline struct tcp_friend *tcp_friend_hold(struct sock *sk,
+ struct sock *osk,
+ struct tcp_friend *otf)
+{
+ struct tcp_friend *tf;
+ u64 was;
+
+ tf = kmalloc(sizeof(*tf), GFP_ATOMIC);
+ if (!tf)
+ return NULL;
+
+ tf->mode = DATA_BYPASS;
+ sock_hold(osk);
+ tf->other_sk = osk;
+ if (otf) {
+ otf->ref++;
+ tf->other_tf = otf;
+ tf->use_seq = &otf->using_seq;
+ tf->ref = 2;
+ otf->other_tf = tf;
+ } else {
+ tf->other_tf = NULL;
+ tf->use_seq = &tf->using_seq;
+ atomic_set(tf->use_seq, 0);
+ tf->ref = 1;
+ }
+ tf->send_tail = 0;
+ tf->send_pend = 0;
+ tf->have_rspace = 1;
+ spin_lock_init(&tf->lock);
+
+ was = atomic_long_xchg(&sk->sk_friend, (u64)tf);
+ if (was == (u64)tcp_friend_CONNECTING) {
+ /* sk_friend was CONNECTING may be in wait_connect() */
+ bh_lock_sock(sk);
+ sk->sk_state_change(sk);
+ bh_unlock_sock(sk);
+ } else if (was == (u64)tcp_friend_EARLYCLOSE) {
+ /* Close race, closed already, abort */
+ tf->ref--;
+ tcp_friend_release(tf);
+ otf->ref--;
+ tf = NULL;
+ }
+
+ return tf;
+}
+
+/*
+ * tcp_friend_connect() - called in one of two ways; 1) called from the
+ * listen()er context with a new *sk to be returned as the accept() socket
+ * and *req socket from connect(), 2) called from the connect()ing context
+ * with it's *sk socket and a NULL *req.
+ *
+ * For 1) put a friend_hold() on *sk and *req to make friends.
+ *
+ * For 2) if called before 1) attempt to set sk_friend to CONNECTING if NULL
+ * as a sendmsg()/recvmsg() barrier.
+ */
+void tcp_friend_connect(struct sock *sk, struct sock *req)
+{
+ struct tcp_friend *tf;
+ struct tcp_friend *otf;
+
+ if (!req) {
+ /* Case 2), atomically swap CONNECTING if NULL */
+ atomic_long_cmpxchg(&sk->sk_friend, 0,
+ (u64)tcp_friend_CONNECTING);
+ return;
+ }
+
+ tf = tcp_friend_hold(sk, req, NULL);
+ if (!tf)
+ return;
+
+ otf = tcp_friend_hold(req, sk, tf);
+ if (!otf) {
+ sk->sk_friend = NULL;
+ req->sk_friend = NULL;
+ tcp_friend_release(tf);
+ return;
+ }
+}
+
+static void tcp_friend_use_seq(struct sock *sk, struct sock *osk)
+{
+ struct tcp_sock *tp = tcp_sk(sk);
+ struct tcp_sock *otp = tcp_sk(osk);
+
+ /*
+ * Note, during data bypass mode only rcv_nxt and copied_seq
+ * values are maintained, now sk <> osk control segments need
+ * to flow so need to reinitialize all sk/osk values.
+ *
+ * Note, any recvmsg(osk) drain and sendmsg(osk) -> recvmsg(sk)
+ * data will maintain all TCP sequence values.
+ */
+
+ /* Our sequence values */
+ tp->rcv_wup = tp->rcv_nxt;
+
+ tp->snd_nxt = otp->rcv_nxt;
+ tp->write_seq = otp->rcv_nxt;
+ tp->pushed_seq = otp->rcv_nxt;
+ tp->snd_una = otp->rcv_nxt;
+ tp->snd_up = otp->rcv_nxt;
+
+ /* Other's sequence values */
+ otp->rcv_wup = otp->rcv_nxt;
+
+ otp->snd_nxt = tp->rcv_nxt;
+ otp->write_seq = tp->rcv_nxt;
+ otp->pushed_seq = tp->rcv_nxt;
+ otp->snd_una = tp->rcv_nxt;
+ otp->snd_up = tp->rcv_nxt;
+}
+
+
+/*
+ * On close()/shutdown() called when sk_friend == CONNECTING, need to
+ * handle possile connect()/close() race with accept(), try to atomically
+ * mark sk_friend with EARLYCLOSE, if successful return NULL as accept()
+ * never completed, else accept() completed so return tf.
+ */
+static struct tcp_friend *tcp_friend_close_connect(struct sock *sk)
+{
+ struct tcp_friend *tf;
+ tf = (struct tcp_friend *)atomic_long_cmpxchg(&sk->sk_friend,
+ (u64)tcp_friend_CONNECTING, (u64)tcp_friend_EARLYCLOSE);
+ if (tf == tcp_friend_CONNECTING)
+ return NULL;
+
+ return tf;
+}
+
+/*
+ * tcp_friend_shutdown() - friends shim on tcp_shutdown().
+ */
+void tcp_friend_shutdown(struct sock *sk, int how)
+{
+ struct tcp_friend *tf = tcp_sk_friend(sk);
+ struct tcp_friend *otf;
+ struct sock *osk;
+
+ if (tf == tcp_friend_CONNECTING) {
+ tf = tcp_friend_close_connect(sk);
+ if (!tf)
+ return;
+ }
+ otf = tf->other_tf;
+ osk = tf->other_sk;
+
+ if (how & RCV_SHUTDOWN) {
+ struct sk_buff *skb, *tmp;
+
+ spin_lock(&tf->lock);
+ skb_queue_walk_safe(&sk->sk_receive_queue, skb, tmp) {
+ if (skb->friend) {
+ __skb_unlink(skb, &sk->sk_receive_queue);
+ __kfree_skb(skb);
+ }
+ }
+ if (tf->mode == DATA_HIWAT)
+ tf->mode = DATA_BYPASS;
+ osk->sk_write_space(osk);
+ spin_unlock(&tf->lock);
+ }
+
+ if (how & SEND_SHUTDOWN) {
+ spin_lock(&otf->lock);
+ if (otf->mode != SHUTDOWN) {
+ otf->mode = SHUTDOWN;
+ if (atomic_inc_return(tf->use_seq) == 1) {
+ /*
+ * 1st friend to shutdown so switch to
+ * updating full TCP sequence state.
+ */
+ spin_lock(&tf->lock);
+ tcp_friend_use_seq(sk, osk);
+ spin_unlock(&tf->lock);
+ }
+ }
+
+ tcp_friend_have_rspace(otf, 1);
+ osk->sk_data_ready(osk, 0);
+ spin_unlock(&otf->lock);
+ }
+}
+
+/*
+ * tcp_friend_close() - friends shim on tcp_close().
+ */
+void tcp_friend_close(struct sock *sk)
+{
+ struct tcp_friend *tf = tcp_sk_friend(sk);
+ struct tcp_friend *otf;
+
+ if (tf == tcp_friend_CONNECTING) {
+ tf = tcp_friend_close_connect(sk);
+ if (!tf)
+ return;
+ }
+ otf = tf->other_tf;
+
+ tcp_friend_shutdown(sk, SHUTDOWN_MASK);
+
+ /* Release other's ref on us */
+ tcp_friend_release(otf);
+
+ /* Relase our ref on other */
+ tcp_friend_release(tf);
+
+ sk->sk_friend = tcp_friend_CLOSED;
+}
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 52b5c2d..7918056 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -4686,7 +4686,7 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list,
restart:
end_of_skbs = true;
skb_queue_walk_from_safe(list, skb, n) {
- if (skb == tail)
+ if (skb == tail || skb->friend)
break;
/* No new bits? It is possible on ofo queue. */
if (!before(start, TCP_SKB_CB(skb)->end_seq)) {
@@ -5641,6 +5641,9 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
}
}
+ if (skb->friend)
+ tcp_friend_connect(sk, NULL);
+
smp_mb();
tcp_set_state(sk, TCP_ESTABLISHED);
@@ -5673,9 +5676,9 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
sk_wake_async(sk, SOCK_WAKE_IO, POLL_OUT);
}
- if (sk->sk_write_pending ||
+ if (!skb->friend && (sk->sk_write_pending ||
icsk->icsk_accept_queue.rskq_defer_accept ||
- icsk->icsk_ack.pingpong) {
+ icsk->icsk_ack.pingpong)) {
/* Save one ACK. Data will be ready after
* several ticks, if write_pending is set.
*
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index 0ea10ee..f2430d8 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -1291,6 +1291,7 @@ int tcp_v4_conn_request(struct sock *sk, struct sk_buff *skb)
tcp_rsk(req)->af_specific = &tcp_request_sock_ipv4_ops;
#endif
+ req->friend = skb->friend;
tcp_clear_options(&tmp_opt);
tmp_opt.mss_clamp = TCP_MSS_DEFAULT;
tmp_opt.user_mss = tp->rx_opt.user_mss;
diff --git a/net/ipv4/tcp_minisocks.c b/net/ipv4/tcp_minisocks.c
index 85a2fbe..5d57255 100644
--- a/net/ipv4/tcp_minisocks.c
+++ b/net/ipv4/tcp_minisocks.c
@@ -318,6 +318,11 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)
const struct tcp_sock *tp = tcp_sk(sk);
int recycle_ok = 0;
+ if (tcp_had_friend(sk)) {
+ tcp_done(sk);
+ return;
+ }
+
if (tcp_death_row.sysctl_tw_recycle && tp->rx_opt.ts_recent_stamp)
recycle_ok = tcp_remember_stamp(sk);
diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index 980b98f..0e2a68e 100644
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -782,6 +782,8 @@ static unsigned tcp_established_options(struct sock *sk, struct sk_buff *skb,
return size;
}
+extern int sysctl_tcp_friends;
+
/* This routine actually transmits TCP packets queued in by
* tcp_do_sendmsg(). This is used by both the initial
* transmission and possible later retransmissions.
@@ -828,9 +830,14 @@ static int tcp_transmit_skb(struct sock *sk, struct sk_buff *skb, int clone_it,
tcb = TCP_SKB_CB(skb);
memset(&opts, 0, sizeof(opts));
- if (unlikely(tcb->tcp_flags & TCPHDR_SYN))
+ if (unlikely(tcb->tcp_flags & TCPHDR_SYN)) {
+ if (sysctl_tcp_friends) {
+ /* Only try to make friends if enabled */
+ skb->friend = sk;
+ }
+
tcp_options_size = tcp_syn_options(sk, skb, &opts, &md5);
- else
+ } else
tcp_options_size = tcp_established_options(sk, skb, &opts,
&md5);
tcp_header_size = tcp_options_size + sizeof(struct tcphdr);
@@ -2468,6 +2475,12 @@ struct sk_buff *tcp_make_synack(struct sock *sk, struct dst_entry *dst,
}
memset(&opts, 0, sizeof(opts));
+
+ if (sysctl_tcp_friends) {
+ /* Only try to make friends if enabled */
+ skb->friend = sk;
+ }
+
#ifdef CONFIG_SYN_COOKIES
if (unlikely(req->cookie_ts))
TCP_SKB_CB(skb)->when = cookie_init_timestamp(req);
diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
index c8683fc..44ede0a 100644
--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c
@@ -1194,6 +1194,7 @@ static int tcp_v6_conn_request(struct sock *sk, struct sk_buff *skb)
tcp_rsk(req)->af_specific = &tcp_request_sock_ipv6_ops;
#endif
+ req->friend = skb->friend;
tcp_clear_options(&tmp_opt);
tmp_opt.mss_clamp = IPV6_MIN_MTU - sizeof(struct tcphdr) - sizeof(struct ipv6hdr);
tmp_opt.user_mss = tp->rx_opt.user_mss;
--
1.7.3.1
^ permalink raw reply related
* Re: [PATCH RFC] virtio_net: fix refill related races
From: Michael S. Tsirkin @ 2011-12-20 19:45 UTC (permalink / raw)
To: Tejun Heo; +Cc: Amit Shah, netdev, linux-kernel, virtualization
In-Reply-To: <20111220193154.GG10752@google.com>
On Tue, Dec 20, 2011 at 11:31:54AM -0800, Tejun Heo wrote:
> On Tue, Dec 20, 2011 at 09:30:55PM +0200, Michael S. Tsirkin wrote:
> > Hmm, in that case it looks like a nasty race could get
> > triggered, with try_fill_recv run on multiple CPUs in parallel,
> > corrupting the linked list within the vq.
> >
> > Using the mutex as my patch did will fix that naturally, as well.
>
> Don't know the code but just use nrt wq. There's even a system one
> called system_nrq_wq.
>
> Thanks.
We can, but we need the mutex for other reasons, anyway.
> --
> tejun
^ permalink raw reply
* Re: [PATCH 1/1] tcp: Replace constants with #define macros
From: Vijay Subramanian @ 2011-12-20 20:02 UTC (permalink / raw)
To: David Laight
Cc: netdev, David S. Miller, Eric Dumazet, ilpo.jarvinen,
H.K. Jerry Chu
In-Reply-To: <AE90C24D6B3A694183C094C60CF0A2F6D8AF1C@saturn3.aculab.com>
Thanks a lot for reviewing the patch. Please see responses below.
On 20 December 2011 02:27, David Laight <David.Laight@aculab.com> wrote:
>
>> +/*These are used to set the sack_ok field in struct
>> tcp_options_received */
>> +#define SACK_SEEN (1 << 0) /*1 = peer is SACK capable, */
>> +#define FACK_ENABLED (1 << 1) /*1 = FACK is enabled locally*/
>> +#define DSACK_SEEN (1 << 2) /*1 = DSACK was received from peer*/
>> +
>
> Since that is a fairly public header, some namespace protection
> might be sensible.
I assume you are talking about collision with userspace. These
definitions are already protected by #ifdef __KERNEL__.
That should be enough to protect against collisions with userspace I think.
>
>> - tcp_opt->sack_ok = (options >> 4) & 0x1;
>> + tcp_opt->sack_ok = (options >> 4) & SACK_SEEN;
>
> Looks to me like that 0x1 isn't SAC_SEEK! So this is now misleading.
Can you please elaborate? Are you saying this is wrong or that it
could be done better?
Thanks for your time!
Vijay
^ permalink raw reply
* Re: [PATCH 1/1] tcp: Replace constants with #define macros
From: David Miller @ 2011-12-20 20:18 UTC (permalink / raw)
To: subramanian.vijay
Cc: David.Laight, netdev, eric.dumazet, ilpo.jarvinen, hkchu
In-Reply-To: <CAGK4HS9LuvgOoY+cSXq-Vjbx7x5gsQ03wj6Tb1UDsO=89Wkrpw@mail.gmail.com>
From: Vijay Subramanian <subramanian.vijay@gmail.com>
Date: Tue, 20 Dec 2011 12:02:52 -0800
> I assume you are talking about collision with userspace. These
> definitions are already protected by #ifdef __KERNEL__.
> That should be enough to protect against collisions with userspace I think.
Within the kernel he means, please put a TCP_* prefix onto these
values.
>>
>>> - tcp_opt->sack_ok = (options >> 4) & 0x1;
>>> + tcp_opt->sack_ok = (options >> 4) & SACK_SEEN;
>>
>> Looks to me like that 0x1 isn't SAC_SEEK! So this is now misleading.
>
> Can you please elaborate? Are you saying this is wrong or that it
> could be done better?
I think he's saying it could be done better. Probably something
like:
tcp_opt->sack_ok = (options & (1 << 4)) ? SACK_SEEN : 0;
^ permalink raw reply
* Re: [RFC][PATCH] net-tcp: TCP/IP stack bypass for loopback connections.
From: David Miller @ 2011-12-20 20:19 UTC (permalink / raw)
To: brutus; +Cc: netdev
In-Reply-To: <20111220193614.0733F160AA8@brutus.mtv.corp.google.com>
From: Bruce "Brutus" Curtis <brutus@google.com>
Date: Tue, 22 Nov 2011 16:09:33 -0800
> TCP/IP loopback socket pair stack bypass, based on an idea by, and
> rough upstream patch from, David Miller <davem@davemloft.net> called
> "friends", the data structure modifcations and connection scheme are
> reused with new dedicated code for the data path.
Thanks for doing this work.
It's very disappoing that the whole of the data sending system
calls need to be duplicated into this new tcp_friends.c file.
My entire hope was that we could integrate the logic into the
existing sendmsg/recvmsg implementations.
Please reimplement things this way. With so much code duplication
I really cannot justify putting this feature in as-is.
^ permalink raw reply
* Re: [PATCH net-next] igb: add basic runtime PM support
From: Alexander Duyck @ 2011-12-20 20:42 UTC (permalink / raw)
To: Yan, Zheng
Cc: e1000-devel, netdev@vger.kernel.org, bruce.w.allan,
jesse.brandeburg, rjw, john.ronciak, davem@davemloft.net
In-Reply-To: <4EEEC688.2010601@intel.com>
I am curious on if you have done any testing on an actual igb device
with this code enabled? My main concern is that the PHY is integrated
into these parts, thus putting the MAC into D3 will result in behaviour
changes in the PHY. Specifically in the case of D3 the PHY on many of
the igb parts will go into a low power link up mode, and may exclude a
1Gb/s link. If this occurs I would expect to see a number of link
issues on the device.
From what I can tell it looks like if you were to perform any ethtool
operations while the link is down and this code is enabled it would
likely result in multiple errors. I am not sure what would happen if
you were to run an "ethtool -t" test while the device was in a powered
down state.
Also have you done any tests to see what the actual power savings of
this patch might be? I'm concerned there may be none due to the fact
that putting a single function of an igb device into D3 will not result
in any power savings. In order to see any power savings you would need
to put all of the functions on a given MAC into D3hot before the device
will actually switch the PCIe bus to L1. On e1000e parts this made
sense since most parts are only single port, however since igb supports
dual and quad port adapters I don't know how effective this would be.
Thanks,
Alex
On 12/18/2011 09:07 PM, Yan, Zheng wrote:
> Use the runtime power management framework to add basic runtime PM support
> to the igb driver. Namely, make the driver suspend the device when the link
> is off and set it up for generating a wakeup event after the link has been
> detected again. Also make the driver suspend the device when the interface
> is being shut down. This feature is disabled by default.
>
> Based on e1000e's runtime PM code.
>
> Signed-off-by: Zheng Yan<zheng.z.yan@intel.com>
> ---
> drivers/net/ethernet/intel/igb/igb_main.c | 151 ++++++++++++++++++++++++-----
> 1 files changed, 126 insertions(+), 25 deletions(-)
>
> diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c
> index 89d576c..aac529b 100644
> --- a/drivers/net/ethernet/intel/igb/igb_main.c
> +++ b/drivers/net/ethernet/intel/igb/igb_main.c
> @@ -53,6 +53,7 @@
> #include<linux/if_ether.h>
> #include<linux/aer.h>
> #include<linux/prefetch.h>
> +#include<linux/pm_runtime.h>
> #ifdef CONFIG_IGB_DCA
> #include<linux/dca.h>
> #endif
> @@ -172,8 +173,18 @@ static int igb_check_vf_assignment(struct igb_adapter *adapter);
> #endif
>
> #ifdef CONFIG_PM
> -static int igb_suspend(struct pci_dev *, pm_message_t);
> -static int igb_resume(struct pci_dev *);
> +static int igb_suspend(struct device *);
> +static int igb_resume(struct device *);
> +#ifdef CONFIG_PM_RUNTIME
> +static int igb_runtime_suspend(struct device *dev);
> +static int igb_runtime_resume(struct device *dev);
> +static int igb_runtime_idle(struct device *dev);
> +#endif
> +static const struct dev_pm_ops igb_pm_ops = {
> + SET_SYSTEM_SLEEP_PM_OPS(igb_suspend, igb_resume)
> + SET_RUNTIME_PM_OPS(igb_runtime_suspend, igb_runtime_resume,
> + igb_runtime_idle)
> +};
> #endif
> static void igb_shutdown(struct pci_dev *);
> #ifdef CONFIG_IGB_DCA
> @@ -214,9 +225,7 @@ static struct pci_driver igb_driver = {
> .probe = igb_probe,
> .remove = __devexit_p(igb_remove),
> #ifdef CONFIG_PM
> - /* Power Management Hooks */
> - .suspend = igb_suspend,
> - .resume = igb_resume,
> + .driver.pm =&igb_pm_ops,
> #endif
> .shutdown = igb_shutdown,
> .err_handler =&igb_err_handler
> @@ -2111,6 +2120,8 @@ static int __devinit igb_probe(struct pci_dev *pdev,
> default:
> break;
> }
> +
> + pm_runtime_put_noidle(&pdev->dev);
> return 0;
>
> err_register:
> @@ -2150,6 +2161,8 @@ static void __devexit igb_remove(struct pci_dev *pdev)
> struct igb_adapter *adapter = netdev_priv(netdev);
> struct e1000_hw *hw =&adapter->hw;
>
> + pm_runtime_get_noresume(&pdev->dev);
> +
> /*
> * The watchdog timer may be rescheduled, so explicitly
> * disable watchdog from being rescheduled.
> @@ -2472,16 +2485,23 @@ static int __devinit igb_sw_init(struct igb_adapter *adapter)
> * handler is registered with the OS, the watchdog timer is started,
> * and the stack is notified that the interface is ready.
> **/
> -static int igb_open(struct net_device *netdev)
> +static int __igb_open(struct net_device *netdev, bool resuming)
> {
> struct igb_adapter *adapter = netdev_priv(netdev);
> struct e1000_hw *hw =&adapter->hw;
> + struct pci_dev *pdev = adapter->pdev;
> int err;
> int i;
>
> - /* disallow open during test */
> - if (test_bit(__IGB_TESTING,&adapter->state))
> - return -EBUSY;
> + if (!resuming) {
> + /* disallow open during test */
> + if (test_bit(__IGB_TESTING,&adapter->state))
> + return -EBUSY;
> + pm_runtime_get_sync(&pdev->dev);
> + } else if (test_bit(__IGB_TESTING,&adapter->state)) {
> + while (test_bit(__IGB_TESTING,&adapter->state))
> + msleep(100);
> + }
>
> netif_carrier_off(netdev);
>
> @@ -2527,6 +2547,9 @@ static int igb_open(struct net_device *netdev)
>
> netif_tx_start_all_queues(netdev);
>
> + if (!resuming)
> + pm_runtime_put(&pdev->dev);
> +
> /* start the watchdog. */
> hw->mac.get_link_status = 1;
> schedule_work(&adapter->watchdog_task);
> @@ -2541,10 +2564,17 @@ err_setup_rx:
> igb_free_all_tx_resources(adapter);
> err_setup_tx:
> igb_reset(adapter);
> + if (!resuming)
> + pm_runtime_put(&pdev->dev);
>
> return err;
> }
>
> +static int igb_open(struct net_device *netdev)
> +{
> + return __igb_open(netdev, false);
> +}
> +
> /**
> * igb_close - Disables a network interface
> * @netdev: network interface device structure
> @@ -2556,21 +2586,32 @@ err_setup_tx:
> * needs to be disabled. A global MAC reset is issued to stop the
> * hardware, and all transmit and receive resources are freed.
> **/
> -static int igb_close(struct net_device *netdev)
> +static int __igb_close(struct net_device *netdev, bool suspending)
> {
> struct igb_adapter *adapter = netdev_priv(netdev);
> + struct pci_dev *pdev = adapter->pdev;
>
> WARN_ON(test_bit(__IGB_RESETTING,&adapter->state));
> - igb_down(adapter);
>
> + if (!suspending)
> + pm_runtime_get_sync(&pdev->dev);
> +
> + igb_down(adapter);
> igb_free_irq(adapter);
>
> igb_free_all_tx_resources(adapter);
> igb_free_all_rx_resources(adapter);
>
> + if (!suspending)
> + pm_runtime_put_sync(&pdev->dev);
> return 0;
> }
>
> +static int igb_close(struct net_device *netdev)
> +{
> + return __igb_close(netdev, false);
> +}
> +
> /**
> * igb_setup_tx_resources - allocate Tx resources (Descriptors)
> * @tx_ring: tx descriptor ring (for a specific queue) to setup
> @@ -3630,6 +3671,9 @@ static void igb_watchdog_task(struct work_struct *work)
>
> link = igb_has_link(adapter);
> if (link) {
> + /* Cancel scheduled suspend requests. */
> + pm_runtime_resume(netdev->dev.parent);
> +
> if (!netif_carrier_ok(netdev)) {
> u32 ctrl;
> hw->mac.ops.get_speed_and_duplex(hw,
> @@ -3701,6 +3745,9 @@ static void igb_watchdog_task(struct work_struct *work)
> if (!test_bit(__IGB_DOWN,&adapter->state))
> mod_timer(&adapter->phy_info_timer,
> round_jiffies(jiffies + 2 * HZ));
> +
> + pm_schedule_suspend(netdev->dev.parent,
> + MSEC_PER_SEC * 5);
> }
> }
>
> @@ -6583,21 +6630,24 @@ err_inval:
> return -EINVAL;
> }
>
> -static int __igb_shutdown(struct pci_dev *pdev, bool *enable_wake)
> +static int __igb_shutdown(struct pci_dev *pdev, bool *enable_wake,
> + bool runtime)
> {
> struct net_device *netdev = pci_get_drvdata(pdev);
> struct igb_adapter *adapter = netdev_priv(netdev);
> struct e1000_hw *hw =&adapter->hw;
> u32 ctrl, rctl, status;
> - u32 wufc = adapter->wol;
> + u32 wufc = runtime ? E1000_WUFC_LNKC : adapter->wol;
> #ifdef CONFIG_PM
> int retval = 0;
> #endif
>
> - netif_device_detach(netdev);
> -
> - if (netif_running(netdev))
> - igb_close(netdev);
> + if (netif_running(netdev)) {
> + netif_device_detach(netdev);
> + __igb_close(netdev, true);
> + } else {
> + wufc&= ~E1000_WUFC_LNKC;
> + }
>
> igb_clear_interrupt_scheme(adapter);
>
> @@ -6656,12 +6706,13 @@ static int __igb_shutdown(struct pci_dev *pdev, bool *enable_wake)
> }
>
> #ifdef CONFIG_PM
> -static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
> +static int igb_suspend(struct device *dev)
> {
> int retval;
> bool wake;
> + struct pci_dev *pdev = to_pci_dev(dev);
>
> - retval = __igb_shutdown(pdev,&wake);
> + retval = __igb_shutdown(pdev,&wake, 0);
> if (retval)
> return retval;
>
> @@ -6675,8 +6726,9 @@ static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
> return 0;
> }
>
> -static int igb_resume(struct pci_dev *pdev)
> +static int igb_resume(struct device *dev)
> {
> + struct pci_dev *pdev = to_pci_dev(dev);
> struct net_device *netdev = pci_get_drvdata(pdev);
> struct igb_adapter *adapter = netdev_priv(netdev);
> struct e1000_hw *hw =&adapter->hw;
> @@ -6697,7 +6749,18 @@ static int igb_resume(struct pci_dev *pdev)
> pci_enable_wake(pdev, PCI_D3hot, 0);
> pci_enable_wake(pdev, PCI_D3cold, 0);
>
> - if (igb_init_interrupt_scheme(adapter)) {
> + if (!rtnl_is_locked()) {
> + /*
> + * shut up ASSERT_RTNL() warning in
> + * netif_set_real_num_tx/rx_queues.
> + */
> + rtnl_lock();
> + err = igb_init_interrupt_scheme(adapter);
> + rtnl_unlock();
> + } else {
> + err = igb_init_interrupt_scheme(adapter);
> + }
> + if (err) {
> dev_err(&pdev->dev, "Unable to allocate memory for queues\n");
> return -ENOMEM;
> }
> @@ -6710,23 +6773,61 @@ static int igb_resume(struct pci_dev *pdev)
>
> wr32(E1000_WUS, ~0);
>
> - if (netif_running(netdev)) {
> - err = igb_open(netdev);
> + if (netdev->flags& IFF_UP) {
> + err = __igb_open(netdev, true);
> if (err)
> return err;
> + netif_device_attach(netdev);
> }
>
> - netif_device_attach(netdev);
> + return 0;
> +}
> +
> +#ifdef CONFIG_PM_RUNTIME
> +static int igb_runtime_idle(struct device *dev)
> +{
> + struct pci_dev *pdev = to_pci_dev(dev);
> + struct net_device *netdev = pci_get_drvdata(pdev);
> + struct igb_adapter *adapter = netdev_priv(netdev);
> +
> + if (!netif_running(netdev) || !igb_has_link(adapter))
> + pm_schedule_suspend(dev, MSEC_PER_SEC * 5);
> +
> + return -EBUSY;
> +}
> +
> +static int igb_runtime_suspend(struct device *dev)
> +{
> + struct pci_dev *pdev = to_pci_dev(dev);
> + int retval;
> + bool wake;
> +
> + retval = __igb_shutdown(pdev,&wake, 1);
> + if (retval)
> + return retval;
> +
> + if (wake) {
> + pci_prepare_to_sleep(pdev);
> + } else {
> + pci_wake_from_d3(pdev, false);
> + pci_set_power_state(pdev, PCI_D3hot);
> + }
>
> return 0;
> }
> +
> +static int igb_runtime_resume(struct device *dev)
> +{
> + return igb_resume(dev);
> +}
> +#endif /* CONFIG_PM_RUNTIME */
> #endif
>
> static void igb_shutdown(struct pci_dev *pdev)
> {
> bool wake;
>
> - __igb_shutdown(pdev,&wake);
> + __igb_shutdown(pdev,&wake, 0);
>
> if (system_state == SYSTEM_POWER_OFF) {
> pci_wake_from_d3(pdev, wake);
------------------------------------------------------------------------------
Write once. Port to many.
Get the SDK and tools to simplify cross-platform app development. Create
new or port existing apps to sell to consumers worldwide. Explore the
Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join
http://p.sf.net/sfu/intel-appdev
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
^ permalink raw reply
* Re: [B.A.T.M.A.N.] [PATCH] batman-adv: checkpatch cleanup - remove lines longer than 80 chars
From: Ben Hutchings @ 2011-12-20 20:56 UTC (permalink / raw)
To: Marek Lindner; +Cc: b.a.t.m.a.n, David Miller, netdev
In-Reply-To: <201112210315.02171.lindner_marek@yahoo.de>
On Wed, 2011-12-21 at 03:15 +0800, Marek Lindner wrote:
> On Wednesday, December 21, 2011 03:02:09 David Miller wrote:
> > From: Marek Lindner <lindner_marek@yahoo.de>
> > Date: Wed, 21 Dec 2011 02:59:41 +0800
> >
> > > On Wednesday, December 21, 2011 02:32:30 David Miller wrote:
> > >> From: Marek Lindner <lindner_marek@yahoo.de>
> > >> Date: Tue, 20 Dec 2011 18:38:24 +0800
> > >>
> > >> > The long line was introduced with b26e478f8f.
> > >> >
> > >> > Signed-off-by: Marek Lindner <lindner_marek@yahoo.de>
> > >>
> > >> I'm not applying this.
> > >>
> > >> I tell people to make sure arguments line up correctly to the
> > >> openning parenthesis on the previous line, and that is what is
> > >> happening here.
> > >
> > > I am not against lining up the arguments but what about checkpatch ? Are
> > > we going to ignore the complaints or is this line limit about to be
> > > changed ?
> >
> > Refactor the code so that both constraints can be satisfied.
> >
> > Is this so hard to understand?
>
> Well, my crystal ball did not unveil that a checkpatch complaint you silently
> introduced would mean somebody else has to refactor the code. Guess I have to
> get a replacement unit.
No-one has to refactor the code. I think David's point is that he's
more concerned about proper alignment than having every line under 80
characters. If you want to satisfy the latter constraint, you still
have to satisfy the former.
Ben.
--
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
^ permalink raw reply
* [PATCH] virtio_net: fix refill related races
From: Michael S. Tsirkin @ 2011-12-20 21:01 UTC (permalink / raw)
Cc: Michael S. Tsirkin, netdev, linux-kernel, virtualization,
Amit Shah, Tejun Heo
Fix theoretical races related to refill work:
1. After napi is disabled by ndo_stop, refill work
can run and re-enable it.
2. Refill can get scheduled on many cpus in parallel.
if this happens it will corrupt the vq linked list
as there's no locking
3. refill work is cancelled after unregister netdev
For small bufs this means it can alloc an skb
for a device which is unregistered which is
not a good idea.
As a solution, add flag to track napi state
and toggle it on start/stop
check on refill. Add a mutex to pretect the flag,
as well as serial refills.
Move refill cancellation to after unregister.
refill work structure and new fields aren't used
on data path, so put them together near the end of
struct virtnet_info.
TODO: consider using system_nrq_wq as suggested by
Tejun Heo. Probably be a good idea but not a must for
correctness.
Lightly tested.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
drivers/net/virtio_net.c | 34 +++++++++++++++++++++++++++-------
1 files changed, 27 insertions(+), 7 deletions(-)
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 6ee8410..452f186 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -21,6 +21,7 @@
#include <linux/etherdevice.h>
#include <linux/ethtool.h>
#include <linux/module.h>
+#include <linux/mutex.h>
#include <linux/virtio.h>
#include <linux/virtio_net.h>
#include <linux/scatterlist.h>
@@ -68,15 +69,21 @@ struct virtnet_info {
/* Active statistics */
struct virtnet_stats __percpu *stats;
- /* Work struct for refilling if we run low on memory. */
- struct delayed_work refill;
-
/* Chain pages by the private ptr. */
struct page *pages;
/* fragments + linear part + virtio header */
struct scatterlist rx_sg[MAX_SKB_FRAGS + 2];
struct scatterlist tx_sg[MAX_SKB_FRAGS + 2];
+
+ /* Work struct for refilling if we run low on memory. */
+ struct delayed_work refill;
+
+ /* Whether napi is enabled, protected by a refill_lock. */
+ bool napi_enable;
+
+ /* Lock to protect refill and napi enable/disable operations. */
+ struct mutex refill_lock;
};
struct skb_vnet_hdr {
@@ -494,14 +501,20 @@ static void refill_work(struct work_struct *work)
bool still_empty;
vi = container_of(work, struct virtnet_info, refill.work);
- napi_disable(&vi->napi);
+
+ mutex_lock(&vi->refill_lock);
+ if (vi->napi_enable)
+ napi_disable(&vi->napi);
still_empty = !try_fill_recv(vi, GFP_KERNEL);
- virtnet_napi_enable(vi);
+ if (vi->napi_enable)
+ virtnet_napi_enable(vi);
/* In theory, this can happen: if we don't get any buffers in
* we will *never* try to fill again. */
if (still_empty)
schedule_delayed_work(&vi->refill, HZ/2);
+
+ mutex_unlock(&vi->refill_lock);
}
static int virtnet_poll(struct napi_struct *napi, int budget)
@@ -719,7 +732,10 @@ static int virtnet_open(struct net_device *dev)
{
struct virtnet_info *vi = netdev_priv(dev);
+ mutex_lock(&vi->refill_lock);
+ vi->napi_enable = true;
virtnet_napi_enable(vi);
+ mutex_unlock(&vi->refill_lock);
return 0;
}
@@ -772,7 +788,10 @@ static int virtnet_close(struct net_device *dev)
{
struct virtnet_info *vi = netdev_priv(dev);
+ mutex_lock(&vi->refill_lock);
+ vi->napi_enable = false;
napi_disable(&vi->napi);
+ mutex_unlock(&vi->refill_lock);
return 0;
}
@@ -1021,6 +1040,7 @@ static int virtnet_probe(struct virtio_device *vdev)
if (vi->stats == NULL)
goto free;
+ mutex_init(&vi->refill_lock);
INIT_DELAYED_WORK(&vi->refill, refill_work);
sg_init_table(vi->rx_sg, ARRAY_SIZE(vi->rx_sg));
sg_init_table(vi->tx_sg, ARRAY_SIZE(vi->tx_sg));
@@ -1081,8 +1101,8 @@ static int virtnet_probe(struct virtio_device *vdev)
return 0;
unregister:
- unregister_netdev(dev);
cancel_delayed_work_sync(&vi->refill);
+ unregister_netdev(dev);
free_vqs:
vdev->config->del_vqs(vdev);
free_stats:
@@ -1121,9 +1141,9 @@ static void __devexit virtnet_remove(struct virtio_device *vdev)
/* Stop all the virtqueues. */
vdev->config->reset(vdev);
+ cancel_delayed_work_sync(&vi->refill);
unregister_netdev(vi->dev);
- cancel_delayed_work_sync(&vi->refill);
/* Free unused buffers in both send and recv, if any. */
free_unused_bufs(vi);
--
1.7.5.53.gc233e
^ permalink raw reply related
* RE: twice past the taps, thence out to net?
From: Wyborny, Carolyn @ 2011-12-20 21:21 UTC (permalink / raw)
To: Eric Dumazet, Rick Jones
Cc: Stephen Hemminger, Vijay Subramanian,
tcpdump-workers@lists.tcpdump.org, netdev@vger.kernel.org,
Vick, Matthew, Kirsher, Jeffrey T
In-Reply-To: <1324064689.2621.24.camel@edumazet-laptop>
>-----Original Message-----
>From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org]
>On Behalf Of Eric Dumazet
>Sent: Friday, December 16, 2011 11:45 AM
>To: Rick Jones
>Cc: Stephen Hemminger; Vijay Subramanian; tcpdump-
>workers@lists.tcpdump.org; netdev@vger.kernel.org; Vick, Matthew;
>Kirsher, Jeffrey T
>Subject: Re: twice past the taps, thence out to net?
>
>Le vendredi 16 décembre 2011 à 11:35 -0800, Rick Jones a écrit :
>
>> I would *love* to. All my accessible igb-driven hardware is in an
>> environment locked to the kernels already there :( Not that it makes
>it
>> more possible for me to do it, but I suspect it does not require 30
>> receivers to reproduce the dups with netperf TCP_STREAM. Particularly
>> if the tx queue len is at 256 it may only take 6 or 8. In fact let me
>> try that now...
>>
>> Yep, with just 8 destinations/concurrent TCP_STREAM tests from the one
>> system one can still see the duplicates in the packet trace taken on
>the
>> sender.
>>
>> Perhaps we can trouble the Intel guys to try to reproduce what I've
>seen?
>>
>
>I do have an igb card somewhere (in fact two dual ports), I'll do the
>test myself !
>
>Thanks
>
>
>--
>To unsubscribe from this list: send the line "unsubscribe netdev" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at http://vger.kernel.org/majordomo-info.html
Let me know if I can do anything to assist. Sorry to have overlooked this thread for a bit.
Thanks,
Carolyn
Carolyn Wyborny
Linux Development
LAN Access Division
Intel Corporation
^ permalink raw reply
* [PATCH] net: macb: fix build break with !CONFIG_OF
From: Olof Johansson @ 2011-12-20 21:20 UTC (permalink / raw)
To: Nicolas Ferre
Cc: netdev, Jean-Christophe PLAGNIOL-VILLARD, Jamie Iles,
linux-arm-kernel, Olof Johansson
This fixes the build of the macb driver that was broken by the DT changes:
drivers/net/ethernet/cadence/macb.c:1503: error: implicit declaration of function 'of_match_ptr'
drivers/net/ethernet/cadence/macb.c:1503: error: 'macb_dt_ids' undeclared here (not in a function)
drivers/net/ethernet/cadence/macb.c:1503: error: initializer element is not constant
drivers/net/ethernet/cadence/macb.c:1503: error: (near initialization for 'macb_driver.driver.of_match_table')
Signed-off-by: Olof Johansson <olof@lixom.net>
---
Nicolas, care to ack this so I can apply it on top of the macb DT branch in arm-soc?
drivers/net/ethernet/cadence/macb.c | 1 +
1 files changed, 1 insertions(+), 0 deletions(-)
diff --git a/drivers/net/ethernet/cadence/macb.c b/drivers/net/ethernet/cadence/macb.c
index baf1a0d..f3d5c65 100644
--- a/drivers/net/ethernet/cadence/macb.c
+++ b/drivers/net/ethernet/cadence/macb.c
@@ -23,6 +23,7 @@
#include <linux/platform_data/macb.h>
#include <linux/platform_device.h>
#include <linux/phy.h>
+#include <linux/of.h>
#include <linux/of_device.h>
#include <linux/of_net.h>
--
1.7.5.4
^ permalink raw reply related
* [PATCH iproute2 0/3] netem iproute2 userspace parts
From: Hagen Paul Pfeifer @ 2011-12-20 21:28 UTC (permalink / raw)
To: netdev; +Cc: shemminger, Hagen Paul Pfeifer
Hello Stephen,
the follow serie provides support for netem rate extension. The s32 parser was
already discussed. I extend the man page to describe the extension.
Hagen
Hagen Paul Pfeifer (3):
utils: add s32 parser
tc: netem rate shaping and cell extension
netem: extend man page for rate extension
include/linux/pkt_sched.h | 8 +++++
include/utils.h | 1 +
lib/utils.c | 19 +++++++++++++
man/man8/tc-netem.8 | 66 +++++++++++++++++++++++++++++++++++---------
tc/q_netem.c | 53 +++++++++++++++++++++++++++++++++++-
5 files changed, 132 insertions(+), 15 deletions(-)
--
1.7.7.3
^ permalink raw reply
* [PATCH iproute2 1/3] utils: add s32 parser
From: Hagen Paul Pfeifer @ 2011-12-20 21:28 UTC (permalink / raw)
To: netdev; +Cc: shemminger, Hagen Paul Pfeifer
In-Reply-To: <1324416528-3845-1-git-send-email-hagen@jauu.net>
Signed-off-by: Hagen Paul Pfeifer <hagen@jauu.net>
---
include/utils.h | 1 +
lib/utils.c | 19 +++++++++++++++++++
2 files changed, 20 insertions(+), 0 deletions(-)
diff --git a/include/utils.h b/include/utils.h
index 47f8e07..496db68 100644
--- a/include/utils.h
+++ b/include/utils.h
@@ -85,6 +85,7 @@ extern int get_time_rtt(unsigned *val, const char *arg, int *raw);
#define get_short get_s16
extern int get_u64(__u64 *val, const char *arg, int base);
extern int get_u32(__u32 *val, const char *arg, int base);
+extern int get_s32(__s32 *val, const char *arg, int base);
extern int get_u16(__u16 *val, const char *arg, int base);
extern int get_s16(__s16 *val, const char *arg, int base);
extern int get_u8(__u8 *val, const char *arg, int base);
diff --git a/lib/utils.c b/lib/utils.c
index efaf377..d80f79b 100644
--- a/lib/utils.c
+++ b/lib/utils.c
@@ -25,6 +25,7 @@
#include <linux/pkt_sched.h>
#include <time.h>
#include <sys/time.h>
+#include <errno.h>
#include "utils.h"
@@ -198,6 +199,24 @@ int get_u8(__u8 *val, const char *arg, int base)
return 0;
}
+int get_s32(__s32 *val, const char *arg, int base)
+{
+ long res;
+ char *ptr;
+
+ errno = 0;
+
+ if (!arg || !*arg)
+ return -1;
+ res = strtol(arg, &ptr, base);
+ if (ptr == arg || *ptr ||
+ ((res == LONG_MIN || res == LONG_MAX) && errno == ERANGE) ||
+ res > INT32_MAX || res < INT32_MIN)
+ return -1;
+ *val = res;
+ return 0;
+}
+
int get_s16(__s16 *val, const char *arg, int base)
{
long res;
--
1.7.7.3
^ permalink raw reply related
* [PATCH iproute2 2/3] tc: netem rate shaping and cell extension
From: Hagen Paul Pfeifer @ 2011-12-20 21:28 UTC (permalink / raw)
To: netdev; +Cc: shemminger, Hagen Paul Pfeifer
In-Reply-To: <1324416528-3845-1-git-send-email-hagen@jauu.net>
This patch add rate shaping as well as cell support. The link-rate can be
specified via rate options. Three optional arguments control the cell
knobs: packet-overhead, cell-size, cell-overhead. To ratelimit eth0 root
queue to 5kbit/s, with a 20 byte packet overhead, 100 byte cell size and
a 5 byte per cell overhead:
tc qdisc add dev eth0 root netem rate 5kbit 20 100 5
Signed-off-by: Hagen Paul Pfeifer <hagen@jauu.net>
---
include/linux/pkt_sched.h | 8 ++++++
tc/q_netem.c | 53 ++++++++++++++++++++++++++++++++++++++++++++-
2 files changed, 60 insertions(+), 1 deletions(-)
diff --git a/include/linux/pkt_sched.h b/include/linux/pkt_sched.h
index c533670..eaf4e9e 100644
--- a/include/linux/pkt_sched.h
+++ b/include/linux/pkt_sched.h
@@ -465,6 +465,7 @@ enum {
TCA_NETEM_REORDER,
TCA_NETEM_CORRUPT,
TCA_NETEM_LOSS,
+ TCA_NETEM_RATE,
__TCA_NETEM_MAX,
};
@@ -495,6 +496,13 @@ struct tc_netem_corrupt {
__u32 correlation;
};
+struct tc_netem_rate {
+ __u32 rate; /* byte/s */
+ __s32 packet_overhead;
+ __u32 cell_size;
+ __s32 cell_overhead;
+};
+
enum {
NETEM_LOSS_UNSPEC,
NETEM_LOSS_GI, /* General Intuitive - 4 state model */
diff --git a/tc/q_netem.c b/tc/q_netem.c
index 6dc40bd..1fdfa44 100644
--- a/tc/q_netem.c
+++ b/tc/q_netem.c
@@ -34,7 +34,8 @@ static void explain(void)
" [ drop PERCENT [CORRELATION]] \n" \
" [ corrupt PERCENT [CORRELATION]] \n" \
" [ duplicate PERCENT [CORRELATION]]\n" \
-" [ reorder PRECENT [CORRELATION] [ gap DISTANCE ]]\n");
+" [ reorder PRECENT [CORRELATION] [ gap DISTANCE ]]\n" \
+" [ rate RATE [PACKETOVERHEAD] [CELLSIZE] [CELLOVERHEAD]]\n");
}
static void explain1(const char *arg)
@@ -131,6 +132,7 @@ static int netem_parse_opt(struct qdisc_util *qu, int argc, char **argv,
struct tc_netem_corr cor;
struct tc_netem_reorder reorder;
struct tc_netem_corrupt corrupt;
+ struct tc_netem_rate rate;
__s16 *dist_data = NULL;
int present[__TCA_NETEM_MAX];
@@ -139,6 +141,7 @@ static int netem_parse_opt(struct qdisc_util *qu, int argc, char **argv,
memset(&cor, 0, sizeof(cor));
memset(&reorder, 0, sizeof(reorder));
memset(&corrupt, 0, sizeof(corrupt));
+ memset(&rate, 0, sizeof(rate));
memset(present, 0, sizeof(present));
while (argc > 0) {
@@ -244,6 +247,34 @@ static int netem_parse_opt(struct qdisc_util *qu, int argc, char **argv,
free(dist_data);
return -1;
}
+ } else if (matches(*argv, "rate") == 0) {
+ ++present[TCA_NETEM_RATE];
+ NEXT_ARG();
+ if (get_rate(&rate.rate, *argv)) {
+ explain1("rate");
+ return -1;
+ }
+ if (NEXT_IS_NUMBER()) {
+ NEXT_ARG();
+ if (get_s32(&rate.packet_overhead, *argv, 0)) {
+ explain1("rate");
+ return -1;
+ }
+ }
+ if (NEXT_IS_NUMBER()) {
+ NEXT_ARG();
+ if (get_u32(&rate.cell_size, *argv, 0)) {
+ explain1("rate");
+ return -1;
+ }
+ }
+ if (NEXT_IS_NUMBER()) {
+ NEXT_ARG();
+ if (get_s32(&rate.cell_overhead, *argv, 0)) {
+ explain1("rate");
+ return -1;
+ }
+ }
} else if (strcmp(*argv, "help") == 0) {
explain();
return -1;
@@ -290,6 +321,10 @@ static int netem_parse_opt(struct qdisc_util *qu, int argc, char **argv,
addattr_l(n, 1024, TCA_NETEM_CORRUPT, &corrupt, sizeof(corrupt)) < 0)
return -1;
+ if (present[TCA_NETEM_RATE] &&
+ addattr_l(n, 1024, TCA_NETEM_RATE, &rate, sizeof(rate)) < 0)
+ return -1;
+
if (dist_data) {
if (addattr_l(n, MAX_DIST * sizeof(dist_data[0]),
TCA_NETEM_DELAY_DIST,
@@ -306,6 +341,7 @@ static int netem_print_opt(struct qdisc_util *qu, FILE *f, struct rtattr *opt)
const struct tc_netem_corr *cor = NULL;
const struct tc_netem_reorder *reorder = NULL;
const struct tc_netem_corrupt *corrupt = NULL;
+ const struct tc_netem_rate *rate = NULL;
struct tc_netem_qopt qopt;
int len = RTA_PAYLOAD(opt) - sizeof(qopt);
SPRINT_BUF(b1);
@@ -339,6 +375,11 @@ static int netem_print_opt(struct qdisc_util *qu, FILE *f, struct rtattr *opt)
return -1;
corrupt = RTA_DATA(tb[TCA_NETEM_CORRUPT]);
}
+ if (tb[TCA_NETEM_RATE]) {
+ if (RTA_PAYLOAD(tb[TCA_NETEM_RATE]) < sizeof(*rate))
+ return -1;
+ rate = RTA_DATA(tb[TCA_NETEM_RATE]);
+ }
}
fprintf(f, "limit %d", qopt.limit);
@@ -382,6 +423,16 @@ static int netem_print_opt(struct qdisc_util *qu, FILE *f, struct rtattr *opt)
sprint_percent(corrupt->correlation, b1));
}
+ if (rate && rate->rate) {
+ fprintf(f, " rate %s", sprint_rate(rate->rate, b1));
+ if (rate->packet_overhead)
+ fprintf(f, " packetoverhead %d", rate->packet_overhead);
+ if (rate->cell_size)
+ fprintf(f, " cellsize %u", rate->cell_size);
+ if (rate->cell_overhead)
+ fprintf(f, " celloverhead %d", rate->cell_overhead);
+ }
+
if (qopt.gap)
fprintf(f, " gap %lu", (unsigned long)qopt.gap);
--
1.7.7.3
^ permalink raw reply related
* [PATCH iproute2 3/3] netem: extend man page for rate extension
From: Hagen Paul Pfeifer @ 2011-12-20 21:28 UTC (permalink / raw)
To: netdev; +Cc: shemminger, Hagen Paul Pfeifer
In-Reply-To: <1324416528-3845-1-git-send-email-hagen@jauu.net>
Signed-off-by: Hagen Paul Pfeifer <hagen@jauu.net>
---
man/man8/tc-netem.8 | 66 ++++++++++++++++++++++++++++++++++++++++-----------
1 files changed, 52 insertions(+), 14 deletions(-)
diff --git a/man/man8/tc-netem.8 b/man/man8/tc-netem.8
index c8ed292..ef2a63b 100644
--- a/man/man8/tc-netem.8
+++ b/man/man8/tc-netem.8
@@ -3,90 +3,96 @@
NetEm \- Network Emulator
.SH SYNOPSIS
.B tc qdisc ... dev
-dev
+DEVICE
.B ] add netem [ limit
packets
.B ]
.B tc qdisc ... dev
-dev
+DEVICE
.B ] add netem [ logging
LEVEL ]
.B ]
.B tc qdisc ... dev
-dev
+DEVICE
.B ] add netem [ delay
TIME [ JITTER [CORRELATION]]]
.B ]
.B tc qdisc ... dev
-dev
+DEVICE
.B ] add netem [ distribution
{uniform|normal|pareto|paretonormal} ]
.B ]
.B tc qdisc ... dev
-dev
+DEVICE
.B ] add netem [ drop
PERCENT [CORRELATION]]
.B ]
.B tc qdisc ... dev
-dev
+DEVICE
.B ] add netem [ loss
PERCENT [CORRELATION]]
.B ]
.B tc qdisc ... dev
-dev
+DEVICE
.B ] add netem [ query ] [ loss_GI
ploss [burst_length [density [pisol [good_burst_length]]]]
.B ]
.B tc qdisc ... dev
-dev
+DEVICE
.B ] add netem [ query ] [ loss_4state
[p13 [p31 [p32 [p23 [p14]]]]]
.B ]
.B tc qdisc ... dev
-dev
+DEVICE
.B ] add netem [ loss_gilb_ell
p [r [1-h [1-k]]]
.B ]
.B tc qdisc ... dev
-dev
+DEVICE
.B ] add netem [ loss_gilb_ell_4s
p [r [1-h [1-k]]]
.B ]
.B tc qdisc ... dev
-dev
+DEVICE
.B ] add netem [ loss_pattern
FILENAME [REPETITIONS]]
.B ]
.B tc qdisc ... dev
-dev
+DEVICE
.B ] add netem [ corrupt
PERCENT [CORRELATION]]
.B ]
.B tc qdisc ... dev
-dev
+DEVICE
.B ] add netem [ duplicate
PERCENT [CORRELATION]]
.B ]
.B tc qdisc ... dev
-dev
+DEVICE
.B ] add netem [ reorder
PRECENT [CORRELATION] [ gap DISTANCE ]]
.B ]
+.B tc qdisc ... dev
+DEVICE
+.B ] add netem [ rate
+RATE [PACKETOVERHEAD [CELLSIZE [CELLOVERHEAD]]]]
+.B ]
+
.SH DESCRIPTION
NetEm is an enhancement of the Linux traffic control facilities
that allow to add delay, packet loss, duplication and more other
@@ -240,6 +246,38 @@ other packets are delayed by 10 ms
in this second example 25% of packets are sent immediately (with correlation of
50%) while the other are delayed by 10 ms.
+
+.B rate
+RATE [PACKETOVERHEAD [CELLSIZE [CELLOVERHEAD]]]
+
+delay packets based on packet size and is a replacement for TBF. Rate can be
+specified in common units (e.g. 100kbit). Optional PACKETOVERHEAD (in bytes)
+specify an per packet overhead and can be negative. A positive value can be
+used to simulate additional link layer headers. A negative value can be used to
+artificial strip the Ethernet header (e.g. -14) and/or simulate a link layer
+header compression scheme. The third parameter - an unsigned value - specify
+the cellsize. Cellsize can be used to simulate link layer schemes. ATM for
+example has an payload cellsize of 48 bytes and 5 byte per cell header. If a
+packet is 50 byte then ATM must use two cells: 2 * 48 bytes payload including 2
+* 5 byte header, thus consume 106 byte on the wire. The last optional value
+CELLOVERHEAD can be used to specify per cell overhead - for our ATM example 5.
+CELLOVERHEAD can be negative, but use negative values with caution.
+
+Note that rate throttling is limited by several factors: the kernel clock
+granularity avoid a perfect shaping at a specific level. This will show up in
+an artificial packet compression (bursts). Another influence factor are network
+adapter buffers which can also add artificial delay.
+
+To delay all outgoing packets on devive eth0 with a rate of 5kbit, a per packet
+overhead of 20 byte, a cellsize of 100 byte and a per celloverhead of 5 byte:
+
+tc qdisc add dev eth0 root netem rate 5kbit 20 100 5
+
+And show:
+
+tc qdisc show dev eth0
+
+
.SH LIMITATIONS
The main known limitation of Netem are related to timer granularity, since
Linux is not a real-time operating system; to the choice of Pseudo-Random
--
1.7.7.3
^ permalink raw reply related
* [PATCH 0/5] Respin of routing cache removal
From: David Miller @ 2011-12-20 21:35 UTC (permalink / raw)
To: netdev
This is just a refresh since half of these patches have generated
some rejects over time.
^ permalink raw reply
* [PATCH 1/5] ipv4: Delete routing cache.
From: David Miller @ 2011-12-20 21:35 UTC (permalink / raw)
To: netdev
Signed-off-by: David S. Miller <davem@davemloft.net>
---
include/net/route.h | 1 -
net/ipv4/fib_frontend.c | 5 -
net/ipv4/route.c | 944 ++---------------------------------------------
3 files changed, 27 insertions(+), 923 deletions(-)
diff --git a/include/net/route.h b/include/net/route.h
index 91855d1..ae32c9f 100644
--- a/include/net/route.h
+++ b/include/net/route.h
@@ -114,7 +114,6 @@ extern int ip_rt_init(void);
extern void ip_rt_redirect(__be32 old_gw, __be32 dst, __be32 new_gw,
__be32 src, struct net_device *dev);
extern void rt_cache_flush(struct net *net, int how);
-extern void rt_cache_flush_batch(struct net *net);
extern struct rtable *__ip_route_output_key(struct net *, struct flowi4 *flp);
extern struct rtable *ip_route_output_flow(struct net *, struct flowi4 *flp,
struct sock *sk);
diff --git a/net/ipv4/fib_frontend.c b/net/ipv4/fib_frontend.c
index 92fc5f6..fb15bc6 100644
--- a/net/ipv4/fib_frontend.c
+++ b/net/ipv4/fib_frontend.c
@@ -1022,11 +1022,6 @@ static int fib_netdev_event(struct notifier_block *this, unsigned long event, vo
rt_cache_flush(dev_net(dev), 0);
break;
case NETDEV_UNREGISTER_BATCH:
- /* The batch unregister is only called on the first
- * device in the list of devices being unregistered.
- * Therefore we should not pass dev_net(dev) in here.
- */
- rt_cache_flush_batch(NULL);
break;
}
return NOTIFY_DONE;
diff --git a/net/ipv4/route.c b/net/ipv4/route.c
index f30112f..68e903a 100644
--- a/net/ipv4/route.c
+++ b/net/ipv4/route.c
@@ -129,7 +129,6 @@ static int ip_rt_gc_elasticity __read_mostly = 8;
static int ip_rt_mtu_expires __read_mostly = 10 * 60 * HZ;
static int ip_rt_min_pmtu __read_mostly = 512 + 20 + 20;
static int ip_rt_min_advmss __read_mostly = 256;
-static int rt_chain_length_max __read_mostly = 20;
static int redirect_genid;
/*
@@ -143,7 +142,6 @@ static void ipv4_dst_destroy(struct dst_entry *dst);
static struct dst_entry *ipv4_negative_advice(struct dst_entry *dst);
static void ipv4_link_failure(struct sk_buff *skb);
static void ip_rt_update_pmtu(struct dst_entry *dst, u32 mtu);
-static int rt_garbage_collect(struct dst_ops *ops);
static void ipv4_dst_ifdown(struct dst_entry *dst, struct net_device *dev,
int how)
@@ -190,7 +188,6 @@ static struct neighbour *ipv4_neigh_lookup(const struct dst_entry *dst, const vo
static struct dst_ops ipv4_dst_ops = {
.family = AF_INET,
.protocol = cpu_to_be16(ETH_P_IP),
- .gc = rt_garbage_collect,
.check = ipv4_dst_check,
.default_advmss = ipv4_default_advmss,
.mtu = ipv4_mtu,
@@ -226,184 +223,30 @@ const __u8 ip_tos2prio[16] = {
};
-/*
- * Route cache.
- */
-
-/* The locking scheme is rather straight forward:
- *
- * 1) Read-Copy Update protects the buckets of the central route hash.
- * 2) Only writers remove entries, and they hold the lock
- * as they look at rtable reference counts.
- * 3) Only readers acquire references to rtable entries,
- * they do so with atomic increments and with the
- * lock held.
- */
-
-struct rt_hash_bucket {
- struct rtable __rcu *chain;
-};
-
-#if defined(CONFIG_SMP) || defined(CONFIG_DEBUG_SPINLOCK) || \
- defined(CONFIG_PROVE_LOCKING)
-/*
- * Instead of using one spinlock for each rt_hash_bucket, we use a table of spinlocks
- * The size of this table is a power of two and depends on the number of CPUS.
- * (on lockdep we have a quite big spinlock_t, so keep the size down there)
- */
-#ifdef CONFIG_LOCKDEP
-# define RT_HASH_LOCK_SZ 256
-#else
-# if NR_CPUS >= 32
-# define RT_HASH_LOCK_SZ 4096
-# elif NR_CPUS >= 16
-# define RT_HASH_LOCK_SZ 2048
-# elif NR_CPUS >= 8
-# define RT_HASH_LOCK_SZ 1024
-# elif NR_CPUS >= 4
-# define RT_HASH_LOCK_SZ 512
-# else
-# define RT_HASH_LOCK_SZ 256
-# endif
-#endif
-
-static spinlock_t *rt_hash_locks;
-# define rt_hash_lock_addr(slot) &rt_hash_locks[(slot) & (RT_HASH_LOCK_SZ - 1)]
-
-static __init void rt_hash_lock_init(void)
-{
- int i;
-
- rt_hash_locks = kmalloc(sizeof(spinlock_t) * RT_HASH_LOCK_SZ,
- GFP_KERNEL);
- if (!rt_hash_locks)
- panic("IP: failed to allocate rt_hash_locks\n");
-
- for (i = 0; i < RT_HASH_LOCK_SZ; i++)
- spin_lock_init(&rt_hash_locks[i]);
-}
-#else
-# define rt_hash_lock_addr(slot) NULL
-
-static inline void rt_hash_lock_init(void)
-{
-}
-#endif
-
-static struct rt_hash_bucket *rt_hash_table __read_mostly;
-static unsigned rt_hash_mask __read_mostly;
-static unsigned int rt_hash_log __read_mostly;
-
static DEFINE_PER_CPU(struct rt_cache_stat, rt_cache_stat);
#define RT_CACHE_STAT_INC(field) __this_cpu_inc(rt_cache_stat.field)
-static inline unsigned int rt_hash(__be32 daddr, __be32 saddr, int idx,
- int genid)
-{
- return jhash_3words((__force u32)daddr, (__force u32)saddr,
- idx, genid)
- & rt_hash_mask;
-}
-
static inline int rt_genid(struct net *net)
{
return atomic_read(&net->ipv4.rt_genid);
}
#ifdef CONFIG_PROC_FS
-struct rt_cache_iter_state {
- struct seq_net_private p;
- int bucket;
- int genid;
-};
-
-static struct rtable *rt_cache_get_first(struct seq_file *seq)
-{
- struct rt_cache_iter_state *st = seq->private;
- struct rtable *r = NULL;
-
- for (st->bucket = rt_hash_mask; st->bucket >= 0; --st->bucket) {
- if (!rcu_access_pointer(rt_hash_table[st->bucket].chain))
- continue;
- rcu_read_lock_bh();
- r = rcu_dereference_bh(rt_hash_table[st->bucket].chain);
- while (r) {
- if (dev_net(r->dst.dev) == seq_file_net(seq) &&
- r->rt_genid == st->genid)
- return r;
- r = rcu_dereference_bh(r->dst.rt_next);
- }
- rcu_read_unlock_bh();
- }
- return r;
-}
-
-static struct rtable *__rt_cache_get_next(struct seq_file *seq,
- struct rtable *r)
-{
- struct rt_cache_iter_state *st = seq->private;
-
- r = rcu_dereference_bh(r->dst.rt_next);
- while (!r) {
- rcu_read_unlock_bh();
- do {
- if (--st->bucket < 0)
- return NULL;
- } while (!rcu_access_pointer(rt_hash_table[st->bucket].chain));
- rcu_read_lock_bh();
- r = rcu_dereference_bh(rt_hash_table[st->bucket].chain);
- }
- return r;
-}
-
-static struct rtable *rt_cache_get_next(struct seq_file *seq,
- struct rtable *r)
-{
- struct rt_cache_iter_state *st = seq->private;
- while ((r = __rt_cache_get_next(seq, r)) != NULL) {
- if (dev_net(r->dst.dev) != seq_file_net(seq))
- continue;
- if (r->rt_genid == st->genid)
- break;
- }
- return r;
-}
-
-static struct rtable *rt_cache_get_idx(struct seq_file *seq, loff_t pos)
-{
- struct rtable *r = rt_cache_get_first(seq);
-
- if (r)
- while (pos && (r = rt_cache_get_next(seq, r)))
- --pos;
- return pos ? NULL : r;
-}
-
static void *rt_cache_seq_start(struct seq_file *seq, loff_t *pos)
{
- struct rt_cache_iter_state *st = seq->private;
if (*pos)
- return rt_cache_get_idx(seq, *pos - 1);
- st->genid = rt_genid(seq_file_net(seq));
+ return NULL;
return SEQ_START_TOKEN;
}
static void *rt_cache_seq_next(struct seq_file *seq, void *v, loff_t *pos)
{
- struct rtable *r;
-
- if (v == SEQ_START_TOKEN)
- r = rt_cache_get_first(seq);
- else
- r = rt_cache_get_next(seq, v);
++*pos;
- return r;
+ return NULL;
}
static void rt_cache_seq_stop(struct seq_file *seq, void *v)
{
- if (v && v != SEQ_START_TOKEN)
- rcu_read_unlock_bh();
}
static int rt_cache_seq_show(struct seq_file *seq, void *v)
@@ -413,34 +256,6 @@ static int rt_cache_seq_show(struct seq_file *seq, void *v)
"Iface\tDestination\tGateway \tFlags\t\tRefCnt\tUse\t"
"Metric\tSource\t\tMTU\tWindow\tIRTT\tTOS\tHHRef\t"
"HHUptod\tSpecDst");
- else {
- struct rtable *r = v;
- struct neighbour *n;
- int len, HHUptod;
-
- rcu_read_lock();
- n = dst_get_neighbour_noref(&r->dst);
- HHUptod = (n && (n->nud_state & NUD_CONNECTED)) ? 1 : 0;
- rcu_read_unlock();
-
- seq_printf(seq, "%s\t%08X\t%08X\t%8X\t%d\t%u\t%d\t"
- "%08X\t%d\t%u\t%u\t%02X\t%d\t%1d\t%08X%n",
- r->dst.dev ? r->dst.dev->name : "*",
- (__force u32)r->rt_dst,
- (__force u32)r->rt_gateway,
- r->rt_flags, atomic_read(&r->dst.__refcnt),
- r->dst.__use, 0, (__force u32)r->rt_src,
- dst_metric_advmss(&r->dst) + 40,
- dst_metric(&r->dst, RTAX_WINDOW),
- (int)((dst_metric(&r->dst, RTAX_RTT) >> 3) +
- dst_metric(&r->dst, RTAX_RTTVAR)),
- r->rt_key_tos,
- -1,
- HHUptod,
- r->rt_spec_dst, &len);
-
- seq_printf(seq, "%*s\n", 127 - len, "");
- }
return 0;
}
@@ -453,8 +268,7 @@ static const struct seq_operations rt_cache_seq_ops = {
static int rt_cache_seq_open(struct inode *inode, struct file *file)
{
- return seq_open_net(inode, file, &rt_cache_seq_ops,
- sizeof(struct rt_cache_iter_state));
+ return seq_open(file, &rt_cache_seq_ops);
}
static const struct file_operations rt_cache_seq_fops = {
@@ -462,7 +276,7 @@ static const struct file_operations rt_cache_seq_fops = {
.open = rt_cache_seq_open,
.read = seq_read,
.llseek = seq_lseek,
- .release = seq_release_net,
+ .release = seq_release,
};
@@ -652,184 +466,12 @@ static inline int ip_rt_proc_init(void)
}
#endif /* CONFIG_PROC_FS */
-static inline void rt_free(struct rtable *rt)
-{
- call_rcu_bh(&rt->dst.rcu_head, dst_rcu_free);
-}
-
-static inline void rt_drop(struct rtable *rt)
-{
- ip_rt_put(rt);
- call_rcu_bh(&rt->dst.rcu_head, dst_rcu_free);
-}
-
-static inline int rt_fast_clean(struct rtable *rth)
-{
- /* Kill broadcast/multicast entries very aggresively, if they
- collide in hash table with more useful entries */
- return (rth->rt_flags & (RTCF_BROADCAST | RTCF_MULTICAST)) &&
- rt_is_input_route(rth) && rth->dst.rt_next;
-}
-
-static inline int rt_valuable(struct rtable *rth)
-{
- return (rth->rt_flags & (RTCF_REDIRECTED | RTCF_NOTIFY)) ||
- (rth->peer && rth->peer->pmtu_expires);
-}
-
-static int rt_may_expire(struct rtable *rth, unsigned long tmo1, unsigned long tmo2)
-{
- unsigned long age;
- int ret = 0;
-
- if (atomic_read(&rth->dst.__refcnt))
- goto out;
-
- age = jiffies - rth->dst.lastuse;
- if ((age <= tmo1 && !rt_fast_clean(rth)) ||
- (age <= tmo2 && rt_valuable(rth)))
- goto out;
- ret = 1;
-out: return ret;
-}
-
-/* Bits of score are:
- * 31: very valuable
- * 30: not quite useless
- * 29..0: usage counter
- */
-static inline u32 rt_score(struct rtable *rt)
-{
- u32 score = jiffies - rt->dst.lastuse;
-
- score = ~score & ~(3<<30);
-
- if (rt_valuable(rt))
- score |= (1<<31);
-
- if (rt_is_output_route(rt) ||
- !(rt->rt_flags & (RTCF_BROADCAST|RTCF_MULTICAST|RTCF_LOCAL)))
- score |= (1<<30);
-
- return score;
-}
-
-static inline bool rt_caching(const struct net *net)
-{
- return net->ipv4.current_rt_cache_rebuild_count <=
- net->ipv4.sysctl_rt_cache_rebuild_count;
-}
-
-static inline bool compare_hash_inputs(const struct rtable *rt1,
- const struct rtable *rt2)
-{
- return ((((__force u32)rt1->rt_key_dst ^ (__force u32)rt2->rt_key_dst) |
- ((__force u32)rt1->rt_key_src ^ (__force u32)rt2->rt_key_src) |
- (rt1->rt_route_iif ^ rt2->rt_route_iif)) == 0);
-}
-
-static inline int compare_keys(struct rtable *rt1, struct rtable *rt2)
-{
- return (((__force u32)rt1->rt_key_dst ^ (__force u32)rt2->rt_key_dst) |
- ((__force u32)rt1->rt_key_src ^ (__force u32)rt2->rt_key_src) |
- (rt1->rt_mark ^ rt2->rt_mark) |
- (rt1->rt_key_tos ^ rt2->rt_key_tos) |
- (rt1->rt_route_iif ^ rt2->rt_route_iif) |
- (rt1->rt_oif ^ rt2->rt_oif)) == 0;
-}
-
-static inline int compare_netns(struct rtable *rt1, struct rtable *rt2)
-{
- return net_eq(dev_net(rt1->dst.dev), dev_net(rt2->dst.dev));
-}
-
static inline int rt_is_expired(struct rtable *rth)
{
return rth->rt_genid != rt_genid(dev_net(rth->dst.dev));
}
/*
- * Perform a full scan of hash table and free all entries.
- * Can be called by a softirq or a process.
- * In the later case, we want to be reschedule if necessary
- */
-static void rt_do_flush(struct net *net, int process_context)
-{
- unsigned int i;
- struct rtable *rth, *next;
-
- for (i = 0; i <= rt_hash_mask; i++) {
- struct rtable __rcu **pprev;
- struct rtable *list;
-
- if (process_context && need_resched())
- cond_resched();
- rth = rcu_access_pointer(rt_hash_table[i].chain);
- if (!rth)
- continue;
-
- spin_lock_bh(rt_hash_lock_addr(i));
-
- list = NULL;
- pprev = &rt_hash_table[i].chain;
- rth = rcu_dereference_protected(*pprev,
- lockdep_is_held(rt_hash_lock_addr(i)));
-
- while (rth) {
- next = rcu_dereference_protected(rth->dst.rt_next,
- lockdep_is_held(rt_hash_lock_addr(i)));
-
- if (!net ||
- net_eq(dev_net(rth->dst.dev), net)) {
- rcu_assign_pointer(*pprev, next);
- rcu_assign_pointer(rth->dst.rt_next, list);
- list = rth;
- } else {
- pprev = &rth->dst.rt_next;
- }
- rth = next;
- }
-
- spin_unlock_bh(rt_hash_lock_addr(i));
-
- for (; list; list = next) {
- next = rcu_dereference_protected(list->dst.rt_next, 1);
- rt_free(list);
- }
- }
-}
-
-/*
- * While freeing expired entries, we compute average chain length
- * and standard deviation, using fixed-point arithmetic.
- * This to have an estimation of rt_chain_length_max
- * rt_chain_length_max = max(elasticity, AVG + 4*SD)
- * We use 3 bits for frational part, and 29 (or 61) for magnitude.
- */
-
-#define FRACT_BITS 3
-#define ONE (1UL << FRACT_BITS)
-
-/*
- * Given a hash chain and an item in this hash chain,
- * find if a previous entry has the same hash_inputs
- * (but differs on tos, mark or oif)
- * Returns 0 if an alias is found.
- * Returns ONE if rth has no alias before itself.
- */
-static int has_noalias(const struct rtable *head, const struct rtable *rth)
-{
- const struct rtable *aux = head;
-
- while (aux != rth) {
- if (compare_hash_inputs(aux, rth))
- return 0;
- aux = rcu_dereference_protected(aux->dst.rt_next, 1);
- }
- return ONE;
-}
-
-/*
* Perturbation of rt_genid by a small quantity [1..256]
* Using 8 bits of shuffling ensure we can call rt_cache_invalidate()
* many times (2^24) without giving recent rt_genid.
@@ -851,169 +493,6 @@ static void rt_cache_invalidate(struct net *net)
void rt_cache_flush(struct net *net, int delay)
{
rt_cache_invalidate(net);
- if (delay >= 0)
- rt_do_flush(net, !in_softirq());
-}
-
-/* Flush previous cache invalidated entries from the cache */
-void rt_cache_flush_batch(struct net *net)
-{
- rt_do_flush(net, !in_softirq());
-}
-
-static void rt_emergency_hash_rebuild(struct net *net)
-{
- if (net_ratelimit())
- printk(KERN_WARNING "Route hash chain too long!\n");
- rt_cache_invalidate(net);
-}
-
-/*
- Short description of GC goals.
-
- We want to build algorithm, which will keep routing cache
- at some equilibrium point, when number of aged off entries
- is kept approximately equal to newly generated ones.
-
- Current expiration strength is variable "expire".
- We try to adjust it dynamically, so that if networking
- is idle expires is large enough to keep enough of warm entries,
- and when load increases it reduces to limit cache size.
- */
-
-static int rt_garbage_collect(struct dst_ops *ops)
-{
- static unsigned long expire = RT_GC_TIMEOUT;
- static unsigned long last_gc;
- static int rover;
- static int equilibrium;
- struct rtable *rth;
- struct rtable __rcu **rthp;
- unsigned long now = jiffies;
- int goal;
- int entries = dst_entries_get_fast(&ipv4_dst_ops);
-
- /*
- * Garbage collection is pretty expensive,
- * do not make it too frequently.
- */
-
- RT_CACHE_STAT_INC(gc_total);
-
- if (now - last_gc < ip_rt_gc_min_interval &&
- entries < ip_rt_max_size) {
- RT_CACHE_STAT_INC(gc_ignored);
- goto out;
- }
-
- entries = dst_entries_get_slow(&ipv4_dst_ops);
- /* Calculate number of entries, which we want to expire now. */
- goal = entries - (ip_rt_gc_elasticity << rt_hash_log);
- if (goal <= 0) {
- if (equilibrium < ipv4_dst_ops.gc_thresh)
- equilibrium = ipv4_dst_ops.gc_thresh;
- goal = entries - equilibrium;
- if (goal > 0) {
- equilibrium += min_t(unsigned int, goal >> 1, rt_hash_mask + 1);
- goal = entries - equilibrium;
- }
- } else {
- /* We are in dangerous area. Try to reduce cache really
- * aggressively.
- */
- goal = max_t(unsigned int, goal >> 1, rt_hash_mask + 1);
- equilibrium = entries - goal;
- }
-
- if (now - last_gc >= ip_rt_gc_min_interval)
- last_gc = now;
-
- if (goal <= 0) {
- equilibrium += goal;
- goto work_done;
- }
-
- do {
- int i, k;
-
- for (i = rt_hash_mask, k = rover; i >= 0; i--) {
- unsigned long tmo = expire;
-
- k = (k + 1) & rt_hash_mask;
- rthp = &rt_hash_table[k].chain;
- spin_lock_bh(rt_hash_lock_addr(k));
- while ((rth = rcu_dereference_protected(*rthp,
- lockdep_is_held(rt_hash_lock_addr(k)))) != NULL) {
- if (!rt_is_expired(rth) &&
- !rt_may_expire(rth, tmo, expire)) {
- tmo >>= 1;
- rthp = &rth->dst.rt_next;
- continue;
- }
- *rthp = rth->dst.rt_next;
- rt_free(rth);
- goal--;
- }
- spin_unlock_bh(rt_hash_lock_addr(k));
- if (goal <= 0)
- break;
- }
- rover = k;
-
- if (goal <= 0)
- goto work_done;
-
- /* Goal is not achieved. We stop process if:
-
- - if expire reduced to zero. Otherwise, expire is halfed.
- - if table is not full.
- - if we are called from interrupt.
- - jiffies check is just fallback/debug loop breaker.
- We will not spin here for long time in any case.
- */
-
- RT_CACHE_STAT_INC(gc_goal_miss);
-
- if (expire == 0)
- break;
-
- expire >>= 1;
-
- if (dst_entries_get_fast(&ipv4_dst_ops) < ip_rt_max_size)
- goto out;
- } while (!in_softirq() && time_before_eq(jiffies, now));
-
- if (dst_entries_get_fast(&ipv4_dst_ops) < ip_rt_max_size)
- goto out;
- if (dst_entries_get_slow(&ipv4_dst_ops) < ip_rt_max_size)
- goto out;
- if (net_ratelimit())
- printk(KERN_WARNING "dst cache overflow\n");
- RT_CACHE_STAT_INC(gc_dst_overflow);
- return 1;
-
-work_done:
- expire += ip_rt_gc_min_interval;
- if (expire > ip_rt_gc_timeout ||
- dst_entries_get_fast(&ipv4_dst_ops) < ipv4_dst_ops.gc_thresh ||
- dst_entries_get_slow(&ipv4_dst_ops) < ipv4_dst_ops.gc_thresh)
- expire = ip_rt_gc_timeout;
-out: return 0;
-}
-
-/*
- * Returns number of entries in a hash chain that have different hash_inputs
- */
-static int slow_chain_length(const struct rtable *head)
-{
- int length = 0;
- const struct rtable *rth = head;
-
- while (rth) {
- length += has_noalias(head, rth);
- rth = rcu_dereference_protected(rth->dst.rt_next, 1);
- }
- return length >> FRACT_BITS;
}
static struct neighbour *ipv4_neigh_lookup(const struct dst_entry *dst, const void *daddr)
@@ -1042,181 +521,19 @@ static int rt_bind_neighbour(struct rtable *rt)
return 0;
}
-static struct rtable *rt_intern_hash(unsigned hash, struct rtable *rt,
- struct sk_buff *skb, int ifindex)
+static struct rtable *rt_finalize(struct rtable *rt, struct sk_buff *skb)
{
- struct rtable *rth, *cand;
- struct rtable __rcu **rthp, **candp;
- unsigned long now;
- u32 min_score;
- int chain_length;
- int attempts = !in_softirq();
-
-restart:
- chain_length = 0;
- min_score = ~(u32)0;
- cand = NULL;
- candp = NULL;
- now = jiffies;
-
- if (!rt_caching(dev_net(rt->dst.dev))) {
- /*
- * If we're not caching, just tell the caller we
- * were successful and don't touch the route. The
- * caller hold the sole reference to the cache entry, and
- * it will be released when the caller is done with it.
- * If we drop it here, the callers have no way to resolve routes
- * when we're not caching. Instead, just point *rp at rt, so
- * the caller gets a single use out of the route
- * Note that we do rt_free on this new route entry, so that
- * once its refcount hits zero, we are still able to reap it
- * (Thanks Alexey)
- * Note: To avoid expensive rcu stuff for this uncached dst,
- * we set DST_NOCACHE so that dst_release() can free dst without
- * waiting a grace period.
- */
-
- rt->dst.flags |= DST_NOCACHE;
- if (rt->rt_type == RTN_UNICAST || rt_is_output_route(rt)) {
- int err = rt_bind_neighbour(rt);
- if (err) {
- if (net_ratelimit())
- printk(KERN_WARNING
- "Neighbour table failure & not caching routes.\n");
- ip_rt_put(rt);
- return ERR_PTR(err);
- }
- }
-
- goto skip_hashing;
- }
-
- rthp = &rt_hash_table[hash].chain;
-
- spin_lock_bh(rt_hash_lock_addr(hash));
- while ((rth = rcu_dereference_protected(*rthp,
- lockdep_is_held(rt_hash_lock_addr(hash)))) != NULL) {
- if (rt_is_expired(rth)) {
- *rthp = rth->dst.rt_next;
- rt_free(rth);
- continue;
- }
- if (compare_keys(rth, rt) && compare_netns(rth, rt)) {
- /* Put it first */
- *rthp = rth->dst.rt_next;
- /*
- * Since lookup is lockfree, the deletion
- * must be visible to another weakly ordered CPU before
- * the insertion at the start of the hash chain.
- */
- rcu_assign_pointer(rth->dst.rt_next,
- rt_hash_table[hash].chain);
- /*
- * Since lookup is lockfree, the update writes
- * must be ordered for consistency on SMP.
- */
- rcu_assign_pointer(rt_hash_table[hash].chain, rth);
-
- dst_use(&rth->dst, now);
- spin_unlock_bh(rt_hash_lock_addr(hash));
-
- rt_drop(rt);
- if (skb)
- skb_dst_set(skb, &rth->dst);
- return rth;
- }
-
- if (!atomic_read(&rth->dst.__refcnt)) {
- u32 score = rt_score(rth);
-
- if (score <= min_score) {
- cand = rth;
- candp = rthp;
- min_score = score;
- }
- }
-
- chain_length++;
-
- rthp = &rth->dst.rt_next;
- }
-
- if (cand) {
- /* ip_rt_gc_elasticity used to be average length of chain
- * length, when exceeded gc becomes really aggressive.
- *
- * The second limit is less certain. At the moment it allows
- * only 2 entries per bucket. We will see.
- */
- if (chain_length > ip_rt_gc_elasticity) {
- *candp = cand->dst.rt_next;
- rt_free(cand);
- }
- } else {
- if (chain_length > rt_chain_length_max &&
- slow_chain_length(rt_hash_table[hash].chain) > rt_chain_length_max) {
- struct net *net = dev_net(rt->dst.dev);
- int num = ++net->ipv4.current_rt_cache_rebuild_count;
- if (!rt_caching(net)) {
- printk(KERN_WARNING "%s: %d rebuilds is over limit, route caching disabled\n",
- rt->dst.dev->name, num);
- }
- rt_emergency_hash_rebuild(net);
- spin_unlock_bh(rt_hash_lock_addr(hash));
-
- hash = rt_hash(rt->rt_key_dst, rt->rt_key_src,
- ifindex, rt_genid(net));
- goto restart;
- }
- }
-
- /* Try to bind route to arp only if it is output
- route or unicast forwarding path.
- */
if (rt->rt_type == RTN_UNICAST || rt_is_output_route(rt)) {
int err = rt_bind_neighbour(rt);
if (err) {
- spin_unlock_bh(rt_hash_lock_addr(hash));
-
- if (err != -ENOBUFS) {
- rt_drop(rt);
- return ERR_PTR(err);
- }
-
- /* Neighbour tables are full and nothing
- can be released. Try to shrink route cache,
- it is most likely it holds some neighbour records.
- */
- if (attempts-- > 0) {
- int saved_elasticity = ip_rt_gc_elasticity;
- int saved_int = ip_rt_gc_min_interval;
- ip_rt_gc_elasticity = 1;
- ip_rt_gc_min_interval = 0;
- rt_garbage_collect(&ipv4_dst_ops);
- ip_rt_gc_min_interval = saved_int;
- ip_rt_gc_elasticity = saved_elasticity;
- goto restart;
- }
-
if (net_ratelimit())
- printk(KERN_WARNING "ipv4: Neighbour table overflow.\n");
- rt_drop(rt);
- return ERR_PTR(-ENOBUFS);
+ printk(KERN_WARNING
+ "Neighbour table failure & not caching routes.\n");
+ ip_rt_put(rt);
+ return ERR_PTR(err);
}
}
- rt->dst.rt_next = rt_hash_table[hash].chain;
-
- /*
- * Since lookup is lockfree, we must make sure
- * previous writes to rt are committed to memory
- * before making rt visible to other CPUS.
- */
- rcu_assign_pointer(rt_hash_table[hash].chain, rt);
-
- spin_unlock_bh(rt_hash_lock_addr(hash));
-
-skip_hashing:
if (skb)
skb_dst_set(skb, &rt->dst);
return rt;
@@ -1284,26 +601,6 @@ void __ip_select_ident(struct iphdr *iph, struct dst_entry *dst, int more)
}
EXPORT_SYMBOL(__ip_select_ident);
-static void rt_del(unsigned hash, struct rtable *rt)
-{
- struct rtable __rcu **rthp;
- struct rtable *aux;
-
- rthp = &rt_hash_table[hash].chain;
- spin_lock_bh(rt_hash_lock_addr(hash));
- ip_rt_put(rt);
- while ((aux = rcu_dereference_protected(*rthp,
- lockdep_is_held(rt_hash_lock_addr(hash)))) != NULL) {
- if (aux == rt || rt_is_expired(aux)) {
- *rthp = aux->dst.rt_next;
- rt_free(aux);
- continue;
- }
- rthp = &aux->dst.rt_next;
- }
- spin_unlock_bh(rt_hash_lock_addr(hash));
-}
-
static void check_peer_redir(struct dst_entry *dst, struct inet_peer *peer)
{
struct rtable *rt = (struct rtable *) dst;
@@ -1334,10 +631,7 @@ static void check_peer_redir(struct dst_entry *dst, struct inet_peer *peer)
void ip_rt_redirect(__be32 old_gw, __be32 daddr, __be32 new_gw,
__be32 saddr, struct net_device *dev)
{
- int s, i;
struct in_device *in_dev = __in_dev_get_rcu(dev);
- __be32 skeys[2] = { saddr, 0 };
- int ikeys[2] = { dev->ifindex, 0 };
struct inet_peer *peer;
struct net *net;
@@ -1360,45 +654,15 @@ void ip_rt_redirect(__be32 old_gw, __be32 daddr, __be32 new_gw,
goto reject_redirect;
}
- for (s = 0; s < 2; s++) {
- for (i = 0; i < 2; i++) {
- unsigned int hash;
- struct rtable __rcu **rthp;
- struct rtable *rt;
-
- hash = rt_hash(daddr, skeys[s], ikeys[i], rt_genid(net));
-
- rthp = &rt_hash_table[hash].chain;
-
- while ((rt = rcu_dereference(*rthp)) != NULL) {
- rthp = &rt->dst.rt_next;
-
- if (rt->rt_key_dst != daddr ||
- rt->rt_key_src != skeys[s] ||
- rt->rt_oif != ikeys[i] ||
- rt_is_input_route(rt) ||
- rt_is_expired(rt) ||
- !net_eq(dev_net(rt->dst.dev), net) ||
- rt->dst.error ||
- rt->dst.dev != dev ||
- rt->rt_gateway != old_gw)
- continue;
-
- if (!rt->peer)
- rt_bind_peer(rt, rt->rt_dst, 1);
-
- peer = rt->peer;
- if (peer) {
- if (peer->redirect_learned.a4 != new_gw ||
- peer->redirect_genid != redirect_genid) {
- peer->redirect_learned.a4 = new_gw;
- peer->redirect_genid = redirect_genid;
- atomic_inc(&__rt_peer_genid);
- }
- check_peer_redir(&rt->dst, peer);
- }
- }
+ peer = inet_getpeer_v4(daddr, 1);
+ if (peer) {
+ if (peer->redirect_learned.a4 != new_gw ||
+ peer->redirect_genid != redirect_genid) {
+ peer->redirect_learned.a4 = new_gw;
+ peer->redirect_genid = redirect_genid;
+ atomic_inc(&__rt_peer_genid);
}
+ inet_putpeer(peer);
}
return;
@@ -1440,10 +704,7 @@ static struct dst_entry *ipv4_negative_advice(struct dst_entry *dst)
ip_rt_put(rt);
ret = NULL;
} else if (rt->rt_flags & RTCF_REDIRECTED) {
- unsigned hash = rt_hash(rt->rt_key_dst, rt->rt_key_src,
- rt->rt_oif,
- rt_genid(dev_net(dst->dev)));
- rt_del(hash, rt);
+ ip_rt_put(rt);
ret = NULL;
} else if (rt->peer && peer_pmtu_expired(rt->peer)) {
dst_metric_set(dst, RTAX_MTU, rt->peer->pmtu_orig);
@@ -1911,7 +1172,7 @@ static struct rtable *rt_dst_alloc(struct net_device *dev,
bool nopolicy, bool noxfrm)
{
return dst_alloc(&ipv4_dst_ops, dev, 1, -1,
- DST_HOST |
+ DST_HOST | DST_NOCACHE |
(nopolicy ? DST_NOPOLICY : 0) |
(noxfrm ? DST_NOXFRM : 0));
}
@@ -1920,7 +1181,6 @@ static struct rtable *rt_dst_alloc(struct net_device *dev,
static int ip_route_input_mc(struct sk_buff *skb, __be32 daddr, __be32 saddr,
u8 tos, struct net_device *dev, int our)
{
- unsigned int hash;
struct rtable *rth;
__be32 spec_dst;
struct in_device *in_dev = __in_dev_get_rcu(dev);
@@ -1984,8 +1244,7 @@ static int ip_route_input_mc(struct sk_buff *skb, __be32 daddr, __be32 saddr,
#endif
RT_CACHE_STAT_INC(in_slow_mc);
- hash = rt_hash(daddr, saddr, dev->ifindex, rt_genid(dev_net(dev)));
- rth = rt_intern_hash(hash, rth, skb, dev->ifindex);
+ rth = rt_finalize(rth, skb);
return IS_ERR(rth) ? PTR_ERR(rth) : 0;
e_nobufs:
@@ -2128,7 +1387,6 @@ static int ip_mkroute_input(struct sk_buff *skb,
{
struct rtable* rth = NULL;
int err;
- unsigned hash;
#ifdef CONFIG_IP_ROUTE_MULTIPATH
if (res->fi && res->fi->fib_nhs > 1)
@@ -2140,10 +1398,7 @@ static int ip_mkroute_input(struct sk_buff *skb,
if (err)
return err;
- /* put it into the cache */
- hash = rt_hash(daddr, saddr, fl4->flowi4_iif,
- rt_genid(dev_net(rth->dst.dev)));
- rth = rt_intern_hash(hash, rth, skb, fl4->flowi4_iif);
+ rth = rt_finalize(rth, skb);
if (IS_ERR(rth))
return PTR_ERR(rth);
return 0;
@@ -2169,7 +1424,6 @@ static int ip_route_input_slow(struct sk_buff *skb, __be32 daddr, __be32 saddr,
unsigned flags = 0;
u32 itag = 0;
struct rtable * rth;
- unsigned hash;
__be32 spec_dst;
int err = -EINVAL;
struct net * net = dev_net(dev);
@@ -2296,8 +1550,7 @@ local_input:
rth->dst.error= -err;
rth->rt_flags &= ~RTCF_LOCAL;
}
- hash = rt_hash(daddr, saddr, fl4.flowi4_iif, rt_genid(net));
- rth = rt_intern_hash(hash, rth, skb, fl4.flowi4_iif);
+ rth = rt_finalize(rth, skb);
err = 0;
if (IS_ERR(rth))
err = PTR_ERR(rth);
@@ -2344,47 +1597,10 @@ martian_source_keep_err:
int ip_route_input_common(struct sk_buff *skb, __be32 daddr, __be32 saddr,
u8 tos, struct net_device *dev, bool noref)
{
- struct rtable * rth;
- unsigned hash;
- int iif = dev->ifindex;
- struct net *net;
int res;
- net = dev_net(dev);
-
rcu_read_lock();
- if (!rt_caching(net))
- goto skip_cache;
-
- tos &= IPTOS_RT_MASK;
- hash = rt_hash(daddr, saddr, iif, rt_genid(net));
-
- for (rth = rcu_dereference(rt_hash_table[hash].chain); rth;
- rth = rcu_dereference(rth->dst.rt_next)) {
- if ((((__force u32)rth->rt_key_dst ^ (__force u32)daddr) |
- ((__force u32)rth->rt_key_src ^ (__force u32)saddr) |
- (rth->rt_route_iif ^ iif) |
- (rth->rt_key_tos ^ tos)) == 0 &&
- rth->rt_mark == skb->mark &&
- net_eq(dev_net(rth->dst.dev), net) &&
- !rt_is_expired(rth)) {
- ipv4_validate_peer(rth);
- if (noref) {
- dst_use_noref(&rth->dst, jiffies);
- skb_dst_set_noref(skb, &rth->dst);
- } else {
- dst_use(&rth->dst, jiffies);
- skb_dst_set(skb, &rth->dst);
- }
- RT_CACHE_STAT_INC(in_hit);
- rcu_read_unlock();
- return 0;
- }
- RT_CACHE_STAT_INC(in_hlist_search);
- }
-
-skip_cache:
/* Multicast recognition logic is moved from route cache to here.
The problem was that too many Ethernet cards have broken/missing
hardware multicast filters :-( As result the host on multicasting
@@ -2527,10 +1743,9 @@ static struct rtable *__mkroute_output(const struct fib_result *res,
/*
* Major route resolver routine.
- * called with rcu_read_lock();
*/
-static struct rtable *ip_route_output_slow(struct net *net, struct flowi4 *fl4)
+struct rtable *__ip_route_output_key(struct net *net, struct flowi4 *fl4)
{
struct net_device *dev_out = NULL;
__u8 tos = RT_FL_TOS(fl4);
@@ -2710,58 +1925,13 @@ static struct rtable *ip_route_output_slow(struct net *net, struct flowi4 *fl4)
make_route:
rth = __mkroute_output(&res, fl4, orig_daddr, orig_saddr, orig_oif,
tos, dev_out, flags);
- if (!IS_ERR(rth)) {
- unsigned int hash;
-
- hash = rt_hash(orig_daddr, orig_saddr, orig_oif,
- rt_genid(dev_net(dev_out)));
- rth = rt_intern_hash(hash, rth, NULL, orig_oif);
- }
+ if (!IS_ERR(rth))
+ rth = rt_finalize(rth, NULL);
out:
rcu_read_unlock();
return rth;
}
-
-struct rtable *__ip_route_output_key(struct net *net, struct flowi4 *flp4)
-{
- struct rtable *rth;
- unsigned int hash;
-
- if (!rt_caching(net))
- goto slow_output;
-
- hash = rt_hash(flp4->daddr, flp4->saddr, flp4->flowi4_oif, rt_genid(net));
-
- rcu_read_lock_bh();
- for (rth = rcu_dereference_bh(rt_hash_table[hash].chain); rth;
- rth = rcu_dereference_bh(rth->dst.rt_next)) {
- if (rth->rt_key_dst == flp4->daddr &&
- rth->rt_key_src == flp4->saddr &&
- rt_is_output_route(rth) &&
- rth->rt_oif == flp4->flowi4_oif &&
- rth->rt_mark == flp4->flowi4_mark &&
- !((rth->rt_key_tos ^ flp4->flowi4_tos) &
- (IPTOS_RT_MASK | RTO_ONLINK)) &&
- net_eq(dev_net(rth->dst.dev), net) &&
- !rt_is_expired(rth)) {
- ipv4_validate_peer(rth);
- dst_use(&rth->dst, jiffies);
- RT_CACHE_STAT_INC(out_hit);
- rcu_read_unlock_bh();
- if (!flp4->saddr)
- flp4->saddr = rth->rt_src;
- if (!flp4->daddr)
- flp4->daddr = rth->rt_dst;
- return rth;
- }
- RT_CACHE_STAT_INC(out_hlist_search);
- }
- rcu_read_unlock_bh();
-
-slow_output:
- return ip_route_output_slow(net, flp4);
-}
EXPORT_SYMBOL_GPL(__ip_route_output_key);
static struct dst_entry *ipv4_blackhole_dst_check(struct dst_entry *dst, u32 cookie)
@@ -3067,43 +2237,6 @@ errout_free:
int ip_rt_dump(struct sk_buff *skb, struct netlink_callback *cb)
{
- struct rtable *rt;
- int h, s_h;
- int idx, s_idx;
- struct net *net;
-
- net = sock_net(skb->sk);
-
- s_h = cb->args[0];
- if (s_h < 0)
- s_h = 0;
- s_idx = idx = cb->args[1];
- for (h = s_h; h <= rt_hash_mask; h++, s_idx = 0) {
- if (!rt_hash_table[h].chain)
- continue;
- rcu_read_lock_bh();
- for (rt = rcu_dereference_bh(rt_hash_table[h].chain), idx = 0; rt;
- rt = rcu_dereference_bh(rt->dst.rt_next), idx++) {
- if (!net_eq(dev_net(rt->dst.dev), net) || idx < s_idx)
- continue;
- if (rt_is_expired(rt))
- continue;
- skb_dst_set_noref(skb, &rt->dst);
- if (rt_fill_info(net, skb, NETLINK_CB(cb->skb).pid,
- cb->nlh->nlmsg_seq, RTM_NEWROUTE,
- 1, NLM_F_MULTI) <= 0) {
- skb_dst_drop(skb);
- rcu_read_unlock_bh();
- goto done;
- }
- skb_dst_drop(skb);
- }
- rcu_read_unlock_bh();
- }
-
-done:
- cb->args[0] = h;
- cb->args[1] = idx;
return skb->len;
}
@@ -3331,16 +2464,6 @@ static __net_initdata struct pernet_operations rt_genid_ops = {
struct ip_rt_acct __percpu *ip_rt_acct __read_mostly;
#endif /* CONFIG_IP_ROUTE_CLASSID */
-static __initdata unsigned long rhash_entries;
-static int __init set_rhash_entries(char *str)
-{
- if (!str)
- return 0;
- rhash_entries = simple_strtoul(str, &str, 0);
- return 1;
-}
-__setup("rhash_entries=", set_rhash_entries);
-
int __init ip_rt_init(void)
{
int rc = 0;
@@ -3363,21 +2486,8 @@ int __init ip_rt_init(void)
if (dst_entries_init(&ipv4_dst_blackhole_ops) < 0)
panic("IP: failed to allocate ipv4_dst_blackhole_ops counter\n");
- rt_hash_table = (struct rt_hash_bucket *)
- alloc_large_system_hash("IP route cache",
- sizeof(struct rt_hash_bucket),
- rhash_entries,
- (totalram_pages >= 128 * 1024) ?
- 15 : 17,
- 0,
- &rt_hash_log,
- &rt_hash_mask,
- rhash_entries ? 0 : 512 * 1024);
- memset(rt_hash_table, 0, (rt_hash_mask + 1) * sizeof(struct rt_hash_bucket));
- rt_hash_lock_init();
-
- ipv4_dst_ops.gc_thresh = (rt_hash_mask + 1);
- ip_rt_max_size = (rt_hash_mask + 1) * 16;
+ ipv4_dst_ops.gc_thresh = ~0;
+ ip_rt_max_size = INT_MAX;
devinet_init();
ip_fib_init();
--
1.7.6.401.g6a319
^ permalink raw reply related
* [PATCH 2/5] ipv4: Kill ip_route_input_noref().
From: David Miller @ 2011-12-20 21:35 UTC (permalink / raw)
To: netdev
The "noref" argument to ip_route_input_common() is now always ignored
because we do not cache routes, and in that case we must always grab
a reference to the resulting 'dst'.
Signed-off-by: David S. Miller <davem@davemloft.net>
---
include/net/route.h | 16 ++--------------
net/ipv4/arp.c | 2 +-
net/ipv4/ip_fragment.c | 4 ++--
net/ipv4/ip_input.c | 4 ++--
net/ipv4/route.c | 6 +++---
net/ipv4/xfrm4_input.c | 4 ++--
6 files changed, 12 insertions(+), 24 deletions(-)
diff --git a/include/net/route.h b/include/net/route.h
index ae32c9f..9675965 100644
--- a/include/net/route.h
+++ b/include/net/route.h
@@ -165,20 +165,8 @@ static inline struct rtable *ip_route_output_gre(struct net *net, struct flowi4
return ip_route_output_key(net, fl4);
}
-extern int ip_route_input_common(struct sk_buff *skb, __be32 dst, __be32 src,
- u8 tos, struct net_device *devin, bool noref);
-
-static inline int ip_route_input(struct sk_buff *skb, __be32 dst, __be32 src,
- u8 tos, struct net_device *devin)
-{
- return ip_route_input_common(skb, dst, src, tos, devin, false);
-}
-
-static inline int ip_route_input_noref(struct sk_buff *skb, __be32 dst, __be32 src,
- u8 tos, struct net_device *devin)
-{
- return ip_route_input_common(skb, dst, src, tos, devin, true);
-}
+extern int ip_route_input(struct sk_buff *skb, __be32 dst, __be32 src,
+ u8 tos, struct net_device *devin);
extern unsigned short ip_rt_frag_needed(struct net *net, const struct iphdr *iph,
unsigned short new_mtu, struct net_device *dev);
diff --git a/net/ipv4/arp.c b/net/ipv4/arp.c
index 381a087..e6a3cbd 100644
--- a/net/ipv4/arp.c
+++ b/net/ipv4/arp.c
@@ -838,7 +838,7 @@ static int arp_process(struct sk_buff *skb)
}
if (arp->ar_op == htons(ARPOP_REQUEST) &&
- ip_route_input_noref(skb, tip, sip, 0, dev) == 0) {
+ ip_route_input(skb, tip, sip, 0, dev) == 0) {
rt = skb_rtable(skb);
addr_type = rt->rt_type;
diff --git a/net/ipv4/ip_fragment.c b/net/ipv4/ip_fragment.c
index fdaabf2..b9423cd 100644
--- a/net/ipv4/ip_fragment.c
+++ b/net/ipv4/ip_fragment.c
@@ -252,8 +252,8 @@ static void ip_expire(unsigned long arg)
/* skb dst is stale, drop it, and perform route lookup again */
skb_dst_drop(head);
iph = ip_hdr(head);
- err = ip_route_input_noref(head, iph->daddr, iph->saddr,
- iph->tos, head->dev);
+ err = ip_route_input(head, iph->daddr, iph->saddr,
+ iph->tos, head->dev);
if (err)
goto out_rcu_unlock;
diff --git a/net/ipv4/ip_input.c b/net/ipv4/ip_input.c
index 073a9b0..ee753c3 100644
--- a/net/ipv4/ip_input.c
+++ b/net/ipv4/ip_input.c
@@ -324,8 +324,8 @@ static int ip_rcv_finish(struct sk_buff *skb)
* how the packet travels inside Linux networking.
*/
if (skb_dst(skb) == NULL) {
- int err = ip_route_input_noref(skb, iph->daddr, iph->saddr,
- iph->tos, skb->dev);
+ int err = ip_route_input(skb, iph->daddr, iph->saddr,
+ iph->tos, skb->dev);
if (unlikely(err)) {
if (err == -EHOSTUNREACH)
IP_INC_STATS_BH(dev_net(skb->dev),
diff --git a/net/ipv4/route.c b/net/ipv4/route.c
index 68e903a..edc91aa 100644
--- a/net/ipv4/route.c
+++ b/net/ipv4/route.c
@@ -1594,8 +1594,8 @@ martian_source_keep_err:
goto out;
}
-int ip_route_input_common(struct sk_buff *skb, __be32 daddr, __be32 saddr,
- u8 tos, struct net_device *dev, bool noref)
+int ip_route_input(struct sk_buff *skb, __be32 daddr, __be32 saddr,
+ u8 tos, struct net_device *dev)
{
int res;
@@ -1638,7 +1638,7 @@ int ip_route_input_common(struct sk_buff *skb, __be32 daddr, __be32 saddr,
rcu_read_unlock();
return res;
}
-EXPORT_SYMBOL(ip_route_input_common);
+EXPORT_SYMBOL(ip_route_input);
/* called with rcu_read_lock() */
static struct rtable *__mkroute_output(const struct fib_result *res,
diff --git a/net/ipv4/xfrm4_input.c b/net/ipv4/xfrm4_input.c
index 06814b6..58d23a5 100644
--- a/net/ipv4/xfrm4_input.c
+++ b/net/ipv4/xfrm4_input.c
@@ -27,8 +27,8 @@ static inline int xfrm4_rcv_encap_finish(struct sk_buff *skb)
if (skb_dst(skb) == NULL) {
const struct iphdr *iph = ip_hdr(skb);
- if (ip_route_input_noref(skb, iph->daddr, iph->saddr,
- iph->tos, skb->dev))
+ if (ip_route_input(skb, iph->daddr, iph->saddr,
+ iph->tos, skb->dev))
goto drop;
}
return dst_input(skb);
--
1.7.6.401.g6a319
^ permalink raw reply related
* [PATCH 3/5] ipv4: Remove rt_key_{src,dst,tos} from struct rtable.
From: David Miller @ 2011-12-20 21:35 UTC (permalink / raw)
To: netdev
They are always used in contexts where they can be reconstituted,
or where the finally resolved rt->rt_{src,dst} is semantically
equivalent.
Signed-off-by: David S. Miller <davem@davemloft.net>
---
include/net/route.h | 5 -----
net/ipv4/route.c | 39 +++++++++------------------------------
net/ipv4/xfrm4_policy.c | 3 ---
3 files changed, 9 insertions(+), 38 deletions(-)
diff --git a/include/net/route.h b/include/net/route.h
index 9675965..a2f250d0 100644
--- a/include/net/route.h
+++ b/include/net/route.h
@@ -45,14 +45,9 @@ struct fib_info;
struct rtable {
struct dst_entry dst;
- /* Lookup key. */
- __be32 rt_key_dst;
- __be32 rt_key_src;
-
int rt_genid;
unsigned rt_flags;
__u16 rt_type;
- __u8 rt_key_tos;
__be32 rt_dst; /* Path destination */
__be32 rt_src; /* Path source */
diff --git a/net/ipv4/route.c b/net/ipv4/route.c
index edc91aa..a30611c 100644
--- a/net/ipv4/route.c
+++ b/net/ipv4/route.c
@@ -1216,12 +1216,9 @@ static int ip_route_input_mc(struct sk_buff *skb, __be32 daddr, __be32 saddr,
#endif
rth->dst.output = ip_rt_bug;
- rth->rt_key_dst = daddr;
- rth->rt_key_src = saddr;
rth->rt_genid = rt_genid(dev_net(dev));
rth->rt_flags = RTCF_MULTICAST;
rth->rt_type = RTN_MULTICAST;
- rth->rt_key_tos = tos;
rth->rt_dst = daddr;
rth->rt_src = saddr;
rth->rt_route_iif = dev->ifindex;
@@ -1350,12 +1347,9 @@ static int __mkroute_input(struct sk_buff *skb,
goto cleanup;
}
- rth->rt_key_dst = daddr;
- rth->rt_key_src = saddr;
rth->rt_genid = rt_genid(dev_net(rth->dst.dev));
rth->rt_flags = flags;
rth->rt_type = res->type;
- rth->rt_key_tos = tos;
rth->rt_dst = daddr;
rth->rt_src = saddr;
rth->rt_route_iif = in_dev->dev->ifindex;
@@ -1525,12 +1519,9 @@ local_input:
rth->dst.tclassid = itag;
#endif
- rth->rt_key_dst = daddr;
- rth->rt_key_src = saddr;
rth->rt_genid = rt_genid(net);
rth->rt_flags = flags|RTCF_LOCAL;
rth->rt_type = res.type;
- rth->rt_key_tos = tos;
rth->rt_dst = daddr;
rth->rt_src = saddr;
#ifdef CONFIG_IP_ROUTE_CLASSID
@@ -1642,9 +1633,7 @@ EXPORT_SYMBOL(ip_route_input);
/* called with rcu_read_lock() */
static struct rtable *__mkroute_output(const struct fib_result *res,
- const struct flowi4 *fl4,
- __be32 orig_daddr, __be32 orig_saddr,
- int orig_oif, __u8 orig_rtos,
+ const struct flowi4 *fl4, int orig_oif,
struct net_device *dev_out,
unsigned int flags)
{
@@ -1694,12 +1683,9 @@ static struct rtable *__mkroute_output(const struct fib_result *res,
rth->dst.output = ip_output;
- rth->rt_key_dst = orig_daddr;
- rth->rt_key_src = orig_saddr;
rth->rt_genid = rt_genid(dev_net(dev_out));
rth->rt_flags = flags;
rth->rt_type = type;
- rth->rt_key_tos = orig_rtos;
rth->rt_dst = fl4->daddr;
rth->rt_src = fl4->saddr;
rth->rt_route_iif = 0;
@@ -1752,8 +1738,6 @@ struct rtable *__ip_route_output_key(struct net *net, struct flowi4 *fl4)
unsigned int flags = 0;
struct fib_result res;
struct rtable *rth;
- __be32 orig_daddr;
- __be32 orig_saddr;
int orig_oif;
res.fi = NULL;
@@ -1761,8 +1745,6 @@ struct rtable *__ip_route_output_key(struct net *net, struct flowi4 *fl4)
res.r = NULL;
#endif
- orig_daddr = fl4->daddr;
- orig_saddr = fl4->saddr;
orig_oif = fl4->flowi4_oif;
fl4->flowi4_iif = net->loopback_dev->ifindex;
@@ -1923,8 +1905,7 @@ struct rtable *__ip_route_output_key(struct net *net, struct flowi4 *fl4)
make_route:
- rth = __mkroute_output(&res, fl4, orig_daddr, orig_saddr, orig_oif,
- tos, dev_out, flags);
+ rth = __mkroute_output(&res, fl4, orig_oif, dev_out, flags);
if (!IS_ERR(rth))
rth = rt_finalize(rth, NULL);
@@ -1985,9 +1966,6 @@ struct dst_entry *ipv4_blackhole_route(struct net *net, struct dst_entry *dst_or
if (new->dev)
dev_hold(new->dev);
- rt->rt_key_dst = ort->rt_key_dst;
- rt->rt_key_src = ort->rt_key_src;
- rt->rt_key_tos = ort->rt_key_tos;
rt->rt_route_iif = ort->rt_route_iif;
rt->rt_iif = ort->rt_iif;
rt->rt_oif = ort->rt_oif;
@@ -2032,7 +2010,7 @@ struct rtable *ip_route_output_flow(struct net *net, struct flowi4 *flp4,
}
EXPORT_SYMBOL_GPL(ip_route_output_flow);
-static int rt_fill_info(struct net *net,
+static int rt_fill_info(struct net *net, __be32 src, u8 tos,
struct sk_buff *skb, u32 pid, u32 seq, int event,
int nowait, unsigned int flags)
{
@@ -2051,7 +2029,7 @@ static int rt_fill_info(struct net *net,
r->rtm_family = AF_INET;
r->rtm_dst_len = 32;
r->rtm_src_len = 0;
- r->rtm_tos = rt->rt_key_tos;
+ r->rtm_tos = tos;
r->rtm_table = RT_TABLE_MAIN;
NLA_PUT_U32(skb, RTA_TABLE, RT_TABLE_MAIN);
r->rtm_type = rt->rt_type;
@@ -2063,9 +2041,9 @@ static int rt_fill_info(struct net *net,
NLA_PUT_BE32(skb, RTA_DST, rt->rt_dst);
- if (rt->rt_key_src) {
+ if (src) {
r->rtm_src_len = 32;
- NLA_PUT_BE32(skb, RTA_SRC, rt->rt_key_src);
+ NLA_PUT_BE32(skb, RTA_SRC, src);
}
if (rt->dst.dev)
NLA_PUT_U32(skb, RTA_OIF, rt->dst.dev->ifindex);
@@ -2075,7 +2053,7 @@ static int rt_fill_info(struct net *net,
#endif
if (rt_is_input_route(rt))
NLA_PUT_BE32(skb, RTA_PREFSRC, rt->rt_spec_dst);
- else if (rt->rt_src != rt->rt_key_src)
+ else if (rt->rt_src != src)
NLA_PUT_BE32(skb, RTA_PREFSRC, rt->rt_src);
if (rt->rt_dst != rt->rt_gateway)
@@ -2221,7 +2199,8 @@ static int inet_rtm_getroute(struct sk_buff *in_skb, struct nlmsghdr* nlh, void
if (rtm->rtm_flags & RTM_F_NOTIFY)
rt->rt_flags |= RTCF_NOTIFY;
- err = rt_fill_info(net, skb, NETLINK_CB(in_skb).pid, nlh->nlmsg_seq,
+ err = rt_fill_info(net, src, rtm->rtm_tos, skb,
+ NETLINK_CB(in_skb).pid, nlh->nlmsg_seq,
RTM_NEWROUTE, 0, 0);
if (err <= 0)
goto errout_free;
diff --git a/net/ipv4/xfrm4_policy.c b/net/ipv4/xfrm4_policy.c
index a0b4c5d..a659008 100644
--- a/net/ipv4/xfrm4_policy.c
+++ b/net/ipv4/xfrm4_policy.c
@@ -79,9 +79,6 @@ static int xfrm4_fill_dst(struct xfrm_dst *xdst, struct net_device *dev,
struct rtable *rt = (struct rtable *)xdst->route;
const struct flowi4 *fl4 = &fl->u.ip4;
- xdst->u.rt.rt_key_dst = fl4->daddr;
- xdst->u.rt.rt_key_src = fl4->saddr;
- xdst->u.rt.rt_key_tos = fl4->flowi4_tos;
xdst->u.rt.rt_route_iif = fl4->flowi4_iif;
xdst->u.rt.rt_iif = fl4->flowi4_iif;
xdst->u.rt.rt_oif = fl4->flowi4_oif;
--
1.7.6.401.g6a319
^ permalink raw reply related
* [PATCH 4/5] ipv4: Kill 'rt_src' from 'struct rtable'
From: David Miller @ 2011-12-20 21:35 UTC (permalink / raw)
To: netdev
Signed-off-by: David S. Miller <davem@davemloft.net>
---
include/net/route.h | 1 -
net/ipv4/route.c | 34 +++++++++++++++-------------------
net/ipv4/xfrm4_policy.c | 1 -
3 files changed, 15 insertions(+), 21 deletions(-)
diff --git a/include/net/route.h b/include/net/route.h
index a2f250d0..2c5d300 100644
--- a/include/net/route.h
+++ b/include/net/route.h
@@ -50,7 +50,6 @@ struct rtable {
__u16 rt_type;
__be32 rt_dst; /* Path destination */
- __be32 rt_src; /* Path source */
int rt_route_iif;
int rt_iif;
int rt_oif;
diff --git a/net/ipv4/route.c b/net/ipv4/route.c
index a30611c..208be60 100644
--- a/net/ipv4/route.c
+++ b/net/ipv4/route.c
@@ -1220,7 +1220,6 @@ static int ip_route_input_mc(struct sk_buff *skb, __be32 daddr, __be32 saddr,
rth->rt_flags = RTCF_MULTICAST;
rth->rt_type = RTN_MULTICAST;
rth->rt_dst = daddr;
- rth->rt_src = saddr;
rth->rt_route_iif = dev->ifindex;
rth->rt_iif = dev->ifindex;
rth->rt_oif = 0;
@@ -1351,7 +1350,6 @@ static int __mkroute_input(struct sk_buff *skb,
rth->rt_flags = flags;
rth->rt_type = res->type;
rth->rt_dst = daddr;
- rth->rt_src = saddr;
rth->rt_route_iif = in_dev->dev->ifindex;
rth->rt_iif = in_dev->dev->ifindex;
rth->rt_oif = 0;
@@ -1523,7 +1521,6 @@ local_input:
rth->rt_flags = flags|RTCF_LOCAL;
rth->rt_type = res.type;
rth->rt_dst = daddr;
- rth->rt_src = saddr;
#ifdef CONFIG_IP_ROUTE_CLASSID
rth->dst.tclassid = itag;
#endif
@@ -1687,7 +1684,6 @@ static struct rtable *__mkroute_output(const struct fib_result *res,
rth->rt_flags = flags;
rth->rt_type = type;
rth->rt_dst = fl4->daddr;
- rth->rt_src = fl4->saddr;
rth->rt_route_iif = 0;
rth->rt_iif = orig_oif ? : dev_out->ifindex;
rth->rt_oif = orig_oif;
@@ -1975,7 +1971,6 @@ struct dst_entry *ipv4_blackhole_route(struct net *net, struct dst_entry *dst_or
rt->rt_flags = ort->rt_flags;
rt->rt_type = ort->rt_type;
rt->rt_dst = ort->rt_dst;
- rt->rt_src = ort->rt_src;
rt->rt_gateway = ort->rt_gateway;
rt->rt_spec_dst = ort->rt_spec_dst;
rt->peer = ort->peer;
@@ -2010,7 +2005,7 @@ struct rtable *ip_route_output_flow(struct net *net, struct flowi4 *flp4,
}
EXPORT_SYMBOL_GPL(ip_route_output_flow);
-static int rt_fill_info(struct net *net, __be32 src, u8 tos,
+static int rt_fill_info(struct net *net, __be32 src, struct flowi4 *fl4,
struct sk_buff *skb, u32 pid, u32 seq, int event,
int nowait, unsigned int flags)
{
@@ -2029,7 +2024,7 @@ static int rt_fill_info(struct net *net, __be32 src, u8 tos,
r->rtm_family = AF_INET;
r->rtm_dst_len = 32;
r->rtm_src_len = 0;
- r->rtm_tos = tos;
+ r->rtm_tos = fl4->flowi4_tos;
r->rtm_table = RT_TABLE_MAIN;
NLA_PUT_U32(skb, RTA_TABLE, RT_TABLE_MAIN);
r->rtm_type = rt->rt_type;
@@ -2053,10 +2048,10 @@ static int rt_fill_info(struct net *net, __be32 src, u8 tos,
#endif
if (rt_is_input_route(rt))
NLA_PUT_BE32(skb, RTA_PREFSRC, rt->rt_spec_dst);
- else if (rt->rt_src != src)
- NLA_PUT_BE32(skb, RTA_PREFSRC, rt->rt_src);
+ else if (fl4->saddr != src)
+ NLA_PUT_BE32(skb, RTA_PREFSRC, fl4->saddr);
- if (rt->rt_dst != rt->rt_gateway)
+ if (fl4->daddr != rt->rt_gateway)
NLA_PUT_BE32(skb, RTA_GATEWAY, rt->rt_gateway);
if (rtnetlink_put_metrics(skb, dst_metrics_ptr(&rt->dst)) < 0)
@@ -2089,7 +2084,7 @@ static int rt_fill_info(struct net *net, __be32 src, u8 tos,
if (ipv4_is_multicast(dst) && !ipv4_is_local_multicast(dst) &&
IPV4_DEVCONF_ALL(net, MC_FORWARDING)) {
int err = ipmr_get_route(net, skb,
- rt->rt_src, rt->rt_dst,
+ fl4->saddr, fl4->daddr,
r, nowait);
if (err <= 0) {
if (!nowait) {
@@ -2124,6 +2119,7 @@ static int inet_rtm_getroute(struct sk_buff *in_skb, struct nlmsghdr* nlh, void
struct rtmsg *rtm;
struct nlattr *tb[RTA_MAX+1];
struct rtable *rt = NULL;
+ struct flowi4 fl4;
__be32 dst = 0;
__be32 src = 0;
u32 iif;
@@ -2158,6 +2154,13 @@ static int inet_rtm_getroute(struct sk_buff *in_skb, struct nlmsghdr* nlh, void
iif = tb[RTA_IIF] ? nla_get_u32(tb[RTA_IIF]) : 0;
mark = tb[RTA_MARK] ? nla_get_u32(tb[RTA_MARK]) : 0;
+ memset(&fl4, 0, sizeof(fl4));
+ fl4.daddr = dst;
+ fl4.saddr = src;
+ fl4.flowi4_tos = rtm->rtm_tos;
+ fl4.flowi4_oif = tb[RTA_OIF] ? nla_get_u32(tb[RTA_OIF]) : 0;
+ fl4.flowi4_mark = mark;
+
if (iif) {
struct net_device *dev;
@@ -2178,13 +2181,6 @@ static int inet_rtm_getroute(struct sk_buff *in_skb, struct nlmsghdr* nlh, void
if (err == 0 && rt->dst.error)
err = -rt->dst.error;
} else {
- struct flowi4 fl4 = {
- .daddr = dst,
- .saddr = src,
- .flowi4_tos = rtm->rtm_tos,
- .flowi4_oif = tb[RTA_OIF] ? nla_get_u32(tb[RTA_OIF]) : 0,
- .flowi4_mark = mark,
- };
rt = ip_route_output_key(net, &fl4);
err = 0;
@@ -2199,7 +2195,7 @@ static int inet_rtm_getroute(struct sk_buff *in_skb, struct nlmsghdr* nlh, void
if (rtm->rtm_flags & RTM_F_NOTIFY)
rt->rt_flags |= RTCF_NOTIFY;
- err = rt_fill_info(net, src, rtm->rtm_tos, skb,
+ err = rt_fill_info(net, src, &fl4, skb,
NETLINK_CB(in_skb).pid, nlh->nlmsg_seq,
RTM_NEWROUTE, 0, 0);
if (err <= 0)
diff --git a/net/ipv4/xfrm4_policy.c b/net/ipv4/xfrm4_policy.c
index a659008..210da0c 100644
--- a/net/ipv4/xfrm4_policy.c
+++ b/net/ipv4/xfrm4_policy.c
@@ -96,7 +96,6 @@ static int xfrm4_fill_dst(struct xfrm_dst *xdst, struct net_device *dev,
xdst->u.rt.rt_flags = rt->rt_flags & (RTCF_BROADCAST | RTCF_MULTICAST |
RTCF_LOCAL);
xdst->u.rt.rt_type = rt->rt_type;
- xdst->u.rt.rt_src = rt->rt_src;
xdst->u.rt.rt_dst = rt->rt_dst;
xdst->u.rt.rt_gateway = rt->rt_gateway;
xdst->u.rt.rt_spec_dst = rt->rt_spec_dst;
--
1.7.6.401.g6a319
^ permalink raw reply related
* [PATCH 5/5] ipv4: Remove 'rt_mark' from 'struct rtable'
From: David Miller @ 2011-12-20 21:35 UTC (permalink / raw)
To: netdev
Signed-off-by: David S. Miller <davem@davemloft.net>
---
include/net/route.h | 1 -
net/ipv4/ipmr.c | 2 +-
net/ipv4/route.c | 9 ++-------
net/ipv4/xfrm4_policy.c | 1 -
4 files changed, 3 insertions(+), 10 deletions(-)
diff --git a/include/net/route.h b/include/net/route.h
index 2c5d300..1664745 100644
--- a/include/net/route.h
+++ b/include/net/route.h
@@ -53,7 +53,6 @@ struct rtable {
int rt_route_iif;
int rt_iif;
int rt_oif;
- __u32 rt_mark;
/* Info on neighbour */
__be32 rt_gateway;
diff --git a/net/ipv4/ipmr.c b/net/ipv4/ipmr.c
index 8e54490..49a9f77 100644
--- a/net/ipv4/ipmr.c
+++ b/net/ipv4/ipmr.c
@@ -1798,7 +1798,7 @@ static struct mr_table *ipmr_rt_fib_lookup(struct net *net, struct sk_buff *skb)
.flowi4_tos = RT_TOS(iph->tos),
.flowi4_oif = rt->rt_oif,
.flowi4_iif = rt->rt_iif,
- .flowi4_mark = rt->rt_mark,
+ .flowi4_mark = skb->mark,
};
struct mr_table *mrt;
int err;
diff --git a/net/ipv4/route.c b/net/ipv4/route.c
index 208be60..c48d6fc2 100644
--- a/net/ipv4/route.c
+++ b/net/ipv4/route.c
@@ -1223,7 +1223,6 @@ static int ip_route_input_mc(struct sk_buff *skb, __be32 daddr, __be32 saddr,
rth->rt_route_iif = dev->ifindex;
rth->rt_iif = dev->ifindex;
rth->rt_oif = 0;
- rth->rt_mark = skb->mark;
rth->rt_gateway = daddr;
rth->rt_spec_dst= spec_dst;
rth->rt_peer_genid = 0;
@@ -1353,7 +1352,6 @@ static int __mkroute_input(struct sk_buff *skb,
rth->rt_route_iif = in_dev->dev->ifindex;
rth->rt_iif = in_dev->dev->ifindex;
rth->rt_oif = 0;
- rth->rt_mark = skb->mark;
rth->rt_gateway = daddr;
rth->rt_spec_dst= spec_dst;
rth->rt_peer_genid = 0;
@@ -1527,7 +1525,6 @@ local_input:
rth->rt_route_iif = dev->ifindex;
rth->rt_iif = dev->ifindex;
rth->rt_oif = 0;
- rth->rt_mark = skb->mark;
rth->rt_gateway = daddr;
rth->rt_spec_dst= spec_dst;
rth->rt_peer_genid = 0;
@@ -1687,7 +1684,6 @@ static struct rtable *__mkroute_output(const struct fib_result *res,
rth->rt_route_iif = 0;
rth->rt_iif = orig_oif ? : dev_out->ifindex;
rth->rt_oif = orig_oif;
- rth->rt_mark = fl4->flowi4_mark;
rth->rt_gateway = fl4->daddr;
rth->rt_spec_dst= fl4->saddr;
rth->rt_peer_genid = 0;
@@ -1965,7 +1961,6 @@ struct dst_entry *ipv4_blackhole_route(struct net *net, struct dst_entry *dst_or
rt->rt_route_iif = ort->rt_route_iif;
rt->rt_iif = ort->rt_iif;
rt->rt_oif = ort->rt_oif;
- rt->rt_mark = ort->rt_mark;
rt->rt_genid = rt_genid(net);
rt->rt_flags = ort->rt_flags;
@@ -2057,8 +2052,8 @@ static int rt_fill_info(struct net *net, __be32 src, struct flowi4 *fl4,
if (rtnetlink_put_metrics(skb, dst_metrics_ptr(&rt->dst)) < 0)
goto nla_put_failure;
- if (rt->rt_mark)
- NLA_PUT_BE32(skb, RTA_MARK, rt->rt_mark);
+ if (fl4->flowi4_mark)
+ NLA_PUT_BE32(skb, RTA_MARK, fl4->flowi4_mark);
error = rt->dst.error;
if (peer) {
diff --git a/net/ipv4/xfrm4_policy.c b/net/ipv4/xfrm4_policy.c
index 210da0c..677a317 100644
--- a/net/ipv4/xfrm4_policy.c
+++ b/net/ipv4/xfrm4_policy.c
@@ -82,7 +82,6 @@ static int xfrm4_fill_dst(struct xfrm_dst *xdst, struct net_device *dev,
xdst->u.rt.rt_route_iif = fl4->flowi4_iif;
xdst->u.rt.rt_iif = fl4->flowi4_iif;
xdst->u.rt.rt_oif = fl4->flowi4_oif;
- xdst->u.rt.rt_mark = fl4->flowi4_mark;
xdst->u.dst.dev = dev;
dev_hold(dev);
--
1.7.6.401.g6a319
^ permalink raw reply related
* tcp_mtu_probe implementation details
From: Anatoly Sivov @ 2011-12-20 21:49 UTC (permalink / raw)
To: netdev
Hi all,
Looking on TCP implementation in Linux I see that there are some things
that I don't understand in tcp_mtu_probe() implementation.
Could someone clear them for me or share the link to Linux PMTUD
documentation, please?
The first thing I don't understand is tp->snd_cwnd < 11 in
if (!icsk->icsk_mtup.enabled ||
icsk->icsk_mtup.probe_size ||
inet_csk(sk)->icsk_ca_state != TCP_CA_Open ||
tp->snd_cwnd < 11 ||
tp->rx_opt.num_sacks || tp->rx_opt.dsack)
return -1;
What is magic number "11" here?
The other question is about size_needed variable.
It is assigned to value probe_size + (tp->reordering + 1) * tp->mss_cache
And that is not clear for me.
What is this "(tp->reordering + 1) * tp->mss_cache" addition?
Thanks.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox