* Re: [PATCH v2 1/3] ipv6: Select fragment id during UFO segmentation if not set.
From: Michael S. Tsirkin @ 2015-01-31 17:11 UTC (permalink / raw)
To: Vladislav Yasevich; +Cc: eric.dumazet, netdev, virtualization, hannes, ben
In-Reply-To: <1422646047-13168-2-git-send-email-vyasevic@redhat.com>
On Fri, Jan 30, 2015 at 02:27:25PM -0500, Vladislav Yasevich wrote:
> If the IPv6 fragment id has not been set and we perform
> fragmentation due to UFO, select a new fragment id.
> We now consider a fragment id of 0 as unset and if id selection
> process returns 0 (after all the pertrubations), we set it to
> 0x80000000, thus giving us ample space not to create collisions
> with the next packet we may have to fragment.
>
> When doing UFO integrity checking, we also select the
> fragment id if it has not be set yet. This is stored into
s/be/been/
> the skb_shinfo() thus allowing UFO to function correclty.
>
> This patch also removes duplicate fragment id generation code
> and moves ipv6_select_ident() into the header as it may be
> used during GSO.
>
> Signed-off-by: Vladislav Yasevich <vyasevic@redhat.com>
Acked-by: Michael S. Tsirkin <mst@redhat.com>
> ---
> include/net/ipv6.h | 3 +++
> net/ipv6/ip6_output.c | 10 ++++------
> net/ipv6/output_core.c | 28 ++++++++++++++++++++++------
> net/ipv6/udp_offload.c | 10 +++++++++-
> 4 files changed, 38 insertions(+), 13 deletions(-)
>
> diff --git a/include/net/ipv6.h b/include/net/ipv6.h
> index 4292929..9bf85d3 100644
> --- a/include/net/ipv6.h
> +++ b/include/net/ipv6.h
> @@ -671,6 +671,9 @@ static inline int ipv6_addr_diff(const struct in6_addr *a1, const struct in6_add
> return __ipv6_addr_diff(a1, a2, sizeof(struct in6_addr));
> }
>
> +u32 __ipv6_select_ident(u32 hashrnd, struct in6_addr *dst,
> + struct in6_addr *src);
> +void ipv6_select_ident(struct frag_hdr *fhdr, struct rt6_info *rt);
> void ipv6_proxy_select_ident(struct sk_buff *skb);
>
> int ip6_dst_hoplimit(struct dst_entry *dst);
> diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
> index ce69a12..5dc91fe 100644
> --- a/net/ipv6/ip6_output.c
> +++ b/net/ipv6/ip6_output.c
> @@ -537,17 +537,15 @@ static void ip6_copy_metadata(struct sk_buff *to, struct sk_buff *from)
> skb_copy_secmark(to, from);
> }
>
> -static void ipv6_select_ident(struct frag_hdr *fhdr, struct rt6_info *rt)
> +void ipv6_select_ident(struct frag_hdr *fhdr, struct rt6_info *rt)
> {
> static u32 ip6_idents_hashrnd __read_mostly;
> - u32 hash, id;
> + u32 id;
>
> net_get_random_once(&ip6_idents_hashrnd, sizeof(ip6_idents_hashrnd));
>
> - hash = __ipv6_addr_jhash(&rt->rt6i_dst.addr, ip6_idents_hashrnd);
> - hash = __ipv6_addr_jhash(&rt->rt6i_src.addr, hash);
> -
> - id = ip_idents_reserve(hash, 1);
> + id = __ipv6_select_ident(ip6_idents_hashrnd, &rt->rt6i_dst.addr,
> + &rt->rt6i_src.addr);
> fhdr->identification = htonl(id);
> }
>
> diff --git a/net/ipv6/output_core.c b/net/ipv6/output_core.c
> index 97f41a3..ceb857d 100644
> --- a/net/ipv6/output_core.c
> +++ b/net/ipv6/output_core.c
> @@ -9,6 +9,24 @@
> #include <net/addrconf.h>
> #include <net/secure_seq.h>
>
> +u32 __ipv6_select_ident(u32 hashrnd, struct in6_addr *dst, struct in6_addr *src)
> +{
> + u32 hash, id;
> +
> + hash = __ipv6_addr_jhash(dst, hashrnd);
> + hash = __ipv6_addr_jhash(src, hash);
> +
> + /* Treat id of 0 as unset and if we get 0 back from ip_idents_reserve,
> + * set the hight order instead thus minimizing possible future
> + * collisions.
> + */
> + id = ip_idents_reserve(hash, 1);
> + if (unlikely(!id))
> + id = 1 << 31;
> +
> + return id;
> +}
> +
> /* This function exists only for tap drivers that must support broken
> * clients requesting UFO without specifying an IPv6 fragment ID.
> *
> @@ -22,7 +40,7 @@ void ipv6_proxy_select_ident(struct sk_buff *skb)
> static u32 ip6_proxy_idents_hashrnd __read_mostly;
> struct in6_addr buf[2];
> struct in6_addr *addrs;
> - u32 hash, id;
> + u32 id;
>
> addrs = skb_header_pointer(skb,
> skb_network_offset(skb) +
> @@ -34,11 +52,9 @@ void ipv6_proxy_select_ident(struct sk_buff *skb)
> net_get_random_once(&ip6_proxy_idents_hashrnd,
> sizeof(ip6_proxy_idents_hashrnd));
>
> - hash = __ipv6_addr_jhash(&addrs[1], ip6_proxy_idents_hashrnd);
> - hash = __ipv6_addr_jhash(&addrs[0], hash);
> -
> - id = ip_idents_reserve(hash, 1);
> - skb_shinfo(skb)->ip6_frag_id = htonl(id);
> + id = __ipv6_select_ident(ip6_proxy_idents_hashrnd,
> + &addrs[1], &addrs[0]);
> + skb_shinfo(skb)->ip6_frag_id = id;
> }
> EXPORT_SYMBOL_GPL(ipv6_proxy_select_ident);
>
> diff --git a/net/ipv6/udp_offload.c b/net/ipv6/udp_offload.c
> index b6aa8ed..a562769 100644
> --- a/net/ipv6/udp_offload.c
> +++ b/net/ipv6/udp_offload.c
> @@ -52,6 +52,10 @@ static struct sk_buff *udp6_ufo_fragment(struct sk_buff *skb,
>
> skb_shinfo(skb)->gso_segs = DIV_ROUND_UP(skb->len, mss);
>
> + /* Set the IPv6 fragment id if not set yet */
> + if (!skb_shinfo(skb)->ip6_frag_id)
> + ipv6_proxy_select_ident(skb);
> +
> segs = NULL;
> goto out;
> }
> @@ -108,7 +112,11 @@ static struct sk_buff *udp6_ufo_fragment(struct sk_buff *skb,
> fptr = (struct frag_hdr *)(skb_network_header(skb) + unfrag_ip6hlen);
> fptr->nexthdr = nexthdr;
> fptr->reserved = 0;
> - fptr->identification = skb_shinfo(skb)->ip6_frag_id;
> + if (skb_shinfo(skb)->ip6_frag_id)
> + fptr->identification = skb_shinfo(skb)->ip6_frag_id;
> + else
> + ipv6_select_ident(fptr,
> + (struct rt6_info *)skb_dst(skb));
>
> /* Fragment the skb. ipv6 header and the remaining fields of the
> * fragment header are updated in ipv6_gso_segment()
> --
> 1.9.3
^ permalink raw reply
* [PATCH] net: sched: One function call less in em_meta_change() after error detection
From: SF Markus Elfring @ 2015-01-31 16:34 UTC (permalink / raw)
To: David S. Miller, Jamal Hadi Salim, netdev
Cc: LKML, kernel-janitors, Julia Lawall
In-Reply-To: <5317A59D.4@users.sourceforge.net>
From: Markus Elfring <elfring@users.sourceforge.net>
Date: Sat, 31 Jan 2015 17:18:48 +0100
The meta_delete() function could be called in four cases by the
em_meta_change() function during error handling even if the passed
variable "meta" contained still a null pointer.
* This implementation detail could be improved by adjustments for jump labels.
* Let us return immediately after the first failed function call according to
the current Linux coding style convention.
* Let us delete also unnecessary checks for the variables "err" and
"meta" there.
Signed-off-by: Markus Elfring <elfring@users.sourceforge.net>
---
net/sched/em_meta.c | 20 ++++++++++----------
1 file changed, 10 insertions(+), 10 deletions(-)
diff --git a/net/sched/em_meta.c b/net/sched/em_meta.c
index b5294ce..2aa67b1 100644
--- a/net/sched/em_meta.c
+++ b/net/sched/em_meta.c
@@ -866,23 +866,23 @@ static int em_meta_change(struct net *net, void *data, int len,
err = nla_parse(tb, TCA_EM_META_MAX, data, len, meta_policy);
if (err < 0)
- goto errout;
+ return err;
err = -EINVAL;
if (tb[TCA_EM_META_HDR] == NULL)
- goto errout;
+ goto exit;
hdr = nla_data(tb[TCA_EM_META_HDR]);
if (TCF_META_TYPE(hdr->left.kind) != TCF_META_TYPE(hdr->right.kind) ||
TCF_META_TYPE(hdr->left.kind) > TCF_META_TYPE_MAX ||
TCF_META_ID(hdr->left.kind) > TCF_META_ID_MAX ||
TCF_META_ID(hdr->right.kind) > TCF_META_ID_MAX)
- goto errout;
+ goto exit;
meta = kzalloc(sizeof(*meta), GFP_KERNEL);
if (meta == NULL) {
err = -ENOMEM;
- goto errout;
+ goto exit;
}
memcpy(&meta->lvalue.hdr, &hdr->left, sizeof(hdr->left));
@@ -891,20 +891,20 @@ static int em_meta_change(struct net *net, void *data, int len,
if (!meta_is_supported(&meta->lvalue) ||
!meta_is_supported(&meta->rvalue)) {
err = -EOPNOTSUPP;
- goto errout;
+ goto delete_meta;
}
if (meta_change_data(&meta->lvalue, tb[TCA_EM_META_LVALUE]) < 0 ||
meta_change_data(&meta->rvalue, tb[TCA_EM_META_RVALUE]) < 0)
- goto errout;
+ goto delete_meta;
m->datalen = sizeof(*meta);
m->data = (unsigned long) meta;
- err = 0;
-errout:
- if (err && meta)
- meta_delete(meta);
+ return 0;
+delete_meta:
+ meta_delete(meta);
+exit:
return err;
}
--
2.2.2
^ permalink raw reply related
* [PATCH net-next 6/6] ipv6: Allow for partial checksums on non-ufo packets
From: Vladislav Yasevich @ 2015-01-31 15:40 UTC (permalink / raw)
To: netdev; +Cc: herbert, hannes, Vladislav Yasevich
In-Reply-To: <1422718818-21093-1-git-send-email-vyasevic@redhat.com>
Currntly, if we are not doing UFO on the packet, all UDP
packets will start with CHECKSUM_NONE and thus perform full
checksum computations in software even if device support
IPv6 checksum offloading.
Let's start start with CHECKSUM_PARTIAL if the device
supports it and we are sending only a single packet at
or below mtu size.
Signed-off-by: Vladislav Yasevich <vyasevic@redhat.com>
---
net/ipv6/ip6_output.c | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
index b89d3c2..1a036f3 100644
--- a/net/ipv6/ip6_output.c
+++ b/net/ipv6/ip6_output.c
@@ -1227,6 +1227,7 @@ static int __ip6_append_data(struct sock *sk,
u32 tskey = 0;
struct rt6_info *rt = (struct rt6_info *)cork->dst;
struct ipv6_txoptions *opt = v6_cork->opt;
+ int csummode = CHECKSUM_NONE;
skb = skb_peek_tail(queue);
if (!skb) {
@@ -1283,6 +1284,14 @@ emsgsize:
tskey = sk->sk_tskey++;
}
+ /* If this is the first and only packet and device
+ * supports checksum offloading, let's use it.
+ */
+ if (!skb &&
+ length + fragheaderlen < mtu &&
+ rt->dst.dev->features & NETIF_F_V6_CSUM &&
+ !exthdrlen)
+ csummode = CHECKSUM_PARTIAL;
/*
* Let's try using as much space as possible.
* Use MTU if total length of the message fits into the MTU.
@@ -1395,7 +1404,7 @@ alloc_new_skb:
* Fill in the control structures
*/
skb->protocol = htons(ETH_P_IPV6);
- skb->ip_summed = CHECKSUM_NONE;
+ skb->ip_summed = csummode;
skb->csum = 0;
/* reserve for fragmentation and ipsec header */
skb_reserve(skb, hh_len + sizeof(struct frag_hdr) +
--
1.9.3
^ permalink raw reply related
* [PATCH net-next 5/6] udpv6: Add lockless sendmsg() support
From: Vladislav Yasevich @ 2015-01-31 15:40 UTC (permalink / raw)
To: netdev; +Cc: herbert, hannes, Vladislav Yasevich
In-Reply-To: <1422718818-21093-1-git-send-email-vyasevic@redhat.com>
This commit adds the same functionaliy to IPv6 that
commit 903ab86d195cca295379699299c5fc10beba31c7
Author: Herbert Xu <herbert@gondor.apana.org.au>
Date: Tue Mar 1 02:36:48 2011 +0000
udp: Add lockless transmit path
added to IPv4.
UDP transmit path can now run without a socket lock,
thus allowing multiple threads to send to a single socket
more efficiently.
This is only used when corking/MSG_MORE is not used.
Signed-off-by: Vladislav Yasevich <vyasevic@redhat.com>
---
net/ipv6/udp.c | 24 ++++++++++++++++++++----
1 file changed, 20 insertions(+), 4 deletions(-)
diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c
index 67a3d70..d048d46 100644
--- a/net/ipv6/udp.c
+++ b/net/ipv6/udp.c
@@ -1177,6 +1177,7 @@ do_udp_sendmsg:
if (len > INT_MAX - sizeof(struct udphdr))
return -EMSGSIZE;
+ getfrag = is_udplite ? udplite_getfrag : ip_generic_getfrag;
if (up->pending) {
/*
* There are pending frames.
@@ -1307,6 +1308,20 @@ do_udp_sendmsg:
goto do_confirm;
back_from_confirm:
+ /* Lockless fast path for the non-corking case */
+ if (!corkreq) {
+ struct sk_buff *skb;
+
+ skb = ip6_make_skb(sk, getfrag, msg, ulen,
+ sizeof(struct udphdr), hlimit, tclass, opt,
+ &fl6, (struct rt6_info *)dst,
+ msg->msg_flags, dontfrag);
+ err = PTR_ERR(skb);
+ if (!IS_ERR_OR_NULL(skb))
+ err = udp_v6_send_skb(skb, &fl6);
+ goto release_dst;
+ }
+
lock_sock(sk);
if (unlikely(up->pending)) {
/* The socket is already corked while preparing it. */
@@ -1324,7 +1339,6 @@ do_append_data:
if (dontfrag < 0)
dontfrag = np->dontfrag;
up->len += ulen;
- getfrag = is_udplite ? udplite_getfrag : ip_generic_getfrag;
err = ip6_append_data(sk, getfrag, msg, ulen,
sizeof(struct udphdr), hlimit, tclass, opt, &fl6,
(struct rt6_info *)dst,
@@ -1336,6 +1350,11 @@ do_append_data:
else if (unlikely(skb_queue_empty(&sk->sk_write_queue)))
up->pending = 0;
+ if (err > 0)
+ err = np->recverr ? net_xmit_errno(err) : 0;
+ release_sock(sk);
+
+release_dst:
if (dst) {
if (connected) {
ip6_dst_store(sk, dst,
@@ -1352,9 +1371,6 @@ do_append_data:
dst = NULL;
}
- if (err > 0)
- err = np->recverr ? net_xmit_errno(err) : 0;
- release_sock(sk);
out:
dst_release(dst);
fl6_sock_release(flowlabel);
--
1.9.3
^ permalink raw reply related
* [PATCH net-next 4/6] ipv6: Introduce udpv6_send_skb()
From: Vladislav Yasevich @ 2015-01-31 15:40 UTC (permalink / raw)
To: netdev; +Cc: herbert, hannes, Vladislav Yasevich
In-Reply-To: <1422718818-21093-1-git-send-email-vyasevic@redhat.com>
Now that we can individually construct IPv6 skbs to send, add a
udpv6_send_skb() function to populate the udp header and send the
skb. This allows udp_v6_push_pending_frames() to re-use this
function as well as enables us to add lockless sendmsg() support.
Signed-off-by: Vladislav Yasevich <vyasevic@redhat.com>
---
net/ipv6/udp.c | 67 +++++++++++++++++++++++++++++++++++-----------------------
1 file changed, 40 insertions(+), 27 deletions(-)
diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c
index e41f017..67a3d70 100644
--- a/net/ipv6/udp.c
+++ b/net/ipv6/udp.c
@@ -990,9 +990,10 @@ static void udp6_hwcsum_outgoing(struct sock *sk, struct sk_buff *skb,
{
unsigned int offset;
struct udphdr *uh = udp_hdr(skb);
+ struct sk_buff *frags = skb_shinfo(skb)->frag_list;
__wsum csum = 0;
- if (skb_queue_len(&sk->sk_write_queue) == 1) {
+ if (!frags) {
/* Only one fragment on the socket. */
skb->csum_start = skb_transport_header(skb) - skb->head;
skb->csum_offset = offsetof(struct udphdr, check);
@@ -1008,9 +1009,9 @@ static void udp6_hwcsum_outgoing(struct sock *sk, struct sk_buff *skb,
skb->ip_summed = CHECKSUM_NONE;
- skb_queue_walk(&sk->sk_write_queue, skb) {
- csum = csum_add(csum, skb->csum);
- }
+ do {
+ csum = csum_add(csum, frags->csum);
+ } while ((frags = frags->next));
uh->check = csum_ipv6_magic(saddr, daddr, len, IPPROTO_UDP,
csum);
@@ -1023,26 +1024,15 @@ static void udp6_hwcsum_outgoing(struct sock *sk, struct sk_buff *skb,
* Sending
*/
-static int udp_v6_push_pending_frames(struct sock *sk)
+static int udp_v6_send_skb(struct sk_buff *skb, struct flowi6 *fl6)
{
- struct sk_buff *skb;
+ struct sock *sk = skb->sk;
struct udphdr *uh;
- struct udp_sock *up = udp_sk(sk);
- struct inet_sock *inet = inet_sk(sk);
- struct flowi6 *fl6;
int err = 0;
int is_udplite = IS_UDPLITE(sk);
__wsum csum = 0;
-
- if (up->pending == AF_INET)
- return udp_push_pending_frames(sk);
-
- fl6 = &inet->cork.fl.u.ip6;
-
- /* Grab the skbuff where UDP header space exists. */
- skb = skb_peek(&sk->sk_write_queue);
- if (skb == NULL)
- goto out;
+ int offset = skb_transport_offset(skb);
+ int len = skb->len - offset;
/*
* Create a UDP header
@@ -1050,29 +1040,28 @@ static int udp_v6_push_pending_frames(struct sock *sk)
uh = udp_hdr(skb);
uh->source = fl6->fl6_sport;
uh->dest = fl6->fl6_dport;
- uh->len = htons(up->len);
+ uh->len = htons(len);
uh->check = 0;
if (is_udplite)
- csum = udplite_csum_outgoing(sk, skb);
- else if (up->no_check6_tx) { /* UDP csum disabled */
+ csum = udplite_csum(skb);
+ else if (udp_sk(sk)->no_check6_tx) { /* UDP csum disabled */
skb->ip_summed = CHECKSUM_NONE;
goto send;
} else if (skb->ip_summed == CHECKSUM_PARTIAL) { /* UDP hardware csum */
- udp6_hwcsum_outgoing(sk, skb, &fl6->saddr, &fl6->daddr,
- up->len);
+ udp6_hwcsum_outgoing(sk, skb, &fl6->saddr, &fl6->daddr, len);
goto send;
} else
- csum = udp_csum_outgoing(sk, skb);
+ csum = udp_csum(skb);
/* add protocol-dependent pseudo-header */
uh->check = csum_ipv6_magic(&fl6->saddr, &fl6->daddr,
- up->len, fl6->flowi6_proto, csum);
+ len, fl6->flowi6_proto, csum);
if (uh->check == 0)
uh->check = CSUM_MANGLED_0;
send:
- err = ip6_push_pending_frames(sk);
+ err = ip6_send_skb(skb);
if (err) {
if (err == -ENOBUFS && !inet6_sk(sk)->recverr) {
UDP6_INC_STATS_USER(sock_net(sk),
@@ -1082,6 +1071,30 @@ send:
} else
UDP6_INC_STATS_USER(sock_net(sk),
UDP_MIB_OUTDATAGRAMS, is_udplite);
+ return err;
+}
+
+static int udp_v6_push_pending_frames(struct sock *sk)
+{
+ struct sk_buff *skb;
+ struct udp_sock *up = udp_sk(sk);
+ struct flowi6 fl6;
+ int err = 0;
+
+ if (up->pending == AF_INET)
+ return udp_push_pending_frames(sk);
+
+ /* ip6_finish_skb will release the cork, so make a copy of
+ * fl6 here.
+ */
+ fl6 = inet_sk(sk)->cork.fl.u.ip6;
+
+ skb = ip6_finish_skb(sk);
+ if (!skb)
+ goto out;
+
+ err = udp_v6_send_skb(skb, &fl6);
+
out:
up->len = 0;
up->pending = 0;
--
1.9.3
^ permalink raw reply related
* [PATCH net-next 3/6] ipv6: introduce ipv6_make_skb
From: Vladislav Yasevich @ 2015-01-31 15:40 UTC (permalink / raw)
To: netdev; +Cc: herbert, hannes, Vladislav Yasevich
In-Reply-To: <1422718818-21093-1-git-send-email-vyasevic@redhat.com>
This commit is very similar to
commit 1c32c5ad6fac8cee1a77449f5abf211e911ff830
Author: Herbert Xu <herbert@gondor.apana.org.au>
Date: Tue Mar 1 02:36:47 2011 +0000
inet: Add ip_make_skb and ip_finish_skb
It adds IPv6 version of the helpers ip6_make_skb and ip6_finish_skb.
The job of ip6_make_skb is to collect messages into an ipv6 packet
and poplulate ipv6 eader. The job of ip6_finish_skb is to transmit
the generated skb. Together they replicated the job of
ip6_push_pending_frames() while also provide the capability to be
called independently. This will be needed to add lockless UDP sendmsg
support.
Signed-off-by: Vladislav Yasevich <vyasevic@redhat.com>
---
include/net/ipv6.h | 19 ++++++++++
net/ipv6/ip6_output.c | 103 ++++++++++++++++++++++++++++++++++++++++----------
2 files changed, 103 insertions(+), 19 deletions(-)
diff --git a/include/net/ipv6.h b/include/net/ipv6.h
index 4292929..8027ca5 100644
--- a/include/net/ipv6.h
+++ b/include/net/ipv6.h
@@ -788,6 +788,25 @@ int ip6_push_pending_frames(struct sock *sk);
void ip6_flush_pending_frames(struct sock *sk);
+int ip6_send_skb(struct sk_buff *skb);
+
+struct sk_buff *__ip6_make_skb(struct sock *sk, struct sk_buff_head *queue,
+ struct inet_cork_full *cork,
+ struct inet6_cork *v6_cork);
+struct sk_buff *ip6_make_skb(struct sock *sk,
+ int getfrag(void *from, char *to, int offset,
+ int len, int odd, struct sk_buff *skb),
+ void *from, int length, int transhdrlen,
+ int hlimit, int tclass, struct ipv6_txoptions *opt,
+ struct flowi6 *fl6, struct rt6_info *rt,
+ unsigned int flags, int dontfrag);
+
+static inline struct sk_buff *ip6_finish_skb(struct sock *sk)
+{
+ return __ip6_make_skb(sk, &sk->sk_write_queue, &inet_sk(sk)->cork,
+ &inet6_sk(sk)->cork);
+}
+
int ip6_dst_lookup(struct sock *sk, struct dst_entry **dst, struct flowi6 *fl6);
struct dst_entry *ip6_dst_lookup_flow(struct sock *sk, struct flowi6 *fl6,
const struct in6_addr *final_dst);
diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
index 1b66453..b89d3c2 100644
--- a/net/ipv6/ip6_output.c
+++ b/net/ipv6/ip6_output.c
@@ -1564,22 +1564,23 @@ static void ip6_cork_release(struct inet_cork_full *cork,
memset(&cork->fl, 0, sizeof(cork->fl));
}
-int ip6_push_pending_frames(struct sock *sk)
+struct sk_buff *__ip6_make_skb(struct sock *sk,
+ struct sk_buff_head *queue,
+ struct inet_cork_full *cork,
+ struct inet6_cork *v6_cork)
{
struct sk_buff *skb, *tmp_skb;
struct sk_buff **tail_skb;
struct in6_addr final_dst_buf, *final_dst = &final_dst_buf;
- struct inet_sock *inet = inet_sk(sk);
struct ipv6_pinfo *np = inet6_sk(sk);
struct net *net = sock_net(sk);
struct ipv6hdr *hdr;
- struct ipv6_txoptions *opt = np->cork.opt;
- struct rt6_info *rt = (struct rt6_info *)inet->cork.base.dst;
- struct flowi6 *fl6 = &inet->cork.fl.u.ip6;
+ struct ipv6_txoptions *opt = v6_cork->opt;
+ struct rt6_info *rt = (struct rt6_info *)cork->base.dst;
+ struct flowi6 *fl6 = &cork->fl.u.ip6;
unsigned char proto = fl6->flowi6_proto;
- int err = 0;
- skb = __skb_dequeue(&sk->sk_write_queue);
+ skb = __skb_dequeue(queue);
if (skb == NULL)
goto out;
tail_skb = &(skb_shinfo(skb)->frag_list);
@@ -1587,7 +1588,7 @@ int ip6_push_pending_frames(struct sock *sk)
/* move skb->data to ip header from ext header */
if (skb->data < skb_network_header(skb))
__skb_pull(skb, skb_network_offset(skb));
- while ((tmp_skb = __skb_dequeue(&sk->sk_write_queue)) != NULL) {
+ while ((tmp_skb = __skb_dequeue(queue)) != NULL) {
__skb_pull(tmp_skb, skb_network_header_len(skb));
*tail_skb = tmp_skb;
tail_skb = &(tmp_skb->next);
@@ -1612,10 +1613,10 @@ int ip6_push_pending_frames(struct sock *sk)
skb_reset_network_header(skb);
hdr = ipv6_hdr(skb);
- ip6_flow_hdr(hdr, np->cork.tclass,
+ ip6_flow_hdr(hdr, v6_cork->tclass,
ip6_make_flowlabel(net, skb, fl6->flowlabel,
np->autoflowlabel));
- hdr->hop_limit = np->cork.hop_limit;
+ hdr->hop_limit = v6_cork->hop_limit;
hdr->nexthdr = proto;
hdr->saddr = fl6->saddr;
hdr->daddr = *final_dst;
@@ -1632,25 +1633,45 @@ int ip6_push_pending_frames(struct sock *sk)
ICMP6_INC_STATS(net, idev, ICMP6_MIB_OUTMSGS);
}
+ ip6_cork_release(cork, v6_cork);
+out:
+ return skb;
+}
+
+int ip6_send_skb(struct sk_buff *skb)
+{
+ struct net *net = sock_net(skb->sk);
+ struct rt6_info *rt = (struct rt6_info *)skb_dst(skb);
+ int err;
+
err = ip6_local_out(skb);
if (err) {
if (err > 0)
err = net_xmit_errno(err);
if (err)
- goto error;
+ IP6_INC_STATS(net, rt->rt6i_idev,
+ IPSTATS_MIB_OUTDISCARDS);
}
-out:
- ip6_cork_release(&inet->cork, &np->cork);
return err;
-error:
- IP6_INC_STATS(net, rt->rt6i_idev, IPSTATS_MIB_OUTDISCARDS);
- goto out;
+}
+
+int ip6_push_pending_frames(struct sock *sk)
+{
+ struct sk_buff *skb;
+
+ skb = ip6_finish_skb(sk);
+ if (!skb)
+ return 0;
+
+ return ip6_send_skb(skb);
}
EXPORT_SYMBOL_GPL(ip6_push_pending_frames);
static void __ip6_flush_pending_frames(struct sock *sk,
- struct sk_buff_head *queue)
+ struct sk_buff_head *queue,
+ struct inet_cork_full *cork,
+ struct inet6_cork *v6_cork)
{
struct sk_buff *skb;
@@ -1661,11 +1682,55 @@ static void __ip6_flush_pending_frames(struct sock *sk,
kfree_skb(skb);
}
- ip6_cork_release(&inet_sk(sk)->cork, &inet6_sk(sk)->cork);
+ ip6_cork_release(cork, v6_cork);
}
void ip6_flush_pending_frames(struct sock *sk)
{
- __ip6_flush_pending_frames(sk, &sk->sk_write_queue);
+ __ip6_flush_pending_frames(sk, &sk->sk_write_queue,
+ &inet_sk(sk)->cork, &inet6_sk(sk)->cork);
}
EXPORT_SYMBOL_GPL(ip6_flush_pending_frames);
+
+struct sk_buff *ip6_make_skb(struct sock *sk,
+ int getfrag(void *from, char *to, int offset,
+ int len, int odd, struct sk_buff *skb),
+ void *from, int length, int transhdrlen,
+ int hlimit, int tclass,
+ struct ipv6_txoptions *opt, struct flowi6 *fl6,
+ struct rt6_info *rt, unsigned int flags,
+ int dontfrag)
+{
+ struct inet_cork_full cork;
+ struct inet6_cork v6_cork;
+ struct sk_buff_head queue;
+ int exthdrlen = (opt ? opt->opt_flen : 0);
+ int err;
+
+ if (flags & MSG_PROBE)
+ return NULL;
+
+ __skb_queue_head_init(&queue);
+
+ cork.base.flags = 0;
+ cork.base.addr = 0;
+ cork.base.opt = NULL;
+ v6_cork.opt = NULL;
+ err = ip6_setup_cork(sk, &cork, &v6_cork, hlimit, tclass, opt, rt, fl6);
+ if (err)
+ return ERR_PTR(err);
+
+ if (dontfrag < 0)
+ dontfrag = inet6_sk(sk)->dontfrag;
+
+ err = __ip6_append_data(sk, fl6, &queue, &cork.base, &v6_cork,
+ ¤t->task_frag, getfrag, from,
+ length + exthdrlen, transhdrlen + exthdrlen,
+ flags, dontfrag);
+ if (err) {
+ __ip6_flush_pending_frames(sk, &queue, &cork, &v6_cork);
+ return ERR_PTR(err);
+ }
+
+ return __ip6_make_skb(sk, &queue, &cork, &v6_cork);
+}
--
1.9.3
^ permalink raw reply related
* [PATCH net-next 2/6] ipv6: Append sending data to arbitrary queue
From: Vladislav Yasevich @ 2015-01-31 15:40 UTC (permalink / raw)
To: netdev; +Cc: herbert, hannes, Vladislav Yasevich
In-Reply-To: <1422718818-21093-1-git-send-email-vyasevic@redhat.com>
Add the ability to append data to arbitrary queue. This
will be needed later to implement lockless UDP sends.
Signed-off-by: Vladislav Yasevich <vyasevic@redhat.com>
---
net/ipv6/ip6_output.c | 106 +++++++++++++++++++++++++++++++-------------------
1 file changed, 67 insertions(+), 39 deletions(-)
diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
index f9f08c4..1b66453 100644
--- a/net/ipv6/ip6_output.c
+++ b/net/ipv6/ip6_output.c
@@ -1041,6 +1041,7 @@ struct dst_entry *ip6_sk_dst_lookup_flow(struct sock *sk, struct flowi6 *fl6,
EXPORT_SYMBOL_GPL(ip6_sk_dst_lookup_flow);
static inline int ip6_ufo_append_data(struct sock *sk,
+ struct sk_buff_head *queue,
int getfrag(void *from, char *to, int offset, int len,
int odd, struct sk_buff *skb),
void *from, int length, int hh_len, int fragheaderlen,
@@ -1056,7 +1057,7 @@ static inline int ip6_ufo_append_data(struct sock *sk,
* device, so create one single skb packet containing complete
* udp datagram
*/
- skb = skb_peek_tail(&sk->sk_write_queue);
+ skb = skb_peek_tail(queue);
if (skb == NULL) {
skb = sock_alloc_send_skb(sk,
hh_len + fragheaderlen + transhdrlen + 20,
@@ -1079,7 +1080,7 @@ static inline int ip6_ufo_append_data(struct sock *sk,
skb->protocol = htons(ETH_P_IPV6);
skb->csum = 0;
- __skb_queue_tail(&sk->sk_write_queue, skb);
+ __skb_queue_tail(queue, skb);
} else if (skb_is_gso(skb)) {
goto append;
}
@@ -1203,49 +1204,36 @@ static int ip6_setup_cork(struct sock *sk, struct inet_cork_full *cork,
return 0;
}
-int ip6_append_data(struct sock *sk, int getfrag(void *from, char *to,
- int offset, int len, int odd, struct sk_buff *skb),
- void *from, int length, int transhdrlen,
- int hlimit, int tclass, struct ipv6_txoptions *opt, struct flowi6 *fl6,
- struct rt6_info *rt, unsigned int flags, int dontfrag)
+static int __ip6_append_data(struct sock *sk,
+ struct flowi6 *fl6,
+ struct sk_buff_head *queue,
+ struct inet_cork *cork,
+ struct inet6_cork *v6_cork,
+ struct page_frag *pfrag,
+ int getfrag(void *from, char *to, int offset,
+ int len, int odd, struct sk_buff *skb),
+ void *from, int length, int transhdrlen,
+ unsigned int flags, int dontfrag)
{
- struct inet_sock *inet = inet_sk(sk);
- struct ipv6_pinfo *np = inet6_sk(sk);
- struct inet_cork *cork;
struct sk_buff *skb, *skb_prev = NULL;
unsigned int maxfraglen, fragheaderlen, mtu, orig_mtu;
- int exthdrlen;
- int dst_exthdrlen;
+ int exthdrlen = 0;
+ int dst_exthdrlen = 0;
int hh_len;
int copy;
int err;
int offset = 0;
__u8 tx_flags = 0;
u32 tskey = 0;
+ struct rt6_info *rt = (struct rt6_info *)cork->dst;
+ struct ipv6_txoptions *opt = v6_cork->opt;
- if (flags&MSG_PROBE)
- return 0;
- cork = &inet->cork.base;
- if (skb_queue_empty(&sk->sk_write_queue)) {
- /*
- * setup for corking
- */
- err = ip6_setup_cork(sk, &inet->cork, &np->cork, hlimit,
- tclass, opt, rt, fl6);
- if (err)
- return err;
- exthdrlen = (opt ? opt->opt_flen : 0);
- length += exthdrlen;
- transhdrlen += exthdrlen;
+ skb = skb_peek_tail(queue);
+ if (!skb) {
+ exthdrlen = opt ? opt->opt_flen : 0;
dst_exthdrlen = rt->dst.header_len - rt->rt6i_nfheader_len;
- } else {
- rt = (struct rt6_info *)cork->dst;
- fl6 = &inet->cork.fl.u.ip6;
- opt = np->cork.opt;
- transhdrlen = 0;
- exthdrlen = 0;
- dst_exthdrlen = 0;
}
+
mtu = cork->fragsize;
orig_mtu = mtu;
@@ -1311,13 +1299,12 @@ emsgsize:
* --yoshfuji
*/
- skb = skb_peek_tail(&sk->sk_write_queue);
cork->length += length;
if (((length > mtu) ||
(skb && skb_is_gso(skb))) &&
(sk->sk_protocol == IPPROTO_UDP) &&
(rt->dst.dev->features & NETIF_F_UFO)) {
- err = ip6_ufo_append_data(sk, getfrag, from, length,
+ err = ip6_ufo_append_data(sk, queue, getfrag, from, length,
hh_len, fragheaderlen,
transhdrlen, mtu, flags, rt);
if (err)
@@ -1458,7 +1445,7 @@ alloc_new_skb:
/*
* Put the packet on the pending queue
*/
- __skb_queue_tail(&sk->sk_write_queue, skb);
+ __skb_queue_tail(queue, skb);
continue;
}
@@ -1477,7 +1464,6 @@ alloc_new_skb:
}
} else {
int i = skb_shinfo(skb)->nr_frags;
- struct page_frag *pfrag = sk_page_frag(sk);
err = -ENOMEM;
if (!sk_page_frag_refill(sk, pfrag))
@@ -1520,6 +1506,42 @@ error:
IP6_INC_STATS(sock_net(sk), rt->rt6i_idev, IPSTATS_MIB_OUTDISCARDS);
return err;
}
+
+int ip6_append_data(struct sock *sk,
+ int getfrag(void *from, char *to, int offset, int len,
+ int odd, struct sk_buff *skb),
+ void *from, int length, int transhdrlen, int hlimit,
+ int tclass, struct ipv6_txoptions *opt, struct flowi6 *fl6,
+ struct rt6_info *rt, unsigned int flags, int dontfrag)
+{
+ struct inet_sock *inet = inet_sk(sk);
+ struct ipv6_pinfo *np = inet6_sk(sk);
+ int exthdrlen;
+ int err;
+
+ if (flags&MSG_PROBE)
+ return 0;
+ if (skb_queue_empty(&sk->sk_write_queue)) {
+ /*
+ * setup for corking
+ */
+ err = ip6_setup_cork(sk, &inet->cork, &np->cork, hlimit,
+ tclass, opt, rt, fl6);
+ if (err)
+ return err;
+
+ exthdrlen = (opt ? opt->opt_flen : 0);
+ length += exthdrlen;
+ transhdrlen += exthdrlen;
+ } else {
+ fl6 = &inet->cork.fl.u.ip6;
+ transhdrlen = 0;
+ }
+
+ return __ip6_append_data(sk, fl6, &sk->sk_write_queue, &inet->cork.base,
+ &np->cork, sk_page_frag(sk), getfrag,
+ from, length, transhdrlen, flags, dontfrag);
+}
EXPORT_SYMBOL_GPL(ip6_append_data);
static void ip6_cork_release(struct inet_cork_full *cork,
@@ -1627,11 +1649,12 @@ error:
}
EXPORT_SYMBOL_GPL(ip6_push_pending_frames);
-void ip6_flush_pending_frames(struct sock *sk)
+static void __ip6_flush_pending_frames(struct sock *sk,
+ struct sk_buff_head *queue)
{
struct sk_buff *skb;
- while ((skb = __skb_dequeue_tail(&sk->sk_write_queue)) != NULL) {
+ while ((skb = __skb_dequeue_tail(queue)) != NULL) {
if (skb_dst(skb))
IP6_INC_STATS(sock_net(sk), ip6_dst_idev(skb_dst(skb)),
IPSTATS_MIB_OUTDISCARDS);
@@ -1640,4 +1663,9 @@ void ip6_flush_pending_frames(struct sock *sk)
ip6_cork_release(&inet_sk(sk)->cork, &inet6_sk(sk)->cork);
}
+
+void ip6_flush_pending_frames(struct sock *sk)
+{
+ __ip6_flush_pending_frames(sk, &sk->sk_write_queue);
+}
EXPORT_SYMBOL_GPL(ip6_flush_pending_frames);
--
1.9.3
^ permalink raw reply related
* [PATCH net-next 1/6] ipv6: pull cork initialization into its own function.
From: Vladislav Yasevich @ 2015-01-31 15:40 UTC (permalink / raw)
To: netdev; +Cc: herbert, hannes, Vladislav Yasevich
In-Reply-To: <1422718818-21093-1-git-send-email-vyasevic@redhat.com>
Pull IPv6 cork initialization into its own function that
can be re-used. IPv6 specific cork data did not have an
explicit data structure. This patch creats eone so that
just ipv6 cork data can be as arguemts. Also, since
IPv6 tries to save the flow label into inet_cork_full
tructure, pass the full cork.
Adjust ip6_cork_release() to take cork data structures.
Signed-off-by: Vladislav Yasevich <vyasevic@redhat.com>
---
include/linux/ipv6.h | 12 ++--
net/ipv6/ip6_output.c | 158 ++++++++++++++++++++++++++++----------------------
2 files changed, 96 insertions(+), 74 deletions(-)
diff --git a/include/linux/ipv6.h b/include/linux/ipv6.h
index 2805062..4d5169f 100644
--- a/include/linux/ipv6.h
+++ b/include/linux/ipv6.h
@@ -125,6 +125,12 @@ struct ipv6_mc_socklist;
struct ipv6_ac_socklist;
struct ipv6_fl_socklist;
+struct inet6_cork {
+ struct ipv6_txoptions *opt;
+ u8 hop_limit;
+ u8 tclass;
+};
+
/**
* struct ipv6_pinfo - ipv6 private area
*
@@ -217,11 +223,7 @@ struct ipv6_pinfo {
struct ipv6_txoptions *opt;
struct sk_buff *pktoptions;
struct sk_buff *rxpmtu;
- struct {
- struct ipv6_txoptions *opt;
- u8 hop_limit;
- u8 tclass;
- } cork;
+ struct inet6_cork cork;
};
/* WARNING: don't change the layout of the members in {raw,udp,tcp}6_sock! */
diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
index ce69a12..f9f08c4 100644
--- a/net/ipv6/ip6_output.c
+++ b/net/ipv6/ip6_output.c
@@ -1135,6 +1135,74 @@ static void ip6_append_data_mtu(unsigned int *mtu,
}
}
+static int ip6_setup_cork(struct sock *sk, struct inet_cork_full *cork,
+ struct inet6_cork *v6_cork,
+ int hlimit, int tclass, struct ipv6_txoptions *opt,
+ struct rt6_info *rt, struct flowi6 *fl6)
+{
+ struct ipv6_pinfo *np = inet6_sk(sk);
+ unsigned int mtu;
+
+ /*
+ * setup for corking
+ */
+ if (opt) {
+ if (WARN_ON(v6_cork->opt))
+ return -EINVAL;
+
+ v6_cork->opt = kzalloc(opt->tot_len, sk->sk_allocation);
+ if (unlikely(v6_cork->opt == NULL))
+ return -ENOBUFS;
+
+ v6_cork->opt->tot_len = opt->tot_len;
+ v6_cork->opt->opt_flen = opt->opt_flen;
+ v6_cork->opt->opt_nflen = opt->opt_nflen;
+
+ v6_cork->opt->dst0opt = ip6_opt_dup(opt->dst0opt,
+ sk->sk_allocation);
+ if (opt->dst0opt && !v6_cork->opt->dst0opt)
+ return -ENOBUFS;
+
+ v6_cork->opt->dst1opt = ip6_opt_dup(opt->dst1opt,
+ sk->sk_allocation);
+ if (opt->dst1opt && !v6_cork->opt->dst1opt)
+ return -ENOBUFS;
+
+ v6_cork->opt->hopopt = ip6_opt_dup(opt->hopopt,
+ sk->sk_allocation);
+ if (opt->hopopt && !v6_cork->opt->hopopt)
+ return -ENOBUFS;
+
+ v6_cork->opt->srcrt = ip6_rthdr_dup(opt->srcrt,
+ sk->sk_allocation);
+ if (opt->srcrt && !v6_cork->opt->srcrt)
+ return -ENOBUFS;
+
+ /* need source address above miyazawa*/
+ }
+ dst_hold(&rt->dst);
+ cork->base.dst = &rt->dst;
+ cork->fl.u.ip6 = *fl6;
+ v6_cork->hop_limit = hlimit;
+ v6_cork->tclass = tclass;
+ if (rt->dst.flags & DST_XFRM_TUNNEL)
+ mtu = np->pmtudisc >= IPV6_PMTUDISC_PROBE ?
+ rt->dst.dev->mtu : dst_mtu(&rt->dst);
+ else
+ mtu = np->pmtudisc >= IPV6_PMTUDISC_PROBE ?
+ rt->dst.dev->mtu : dst_mtu(rt->dst.path);
+ if (np->frag_size < mtu) {
+ if (np->frag_size)
+ mtu = np->frag_size;
+ }
+ cork->base.fragsize = mtu;
+ if (dst_allfrag(rt->dst.path))
+ cork->base.flags |= IPCORK_ALLFRAG;
+ cork->base.length = 0;
+
+ return 0;
+}
+
int ip6_append_data(struct sock *sk, int getfrag(void *from, char *to,
int offset, int len, int odd, struct sk_buff *skb),
void *from, int length, int transhdrlen,
@@ -1162,59 +1230,10 @@ int ip6_append_data(struct sock *sk, int getfrag(void *from, char *to,
/*
* setup for corking
*/
- if (opt) {
- if (WARN_ON(np->cork.opt))
- return -EINVAL;
-
- np->cork.opt = kzalloc(opt->tot_len, sk->sk_allocation);
- if (unlikely(np->cork.opt == NULL))
- return -ENOBUFS;
-
- np->cork.opt->tot_len = opt->tot_len;
- np->cork.opt->opt_flen = opt->opt_flen;
- np->cork.opt->opt_nflen = opt->opt_nflen;
-
- np->cork.opt->dst0opt = ip6_opt_dup(opt->dst0opt,
- sk->sk_allocation);
- if (opt->dst0opt && !np->cork.opt->dst0opt)
- return -ENOBUFS;
-
- np->cork.opt->dst1opt = ip6_opt_dup(opt->dst1opt,
- sk->sk_allocation);
- if (opt->dst1opt && !np->cork.opt->dst1opt)
- return -ENOBUFS;
-
- np->cork.opt->hopopt = ip6_opt_dup(opt->hopopt,
- sk->sk_allocation);
- if (opt->hopopt && !np->cork.opt->hopopt)
- return -ENOBUFS;
-
- np->cork.opt->srcrt = ip6_rthdr_dup(opt->srcrt,
- sk->sk_allocation);
- if (opt->srcrt && !np->cork.opt->srcrt)
- return -ENOBUFS;
-
- /* need source address above miyazawa*/
- }
- dst_hold(&rt->dst);
- cork->dst = &rt->dst;
- inet->cork.fl.u.ip6 = *fl6;
- np->cork.hop_limit = hlimit;
- np->cork.tclass = tclass;
- if (rt->dst.flags & DST_XFRM_TUNNEL)
- mtu = np->pmtudisc >= IPV6_PMTUDISC_PROBE ?
- rt->dst.dev->mtu : dst_mtu(&rt->dst);
- else
- mtu = np->pmtudisc >= IPV6_PMTUDISC_PROBE ?
- rt->dst.dev->mtu : dst_mtu(rt->dst.path);
- if (np->frag_size < mtu) {
- if (np->frag_size)
- mtu = np->frag_size;
- }
- cork->fragsize = mtu;
- if (dst_allfrag(rt->dst.path))
- cork->flags |= IPCORK_ALLFRAG;
- cork->length = 0;
+ err = ip6_setup_cork(sk, &inet->cork, &np->cork, hlimit,
+ tclass, opt, rt, fl6);
+ if (err)
+ return err;
exthdrlen = (opt ? opt->opt_flen : 0);
length += exthdrlen;
transhdrlen += exthdrlen;
@@ -1226,8 +1245,8 @@ int ip6_append_data(struct sock *sk, int getfrag(void *from, char *to,
transhdrlen = 0;
exthdrlen = 0;
dst_exthdrlen = 0;
- mtu = cork->fragsize;
}
+ mtu = cork->fragsize;
orig_mtu = mtu;
hh_len = LL_RESERVED_SPACE(rt->dst.dev);
@@ -1503,23 +1522,24 @@ error:
}
EXPORT_SYMBOL_GPL(ip6_append_data);
-static void ip6_cork_release(struct inet_sock *inet, struct ipv6_pinfo *np)
+static void ip6_cork_release(struct inet_cork_full *cork,
+ struct inet6_cork *v6_cork)
{
- if (np->cork.opt) {
- kfree(np->cork.opt->dst0opt);
- kfree(np->cork.opt->dst1opt);
- kfree(np->cork.opt->hopopt);
- kfree(np->cork.opt->srcrt);
- kfree(np->cork.opt);
- np->cork.opt = NULL;
+ if (v6_cork->opt) {
+ kfree(v6_cork->opt->dst0opt);
+ kfree(v6_cork->opt->dst1opt);
+ kfree(v6_cork->opt->hopopt);
+ kfree(v6_cork->opt->srcrt);
+ kfree(v6_cork->opt);
+ v6_cork->opt = NULL;
}
- if (inet->cork.base.dst) {
- dst_release(inet->cork.base.dst);
- inet->cork.base.dst = NULL;
- inet->cork.base.flags &= ~IPCORK_ALLFRAG;
+ if (cork->base.dst) {
+ dst_release(cork->base.dst);
+ cork->base.dst = NULL;
+ cork->base.flags &= ~IPCORK_ALLFRAG;
}
- memset(&inet->cork.fl, 0, sizeof(inet->cork.fl));
+ memset(&cork->fl, 0, sizeof(cork->fl));
}
int ip6_push_pending_frames(struct sock *sk)
@@ -1599,7 +1619,7 @@ int ip6_push_pending_frames(struct sock *sk)
}
out:
- ip6_cork_release(inet, np);
+ ip6_cork_release(&inet->cork, &np->cork);
return err;
error:
IP6_INC_STATS(net, rt->rt6i_idev, IPSTATS_MIB_OUTDISCARDS);
@@ -1618,6 +1638,6 @@ void ip6_flush_pending_frames(struct sock *sk)
kfree_skb(skb);
}
- ip6_cork_release(inet_sk(sk), inet6_sk(sk));
+ ip6_cork_release(&inet_sk(sk)->cork, &inet6_sk(sk)->cork);
}
EXPORT_SYMBOL_GPL(ip6_flush_pending_frames);
--
1.9.3
^ permalink raw reply related
* [PATCH net-next 0/6] ipv6: Add lockless UDP send path
From: Vladislav Yasevich @ 2015-01-31 15:40 UTC (permalink / raw)
To: netdev; +Cc: herbert, hannes, Vladislav Yasevich
This series introduces a lockless UDPv6 send path similar to
what Herbert Xu did for IPv4 a while ago.
There are some difference from IPv4. IPv6 caching for flow
label is a bit different, as well as it requires another cork
cork structure that holds the IPv6 ancillary data.
Please take a look.
Thanks
-vlad
Vladislav Yasevich (5):
ipv6: pull cork initialization into its own function.
ipv6: Append sending data to arbitrary queue
ipv6: introduce ipv6_make_skb
ipv6: Introduce udpv6_send_skb()
udpv6: Add lockless sendmsg() support
include/linux/ipv6.h | 12 +-
include/net/ipv6.h | 19 +++
net/ipv6/ip6_output.c | 349 +++++++++++++++++++++++++++++++++-----------------
net/ipv6/udp.c | 91 ++++++++-----
4 files changed, 317 insertions(+), 154 deletions(-)
--
1.9.3
^ permalink raw reply
* Hello B !! !!
From: Ahmed Mohammed @ 2015-01-31 13:29 UTC (permalink / raw)
To: netdev
Für Ihre Aufmerksamkeit, ich werde mich sehr freuen, wenn Sie mir helfen, eine Summe (€ 69,500,000.00 Euro) auf Ihr persönliches Bankkonto zu Gunsten der uns beide und zukünftige Investitionen in Ihrem Land.
Beachten Sie, bin ich mit der Bundesregierung Rohöl Unternehmen Einrichtungen arbeiten hier, und wir sind nicht gestattet oder erlaubt, ausländisches Bankkonto haben, und wenn dieses Geld in meinem lokalen Konto zu sehen hier dann Regierung wird mich vermuten, daher brauche ich Ihre Hilfe, um die Empfangs riesige Mittel in meinem Namen, das werde ich für Ihre Unterstützung erhalten Sie 30%. Ich werde Ihnen mehr Details, sobald ich von Ihnen höre senden
Bitte senden Sie mir dringend Ihren vollständigen Namen und Ihre Adresse, Ihre Telefonnummern anruft, Ihre Handynummer und Ihre Bankverbindung für die Übertragung des Sondervermögens für mein Unternehmen Investitionen in Ihrem Land
Grüße
Ahmed Mohammed
^ permalink raw reply
* Re: [PATCH 2/6] rhashtable: Use a single bucket lock for sibling buckets
From: Thomas Graf @ 2015-01-31 12:50 UTC (permalink / raw)
To: Herbert Xu; +Cc: davem, netdev, ying.xue
In-Reply-To: <20150131093812.GA29240@gondor.apana.org.au>
On 01/31/15 at 08:38pm, Herbert Xu wrote:
> On Sat, Jan 31, 2015 at 08:41:27AM +0000, Thomas Graf wrote:
> > On 01/31/15 at 03:34pm, Herbert Xu wrote:
> > > Thomas, could you please hold off on these changes? They totally
> > > conflict with my rehash work, which is going to render these changes
> > > moot anyway since it'll completely change how expansion/shrinking
> > > works.
> >
> > Can you share that work?
>
> I will post it when it's ready. But if you keep moving the goal
> posts then I'll never get there.
My intent is definitely not to make it harder for you. My intent
is fix the remaining races which were reported before net-next gets
merged. I'm happy to hold off if you plan on proposing your rehashing
in the current net-next cycle.
^ permalink raw reply
* [PATCH net-next 4/4] enic: add ethtool support for changing alloc order
From: Govindarajulu Varadarajan @ 2015-01-31 12:28 UTC (permalink / raw)
To: davem, netdev; +Cc: ssujith, benve, edumazet, ben, Govindarajulu Varadarajan
In-Reply-To: <1422707290-939-1-git-send-email-_govind@gmx.com>
Adds support for changing page order of enic frag allocator.
In case of changing mtu, if size of new mtu is greater than size of compound
page allocated in enic frag allocator, we change the order to min order required
for the new mtu. We would like to give high priority for changing mtu than order
value.
Signed-off-by: Govindarajulu Varadarajan <_govind@gmx.com>
---
drivers/net/ethernet/cisco/enic/enic_ethtool.c | 15 +++++++++++++++
drivers/net/ethernet/cisco/enic/enic_main.c | 18 ++++++++++++++++++
2 files changed, 33 insertions(+)
diff --git a/drivers/net/ethernet/cisco/enic/enic_ethtool.c b/drivers/net/ethernet/cisco/enic/enic_ethtool.c
index 3f9d91b..e29423c 100644
--- a/drivers/net/ethernet/cisco/enic/enic_ethtool.c
+++ b/drivers/net/ethernet/cisco/enic/enic_ethtool.c
@@ -18,6 +18,7 @@
#include <linux/netdevice.h>
#include <linux/ethtool.h>
+#include <linux/if_vlan.h>
#include "enic_res.h"
#include "enic.h"
@@ -409,6 +410,9 @@ static int enic_get_tunable(struct net_device *dev,
case ETHTOOL_RX_COPYBREAK:
*(u32 *)data = enic->rx_copybreak;
break;
+ case ETHTOOL_RX_ALLOC_ORDER:
+ *(u8 *)data = enic->alloc_order;
+ break;
default:
ret = -EINVAL;
break;
@@ -428,6 +432,17 @@ static int enic_set_tunable(struct net_device *dev,
case ETHTOOL_RX_COPYBREAK:
enic->rx_copybreak = *(u32 *)data;
break;
+ case ETHTOOL_RX_ALLOC_ORDER:
+ ret = dev->mtu + VLAN_ETH_HLEN + NET_IP_ALIGN + NET_SKB_PAD;
+ ret = SKB_DATA_ALIGN(ret) +
+ SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
+ if (*(u8 *)data < get_order(ret)) {
+ ret = -EINVAL;
+ break;
+ }
+ ret = 0;
+ enic->alloc_order = *(u8 *)data;
+ break;
default:
ret = -EINVAL;
break;
diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c
index f15687d..4a759a0 100644
--- a/drivers/net/ethernet/cisco/enic/enic_main.c
+++ b/drivers/net/ethernet/cisco/enic/enic_main.c
@@ -1872,7 +1872,11 @@ static int enic_change_mtu(struct net_device *netdev, int new_mtu)
{
struct enic *enic = netdev_priv(netdev);
int running = netif_running(netdev);
+ size_t len;
+ len = new_mtu + VLAN_ETH_HLEN + NET_IP_ALIGN + NET_SKB_PAD;
+ len = SKB_DATA_ALIGN(len) +
+ SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
if (new_mtu < ENIC_MIN_MTU || new_mtu > ENIC_MAX_MTU)
return -EINVAL;
@@ -1884,6 +1888,11 @@ static int enic_change_mtu(struct net_device *netdev, int new_mtu)
netdev->mtu = new_mtu;
+ if (len > (PAGE_SIZE << enic->alloc_order)) {
+ enic->alloc_order = get_order(len);
+ netdev_warn(netdev, "new mtu is greater than size of rx alloc_page, resetting enic->alloc_order to :%d\n",
+ enic->alloc_order);
+ }
if (netdev->mtu > enic->port_mtu)
netdev_warn(netdev,
"interface MTU (%d) set higher than port MTU (%d)\n",
@@ -1902,8 +1911,12 @@ static void enic_change_mtu_work(struct work_struct *work)
int new_mtu = vnic_dev_mtu(enic->vdev);
int err;
unsigned int i;
+ size_t len;
new_mtu = max_t(int, ENIC_MIN_MTU, min_t(int, ENIC_MAX_MTU, new_mtu));
+ len = new_mtu + VLAN_ETH_HLEN + NET_IP_ALIGN + NET_SKB_PAD;
+ len = SKB_DATA_ALIGN(len) +
+ SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
rtnl_lock();
@@ -1927,6 +1940,11 @@ static void enic_change_mtu_work(struct work_struct *work)
/* Fill RQ with new_mtu-sized buffers */
netdev->mtu = new_mtu;
+ if (len > (PAGE_SIZE << enic->alloc_order)) {
+ enic->alloc_order = get_order(len);
+ netdev_warn(netdev, "new mtu is greater than size of rx alloc_page, resetting enic->alloc_order to :%d\n",
+ enic->alloc_order);
+ }
vnic_rq_fill(&enic->rq[0], enic_rq_alloc_buf);
/* Need at least one buffer on ring to get going */
if (vnic_rq_desc_used(&enic->rq[0]) == 0) {
--
2.2.2
^ permalink raw reply related
* [PATCH net-next 3/4] ethtool: add RX_ALLOC_ORDER to tunable
From: Govindarajulu Varadarajan @ 2015-01-31 12:28 UTC (permalink / raw)
To: davem, netdev; +Cc: ssujith, benve, edumazet, ben, Govindarajulu Varadarajan
In-Reply-To: <1422707290-939-1-git-send-email-_govind@gmx.com>
Signed-off-by: Govindarajulu Varadarajan <_govind@gmx.com>
---
include/uapi/linux/ethtool.h | 1 +
net/core/ethtool.c | 5 +++++
2 files changed, 6 insertions(+)
diff --git a/include/uapi/linux/ethtool.h b/include/uapi/linux/ethtool.h
index 5f66d9c..59362f8 100644
--- a/include/uapi/linux/ethtool.h
+++ b/include/uapi/linux/ethtool.h
@@ -213,6 +213,7 @@ enum tunable_id {
ETHTOOL_ID_UNSPEC,
ETHTOOL_RX_COPYBREAK,
ETHTOOL_TX_COPYBREAK,
+ ETHTOOL_RX_ALLOC_ORDER,
};
enum tunable_type_id {
diff --git a/net/core/ethtool.c b/net/core/ethtool.c
index 91f74f3..5fd7ebf 100644
--- a/net/core/ethtool.c
+++ b/net/core/ethtool.c
@@ -1670,6 +1670,11 @@ static int ethtool_tunable_valid(const struct ethtool_tunable *tuna)
tuna->type_id != ETHTOOL_TUNABLE_U32)
return -EINVAL;
break;
+ case ETHTOOL_RX_ALLOC_ORDER:
+ if (tuna->len != sizeof(u8) ||
+ tuna->type_id != ETHTOOL_TUNABLE_U8)
+ return -EINVAL;
+ break;
default:
return -EINVAL;
}
--
2.2.2
^ permalink raw reply related
* [PATCH net-next 2/4] enic: Add rq allocation failure stats
From: Govindarajulu Varadarajan @ 2015-01-31 12:28 UTC (permalink / raw)
To: davem, netdev; +Cc: ssujith, benve, edumazet, ben, Govindarajulu Varadarajan
In-Reply-To: <1422707290-939-1-git-send-email-_govind@gmx.com>
This patch adds rq buff allocation failure stats.
cache_alloc_err: incremented when higher order page allocation fails.
enic_rq_alloc_buf: incremented rq buff fails. Either due to page alloc
failure or build_skb.
Signed-off-by: Govindarajulu Varadarajan <_govind@gmx.com>
---
drivers/net/ethernet/cisco/enic/enic_ethtool.c | 2 ++
drivers/net/ethernet/cisco/enic/enic_main.c | 3 +++
drivers/net/ethernet/cisco/enic/vnic_stats.h | 2 ++
3 files changed, 7 insertions(+)
diff --git a/drivers/net/ethernet/cisco/enic/enic_ethtool.c b/drivers/net/ethernet/cisco/enic/enic_ethtool.c
index 0c396c1..3f9d91b 100644
--- a/drivers/net/ethernet/cisco/enic/enic_ethtool.c
+++ b/drivers/net/ethernet/cisco/enic/enic_ethtool.c
@@ -86,6 +86,8 @@ static const struct enic_stat enic_rx_stats[] = {
static const struct enic_stat enic_gen_stats[] = {
ENIC_GEN_STAT(dma_map_error),
+ ENIC_GEN_STAT(cache_alloc_err),
+ ENIC_GEN_STAT(rq_alloc_err),
};
static const unsigned int enic_n_tx_stats = ARRAY_SIZE(enic_tx_stats);
diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c
index d9cad93..f15687d 100644
--- a/drivers/net/ethernet/cisco/enic/enic_main.c
+++ b/drivers/net/ethernet/cisco/enic/enic_main.c
@@ -962,6 +962,7 @@ struct enic_alloc_cache *enic_page_refill(struct enic *enic, size_t sz,
goto no_ec;
ec->frag.page = alloc_pages_node(NUMA_NO_NODE, gfp_comp, order);
if (unlikely(!ec->frag.page)) {
+ enic->gen_stats.cache_alloc_err++;
order = get_order(sz);
ec->frag.page = alloc_pages_node(NUMA_NO_NODE, gfp, order);
if (!ec->frag.page)
@@ -1105,6 +1106,8 @@ static int enic_rq_alloc_buf(struct vnic_rq *rq)
return 0;
alloc_fail:
+ enic->gen_stats.rq_alloc_err++;
+
return -ENOMEM;
}
diff --git a/drivers/net/ethernet/cisco/enic/vnic_stats.h b/drivers/net/ethernet/cisco/enic/vnic_stats.h
index 74c81ed..b2a4528 100644
--- a/drivers/net/ethernet/cisco/enic/vnic_stats.h
+++ b/drivers/net/ethernet/cisco/enic/vnic_stats.h
@@ -65,6 +65,8 @@ struct vnic_rx_stats {
/* Generic statistics */
struct vnic_gen_stats {
u64 dma_map_error;
+ u64 cache_alloc_err; /* alloc_pages(enic->order) failures */
+ u64 rq_alloc_err; /* rq buf + skb alloc failures */
};
struct vnic_stats {
--
2.2.2
^ permalink raw reply related
* [PATCH net-next 1/4] enic: implement frag allocator
From: Govindarajulu Varadarajan @ 2015-01-31 12:28 UTC (permalink / raw)
To: davem, netdev; +Cc: ssujith, benve, edumazet, ben, Govindarajulu Varadarajan
In-Reply-To: <1422707290-939-1-git-send-email-_govind@gmx.com>
This patch implements frag allocator for rq buffer. This is based on
__alloc_page_frag & __page_frag_refill implementation in net/core/skbuff.c
In addition to frag allocation from order(3) page in __alloc_page_frag,
we also maintain dma address of the page. While allocating a frag for rx buffer
we return va + offset for virtual address of the frag, and pa + offset for
dma address of the frag. This reduces the number of calls to dma_map()
by 1/3 for 9k mtu and by 1/20 for 1500 mtu.
__alloc_page_frag is limited to max buffer size of PAGE_SIZE, i.e 4096 in most
of the cases. So 9k buffer allocation goes through kmalloc which return
page of order 2, 16k. We waste 7k bytes for every 9k buffer.
We maintain dma_count variable which is incremented when we allocate a frag.
enic_unmap_dma will decrement the dma_count and unmap it when there is no user
of that page in rx ring.
This reduces the memory utilization for 9k mtu by 33%.
enic_alloc_cache struct, which stores the page details, is declared per rq.
And all calls to allocation, free, dmap_unmap is serialized. So we do not need
locks.
Signed-off-by: Govindarajulu Varadarajan <_govind@gmx.com>
---
drivers/net/ethernet/cisco/enic/enic.h | 16 +++
drivers/net/ethernet/cisco/enic/enic_main.c | 156 +++++++++++++++++++++++-----
drivers/net/ethernet/cisco/enic/vnic_rq.c | 13 +++
drivers/net/ethernet/cisco/enic/vnic_rq.h | 2 +
4 files changed, 163 insertions(+), 24 deletions(-)
diff --git a/drivers/net/ethernet/cisco/enic/enic.h b/drivers/net/ethernet/cisco/enic/enic.h
index 84b6a2b..7fd3db1 100644
--- a/drivers/net/ethernet/cisco/enic/enic.h
+++ b/drivers/net/ethernet/cisco/enic/enic.h
@@ -20,6 +20,11 @@
#ifndef _ENIC_H_
#define _ENIC_H_
+#include <linux/if.h>
+#include <linux/if_link.h>
+#include <linux/if_ether.h>
+#include <linux/netdevice.h>
+
#include "vnic_enet.h"
#include "vnic_dev.h"
#include "vnic_wq.h"
@@ -176,6 +181,7 @@ struct enic {
u64 rq_truncated_pkts;
u64 rq_bad_fcs;
struct napi_struct napi[ENIC_RQ_MAX + ENIC_WQ_MAX];
+ u8 alloc_order;
/* interrupt resource cache line section */
____cacheline_aligned struct vnic_intr intr[ENIC_INTR_MAX];
@@ -191,6 +197,16 @@ struct enic {
struct vnic_gen_stats gen_stats;
};
+#define ENIC_ALLOC_ORDER get_order(32 * 1024)
+
+struct enic_alloc_cache {
+ struct page_frag frag;
+ unsigned int pagecnt_bias;
+ int dma_count;
+ void *va;
+ dma_addr_t pa;
+};
+
static inline struct device *enic_get_dev(struct enic *enic)
{
return &(enic->pdev->dev);
diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c
index ee44c82..d9cad93 100644
--- a/drivers/net/ethernet/cisco/enic/enic_main.c
+++ b/drivers/net/ethernet/cisco/enic/enic_main.c
@@ -950,6 +950,105 @@ nla_put_failure:
return -EMSGSIZE;
}
+struct enic_alloc_cache *enic_page_refill(struct enic *enic, size_t sz,
+ gfp_t gfp)
+{
+ struct enic_alloc_cache *ec;
+ gfp_t gfp_comp = gfp | __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY;
+ u8 order = enic->alloc_order;
+
+ ec = kzalloc(sizeof(*ec), GFP_ATOMIC);
+ if (unlikely(!ec))
+ goto no_ec;
+ ec->frag.page = alloc_pages_node(NUMA_NO_NODE, gfp_comp, order);
+ if (unlikely(!ec->frag.page)) {
+ order = get_order(sz);
+ ec->frag.page = alloc_pages_node(NUMA_NO_NODE, gfp, order);
+ if (!ec->frag.page)
+ goto free_ec;
+ }
+
+ ec->frag.size = (PAGE_SIZE << order);
+ ec->va = page_address(ec->frag.page);
+ ec->pa = pci_map_single(enic->pdev, ec->va, ec->frag.size,
+ PCI_DMA_FROMDEVICE);
+ if (unlikely(enic_dma_map_check(enic, ec->pa)))
+ goto free_page;
+ atomic_add(ec->frag.size - 1, &ec->frag.page->_count);
+ ec->pagecnt_bias = ec->frag.size;
+ ec->frag.offset = ec->frag.size;
+
+ return ec;
+
+free_page:
+ __free_pages(ec->frag.page, order);
+free_ec:
+ kfree(ec);
+no_ec:
+ return NULL;
+}
+
+struct enic_alloc_cache *enic_alloc_frag(struct vnic_rq *rq, size_t sz)
+{
+ struct enic *enic = vnic_dev_priv(rq->vdev);
+ struct enic_alloc_cache *ec = rq->ec;
+ int offset;
+
+ if (unlikely(!ec)) {
+refill:
+ ec = enic_page_refill(enic, sz, GFP_ATOMIC);
+ rq->ec = ec;
+
+ if (unlikely(!ec))
+ return NULL;
+ }
+
+ offset = ec->frag.offset - sz;
+ if (offset < 0) {
+ if (!atomic_sub_and_test(ec->pagecnt_bias,
+ &ec->frag.page->_count)) {
+ /* rq cleanup service has processed all the frags
+ * belonging to this page. Since page->_count is not 0
+ * and ec->dma_count is 0 these frags should be in
+ * stack. We should unmap the page here.
+ */
+ if (!ec->dma_count) {
+ pci_unmap_single(enic->pdev, ec->pa,
+ ec->frag.size,
+ PCI_DMA_FROMDEVICE);
+ kfree(ec);
+ } else {
+ /* frags from this page are still in rx queue. Let the
+ * rx cleanup service unmap the page in enic_unmap_dma.
+ */
+ ec->pagecnt_bias = 0;
+ }
+ goto refill;
+ }
+ WARN_ON(ec->dma_count);
+ atomic_set(&ec->frag.page->_count, ec->frag.size);
+ ec->pagecnt_bias = ec->frag.size;
+ offset = ec->frag.size - sz;
+ }
+ ec->pagecnt_bias--;
+ ec->dma_count++;
+ ec->frag.offset = offset;
+
+ return ec;
+}
+
+void enic_unmap_dma(struct enic *enic, struct enic_alloc_cache *ec)
+{
+ /* enic_alloc_frag is done using this page. We should be free to unmap
+ * the page if there are no pending frags in the queue.
+ */
+ if (!--ec->dma_count && !ec->pagecnt_bias) {
+ pci_unmap_single(enic->pdev, ec->pa, ec->frag.size,
+ PCI_DMA_FROMDEVICE);
+ kfree(ec);
+ }
+}
+
static void enic_free_rq_buf(struct vnic_rq *rq, struct vnic_rq_buf *buf)
{
struct enic *enic = vnic_dev_priv(rq->vdev);
@@ -957,8 +1056,7 @@ static void enic_free_rq_buf(struct vnic_rq *rq, struct vnic_rq_buf *buf)
if (!buf->os_buf)
return;
- pci_unmap_single(enic->pdev, buf->dma_addr,
- buf->len, PCI_DMA_FROMDEVICE);
+ enic_unmap_dma(enic, buf->ec);
dev_kfree_skb_any(buf->os_buf);
buf->os_buf = NULL;
}
@@ -968,10 +1066,12 @@ static int enic_rq_alloc_buf(struct vnic_rq *rq)
struct enic *enic = vnic_dev_priv(rq->vdev);
struct net_device *netdev = enic->netdev;
struct sk_buff *skb;
- unsigned int len = netdev->mtu + VLAN_ETH_HLEN;
+ unsigned int len;
unsigned int os_buf_index = 0;
dma_addr_t dma_addr;
struct vnic_rq_buf *buf = rq->to_use;
+ struct enic_alloc_cache *ec;
+ void *va;
if (buf->os_buf) {
enic_queue_rq_desc(rq, buf->os_buf, os_buf_index, buf->dma_addr,
@@ -979,21 +1079,33 @@ static int enic_rq_alloc_buf(struct vnic_rq *rq)
return 0;
}
- skb = netdev_alloc_skb_ip_align(netdev, len);
- if (!skb)
- return -ENOMEM;
- dma_addr = pci_map_single(enic->pdev, skb->data, len,
- PCI_DMA_FROMDEVICE);
- if (unlikely(enic_dma_map_check(enic, dma_addr))) {
- dev_kfree_skb(skb);
- return -ENOMEM;
- }
+ len = netdev->mtu + VLAN_ETH_HLEN + NET_IP_ALIGN + NET_SKB_PAD;
+ len = SKB_DATA_ALIGN(len) +
+ SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
- enic_queue_rq_desc(rq, skb, os_buf_index,
- dma_addr, len);
+ ec = enic_alloc_frag(rq, len);
+ if (unlikely(!ec))
+ goto alloc_fail;
+ va = ec->va + ec->frag.offset;
+ skb = build_skb(va, len);
+ if (unlikely(!skb)) {
+ ec->pagecnt_bias++;
+ ec->frag.offset += len;
+ ec->dma_count--;
+
+ goto alloc_fail;
+ }
+ skb_reserve(skb, NET_SKB_PAD + NET_IP_ALIGN);
+ dma_addr = ec->pa + ec->frag.offset + NET_SKB_PAD + NET_IP_ALIGN;
+ buf->ec = ec;
+ enic_queue_rq_desc(rq, skb, os_buf_index, dma_addr,
+ netdev->mtu + VLAN_ETH_HLEN);
return 0;
+
+alloc_fail:
+ return -ENOMEM;
}
static void enic_intr_update_pkt_size(struct vnic_rx_bytes_counter *pkt_size,
@@ -1016,8 +1128,6 @@ static bool enic_rxcopybreak(struct net_device *netdev, struct sk_buff **skb,
new_skb = netdev_alloc_skb_ip_align(netdev, len);
if (!new_skb)
return false;
- pci_dma_sync_single_for_cpu(enic->pdev, buf->dma_addr, len,
- DMA_FROM_DEVICE);
memcpy(new_skb->data, (*skb)->data, len);
*skb = new_skb;
@@ -1065,8 +1175,7 @@ static void enic_rq_indicate_buf(struct vnic_rq *rq,
enic->rq_truncated_pkts++;
}
- pci_unmap_single(enic->pdev, buf->dma_addr, buf->len,
- PCI_DMA_FROMDEVICE);
+ enic_unmap_dma(enic, buf->ec);
dev_kfree_skb_any(skb);
buf->os_buf = NULL;
@@ -1077,11 +1186,11 @@ static void enic_rq_indicate_buf(struct vnic_rq *rq,
/* Good receive
*/
-
+ pci_dma_sync_single_for_cpu(enic->pdev, buf->dma_addr,
+ bytes_written, DMA_FROM_DEVICE);
if (!enic_rxcopybreak(netdev, &skb, buf, bytes_written)) {
buf->os_buf = NULL;
- pci_unmap_single(enic->pdev, buf->dma_addr, buf->len,
- PCI_DMA_FROMDEVICE);
+ enic_unmap_dma(enic, buf->ec);
}
prefetch(skb->data - NET_IP_ALIGN);
@@ -1122,9 +1231,7 @@ static void enic_rq_indicate_buf(struct vnic_rq *rq,
/* Buffer overflow
*/
-
- pci_unmap_single(enic->pdev, buf->dma_addr, buf->len,
- PCI_DMA_FROMDEVICE);
+ enic_unmap_dma(enic, buf->ec);
dev_kfree_skb_any(skb);
buf->os_buf = NULL;
}
@@ -2637,6 +2744,7 @@ static int enic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
goto err_out_dev_deinit;
}
enic->rx_copybreak = RX_COPYBREAK_DEFAULT;
+ enic->alloc_order = ENIC_ALLOC_ORDER;
return 0;
diff --git a/drivers/net/ethernet/cisco/enic/vnic_rq.c b/drivers/net/ethernet/cisco/enic/vnic_rq.c
index 36a2ed6..c31669f 100644
--- a/drivers/net/ethernet/cisco/enic/vnic_rq.c
+++ b/drivers/net/ethernet/cisco/enic/vnic_rq.c
@@ -26,6 +26,7 @@
#include "vnic_dev.h"
#include "vnic_rq.h"
+#include "enic.h"
static int vnic_rq_alloc_bufs(struct vnic_rq *rq)
{
@@ -199,6 +200,18 @@ void vnic_rq_clean(struct vnic_rq *rq,
rq->ring.desc_avail++;
}
+ if (rq->ec) {
+ struct enic *enic = vnic_dev_priv(rq->vdev);
+ struct enic_alloc_cache *ec = rq->ec;
+
+ WARN_ON(ec->dma_count);
+ pci_unmap_single(enic->pdev, ec->pa, ec->frag.size,
+ PCI_DMA_FROMDEVICE);
+ atomic_sub(ec->pagecnt_bias - 1, &ec->frag.page->_count);
+ __free_pages(ec->frag.page, get_order(ec->frag.size));
+ kfree(ec);
+ rq->ec = NULL;
+ }
/* Use current fetch_index as the ring starting point */
fetch_index = ioread32(&rq->ctrl->fetch_index);
diff --git a/drivers/net/ethernet/cisco/enic/vnic_rq.h b/drivers/net/ethernet/cisco/enic/vnic_rq.h
index 8111d52..2e4815a 100644
--- a/drivers/net/ethernet/cisco/enic/vnic_rq.h
+++ b/drivers/net/ethernet/cisco/enic/vnic_rq.h
@@ -73,6 +73,7 @@ struct vnic_rq_buf {
unsigned int index;
void *desc;
uint64_t wr_id;
+ struct enic_alloc_cache *ec;
};
struct vnic_rq {
@@ -100,6 +101,7 @@ struct vnic_rq {
unsigned int bpoll_state;
spinlock_t bpoll_lock;
#endif /* CONFIG_NET_RX_BUSY_POLL */
+ struct enic_alloc_cache *ec;
};
static inline unsigned int vnic_rq_desc_avail(struct vnic_rq *rq)
--
2.2.2
^ permalink raw reply related
* [PATCH net-next 0/4] enic: improve rq buff allocation and reduce dma mapping
From: Govindarajulu Varadarajan @ 2015-01-31 12:28 UTC (permalink / raw)
To: davem, netdev; +Cc: ssujith, benve, edumazet, ben, Govindarajulu Varadarajan
The following series tries to address these two problem in rq buff allocation.
* Memory wastage because of large 9k allocation using kmalloc:
For 9k mtu buffer, netdev_alloc_skb_ip_align internally calls kmalloc for
size > 4096. In case of 9k buff, kmalloc returns pages for order 2, 16k.
And we use only ~9k of 16k. 7k memory wasted. Using the frag the frag
allocator in patch 1/2, we can allocate three 9k buffs in a 32k page size.
Typical enic configuration has 8 rq, and desc ring of size 4096.
Thats 8 * 4096 * (16*1024) = 512 MB. Using this frag allocator:
8 * 4096 * (32*1024/3) = 341 MB. Thats 171 MB of memory save.
* frequent dma_map() calls:
we call dma_map() for every buff we allocate. When iommu is on, This is very
cpu time consuming. From my testing, most of the cpu cycles are wasted
spinning on spin_lock_irqsave(&iovad->iova_rbtree_lock, flags) in
intel_map_page() .. -> ..__alloc_and_insert_iova_range()
With this patch, we call dma_map() once for 32k page. i.e once for every three
9k desc, and once every twenty 1500 bytes desc.
Here are testing result with 8 rq, 4096 ring size and 9k mtu. irq of each rq
is affinitized with different CPU. Ran iperf with 32 threads. Link is 10G.
iommu is on.
CPU utilization throughput
without patch 100% 1.8 Gbps
with patch 13% 9.8 Gbps
Govindarajulu Varadarajan (4):
enic: implement frag allocator
enic: Add rq allocation failure stats
ethtool: add RX_ALLOC_ORDER to tunable
enic: add ethtool support for changing alloc order
drivers/net/ethernet/cisco/enic/enic.h | 16 +++
drivers/net/ethernet/cisco/enic/enic_ethtool.c | 17 +++
drivers/net/ethernet/cisco/enic/enic_main.c | 177 +++++++++++++++++++++----
drivers/net/ethernet/cisco/enic/vnic_rq.c | 13 ++
drivers/net/ethernet/cisco/enic/vnic_rq.h | 2 +
drivers/net/ethernet/cisco/enic/vnic_stats.h | 2 +
include/uapi/linux/ethtool.h | 1 +
net/core/ethtool.c | 5 +
8 files changed, 209 insertions(+), 24 deletions(-)
--
2.2.2
^ permalink raw reply
* Re: [PATCH 7/9] netfilter: nft_hash: add support for timeouts
From: Patrick McHardy @ 2015-01-31 12:16 UTC (permalink / raw)
To: Herbert Xu
Cc: tgraf, davem, David.Laight, ying.xue, paulmck, netdev,
netfilter-devel
In-Reply-To: <20150131042920.GA27480@gondor.apana.org.au>
On 31.01, Herbert Xu wrote:
> On Fri, Jan 30, 2015 at 07:46:32AM +0000, Patrick McHardy wrote:
> >
> > + mutex_lock(&priv->ht.mutex);
> > + tbl = rht_dereference(priv->ht.tbl, &priv->ht);
> > + for (i = 0; i < tbl->size; i++) {
> > + rht_for_each_entry_safe(he, pos, next, tbl, i, node) {
> > + if (!nft_set_ext_exists(&he->ext, NFT_SET_EXT_TIMEOUT))
> > + continue;
> > + timeout = *nft_set_ext_timeout(&he->ext);
> > + if (time_before(jiffies, timeout))
> > + continue;
> > +
> > + rhashtable_remove(&priv->ht, &he->node);
> > + nft_hash_elem_destroy(set, he);
> > + }
> > + }
> > + mutex_unlock(&priv->ht.mutex);
>
> What if somebody is currently walking over the table? Shouldn't
> you do an RCU free here instead of immediately destroying the
> element?
Yes, that's what I meant in mail 0/x regarding the existing races.
Probably will add some fixed sized batching here.
^ permalink raw reply
* Re: rhashtable: Fix potential crash on destroy in rhashtable_shrink
From: Thomas Graf @ 2015-01-31 12:15 UTC (permalink / raw)
To: Herbert Xu; +Cc: netdev
In-Reply-To: <20150131112209.GA29869@gondor.apana.org.au>
On 01/31/15 at 10:22pm, Herbert Xu wrote:
> That's too late. nft_hash will have freed all the elements
> before rhashtable_destroy gets called.
I see, so this is to accomodate nft_hash which doesn't remove
the elements from the hash but just frees them.
Acked-by: Thomas Graf <tgraf@suug.ch>
^ permalink raw reply
* Re: rhashtable: Fix potential crash on destroy in rhashtable_shrink
From: Herbert Xu @ 2015-01-31 11:22 UTC (permalink / raw)
To: Thomas Graf; +Cc: netdev
In-Reply-To: <20150131111652.GA22448@casper.infradead.org>
On Sat, Jan 31, 2015 at 11:16:52AM +0000, Thomas Graf wrote:
> On 01/31/15 at 08:36pm, Herbert Xu wrote:
> > The current being_destroyed check in rhashtable_expand is not
> > enough since if we start a shrinking process after freeing all
> > elements in the table that's also going to crash.
>
> (The check in expand() is just an optimization to drop out of
> work cycles if it does not make sense to continue anymore.)
>
> >
> > This patch adds a being_destroyed check to the deferred worker
> > thread so that we bail out as soon as we take the lock.
>
> Shouldn't the cancel_work_sync() in rhashtable_destroy() block
> until the deferred worker is done and cancelled?
That's too late. nft_hash will have freed all the elements
before rhashtable_destroy gets called.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: rhashtable: Move hash_rnd into bucket_table
From: Thomas Graf @ 2015-01-31 11:17 UTC (permalink / raw)
To: Herbert Xu; +Cc: netdev
In-Reply-To: <20150131102150.GA29498@gondor.apana.org.au>
On 01/31/15 at 09:21pm, Herbert Xu wrote:
> Currently hash_rnd is a parameter that users can set. However,
> no existing users set this parameter. It is also something that
> people are unlikely to want to set directly since it's just a
> random number.
>
> In preparation for allowing the reseeding/rehashing of rhashtable,
> this patch moves hash_rnd into bucket_table so that it's now an
> internal state rather than a parameter.
>
> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
For net-next
Acked-by: Thomas Graf <tgraf@suug.ch>
^ permalink raw reply
* Re: rhashtable: Fix potential crash on destroy in rhashtable_shrink
From: Thomas Graf @ 2015-01-31 11:16 UTC (permalink / raw)
To: Herbert Xu; +Cc: netdev
In-Reply-To: <20150131093637.GA29106@gondor.apana.org.au>
On 01/31/15 at 08:36pm, Herbert Xu wrote:
> The current being_destroyed check in rhashtable_expand is not
> enough since if we start a shrinking process after freeing all
> elements in the table that's also going to crash.
(The check in expand() is just an optimization to drop out of
work cycles if it does not make sense to continue anymore.)
>
> This patch adds a being_destroyed check to the deferred worker
> thread so that we bail out as soon as we take the lock.
Shouldn't the cancel_work_sync() in rhashtable_destroy() block
until the deferred worker is done and cancelled?
^ permalink raw reply
* [RFC] rhashtable: rhashtable_rehash
From: Herbert Xu @ 2015-01-31 10:23 UTC (permalink / raw)
To: Thomas Graf, netdev
In-Reply-To: <20150131102150.GA29498@gondor.apana.org.au>
Here is a totally untested patch that illustrates what I want to
do with rehash.
Note that this depends on the hash rnd patch I just sent.
diff --git a/lib/rhashtable.c b/lib/rhashtable.c
index 4c3da1f..ebd5e44 100644
--- a/lib/rhashtable.c
+++ b/lib/rhashtable.c
@@ -79,9 +79,9 @@ static u32 rht_bucket_index(const struct bucket_table *tbl, u32 hash)
return hash & (tbl->size - 1);
}
-static u32 obj_raw_hashfn(const struct rhashtable *ht, const void *ptr)
+static u32 obj_raw_hashfn(const struct rhashtable *ht,
+ struct bucket_table *tbl, const void *ptr)
{
- struct bucket_table *tbl = rht_dereference_rcu(ht->tbl, ht);
u32 hash;
if (unlikely(!ht->p.key_len))
@@ -93,9 +93,9 @@ static u32 obj_raw_hashfn(const struct rhashtable *ht, const void *ptr)
return hash >> HASH_RESERVED_SPACE;
}
-static u32 key_hashfn(struct rhashtable *ht, const void *key, u32 len)
+static u32 key_hashfn(struct rhashtable *ht, struct bucket_table *tbl,
+ const void *key, u32 len)
{
- struct bucket_table *tbl = rht_dereference_rcu(ht->tbl, ht);
u32 hash;
hash = ht->p.hashfn(key, len, tbl->hash_rnd);
@@ -295,6 +295,89 @@ static void link_old_to_new(struct bucket_table *new_tbl,
spin_unlock_bh(new_bucket_lock);
}
+static void rhashtable_rehash_one(struct rhashtable *ht, unsigned int old_hash)
+{
+ struct *old_tbl = rht_dereference(ht->tbl, ht);
+ struct *new_tbl = rht_dereference(ht->future_tbl, ht);
+ struct rhash_head *head, *next, *entry;
+ struct rhash_head __rcu **pprev;
+ spinlock_t *new_bucket_lock;
+ unsigned int new_hash;
+
+ pprev = &old_tbl->buckets[old_hash];
+ rht_for_each(entry, old_tbl, old_hash) {
+ next = rht_dereference_bucket(entry->next, old_tbl, old_hash);
+
+ if (rht_is_a_nulls(next))
+ break;
+
+ pprev = &entry->next;
+ }
+
+ new_hash = head_hashfn(ht, new_tbl, entry);
+
+ new_bucket_lock = bucket_lock(new_tbl, new_hash);
+
+ spin_lock(new_bucket_lock);
+ head = rht_dereference_bucket(new_tbl->buckets[new_hash],
+ new_tbl, new_hash);
+
+ if (rht_is_a_nulls(head))
+ INIT_RHT_NULLS_HEAD(entry->next, ht, hash);
+ else
+ RCU_INIT_POINTER(entry->next, head);
+
+ rcu_assign_pointer(new_tbl->buckets[new_hash], entry);
+ spin_unlock(new_bucket_lock);
+
+ rcu_assign_pointer(*pprev, next);
+}
+
+static void rhashtable_rehash_chain(struct rhashtable *ht,
+ unsigned int old_hash)
+{
+ struct *old_tbl = rht_dereference(ht->tbl, ht);
+ spinlock_t *old_bucket_lock;
+
+ old_bucket_lock = bucket_lock(old_tbl, old_hash);
+
+ spin_lock_bh(old_bucket_lock);
+ while (!rht_is_a_nulls(rht_dereference_bucket(
+ old_tbl->buckets[old_hash], old_tbl, old_hash)))
+ rhashtable_rehash_one(ht, old_hash);
+ spin_unlock_bh(old_bucket_lock);
+}
+
+static void rhashtable_rehash(struct rhashtable *ht,
+ struct bucket_table *new_tbl)
+{
+ struct *old_tbl = rht_dereference(ht->tbl, ht);
+
+ get_random_bytes(&new_tbl->hash_rnd, sizeof(new_tbl->hash_rnd));
+
+ /* Make insertions go into the new, empty table right away. Deletions
+ * and lookups will be attempted in both tables until we synchronize.
+ * The synchronize_rcu() guarantees for the new table to be picked up
+ * so no new additions go into the old table while we relink.
+ */
+ rcu_assign_pointer(ht->future_tbl, new_tbl);
+ synchronize_rcu();
+
+ for (old_hash = 0; i < old_tbl->size; old_hash++)
+ rhashtable_rehash_chain(ht, old_hash);
+
+ /* Publish the new table pointer. */
+ rcu_assign_pointer(ht->tbl, new_tbl);
+
+ /* Wait for readers. All new readers will see the new
+ * table, and thus no references to the old table will
+ * remain.
+ */
+ synchronize_rcu();
+
+ bucket_table_free(old_tbl);
+}
+
/**
* rhashtable_expand - Expand hash table while allowing concurrent lookups
* @ht: the hash table to expand
@@ -327,72 +410,8 @@ int rhashtable_expand(struct rhashtable *ht)
atomic_inc(&ht->shift);
- /* Make insertions go into the new, empty table right away. Deletions
- * and lookups will be attempted in both tables until we synchronize.
- * The synchronize_rcu() guarantees for the new table to be picked up
- * so no new additions go into the old table while we relink.
- */
- rcu_assign_pointer(ht->future_tbl, new_tbl);
- synchronize_rcu();
-
- /* For each new bucket, search the corresponding old bucket for the
- * first entry that hashes to the new bucket, and link the end of
- * newly formed bucket chain (containing entries added to future
- * table) to that entry. Since all the entries which will end up in
- * the new bucket appear in the same old bucket, this constructs an
- * entirely valid new hash table, but with multiple buckets
- * "zipped" together into a single imprecise chain.
- */
- for (new_hash = 0; new_hash < new_tbl->size; new_hash++) {
- old_hash = rht_bucket_index(old_tbl, new_hash);
- old_bucket_lock = bucket_lock(old_tbl, old_hash);
-
- spin_lock_bh(old_bucket_lock);
- rht_for_each(he, old_tbl, old_hash) {
- if (head_hashfn(ht, new_tbl, he) == new_hash) {
- link_old_to_new(new_tbl, new_hash, he);
- break;
- }
- }
- spin_unlock_bh(old_bucket_lock);
- }
-
- /* Publish the new table pointer. Lookups may now traverse
- * the new table, but they will not benefit from any
- * additional efficiency until later steps unzip the buckets.
- */
- rcu_assign_pointer(ht->tbl, new_tbl);
-
- /* Unzip interleaved hash chains */
- while (!complete && !ht->being_destroyed) {
- /* Wait for readers. All new readers will see the new
- * table, and thus no references to the old table will
- * remain.
- */
- synchronize_rcu();
-
- /* For each bucket in the old table (each of which
- * contains items from multiple buckets of the new
- * table): ...
- */
- complete = true;
- for (old_hash = 0; old_hash < old_tbl->size; old_hash++) {
- struct rhash_head *head;
-
- old_bucket_lock = bucket_lock(old_tbl, old_hash);
- spin_lock_bh(old_bucket_lock);
-
- hashtable_chain_unzip(ht, new_tbl, old_tbl, old_hash);
- head = rht_dereference_bucket(old_tbl->buckets[old_hash],
- old_tbl, old_hash);
- if (!rht_is_a_nulls(head))
- complete = false;
-
- spin_unlock_bh(old_bucket_lock);
- }
- }
+ rhashtable_rehash(ht, new_tbl);
- bucket_table_free(old_tbl);
return 0;
}
EXPORT_SYMBOL_GPL(rhashtable_expand);
@@ -425,57 +444,7 @@ int rhashtable_shrink(struct rhashtable *ht)
if (new_tbl == NULL)
return -ENOMEM;
- rcu_assign_pointer(ht->future_tbl, new_tbl);
- synchronize_rcu();
-
- /* Link the first entry in the old bucket to the end of the
- * bucket in the new table. As entries are concurrently being
- * added to the new table, lock down the new bucket. As we
- * always divide the size in half when shrinking, each bucket
- * in the new table maps to exactly two buckets in the old
- * table.
- *
- * As removals can occur concurrently on the old table, we need
- * to lock down both matching buckets in the old table.
- */
- for (new_hash = 0; new_hash < new_tbl->size; new_hash++) {
- old_bucket_lock1 = bucket_lock(tbl, new_hash);
- old_bucket_lock2 = bucket_lock(tbl, new_hash + new_tbl->size);
- new_bucket_lock = bucket_lock(new_tbl, new_hash);
-
- spin_lock_bh(old_bucket_lock1);
-
- /* Depending on the lock per buckets mapping, the bucket in
- * the lower and upper region may map to the same lock.
- */
- if (old_bucket_lock1 != old_bucket_lock2) {
- spin_lock_bh_nested(old_bucket_lock2, RHT_LOCK_NESTED);
- spin_lock_bh_nested(new_bucket_lock, RHT_LOCK_NESTED2);
- } else {
- spin_lock_bh_nested(new_bucket_lock, RHT_LOCK_NESTED);
- }
-
- rcu_assign_pointer(*bucket_tail(new_tbl, new_hash),
- tbl->buckets[new_hash]);
- rcu_assign_pointer(*bucket_tail(new_tbl, new_hash),
- tbl->buckets[new_hash + new_tbl->size]);
-
- spin_unlock_bh(new_bucket_lock);
- if (old_bucket_lock1 != old_bucket_lock2)
- spin_unlock_bh(old_bucket_lock2);
- spin_unlock_bh(old_bucket_lock1);
- }
-
- /* Publish the new, valid hash table */
- rcu_assign_pointer(ht->tbl, new_tbl);
- atomic_dec(&ht->shift);
-
- /* Wait for readers. No new readers will have references to the
- * old hash table.
- */
- synchronize_rcu();
-
- bucket_table_free(tbl);
+ rhashtable_rehash(ht, new_tbl);
return 0;
}
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply related
* rhashtable: Move hash_rnd into bucket_table
From: Herbert Xu @ 2015-01-31 10:21 UTC (permalink / raw)
To: Thomas Graf, netdev
Currently hash_rnd is a parameter that users can set. However,
no existing users set this parameter. It is also something that
people are unlikely to want to set directly since it's just a
random number.
In preparation for allowing the reseeding/rehashing of rhashtable,
this patch moves hash_rnd into bucket_table so that it's now an
internal state rather than a parameter.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
diff --git a/include/linux/rhashtable.h b/include/linux/rhashtable.h
index a2562ed..6d7e840 100644
--- a/include/linux/rhashtable.h
+++ b/include/linux/rhashtable.h
@@ -55,6 +55,7 @@ struct rhash_head {
struct bucket_table {
size_t size;
unsigned int locks_mask;
+ u32 hash_rnd;
spinlock_t *locks;
struct rhash_head __rcu *buckets[];
};
@@ -70,7 +71,6 @@ struct rhashtable;
* @key_len: Length of key
* @key_offset: Offset of key in struct to be hashed
* @head_offset: Offset of rhash_head in struct to be hashed
- * @hash_rnd: Seed to use while hashing
* @max_shift: Maximum number of shifts while expanding
* @min_shift: Minimum number of shifts while shrinking
* @nulls_base: Base value to generate nulls marker
@@ -89,7 +89,6 @@ struct rhashtable_params {
size_t key_len;
size_t key_offset;
size_t head_offset;
- u32 hash_rnd;
size_t max_shift;
size_t min_shift;
u32 nulls_base;
diff --git a/lib/rhashtable.c b/lib/rhashtable.c
index 84a78e3..71c6aa1 100644
--- a/lib/rhashtable.c
+++ b/lib/rhashtable.c
@@ -81,13 +81,14 @@ static u32 rht_bucket_index(const struct bucket_table *tbl, u32 hash)
static u32 obj_raw_hashfn(const struct rhashtable *ht, const void *ptr)
{
+ struct bucket_table *tbl = rht_dereference_rcu(ht->tbl, ht);
u32 hash;
if (unlikely(!ht->p.key_len))
- hash = ht->p.obj_hashfn(ptr, ht->p.hash_rnd);
+ hash = ht->p.obj_hashfn(ptr, tbl->hash_rnd);
else
hash = ht->p.hashfn(ptr + ht->p.key_offset, ht->p.key_len,
- ht->p.hash_rnd);
+ tbl->hash_rnd);
return hash >> HASH_RESERVED_SPACE;
}
@@ -97,7 +98,7 @@ static u32 key_hashfn(struct rhashtable *ht, const void *key, u32 len)
struct bucket_table *tbl = rht_dereference_rcu(ht->tbl, ht);
u32 hash;
- hash = ht->p.hashfn(key, len, ht->p.hash_rnd);
+ hash = ht->p.hashfn(key, len, tbl->hash_rnd);
hash >>= HASH_RESERVED_SPACE;
return rht_bucket_index(tbl, hash);
@@ -894,14 +895,13 @@ int rhashtable_init(struct rhashtable *ht, struct rhashtable_params *params)
if (tbl == NULL)
return -ENOMEM;
+ get_random_bytes(&tbl->hash_rnd, sizeof(tbl->hash_rnd));
+
atomic_set(&ht->nelems, 0);
atomic_set(&ht->shift, ilog2(tbl->size));
RCU_INIT_POINTER(ht->tbl, tbl);
RCU_INIT_POINTER(ht->future_tbl, tbl);
- if (!ht->p.hash_rnd)
- get_random_bytes(&ht->p.hash_rnd, sizeof(ht->p.hash_rnd));
-
if (ht->p.grow_decision || ht->p.shrink_decision)
INIT_WORK(&ht->run_work, rht_deferred_worker);
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply related
* Re: [PATCH 2/6] rhashtable: Use a single bucket lock for sibling buckets
From: Herbert Xu @ 2015-01-31 9:38 UTC (permalink / raw)
To: Thomas Graf; +Cc: davem, netdev, ying.xue
In-Reply-To: <20150131084127.GA13841@casper.infradead.org>
On Sat, Jan 31, 2015 at 08:41:27AM +0000, Thomas Graf wrote:
> On 01/31/15 at 03:34pm, Herbert Xu wrote:
> > Thomas, could you please hold off on these changes? They totally
> > conflict with my rehash work, which is going to render these changes
> > moot anyway since it'll completely change how expansion/shrinking
> > works.
>
> Can you share that work?
I will post it when it's ready. But if you keep moving the goal
posts then I'll never get there.
Thanks,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* rhashtable: Fix potential crash on destroy in rhashtable_shrink
From: Herbert Xu @ 2015-01-31 9:36 UTC (permalink / raw)
To: Thomas Graf, netdev
The current being_destroyed check in rhashtable_expand is not
enough since if we start a shrinking process after freeing all
elements in the table that's also going to crash.
This patch adds a being_destroyed check to the deferred worker
thread so that we bail out as soon as we take the lock.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
diff --git a/lib/rhashtable.c b/lib/rhashtable.c
index 69a4eb0..4c3da1f 100644
--- a/lib/rhashtable.c
+++ b/lib/rhashtable.c
@@ -489,6 +489,9 @@ static void rht_deferred_worker(struct work_struct *work)
ht = container_of(work, struct rhashtable, run_work);
mutex_lock(&ht->mutex);
+ if (ht->being_destroyed)
+ goto unlock;
+
tbl = rht_dereference(ht->tbl, ht);
list_for_each_entry(walker, &ht->walkers, list)
@@ -499,6 +502,7 @@ static void rht_deferred_worker(struct work_struct *work)
else if (ht->p.shrink_decision && ht->p.shrink_decision(ht, tbl->size))
rhashtable_shrink(ht);
+unlock:
mutex_unlock(&ht->mutex);
}
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox