* [PATCH net-next] net: speedup skb_splice_bits() @ 2012-04-22 22:26 Eric Dumazet 2012-04-24 3:02 ` David Miller 0 siblings, 1 reply; 5+ messages in thread From: Eric Dumazet @ 2012-04-22 22:26 UTC (permalink / raw) To: David Miller; +Cc: netdev, Tom Herbert, Jens Axboe From: Eric Dumazet <edumazet@google.com> Commit 35f3d14db (pipe: add support for shrinking and growing pipes) added a slowdown for splice(socket -> pipe), as we might grow the spd used in skb_splice_bits() for each skb we process in splice() syscall. Its not needed since skb lengths are capped. The default on-stack arrays are more than enough. Use MAX_SKB_FRAGS instead of PIPE_DEF_BUFFERS to describe the reasonable limit per skb. Add coalescing support to help splicing of GRO skbs built from linear skbs (linked into frag_list) Signed-off-by: Eric Dumazet <edumazet@google.com> Cc: Jens Axboe <axboe@kernel.dk> Cc: Tom Herbert <therbert@google.com> --- net/core/skbuff.c | 30 +++++++++++++++++++----------- 1 file changed, 19 insertions(+), 11 deletions(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index bf257de..dfb3040 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -1581,11 +1581,20 @@ new_page: memcpy(page_address(p) + off, page_address(page) + *offset, *len); sk->sk_sndmsg_off += *len; *offset = off; - get_page(p); return p; } +static bool spd_can_coalesce(const struct splice_pipe_desc *spd, + struct page *page, + unsigned int offset) +{ + return spd->nr_pages && + spd->pages[spd->nr_pages - 1] == page && + (spd->partial[spd->nr_pages - 1].offset + + spd->partial[spd->nr_pages - 1].len == offset); +} + /* * Fill page/offset/length into spd, if it can hold more pages. */ @@ -1595,16 +1604,19 @@ static inline int spd_fill_page(struct splice_pipe_desc *spd, struct sk_buff *skb, int linear, struct sock *sk) { - if (unlikely(spd->nr_pages == pipe->buffers)) + if (unlikely(spd->nr_pages == MAX_SKB_FRAGS)) return 1; if (linear) { page = linear_to_page(page, len, &offset, skb, sk); if (!page) return 1; - } else - get_page(page); - + } + if (spd_can_coalesce(spd, page, offset)) { + spd->partial[spd->nr_pages - 1].len += *len; + return 0; + } + get_page(page); spd->pages[spd->nr_pages] = page; spd->partial[spd->nr_pages].len = *len; spd->partial[spd->nr_pages].offset = offset; @@ -1710,8 +1722,8 @@ int skb_splice_bits(struct sk_buff *skb, unsigned int offset, struct pipe_inode_info *pipe, unsigned int tlen, unsigned int flags) { - struct partial_page partial[PIPE_DEF_BUFFERS]; - struct page *pages[PIPE_DEF_BUFFERS]; + struct partial_page partial[MAX_SKB_FRAGS]; + struct page *pages[MAX_SKB_FRAGS]; struct splice_pipe_desc spd = { .pages = pages, .partial = partial, @@ -1723,9 +1735,6 @@ int skb_splice_bits(struct sk_buff *skb, unsigned int offset, struct sock *sk = skb->sk; int ret = 0; - if (splice_grow_spd(pipe, &spd)) - return -ENOMEM; - /* * __skb_splice_bits() only fails if the output has no room left, * so no point in going over the frag_list for the error case. @@ -1761,7 +1770,6 @@ done: lock_sock(sk); } - splice_shrink_spd(pipe, &spd); return ret; } ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH net-next] net: speedup skb_splice_bits() 2012-04-22 22:26 [PATCH net-next] net: speedup skb_splice_bits() Eric Dumazet @ 2012-04-24 3:02 ` David Miller 2012-04-24 3:06 ` David Miller 0 siblings, 1 reply; 5+ messages in thread From: David Miller @ 2012-04-24 3:02 UTC (permalink / raw) To: eric.dumazet; +Cc: netdev, therbert, axboe From: Eric Dumazet <eric.dumazet@gmail.com> Date: Mon, 23 Apr 2012 00:26:16 +0200 > From: Eric Dumazet <edumazet@google.com> > > Commit 35f3d14db (pipe: add support for shrinking and growing pipes) > added a slowdown for splice(socket -> pipe), as we might grow the spd > used in skb_splice_bits() for each skb we process in splice() syscall. > > Its not needed since skb lengths are capped. The default on-stack arrays > are more than enough. > > Use MAX_SKB_FRAGS instead of PIPE_DEF_BUFFERS to describe the reasonable > limit per skb. > > Add coalescing support to help splicing of GRO skbs built from linear > skbs (linked into frag_list) > > Signed-off-by: Eric Dumazet <edumazet@google.com> > Cc: Jens Axboe <axboe@kernel.dk> > Cc: Tom Herbert <therbert@google.com> Applied thanks. I'm going to cleanup spd_fill_page() to return a bool. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH net-next] net: speedup skb_splice_bits() 2012-04-24 3:02 ` David Miller @ 2012-04-24 3:06 ` David Miller 2012-04-24 3:18 ` Eric Dumazet 0 siblings, 1 reply; 5+ messages in thread From: David Miller @ 2012-04-24 3:06 UTC (permalink / raw) To: eric.dumazet; +Cc: netdev, therbert, axboe From: David Miller <davem@davemloft.net> Date: Mon, 23 Apr 2012 23:02:05 -0400 (EDT) > I'm going to cleanup spd_fill_page() to return a bool. Done, thusly: -------------------- net: Use bool and remove inline in skb_splice_bits() code. Signed-off-by: David S. Miller <davem@davemloft.net> --- net/core/skbuff.c | 58 ++++++++++++++++++++++++++--------------------------- 1 file changed, 29 insertions(+), 29 deletions(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index dfb3040..aaf4abc 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -1547,9 +1547,9 @@ static void sock_spd_release(struct splice_pipe_desc *spd, unsigned int i) put_page(spd->pages[i]); } -static inline struct page *linear_to_page(struct page *page, unsigned int *len, - unsigned int *offset, - struct sk_buff *skb, struct sock *sk) +static struct page *linear_to_page(struct page *page, unsigned int *len, + unsigned int *offset, + struct sk_buff *skb, struct sock *sk) { struct page *p = sk->sk_sndmsg_page; unsigned int off; @@ -1598,23 +1598,23 @@ static bool spd_can_coalesce(const struct splice_pipe_desc *spd, /* * Fill page/offset/length into spd, if it can hold more pages. */ -static inline int spd_fill_page(struct splice_pipe_desc *spd, - struct pipe_inode_info *pipe, struct page *page, - unsigned int *len, unsigned int offset, - struct sk_buff *skb, int linear, - struct sock *sk) +static bool spd_fill_page(struct splice_pipe_desc *spd, + struct pipe_inode_info *pipe, struct page *page, + unsigned int *len, unsigned int offset, + struct sk_buff *skb, int linear, + struct sock *sk) { if (unlikely(spd->nr_pages == MAX_SKB_FRAGS)) - return 1; + return true; if (linear) { page = linear_to_page(page, len, &offset, skb, sk); if (!page) - return 1; + return true; } if (spd_can_coalesce(spd, page, offset)) { spd->partial[spd->nr_pages - 1].len += *len; - return 0; + return false; } get_page(page); spd->pages[spd->nr_pages] = page; @@ -1622,7 +1622,7 @@ static inline int spd_fill_page(struct splice_pipe_desc *spd, spd->partial[spd->nr_pages].offset = offset; spd->nr_pages++; - return 0; + return false; } static inline void __segment_seek(struct page **page, unsigned int *poff, @@ -1639,20 +1639,20 @@ static inline void __segment_seek(struct page **page, unsigned int *poff, *plen -= off; } -static inline int __splice_segment(struct page *page, unsigned int poff, - unsigned int plen, unsigned int *off, - unsigned int *len, struct sk_buff *skb, - struct splice_pipe_desc *spd, int linear, - struct sock *sk, - struct pipe_inode_info *pipe) +static bool __splice_segment(struct page *page, unsigned int poff, + unsigned int plen, unsigned int *off, + unsigned int *len, struct sk_buff *skb, + struct splice_pipe_desc *spd, int linear, + struct sock *sk, + struct pipe_inode_info *pipe) { if (!*len) - return 1; + return true; /* skip this segment if already processed */ if (*off >= plen) { *off -= plen; - return 0; + return false; } /* ignore any bits we already processed */ @@ -1668,23 +1668,23 @@ static inline int __splice_segment(struct page *page, unsigned int poff, flen = min_t(unsigned int, flen, PAGE_SIZE - poff); if (spd_fill_page(spd, pipe, page, &flen, poff, skb, linear, sk)) - return 1; + return true; __segment_seek(&page, &poff, &plen, flen); *len -= flen; } while (*len && plen); - return 0; + return false; } /* - * Map linear and fragment data from the skb to spd. It reports failure if the + * Map linear and fragment data from the skb to spd. It reports true if the * pipe is full or if we already spliced the requested length. */ -static int __skb_splice_bits(struct sk_buff *skb, struct pipe_inode_info *pipe, - unsigned int *offset, unsigned int *len, - struct splice_pipe_desc *spd, struct sock *sk) +static bool __skb_splice_bits(struct sk_buff *skb, struct pipe_inode_info *pipe, + unsigned int *offset, unsigned int *len, + struct splice_pipe_desc *spd, struct sock *sk) { int seg; @@ -1695,7 +1695,7 @@ static int __skb_splice_bits(struct sk_buff *skb, struct pipe_inode_info *pipe, (unsigned long) skb->data & (PAGE_SIZE - 1), skb_headlen(skb), offset, len, skb, spd, 1, sk, pipe)) - return 1; + return true; /* * then map the fragments @@ -1706,10 +1706,10 @@ static int __skb_splice_bits(struct sk_buff *skb, struct pipe_inode_info *pipe, if (__splice_segment(skb_frag_page(f), f->page_offset, skb_frag_size(f), offset, len, skb, spd, 0, sk, pipe)) - return 1; + return true; } - return 0; + return false; } /* -- 1.7.10 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH net-next] net: speedup skb_splice_bits() 2012-04-24 3:06 ` David Miller @ 2012-04-24 3:18 ` Eric Dumazet 2012-04-24 3:35 ` David Miller 0 siblings, 1 reply; 5+ messages in thread From: Eric Dumazet @ 2012-04-24 3:18 UTC (permalink / raw) To: David Miller; +Cc: netdev, therbert, axboe On Mon, 2012-04-23 at 23:06 -0400, David Miller wrote: > From: David Miller <davem@davemloft.net> > Date: Mon, 23 Apr 2012 23:02:05 -0400 (EDT) > > > I'm going to cleanup spd_fill_page() to return a bool. > > Done, thusly: > > -------------------- > net: Use bool and remove inline in skb_splice_bits() code. > > Signed-off-by: David S. Miller <davem@davemloft.net> > --- > net/core/skbuff.c | 58 ++++++++++++++++++++++++++--------------------------- > 1 file changed, 29 insertions(+), 29 deletions(-) > > diff --git a/net/core/skbuff.c b/net/core/skbuff.c > index dfb3040..aaf4abc 100644 > --- a/net/core/skbuff.c > +++ b/net/core/skbuff.c > @@ -1547,9 +1547,9 @@ static void sock_spd_release(struct splice_pipe_desc *spd, unsigned int i) > put_page(spd->pages[i]); > } > > -static inline struct page *linear_to_page(struct page *page, unsigned int *len, > - unsigned int *offset, > - struct sk_buff *skb, struct sock *sk) > +static struct page *linear_to_page(struct page *page, unsigned int *len, > + unsigned int *offset, > + struct sk_buff *skb, struct sock *sk) > { > struct page *p = sk->sk_sndmsg_page; > unsigned int off; > @@ -1598,23 +1598,23 @@ static bool spd_can_coalesce(const struct splice_pipe_desc *spd, > /* > * Fill page/offset/length into spd, if it can hold more pages. > */ > -static inline int spd_fill_page(struct splice_pipe_desc *spd, > - struct pipe_inode_info *pipe, struct page *page, > - unsigned int *len, unsigned int offset, > - struct sk_buff *skb, int linear, > - struct sock *sk) > +static bool spd_fill_page(struct splice_pipe_desc *spd, > + struct pipe_inode_info *pipe, struct page *page, > + unsigned int *len, unsigned int offset, > + struct sk_buff *skb, int linear, linear could be a bool too [PATCH] net: make spd_fill_page() linear argument a bool Signed-off-by: Eric Dumazet <edumazet@google.com> --- net/core/skbuff.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index aaf4abc..2342a72 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -1601,7 +1601,7 @@ static bool spd_can_coalesce(const struct splice_pipe_desc *spd, static bool spd_fill_page(struct splice_pipe_desc *spd, struct pipe_inode_info *pipe, struct page *page, unsigned int *len, unsigned int offset, - struct sk_buff *skb, int linear, + struct sk_buff *skb, bool linear, struct sock *sk) { if (unlikely(spd->nr_pages == MAX_SKB_FRAGS)) @@ -1642,7 +1642,7 @@ static inline void __segment_seek(struct page **page, unsigned int *poff, static bool __splice_segment(struct page *page, unsigned int poff, unsigned int plen, unsigned int *off, unsigned int *len, struct sk_buff *skb, - struct splice_pipe_desc *spd, int linear, + struct splice_pipe_desc *spd, bool linear, struct sock *sk, struct pipe_inode_info *pipe) { @@ -1694,7 +1694,7 @@ static bool __skb_splice_bits(struct sk_buff *skb, struct pipe_inode_info *pipe, if (__splice_segment(virt_to_page(skb->data), (unsigned long) skb->data & (PAGE_SIZE - 1), skb_headlen(skb), - offset, len, skb, spd, 1, sk, pipe)) + offset, len, skb, spd, true, sk, pipe)) return true; /* @@ -1705,7 +1705,7 @@ static bool __skb_splice_bits(struct sk_buff *skb, struct pipe_inode_info *pipe, if (__splice_segment(skb_frag_page(f), f->page_offset, skb_frag_size(f), - offset, len, skb, spd, 0, sk, pipe)) + offset, len, skb, spd, false, sk, pipe)) return true; } ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH net-next] net: speedup skb_splice_bits() 2012-04-24 3:18 ` Eric Dumazet @ 2012-04-24 3:35 ` David Miller 0 siblings, 0 replies; 5+ messages in thread From: David Miller @ 2012-04-24 3:35 UTC (permalink / raw) To: eric.dumazet; +Cc: netdev, therbert, axboe From: Eric Dumazet <eric.dumazet@gmail.com> Date: Tue, 24 Apr 2012 05:18:23 +0200 > linear could be a bool too Yep. > [PATCH] net: make spd_fill_page() linear argument a bool > > Signed-off-by: Eric Dumazet <edumazet@google.com> Applied, thanks! ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2012-04-24 3:35 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-04-22 22:26 [PATCH net-next] net: speedup skb_splice_bits() Eric Dumazet 2012-04-24 3:02 ` David Miller 2012-04-24 3:06 ` David Miller 2012-04-24 3:18 ` Eric Dumazet 2012-04-24 3:35 ` David Miller
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).