All of lore.kernel.org
 help / color / mirror / Atom feed
From: Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
To: Roland Dreier <rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>
Cc: Dan Carpenter <error27-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>,
	Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>,
	Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>,
	Hal Rosenstock
	<hal.rosenstock-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>,
	linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
	kernel-janitors-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: Re: [patch v4] infiniband: uverbs: handle large number of entries
Date: Thu, 25 Nov 2010 04:13:37 +0000	[thread overview]
Message-ID: <20101125041337.GA11049@obsidianresearch.com> (raw)
In-Reply-To: <adad3pugi90.fsf-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>

On Wed, Nov 24, 2010 at 08:05:47PM -0800, Roland Dreier wrote:

>  > So if you are worried about how many times ib_poll_cq is called then
>  > bound the kzalloc size and wrap the whole thing in a loop, but
>  > realistically I have to think the performance trade off of
>  > kzalloc/free vs calling ib_poll more often is not entirely obvious.
> 
> That's true... maybe doing things one at a time but avoiding the allocs
> is the right tradeoff.

Hmm, considering your list is everything but Mellanox, maybe it makes
much more sense to push the copy_to_user down into the driver - 
ie a ibv_poll_cq_user - then the driver can construct each CQ entry on
the stack and copy it to userspace, avoid the double copy, allocation
and avoid any fixed overhead of ibv_poll_cq.

A bigger change to be sure, but remember this old thread:

http://www.mail-archive.com/linux-rdma@vger.kernel.org/msg05114.html

2x improvement by removing allocs on the post path..

Jason

WARNING: multiple messages have this Message-ID (diff)
From: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
To: Roland Dreier <rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>
Cc: Dan Carpenter <error27-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>,
	Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>,
	Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>,
	Hal Rosenstock
	<hal.rosenstock-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>,
	linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
	kernel-janitors-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: Re: [patch v4] infiniband: uverbs: handle large number of entries
Date: Wed, 24 Nov 2010 21:13:37 -0700	[thread overview]
Message-ID: <20101125041337.GA11049@obsidianresearch.com> (raw)
In-Reply-To: <adad3pugi90.fsf-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>

On Wed, Nov 24, 2010 at 08:05:47PM -0800, Roland Dreier wrote:

>  > So if you are worried about how many times ib_poll_cq is called then
>  > bound the kzalloc size and wrap the whole thing in a loop, but
>  > realistically I have to think the performance trade off of
>  > kzalloc/free vs calling ib_poll more often is not entirely obvious.
> 
> That's true... maybe doing things one at a time but avoiding the allocs
> is the right tradeoff.

Hmm, considering your list is everything but Mellanox, maybe it makes
much more sense to push the copy_to_user down into the driver - 
ie a ibv_poll_cq_user - then the driver can construct each CQ entry on
the stack and copy it to userspace, avoid the double copy, allocation
and avoid any fixed overhead of ibv_poll_cq.

A bigger change to be sure, but remember this old thread:

http://www.mail-archive.com/linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg05114.html

2x improvement by removing allocs on the post path..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

  parent reply	other threads:[~2010-11-25  4:13 UTC|newest]

Thread overview: 48+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2010-10-07  7:16 [patch] infiniband: uverbs: limit the number of entries Dan Carpenter
2010-10-07  7:16 ` Dan Carpenter
2010-10-07 16:16 ` Jason Gunthorpe
2010-10-07 16:16   ` Jason Gunthorpe
     [not found]   ` <20101007161649.GD21206-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2010-10-07 16:59     ` Dan Carpenter
2010-10-07 16:59       ` Dan Carpenter
2010-10-08  7:59       ` Nicolas Palix
2010-10-08  7:59         ` Nicolas Palix
     [not found]         ` <AANLkTin5zou2JHsdDyhGESuxyPonOs3kLo9Th0vg-kd8-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2010-10-08 14:25           ` [patch v2] " Dan Carpenter
2010-10-08 14:25             ` Dan Carpenter
2010-10-09 23:16       ` [patch] " Jason Gunthorpe
2010-10-09 23:16         ` Jason Gunthorpe
     [not found]         ` <20101009231607.GA24649-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2010-10-12 11:31           ` [patch v3] infiniband: uverbs: handle large " Dan Carpenter
2010-10-12 11:31             ` Dan Carpenter
2010-10-12 21:01             ` Jason Gunthorpe
2010-10-12 21:01               ` Jason Gunthorpe
     [not found]               ` <20101012210118.GR24268-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2010-10-13  9:05                 ` Dan Carpenter
2010-10-13  9:05                   ` Dan Carpenter
2010-10-13  9:13                 ` [patch v4] " Dan Carpenter
2010-10-13  9:13                   ` Dan Carpenter
2010-11-23  7:10                   ` Dan Carpenter
2010-11-23  7:10                     ` Dan Carpenter
2010-11-24 22:07                     ` Roland Dreier
2010-11-24 22:07                       ` Roland Dreier
     [not found]                       ` <adahbf6gytv.fsf-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>
2010-11-24 22:18                         ` Jason Gunthorpe
2010-11-24 22:18                           ` Jason Gunthorpe
     [not found]                           ` <20101124221845.GH2369-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2010-11-25  4:05                             ` Roland Dreier
2010-11-25  4:05                               ` Roland Dreier
     [not found]                               ` <adad3pugi90.fsf-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>
2010-11-25  4:13                                 ` Jason Gunthorpe [this message]
2010-11-25  4:13                                   ` Jason Gunthorpe
     [not found]                                   ` <20101125041337.GA11049-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2010-11-25 15:00                                     ` ibv_post_send/recv kernel path optimizations (was: uverbs: handle large number of entries) Or Gerlitz
     [not found]                                       ` <4CEE7A22.2040706-smomgflXvOZWk0Htik3J/w@public.gmane.org>
2010-11-26 11:56                                         ` Walukiewicz, Miroslaw
     [not found]                                           ` <BE2BFE91933D1B4089447C644860408064B44854-IGOiFh9zz4wLt2AQoY/u9bfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2010-12-01  8:11                                             ` ibv_post_send/recv kernel path optimizations Or Gerlitz
     [not found]                                               ` <4CF60343.7050602-smomgflXvOZWk0Htik3J/w@public.gmane.org>
2010-12-08 12:14                                                 ` Walukiewicz, Miroslaw
     [not found]                                                   ` <BE2BFE91933D1B4089447C64486040806673CF38-IGOiFh9zz4wLt2AQoY/u9bfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2010-12-08 18:30                                                     ` Jason Gunthorpe
2010-12-08 19:04                                                     ` Roland Dreier
2010-12-14 14:12                                                 ` Walukiewicz, Miroslaw
     [not found]                                                   ` <BE2BFE91933D1B4089447C644860408066ABCF66-IGOiFh9zz4wLt2AQoY/u9bfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2010-12-14 18:17                                                     ` Jason Gunthorpe
     [not found]                                                       ` <20101214181735.GA2506-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2010-12-27 12:38                                                         ` Or Gerlitz
     [not found]                                                           ` <4D1888CB.2010101-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org>
2010-12-27 15:18                                                             ` Walukiewicz, Miroslaw
     [not found]                                                               ` <BE2BFE91933D1B4089447C644860408066C547E0-IGOiFh9zz4wLt2AQoY/u9bfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2010-12-27 15:22                                                                 ` Or Gerlitz
     [not found]                                                                   ` <4D18AF2D.1010109-smomgflXvOZWk0Htik3J/w@public.gmane.org>
2010-12-27 15:40                                                                     ` Walukiewicz, Miroslaw
2011-01-05 18:16                                                                 ` Roland Dreier
     [not found]                                                                   ` <ada4o9nfc6e.fsf-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>
2011-01-10 14:15                                                                     ` Walukiewicz, Miroslaw
     [not found]                                                                       ` <BE2BFE91933D1B4089447C644860408066DDDF31-IGOiFh9zz4wLt2AQoY/u9bfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2011-01-10 20:38                                                                         ` Roland Dreier
     [not found]                                                                           ` <adawrmc7av2.fsf-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>
2011-01-21 11:41                                                                             ` Walukiewicz, Miroslaw
     [not found]                                                                               ` <BE2BFE91933D1B4089447C644860408066F95285-IGOiFh9zz4wLt2AQoY/u9bfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2011-01-21 15:49                                                                                 ` Hefty, Sean
     [not found]                                                                                   ` <CF9C39F99A89134C9CF9C4CCB68B8DDF25C1D51C31-osO9UTpF0USkrb+BlOpmy7fspsVTdybXVpNB7YpNyf8@public.gmane.org>
2011-01-24 17:03                                                                                     ` Walukiewicz, Miroslaw

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20101125041337.GA11049@obsidianresearch.com \
    --to=jgunthorpe@obsidianresearch.com \
    --cc=error27-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org \
    --cc=hal.rosenstock-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org \
    --cc=kernel-janitors-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
    --cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
    --cc=rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org \
    --cc=rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org \
    --cc=sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.