From: Jason Gunthorpe <jgg@nvidia.com>
To: Bob Pearson <rpearsonhpe@gmail.com>
Cc: leon@kernel.org, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org
Subject: Re: [PATCH for-next v2 06/18] RDMA/rxe: Add rxe_add_frag() to rxe_mr.c
Date: Thu, 1 Dec 2022 14:00:35 -0400 [thread overview]
Message-ID: <Y4jrwwjzgj/AUg9n@nvidia.com> (raw)
In-Reply-To: <24f77b1b-b2e0-17f7-06b3-7ac17a247efc@gmail.com>
On Thu, Dec 01, 2022 at 11:11:10AM -0600, Bob Pearson wrote:
> On 12/1/22 09:39, Jason Gunthorpe wrote:
> > On Thu, Dec 01, 2022 at 09:38:10AM -0600, Bob Pearson wrote:
> >
> >> Further, looking at ipoib as an example, it builds sge lists using the lkey from get_dma_mr()
> >> and sets the sge->addr to a kernel virtual memory address after previously calling
> >> ib_dma_map_single() so the addresses are mapped for dma access and visible before use.
> >> They are unmapped after the read/write operation completes. What is the point of kmapping
> >> the address after dma mapping them?
> >
> > Because not everything is ipoib, and things like block will map sgls
> > with struct pages, not kva.
> >
> > Jason
>
> OK it's working now but there is a bug in your rxe_mr_fill_pages_from_sgt() routine.
> You have a
>
> if (xas_xa_index && WARN_ON(sg_iter.sg_pgoffset % PAGE_SIZE)) {...}
>
> which seems to assume that sg_pgoffset contains the byte offset in the current page.
> But looking at __sg_page_iter_next() it appears that it is the number of pages offset
> in the current sg entry which results in a splat when I run
> ib_send_bw.
The pgoffset depends on the creator of the SGL, I guess something must
be pointing the SGL ad a folio and using the sg_pgoffset as a sub
folio index?
If so then the calculation of the kmap has to be adjusted to extract
the number of pages from the sg_pgoffset
Jason
next prev parent reply other threads:[~2022-12-01 18:00 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-10-31 20:27 [PATCH for-next v2 01/18] RDMA/rxe: Isolate code to fill request roce headers Bob Pearson
2022-10-31 20:27 ` [PATCH for-next v2 02/18] RDMA/rxe: Isolate request payload code in a subroutine Bob Pearson
2022-10-31 20:27 ` [PATCH for-next v2 03/18] RDMA/rxe: Remove paylen parameter from rxe_init_packet Bob Pearson
2022-10-31 20:27 ` [PATCH for-next v2 04/18] RDMA/rxe: Isolate code to build request packet Bob Pearson
2022-10-31 20:27 ` [PATCH for-next v2 05/18] RDMA/rxe: Add sg fragment ops Bob Pearson
2022-11-24 19:05 ` Jason Gunthorpe
2022-10-31 20:27 ` [PATCH for-next v2 06/18] RDMA/rxe: Add rxe_add_frag() to rxe_mr.c Bob Pearson
2022-11-24 19:10 ` Jason Gunthorpe
2022-11-30 20:53 ` Bob Pearson
2022-11-30 23:36 ` Jason Gunthorpe
2022-12-01 0:16 ` Bob Pearson
2022-12-01 0:20 ` Jason Gunthorpe
2022-12-01 0:36 ` Bob Pearson
2022-12-01 0:41 ` Jason Gunthorpe
2022-12-01 5:05 ` Bob Pearson
2022-12-01 12:51 ` Jason Gunthorpe
2022-12-01 15:04 ` Bob Pearson
2022-12-01 15:16 ` Bob Pearson
2022-12-01 15:38 ` Bob Pearson
2022-12-01 15:39 ` Jason Gunthorpe
2022-12-01 17:11 ` Bob Pearson
2022-12-01 18:00 ` Jason Gunthorpe [this message]
2022-10-31 20:27 ` [PATCH for-next v2 07/18] RDMA/rxe: Add routine to compute the number of frags Bob Pearson
2022-11-24 19:15 ` Jason Gunthorpe
2022-10-31 20:27 ` [PATCH for-next v2 08/18] RDMA/rxe: Extend rxe_mr_copy to support skb frags Bob Pearson
2022-10-31 20:27 ` [PATCH for-next v2 09/18] RDMA/rxe: Add routine to compute number of frags for dma Bob Pearson
2022-11-24 19:16 ` Jason Gunthorpe
2022-10-31 20:27 ` [PATCH for-next v2 10/18] RDMA/rxe: Extend copy_data to support skb frags Bob Pearson
2022-10-31 20:28 ` [PATCH for-next v2 11/18] RDMA/rxe: Replace rxe by qp as a parameter Bob Pearson
2022-10-31 20:28 ` [PATCH for-next v2 12/18] RDMA/rxe: Extend rxe_init_packet() to support frags Bob Pearson
2022-10-31 20:28 ` [PATCH for-next v2 13/18] RDMA/rxe: Extend rxe_icrc.c " Bob Pearson
2022-10-31 20:28 ` [PATCH for-next v2 14/18] RDMA/rxe: Extend rxe_init_req_packet() for frags Bob Pearson
2022-10-31 20:28 ` [PATCH for-next v2 15/18] RDMA/rxe: Extend response packets " Bob Pearson
2022-10-31 20:28 ` [PATCH for-next v2 16/18] RDMA/rxe: Extend send/write_data_in() " Bob Pearson
2022-10-31 20:28 ` [PATCH for-next v2 17/18] RDMA/rxe: Extend do_read() in rxe_comp,c " Bob Pearson
2022-10-31 20:28 ` [PATCH for-next v2 18/18] RDMA/rxe: Enable sg code in rxe Bob Pearson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Y4jrwwjzgj/AUg9n@nvidia.com \
--to=jgg@nvidia.com \
--cc=leon@kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=rpearsonhpe@gmail.com \
--cc=zyjzyj2000@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox