All of lore.kernel.org
 help / color / mirror / Atom feed
From: Christoph Hellwig <hch@lst.de>
To: Jason Gunthorpe <jgg@ziepe.ca>
Cc: linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org,
	iommu@lists.linux-foundation.org,
	Bjorn Helgaas <bhelgaas@google.com>,
	Logan Gunthorpe <logang@deltatee.com>,
	Christoph Hellwig <hch@lst.de>
Subject: Re: [PATCH 2/5] RDMA/core: remove use of dma_virt_ops
Date: Wed, 4 Nov 2020 15:01:08 +0100	[thread overview]
Message-ID: <20201104140108.GA5674@lst.de> (raw)
In-Reply-To: <20201104134241.GP36674@ziepe.ca>

On Wed, Nov 04, 2020 at 09:42:41AM -0400, Jason Gunthorpe wrote:
> On Wed, Nov 04, 2020 at 10:50:49AM +0100, Christoph Hellwig wrote:
> 
> > +int ib_dma_virt_map_sg(struct ib_device *dev, struct scatterlist *sg, int nents)
> > +{
> > +	struct scatterlist *s;
> > +	int i;
> > +
> > +	for_each_sg(sg, s, nents, i) {
> > +		sg_dma_address(s) = (uintptr_t)sg_virt(s);
> > +		sg_dma_len(s) = s->length;
> 
> Hmm.. There is nothing ensuring the page is mapped here for this
> sg_virt(). Before maybe some of the kconfig stuff prevented highmem
> systems indirectly, I wonder if we should add something more direct to
> exclude highmem for these drivers?

I had actually noticed this earlier as well and then completely forgot
about it..

rdmavt depends on X86_64, so it can't be used with highmem, but for
rxe and siw there weren't any such dependencies so I think we were just
lucky.  Let me send a fix to add explicit depencies and then respin this
series on top of that..

> Sigh. I think the proper fix is to replace addr/length with a
> scatterlist pointer in the struct ib_sge, then have SW drivers
> directly use the page pointer properly.

The proper fix is to move the DMA mapping into the RDMA core, yes.
And as you said it will be hard.  But I don't think scatterlists
are the right interface.  IMHO we can keep re-use the existing
struct ib_sge:

struct ib_ge {
	u64     addr;
	u32     length;
	u32     lkey;
};

with the difference that if lkey is not a MR, addr is the physical
address of the memory, not a dma_addr_t or virtual address.
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

WARNING: multiple messages have this Message-ID (diff)
From: Christoph Hellwig <hch@lst.de>
To: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Christoph Hellwig <hch@lst.de>,
	Bjorn Helgaas <bhelgaas@google.com>,
	Logan Gunthorpe <logang@deltatee.com>,
	linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org,
	iommu@lists.linux-foundation.org
Subject: Re: [PATCH 2/5] RDMA/core: remove use of dma_virt_ops
Date: Wed, 4 Nov 2020 15:01:08 +0100	[thread overview]
Message-ID: <20201104140108.GA5674@lst.de> (raw)
In-Reply-To: <20201104134241.GP36674@ziepe.ca>

On Wed, Nov 04, 2020 at 09:42:41AM -0400, Jason Gunthorpe wrote:
> On Wed, Nov 04, 2020 at 10:50:49AM +0100, Christoph Hellwig wrote:
> 
> > +int ib_dma_virt_map_sg(struct ib_device *dev, struct scatterlist *sg, int nents)
> > +{
> > +	struct scatterlist *s;
> > +	int i;
> > +
> > +	for_each_sg(sg, s, nents, i) {
> > +		sg_dma_address(s) = (uintptr_t)sg_virt(s);
> > +		sg_dma_len(s) = s->length;
> 
> Hmm.. There is nothing ensuring the page is mapped here for this
> sg_virt(). Before maybe some of the kconfig stuff prevented highmem
> systems indirectly, I wonder if we should add something more direct to
> exclude highmem for these drivers?

I had actually noticed this earlier as well and then completely forgot
about it..

rdmavt depends on X86_64, so it can't be used with highmem, but for
rxe and siw there weren't any such dependencies so I think we were just
lucky.  Let me send a fix to add explicit depencies and then respin this
series on top of that..

> Sigh. I think the proper fix is to replace addr/length with a
> scatterlist pointer in the struct ib_sge, then have SW drivers
> directly use the page pointer properly.

The proper fix is to move the DMA mapping into the RDMA core, yes.
And as you said it will be hard.  But I don't think scatterlists
are the right interface.  IMHO we can keep re-use the existing
struct ib_sge:

struct ib_ge {
	u64     addr;
	u32     length;
	u32     lkey;
};

with the difference that if lkey is not a MR, addr is the physical
address of the memory, not a dma_addr_t or virtual address.

  reply	other threads:[~2020-11-04 14:01 UTC|newest]

Thread overview: 32+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-11-04  9:50 remove dma_virt_ops Christoph Hellwig
2020-11-04  9:50 ` Christoph Hellwig
2020-11-04  9:50 ` [PATCH 1/5] RDMA/core: remove ib_dma_{alloc,free}_coherent Christoph Hellwig
2020-11-04  9:50   ` Christoph Hellwig
2020-11-04  9:50 ` [PATCH 2/5] RDMA/core: remove use of dma_virt_ops Christoph Hellwig
2020-11-04  9:50   ` Christoph Hellwig
2020-11-04 13:42   ` Jason Gunthorpe
2020-11-04 13:42     ` Jason Gunthorpe
2020-11-04 14:01     ` Christoph Hellwig [this message]
2020-11-04 14:01       ` Christoph Hellwig
2020-11-04 15:09       ` Bernard Metzler
2020-11-04 15:09         ` Bernard Metzler
2020-11-04 18:10         ` Jason Gunthorpe
2020-11-04 18:10           ` Jason Gunthorpe
2020-11-04 15:52       ` Jason Gunthorpe
2020-11-04 15:52         ` Jason Gunthorpe
2020-11-04 16:31         ` Christoph Hellwig
2020-11-04 16:31           ` Christoph Hellwig
2020-11-04 18:09           ` Jason Gunthorpe
2020-11-04 18:09             ` Jason Gunthorpe
2020-11-04  9:50 ` [PATCH 3/5] PCI/p2p: remove the DMA_VIRT_OPS hacks Christoph Hellwig
2020-11-04  9:50   ` Christoph Hellwig
2020-11-04 16:53   ` Bjorn Helgaas
2020-11-04 16:53     ` Bjorn Helgaas
2020-11-04 17:00   ` Logan Gunthorpe
2020-11-04 17:00     ` Logan Gunthorpe
2020-11-04  9:50 ` [PATCH 4/5] PCI/p2p: cleanup up __pci_p2pdma_map_sg a bit Christoph Hellwig
2020-11-04  9:50   ` Christoph Hellwig
2020-11-04 16:54   ` Bjorn Helgaas
2020-11-04 16:54     ` Bjorn Helgaas
2020-11-04  9:50 ` [PATCH 5/5] dma-mapping: remove dma_virt_ops Christoph Hellwig
2020-11-04  9:50   ` Christoph Hellwig

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20201104140108.GA5674@lst.de \
    --to=hch@lst.de \
    --cc=bhelgaas@google.com \
    --cc=iommu@lists.linux-foundation.org \
    --cc=jgg@ziepe.ca \
    --cc=linux-pci@vger.kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=logang@deltatee.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.