Re: Enabling peer to peer device transactions for PCIe devices

Logan Gunthorpe
2016-12-05T18:39:13Z

On 05/12/16 11:08 AM, Dan Williams wrote:
> I've already recommended that iopmem not be a block device and instead
> be a device-dax instance. I also don't think it should claim the PCI
> ID, rather the driver that wants to map one of its bars this way can
> register the memory region with the device-dax core.
>
> I'm not sure there are enough device drivers that want to do this to
> have it be a generic /sys/.../resource_dmableX capability. It still
> seems to be an exotic one-off type of configuration.

Yes, this is essentially my thinking. Except I think the userspace
interface should really depend on the device itself. Device dax is a
good choice for many and I agree the block device approach wouldn't be
ideal.

Specifically for NVME CMB: I think it would make a lot of sense to just
hand out these mappings with an mmap call on /dev/nvmeX. I expect CMB
buffers would be volatile and thus you wouldn't need to keep track of
where in the BAR the region came from. Thus, the mmap call would just be
an allocator from BAR memory. If device-dax were used, userspace would
need to lookup which device-dax instance corresponds to which nvme drive.

Logan