Mapping PCIe BAR as PMEM

Mapping PCIe BAR as PMEM

[Vijairaj]

Hi,
I am running the 4.4 kernel on AMD64 and was wondering what's a good way of
mapping the 16M of battery backed SRAM on a custom PCIe card as PMEM. Once
mapped, I will create a file system on the block device.

I am thinking about doing the following:
    - Use the kernel parameter memmap=nn!ss to reserve the PCIe BAR
    - Write a PCI driver to enable the PCI device `pci_enable_device()`
    - Load the PCI driver
    - Load the nd_pmem driver

Is this sufficient or is there anything else required to be done in the PCI
driver?

thanks,
Vijairaj
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: Mapping PCIe BAR as PMEM

[Dan Williams]

On Wed, Jun 15, 2016 at 6:57 AM, Vijairaj <vijairaj.r@gmail.com> wrote:
> Hi,
> I am running the 4.4 kernel on AMD64 and was wondering what's a good way of
> mapping the 16M of battery backed SRAM on a custom PCIe card as PMEM. Once
> mapped, I will create a file system on the block device.
>
> I am thinking about doing the following:
>     - Use the kernel parameter memmap=nn!ss to reserve the PCIe BAR

You don't need memmap= because the kernel can discover the PCIe BAR
like any other pci device (i.e. with a struct pci_device_id table).

>     - Write a PCI driver to enable the PCI device `pci_enable_device()`
>     - Load the PCI driver
>     - Load the nd_pmem driver

The pmem driver will load automatically if you register a pmem region
with nvdimm_pmem_region_create().

> Is this sufficient or is there anything else required to be done in the PCI
> driver?

Take a look at drivers/nvdimm/e820.c and copy/convert it into
drivers/nvdimm/pci.c.

One thing to note is that a PCIe card is not I/O cache coherent
meaning it can't be used in the same scenarios a host
memory-controller based persistent memory.  One change you should
probably make compared to the e820 driver is to not set the
ND_REGION_PAGEMAP bit.
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: Mapping PCIe BAR as PMEM

[Logan Gunthorpe]

Hey Vijairaj,

This is essentially what we did with our iopmem work [1].

We wrote a PCI driver (see [2] for an example) and had a small patch to 
the kernel to enable mappings with write combining.

Logan

[1] http://www.spinics.net/lists/linux-mm/msg103990.html
[2] https://github.com/sbates130272/iopmem

On 15/06/16 07:57 AM, Vijairaj wrote:
> Hi,
> I am running the 4.4 kernel on AMD64 and was wondering what's a good way of
> mapping the 16M of battery backed SRAM on a custom PCIe card as PMEM. Once
> mapped, I will create a file system on the block device.
>
> I am thinking about doing the following:
>      - Use the kernel parameter memmap=nn!ss to reserve the PCIe BAR
>      - Write a PCI driver to enable the PCI device `pci_enable_device()`
>      - Load the PCI driver
>      - Load the nd_pmem driver
>
> Is this sufficient or is there anything else required to be done in the PCI
> driver?
>
> thanks,
> Vijairaj
> _______________________________________________
> Linux-nvdimm mailing list
> Linux-nvdimm@lists.01.org
> https://lists.01.org/mailman/listinfo/linux-nvdimm
>
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm