[SPDK] SPDK failed to access CMB cards

[SPDK] SPDK failed to access CMB cards

[Srinath Mannam]

[-- Attachment #1: Type: text/plain, Size: 2273 bytes --]

Hi,

SQ address for CMB cards is configured as offset of CMB BAR physical
address(CPU address).

// SPDK source lib/nvme/nvme_pcie.c

/* cmd and cpl rings must be aligned on page size boundaries. */
        if (ctrlr->opts.use_cmb_sqs) {
                if (nvme_pcie_ctrlr_alloc_cmb(ctrlr, pqpair->num_entries *
sizeof(struct spdk_nvme_cmd),
                                              page_size, &offset) == 0) {
                        pqpair->cmd = pctrlr->cmb_bar_virt_addr + offset;

                        pqpair->cmd_bus_addr = pctrlr->cmb_bar_phys_addr +
offset;     // physical address of cmb bar is assigned to SQ address.

                        pqpair->sq_in_cmb = true;
                }
        }

Outbound mapping in our platforms, CPU address is not same as PCI bus
address.
If we program CPU address (CMB bar physical address) as SQ address. then
CMB card will send read request  to corresponding CPU address over PCIe
bus, which is invalid inbound address to our RC so it is rejected as UR.

For the same issue, we have a fix in kernel driver. The fix is, SQ address
has to configure rather PCI bus address than CPU address.
In this case CMB card read SQ from its local memory (CMB bar memory) without
generating memory read on PCIe bus.

We need same type of fix in spdk nvme library also.

We think two different solutions to solve this issue.
1. Get PCI bus address to corresponding CPU address (SMB bar address)
through sysfs entry, in the same way we get BAR phy address with the read
of "resource" file in sysfs. This case we program PCI bus address as SQ
address.  This fix is similar to the kernel driver fix.
2. Create IOMMU entry for CPU address to make it as valid inbound address.
In the case we program CPU address as SQ address.

With the second solution we observed throughput drop to two third of first
solution. Because every read to SQ from CMB card goes to RC with CPU
address and route back to CMB card with PCI bus address.
In first solution CMB card directly read SQ on its own memory without
generating memory read on PCIe bus.

We are using SPDK 18.04 version.
Please advice if any solution available in the latest release version or
pending patches.


Regards,
Srinath.

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 3705 bytes --]

Re: [SPDK] SPDK failed to access CMB cards

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 1100 bytes --]

On Mon, 2018-06-18 at 20:33 +0530, Srinath Mannam wrote:
> Outbound mapping in our platforms, CPU address is not same as PCI bus address.
> If we program CPU address (CMB bar physical address) as SQ address. then CMB
> card will send read request  to corresponding CPU address over PCIe bus, which
> is invalid inbound address to our RC so it is rejected as UR.
> 
> For the same issue, we have a fix in kernel driver. The fix is, SQ address has
> to configure rather PCI bus address than CPU address.
> In this case CMB card read SQ from its local memory (CMB bar memory) without
> generating memory read on PCIe bus.
> 
> We need same type of fix in spdk nvme library also.
> 
> We think two different solutions to solve this issue.
> 1. Get PCI bus address to corresponding CPU address (SMB bar address) through
> sysfs entry, in the same way we get BAR phy address with the read of
> "resource" file in sysfs. This case we program PCI bus address as SQ address. 
> This fix is similar to the kernel driver fix. 

Which sysfs entry contains the proper address?

Thanks,
Ben

Re: [SPDK] SPDK failed to access CMB cards

[Srinath Mannam]

[-- Attachment #1: Type: text/plain, Size: 2549 bytes --]

++

Hi,

Please provide me your inputs for the issue described in the below email.

On Mon, Jun 18, 2018 at 8:33 PM, Srinath Mannam
<srinath.mannam(a)broadcom.com> wrote:
> Hi,
>
> SQ address for CMB cards is configured as offset of CMB BAR physical
> address(CPU address).
>
> // SPDK source lib/nvme/nvme_pcie.c
>
> /* cmd and cpl rings must be aligned on page size boundaries. */
>         if (ctrlr->opts.use_cmb_sqs) {
>                 if (nvme_pcie_ctrlr_alloc_cmb(ctrlr, pqpair->num_entries *
> sizeof(struct spdk_nvme_cmd),
>                                               page_size, &offset) == 0) {
>                         pqpair->cmd = pctrlr->cmb_bar_virt_addr + offset;
>
>                         pqpair->cmd_bus_addr = pctrlr->cmb_bar_phys_addr +
> offset;     // physical address of cmb bar is assigned to SQ address.
>
>                         pqpair->sq_in_cmb = true;
>                 }
>         }
>
> Outbound mapping in our platforms, CPU address is not same as PCI bus
> address.
> If we program CPU address (CMB bar physical address) as SQ address. then CMB
> card will send read request  to corresponding CPU address over PCIe bus,
> which is invalid inbound address to our RC so it is rejected as UR.
>
> For the same issue, we have a fix in kernel driver. The fix is, SQ address
> has to configure rather PCI bus address than CPU address.
> In this case CMB card read SQ from its local memory (CMB bar memory) without
> generating memory read on PCIe bus.
>
> We need same type of fix in spdk nvme library also.
>
> We think two different solutions to solve this issue.
> 1. Get PCI bus address to corresponding CPU address (SMB bar address)
> through sysfs entry, in the same way we get BAR phy address with the read of
> "resource" file in sysfs. This case we program PCI bus address as SQ
> address.  This fix is similar to the kernel driver fix.
> 2. Create IOMMU entry for CPU address to make it as valid inbound address.
> In the case we program CPU address as SQ address.
>
> With the second solution we observed throughput drop to two third of first
> solution. Because every read to SQ from CMB card goes to RC with CPU address
> and route back to CMB card with PCI bus address.
> In first solution CMB card directly read SQ on its own memory without
> generating memory read on PCIe bus.
>
> We are using SPDK 18.04 version.
> Please advice if any solution available in the latest release version or
> pending patches.
>
>
> Regards,
> Srinath.