[SPDK] Controller Memory Buffer

[SPDK] Controller Memory Buffer

[Jonas Pfefferle]

[-- Attachment #1: Type: text/plain, Size: 166 bytes --]

Hi @all,

Is there any NVMe drive available which supports the optional "Controller 
Memory Buffer" feature introduced in NVMe 1.2?

Thanks & Regards,
Jonas

Re: [SPDK] Controller Memory Buffer

[Stephen Bates]

[-- Attachment #1: Type: text/plain, Size: 879 bytes --]

Jonas

There are some Intel drives that include a small (a few MB) CMB that supports SQS only. Both Eideticom and Everspin have NVMe devices that have CMBs that support the WDS and RDS modes as well as SQS.

I know that other vendors are pretty actively looking at this so I’d expect more NVMe SSDs with a variety of CMB modes later this year. I’d suggest pinging those companies and getting roadmap updates from them to understand their timelines.

Cheers

Stephen

> On Mar 9, 2018, at 9:15 AM, Jonas Pfefferle <pepperjo(a)japf.ch> wrote:
> 
> Hi @all,
> 
> Is there any NVMe drive available which supports the optional "Controller Memory Buffer" feature introduced in NVMe 1.2?
> 
> Thanks & Regards,
> Jonas
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] Controller Memory Buffer

[Jonas Pfefferle]

[-- Attachment #1: Type: text/plain, Size: 1761 bytes --]

Hi Stephen,

Thanks for the hints. Do you happen to know which Intel drives these are? I 
have a few Intel drives: DC P3600/3700 and 900p/P4800X but neither of those 
devices have CMB.

I'm experimenting with Mellanox ConnectX-5 target offloading so it would be 
nice to see the effects of having SQ/CQ and data staging on the device. From 
the experiments I've done so far it looks like the latencies are higher with 
the offloaded target compared to SPDK or the Kernel target. I assume this is 
partially due to the additional PCIe round trip times.

Thanks & Cheers,
Jonas

  On Sat, 10 Mar 2018 21:11:01 +0000
  "Stephen  Bates" <sbates(a)raithlin.com> wrote:
> Jonas
> 
> There are some Intel drives that include a small (a few MB) CMB that 
>supports SQS only. Both Eideticom and Everspin have NVMe devices that 
>have CMBs that support the WDS and RDS modes as well as SQS.
> 
> I know that other vendors are pretty actively looking at this so I’d 
>expect more NVMe SSDs with a variety of CMB modes later this year. 
>I’d suggest pinging those companies and getting roadmap updates from 
>them to understand their timelines.
> 
> Cheers
> 
> Stephen
> 
>> On Mar 9, 2018, at 9:15 AM, Jonas Pfefferle <pepperjo(a)japf.ch> 
>>wrote:
>> 
>> Hi @all,
>> 
>> Is there any NVMe drive available which supports the optional 
>>"Controller Memory Buffer" feature introduced in NVMe 1.2?
>> 
>> Thanks & Regards,
>> Jonas
>> 
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] Controller Memory Buffer

[Stephen Bates]

[-- Attachment #1: Type: text/plain, Size: 859 bytes --]

> Thanks for the hints. Do you happen to know which Intel drives these are? 

I don't have model numbers. Contact your Intel rep for those.
    
> I'm experimenting with Mellanox ConnectX-5 target offloading 

Cool. We are playing with that too! In fact we have a Linux kernel fork that mashes up Max's offload driver with our p2pdma framework for CMBs here - https://github.com/lsgunth/linux/tree/max-mlnx-offload-p2pdma

> the latencies are higher 

Do you have a PCIe switch or are the ConnectX-5 and NVMe SSD connected via the root complex on the (I presume Intel) CPU? If the later then I suspect some of your latency is coming from the non-performant MemWr TLP path that connects the root ports together in the root complex.

Stephen

PS Since we have completely diverged from SPDK at this point I suggest taking this off list ;-)...

Re: [SPDK] Controller Memory Buffer

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 442 bytes --]



On 3/11/18, 4:31 AM, "SPDK on behalf of Jonas Pfefferle" <spdk-bounces(a)lists.01.org on behalf of pepperjo(a)japf.ch> wrote:

    Hi Stephen,
    
    Thanks for the hints. Do you happen to know which Intel drives these are? I 
    have a few Intel drives: DC P3600/3700 and 900p/P4800X but neither of those 
  devices have CMB.
  
The Intel D3600/3700 (dual-port) SSDs have the SQS-only CMB that Stephen mentioned.

-Jim