NVMe and SR-IOV, how to use together?

NVMe and SR-IOV, how to use together?

[Bogdan Rudas]

Hi!

Today I've received from servers vendor modern Samsung SSD 1725a
MZPLL1T6HEHP for evaluation. I would like to use it as storage for
virtual machines. Before I will start doing anything with a device I
would like to re-check if I understand technology basics:

To divide the device among virtual machines I have to delete default
namespace and create a dedicated namespace for each VM.
When I will have at least 2 namespace, every of them will be exposed
in host operating system as virtual function of PCI device (i.e. I
will see lot of Virtual function for Non-Volatile memory controller)
Then I can do PCI passthru like I do it for any other SR-IOV PCI device.
Finally I have to remove all namespaces and re-create the single one
which will be exposed as single PCI device with no virtual functions.

Could someone review statements above and point me to useful
additional information? There are no user friendly information about
SR-IOV on NVMe devices in Linux, so I have to disturb the development
community.


Thank you!

-- 


CONFIDENTIALITY NOTICE: This email and files attached to it are 
confidential. If you are not the intended recipient you are hereby notified 
that using, copying, distributing or taking any action in reliance on the 
contents of this information is strictly prohibited. If you have received 
this email in error please notify the sender and delete this email.

NVMe and SR-IOV, how to use together?

[Christoph Hellwig]

SR-IOV itself only allows you to create new virtual functions,
you can look at any number of Linux howtos on creating them using
the sysfs interface.

As far as NVMe is concerned there are multiple levels of interaction
that could be possible, all optional.

First NVMe since version 1.1 supports subsystems with multiple
controlles, and you can use namespace management (if support by the
device) to assign namespaces to one or more controllers.  On Linux
you can use the nvme-cli tool for that.

Since NVMe 1.3 there are virtualization extensions that also allow to
assign resources like queue pairs to specific VMs.  There is no support
in Linux or nvme-cli for that at the moment.

NVMe and SR-IOV, how to use together?

[Bogdan Rudas]

I run kernel 4.4.27 which don't have a support for SR-IOV in NVMe
driver, that's why I do not see a VFs in lspci output.

Thank you.

On Mon, Sep 18, 2017@7:11 PM, Christoph Hellwig <hch@infradead.org> wrote:
> SR-IOV itself only allows you to create new virtual functions,
> you can look at any number of Linux howtos on creating them using
> the sysfs interface.
>
> As far as NVMe is concerned there are multiple levels of interaction
> that could be possible, all optional.
>
> First NVMe since version 1.1 supports subsystems with multiple
> controlles, and you can use namespace management (if support by the
> device) to assign namespaces to one or more controllers.  On Linux
> you can use the nvme-cli tool for that.
>
> Since NVMe 1.3 there are virtualization extensions that also allow to
> assign resources like queue pairs to specific VMs.  There is no support
> in Linux or nvme-cli for that at the moment.



-- 
Bogdan Rudas
Director of IT Europe
Exadel Inc.
http://www.exadel.com/
E-mail: brudas at exadel.com
Skype ID: bogdan.rudas

-- 


CONFIDENTIALITY NOTICE: This email and files attached to it are 
confidential. If you are not the intended recipient you are hereby notified 
that using, copying, distributing or taking any action in reliance on the 
contents of this information is strictly prohibited. If you have received 
this email in error please notify the sender and delete this email.

NVMe and SR-IOV, how to use together?

[Keith Busch]

On Mon, Sep 18, 2017@09:20:16PM +0300, Bogdan Rudas wrote:
> I run kernel 4.4.27 which don't have a support for SR-IOV in NVMe
> driver, that's why I do not see a VFs in lspci output.

The driver has nothing to do with what lspci will output. If you don't
see the SR-IOV capability structure there, that means your drive doesn't
support it.

NVMe and SR-IOV, how to use together?

[Bogdan Rudas]

Hi!

I've checked the device on 4.14.0 rc1 kernel. And see no SR-IOV files
for SSD PCI device in /sys/devices/..../sriov_*, SR-IOV works for
Intel I350 LOM at the same host.
However manufactures brochure claims the SR-IOV support for this
series. Should I report this as a bug in Linux nvme driver? Is so,
what the information should I gather?

Here is a brochure with SR-IOV mention:
http://www.samsung.com/semiconductor/global/file/insight/2016/08/Samsung_PM1725a-1.pdf

Here is SSD information:
root at ubuntu-test:~# nvme list
Node             SN                   Model
        Namespace Usage                      Format           FW Rev
---------------- --------------------
---------------------------------------- ---------
-------------------------- ---------------- --------
/dev/nvme0n1     S3HBNA0J600073       SAMSUNG MZPLL1T6HEHP-00003
        1           1.60  TB /   1.60  TB    512   B +  0 B   GPNA4B3Q

On Mon, Sep 18, 2017@9:41 PM, Keith Busch <keith.busch@intel.com> wrote:
> On Mon, Sep 18, 2017@09:20:16PM +0300, Bogdan Rudas wrote:
>> I run kernel 4.4.27 which don't have a support for SR-IOV in NVMe
>> driver, that's why I do not see a VFs in lspci output.
>
> The driver has nothing to do with what lspci will output. If you don't
> see the SR-IOV capability structure there, that means your drive doesn't
> support it.



-- 
Bogdan Rudas
Director of IT Europe
Exadel Inc.
http://www.exadel.com/
E-mail: brudas at exadel.com
Skype ID: bogdan.rudas

-- 


CONFIDENTIALITY NOTICE: This email and files attached to it are 
confidential. If you are not the intended recipient you are hereby notified 
that using, copying, distributing or taking any action in reliance on the 
contents of this information is strictly prohibited. If you have received 
this email in error please notify the sender and delete this email.

NVMe and SR-IOV, how to use together?

[Gruher, Joseph R]

It would be interesting to confirm the device really supports SR-IOV. At launch the device claimed it on paper but we were told firmware support had not been implemented. Do you have any support from the manufacturer to confirm the device HW/FW really supports SR-IOV today?


> On Sep 19, 2017,@1:31 PM, Bogdan Rudas <brudas@exadel.com> wrote:
> 
> Hi!
> 
> I've checked the device on 4.14.0 rc1 kernel. And see no SR-IOV files
> for SSD PCI device in /sys/devices/..../sriov_*, SR-IOV works for
> Intel I350 LOM at the same host.
> However manufactures brochure claims the SR-IOV support for this
> series. Should I report this as a bug in Linux nvme driver? Is so,
> what the information should I gather?
> 
> Here is a brochure with SR-IOV mention:
> http://www.samsung.com/semiconductor/global/file/insight/2016/08/Samsung_PM1725a-1.pdf
> 
> Here is SSD information:
> root at ubuntu-test:~# nvme list
> Node             SN                   Model
>        Namespace Usage                      Format           FW Rev
> ---------------- --------------------
> ---------------------------------------- ---------
> -------------------------- ---------------- --------
> /dev/nvme0n1     S3HBNA0J600073       SAMSUNG MZPLL1T6HEHP-00003
>        1           1.60  TB /   1.60  TB    512   B +  0 B   GPNA4B3Q
> 
>> On Mon, Sep 18, 2017@9:41 PM, Keith Busch <keith.busch@intel.com> wrote:
>>> On Mon, Sep 18, 2017@09:20:16PM +0300, Bogdan Rudas wrote:
>>> I run kernel 4.4.27 which don't have a support for SR-IOV in NVMe
>>> driver, that's why I do not see a VFs in lspci output.
>> 
>> The driver has nothing to do with what lspci will output. If you don't
>> see the SR-IOV capability structure there, that means your drive doesn't
>> support it.
> 
> 
> 
> -- 
> Bogdan Rudas
> Director of IT Europe
> Exadel Inc.
> http://www.exadel.com/
> E-mail: brudas at exadel.com
> Skype ID: bogdan.rudas
> 
> -- 
> 
> 
> CONFIDENTIALITY NOTICE: This email and files attached to it are 
> confidential. If you are not the intended recipient you are hereby notified 
> that using, copying, distributing or taking any action in reliance on the 
> contents of this information is strictly prohibited. If you have received 
> this email in error please notify the sender and delete this email.
> 
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme

NVMe and SR-IOV, how to use together?

[Bogdan Rudas]

Sadly no, I've got the drive for compatibility test from local
hardware supplier. Samsung support said "We are very sorry. We could
support only our direct customer." so I'm going to return the drive to
supplier.

On Tue, Sep 19, 2017 at 3:49 PM, Gruher, Joseph R
<joseph.r.gruher@intel.com> wrote:
> It would be interesting to confirm the device really supports SR-IOV. At launch the device claimed it on paper but we were told firmware support had not been implemented. Do you have any support from the manufacturer to confirm the device HW/FW really supports SR-IOV today?
>
>
>> On Sep 19, 2017,@1:31 PM, Bogdan Rudas <brudas@exadel.com> wrote:
>>
>> Hi!
>>
>> I've checked the device on 4.14.0 rc1 kernel. And see no SR-IOV files
>> for SSD PCI device in /sys/devices/..../sriov_*, SR-IOV works for
>> Intel I350 LOM at the same host.
>> However manufactures brochure claims the SR-IOV support for this
>> series. Should I report this as a bug in Linux nvme driver? Is so,
>> what the information should I gather?
>>
>> Here is a brochure with SR-IOV mention:
>> http://www.samsung.com/semiconductor/global/file/insight/2016/08/Samsung_PM1725a-1.pdf
>>
>> Here is SSD information:
>> root at ubuntu-test:~# nvme list
>> Node             SN                   Model
>>        Namespace Usage                      Format           FW Rev
>> ---------------- --------------------
>> ---------------------------------------- ---------
>> -------------------------- ---------------- --------
>> /dev/nvme0n1     S3HBNA0J600073       SAMSUNG MZPLL1T6HEHP-00003
>>        1           1.60  TB /   1.60  TB    512   B +  0 B   GPNA4B3Q
>>
>>> On Mon, Sep 18, 2017@9:41 PM, Keith Busch <keith.busch@intel.com> wrote:
>>>> On Mon, Sep 18, 2017@09:20:16PM +0300, Bogdan Rudas wrote:
>>>> I run kernel 4.4.27 which don't have a support for SR-IOV in NVMe
>>>> driver, that's why I do not see a VFs in lspci output.
>>>
>>> The driver has nothing to do with what lspci will output. If you don't
>>> see the SR-IOV capability structure there, that means your drive doesn't
>>> support it.
>>
>>
>>
>> --
>> Bogdan Rudas
>> Director of IT Europe
>> Exadel Inc.
>> http://www.exadel.com/
>> E-mail: brudas at exadel.com
>> Skype ID: bogdan.rudas
>>
>> --
>>
>>
>> CONFIDENTIALITY NOTICE: This email and files attached to it are
>> confidential. If you are not the intended recipient you are hereby notified
>> that using, copying, distributing or taking any action in reliance on the
>> contents of this information is strictly prohibited. If you have received
>> this email in error please notify the sender and delete this email.
>>
>> _______________________________________________
>> Linux-nvme mailing list
>> Linux-nvme at lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-nvme



-- 
Bogdan Rudas
Director of IT Europe
Exadel Inc.
http://www.exadel.com/
E-mail: brudas at exadel.com
Skype ID: bogdan.rudas

-- 


CONFIDENTIALITY NOTICE: This email and files attached to it are 
confidential. If you are not the intended recipient you are hereby notified 
that using, copying, distributing or taking any action in reliance on the 
contents of this information is strictly prohibited. If you have received 
this email in error please notify the sender and delete this email.