VFIO/vfio-user: specify NVMe namespace to boot from

VFIO/vfio-user: specify NVMe namespace to boot from

[Thanos Makatos]

We're working on implementing a virtual NVMe controller based on SPDK and a multiprocess-qemu branch that uses the vfio-user. We're facing a problem where the existing API doesn't allow us to tell QEMU from which NVMe namespace we'd like SeaBIOS to boot from.

How can we solve this problem? Can we add a parameter to the '-boot' option, e.g. '-boot path=/devices/pciblah/...@namespace0'? AFAIK VFIO should have the same problem. 

The corresponding SeaBIOS patch can be found in https://mail.coreboot.org/hyperkitty/list/seabios@seabios.org/thread/2Q7NPH7TJNHK6JGPHQL7755HILO23ISN/

Re: VFIO/vfio-user: specify NVMe namespace to boot from

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1064 bytes --]

On Mon, Jul 12, 2021 at 01:24:07PM +0000, Thanos Makatos wrote:
> We're working on implementing a virtual NVMe controller based on SPDK and a multiprocess-qemu branch that uses the vfio-user. We're facing a problem where the existing API doesn't allow us to tell QEMU from which NVMe namespace we'd like SeaBIOS to boot from.
> 
> How can we solve this problem? Can we add a parameter to the '-boot' option, e.g. '-boot path=/devices/pciblah/...@namespace0'? AFAIK VFIO should have the same problem. 
> 
> The corresponding SeaBIOS patch can be found in https://mail.coreboot.org/hyperkitty/list/seabios@seabios.org/thread/2Q7NPH7TJNHK6JGPHQL7755HILO23ISN/

Hi,
I have CCed Gerd Hoffmann on a hunch that he may have ideas.

Yes, I think the path needs to include the Namespace ID similar to how
SCSI boot paths include the target/channel/LUN.

When the SeaBIOS NVMe driver probes the controller is should discover
the available Namespaces and set up individual drives for each
Namespace. That would be analogous to virtio_scsi_add_lun().

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

RE: VFIO/vfio-user: specify NVMe namespace to boot from

[Thanos Makatos]

> On Mon, Jul 12, 2021 at 01:24:07PM +0000, Thanos Makatos wrote:
> > We're working on implementing a virtual NVMe controller based on SPDK
> and a multiprocess-qemu branch that uses the vfio-user. We're facing a
> problem where the existing API doesn't allow us to tell QEMU from which
> NVMe namespace we'd like SeaBIOS to boot from.
> >
> > How can we solve this problem? Can we add a parameter to the '-boot'
> option, e.g. '-boot path=/devices/pciblah/...@namespace0'? AFAIK VFIO
> should have the same problem.
> >
> > The corresponding SeaBIOS patch can be found in
> https://mail.coreboot.org/hyperkitty/list/seabios@seabios.org/thread/2Q7
> NPH7TJNHK6JGPHQL7755HILO23ISN/
> 
> Hi,
> I have CCed Gerd Hoffmann on a hunch that he may have ideas.
> 
> Yes, I think the path needs to include the Namespace ID similar to how
> SCSI boot paths include the target/channel/LUN.

Just to make sure I've explained the problem correctly, what we need to fix is not specific to NVMe: we want to be able to boot from any PCI device, so we'd need something like '-boot /devices/pciblah'. Specifically, for NVMe we might want to specify the namespace (as we can't necessarily assume that the first one must be used), therefore we'd have to add the '/namespace0' bit. For other kinds of PCI devices, we'd have to append something different.

Would such a change be acceptable?

> 
> When the SeaBIOS NVMe driver probes the controller is should discover
> the available Namespaces and set up individual drives for each
> Namespace. That would be analogous to virtio_scsi_add_lun().

Setting up an individual drive can easily exhaust SeaBIOS's limited memory, so it's better to either probe the specified namespace or probe the first active one. I've sent a patch here: https://mail.coreboot.org/hyperkitty/list/seabios@seabios.org/message/TKKUCH5PYZPXZG3NA2F5SALUFUVUYDZG/

Re: VFIO/vfio-user: specify NVMe namespace to boot from

[Philippe Mathieu-Daudé]

On 7/20/21 9:10 PM, Thanos Makatos wrote:
>> On Mon, Jul 12, 2021 at 01:24:07PM +0000, Thanos Makatos wrote:
>>> We're working on implementing a virtual NVMe controller based on SPDK
>> and a multiprocess-qemu branch that uses the vfio-user. We're facing a
>> problem where the existing API doesn't allow us to tell QEMU from which
>> NVMe namespace we'd like SeaBIOS to boot from.
>>>
>>> How can we solve this problem? Can we add a parameter to the '-boot'
>> option, e.g. '-boot path=/devices/pciblah/...@namespace0'? AFAIK VFIO
>> should have the same problem.
>>>
>>> The corresponding SeaBIOS patch can be found in
>> https://mail.coreboot.org/hyperkitty/list/seabios@seabios.org/thread/2Q7
>> NPH7TJNHK6JGPHQL7755HILO23ISN/
>>
>> Hi,
>> I have CCed Gerd Hoffmann on a hunch that he may have ideas.
>>
>> Yes, I think the path needs to include the Namespace ID similar to how
>> SCSI boot paths include the target/channel/LUN.
> 
> Just to make sure I've explained the problem correctly, what we need to fix is not specific to NVMe: we want to be able to boot from any PCI device, so we'd need something like '-boot /devices/pciblah'. Specifically, for NVMe we might want to specify the namespace (as we can't necessarily assume that the first one must be used), therefore we'd have to add the '/namespace0' bit. For other kinds of PCI devices, we'd have to append something different.
> 
> Would such a change be acceptable?

FWIW -boot path= sounds a good way to solve this problem,
as long as you succeed at not breaking the legacy bootdevice.c :)

> 
>>
>> When the SeaBIOS NVMe driver probes the controller is should discover
>> the available Namespaces and set up individual drives for each
>> Namespace. That would be analogous to virtio_scsi_add_lun().
> 
> Setting up an individual drive can easily exhaust SeaBIOS's limited memory, so it's better to either probe the specified namespace or probe the first active one. I've sent a patch here: https://mail.coreboot.org/hyperkitty/list/seabios@seabios.org/message/TKKUCH5PYZPXZG3NA2F5SALUFUVUYDZG/
> 
>