QEMU Userspace NVMe driver and multiple iothreads

QEMU Userspace NVMe driver and multiple iothreads

[Farhan Ali]

Hi,

I have been experimenting with the QEMU Userspace NVMe driver on s390x 
architecture. I have noticed an issue when assigning multiple virtqueues 
and multiple iothreads to the block device. The driver works well with a 
single iothread, but when using more than one iothread we can hit a 
problem where 2 iothreads can update the completion queue head doorbell 
register with the same value within microseconds of each other. As far 
as I understand this would be an invalid doorbell write as per NVMe spec 
(for eg spec version 1.4 section 5.2.1 defines this as an invalid 
write). This causes the NVMe device not to post any further completions. 
As far i understand this doesn't seem to be specific to s390x architecture.

I would appreciate some guidance on this to see if there is some known 
limitations with the userspace NVMe driver and multi queue/multi 
iothread? This is an example xml snippet i used to define the nvme block 
device

...

<disk type='nvme' device='disk'>
       <driver name='qemu' type='raw' queues='8' packed='on'>
             <iothreads>
                   <iothread id='1'/>
             </iothreads>
       </driver>
       <source type='pci' managed='yes' namespace='1'>
             <address domain='0x0004' bus='0x00' slot='0x00' 
function='0x0'/>
       </source>
       <target dev='vde' bus='virtio'/>
       <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0002'/>

</disk>
....

Appreciate any help on this!

Thanks
Farhan

Re: QEMU Userspace NVMe driver and multiple iothreads

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2096 bytes --]

On Thu, Mar 20, 2025 at 11:13:04AM -0700, Farhan Ali wrote:
> Hi,
> 
> I have been experimenting with the QEMU Userspace NVMe driver on s390x
> architecture. I have noticed an issue when assigning multiple virtqueues and
> multiple iothreads to the block device. The driver works well with a single
> iothread, but when using more than one iothread we can hit a problem where 2
> iothreads can update the completion queue head doorbell register with the
> same value within microseconds of each other. As far as I understand this
> would be an invalid doorbell write as per NVMe spec (for eg spec version 1.4
> section 5.2.1 defines this as an invalid write). This causes the NVMe device
> not to post any further completions. As far i understand this doesn't seem
> to be specific to s390x architecture.
> 
> I would appreciate some guidance on this to see if there is some known
> limitations with the userspace NVMe driver and multi queue/multi iothread?

Yes, the block driver only supports 1 IOThread. The code uses
bdrv_get_aio_context(bs) rather than qemu_get_current_aio_context(), so
it will only operate in one AioContext.

Code changes would be necessary to support multiple IOThreads.

By the way, this block driver is experimental. There are several bugs
filed against it in bugzilla.redhat.com. It's currently only useful to
QEMU developers and I wouldn't rely on it for production VMs.

Stefan

> This is an example xml snippet i used to define the nvme block device
> 
> ...
> 
> <disk type='nvme' device='disk'>
>       <driver name='qemu' type='raw' queues='8' packed='on'>
>             <iothreads>
>                   <iothread id='1'/>
>             </iothreads>
>       </driver>
>       <source type='pci' managed='yes' namespace='1'>
>             <address domain='0x0004' bus='0x00' slot='0x00' function='0x0'/>
>       </source>
>       <target dev='vde' bus='virtio'/>
>       <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0002'/>
> 
> </disk>
> ....
> 
> Appreciate any help on this!
> 
> Thanks
> Farhan
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: QEMU Userspace NVMe driver and multiple iothreads

[Farhan Ali]

On 3/20/2025 12:17 PM, Stefan Hajnoczi wrote:
> On Thu, Mar 20, 2025 at 11:13:04AM -0700, Farhan Ali wrote:
>> Hi,
>>
>> I have been experimenting with the QEMU Userspace NVMe driver on s390x
>> architecture. I have noticed an issue when assigning multiple virtqueues and
>> multiple iothreads to the block device. The driver works well with a single
>> iothread, but when using more than one iothread we can hit a problem where 2
>> iothreads can update the completion queue head doorbell register with the
>> same value within microseconds of each other. As far as I understand this
>> would be an invalid doorbell write as per NVMe spec (for eg spec version 1.4
>> section 5.2.1 defines this as an invalid write). This causes the NVMe device
>> not to post any further completions. As far i understand this doesn't seem
>> to be specific to s390x architecture.
>>
>> I would appreciate some guidance on this to see if there is some known
>> limitations with the userspace NVMe driver and multi queue/multi iothread?
> Yes, the block driver only supports 1 IOThread. The code uses
> bdrv_get_aio_context(bs) rather than qemu_get_current_aio_context(), so
> it will only operate in one AioContext.
>
> Code changes would be necessary to support multiple IOThreads.
>
> By the way, this block driver is experimental. There are several bugs
> filed against it in bugzilla.redhat.com. It's currently only useful to
> QEMU developers and I wouldn't rely on it for production VMs.
>
> Stefan

Hi Stefan,

Thanks for your response. This clarifies my suspicions, and provides 
more context on the stability of the driver.

Thanks

Farhan


>> This is an example xml snippet i used to define the nvme block device
>>
>> ...
>>
>> <disk type='nvme' device='disk'>
>>        <driver name='qemu' type='raw' queues='8' packed='on'>
>>              <iothreads>
>>                    <iothread id='1'/>
>>              </iothreads>
>>        </driver>
>>        <source type='pci' managed='yes' namespace='1'>
>>              <address domain='0x0004' bus='0x00' slot='0x00' function='0x0'/>
>>        </source>
>>        <target dev='vde' bus='virtio'/>
>>        <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0002'/>
>>
>> </disk>
>> ....
>>
>> Appreciate any help on this!
>>
>> Thanks
>> Farhan
>>