Re: QEMU Userspace NVMe driver and multiple iothreads

[Farhan Ali <alifm@linux.ibm.com>]

On 3/20/2025 12:17 PM, Stefan Hajnoczi wrote:
> On Thu, Mar 20, 2025 at 11:13:04AM -0700, Farhan Ali wrote:
>> Hi,
>>
>> I have been experimenting with the QEMU Userspace NVMe driver on s390x
>> architecture. I have noticed an issue when assigning multiple virtqueues and
>> multiple iothreads to the block device. The driver works well with a single
>> iothread, but when using more than one iothread we can hit a problem where 2
>> iothreads can update the completion queue head doorbell register with the
>> same value within microseconds of each other. As far as I understand this
>> would be an invalid doorbell write as per NVMe spec (for eg spec version 1.4
>> section 5.2.1 defines this as an invalid write). This causes the NVMe device
>> not to post any further completions. As far i understand this doesn't seem
>> to be specific to s390x architecture.
>>
>> I would appreciate some guidance on this to see if there is some known
>> limitations with the userspace NVMe driver and multi queue/multi iothread?
> Yes, the block driver only supports 1 IOThread. The code uses
> bdrv_get_aio_context(bs) rather than qemu_get_current_aio_context(), so
> it will only operate in one AioContext.
>
> Code changes would be necessary to support multiple IOThreads.
>
> By the way, this block driver is experimental. There are several bugs
> filed against it in bugzilla.redhat.com. It's currently only useful to
> QEMU developers and I wouldn't rely on it for production VMs.
>
> Stefan

Hi Stefan,

Thanks for your response. This clarifies my suspicions, and provides 
more context on the stability of the driver.

Thanks

Farhan


>> This is an example xml snippet i used to define the nvme block device
>>
>> ...
>>
>> <disk type='nvme' device='disk'>
>>        <driver name='qemu' type='raw' queues='8' packed='on'>
>>              <iothreads>
>>                    <iothread id='1'/>
>>              </iothreads>
>>        </driver>
>>        <source type='pci' managed='yes' namespace='1'>
>>              <address domain='0x0004' bus='0x00' slot='0x00' function='0x0'/>
>>        </source>
>>        <target dev='vde' bus='virtio'/>
>>        <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0002'/>
>>
>> </disk>
>> ....
>>
>> Appreciate any help on this!
>>
>> Thanks
>> Farhan
>>