* Re: RFC: control virtqueue size by the vdpa tool [not found] <DM8PR12MB5400FEB0322A9FD6B3271D45AB799@DM8PR12MB5400.namprd12.prod.outlook.com> @ 2022-08-30 8:21 ` Michael S. Tsirkin 2022-08-30 19:58 ` Michael S. Tsirkin 1 sibling, 0 replies; 6+ messages in thread From: Michael S. Tsirkin @ 2022-08-30 8:21 UTC (permalink / raw) To: Eli Cohen; +Cc: eperezma@redhat.com, virtualization@lists.linux-foundation.org On Tue, Aug 30, 2022 at 06:22:31AM +0000, Eli Cohen wrote: > > > Hi, > > > > I have been experimenting with different queue sizes with mlx5_vdpa and noticed > that queue size can affect performance. Absolutely. Can you share the results btw? Would be very interesting. > I would like to propose an extension to vdpa tool to allow to specify the queue > size. Valid values will conform to the max of 32768 specified by the spec. > > > > “vdpa mgmtdev show” will have another line specifying the valid range for a > management device which could be narrower than the spec allows. This range will > be valid for data queues only (not for control VQ). > > Another line will display the default queue size > > > > Example: > > $ vdpa mgmtdev show > > auxiliary/mlx5_core.sf.6: > > supported_classes net > > max_supported_vqs 65 > > dev_features CSUM GUEST_CSUM MTU HOST_TSO4 HOST_TSO6 STATUS CTRL_VQ CTRL_VLAN > MQ CTRL_MAC_ADDR VERSION_1 ACCESS_PLATFORM > > data queue range 256-4096 > > default queue size 256 > > > > When you create the vdpa device you can specify the requested value: > > $ vdpa dev add name vdpa-a mgmtdev auxiliary/mlx5_core.sf.6 max_vqp 1 mtu 9000 > queue_size 1024 > > Makes sense to me, however, note that 1. the value controlled from the host is actually the max queue size not the queue size. queue size can be controlled from guest with the recent reset extension 2. different sizes for rx and tx probably make sense -- MST _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RFC: control virtqueue size by the vdpa tool [not found] <DM8PR12MB5400FEB0322A9FD6B3271D45AB799@DM8PR12MB5400.namprd12.prod.outlook.com> 2022-08-30 8:21 ` RFC: control virtqueue size by the vdpa tool Michael S. Tsirkin @ 2022-08-30 19:58 ` Michael S. Tsirkin 2022-08-30 21:04 ` Si-Wei Liu 1 sibling, 1 reply; 6+ messages in thread From: Michael S. Tsirkin @ 2022-08-30 19:58 UTC (permalink / raw) To: Eli Cohen; +Cc: eperezma@redhat.com, virtualization@lists.linux-foundation.org On Tue, Aug 30, 2022 at 06:22:31AM +0000, Eli Cohen wrote: > > > Hi, > > > > I have been experimenting with different queue sizes with mlx5_vdpa and noticed > that queue size can affect performance. > > I would like to propose an extension to vdpa tool to allow to specify the queue > size. Valid values will conform to the max of 32768 specified by the spec. > > > > “vdpa mgmtdev show” will have another line specifying the valid range for a > management device which could be narrower than the spec allows. This range will > be valid for data queues only (not for control VQ). > > Another line will display the default queue size > > > > Example: > > $ vdpa mgmtdev show > > auxiliary/mlx5_core.sf.6: > > supported_classes net > > max_supported_vqs 65 > > dev_features CSUM GUEST_CSUM MTU HOST_TSO4 HOST_TSO6 STATUS CTRL_VQ CTRL_VLAN > MQ CTRL_MAC_ADDR VERSION_1 ACCESS_PLATFORM > > data queue range 256-4096 > > default queue size 256 > > > > When you create the vdpa device you can specify the requested value: > > $ vdpa dev add name vdpa-a mgmtdev auxiliary/mlx5_core.sf.6 max_vqp 1 mtu 9000 > queue_size 1024 > > A follow up question: isn't it enough to control the size from qemu? do we need ability to control it at the kernel level? -- MST _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RFC: control virtqueue size by the vdpa tool 2022-08-30 19:58 ` Michael S. Tsirkin @ 2022-08-30 21:04 ` Si-Wei Liu 2022-08-30 22:01 ` Michael S. Tsirkin 0 siblings, 1 reply; 6+ messages in thread From: Si-Wei Liu @ 2022-08-30 21:04 UTC (permalink / raw) To: Michael S. Tsirkin, Eli Cohen Cc: eperezma@redhat.com, virtualization@lists.linux-foundation.org On 8/30/2022 12:58 PM, Michael S. Tsirkin wrote: > On Tue, Aug 30, 2022 at 06:22:31AM +0000, Eli Cohen wrote: >> >> >> Hi, >> >> >> >> I have been experimenting with different queue sizes with mlx5_vdpa and noticed >> that queue size can affect performance. >> >> I would like to propose an extension to vdpa tool to allow to specify the queue >> size. Valid values will conform to the max of 32768 specified by the spec. >> >> >> >> “vdpa mgmtdev show” will have another line specifying the valid range for a >> management device which could be narrower than the spec allows. This range will >> be valid for data queues only (not for control VQ). >> >> Another line will display the default queue size >> >> >> >> Example: >> >> $ vdpa mgmtdev show >> >> auxiliary/mlx5_core.sf.6: >> >> supported_classes net >> >> max_supported_vqs 65 >> >> dev_features CSUM GUEST_CSUM MTU HOST_TSO4 HOST_TSO6 STATUS CTRL_VQ CTRL_VLAN >> MQ CTRL_MAC_ADDR VERSION_1 ACCESS_PLATFORM >> >> data queue range 256-4096 >> >> default queue size 256 >> >> >> >> When you create the vdpa device you can specify the requested value: >> >> $ vdpa dev add name vdpa-a mgmtdev auxiliary/mlx5_core.sf.6 max_vqp 1 mtu 9000 >> queue_size 1024 >> >> > > A follow up question: isn't it enough to control the size > from qemu? do we need ability to control it at the kernel level? > Right, I think today we can optionally control the queue size from qemu via rx_queue_size or tx_queue_size, but it has a limit of 1024 (btw why it has such limit, which is relatively lower in my opinion). I think what was missing for QEMU is to query the max number of queue size that the hardware can support from the backend. -Siwei _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RFC: control virtqueue size by the vdpa tool 2022-08-30 21:04 ` Si-Wei Liu @ 2022-08-30 22:01 ` Michael S. Tsirkin 2022-08-30 23:22 ` Si-Wei Liu 0 siblings, 1 reply; 6+ messages in thread From: Michael S. Tsirkin @ 2022-08-30 22:01 UTC (permalink / raw) To: Si-Wei Liu Cc: eperezma@redhat.com, Eli Cohen, virtualization@lists.linux-foundation.org On Tue, Aug 30, 2022 at 02:04:55PM -0700, Si-Wei Liu wrote: > > > On 8/30/2022 12:58 PM, Michael S. Tsirkin wrote: > > On Tue, Aug 30, 2022 at 06:22:31AM +0000, Eli Cohen wrote: > > > > > > Hi, > > > > > > > > > I have been experimenting with different queue sizes with mlx5_vdpa and noticed > > > that queue size can affect performance. > > > > > > I would like to propose an extension to vdpa tool to allow to specify the queue > > > size. Valid values will conform to the max of 32768 specified by the spec. > > > > > > > > > “vdpa mgmtdev show” will have another line specifying the valid range for a > > > management device which could be narrower than the spec allows. This range will > > > be valid for data queues only (not for control VQ). > > > > > > Another line will display the default queue size > > > > > > > > > Example: > > > > > > $ vdpa mgmtdev show > > > > > > auxiliary/mlx5_core.sf.6: > > > > > > supported_classes net > > > > > > max_supported_vqs 65 > > > > > > dev_features CSUM GUEST_CSUM MTU HOST_TSO4 HOST_TSO6 STATUS CTRL_VQ CTRL_VLAN > > > MQ CTRL_MAC_ADDR VERSION_1 ACCESS_PLATFORM > > > > > > data queue range 256-4096 > > > > > > default queue size 256 > > > > > > > > > When you create the vdpa device you can specify the requested value: > > > > > > $ vdpa dev add name vdpa-a mgmtdev auxiliary/mlx5_core.sf.6 max_vqp 1 mtu 9000 > > > queue_size 1024 > > > > > > > A follow up question: isn't it enough to control the size > > from qemu? do we need ability to control it at the kernel level? > > > Right, I think today we can optionally control the queue size from qemu via > rx_queue_size or tx_queue_size, but it has a limit of 1024 (btw why it has > such limit, which is relatively lower in my opinion). I think what was > missing for QEMU is to query the max number of queue size that the hardware > can support from the backend. > > -Siwei > okay sure. my question is how important is it to control it in the kernel? -- MST _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RFC: control virtqueue size by the vdpa tool 2022-08-30 22:01 ` Michael S. Tsirkin @ 2022-08-30 23:22 ` Si-Wei Liu [not found] ` <DM8PR12MB54006D58A2ACA88CC786B9A8AB789@DM8PR12MB5400.namprd12.prod.outlook.com> 0 siblings, 1 reply; 6+ messages in thread From: Si-Wei Liu @ 2022-08-30 23:22 UTC (permalink / raw) To: Michael S. Tsirkin Cc: eperezma@redhat.com, Eli Cohen, virtualization@lists.linux-foundation.org On 8/30/2022 3:01 PM, Michael S. Tsirkin wrote: > On Tue, Aug 30, 2022 at 02:04:55PM -0700, Si-Wei Liu wrote: >> >> On 8/30/2022 12:58 PM, Michael S. Tsirkin wrote: >>> On Tue, Aug 30, 2022 at 06:22:31AM +0000, Eli Cohen wrote: >>>> Hi, >>>> >>>> >>>> I have been experimenting with different queue sizes with mlx5_vdpa and noticed >>>> that queue size can affect performance. >>>> >>>> I would like to propose an extension to vdpa tool to allow to specify the queue >>>> size. Valid values will conform to the max of 32768 specified by the spec. >>>> >>>> >>>> “vdpa mgmtdev show” will have another line specifying the valid range for a >>>> management device which could be narrower than the spec allows. This range will >>>> be valid for data queues only (not for control VQ). >>>> >>>> Another line will display the default queue size >>>> >>>> >>>> Example: >>>> >>>> $ vdpa mgmtdev show >>>> >>>> auxiliary/mlx5_core.sf.6: >>>> >>>> supported_classes net >>>> >>>> max_supported_vqs 65 >>>> >>>> dev_features CSUM GUEST_CSUM MTU HOST_TSO4 HOST_TSO6 STATUS CTRL_VQ CTRL_VLAN >>>> MQ CTRL_MAC_ADDR VERSION_1 ACCESS_PLATFORM >>>> >>>> data queue range 256-4096 >>>> >>>> default queue size 256 >>>> >>>> >>>> When you create the vdpa device you can specify the requested value: >>>> >>>> $ vdpa dev add name vdpa-a mgmtdev auxiliary/mlx5_core.sf.6 max_vqp 1 mtu 9000 >>>> queue_size 1024 >>>> >>> A follow up question: isn't it enough to control the size >>> from qemu? do we need ability to control it at the kernel level? >>> >> Right, I think today we can optionally control the queue size from qemu via >> rx_queue_size or tx_queue_size, but it has a limit of 1024 (btw why it has >> such limit, which is relatively lower in my opinion). I think what was >> missing for QEMU is to query the max number of queue size that the hardware >> can support from the backend. >> >> -Siwei >> > okay sure. my question is how important is it to control it in the > kernel? I don't have a specific use case for that (in kernel) -Siwei _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <DM8PR12MB54006D58A2ACA88CC786B9A8AB789@DM8PR12MB5400.namprd12.prod.outlook.com>]
* Re: RFC: control virtqueue size by the vdpa tool [not found] ` <DM8PR12MB54006D58A2ACA88CC786B9A8AB789@DM8PR12MB5400.namprd12.prod.outlook.com> @ 2022-08-31 22:22 ` Si-Wei Liu 0 siblings, 0 replies; 6+ messages in thread From: Si-Wei Liu @ 2022-08-31 22:22 UTC (permalink / raw) To: Eli Cohen, Michael S. Tsirkin Cc: eperezma@redhat.com, virtualization@lists.linux-foundation.org On 8/30/2022 10:22 PM, Eli Cohen wrote: >> From: Si-Wei Liu <si-wei.liu@oracle.com> >> Sent: Wednesday, August 31, 2022 2:23 AM >> To: Michael S. Tsirkin <mst@redhat.com> >> Cc: Eli Cohen <elic@nvidia.com>; virtualization@lists.linux-foundation.org; >> Jason Wang <jasowang@redhat.com>; eperezma@redhat.com >> Subject: Re: RFC: control virtqueue size by the vdpa tool >> >> >> >> On 8/30/2022 3:01 PM, Michael S. Tsirkin wrote: >>> On Tue, Aug 30, 2022 at 02:04:55PM -0700, Si-Wei Liu wrote: >>>> On 8/30/2022 12:58 PM, Michael S. Tsirkin wrote: >>>>> On Tue, Aug 30, 2022 at 06:22:31AM +0000, Eli Cohen wrote: >>>>>> Hi, >>>>>> >>>>>> >>>>>> I have been experimenting with different queue sizes with mlx5_vdpa >> and noticed >>>>>> that queue size can affect performance. >>>>>> >>>>>> I would like to propose an extension to vdpa tool to allow to specify >> the queue >>>>>> size. Valid values will conform to the max of 32768 specified by the >> spec. >>>>>> >>>>>> “vdpa mgmtdev show” will have another line specifying the valid range >> for a >>>>>> management device which could be narrower than the spec allows. >> This range will >>>>>> be valid for data queues only (not for control VQ). >>>>>> >>>>>> Another line will display the default queue size >>>>>> >>>>>> >>>>>> Example: >>>>>> >>>>>> $ vdpa mgmtdev show >>>>>> >>>>>> auxiliary/mlx5_core.sf.6: >>>>>> >>>>>> supported_classes net >>>>>> >>>>>> max_supported_vqs 65 >>>>>> >>>>>> dev_features CSUM GUEST_CSUM MTU HOST_TSO4 HOST_TSO6 >> STATUS CTRL_VQ CTRL_VLAN >>>>>> MQ CTRL_MAC_ADDR VERSION_1 ACCESS_PLATFORM >>>>>> >>>>>> data queue range 256-4096 >>>>>> >>>>>> default queue size 256 >>>>>> >>>>>> >>>>>> When you create the vdpa device you can specify the requested value: >>>>>> >>>>>> $ vdpa dev add name vdpa-a mgmtdev auxiliary/mlx5_core.sf.6 >> max_vqp 1 mtu 9000 >>>>>> queue_size 1024 >>>>>> >>>>> A follow up question: isn't it enough to control the size >>>>> from qemu? do we need ability to control it at the kernel level? >>>>> >>>> Right, I think today we can optionally control the queue size from qemu >> via >>>> rx_queue_size or tx_queue_size, but it has a limit of 1024 (btw why it has >>>> such limit, which is relatively lower in my opinion). I think what was >>>> missing for QEMU is to query the max number of queue size that the >> hardware >>>> can support from the backend. > I agree that ethtool is the way to go. > BTW, Si-Wei, can you point to the code that limits the configuration to 1024? It's in QEMU's virtio_net_device_realize(): virtio_net_set_config_size(n, n->host_features); virtio_init(vdev, "virtio-net", VIRTIO_ID_NET, n->config_size); /* * We set a lower limit on RX queue size to what it always was. * Guests that want a smaller ring can always resize it without * help from us (using virtio 1 and up). */ if (n->net_conf.rx_queue_size < VIRTIO_NET_RX_QUEUE_MIN_SIZE || n->net_conf.rx_queue_size > VIRTQUEUE_MAX_SIZE || !is_power_of_2(n->net_conf.rx_queue_size)) { error_setg(errp, "Invalid rx_queue_size (= %" PRIu16 "), " "must be a power of 2 between %d and %d.", n->net_conf.rx_queue_size, VIRTIO_NET_RX_QUEUE_MIN_SIZE, VIRTQUEUE_MAX_SIZE); -Siwei > And if ethtool does not provide a way to show the max we can add this support in the future. > >>>> -Siwei >>>> >>> okay sure. my question is how important is it to control it in the >>> kernel? >> I don't have a specific use case for that (in kernel) >> >> -Siwei _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-08-31 22:23 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <DM8PR12MB5400FEB0322A9FD6B3271D45AB799@DM8PR12MB5400.namprd12.prod.outlook.com>
2022-08-30 8:21 ` RFC: control virtqueue size by the vdpa tool Michael S. Tsirkin
2022-08-30 19:58 ` Michael S. Tsirkin
2022-08-30 21:04 ` Si-Wei Liu
2022-08-30 22:01 ` Michael S. Tsirkin
2022-08-30 23:22 ` Si-Wei Liu
[not found] ` <DM8PR12MB54006D58A2ACA88CC786B9A8AB789@DM8PR12MB5400.namprd12.prod.outlook.com>
2022-08-31 22:22 ` Si-Wei Liu
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).