All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Michael S. Tsirkin" <mst@redhat.com>
To: Parav Pandit <parav@nvidia.com>
Cc: Gavin Li <gavinl@nvidia.com>,
	"stephen@networkplumber.org" <stephen@networkplumber.org>,
	"davem@davemloft.net" <davem@davemloft.net>,
	"jesse.brandeburg@intel.com" <jesse.brandeburg@intel.com>,
	"kuba@kernel.org" <kuba@kernel.org>,
	"sridhar.samudrala@intel.com" <sridhar.samudrala@intel.com>,
	"jasowang@redhat.com" <jasowang@redhat.com>,
	"loseweigh@gmail.com" <loseweigh@gmail.com>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"virtualization@lists.linux-foundation.org"
	<virtualization@lists.linux-foundation.org>,
	"virtio-dev@lists.oasis-open.org"
	<virtio-dev@lists.oasis-open.org>, Gavi Teitz <gavi@nvidia.com>,
	Xuan Zhuo <xuanzhuo@linux.alibaba.com>,
	Si-Wei Liu <si-wei.liu@oracle.com>
Subject: [virtio-dev] Re: [PATCH v5 2/2] virtio-net: use mtu size as buffer length for big packets
Date: Wed, 7 Sep 2022 15:23:57 -0400	[thread overview]
Message-ID: <20220907152156-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <PH0PR12MB54811F1234CB7822F47DD1B9DC419@PH0PR12MB5481.namprd12.prod.outlook.com>

On Wed, Sep 07, 2022 at 07:18:06PM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, September 7, 2022 3:12 PM
> 
> > > Because of shallow queue of 16 entries deep.
> > 
> > but why is the queue just 16 entries?
> I explained the calculation in [1] about 16 entries.
> 
> [1] https://lore.kernel.org/netdev/PH0PR12MB54812EC7F4711C1EA4CAA119DC419@PH0PR12MB5481.namprd12.prod.outlook.com/
> 
> > does the device not support indirect?
> >
> Yes, indirect feature bit is disabled on the device.

OK that explains it.

> > because with indirect you get 256 entries, with 16 s/g each.
> > 
> Sure. I explained below that indirect comes with 7x memory cost that is not desired.
> (Ignored the table memory allocation cost and extra latency).

Oh sure, it's a waste. I wonder what effect does the patch have
on bandwidth with indirect enabled though.


> Hence don't want to enable indirect in this scenario.
> This optimization also works with indirect with smaller indirect table.
> 
> > 
> > > With driver turn around time to repost buffers, device is idle without any
> > RQ buffers.
> > > With this improvement, device has 85 buffers instead of 16 to receive
> > packets.
> > >
> > > Enabling indirect in device can help at cost of 7x higher memory per VQ in
> > the guest VM.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org


WARNING: multiple messages have this Message-ID (diff)
From: "Michael S. Tsirkin" <mst@redhat.com>
To: Parav Pandit <parav@nvidia.com>
Cc: "virtio-dev@lists.oasis-open.org"
	<virtio-dev@lists.oasis-open.org>,
	"sridhar.samudrala@intel.com" <sridhar.samudrala@intel.com>,
	"jesse.brandeburg@intel.com" <jesse.brandeburg@intel.com>,
	Gavi Teitz <gavi@nvidia.com>,
	"virtualization@lists.linux-foundation.org"
	<virtualization@lists.linux-foundation.org>,
	"stephen@networkplumber.org" <stephen@networkplumber.org>,
	"loseweigh@gmail.com" <loseweigh@gmail.com>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"kuba@kernel.org" <kuba@kernel.org>,
	"davem@davemloft.net" <davem@davemloft.net>,
	Gavin Li <gavinl@nvidia.com>
Subject: Re: [PATCH v5 2/2] virtio-net: use mtu size as buffer length for big packets
Date: Wed, 7 Sep 2022 15:23:57 -0400	[thread overview]
Message-ID: <20220907152156-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <PH0PR12MB54811F1234CB7822F47DD1B9DC419@PH0PR12MB5481.namprd12.prod.outlook.com>

On Wed, Sep 07, 2022 at 07:18:06PM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, September 7, 2022 3:12 PM
> 
> > > Because of shallow queue of 16 entries deep.
> > 
> > but why is the queue just 16 entries?
> I explained the calculation in [1] about 16 entries.
> 
> [1] https://lore.kernel.org/netdev/PH0PR12MB54812EC7F4711C1EA4CAA119DC419@PH0PR12MB5481.namprd12.prod.outlook.com/
> 
> > does the device not support indirect?
> >
> Yes, indirect feature bit is disabled on the device.

OK that explains it.

> > because with indirect you get 256 entries, with 16 s/g each.
> > 
> Sure. I explained below that indirect comes with 7x memory cost that is not desired.
> (Ignored the table memory allocation cost and extra latency).

Oh sure, it's a waste. I wonder what effect does the patch have
on bandwidth with indirect enabled though.


> Hence don't want to enable indirect in this scenario.
> This optimization also works with indirect with smaller indirect table.
> 
> > 
> > > With driver turn around time to repost buffers, device is idle without any
> > RQ buffers.
> > > With this improvement, device has 85 buffers instead of 16 to receive
> > packets.
> > >
> > > Enabling indirect in device can help at cost of 7x higher memory per VQ in
> > the guest VM.

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

WARNING: multiple messages have this Message-ID (diff)
From: "Michael S. Tsirkin" <mst@redhat.com>
To: Parav Pandit <parav@nvidia.com>
Cc: Gavin Li <gavinl@nvidia.com>,
	"stephen@networkplumber.org" <stephen@networkplumber.org>,
	"davem@davemloft.net" <davem@davemloft.net>,
	"jesse.brandeburg@intel.com" <jesse.brandeburg@intel.com>,
	"kuba@kernel.org" <kuba@kernel.org>,
	"sridhar.samudrala@intel.com" <sridhar.samudrala@intel.com>,
	"jasowang@redhat.com" <jasowang@redhat.com>,
	"loseweigh@gmail.com" <loseweigh@gmail.com>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"virtualization@lists.linux-foundation.org" 
	<virtualization@lists.linux-foundation.org>,
	"virtio-dev@lists.oasis-open.org"
	<virtio-dev@lists.oasis-open.org>, Gavi Teitz <gavi@nvidia.com>,
	Xuan Zhuo <xuanzhuo@linux.alibaba.com>,
	Si-Wei Liu <si-wei.liu@oracle.com>
Subject: Re: [PATCH v5 2/2] virtio-net: use mtu size as buffer length for big packets
Date: Wed, 7 Sep 2022 15:23:57 -0400	[thread overview]
Message-ID: <20220907152156-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <PH0PR12MB54811F1234CB7822F47DD1B9DC419@PH0PR12MB5481.namprd12.prod.outlook.com>

On Wed, Sep 07, 2022 at 07:18:06PM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, September 7, 2022 3:12 PM
> 
> > > Because of shallow queue of 16 entries deep.
> > 
> > but why is the queue just 16 entries?
> I explained the calculation in [1] about 16 entries.
> 
> [1] https://lore.kernel.org/netdev/PH0PR12MB54812EC7F4711C1EA4CAA119DC419@PH0PR12MB5481.namprd12.prod.outlook.com/
> 
> > does the device not support indirect?
> >
> Yes, indirect feature bit is disabled on the device.

OK that explains it.

> > because with indirect you get 256 entries, with 16 s/g each.
> > 
> Sure. I explained below that indirect comes with 7x memory cost that is not desired.
> (Ignored the table memory allocation cost and extra latency).

Oh sure, it's a waste. I wonder what effect does the patch have
on bandwidth with indirect enabled though.


> Hence don't want to enable indirect in this scenario.
> This optimization also works with indirect with smaller indirect table.
> 
> > 
> > > With driver turn around time to repost buffers, device is idle without any
> > RQ buffers.
> > > With this improvement, device has 85 buffers instead of 16 to receive
> > packets.
> > >
> > > Enabling indirect in device can help at cost of 7x higher memory per VQ in
> > the guest VM.


  reply	other threads:[~2022-09-07 19:24 UTC|newest]

Thread overview: 108+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-09-01  2:10 [virtio-dev] [PATCH v5 0/2] Improve virtio performance for 9k mtu Gavin Li
2022-09-01  2:10 ` Gavin Li
2022-09-01  2:10 ` [virtio-dev] [PATCH v5 1/2] virtio-net: introduce and use helper function for guest gso support checks Gavin Li
2022-09-01  2:10   ` Gavin Li
2022-09-07  2:12   ` [virtio-dev] " Jason Wang
2022-09-01  2:10 ` [virtio-dev] [PATCH v5 2/2] virtio-net: use mtu size as buffer length for big packets Gavin Li
2022-09-01  2:10   ` Gavin Li
2022-09-07  2:17   ` [virtio-dev] " Jason Wang
2022-09-07  2:17     ` Jason Wang
2022-09-07  2:17     ` Jason Wang
2022-09-07  3:15     ` Gavin Li
2022-09-07  3:15       ` Gavin Li
2022-09-07  3:29       ` Gavin Li
2022-09-07  3:29         ` Gavin Li
2022-09-07  5:31   ` [virtio-dev] " Michael S. Tsirkin
2022-09-07  5:31     ` Michael S. Tsirkin
2022-09-07  5:31     ` Michael S. Tsirkin
2022-09-07  8:08     ` [virtio-dev] " Gavin Li
2022-09-07  8:08       ` Gavin Li
2022-09-07  9:26       ` [virtio-dev] " Michael S. Tsirkin
2022-09-07  9:26         ` Michael S. Tsirkin
2022-09-07  9:26         ` Michael S. Tsirkin
2022-09-07 14:08         ` [virtio-dev] " Parav Pandit
2022-09-07 14:08           ` Parav Pandit
2022-09-07 14:08           ` Parav Pandit via Virtualization
2022-09-07 14:29           ` [virtio-dev] " Michael S. Tsirkin
2022-09-07 14:29             ` Michael S. Tsirkin
2022-09-07 14:29             ` Michael S. Tsirkin
2022-09-07 14:33             ` [virtio-dev] " Parav Pandit
2022-09-07 14:33               ` Parav Pandit
2022-09-07 14:33               ` Parav Pandit via Virtualization
2022-09-07 14:40               ` [virtio-dev] " Michael S. Tsirkin
2022-09-07 14:40                 ` Michael S. Tsirkin
2022-09-07 14:40                 ` Michael S. Tsirkin
2022-09-07 16:12                 ` [virtio-dev] " Parav Pandit
2022-09-07 16:12                   ` Parav Pandit
2022-09-07 16:12                   ` Parav Pandit via Virtualization
2022-09-07 18:15                   ` [virtio-dev] " Michael S. Tsirkin
2022-09-07 18:15                     ` Michael S. Tsirkin
2022-09-07 18:15                     ` Michael S. Tsirkin
2022-09-07 19:06                     ` [virtio-dev] " Parav Pandit
2022-09-07 19:06                       ` Parav Pandit
2022-09-07 19:06                       ` Parav Pandit via Virtualization
2022-09-07 19:11                       ` [virtio-dev] " Michael S. Tsirkin
2022-09-07 19:11                         ` Michael S. Tsirkin
2022-09-07 19:11                         ` Michael S. Tsirkin
2022-09-07 19:18                         ` [virtio-dev] " Parav Pandit
2022-09-07 19:18                           ` Parav Pandit
2022-09-07 19:18                           ` Parav Pandit via Virtualization
2022-09-07 19:23                           ` Michael S. Tsirkin [this message]
2022-09-07 19:23                             ` Michael S. Tsirkin
2022-09-07 19:23                             ` Michael S. Tsirkin
2022-09-07 19:27                             ` [virtio-dev] " Parav Pandit
2022-09-07 19:27                               ` Parav Pandit
2022-09-07 19:27                               ` Parav Pandit via Virtualization
2022-09-07 19:36                               ` [virtio-dev] " Michael S. Tsirkin
2022-09-07 19:36                                 ` Michael S. Tsirkin
2022-09-07 19:36                                 ` Michael S. Tsirkin
2022-09-07 19:37                                 ` [virtio-dev] " Michael S. Tsirkin
2022-09-07 19:37                                   ` Michael S. Tsirkin
2022-09-07 19:37                                   ` Michael S. Tsirkin
2022-09-07 19:54                                   ` [virtio-dev] " Parav Pandit
2022-09-07 19:54                                     ` Parav Pandit
2022-09-07 19:54                                     ` Parav Pandit via Virtualization
2022-09-07 19:51                                 ` [virtio-dev] " Parav Pandit
2022-09-07 19:51                                   ` Parav Pandit
2022-09-07 19:51                                   ` Parav Pandit via Virtualization
2022-09-07 21:39                                   ` [virtio-dev] " Si-Wei Liu
2022-09-07 21:39                                     ` Si-Wei Liu
2022-09-07 21:39                                     ` Si-Wei Liu
2022-09-07 22:11                                     ` Parav Pandit
2022-09-07 22:11                                       ` Parav Pandit via Virtualization
2022-09-07 22:57                                       ` Si-Wei Liu
2022-09-07 22:57                                         ` Si-Wei Liu
2022-09-07 22:57                                         ` Si-Wei Liu
2022-09-22  9:26                                   ` [virtio-dev] " Michael S. Tsirkin
2022-09-22  9:26                                     ` Michael S. Tsirkin
2022-09-22  9:26                                     ` Michael S. Tsirkin
2022-09-22 10:07                                     ` [virtio-dev] " Parav Pandit
2022-09-22 10:07                                       ` Parav Pandit
2022-09-22 10:07                                       ` Parav Pandit via Virtualization
2022-09-07 20:04                                 ` [virtio-dev] " Parav Pandit
2022-09-07 20:04                                   ` Parav Pandit
2022-09-07 20:04                                   ` Parav Pandit via Virtualization
2022-09-22  9:35   ` [virtio-dev] " Michael S. Tsirkin
2022-09-22  9:35     ` Michael S. Tsirkin
2022-09-22  9:35     ` Michael S. Tsirkin
2022-09-22 10:04     ` [virtio-dev] " Parav Pandit
2022-09-22 10:04       ` Parav Pandit
2022-09-22 10:04       ` Parav Pandit via Virtualization
2022-09-22 10:14       ` [virtio-dev] " Michael S. Tsirkin
2022-09-22 10:14         ` Michael S. Tsirkin
2022-09-22 10:14         ` Michael S. Tsirkin
2022-09-22 10:29         ` [virtio-dev] " Parav Pandit
2022-09-22 10:29           ` Parav Pandit
2022-09-22 10:29           ` Parav Pandit via Virtualization
2022-09-22 12:34         ` Jakub Kicinski
2022-10-05 10:29           ` [virtio-dev] " Parav Pandit
2022-10-05 10:29             ` Parav Pandit
2022-10-05 10:29             ` Parav Pandit via Virtualization
2022-09-06 13:50 ` [virtio-dev] [PATCH v5 0/2] Improve virtio performance for 9k mtu Gavin Li
2022-09-07  2:57 ` Gavin Li
2022-09-07  2:57   ` Gavin Li
2022-09-19 15:35   ` Jakub Kicinski
2022-09-20 13:40     ` Gavin Li
2022-09-20 13:45     ` Gavin Li
2022-09-20 13:45       ` Gavin Li
2022-09-22  0:25       ` Jakub Kicinski

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20220907152156-mutt-send-email-mst@kernel.org \
    --to=mst@redhat.com \
    --cc=davem@davemloft.net \
    --cc=gavi@nvidia.com \
    --cc=gavinl@nvidia.com \
    --cc=jasowang@redhat.com \
    --cc=jesse.brandeburg@intel.com \
    --cc=kuba@kernel.org \
    --cc=loseweigh@gmail.com \
    --cc=netdev@vger.kernel.org \
    --cc=parav@nvidia.com \
    --cc=si-wei.liu@oracle.com \
    --cc=sridhar.samudrala@intel.com \
    --cc=stephen@networkplumber.org \
    --cc=virtio-dev@lists.oasis-open.org \
    --cc=virtualization@lists.linux-foundation.org \
    --cc=xuanzhuo@linux.alibaba.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.