* Re: Performance regression with virtio_net [not found] <20170727170942.ub24lwgrl3x2outp@ubuntu-hedt> @ 2017-07-27 20:38 ` Michael S. Tsirkin [not found] ` <20170727233244-mutt-send-email-mst@kernel.org> 1 sibling, 0 replies; 5+ messages in thread From: Michael S. Tsirkin @ 2017-07-27 20:38 UTC (permalink / raw) To: Seth Forshee; +Cc: netdev, virtualization On Thu, Jul 27, 2017 at 12:09:42PM -0500, Seth Forshee wrote: > I'm seeing a performance regression with virtio_net that looks to have > started in 4.12-rc1. I only see it in one context though, downloading > snap packages from the Ubuntu snap store. For example: > > https://api.snapcraft.io/api/v1/snaps/download/b8X2psL1ryVrPt5WEmpYiqfr5emixTd7_1797.snap > > which redirects to Internap's CDN. Normally this downloads in a few > seconds at ~10 MB/s, but with 4.12 and 4.13 it takes minutes with a rate > of ~150 KB/s. Everything else I've tried downloads as normal speeds. So just wget that URL should be enough? > I bisected this to 680557cf79f8 "virtio_net: rework mergeable buffer > handling". If I revert this on top of 4.13-rc2 (along with other changes > needed to successfully revert it) speeds return to normal. > > Thanks, > Seth Interesting. A more likely suspect would be e377fcc8486d40867c6c217077ad0fa40977e060 - could you please try reverting that one instead? Also, could you please look at mergeable_rx_buffer_size in sysfs with and without the change? -- MST ^ permalink raw reply [flat|nested] 5+ messages in thread
[parent not found: <20170727233244-mutt-send-email-mst@kernel.org>]
* Re: Performance regression with virtio_net [not found] ` <20170727233244-mutt-send-email-mst@kernel.org> @ 2017-07-27 21:14 ` Seth Forshee [not found] ` <20170727211430.viki2jcxbzk2u62c@ubuntu-hedt> 1 sibling, 0 replies; 5+ messages in thread From: Seth Forshee @ 2017-07-27 21:14 UTC (permalink / raw) To: Michael S. Tsirkin; +Cc: netdev, virtualization On Thu, Jul 27, 2017 at 11:38:52PM +0300, Michael S. Tsirkin wrote: > On Thu, Jul 27, 2017 at 12:09:42PM -0500, Seth Forshee wrote: > > I'm seeing a performance regression with virtio_net that looks to have > > started in 4.12-rc1. I only see it in one context though, downloading > > snap packages from the Ubuntu snap store. For example: > > > > https://api.snapcraft.io/api/v1/snaps/download/b8X2psL1ryVrPt5WEmpYiqfr5emixTd7_1797.snap > > > > which redirects to Internap's CDN. Normally this downloads in a few > > seconds at ~10 MB/s, but with 4.12 and 4.13 it takes minutes with a rate > > of ~150 KB/s. Everything else I've tried downloads as normal speeds. > > So just wget that URL should be enough? Yes. Note that sometimes it starts out faster then slows down. > > I bisected this to 680557cf79f8 "virtio_net: rework mergeable buffer > > handling". If I revert this on top of 4.13-rc2 (along with other changes > > needed to successfully revert it) speeds return to normal. > > > > Thanks, > > Seth > > > Interesting. A more likely suspect would be > e377fcc8486d40867c6c217077ad0fa40977e060 - could you please try > reverting that one instead? I tried it, and I still get slow download speeds. I did test at 680557cf79f82623e2c4fd42733077d60a843513 during the bisect so I'm reasonably confident that this is the one where things went bad. > Also, could you please look at mergeable_rx_buffer_size in sysfs with > and without the change? In all cases (stock 4.13-rc2, 680557cf79f8 reverted, and e377fcc8486d reverted) mergeable_rx_buffer_size was 1536. Thanks, Seth ^ permalink raw reply [flat|nested] 5+ messages in thread
[parent not found: <20170727211430.viki2jcxbzk2u62c@ubuntu-hedt>]
* Re: Performance regression with virtio_net [not found] ` <20170727211430.viki2jcxbzk2u62c@ubuntu-hedt> @ 2017-07-27 21:30 ` Michael S. Tsirkin 2017-07-28 13:12 ` Seth Forshee 0 siblings, 1 reply; 5+ messages in thread From: Michael S. Tsirkin @ 2017-07-27 21:30 UTC (permalink / raw) To: Seth Forshee; +Cc: netdev, virtualization On Thu, Jul 27, 2017 at 04:14:30PM -0500, Seth Forshee wrote: > On Thu, Jul 27, 2017 at 11:38:52PM +0300, Michael S. Tsirkin wrote: > > On Thu, Jul 27, 2017 at 12:09:42PM -0500, Seth Forshee wrote: > > > I'm seeing a performance regression with virtio_net that looks to have > > > started in 4.12-rc1. I only see it in one context though, downloading > > > snap packages from the Ubuntu snap store. For example: > > > > > > https://api.snapcraft.io/api/v1/snaps/download/b8X2psL1ryVrPt5WEmpYiqfr5emixTd7_1797.snap > > > > > > which redirects to Internap's CDN. Normally this downloads in a few > > > seconds at ~10 MB/s, but with 4.12 and 4.13 it takes minutes with a rate > > > of ~150 KB/s. Everything else I've tried downloads as normal speeds. > > > > So just wget that URL should be enough? > > Yes. Note that sometimes it starts out faster then slows down. > > > I bisected this to 680557cf79f8 "virtio_net: rework mergeable buffer > > > handling". If I revert this on top of 4.13-rc2 (along with other changes > > > needed to successfully revert it) speeds return to normal. > > > > > > Thanks, > > > Seth > > > > > > Interesting. A more likely suspect would be > > e377fcc8486d40867c6c217077ad0fa40977e060 - could you please try > > reverting that one instead? > > I tried it, and I still get slow download speeds. I did test at > 680557cf79f82623e2c4fd42733077d60a843513 during the bisect so I'm > reasonably confident that this is the one where things went bad. > > Also, could you please look at mergeable_rx_buffer_size in sysfs with > > and without the change? > > In all cases (stock 4.13-rc2, 680557cf79f8 reverted, and e377fcc8486d > reverted) mergeable_rx_buffer_size was 1536. > > Thanks, > Seth Do you see any error counters incrementing after it slows down? -- MST ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Performance regression with virtio_net 2017-07-27 21:30 ` Michael S. Tsirkin @ 2017-07-28 13:12 ` Seth Forshee 0 siblings, 0 replies; 5+ messages in thread From: Seth Forshee @ 2017-07-28 13:12 UTC (permalink / raw) To: Michael S. Tsirkin; +Cc: netdev, virtualization On Fri, Jul 28, 2017 at 12:30:54AM +0300, Michael S. Tsirkin wrote: > On Thu, Jul 27, 2017 at 04:14:30PM -0500, Seth Forshee wrote: > > On Thu, Jul 27, 2017 at 11:38:52PM +0300, Michael S. Tsirkin wrote: > > > On Thu, Jul 27, 2017 at 12:09:42PM -0500, Seth Forshee wrote: > > > > I'm seeing a performance regression with virtio_net that looks to have > > > > started in 4.12-rc1. I only see it in one context though, downloading > > > > snap packages from the Ubuntu snap store. For example: > > > > > > > > https://api.snapcraft.io/api/v1/snaps/download/b8X2psL1ryVrPt5WEmpYiqfr5emixTd7_1797.snap > > > > > > > > which redirects to Internap's CDN. Normally this downloads in a few > > > > seconds at ~10 MB/s, but with 4.12 and 4.13 it takes minutes with a rate > > > > of ~150 KB/s. Everything else I've tried downloads as normal speeds. > > > > > > So just wget that URL should be enough? > > > > Yes. Note that sometimes it starts out faster then slows down. > > > > I bisected this to 680557cf79f8 "virtio_net: rework mergeable buffer > > > > handling". If I revert this on top of 4.13-rc2 (along with other changes > > > > needed to successfully revert it) speeds return to normal. > > > > > > > > Thanks, > > > > Seth > > > > > > > > > Interesting. A more likely suspect would be > > > e377fcc8486d40867c6c217077ad0fa40977e060 - could you please try > > > reverting that one instead? > > > > I tried it, and I still get slow download speeds. I did test at > > 680557cf79f82623e2c4fd42733077d60a843513 during the bisect so I'm > > reasonably confident that this is the one where things went bad. > > > Also, could you please look at mergeable_rx_buffer_size in sysfs with > > > and without the change? > > > > In all cases (stock 4.13-rc2, 680557cf79f8 reverted, and e377fcc8486d > > reverted) mergeable_rx_buffer_size was 1536. > > > > Thanks, > > Seth > > Do you see any error counters incrementing after it slows down? I see rx_dropped and rx_length_errors increasing in lockstep once it slows down. Thanks, Seth ^ permalink raw reply [flat|nested] 5+ messages in thread
* Performance regression with virtio_net @ 2017-07-27 17:09 Seth Forshee 0 siblings, 0 replies; 5+ messages in thread From: Seth Forshee @ 2017-07-27 17:09 UTC (permalink / raw) To: Michael S. Tsirkin, Jason Wang; +Cc: netdev, virtualization I'm seeing a performance regression with virtio_net that looks to have started in 4.12-rc1. I only see it in one context though, downloading snap packages from the Ubuntu snap store. For example: https://api.snapcraft.io/api/v1/snaps/download/b8X2psL1ryVrPt5WEmpYiqfr5emixTd7_1797.snap which redirects to Internap's CDN. Normally this downloads in a few seconds at ~10 MB/s, but with 4.12 and 4.13 it takes minutes with a rate of ~150 KB/s. Everything else I've tried downloads as normal speeds. I bisected this to 680557cf79f8 "virtio_net: rework mergeable buffer handling". If I revert this on top of 4.13-rc2 (along with other changes needed to successfully revert it) speeds return to normal. Thanks, Seth ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2017-07-28 13:12 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20170727170942.ub24lwgrl3x2outp@ubuntu-hedt>
2017-07-27 20:38 ` Performance regression with virtio_net Michael S. Tsirkin
[not found] ` <20170727233244-mutt-send-email-mst@kernel.org>
2017-07-27 21:14 ` Seth Forshee
[not found] ` <20170727211430.viki2jcxbzk2u62c@ubuntu-hedt>
2017-07-27 21:30 ` Michael S. Tsirkin
2017-07-28 13:12 ` Seth Forshee
2017-07-27 17:09 Seth Forshee
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).