From mboxrd@z Thu Jan 1 00:00:00 1970 From: Thomas Lendacky Subject: Re: [PATCH v2 2/2] virtio-ring: Allocate indirect buffers from cache when possible Date: Mon, 10 Sep 2012 10:47:15 -0500 Message-ID: <14037987.hiN6MSGY6z@tomlt1.ibmoffice.com> References: <1346159043-16446-2-git-send-email-levinsasha928@gmail.com> <20120906084526.GE17656@redhat.com> <87txvahfv3.fsf@rustcorp.com.au> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============8336939264513104731==" Return-path: In-Reply-To: <87txvahfv3.fsf@rustcorp.com.au> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: virtualization-bounces@lists.linux-foundation.org Errors-To: virtualization-bounces@lists.linux-foundation.org To: Rusty Russell Cc: kvm@vger.kernel.org, "Michael S. Tsirkin" , linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, avi@redhat.com, Sasha Levin List-Id: virtualization@lists.linuxfoundation.org --===============8336939264513104731== Content-Type: multipart/alternative; boundary="nextPart2991038.hIkfp3otzW" Content-Transfer-Encoding: 7Bit --nextPart2991038.hIkfp3otzW Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" On Friday, September 07, 2012 09:19:04 AM Rusty Russell wrote: > "Michael S. Tsirkin" writes: > > On Thu, Sep 06, 2012 at 05:27:23PM +0930, Rusty Russell wrote: > >> "Michael S. Tsirkin" writes: > >> > Yes without checksum net core always linearizes packets, so yes it is > >> > screwed. > >> > For -net, skb always allocates space for 17 frags + linear part so > >> > it seems sane to do same in virtio core, and allocate, for -net, > >> > up to max_frags + 1 from cache. > >> > We can adjust it: no _SG -> 2 otherwise 18. > >> > >> But I thought it used individual buffers these days? > > > > Yes for receive, no for transmit. That's probably why > > we should have the threshold per vq, not per device, BTW. > > Can someone actually run with my histogram patch and see what the real > numbers are? > I ran some TCP_RR and TCP_STREAM sessions, both host-to-guest and guest-to-host, with a form of the histogram patch applied against a RHEL6.3 kernel. The histogram values were reset after each test. Here are the results: 60 session TCP_RR from host-to-guest with 256 byte request and 256 byte response for 60 seconds: Queue histogram for virtio1: Size distribution for input (max=7818456): 1: 7818456 ################################################################ Size distribution for output (max=7816698): 2: 149 3: 7816698 ################################################################ 4: 2 5: 1 Size distribution for control (max=1): 0: 0 4 session TCP_STREAM from host-to-guest with 4K message size for 60 seconds: Queue histogram for virtio1: Size distribution for input (max=16050941): 1: 16050941 ################################################################ Size distribution for output (max=1877796): 2: 1877796 ################################################################ 3: 5 Size distribution for control (max=1): 0: 0 4 session TCP_STREAM from host-to-guest with 16K message size for 60 seconds: Queue histogram for virtio1: Size distribution for input (max=16831151): 1: 16831151 ################################################################ Size distribution for output (max=1923965): 2: 1923965 ################################################################ 3: 5 Size distribution for control (max=1): 0: 0 4 session TCP_STREAM from guest-to-host with 4K message size for 60 seconds: Queue histogram for virtio1: Size distribution for input (max=1316069): 1: 1316069 ################################################################ Size distribution for output (max=879213): 2: 24 3: 24097 # 4: 23176 # 5: 3412 6: 4446 7: 4663 8: 4195 9: 3772 10: 3388 11: 3666 12: 2885 13: 2759 14: 2997 15: 3060 16: 2651 17: 2235 18: 92721 ###### 19: 879213 ################################################################ Size distribution for control (max=1): 0: 0 4 session TCP_STREAM from guest-to-host with 16K message size for 60 seconds: Queue histogram for virtio1: Size distribution for input (max=1428590): 1: 1428590 ################################################################ Size distribution for output (max=957774): 2: 20 3: 54955 ### 4: 34281 ## 5: 2967 6: 3394 7: 9400 8: 3061 9: 3397 10: 3258 11: 3275 12: 3147 13: 2876 14: 2747 15: 2832 16: 2013 17: 1670 18: 100369 ###### 19: 957774 ################################################################ Size distribution for control (max=1): 0: 0 Thanks, Tom > I'm not convinced that the ideal 17-buffer case actually happens as much > as we think. And if it's not happening with this netperf test, we're > testing the wrong thing. > > Thanks, > Rusty. > > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html --nextPart2991038.hIkfp3otzW Content-Transfer-Encoding: 7Bit Content-Type: text/html; charset="us-ascii"

On Friday, September 07, 2012 09:19:04 AM Rusty Russell wrote:

> "Michael S. Tsirkin" <mst@redhat.com> writes:

> > On Thu, Sep 06, 2012 at 05:27:23PM +0930, Rusty Russell wrote:

> >> "Michael S. Tsirkin" <mst@redhat.com> writes:

> >> > Yes without checksum net core always linearizes packets, so yes it is

> >> > screwed.

> >> > For -net, skb always allocates space for 17 frags + linear part so

> >> > it seems sane to do same in virtio core, and allocate, for -net,

> >> > up to max_frags + 1 from cache.

> >> > We can adjust it: no _SG -> 2 otherwise 18.

> >>

> >> But I thought it used individual buffers these days?

> >

> > Yes for receive, no for transmit. That's probably why

> > we should have the threshold per vq, not per device, BTW.

>

> Can someone actually run with my histogram patch and see what the real

> numbers are?

>

 

I ran some TCP_RR and TCP_STREAM sessions, both host-to-guest and

guest-to-host, with a form of the histogram patch applied against a

RHEL6.3 kernel. The histogram values were reset after each test.

 

Here are the results:

 

60 session TCP_RR from host-to-guest with 256 byte request and 256 byte

response for 60 seconds:

 

Queue histogram for virtio1:

Size distribution for input (max=7818456):

1: 7818456 ################################################################

Size distribution for output (max=7816698):

2: 149

3: 7816698 ################################################################

4: 2

5: 1

Size distribution for control (max=1):

0: 0

 

 

4 session TCP_STREAM from host-to-guest with 4K message size for 60 seconds:

 

Queue histogram for virtio1:

Size distribution for input (max=16050941):

1: 16050941 ################################################################

Size distribution for output (max=1877796):

2: 1877796 ################################################################

3: 5

Size distribution for control (max=1):

0: 0

 

4 session TCP_STREAM from host-to-guest with 16K message size for 60 seconds:

 

Queue histogram for virtio1:

Size distribution for input (max=16831151):

1: 16831151 ################################################################

Size distribution for output (max=1923965):

2: 1923965 ################################################################

3: 5

Size distribution for control (max=1):

0: 0

 

4 session TCP_STREAM from guest-to-host with 4K message size for 60 seconds:

 

Queue histogram for virtio1:

Size distribution for input (max=1316069):

1: 1316069 ################################################################

Size distribution for output (max=879213):

2: 24

3: 24097 #

4: 23176 #

5: 3412

6: 4446

7: 4663

8: 4195

9: 3772

10: 3388

11: 3666

12: 2885

13: 2759

14: 2997

15: 3060

16: 2651

17: 2235

18: 92721 ######

19: 879213 ################################################################

Size distribution for control (max=1):

0: 0

 

4 session TCP_STREAM from guest-to-host with 16K message size for 60 seconds:

 

Queue histogram for virtio1:

Size distribution for input (max=1428590):

1: 1428590 ################################################################

Size distribution for output (max=957774):

2: 20

3: 54955 ###

4: 34281 ##

5: 2967

6: 3394

7: 9400

8: 3061

9: 3397

10: 3258

11: 3275

12: 3147

13: 2876

14: 2747

15: 2832

16: 2013

17: 1670

18: 100369 ######

19: 957774 ################################################################

Size distribution for control (max=1):

0: 0

 

Thanks,

Tom

 

> I'm not convinced that the ideal 17-buffer case actually happens as much

> as we think. And if it's not happening with this netperf test, we're

> testing the wrong thing.

>

> Thanks,

> Rusty.

>

> --

> To unsubscribe from this list: send the line "unsubscribe kvm" in

> the body of a message to majordomo@vger.kernel.org

> More majordomo info at http://vger.kernel.org/majordomo-info.html

 

--nextPart2991038.hIkfp3otzW-- --===============8336939264513104731== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization --===============8336939264513104731==--