Re: -b vs. -n

["Alan D. Brunelle" <Alan.Brunelle@hp.com>]

Alan D. Brunelle wrote:
> Alan D. Brunelle wrote:
>> Jens Axboe wrote:
>>> On Thu, Jan 29 2009, Alan D. Brunelle wrote:
>>>> Has anybody experimented with increasing the _number_ of buffers rather
>>>> than the _size_ of the buffers when confronted with drops? I'm finding
>>>> on a large(ish) system that it is better to have lots of small buffers
>>>> handled by relay rather than fewer larger buffers. In my particular case:
>>>>
>>>> 16 CPUs
>>>> 96 devices
>>>> running some dd's against all the devices...
>>>>
>>>> -b 1024 or -b 2048 still results in drops
>>>>
>>>> but:
>>>>
>>>> -n 512 -b 16 allows things to run smoother.
>>>>
>>>> I _think_ this may have to do with the way relay reports POLLIN: it does
>>>> it only when a buffer switch happens as opposed to when there is data
>>>> ready. Need to look at this some more, but just wondering if others out
>>>> there have found similar things in their testing...
>>> That's interesting. The reason why I exposed both parameters was mainly
>>> that I didn't have the equipment to do adequate testing on what would be
>>> the best setup for this. So perhaps we can change the README to reflect
>>> that it is usually better to bump the number of buffers instead of the
>>> size, if you run into overflow problems?
>>>
>> It's not clear - still running tests. [I know for SMALLER numbers of
>> disks increasing the buffers has worked just fine.] I'm still fighting
>> (part time) with version 2.0 of blktrace, so _that_ may have something
>> to do with it! :-)
>>
>> Alan
> 
> Did some testing over the weekend: purposefully "bad" set up:
> 
> o  48 FC disks having Ext2 file systems created on them (I've found mkfs
> really stresses systems :-) )
> o  5 CCISS disks in LVM w/ an XFS file system used to capture the traces.
> 
> I then did 30 passes for a select set of permutations of -n & -b
> parameters. Here's the averages:
> 
>   -b    4     8    16    32    64   128   256   512  1024  2048  4096
> -n  |----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
>    4|                                           77.9  83.2  88.1  86.7
>    8|                                     77.9  73.7
>   16|                               86.1        65.5
>   32|                         86.0              64.7
>   64|                   85.6
>  128|             85.8
>  256|       85.6
>  512| 86.9
> 1024| 79.8
> 2048| 66.1
> 4096| 70.8
> 
> The values in the chart are percent of traces dropped* (you can see that
> this is a bad set up - >60% drops in all case). (-b is across, and -n is
> down).
> 
> Looking at increasing -b from the default (512) to 4096 whilst keep -n @
> 4 shows _more_ drops: 77.9  83.2  88.1  86.7
> 
> Looking at increasing -n from the default (4) to 2048 whilst keeping -b
> @ 512 shows _fewer_ drops: 77.9, 73.7, 65.5 and then 64.7
> 
> (Doing this with a small buffer size - 4KiB - was a bit inconclusive:
> 86.9 -> 79.8 -> 66.1 but then up to 70.8.)
> 
> The diagonal numbers represent the results from trying to keep the total
> memory consumption level - 4 buffers @ 512K up to 512 buffers @ 4K. Not
> too conclusive, but it seems that there's a benefit to having smaller
> numbers of larger buffers keeping the same memory footprint.
> 
> Anyways, none of this looks too convincing overall - and as noted, it's
> a "bad" case - way too many drops.
> 
> -----------------
> 
> I'm  re-doing this using a more balanced configuration: I'm using the 48
> fibre channel disks to hold the traces, and using 48 CCISS disks to do
> the mkfs operations on. (Preliminary results show we're around the hairy
> edge here - a few % drops on some case (<2.0%).)
> 
> -----------------
> 
> * I have modified blktrace to output the total number of drops, the
> percentage of drops, and changed the suggestion line to read:
> 
> You have 254436128 ( 81.3%) dropped events
> Consider using a larger buffer size (-b) and/or more buffers (-n)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrace" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


Well, the "balanced configuration" didn't help much: going in either
direction (more buffers or larger buffers) both helped. (This is with 48
FC disks set up in an LV & capturing traces generated whilst doing mkfs
on 48 CCISS disks.)

  -b     4     8    16    32    64   128   256   512  1024  2048  4096
-n   |----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
    4|                                            4.4   0.0   0.0   0.0
    8|                                      1.5   0.0
   16|                                0.1         0.0
   32|                          0.8               0.0
   64|                    1.1
  128|              0.8
  256|        2.6
  512|  2.3
 1024|  0.5
 2048|  0.1
 4096|  0.0

 I _think_ having the error message reflect the possibility of upping
 either buffer size OR number of buffers makes sense. I'll also
 back-port the "You have 254436128 ( 81.3%) dropped events" formatted
 message code too.

 Alan