OSDL DBT-2 AS vs. Deadline 2.5.68-mm2

OSDL DBT-2 AS vs. Deadline 2.5.68-mm2

[markw]

I've collected some data from STP to see if it's useful or if there's
anything else that would be useful to collect. I've got some tests
queued up for the newer patches, but I wanted to put out what I had so
far.


METRICS OVER LAST 20 MINUTES:
--------------- -------- ----- ---- -------- -----------------------------------
Kernel          Elevator NOTPM CPU% Blocks/s URL                                
--------------- -------- ----- ---- -------- -----------------------------------
2.5.68-mm2      as        1155 94.3   8940.2 http://khack.osdl.org/stp/271356/  
2.5.68-mm2      deadline  1255 94.9   9598.7 http://khack.osdl.org/stp/271359/  

FUNCTIONS SORTED BY TICKS:
-- ------------------------- ------- ------------------------- -------
 # as 2.5.68-mm2             ticks   deadline 2.5.68-mm2       ticks  
-- ------------------------- ------- ------------------------- -------
 1 default_idle              6103428 default_idle              5359025
 2 bounce_copy_vec             86272 bounce_copy_vec             97696
 3 schedule                    63819 schedule                    70114
 4 __make_request              30397 __blk_queue_bounce          31167
 5 __blk_queue_bounce          26962 scsi_request_fn             26623
 6 scsi_request_fn             24845 __make_request              25012
 7 do_softirq                  21122 do_softirq                  24623
 8 scsi_end_request            14080 system_call                 13056
 9 system_call                 12059 try_to_wake_up              12503
10 try_to_wake_up              11240 dio_bio_end_io              11511

-- 
Mark Wong - - markw@osdl.org
Open Source Development Lab Inc - A non-profit corporation
15275 SW Koll Parkway - Suite H - Beaverton OR, 97006
(503)-626-2455 x 32 (office)
(503)-626-2436      (fax)
http://www.osdl.org/archive/markw/

Re: OSDL DBT-2 AS vs. Deadline 2.5.68-mm2

[William Lee Irwin III]

On Wed, May 07, 2003 at 09:33:29AM -0700, markw@osdl.org wrote:
> FUNCTIONS SORTED BY TICKS:
> -- ------------------------- ------- ------------------------- -------
>  # as 2.5.68-mm2             ticks   deadline 2.5.68-mm2       ticks  
> -- ------------------------- ------- ------------------------- -------
>  1 default_idle              6103428 default_idle              5359025
>  2 bounce_copy_vec             86272 bounce_copy_vec             97696
>  3 schedule                    63819 schedule                    70114
>  4 __make_request              30397 __blk_queue_bounce          31167
>  5 __blk_queue_bounce          26962 scsi_request_fn             26623
>  6 scsi_request_fn             24845 __make_request              25012
>  7 do_softirq                  21122 do_softirq                  24623
>  8 scsi_end_request            14080 system_call                 13056
>  9 system_call                 12059 try_to_wake_up              12503
> 10 try_to_wake_up              11240 dio_bio_end_io              11511

You're already in deeper trouble than elevators can get you out of as
your driver is using bounce buffers. What hardware/driver are you using?


-- wli

Re: OSDL DBT-2 AS vs. Deadline 2.5.68-mm2

[Jens Axboe]

On Wed, May 07 2003, markw@osdl.org wrote:
> I've collected some data from STP to see if it's useful or if there's
> anything else that would be useful to collect. I've got some tests
> queued up for the newer patches, but I wanted to put out what I had so
> far.
> 
> 
> METRICS OVER LAST 20 MINUTES:
> --------------- -------- ----- ---- -------- -----------------------------------
> Kernel          Elevator NOTPM CPU% Blocks/s URL                                
> --------------- -------- ----- ---- -------- -----------------------------------
> 2.5.68-mm2      as        1155 94.3   8940.2 http://khack.osdl.org/stp/271356/  
> 2.5.68-mm2      deadline  1255 94.9   9598.7 http://khack.osdl.org/stp/271359/  
> 
> FUNCTIONS SORTED BY TICKS:
> -- ------------------------- ------- ------------------------- -------
>  # as 2.5.68-mm2             ticks   deadline 2.5.68-mm2       ticks  
> -- ------------------------- ------- ------------------------- -------
>  1 default_idle              6103428 default_idle              5359025
>  2 bounce_copy_vec             86272 bounce_copy_vec             97696
>  3 schedule                    63819 schedule                    70114
>  4 __make_request              30397 __blk_queue_bounce          31167
>  5 __blk_queue_bounce          26962 scsi_request_fn             26623
>  6 scsi_request_fn             24845 __make_request              25012

uhh nasty, you are spending a lot of time bouncing. How much RAM is in
the machine, and what is the scsi hba?

-- 
Jens Axboe

Re: OSDL DBT-2 AS vs. Deadline 2.5.68-mm2

[markw]

On  7 May, Jens Axboe wrote:
> On Wed, May 07 2003, markw@osdl.org wrote:
>> I've collected some data from STP to see if it's useful or if there's
>> anything else that would be useful to collect. I've got some tests
>> queued up for the newer patches, but I wanted to put out what I had so
>> far.
>> 
>> 
>> METRICS OVER LAST 20 MINUTES:
>> --------------- -------- ----- ---- -------- -----------------------------------
>> Kernel          Elevator NOTPM CPU% Blocks/s URL                                
>> --------------- -------- ----- ---- -------- -----------------------------------
>> 2.5.68-mm2      as        1155 94.3   8940.2 http://khack.osdl.org/stp/271356/  
>> 2.5.68-mm2      deadline  1255 94.9   9598.7 http://khack.osdl.org/stp/271359/  
>> 
>> FUNCTIONS SORTED BY TICKS:
>> -- ------------------------- ------- ------------------------- -------
>>  # as 2.5.68-mm2             ticks   deadline 2.5.68-mm2       ticks  
>> -- ------------------------- ------- ------------------------- -------
>>  1 default_idle              6103428 default_idle              5359025
>>  2 bounce_copy_vec             86272 bounce_copy_vec             97696
>>  3 schedule                    63819 schedule                    70114
>>  4 __make_request              30397 __blk_queue_bounce          31167
>>  5 __blk_queue_bounce          26962 scsi_request_fn             26623
>>  6 scsi_request_fn             24845 __make_request              25012
> 
> uhh nasty, you are spending a lot of time bouncing. How much RAM is in
> the machine, and what is the scsi hba?

The system has 4GB of memory and has a DECchip 21554 (aacraid) that the
external drives are connected to.  Mark Haverkamp is currently trying to
address those bounce buffers.

Re: OSDL DBT-2 AS vs. Deadline 2.5.68-mm2

[Jens Axboe]

On Wed, May 07 2003, markw@osdl.org wrote:
> On  7 May, Jens Axboe wrote:
> > On Wed, May 07 2003, markw@osdl.org wrote:
> >> I've collected some data from STP to see if it's useful or if there's
> >> anything else that would be useful to collect. I've got some tests
> >> queued up for the newer patches, but I wanted to put out what I had so
> >> far.
> >> 
> >> 
> >> METRICS OVER LAST 20 MINUTES:
> >> --------------- -------- ----- ---- -------- -----------------------------------
> >> Kernel          Elevator NOTPM CPU% Blocks/s URL                                
> >> --------------- -------- ----- ---- -------- -----------------------------------
> >> 2.5.68-mm2      as        1155 94.3   8940.2 http://khack.osdl.org/stp/271356/  
> >> 2.5.68-mm2      deadline  1255 94.9   9598.7 http://khack.osdl.org/stp/271359/  
> >> 
> >> FUNCTIONS SORTED BY TICKS:
> >> -- ------------------------- ------- ------------------------- -------
> >>  # as 2.5.68-mm2             ticks   deadline 2.5.68-mm2       ticks  
> >> -- ------------------------- ------- ------------------------- -------
> >>  1 default_idle              6103428 default_idle              5359025
> >>  2 bounce_copy_vec             86272 bounce_copy_vec             97696
> >>  3 schedule                    63819 schedule                    70114
> >>  4 __make_request              30397 __blk_queue_bounce          31167
> >>  5 __blk_queue_bounce          26962 scsi_request_fn             26623
> >>  6 scsi_request_fn             24845 __make_request              25012
> > 
> > uhh nasty, you are spending a lot of time bouncing. How much RAM is in
> > the machine, and what is the scsi hba?
> 
> The system has 4GB of memory and has a DECchip 21554 (aacraid) that the
> external drives are connected to.  Mark Haverkamp is currently trying to
> address those bounce buffers.

aacraid actually looks sane enough. so you should just be able to set
host->highmem_io and it should work.

-- 
Jens Axboe

Re: OSDL DBT-2 AS vs. Deadline 2.5.68-mm2

[Mark Haverkamp]

On Wed, 2003-05-07 at 10:36, Jens Axboe wrote:
> On Wed, May 07 2003, markw@osdl.org wrote:
> > On  7 May, Jens Axboe wrote:
> > > On Wed, May 07 2003, markw@osdl.org wrote:
> > >> I've collected some data from STP to see if it's useful or if there's
> > >> anything else that would be useful to collect. I've got some tests
> > >> queued up for the newer patches, but I wanted to put out what I had so
> > >> far.
> > >> 
> > >> 
> > >> METRICS OVER LAST 20 MINUTES:
> > >> --------------- -------- ----- ---- -------- -----------------------------------
> > >> Kernel          Elevator NOTPM CPU% Blocks/s URL                                
> > >> --------------- -------- ----- ---- -------- -----------------------------------
> > >> 2.5.68-mm2      as        1155 94.3   8940.2 http://khack.osdl.org/stp/271356/  
> > >> 2.5.68-mm2      deadline  1255 94.9   9598.7 http://khack.osdl.org/stp/271359/  
> > >> 
> > >> FUNCTIONS SORTED BY TICKS:
> > >> -- ------------------------- ------- ------------------------- -------
> > >>  # as 2.5.68-mm2             ticks   deadline 2.5.68-mm2       ticks  
> > >> -- ------------------------- ------- ------------------------- -------
> > >>  1 default_idle              6103428 default_idle              5359025
> > >>  2 bounce_copy_vec             86272 bounce_copy_vec             97696
> > >>  3 schedule                    63819 schedule                    70114
> > >>  4 __make_request              30397 __blk_queue_bounce          31167
> > >>  5 __blk_queue_bounce          26962 scsi_request_fn             26623
> > >>  6 scsi_request_fn             24845 __make_request              25012
> > > 
> > > uhh nasty, you are spending a lot of time bouncing. How much RAM is in
> > > the machine, and what is the scsi hba?
> > 
> > The system has 4GB of memory and has a DECchip 21554 (aacraid) that the
> > external drives are connected to.  Mark Haverkamp is currently trying to
> > address those bounce buffers.
> 
> aacraid actually looks sane enough. so you should just be able to set
> host->highmem_io and it should work.

That is what I thought, but I wanted to get a machine with lots of
memory to try it out on.

Mark.
-- 
Mark Haverkamp <markh@osdl.org>

Re: OSDL DBT-2 AS vs. Deadline 2.5.68-mm2

[Zwane Mwaikambo]

On Wed, 7 May 2003, Mark Haverkamp wrote:

> That is what I thought, but I wanted to get a machine with lots of
> memory to try it out on.

if(aac->pae_support)
	pci_set_dma_mask(dev, 0xFFFFFFFFFFFFFFFFUL);

This probably wants 0xFFFFFFFFFFFFFFFFULL and checking pci_set_dma_mask() 
return value.

-- 
function.linuxpower.ca

Re: OSDL DBT-2 AS vs. Deadline 2.5.68-mm2

[Martin Josefsson]

On Wed, 2003-05-07 at 18:33, markw@osdl.org wrote:
> I've collected some data from STP to see if it's useful or if there's
> anything else that would be useful to collect. I've got some tests
> queued up for the newer patches, but I wanted to put out what I had so
> far.
> 
> 
> METRICS OVER LAST 20 MINUTES:
> --------------- -------- ----- ---- -------- -----------------------------------
> Kernel          Elevator NOTPM CPU% Blocks/s URL                                
> --------------- -------- ----- ---- -------- -----------------------------------
> 2.5.68-mm2      as        1155 94.3   8940.2 http://khack.osdl.org/stp/271356/  
> 2.5.68-mm2      deadline  1255 94.9   9598.7 http://khack.osdl.org/stp/271359/  
> 
> FUNCTIONS SORTED BY TICKS:
> -- ------------------------- ------- ------------------------- -------
>  # as 2.5.68-mm2             ticks   deadline 2.5.68-mm2       ticks  
> -- ------------------------- ------- ------------------------- -------
>  1 default_idle              6103428 default_idle              5359025
>  2 bounce_copy_vec             86272 bounce_copy_vec             97696
>  3 schedule                    63819 schedule                    70114
>  4 __make_request              30397 __blk_queue_bounce          31167
>  5 __blk_queue_bounce          26962 scsi_request_fn             26623
>  6 scsi_request_fn             24845 __make_request              25012

You are using scsi, what tcq depth are you using? AS doesn't like >4 or
something like that.

-- 
/Martin

Re: OSDL DBT-2 AS vs. Deadline 2.5.68-mm2

[markw]

On  7 May, Martin Josefsson wrote:
> On Wed, 2003-05-07 at 18:33, markw@osdl.org wrote:
>> I've collected some data from STP to see if it's useful or if there's
>> anything else that would be useful to collect. I've got some tests
>> queued up for the newer patches, but I wanted to put out what I had so
>> far.
>> 
>> 
>> METRICS OVER LAST 20 MINUTES:
>> --------------- -------- ----- ---- -------- -----------------------------------
>> Kernel          Elevator NOTPM CPU% Blocks/s URL                                
>> --------------- -------- ----- ---- -------- -----------------------------------
>> 2.5.68-mm2      as        1155 94.3   8940.2 http://khack.osdl.org/stp/271356/  
>> 2.5.68-mm2      deadline  1255 94.9   9598.7 http://khack.osdl.org/stp/271359/  
>> 
>> FUNCTIONS SORTED BY TICKS:
>> -- ------------------------- ------- ------------------------- -------
>>  # as 2.5.68-mm2             ticks   deadline 2.5.68-mm2       ticks  
>> -- ------------------------- ------- ------------------------- -------
>>  1 default_idle              6103428 default_idle              5359025
>>  2 bounce_copy_vec             86272 bounce_copy_vec             97696
>>  3 schedule                    63819 schedule                    70114
>>  4 __make_request              30397 __blk_queue_bounce          31167
>>  5 __blk_queue_bounce          26962 scsi_request_fn             26623
>>  6 scsi_request_fn             24845 __make_request              25012
> 
> You are using scsi, what tcq depth are you using? AS doesn't like >4 or
> something like that.

I'm told that it's set to 128 with this bit of code:

        if(dev->tagged_supported)
                scsi_adjust_queue_depth(dev, MSG_ORDERED_TAG, 128);

Re: OSDL DBT-2 AS vs. Deadline 2.5.68-mm2

[Nick Piggin]

markw@osdl.org wrote:

>I've collected some data from STP to see if it's useful or if there's
>anything else that would be useful to collect. I've got some tests
>queued up for the newer patches, but I wanted to put out what I had so
>far.
>
Thanks. It looks like AS isn't doing too badly here. Newer mm kernels
have some more AS changes, and the dynamic struct request patch which
would be good to test.

Are you using TCQ on your disks?

Re: OSDL DBT-2 AS vs. Deadline 2.5.68-mm2

[markw]

On  8 May, Nick Piggin wrote:
> markw@osdl.org wrote:
> 
>>I've collected some data from STP to see if it's useful or if there's
>>anything else that would be useful to collect. I've got some tests
>>queued up for the newer patches, but I wanted to put out what I had so
>>far.
>>
> Thanks. It looks like AS isn't doing too badly here. Newer mm kernels
> have some more AS changes, and the dynamic struct request patch which
> would be good to test.
> 
> Are you using TCQ on your disks?
> 

There's a queue depth being set.  Is that a good indicator that TCQ is
being used?  If not, I'd be happy to verify it.

Mark

Re: OSDL DBT-2 AS vs. Deadline 2.5.68-mm2

[Jens Axboe]

On Thu, May 08 2003, markw@osdl.org wrote:
> On  8 May, Nick Piggin wrote:
> > markw@osdl.org wrote:
> > 
> >>I've collected some data from STP to see if it's useful or if there's
> >>anything else that would be useful to collect. I've got some tests
> >>queued up for the newer patches, but I wanted to put out what I had so
> >>far.
> >>
> > Thanks. It looks like AS isn't doing too badly here. Newer mm kernels
> > have some more AS changes, and the dynamic struct request patch which
> > would be good to test.
> > 
> > Are you using TCQ on your disks?
> > 
> 
> There's a queue depth being set.  Is that a good indicator that TCQ is
> being used?  If not, I'd be happy to verify it.

The queue depth being set, is the highest queueing depth that the scsi
mid layer will throw at you. The actual TCQ depth may be lower, depends
on the hardware. aacraid, iirc, has a pretty big depth so I woudldn't be
surprised if it could use all of those 128 tags.

It would be interesting to see a forced depth of 2 with AS against the
stock case of 128 (and deadline).

-- 
Jens Axboe