MPTSAS Driver and scatter gather lists

MPTSAS Driver and scatter gather lists

[Steven Pratt]

I was doing some performance comparisons between the aic94xx sas adapter 
and the mptsas (lsi1078 chip) and saw that performance was slightly 
lower on lsi using the same drive. One thing I noticed the mptsas driver 
was only setting sg_tablesize to 40 which restricts the maximum size of 
the IO to 160kb.  This is inefficient and so I changed it to 255 (max 
value) for testing purposes.  The results were not what I expected.  
While CPU efficiency was improved at the larger block sizes (like I had 
hopped), there were drastic improvements at small (<16k) block sizes 
which is completely surprising).  So I decided to see where all the time 
was being spent.  When doing oprofile, loss of idle ticks was confusing 
things so I booted with idle=poll, and magically the original mptsas 
driver now performs like the modified version at the small block sizes.
My suspicion is that we have some some sleep delay which is minimized by 
going into poll mode, but I have no idea why changing the sg_tablesize 
gives the same results regardless of poll mode.  I should reiterate that 
I am not talking about small difference. This is 2x performance at 1/3 cpu.

Also, is there any reason we can't increase sg_tablesize for mptsas?

Any thoughts or experiments to try are welcome.
Steve

RE: MPTSAS Driver and scatter gather lists

[Moore, Eric]

On Tuesday, December 04, 2007 8:45 AM,  Steven Pratt wrote:
> 
> Also, is there any reason we can't increase sg_tablesize for mptsas?
> 

The default 128, set in Kconfig (look at FUSION_MAX_SGE).    It only set
to 40 when that is not defined.  What is in your kernel .config, e.g
look for CONFIG_FUSION_MAX_SGE.

Eric

Re: MPTSAS Driver and scatter gather lists

[Steven Pratt]

Moore, Eric wrote:
> On Tuesday, December 04, 2007 8:45 AM,  Steven Pratt wrote:
>   
>> Also, is there any reason we can't increase sg_tablesize for mptsas?
>>
>>     
>
> The default 128, set in Kconfig (look at FUSION_MAX_SGE).    It only set
> to 40 when that is not defined.  What is in your kernel .config, e.g
> look for CONFIG_FUSION_MAX_SGE.
>   

RHEL5 x86_64 has this:

 >grep CONFIG_FUSION_MAX_SGE /boot/config-2.6.18-8.el5
CONFIG_FUSION_MAX_SGE=40

Steve
> Eric
>   

Re: MPTSAS Driver and scatter gather lists

[Zhao Forrest]

Steve,
Do you use a 10k RPM SAS hard drive? Could you share that what test
tools(or test cases) are you using to do the block IO test? We have a
LSI SAS1068E chip and 10k RPM SAS hard drive in our lab. I'll find
some time to reproduce the issue and do some investigation.

Thanks,
Forrest

On 12/4/07, Steven Pratt <slpratt@austin.ibm.com> wrote:
> I was doing some performance comparisons between the aic94xx sas adapter
> and the mptsas (lsi1078 chip) and saw that performance was slightly
> lower on lsi using the same drive. One thing I noticed the mptsas driver
> was only setting sg_tablesize to 40 which restricts the maximum size of
> the IO to 160kb.  This is inefficient and so I changed it to 255 (max
> value) for testing purposes.  The results were not what I expected.
> While CPU efficiency was improved at the larger block sizes (like I had
> hopped), there were drastic improvements at small (<16k) block sizes
> which is completely surprising).  So I decided to see where all the time
> was being spent.  When doing oprofile, loss of idle ticks was confusing
> things so I booted with idle=poll, and magically the original mptsas
> driver now performs like the modified version at the small block sizes.
> My suspicion is that we have some some sleep delay which is minimized by
> going into poll mode, but I have no idea why changing the sg_tablesize
> gives the same results regardless of poll mode.  I should reiterate that
> I am not talking about small difference. This is 2x performance at 1/3 cpu.
>
> Also, is there any reason we can't increase sg_tablesize for mptsas?
>
> Any thoughts or experiments to try are welcome.
> Steve
> -
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: MPTSAS Driver and scatter gather lists

[Steven Pratt]

Zhao Forrest wrote:
> Steve,
> Do you use a 10k RPM SAS hard drive? Could you share that what test
> tools(or test cases) are you using to do the block IO test? We have a
> LSI SAS1068E chip and 10k RPM SAS hard drive in our lab. I'll find
> some time to reproduce the issue and do some investigation.
>   
I've been using 10k and 15k drives (although these particular tests were 
from 15k).  I've been mostly using an internal IO tool, but this is 
easily reproducible with dd using the iflag=direct to force ODIRECT. 

Steve

> Thanks,
> Forrest
>
> On 12/4/07, Steven Pratt <slpratt@austin.ibm.com> wrote:
>   
>> I was doing some performance comparisons between the aic94xx sas adapter
>> and the mptsas (lsi1078 chip) and saw that performance was slightly
>> lower on lsi using the same drive. One thing I noticed the mptsas driver
>> was only setting sg_tablesize to 40 which restricts the maximum size of
>> the IO to 160kb.  This is inefficient and so I changed it to 255 (max
>> value) for testing purposes.  The results were not what I expected.
>> While CPU efficiency was improved at the larger block sizes (like I had
>> hopped), there were drastic improvements at small (<16k) block sizes
>> which is completely surprising).  So I decided to see where all the time
>> was being spent.  When doing oprofile, loss of idle ticks was confusing
>> things so I booted with idle=poll, and magically the original mptsas
>> driver now performs like the modified version at the small block sizes.
>> My suspicion is that we have some some sleep delay which is minimized by
>> going into poll mode, but I have no idea why changing the sg_tablesize
>> gives the same results regardless of poll mode.  I should reiterate that
>> I am not talking about small difference. This is 2x performance at 1/3 cpu.
>>
>> Also, is there any reason we can't increase sg_tablesize for mptsas?
>>
>> Any thoughts or experiments to try are welcome.
>> Steve
>> -
>> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>     

Re: MPTSAS Driver and scatter gather lists

[Steven Pratt]

Zhao Forrest wrote:
> Steve,
> Do you use a 10k RPM SAS hard drive? Could you share that what test
> tools(or test cases) are you using to do the block IO test? We have a
> LSI SAS1068E chip and 10k RPM SAS hard drive in our lab. I'll find
> some time to reproduce the issue and do some investigation.
>
>   
I've been using 10k and 15k drives (although these particular tests were 
from 15k).  I've been mostly using an internal IO tool, but this is 
easily reproducible with dd using the iflag=direct to force ODIRECT. 

Steve

> Thanks,
> Forrest
>
> On 12/4/07, Steven Pratt <slpratt@austin.ibm.com> wrote:
>   
>> I was doing some performance comparisons between the aic94xx sas adapter
>> and the mptsas (lsi1078 chip) and saw that performance was slightly
>> lower on lsi using the same drive. One thing I noticed the mptsas driver
>> was only setting sg_tablesize to 40 which restricts the maximum size of
>> the IO to 160kb.  This is inefficient and so I changed it to 255 (max
>> value) for testing purposes.  The results were not what I expected.
>> While CPU efficiency was improved at the larger block sizes (like I had
>> hopped), there were drastic improvements at small (<16k) block sizes
>> which is completely surprising).  So I decided to see where all the time
>> was being spent.  When doing oprofile, loss of idle ticks was confusing
>> things so I booted with idle=poll, and magically the original mptsas
>> driver now performs like the modified version at the small block sizes.
>> My suspicion is that we have some some sleep delay which is minimized by
>> going into poll mode, but I have no idea why changing the sg_tablesize
>> gives the same results regardless of poll mode.  I should reiterate that
>> I am not talking about small difference. This is 2x performance at 1/3 cpu.
>>
>> Also, is there any reason we can't increase sg_tablesize for mptsas?
>>
>> Any thoughts or experiments to try are welcome.
>> Steve
>> -
>> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>