Bad SCSI performance with 2.6.14 and Fusion-MPT

Bad SCSI performance with 2.6.14 and Fusion-MPT

[Michael Renner]

Hi,

I'm seeing really bad scsi performance with 2.6.14 in combination with Fusion
MPT and amd64. The same behavior was seen in 2.6.14-git13.

The controller in question identifies itself as "LSI Logic / Symbios Logic
53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07)", PCI ID 1000:0030.

Starting with 2.6.14 (didn't test the releases between 2.6.13 and 2.6.14) the
resync rate of the software raid dropped to about 2mb/sec on a raid1 array over
two scsi disks (normally I get around 60-80mb/sec), doing a dd from one disk to
the other supported those values. I can't give exact measures since a "sync"
call hangs since half an hour (was a bit too much data for testing) but vmstat
shows:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  5      0 23873224 8198000  24288    0    0   463   259   71    23  0  0 83 17
 0  5      0 23872968 8198000  24288    0    0     0  4096  273    48  0  0 75 25
 0  5      0 23873340 8198000  24288    0    0     0     0  271    50  0  0 75 25
 0  4      0 23873084 8198000  24288    0    0     0  4096  270    49  0  0 75 24
 0  4      0 23873120 8198000  24288    0    0     0  4096  274    54  0  0 75 25
 0  4      0 23873120 8198000  24288    0    0     0     0  270    44  0  0 75 25
 0  4      0 23873136 8198000  24288    0    0     0  4096  272    51  0  0 75 25

which is bad.

Any ideas?

best regards,
Michael Renner

re: Bad SCSI performance with 2.6.14 and Fusion-MPT

[Alan D. Brunelle]

I'm seeing what looks to be the same problem (see: U320 SCSI negotiation 
problem in Linux 2.6.13 and later implementations on LSI Logic / Symbios 
Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 08)). [[I think I 
have a patch (perhaps suspect, but it works for me), if you're interested.]]

Alan D. Brunelle

Re: Bad SCSI performance with 2.6.14 and Fusion-MPT

[Michael Renner]

Alan D. Brunelle wrote:

> I'm seeing what looks to be the same problem (see: U320 SCSI negotiation 
> problem in Linux 2.6.13 and later implementations on LSI Logic / Symbios 
> Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 08)). [[I think I 
> have a patch (perhaps suspect, but it works for me), if you're 
> interested.]]

Yeah, I'd like to see it. Interestingly another box with the same 
hardware setup and very slightly different .config works fine with 2.6.14.

best regards,
michael

Re: Bad SCSI performance with 2.6.14 and Fusion-MPT

[Alan D. Brunelle]

Michael Renner wrote:

> Alan D. Brunelle wrote:
>
>> I'm seeing what looks to be the same problem (see: U320 SCSI 
>> negotiation problem in Linux 2.6.13 and later implementations on LSI 
>> Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI 
>> (rev 08)). [[I think I have a patch (perhaps suspect, but it works 
>> for me), if you're interested.]]
>
>
> Yeah, I'd like to see it. Interestingly another box with the same 
> hardware setup and very slightly different .config works fine with 
> 2.6.14.
>
> best regards,
> michael
>
It doesn't surprise me that a different .config or hardware change would 
affect this. If I understand the problem correctly, then it depends upon 
timing - and changing things (hw or sw) could affect the outcome.

The posting by

The following works for me - but as noted previously, it may be suspect 
in general: I'm concerned about the indefinate delay (theoretical).

diff -ur linux-2.6.14.mm2/drivers/message/fusion/mptbase.c 
linux-2.6.14.mm2.adb/drivers/message/fusion/mptbase.c
--- linux-2.6.14.mm2/drivers/message/fusion/mptbase.c   2005-11-17 
09:17:15.000000000 -0500
+++ linux-2.6.14.mm2.adb/drivers/message/fusion/mptbase.c       
2005-11-17 10:11:52.000000000 -0500
@@ -4994,7 +4994,9 @@
        if ((mf = mpt_get_msg_frame(mpt_base_index, ioc)) == NULL) {
                dcprintk((MYIOC_s_WARN_FMT "mpt_config: no msg frames!\n",
                                ioc->name));
-               return -EAGAIN;
+               do {
+                       msleep(25);
+               } while ((mf = mpt_get_msg_frame(mpt_base_index, ioc)) 
== NULL);
        }
        pReq = (Config_t *)mf;
        pReq->Action = pCfg->action;

RE: Bad SCSI performance with 2.6.14 and Fusion-MPT

[Moore, Eric Dean]

On Thursday, November 17, 2005 7:53 AM,  Alan D. Brunelle wrote:
> It doesn't surprise me that a different .config or hardware 
> change would 
> affect this. If I understand the problem correctly, then it 
> depends upon 
> timing - and changing things (hw or sw) could affect the outcome.
> 
> The posting by
> 
> The following works for me - but as noted previously, it may 
> be suspect 
> in general: I'm concerned about the indefinate delay (theoretical).
> 

Please use the patch I posted last night, apply to 2.6.15-rc1-git4.
It addresses this issue.  

Here is the URL to this patch:

http://marc.theaimsgroup.com/?l=linux-scsi&m=113219282408101&w=2

Eric Moore
LSI Logic

Re: Bad SCSI performance with 2.6.14 and Fusion-MPT

[Alan D. Brunelle]

Moore, Eric Dean wrote:

>On Thursday, November 17, 2005 7:53 AM,  Alan D. Brunelle wrote:
>  
>
>>It doesn't surprise me that a different .config or hardware 
>>change would 
>>affect this. If I understand the problem correctly, then it 
>>depends upon 
>>timing - and changing things (hw or sw) could affect the outcome.
>>
>>The posting by
>>
>>The following works for me - but as noted previously, it may 
>>be suspect 
>>in general: I'm concerned about the indefinate delay (theoretical).
>>
>>    
>>
>
>Please use the patch I posted last night, apply to 2.6.15-rc1-git4.
>It addresses this issue.  
>
>Here is the URL to this patch:
>
>http://marc.theaimsgroup.com/?l=linux-scsi&m=113219282408101&w=2
>
>Eric Moore
>LSI Logic
>
>
>  
>
FYI - I did try it earlier this morning with 2.6.14.2 and it did *not* 
fix the issue. I will see about updating to 2.6.15-rc1-git4, and trying 
that again...

Alan

RE: Bad SCSI performance with 2.6.14 and Fusion-MPT

[Moore, Eric Dean]

[-- Attachment #1: Type: text/plain, Size: 856 bytes --]

On  Thursday, November 17, 2005 10:08 AM, Alan D. Brunelle wrote:
> >Please use the patch I posted last night, apply to 2.6.15-rc1-git4.
> >It addresses this issue.  
> >
> >Here is the URL to this patch:
> >
> >http://marc.theaimsgroup.com/?l=linux-scsi&m=113219282408101&w=2
> >
> >Eric Moore
> >LSI Logic
> >
> >
> >  
> >
> FYI - I did try it earlier this morning with 2.6.14.2 and it 
> did *not* 
> fix the issue. I will see about updating to 2.6.15-rc1-git4, 
> and trying 
> that again...
> 

Ok, that make sense, I see your patch is handling mpt_config.

So with the patch I provided at the link above, can you apply
this patch over that. I've added the sync code in mpt_config.
Please let me know the results.  I also have a tool called 
getspeed that I can send you. It dump's the programmed
speed the devices were negotiated at.

Eric Moore




[-- Attachment #2: 1a-perf-fix.patch --]
[-- Type: application/octet-stream, Size: 1101 bytes --]

diff -uarN b/drivers/message/fusion/mptbase.c a/drivers/message/fusion/mptbase.c
--- b/drivers/message/fusion/mptbase.c	2005-11-16 18:19:11.000000000 -0700
+++ a/drivers/message/fusion/mptbase.c	2005-11-17 10:43:18.000000000 -0700
@@ -5053,11 +5053,20 @@
 		return -EPERM;
 	}
 
+	if(mpt_alt_ioc_wait(ioc)!=0) {
+		dcprintk((MYIOC_s_WARN_FMT "alt_ioc busy!\n",
+		    ioc->name));
+		return -EBUSY;
+	}
+
 	/* Get and Populate a free Frame
 	 */
 	if ((mf = mpt_get_msg_frame(mpt_base_index, ioc)) == NULL) {
 		dcprintk((MYIOC_s_WARN_FMT "mpt_config: no msg frames!\n",
 				ioc->name));
+		spin_lock(&ioc->initializing_hba_lock);
+		ioc->initializing_hba_lock_flag=0;
+		spin_unlock(&ioc->initializing_hba_lock);
 		return -EAGAIN;
 	}
 	pReq = (Config_t *)mf;
@@ -5139,6 +5148,10 @@
 	mpt_put_msg_frame(mpt_base_index, ioc, mf);
 	wait_event(mpt_waitq, pCfg->wait_done);
 
+	spin_lock(&ioc->initializing_hba_lock);
+	ioc->initializing_hba_lock_flag=0;
+	spin_unlock(&ioc->initializing_hba_lock);
+
 	/* mf has been freed - do not access */
 
 	rc = pCfg->status;

Re: Bad SCSI performance with 2.6.14 and Fusion-MPT

[Alan D. Brunelle]

Moore, Eric Dean wrote:

>On  Thursday, November 17, 2005 10:08 AM, Alan D. Brunelle wrote:
>  
>
>>>Please use the patch I posted last night, apply to 2.6.15-rc1-git4.
>>>It addresses this issue.  
>>>
>>>Here is the URL to this patch:
>>>
>>>http://marc.theaimsgroup.com/?l=linux-scsi&m=113219282408101&w=2
>>>
>>>Eric Moore
>>>LSI Logic
>>>
>>>
>>> 
>>>
>>>      
>>>
>>FYI - I did try it earlier this morning with 2.6.14.2 and it 
>>did *not* 
>>fix the issue. I will see about updating to 2.6.15-rc1-git4, 
>>and trying 
>>that again...
>>
>>    
>>
>
>Ok, that make sense, I see your patch is handling mpt_config.
>
>So with the patch I provided at the link above, can you apply
>this patch over that. I've added the sync code in mpt_config.
>Please let me know the results.  I also have a tool called 
>getspeed that I can send you. It dump's the programmed
>speed the devices were negotiated at.
>
>Eric Moore
>
>
>
>  
>
Hi Eric -

I just added in this patch on top of the previous one (again, on 
2.6.14.2) and it worked just fine - I'm getting full U320 16-bit 
transfers. I use

% /usr/bin/sginfo -t 0x19,0x3 -Xz <dev>

to determine this - prior to these patches I would see some of these:

/dev/sde  255 0 0 0 2 0 0

Now I see:

/dev/sde  8 127 1 87 2 1 3

Which is *much* better!

Thanks!
Alan
PS. Expanded -

SPI-4 Negotiated Settings mode subpage (0x19,0x3)
--------------------------------------------
Transfer period                    8
REQ/ACK offset                     127
Transfer width exponent            1
Protocol option bits               87
Transciever mode                   2
Sent PCOMP_EN                      1
Received PCOMP_EN                  3

RE: Bad SCSI performance with 2.6.14 and Fusion-MPT

[Moore, Eric Dean]

On Thursday, November 17, 2005 11:24 AM, Alan D. Brunelle wrote:

> Which is *much* better!
> 
> Thanks!
> Alan

Thanks so much.  I will repost later today with a patch, that will
apply over the last patch[7/7] from yesterday.   I'm busy  in the lab
at the moment. Before I leave the office tonight, I will repost it.

Eric

Re: Bad SCSI performance with 2.6.14 and Fusion-MPT

[Douglas Gilbert]

Alan D. Brunelle wrote:
> Moore, Eric Dean wrote:
> 
>> On  Thursday, November 17, 2005 10:08 AM, Alan D. Brunelle wrote:
>>  
>>
>>>> Please use the patch I posted last night, apply to 2.6.15-rc1-git4.
>>>> It addresses this issue. 
>>>> Here is the URL to this patch:
>>>>
>>>> http://marc.theaimsgroup.com/?l=linux-scsi&m=113219282408101&w=2
>>>>
>>>> Eric Moore
>>>> LSI Logic
>>>>
>>>>
>>>>
>>>>
>>>>     
>>>
>>> FYI - I did try it earlier this morning with 2.6.14.2 and it did
>>> *not* fix the issue. I will see about updating to 2.6.15-rc1-git4,
>>> and trying that again...
>>>
>>>   
>>
>>
>> Ok, that make sense, I see your patch is handling mpt_config.
>>
>> So with the patch I provided at the link above, can you apply
>> this patch over that. I've added the sync code in mpt_config.
>> Please let me know the results.  I also have a tool called getspeed
>> that I can send you. It dump's the programmed
>> speed the devices were negotiated at.
>>
>> Eric Moore
>>
>>
>>
>>  
>>
> Hi Eric -
> 
> I just added in this patch on top of the previous one (again, on
> 2.6.14.2) and it worked just fine - I'm getting full U320 16-bit
> transfers. I use
> 
> % /usr/bin/sginfo -t 0x19,0x3 -Xz <dev>
> 
> to determine this - prior to these patches I would see some of these:
> 
> /dev/sde  255 0 0 0 2 0 0
> 
> Now I see:
> 
> /dev/sde  8 127 1 87 2 1 3
> 
> Which is *much* better!
> 
> Thanks!
> Alan
> PS. Expanded -
> 
> SPI-4 Negotiated Settings mode subpage (0x19,0x3)
> --------------------------------------------
> Transfer period                    8
> REQ/ACK offset                     127
> Transfer width exponent            1
> Protocol option bits               87
> Transciever mode                   2
> Sent PCOMP_EN                      1
> Received PCOMP_EN                  3

Alan,
Good to see that sginfo is finding some use. I'm trying
to move most of sginfo's functionality into sdparm.
The sginfo invocation is a little less cryptic in sdparm:

 # sdparm -t spi -p ns -l /dev/sdd
    /dev/sdd: SEAGATE   ST373453LC        XXXX
port: negotiated settings (SPI) mode page:
  PPID_3      1  [cha: n, def:  1, sav:  1]  Port's (transport) protocol identifier
  TPF         8  [cha: n, def:  0, sav:  0]  Transfer period factor
  RAO        63  [cha: n, def:  0, sav:  0]  REQ/ACK offset
  TWE         1  [cha: n, def:  0, sav:  0]  Transfer width exponent
  POB        23  [cha: n, def:  0, sav:  0]  Protocol option bits
  TM          2  [cha: n, def:  0, sav:  0]  Transceiver mode
  SPE         1  [cha: n, def:  0, sav:  0]  Sent PCOMP_EN bit (for current I_T nexus)
  RPE         1  [cha: n, def:  0, sav:  0]  Received PCOMP_EN bit (for current I_T nexus)

The long hand invocation would be:
 # sdparm --transport=spi --page=ns --long /dev/sdd

Parameters can can fetched separately with sdparm:
 # sdparm -t spi -q --get=TPF,RAO /dev/sdd
TPF         8  [cha: n, def:  0, sav:  0]
RAO        63  [cha: n, def:  0, sav:  0]

sdparm is a lot easier to use than sginfo when in comes
to changing mode parameters (not that anything in the
above mode subpage can be changed). I will continue to
maintain sginfo, adding new mode pages and parameters
to both sdparm and sginfo.


BTW I recently fixed the sginfo bug that required "0x" to
appear additionally in front of the subpage number.
So this now works:
 # sginfo -t 0x19,3 -Xz /dev/sdd
8 63 1 23 2 1 1
in the soon to be released sg3_utils 1.18 .

Doug Gilbert