From: Douglas Gilbert <dgilbert@interlog.com>
To: "Moore, Eric" <Eric.Moore@lsi.com>
Cc: Louis-David Mitterrand <vindex+lists-linux-scsi@apartia.org>,
"Desai, Kashyap" <Kashyap.Desai@lsi.com>,
"Prakash, Sathya" <Sathya.Prakash@lsi.com>,
"linux-scsi@vger.kernel.org" <linux-scsi@vger.kernel.org>
Subject: Re: caviar black + mpt2sas horrific performance
Date: Sun, 03 Apr 2011 13:35:24 -0400 [thread overview]
Message-ID: <4D98AFDC.3010401@interlog.com> (raw)
In-Reply-To: <4565AEA676113A449269C2F3A549520F808E4AB6@cosmail03.lsi.com>
On 11-04-02 12:51 AM, Moore, Eric wrote:
> On Thursday, March 31, 2011 9:35 AM, Louis-David Mitterrand wrote:
>> On Thu, Mar 31, 2011 at 09:07:05AM -0600, Moore, Eric wrote:
>>> On Wednesday, March 30, 2011 2:17 PM, Louis-David Mitterrand wrote:
>>>> On Wed, Mar 30, 2011 at 02:02:12PM -0600, Moore, Eric wrote:
>>>>> Hi Louis, can you send me your /var/log/messages and dmesg output?
>>>>> Please capture after you've run your test.
>>>>
>>>> Hi,
>>>>
>>>> My test produces no specific output through dmesg or /var/log/messages
>>>> as there is no apparent error from the controller.
>>>>
>>>> However I can send you the syslog startup output. Let me know if I can
>>>> run any further tests or provide more information.
>>>>
>>>
>>> need you to set the mpt2sas logging level to 0x310 just before you run
>> you test, then capture the logs after.
>>> Please make sure your syslog level is set to capture all KERN_XXX levels
>> to include KERN_DEBUG.
>>>
>>> Example:
>>> # echo 0x310> /sys/module/mpt2sas/parameters/logging_level
>>
>> Hi,
>>
>> Here is the output:
>>
>> Mar 31 17:07:50 zenon kernel: setting logging_level(0x00000310)
>> Mar 31 17:24:29 zenon kernel: sd 0:0:0:0: [sda] CDB: ATA command pass
>> through(16): 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
>> Mar 31 17:24:29 zenon kernel: mpt2sas0:
>> sas_address(0x4433221103000000), phy(3)
>> Mar 31 17:24:29 zenon kernel: mpt2sas0:
>> enclosure_logical_id(0x5842b2b05020c600), slot(0)
>> Mar 31 17:24:29 zenon kernel: mpt2sas0: handle(0x0009),
>> ioc_status(success)(0x0000), smid(1344)
>> Mar 31 17:24:29 zenon kernel: mpt2sas0: request_len(0), underflow(0),
>> resid(0)
>> Mar 31 17:24:29 zenon kernel: mpt2sas0: tag(0), transfer_count(0), sc-
>>> result(0x00000002)
>> Mar 31 17:24:29 zenon kernel: mpt2sas0: scsi_status(check
>> condition)(0x02), scsi_state(autosense valid )(0x01)
>> Mar 31 17:24:29 zenon kernel: mpt2sas0: [sense_key,asc,ascq]:
>> [0x01,0x00,0x1d], count(22)
>> Mar 31 17:24:29 zenon kernel: sd 0:0:0:0: [sda] CDB: ATA command pass
>> through(16): 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
>> Mar 31 17:24:29 zenon kernel: mpt2sas0:
>> sas_address(0x4433221103000000), phy(3)
>> Mar 31 17:24:29 zenon kernel: mpt2sas0:
>> enclosure_logical_id(0x5842b2b05020c600), slot(0)
>> Mar 31 17:24:29 zenon kernel: mpt2sas0: handle(0x0009),
>> ioc_status(success)(0x0000), smid(884)
>> Mar 31 17:24:29 zenon kernel: mpt2sas0: request_len(0), underflow(0),
>> resid(0)
>> Mar 31 17:24:29 zenon kernel: mpt2sas0: tag(0), transfer_count(0), sc-
>>> result(0x00000002)
>> Mar 31 17:24:29 zenon kernel: mpt2sas0: scsi_status(check
>> condition)(0x02), scsi_state(autosense valid )(0x01)
>> Mar 31 17:24:29 zenon kernel: mpt2sas0: [sense_key,asc,ascq]:
>> [0x01,0x00,0x1d], count(22)
>> Mar 31 17:24:30 zenon kernel: sd 0:0:1:0: [sdb] CDB: ATA command pass
>> through(16): 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
>> Mar 31 17:24:30 zenon kernel: mpt2sas0:
>> sas_address(0x4433221102000000), phy(2)
>> Mar 31 17:24:30 zenon kernel: mpt2sas0:
>> enclosure_logical_id(0x5842b2b05020c600), slot(1)
>> Mar 31 17:24:30 zenon kernel: mpt2sas0: handle(0x000a),
>> ioc_status(success)(0x0000), smid(100)
>> Mar 31 17:24:30 zenon kernel: mpt2sas0: request_len(0), underflow(0),
>> resid(0)
>> Mar 31 17:24:30 zenon kernel: mpt2sas0: tag(0), transfer_count(0), sc-
>>> result(0x00000002)
>> Mar 31 17:24:30 zenon kernel: mpt2sas0: scsi_status(check
>> condition)(0x02), scsi_state(autosense valid )(0x01)
>> Mar 31 17:24:30 zenon kernel: mpt2sas0: [sense_key,asc,ascq]:
>> [0x01,0x00,0x1d], count(22)
>> Mar 31 17:24:30 zenon kernel: sd 0:0:1:0: [sdb] CDB: ATA command pass
>> through(16): 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
>> Mar 31 17:24:30 zenon kernel: mpt2sas0:
>> sas_address(0x4433221102000000), phy(2)
>> Mar 31 17:24:30 zenon kernel: mpt2sas0:
>> enclosure_logical_id(0x5842b2b05020c600), slot(1)
>> Mar 31 17:24:30 zenon kernel: mpt2sas0: handle(0x000a),
>> ioc_status(success)(0x0000), smid(510)
>> Mar 31 17:24:30 zenon kernel: mpt2sas0: request_len(0), underflow(0),
>> resid(0)
>> Mar 31 17:24:30 zenon kernel: mpt2sas0: tag(0), transfer_count(0), sc-
>>> result(0x00000002)
>> Mar 31 17:24:30 zenon kernel: mpt2sas0: scsi_status(check
>> condition)(0x02), scsi_state(autosense valid )(0x01)
>> Mar 31 17:24:30 zenon kernel: mpt2sas0: [sense_key,asc,ascq]:
>> [0x01,0x00,0x1d], count(22)
>> Mar 31 17:24:30 zenon kernel: sd 0:0:2:0: [sdc] CDB: ATA command pass
>> through(16): 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
>> Mar 31 17:24:30 zenon kernel: mpt2sas0:
>> sas_address(0x4433221101000000), phy(1)
>> Mar 31 17:24:30 zenon kernel: mpt2sas0:
>> enclosure_logical_id(0x5842b2b05020c600), slot(2)
>> Mar 31 17:24:30 zenon kernel: mpt2sas0: handle(0x000b),
>> ioc_status(success)(0x0000), smid(1455)
>> Mar 31 17:24:30 zenon kernel: mpt2sas0: request_len(0), underflow(0),
>> resid(0)
>> Mar 31 17:24:30 zenon kernel: mpt2sas0: tag(0), transfer_count(0), sc-
>>> result(0x00000002)
>> Mar 31 17:24:30 zenon kernel: mpt2sas0: scsi_status(check
>> condition)(0x02), scsi_state(autosense valid )(0x01)
>> Mar 31 17:24:30 zenon kernel: mpt2sas0: [sense_key,asc,ascq]:
>> [0x01,0x00,0x1d], count(22)
>> Mar 31 17:24:31 zenon kernel: sd 0:0:2:0: [sdc] CDB: ATA command pass
>> through(16): 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
>> Mar 31 17:24:31 zenon kernel: mpt2sas0:
>> sas_address(0x4433221101000000), phy(1)
>> Mar 31 17:24:31 zenon kernel: mpt2sas0:
>> enclosure_logical_id(0x5842b2b05020c600), slot(2)
>> Mar 31 17:24:31 zenon kernel: mpt2sas0: handle(0x000b),
>> ioc_status(success)(0x0000), smid(966)
>> Mar 31 17:24:31 zenon kernel: mpt2sas0: request_len(0), underflow(0),
>> resid(0)
>> Mar 31 17:24:31 zenon kernel: mpt2sas0: tag(0), transfer_count(0), sc-
>>> result(0x00000002)
>> Mar 31 17:24:31 zenon kernel: mpt2sas0: scsi_status(check
>> condition)(0x02), scsi_state(autosense valid )(0x01)
>> Mar 31 17:24:31 zenon kernel: mpt2sas0: [sense_key,asc,ascq]:
>> [0x01,0x00,0x1d], count(22)
>> Mar 31 17:24:31 zenon kernel: sd 0:0:3:0: [sdd] CDB: ATA command pass
>> through(16): 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
>> Mar 31 17:24:31 zenon kernel: mpt2sas0:
>> sas_address(0x4433221100000000), phy(0)
>> Mar 31 17:24:31 zenon kernel: mpt2sas0:
>> enclosure_logical_id(0x5842b2b05020c600), slot(3)
>> Mar 31 17:24:31 zenon kernel: mpt2sas0: handle(0x000c),
>> ioc_status(success)(0x0000), smid(519)
>> Mar 31 17:24:31 zenon kernel: mpt2sas0: request_len(0), underflow(0),
>> resid(0)
>> Mar 31 17:24:31 zenon kernel: mpt2sas0: tag(0), transfer_count(0), sc-
>>> result(0x00000002)
>> Mar 31 17:24:31 zenon kernel: mpt2sas0: scsi_status(check
>> condition)(0x02), scsi_state(autosense valid )(0x01)
>> Mar 31 17:24:31 zenon kernel: mpt2sas0: [sense_key,asc,ascq]:
>> [0x01,0x00,0x1d], count(22)
>> Mar 31 17:24:32 zenon kernel: sd 0:0:3:0: [sdd] CDB: ATA command pass
>> through(16): 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
>> Mar 31 17:24:32 zenon kernel: mpt2sas0:
>> sas_address(0x4433221100000000), phy(0)
>> Mar 31 17:24:32 zenon kernel: mpt2sas0:
>> enclosure_logical_id(0x5842b2b05020c600), slot(3)
>> Mar 31 17:24:32 zenon kernel: mpt2sas0: handle(0x000c),
>> ioc_status(success)(0x0000), smid(1125)
>> Mar 31 17:24:32 zenon kernel: mpt2sas0: request_len(0), underflow(0),
>> resid(0)
>> Mar 31 17:24:32 zenon kernel: mpt2sas0: tag(0), transfer_count(0), sc-
>>> result(0x00000002)
>> Mar 31 17:24:32 zenon kernel: mpt2sas0: scsi_status(check
>> condition)(0x02), scsi_state(autosense valid )(0x01)
>> Mar 31 17:24:32 zenon kernel: mpt2sas0: [sense_key,asc,ascq]:
>> [0x01,0x00,0x1d], count(22)
>> Mar 31 17:24:32 zenon kernel: sd 0:0:4:0: [sde] CDB: ATA command pass
>> through(16): 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
>> Mar 31 17:24:32 zenon kernel: mpt2sas0:
>> sas_address(0x4433221104000000), phy(4)
>> Mar 31 17:24:32 zenon kernel: mpt2sas0:
>> enclosure_logical_id(0x5842b2b05020c600), slot(7)
>> Mar 31 17:24:32 zenon kernel: mpt2sas0: handle(0x000d),
>> ioc_status(success)(0x0000), smid(570)
>> Mar 31 17:24:32 zenon kernel: mpt2sas0: request_len(0), underflow(0),
>> resid(0)
>> Mar 31 17:24:32 zenon kernel: mpt2sas0: tag(0), transfer_count(0), sc-
>>> result(0x00000002)
>> Mar 31 17:24:32 zenon kernel: mpt2sas0: scsi_status(check
>> condition)(0x02), scsi_state(autosense valid )(0x01)
>> Mar 31 17:24:32 zenon kernel: mpt2sas0: [sense_key,asc,ascq]:
>> [0x01,0x00,0x1d], count(22)
>> Mar 31 17:24:32 zenon kernel: sd 0:0:4:0: [sde] CDB: ATA command pass
>> through(16): 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
>> Mar 31 17:24:32 zenon kernel: mpt2sas0:
>> sas_address(0x4433221104000000), phy(4)
>> Mar 31 17:24:32 zenon kernel: mpt2sas0:
>> enclosure_logical_id(0x5842b2b05020c600), slot(7)
>> Mar 31 17:24:32 zenon kernel: mpt2sas0: handle(0x000d),
>> ioc_status(success)(0x0000), smid(28)
>> Mar 31 17:24:32 zenon kernel: mpt2sas0: request_len(0), underflow(0),
>> resid(0)
>> Mar 31 17:24:32 zenon kernel: mpt2sas0: tag(0), transfer_count(0), sc-
>>> result(0x00000002)
>> Mar 31 17:24:32 zenon kernel: mpt2sas0: scsi_status(check
>> condition)(0x02), scsi_state(autosense valid )(0x01)
>> Mar 31 17:24:32 zenon kernel: mpt2sas0: [sense_key,asc,ascq]:
>> [0x01,0x00,0x1d], count(22)
>> Mar 31 17:24:33 zenon kernel: sd 0:0:5:0: [sdf] CDB: ATA command pass
>> through(16): 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
>> Mar 31 17:24:33 zenon kernel: mpt2sas0:
>> sas_address(0x4433221107000000), phy(7)
>> Mar 31 17:24:33 zenon kernel: mpt2sas0:
>> enclosure_logical_id(0x5842b2b05020c600), slot(4)
>> Mar 31 17:24:33 zenon kernel: mpt2sas0: handle(0x000e),
>> ioc_status(success)(0x0000), smid(471)
>> Mar 31 17:24:33 zenon kernel: mpt2sas0: request_len(0), underflow(0),
>> resid(0)
>> Mar 31 17:24:33 zenon kernel: mpt2sas0: tag(0), transfer_count(0), sc-
>>> result(0x00000002)
>> Mar 31 17:24:33 zenon kernel: mpt2sas0: scsi_status(check
>> condition)(0x02), scsi_state(autosense valid )(0x01)
>> Mar 31 17:24:33 zenon kernel: mpt2sas0: [sense_key,asc,ascq]:
>> [0x01,0x00,0x1d], count(22)
>> Mar 31 17:24:33 zenon kernel: sd 0:0:5:0: [sdf] CDB: ATA command pass
>> through(16): 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
>> Mar 31 17:24:33 zenon kernel: mpt2sas0:
>> sas_address(0x4433221107000000), phy(7)
>> Mar 31 17:24:33 zenon kernel: mpt2sas0:
>> enclosure_logical_id(0x5842b2b05020c600), slot(4)
>> Mar 31 17:24:33 zenon kernel: mpt2sas0: handle(0x000e),
>> ioc_status(success)(0x0000), smid(789)
>> Mar 31 17:24:33 zenon kernel: mpt2sas0: request_len(0), underflow(0),
>> resid(0)
>> Mar 31 17:24:33 zenon kernel: mpt2sas0: tag(0), transfer_count(0), sc-
>>> result(0x00000002)
>> Mar 31 17:24:33 zenon kernel: mpt2sas0: scsi_status(check
>> condition)(0x02), scsi_state(autosense valid )(0x01)
>> Mar 31 17:24:33 zenon kernel: mpt2sas0: [sense_key,asc,ascq]:
>> [0x01,0x00,0x1d], count(22)
>> Mar 31 17:24:34 zenon kernel: sd 0:0:6:0: [sdg] CDB: ATA command pass
>> through(16): 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
>> Mar 31 17:24:34 zenon kernel: mpt2sas0:
>> sas_address(0x4433221106000000), phy(6)
>> Mar 31 17:24:34 zenon kernel: mpt2sas0:
>> enclosure_logical_id(0x5842b2b05020c600), slot(5)
>> Mar 31 17:24:34 zenon kernel: mpt2sas0: handle(0x000f),
>> ioc_status(success)(0x0000), smid(594)
>> Mar 31 17:24:34 zenon kernel: mpt2sas0: request_len(0), underflow(0),
>> resid(0)
>> Mar 31 17:24:34 zenon kernel: mpt2sas0: tag(0), transfer_count(0), sc-
>>> result(0x00000002)
>> Mar 31 17:24:34 zenon kernel: mpt2sas0: scsi_status(check
>> condition)(0x02), scsi_state(autosense valid )(0x01)
>> Mar 31 17:24:34 zenon kernel: mpt2sas0: [sense_key,asc,ascq]:
>> [0x01,0x00,0x1d], count(22)
>> Mar 31 17:24:34 zenon kernel: sd 0:0:6:0: [sdg] CDB: ATA command pass
>> through(16): 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
>> Mar 31 17:24:34 zenon kernel: mpt2sas0:
>> sas_address(0x4433221106000000), phy(6)
>> Mar 31 17:24:34 zenon kernel: mpt2sas0:
>> enclosure_logical_id(0x5842b2b05020c600), slot(5)
>> Mar 31 17:24:34 zenon kernel: mpt2sas0: handle(0x000f),
>> ioc_status(success)(0x0000), smid(761)
>> Mar 31 17:24:34 zenon kernel: mpt2sas0: request_len(0), underflow(0),
>> resid(0)
>> Mar 31 17:24:34 zenon kernel: mpt2sas0: tag(0), transfer_count(0), sc-
>>> result(0x00000002)
>> Mar 31 17:24:34 zenon kernel: mpt2sas0: scsi_status(check
>> condition)(0x02), scsi_state(autosense valid )(0x01)
>> Mar 31 17:24:34 zenon kernel: mpt2sas0: [sense_key,asc,ascq]:
>> [0x01,0x00,0x1d], count(22)
>> Mar 31 17:24:35 zenon kernel: sd 0:0:7:0: [sdh] CDB: ATA command pass
>> through(16): 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
>> Mar 31 17:24:35 zenon kernel: mpt2sas0:
>> sas_address(0x4433221105000000), phy(5)
>> Mar 31 17:24:35 zenon kernel: mpt2sas0:
>> enclosure_logical_id(0x5842b2b05020c600), slot(6)
>> Mar 31 17:24:35 zenon kernel: mpt2sas0: handle(0x0010),
>> ioc_status(success)(0x0000), smid(1127)
>> Mar 31 17:24:35 zenon kernel: mpt2sas0: request_len(0), underflow(0),
>> resid(0)
>> Mar 31 17:24:35 zenon kernel: mpt2sas0: tag(0), transfer_count(0), sc-
>>> result(0x00000002)
>> Mar 31 17:24:35 zenon kernel: mpt2sas0: scsi_status(check
>> condition)(0x02), scsi_state(autosense valid )(0x01)
>> Mar 31 17:24:35 zenon kernel: mpt2sas0: [sense_key,asc,ascq]:
>> [0x01,0x00,0x1d], count(22)
>> Mar 31 17:24:35 zenon kernel: sd 0:0:7:0: [sdh] CDB: ATA command pass
>> through(16): 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
>> Mar 31 17:24:35 zenon kernel: mpt2sas0:
>> sas_address(0x4433221105000000), phy(5)
>> Mar 31 17:24:35 zenon kernel: mpt2sas0:
>> enclosure_logical_id(0x5842b2b05020c600), slot(6)
>> Mar 31 17:24:35 zenon kernel: mpt2sas0: handle(0x0010),
>> ioc_status(success)(0x0000), smid(1583)
>> Mar 31 17:24:35 zenon kernel: mpt2sas0: request_len(0), underflow(0),
>> resid(0)
>> Mar 31 17:24:35 zenon kernel: mpt2sas0: tag(0), transfer_count(0), sc-
>>> result(0x00000002)
>> Mar 31 17:24:35 zenon kernel: mpt2sas0: scsi_status(check
>> condition)(0x02), scsi_state(autosense valid )(0x01)
>> Mar 31 17:24:35 zenon kernel: mpt2sas0: [sense_key,asc,ascq]:
>> [0x01,0x00,0x1d], count(22)
>
>
> Well you've got a load of SATA passthru request with RECOVERED_ERROR sense key is probably effecting your performance.
>
> Any chance you could try shutdown the daemons that are sending those request? I doubt its coming via the filesystem and the test your running.
The daemon in question is probably smartd. And RECOVERED
ERROR is the expected sense key when the CK_COND bit is
set in the SCSI ATA PASS-THROUGH command. It requests the
ATA registers after the ATA device has executed the related
command. And if that causes a significant delay in the
mpt2sas driver then IMO that is a bug in the driver.
One piece of additional information that may be useful is
whether fixed or descriptor format sense data is returned.
With SAT-1 only descriptor format was allowed. In SAT-2
(now a standard) it can be fixed or descriptor format with
the preference seeming to be with the former. My tools and
smartmontools (smartd) cannot handle fixed format sense
data from SCSI ATA PASS-THROUGH commands.
Doug Gilbert
next prev parent reply other threads:[~2011-04-03 17:35 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-03-30 8:01 caviar black + mpt2sas horrific performance Louis-David Mitterrand
[not found] ` <4565AEA676113A449269C2F3A549520F8086471A@cosmail03.lsi.com>
2011-03-30 20:17 ` Louis-David Mitterrand
2011-03-31 15:07 ` Moore, Eric
2011-03-31 15:34 ` Louis-David Mitterrand
2011-04-02 4:51 ` Moore, Eric
2011-04-03 17:35 ` Douglas Gilbert [this message]
2011-04-03 23:05 ` Moore, Eric
2011-04-29 14:06 ` Louis-David Mitterrand
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4D98AFDC.3010401@interlog.com \
--to=dgilbert@interlog.com \
--cc=Eric.Moore@lsi.com \
--cc=Kashyap.Desai@lsi.com \
--cc=Sathya.Prakash@lsi.com \
--cc=linux-scsi@vger.kernel.org \
--cc=vindex+lists-linux-scsi@apartia.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.