Re: SRPT and SCST

Re: SRPT and SCST

[Philip Pokorny]

Chris Worley asked that we post this to scst and linux-rdma lists for 
discussion.

We're trying to get IB SRPT working and can't seem to get a stable 
configuration using any of the various SCST, IB_SRPT, and kernel/distro 
versions out there.  In most cases, we're able to crash the connection 
and typically the target within minutes of pounding by 4 initiators 
doing "mkfs.ext3", "tar xf" and "fsck" to the SRP block device.

Our target is a Penguin Computing Altus 2704 with disk expansion 
chassis.  That's a 4-socket AMD hex-core (24 total cores) with 128GB of 
memory and 24 1TB drives attached to two LSI 1068 SAS controllers. (aka 
3801E)  The drives are configured as 12 RAID-1 mirrors and 3-wide LVM 
stripes over those mirrors.  There are an additional 6 SSD's in the 
server in a "fast" VG also RAID-1 mirrored and LVM striped.  Read ahead 
is disabled on the LVM volumes.

LVM volumes are exported via SCST as FILEIO block devices to initiators. 
  50 groups are defined with two LVM volumes/block devices per group. 
One initiator per group. (NODE GUID added to "names" in the group)

With only 4 initiators, almost 100% of I/O is to RAM and no disk I/O is 
seen on the target.

Performance (when it's working) is generally good at 800MB/sec 
aggregate, but we'd like to see better.  It appeared we were getting 
1.3GB/s at one point.

On Wed, Nov 4, 2009 at 5:34 PM, Philip Pokorny
<ppokorny-pabcTyWEv4ZW60MLeMDbCVaTQe2KTcn/@public.gmane.org> wrote:
> We got a serial console attached and ran a test using the SCST and IB_SRPT
> versions that you recommended (Arend set it up so I'll defer to him on the
> exact SVN checkout that he used).
>
>> What sort of crashes are you seeing?  I also have a customer
>> experiencing a crash, but I can't get details out of them.
>
> The client gets SCSI I/O errors and aborts the filesystem (putting it in
> read-only mode).
>
> After about 400 seconds of testing, the server side logs the following:
>
> [ 8418.697830] <6>[12426]: scst_check_sense:2444:Clearing dbl_ua_possible
> flag (dev ffff811816136000, cmd ffff81081017c1c8)
> [ 8418.697836] <6>[12426]: scst_dec_on_dev_cmd:577:cmd ffff81081017c1c8 (tag
> 17): unblocking dev ffff811816136000
> [ 8418.697843] <6>[0]: scst_unblock_dev:4653:Device UNBLOCK(new 0), dev
> ffff811816136000
> [ 8864.258468] ib_mthca 0000:81:00.0: SQ 000405 full (999320 head, 997272
> tail, 2048 max, 0 nreq)
> [ 8864.294450] ***ERROR***: srpt_xfer_data[2374] ret=-12
> [ 8864.326702] <6>[0]: scst_queue_retry_cmd:1099:TGT QUEUE FULL:
> incrementing retry_cmds 1
> [ 8864.326709] <6>[0]: scst_queue_retry_cmd:1106:Some command(s) finished,
> direct retry (finished_cmds=2023031, tgt->finished_cmds=2023137,
> retry_cmds=0)
> [ 8878.447081] ib_mthca 0000:81:00.0: SQ 000406 full (1080498 head, 1078450
> tail, 2048 max, 0 nreq)
> [ 8878.484452] ***ERROR***: srpt_xfer_data[2374] ret=-12
> [ 8878.517595] <6>[0]: scst_queue_retry_cmd:1099:TGT QUEUE FULL:
> incrementing retry_cmds 1
> [ 8878.517608] <6>[0]: scst_queue_retry_cmd:1106:Some command(s) finished,
> direct retry (finished_cmds=2256307, tgt->finished_cmds=2256504,
> retry_cmds=0)
> [ 8882.694684] ib_mthca 0000:81:00.0: SQ 000404 full (1087484 head, 1085436
> tail, 2048 max, 0 nreq)
> [ 8882.732542] ***ERROR***: srpt_xfer_data[2374] ret=-12
> [ 8882.766396] <6>[0]: scst_queue_retry_cmd:1099:TGT QUEUE FULL:
> incrementing retry_cmds 1
> [ 8882.766403] <6>[0]: scst_queue_retry_cmd:1106:Some command(s) finished,
> direct retry (finished_cmds=2310445, tgt->finished_cmds=2310539,
> retry_cmds=0)
> [ 8891.650890] ib_mthca 0000:81:00.0: SQ 000407 full (1155377 head, 1153329
> tail, 2048 max, 0 nreq)
> [ 8891.689016] ***ERROR***: srpt_xfer_data[2374] ret=-12
> [ 8891.723548] <6>[0]: scst_queue_retry_cmd:1099:TGT QUEUE FULL:
> incrementing retry_cmds 1
> [ 8891.723556] <6>[0]: scst_queue_retry_cmd:1106:Some command(s) finished,
> direct retry (finished_cmds=2381910, tgt->finished_cmds=2382001,
> retry_cmds=0)
> [ 8891.723573] ib_mthca 0000:81:00.0: too many gathers
> [ 8891.758000] ***ERROR***: srpt_xfer_data[2374] ret=-22
> [ 8891.792888] <6>[0]: scst: scst_rdy_to_xfer:985:***ERROR***: Target driver
> ib_srpt rdy_to_xfer() returned fatal error
>
> I hope that helps.
>
> I've seen that same "rdy_to_xfer() returned fatal error several times in
> different configurations.  The screen shots we sent earlier had the same
> "ib_mthca ... SQ ... full (xx head..." message at the start.  So that seems
> to be related as well.
>
> Thanks for the help,
> Phil P.
>
> --
> Philip Pokorny, RHCE
> Chief Hardware Architect - Penguin Computing
> Voice: 415-370-0835  Toll free: 888-PENGUIN
> www.penguincomputing.com
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SRPT and SCST

[Vladislav Bolkhovitin]

Philip Pokorny, on 11/05/2009 11:51 AM wrote:
> Chris Worley asked that we post this to scst and linux-rdma lists for 
> discussion.
> 
> We're trying to get IB SRPT working and can't seem to get a stable 
> configuration using any of the various SCST, IB_SRPT, and kernel/distro 
> versions out there.  In most cases, we're able to crash the connection 
> and typically the target within minutes of pounding by 4 initiators 
> doing "mkfs.ext3", "tar xf" and "fsck" to the SRP block device.
> 
> Our target is a Penguin Computing Altus 2704 with disk expansion 
> chassis.  That's a 4-socket AMD hex-core (24 total cores) with 128GB of 
> memory and 24 1TB drives attached to two LSI 1068 SAS controllers. (aka 
> 3801E)  The drives are configured as 12 RAID-1 mirrors and 3-wide LVM 
> stripes over those mirrors.  There are an additional 6 SSD's in the 
> server in a "fast" VG also RAID-1 mirrored and LVM striped.  Read ahead 
> is disabled on the LVM volumes.
> 
> LVM volumes are exported via SCST as FILEIO block devices to initiators. 
>   50 groups are defined with two LVM volumes/block devices per group. 
> One initiator per group. (NODE GUID added to "names" in the group)
> 
> With only 4 initiators, almost 100% of I/O is to RAM and no disk I/O is 
> seen on the target.
> 
> Performance (when it's working) is generally good at 800MB/sec 
> aggregate, but we'd like to see better.  It appeared we were getting 
> 1.3GB/s at one point.
> 
> On Wed, Nov 4, 2009 at 5:34 PM, Philip Pokorny
> <ppokorny-pabcTyWEv4ZW60MLeMDbCVaTQe2KTcn/@public.gmane.org> wrote:
>> We got a serial console attached and ran a test using the SCST and IB_SRPT
>> versions that you recommended (Arend set it up so I'll defer to him on the
>> exact SVN checkout that he used).
>>
>>> What sort of crashes are you seeing?  I also have a customer
>>> experiencing a crash, but I can't get details out of them.
>> The client gets SCSI I/O errors and aborts the filesystem (putting it in
>> read-only mode).
>>
>> After about 400 seconds of testing, the server side logs the following:
>>
>> [ 8418.697830] <6>[12426]: scst_check_sense:2444:Clearing dbl_ua_possible
>> flag (dev ffff811816136000, cmd ffff81081017c1c8)
>> [ 8418.697836] <6>[12426]: scst_dec_on_dev_cmd:577:cmd ffff81081017c1c8 (tag
>> 17): unblocking dev ffff811816136000
>> [ 8418.697843] <6>[0]: scst_unblock_dev:4653:Device UNBLOCK(new 0), dev
>> ffff811816136000
>> [ 8864.258468] ib_mthca 0000:81:00.0: SQ 000405 full (999320 head, 997272
>> tail, 2048 max, 0 nreq)
>> [ 8864.294450] ***ERROR***: srpt_xfer_data[2374] ret=-12
>> [ 8864.326702] <6>[0]: scst_queue_retry_cmd:1099:TGT QUEUE FULL:
>> incrementing retry_cmds 1
>> [ 8864.326709] <6>[0]: scst_queue_retry_cmd:1106:Some command(s) finished,
>> direct retry (finished_cmds=2023031, tgt->finished_cmds=2023137,
>> retry_cmds=0)
>> [ 8878.447081] ib_mthca 0000:81:00.0: SQ 000406 full (1080498 head, 1078450
>> tail, 2048 max, 0 nreq)
>> [ 8878.484452] ***ERROR***: srpt_xfer_data[2374] ret=-12
>> [ 8878.517595] <6>[0]: scst_queue_retry_cmd:1099:TGT QUEUE FULL:
>> incrementing retry_cmds 1
>> [ 8878.517608] <6>[0]: scst_queue_retry_cmd:1106:Some command(s) finished,
>> direct retry (finished_cmds=2256307, tgt->finished_cmds=2256504,
>> retry_cmds=0)
>> [ 8882.694684] ib_mthca 0000:81:00.0: SQ 000404 full (1087484 head, 1085436
>> tail, 2048 max, 0 nreq)
>> [ 8882.732542] ***ERROR***: srpt_xfer_data[2374] ret=-12
>> [ 8882.766396] <6>[0]: scst_queue_retry_cmd:1099:TGT QUEUE FULL:
>> incrementing retry_cmds 1
>> [ 8882.766403] <6>[0]: scst_queue_retry_cmd:1106:Some command(s) finished,
>> direct retry (finished_cmds=2310445, tgt->finished_cmds=2310539,
>> retry_cmds=0)
>> [ 8891.650890] ib_mthca 0000:81:00.0: SQ 000407 full (1155377 head, 1153329
>> tail, 2048 max, 0 nreq)
>> [ 8891.689016] ***ERROR***: srpt_xfer_data[2374] ret=-12
>> [ 8891.723548] <6>[0]: scst_queue_retry_cmd:1099:TGT QUEUE FULL:
>> incrementing retry_cmds 1
>> [ 8891.723556] <6>[0]: scst_queue_retry_cmd:1106:Some command(s) finished,
>> direct retry (finished_cmds=2381910, tgt->finished_cmds=2382001,
>> retry_cmds=0)
>> [ 8891.723573] ib_mthca 0000:81:00.0: too many gathers
>> [ 8891.758000] ***ERROR***: srpt_xfer_data[2374] ret=-22
>> [ 8891.792888] <6>[0]: scst: scst_rdy_to_xfer:985:***ERROR***: Target driver
>> ib_srpt rdy_to_xfer() returned fatal error
>>
>> I hope that helps.
>>
>> I've seen that same "rdy_to_xfer() returned fatal error several times in
>> different configurations.  The screen shots we sent earlier had the same
>> "ib_mthca ... SQ ... full (xx head..." message at the start.  So that seems
>> to be related as well.

Looks like ib_post_send() in srpt_perform_rdmas() returned ENOMEM and 
then srpt_xfer_data() "forgot" to unmapped corresponding SG with all 
related consequences.

>> Thanks for the help,
>> Phil P.
>>
>> --
>> Philip Pokorny, RHCE
>> Chief Hardware Architect - Penguin Computing
>> Voice: 415-370-0835  Toll free: 888-PENGUIN
>> www.penguincomputing.com

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SRPT and SCST

[Bart Van Assche]

On Thu, Nov 5, 2009 at 2:27 PM, Vladislav Bolkhovitin <vst-d+Crzxg7Rs0@public.gmane.org> wrote:
>
> Philip Pokorny, on 11/05/2009 11:51 AM wrote:
>>
>> Chris Worley asked that we post this to scst and linux-rdma lists for discussion.
>>
>> We're trying to get IB SRPT working and can't seem to get a stable configuration using any of the various SCST, IB_SRPT, and kernel/distro versions out there.  In most cases, we're able to crash the connection and typically the target within minutes of pounding by 4 initiators doing "mkfs.ext3", "tar xf" and "fsck" to the SRP block device.
>>
>> Our target is a Penguin Computing Altus 2704 with disk expansion chassis.  That's a 4-socket AMD hex-core (24 total cores) with 128GB of memory and 24 1TB drives attached to two LSI 1068 SAS controllers. (aka 3801E)  The drives are configured as 12 RAID-1 mirrors and 3-wide LVM stripes over those mirrors.  There are an additional 6 SSD's in the server in a "fast" VG also RAID-1 mirrored and LVM striped.  Read ahead is disabled on the LVM volumes.
>>
>> LVM volumes are exported via SCST as FILEIO block devices to initiators.  50 groups are defined with two LVM volumes/block devices per group. One initiator per group. (NODE GUID added to "names" in the group)
>>
>> With only 4 initiators, almost 100% of I/O is to RAM and no disk I/O is seen on the target.
>>
>> Performance (when it's working) is generally good at 800MB/sec aggregate, but we'd like to see better.  It appeared we were getting 1.3GB/s at one point.
>>
>> On Wed, Nov 4, 2009 at 5:34 PM, Philip Pokorny
>> <ppokorny-pabcTyWEv4ZW60MLeMDbCVaTQe2KTcn/@public.gmane.org> wrote:
>>>
>>> We got a serial console attached and ran a test using the SCST and IB_SRPT
>>> versions that you recommended (Arend set it up so I'll defer to him on the
>>> exact SVN checkout that he used).
>>>
>>>> What sort of crashes are you seeing?  I also have a customer
>>>> experiencing a crash, but I can't get details out of them.
>>>
>>> The client gets SCSI I/O errors and aborts the filesystem (putting it in
>>> read-only mode).
>>>
>>> After about 400 seconds of testing, the server side logs the following:
>>>
>>> [ 8418.697830] <6>[12426]: scst_check_sense:2444:Clearing dbl_ua_possible
>>> flag (dev ffff811816136000, cmd ffff81081017c1c8)
>>> [ 8418.697836] <6>[12426]: scst_dec_on_dev_cmd:577:cmd ffff81081017c1c8 (tag
>>> 17): unblocking dev ffff811816136000
>>> [ 8418.697843] <6>[0]: scst_unblock_dev:4653:Device UNBLOCK(new 0), dev
>>> ffff811816136000
>>> [ 8864.258468] ib_mthca 0000:81:00.0: SQ 000405 full (999320 head, 997272
>>> tail, 2048 max, 0 nreq)
>>> [ 8864.294450] ***ERROR***: srpt_xfer_data[2374] ret=-12
>>> [ 8864.326702] <6>[0]: scst_queue_retry_cmd:1099:TGT QUEUE FULL:
>>> incrementing retry_cmds 1
>>> [ 8864.326709] <6>[0]: scst_queue_retry_cmd:1106:Some command(s) finished,
>>> direct retry (finished_cmds=2023031, tgt->finished_cmds=2023137,
>>> retry_cmds=0)
>>> [ 8878.447081] ib_mthca 0000:81:00.0: SQ 000406 full (1080498 head, 1078450
>>> tail, 2048 max, 0 nreq)
>>> [ 8878.484452] ***ERROR***: srpt_xfer_data[2374] ret=-12
>>> [ 8878.517595] <6>[0]: scst_queue_retry_cmd:1099:TGT QUEUE FULL:
>>> incrementing retry_cmds 1
>>> [ 8878.517608] <6>[0]: scst_queue_retry_cmd:1106:Some command(s) finished,
>>> direct retry (finished_cmds=2256307, tgt->finished_cmds=2256504,
>>> retry_cmds=0)
>>> [ 8882.694684] ib_mthca 0000:81:00.0: SQ 000404 full (1087484 head, 1085436
>>> tail, 2048 max, 0 nreq)
>>> [ 8882.732542] ***ERROR***: srpt_xfer_data[2374] ret=-12
>>> [ 8882.766396] <6>[0]: scst_queue_retry_cmd:1099:TGT QUEUE FULL:
>>> incrementing retry_cmds 1
>>> [ 8882.766403] <6>[0]: scst_queue_retry_cmd:1106:Some command(s) finished,
>>> direct retry (finished_cmds=2310445, tgt->finished_cmds=2310539,
>>> retry_cmds=0)
>>> [ 8891.650890] ib_mthca 0000:81:00.0: SQ 000407 full (1155377 head, 1153329
>>> tail, 2048 max, 0 nreq)
>>> [ 8891.689016] ***ERROR***: srpt_xfer_data[2374] ret=-12
>>> [ 8891.723548] <6>[0]: scst_queue_retry_cmd:1099:TGT QUEUE FULL:
>>> incrementing retry_cmds 1
>>> [ 8891.723556] <6>[0]: scst_queue_retry_cmd:1106:Some command(s) finished,
>>> direct retry (finished_cmds=2381910, tgt->finished_cmds=2382001,
>>> retry_cmds=0)
>>> [ 8891.723573] ib_mthca 0000:81:00.0: too many gathers
>>> [ 8891.758000] ***ERROR***: srpt_xfer_data[2374] ret=-22
>>> [ 8891.792888] <6>[0]: scst: scst_rdy_to_xfer:985:***ERROR***: Target driver
>>> ib_srpt rdy_to_xfer() returned fatal error
>>>
>>> I hope that helps.
>>>
>>> I've seen that same "rdy_to_xfer() returned fatal error several times in
>>> different configurations.  The screen shots we sent earlier had the same
>>> "ib_mthca ... SQ ... full (xx head..." message at the start.  So that seems
>>> to be related as well.
>
> Looks like ib_post_send() in srpt_perform_rdmas() returned ENOMEM and then srpt_xfer_data() "forgot" to unmapped corresponding SG with all related consequences.

I would like to add the following to the above analysis:
* The function srpt_xfer_data() calls srpt_perform_rdmas(). The last
function calls ib_post_send() repeatedly. ib_post_send() can fail
because the work queue on which the send request should be posted is
full. When processing an sg list with multiple entries, it is possible
that some entries get posted to the work queue and some not.
* If this happens, the last send posted on the work queue won't
trigger a completion event (this is a bug in srpt_perform_rdmas()).
* If the srpt_perform_rdmas() call inside srpt_xfer_data() fails, it
is possible that some of the sg list entries have been posted on the
IB work queue and some not. So error recovery will be more complex
than just a single call to ib_dma_unmap_sg().

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SRPT and SCST

[Bart Van Assche]

On Fri, Nov 6, 2009 at 3:54 AM, Philip Pokorny
<ppokorny-pabcTyWEv4ZW60MLeMDbCVaTQe2KTcn/@public.gmane.org> wrote:
> ======= Update
> At Chris's suggestion, I changed the scst threading to a *single* thread.  I
> also changed the number of outstanding commands to 32 (from 64)  We have a
> relatively large number of initiators (50, currently testing with only 4)
> but each initiator has it's own dedicated, non-shared set of LVM "block"
> devices.  So it does not seem necessary to set the device queue size to
> initiators*device_queue_size.  I set the total device queued commands to 37.
>
> Re-running the test, the storage is "stable" and completed the first phase
> of the test (create and check the filesystem) but now two of four clients
> have "live-locked" waiting for an I/O to complete for the last 15 minutes or
> so.
>
> On the initiators iostat looks like this:
>
> Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await  svctm  %util
> sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
> 160.02    0.00   0.00 100.01
> sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
> 160.02    0.00   0.00 100.01
> sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
> 160.02    0.00   0.00 100.01
> sdb               0.00     0.00  0.00  0.14     0.00     8.14   114.00
> 159.34 180114.00 7002.00 100.03
> sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
> 160.02    0.00   0.00 100.01
> sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
> 160.02    0.00   0.00 100.01
>
> This tells me that there is a pending I/O waiting to be completed but it
> seems to have been lost on the server, because this is taking much too
> long.  There are 7 seconds "between" each line of output above so that's
> almost 30 seconds of output with *no* change in the I/O status.
>
> The "gzip | tar -x" I was running is "hung"

Hello Phil,

Can you please post the SCST target logs available for the above scenario ?

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SRPT and SCST

[Vladislav Bolkhovitin]

Bart Van Assche, on 11/06/2009 10:06 AM wrote:
> On Fri, Nov 6, 2009 at 3:54 AM, Philip Pokorny
> <ppokorny-pabcTyWEv4ZW60MLeMDbCVaTQe2KTcn/@public.gmane.org> wrote:
>> ======= Update
>> At Chris's suggestion, I changed the scst threading to a *single* thread.  I
>> also changed the number of outstanding commands to 32 (from 64)  We have a
>> relatively large number of initiators (50, currently testing with only 4)
>> but each initiator has it's own dedicated, non-shared set of LVM "block"
>> devices.  So it does not seem necessary to set the device queue size to
>> initiators*device_queue_size.  I set the total device queued commands to 37.
>>
>> Re-running the test, the storage is "stable" and completed the first phase
>> of the test (create and check the filesystem) but now two of four clients
>> have "live-locked" waiting for an I/O to complete for the last 15 minutes or
>> so.
>>
>> On the initiators iostat looks like this:
>>
>> Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz
>> avgqu-sz   await  svctm  %util
>> sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
>> 160.02    0.00   0.00 100.01
>> sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
>> 160.02    0.00   0.00 100.01
>> sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
>> 160.02    0.00   0.00 100.01
>> sdb               0.00     0.00  0.00  0.14     0.00     8.14   114.00
>> 159.34 180114.00 7002.00 100.03
>> sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
>> 160.02    0.00   0.00 100.01
>> sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
>> 160.02    0.00   0.00 100.01
>>
>> This tells me that there is a pending I/O waiting to be completed but it
>> seems to have been lost on the server, because this is taking much too
>> long.  There are 7 seconds "between" each line of output above so that's
>> almost 30 seconds of output with *no* change in the I/O status.
>>
>> The "gzip | tar -x" I was running is "hung"
> 
> Hello Phil,
> 
> Can you please post the SCST target logs available for the above scenario ?

Yes, and please make sure you are running the debug build.

> Bart.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SRPT and SCST

[Bart Van Assche]

On Fri, Nov 6, 2009 at 3:39 PM, Philip Pokorny
<ppokorny-pabcTyWEv4ZW60MLeMDbCVaTQe2KTcn/@public.gmane.org> wrote:
>
> >> This tells me that there is a pending I/O waiting to be completed but it
> >> seems to have been lost on the server, because this is taking much too
> >> long.  There are 7 seconds "between" each line of output above so that's
> >> almost 30 seconds of output with *no* change in the I/O status.
> >>
> >> The "gzip | tar -x" I was running is "hung"
>
> Upon further investigation, I found that the clients had actually aborted SCSI commands that took too long:
>
> sd 7:0:0:1: timing out command, waited 180s
> sd 7:0:0:1: SCSI error: return code = 0x06000000
> end_request: I/O error, dev sdb, sector 60377910
> Buffer I/O error on device sdb1, logical block 30188699
> lost page write due to I/O error on sdb1
> sd 7:0:0:1: timing out command, waited 180s
> sd 7:0:0:1: SCSI error: return code = 0x06000000
> end_request: I/O error, dev sdb, sector 186810934
> EXT3-fs error (device sdb1): ext3_get_inode_loc: unable to read inode block - inode=11675841, block=93405211
> Aborting journal on device sdb1.
>
> I should point out that the IB_SRP CLIENT we are using is from OFED 1.3.2
>
> [root@head0 ~]# modinfo ib_srp
> filename:       /lib/modules/2.6.18-128.1.1.el5.530g0000/kernel/drivers/infiniband/ulp/srp/ib_srp.ko
> license:        Dual BSD/GPL
> description:    InfiniBand SCSI RDMA Protocol initiator v0.2 (November 1, 2005)
>
> These are Red Hat 5 clients and we can upgrade to Red Hat 5.4 with the Red Hat IB_SRP, but it may be the same code.  Anything else will be more work.

It might be a good idea to repeat the test with the SRP initiator
included with RHEL 5 instead of the OFED SRP initiator. At least one
bug that is present in the OFED SRP initiator is not present in the
RHEL 5 SRP initiator. See also
https://bugs.openfabrics.org/show_bug.cgi?id=1745 for an example.

> > Can you please post the SCST target logs available for the above scenario ?
>
> Yes, and please make sure you are running the debug build.
>
> =====
> Sure.  We *are* running the CONFIG_SCST_DEBUG build.
>
> How do I collect the target logs?  I don't see anything obvious in /proc/scsi_tgt/...

The following commands will enable lots of additional tracing
information (probably way too much):
cat /proc/scsi_tgt/help
echo all >/proc/scsi_tgt/trace_level
echo all >/proc/scsi_tgt/vdisk/trace_level
echo all >/proc/scsi_tgt/ib_srpt/trace_level

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SRPT and SCST

[Vladislav Bolkhovitin]

Bart Van Assche, on 11/06/2009 05:53 PM wrote:
> On Fri, Nov 6, 2009 at 3:39 PM, Philip Pokorny
> <ppokorny-pabcTyWEv4ZW60MLeMDbCVaTQe2KTcn/@public.gmane.org> wrote:
>>>> This tells me that there is a pending I/O waiting to be completed but it
>>>> seems to have been lost on the server, because this is taking much too
>>>> long.  There are 7 seconds "between" each line of output above so that's
>>>> almost 30 seconds of output with *no* change in the I/O status.
>>>>
>>>> The "gzip | tar -x" I was running is "hung"
>> Upon further investigation, I found that the clients had actually aborted SCSI commands that took too long:
>>
>> sd 7:0:0:1: timing out command, waited 180s
>> sd 7:0:0:1: SCSI error: return code = 0x06000000
>> end_request: I/O error, dev sdb, sector 60377910
>> Buffer I/O error on device sdb1, logical block 30188699
>> lost page write due to I/O error on sdb1
>> sd 7:0:0:1: timing out command, waited 180s
>> sd 7:0:0:1: SCSI error: return code = 0x06000000
>> end_request: I/O error, dev sdb, sector 186810934
>> EXT3-fs error (device sdb1): ext3_get_inode_loc: unable to read inode block - inode=11675841, block=93405211
>> Aborting journal on device sdb1.
>>
>> I should point out that the IB_SRP CLIENT we are using is from OFED 1.3.2
>>
>> [root@head0 ~]# modinfo ib_srp
>> filename:       /lib/modules/2.6.18-128.1.1.el5.530g0000/kernel/drivers/infiniband/ulp/srp/ib_srp.ko
>> license:        Dual BSD/GPL
>> description:    InfiniBand SCSI RDMA Protocol initiator v0.2 (November 1, 2005)
>>
>> These are Red Hat 5 clients and we can upgrade to Red Hat 5.4 with the Red Hat IB_SRP, but it may be the same code.  Anything else will be more work.
> 
> It might be a good idea to repeat the test with the SRP initiator
> included with RHEL 5 instead of the OFED SRP initiator. At least one
> bug that is present in the OFED SRP initiator is not present in the
> RHEL 5 SRP initiator. See also
> https://bugs.openfabrics.org/show_bug.cgi?id=1745 for an example.
> 
>>> Can you please post the SCST target logs available for the above scenario ?
>> Yes, and please make sure you are running the debug build.
>>
>> =====
>> Sure.  We *are* running the CONFIG_SCST_DEBUG build.
>>
>> How do I collect the target logs?  I don't see anything obvious in /proc/scsi_tgt/...

They are in the regular kernel logs. Refer to you distribution kernel 
logging configuration to find out where they are stored. Usually it is 
/var/log/messages.

> The following commands will enable lots of additional tracing
> information (probably way too much):
> cat /proc/scsi_tgt/help
> echo all >/proc/scsi_tgt/trace_level
> echo all >/proc/scsi_tgt/vdisk/trace_level

The above commands will enable all the logging. But in this case it will 
be an overkill. By default CONFIG_SCST_DEBUG configuration has all the 
minimally necessary logging enabled, so, Philip, for your case you don't 
need to enable anything additional.

> echo all >/proc/scsi_tgt/ib_srpt/trace_level
> 
> Bart.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SRPT and SCST

[Bart Van Assche]

On Fri, Nov 6, 2009 at 6:28 PM, Arend Dittmer
<adittmer-pabcTyWEv4ZW60MLeMDbCVaTQe2KTcn/@public.gmane.org> wrote:
> Please find attached the gzip'ed /var/log/messages.

This log clearly show the login and logout actions from the different
initiators. I couldn't find anything unusual in the posted log file
however. Around which time did the initiator start complaining about
aborted SCSI commands ? Does this issue also happen when using the SRP
initiator included in a vanilla (non-OFED) Linux kernel ?

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SRPT and SCST

[Vladislav Bolkhovitin]

Bart Van Assche, on 11/08/2009 12:49 PM wrote:
> On Fri, Nov 6, 2009 at 6:28 PM, Arend Dittmer
> <adittmer-pabcTyWEv4ZW60MLeMDbCVaTQe2KTcn/@public.gmane.org> wrote:
>> Please find attached the gzip'ed /var/log/messages.
> 
> This log clearly show the login and logout actions from the different
> initiators. I couldn't find anything unusual in the posted log file
> however. Around which time did the initiator start complaining about
> aborted SCSI commands ? Does this issue also happen when using the SRP
> initiator included in a vanilla (non-OFED) Linux kernel ?

It looks painfully similar to what Chris Worley experienced some time 
ago and somehow fixed/workarounded.

Chris, can you comment on this?

> Bart.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SRPT and SCST

[Chris Worley]

On Mon, Nov 9, 2009 at 1:26 PM, Vladislav Bolkhovitin <vst-d+Crzxg7Rs0@public.gmane.org> wrote:
> Bart Van Assche, on 11/08/2009 12:49 PM wrote:
>>
>> On Fri, Nov 6, 2009 at 6:28 PM, Arend Dittmer
>> <adittmer-pabcTyWEv4ZW60MLeMDbCVaTQe2KTcn/@public.gmane.org> wrote:
>>>
>>> Please find attached the gzip'ed /var/log/messages.
>>
>> This log clearly show the login and logout actions from the different
>> initiators. I couldn't find anything unusual in the posted log file
>> however. Around which time did the initiator start complaining about
>> aborted SCSI commands ? Does this issue also happen when using the SRP
>> initiator included in a vanilla (non-OFED) Linux kernel ?
>
> It looks painfully similar to what Chris Worley experienced some time ago
> and somehow fixed/workarounded.
>
> Chris, can you comment on this?

The "thread=1" fixed the problem mostly, but I am working with another
group that says they still get an abort, but haven't gotten around to
providing me with the info I need to look at it.

Chris
>
>> Bart.
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: SRPT and SCST

[Arend Dittmer]

To Bart's earlier question  ... we apologize for not being able to come up with a time and date when the initiators lost contact with the target. We have not been able to test with an initiator from a vanilla kernel. We only tested with the initiator that ships with RedHat 5.3. We built the module against a slightly modified RedHat kernel that includes process management patches that allow for a unified process space for our cluster management software Scyld clusterware. Our patches do not affect any storage components. 

[root@head0 ~]# bpsh 1 modinfo ib_srp
filename:       /lib/modules/2.6.18-128.1.1.el5.530g0000/kernel/drivers/infiniband/ulp/srp/ib_srp.ko

license:        Dual BSD/GPL
description:    InfiniBand SCSI RDMA Protocol initiator v0.2 (November 1, 2005)
author:         Roland Dreier
srcversion:     23B2629641E1A475BF72F44
depends:        ib_core,scsi_mod,ib_cm,ib_sa
vermagic:       2.6.18-128.1.1.el5.530g0000 SMP mod_unload gcc-4.1
parm:           srp_sg_tablesize:Max number of gather/scatter entries per I/O (default is 12) (int)
parm:           topspin_workarounds:Enable workarounds for Topspin/Cisco SRP target bugs if != 0 (in
t)
parm:           mellanox_workarounds:Enable workarounds for Mellanox SRP target bugs if != 0 (int)
parm:           srp_dev_loss_tmo:Default number of seconds that srp transport should              in
sulate the lost of a remote port (default is 60 secs (int)
module_sig:     883f35049c0555e56ccec1c0ba19c3112f87b09e2872185017b618b6026be92291a62b5446018e009d1e
3299cd274ad8e31c3d0b03081b112959d4d84

Also ... we ran with only a single thread.

Thanks

Arend     


-----Original Message-----
From: Chris Worley [mailto:worleys-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org]
Sent: Mon 11/9/2009 3:43 PM
To: Vladislav Bolkhovitin
Cc: Bart Van Assche; Arend Dittmer; Philip Pokorny; scst-devel-5NWGOfrQmnc@public.gmane.orgurceforge.net; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Vu Pham
Subject: Re: SRPT and SCST
 
On Mon, Nov 9, 2009 at 1:26 PM, Vladislav Bolkhovitin <vst-d+Crzxg7Rs0@public.gmane.org> wrote:
> Bart Van Assche, on 11/08/2009 12:49 PM wrote:
>>
>> On Fri, Nov 6, 2009 at 6:28 PM, Arend Dittmer
>> <adittmer-pabcTyWEv4ZW60MLeMDbCVaTQe2KTcn/@public.gmane.org> wrote:
>>>
>>> Please find attached the gzip'ed /var/log/messages.
>>
>> This log clearly show the login and logout actions from the different
>> initiators. I couldn't find anything unusual in the posted log file
>> however. Around which time did the initiator start complaining about
>> aborted SCSI commands ? Does this issue also happen when using the SRP
>> initiator included in a vanilla (non-OFED) Linux kernel ?
>
> It looks painfully similar to what Chris Worley experienced some time ago
> and somehow fixed/workarounded.
>
> Chris, can you comment on this?

The "thread=1" fixed the problem mostly, but I am working with another
group that says they still get an abort, but haven't gotten around to
providing me with the info I need to look at it.

Chris
>
>> Bart.
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SRPT and SCST

[Vladislav Bolkhovitin]

Arend Dittmer, on 11/11/2009 03:33 AM wrote:
> To Bart's earlier question  ... we apologize for not being able to come up with a time and date when the initiators lost contact with the target. We have not been able to test with an initiator from a vanilla kernel. We only tested with the initiator that ships with RedHat 5.3. We built the module against a slightly modified RedHat kernel that includes process management patches that allow for a unified process space for our cluster management software Scyld clusterware. Our patches do not affect any storage components. 
> 
> [root@head0 ~]# bpsh 1 modinfo ib_srp
> filename:       /lib/modules/2.6.18-128.1.1.el5.530g0000/kernel/drivers/infiniband/ulp/srp/ib_srp.ko
> 
> license:        Dual BSD/GPL
> description:    InfiniBand SCSI RDMA Protocol initiator v0.2 (November 1, 2005)
> author:         Roland Dreier
> srcversion:     23B2629641E1A475BF72F44
> depends:        ib_core,scsi_mod,ib_cm,ib_sa
> vermagic:       2.6.18-128.1.1.el5.530g0000 SMP mod_unload gcc-4.1
> parm:           srp_sg_tablesize:Max number of gather/scatter entries per I/O (default is 12) (int)
> parm:           topspin_workarounds:Enable workarounds for Topspin/Cisco SRP target bugs if != 0 (in
> t)
> parm:           mellanox_workarounds:Enable workarounds for Mellanox SRP target bugs if != 0 (int)

Hmm, "workarounds for Mellanox SRP target", i.e. for SCST SRP target? 
For what are those workarounds and why don't fix the corresponding 
problems in the target? From the source code it isn't obvious..

> parm:           srp_dev_loss_tmo:Default number of seconds that srp transport should              in
> sulate the lost of a remote port (default is 60 secs (int)
> module_sig:     883f35049c0555e56ccec1c0ba19c3112f87b09e2872185017b618b6026be92291a62b5446018e009d1e
> 3299cd274ad8e31c3d0b03081b112959d4d84
> 
> Also ... we ran with only a single thread.
> 
> Thanks
> 
> Arend     
> 
> 
> -----Original Message-----
> From: Chris Worley [mailto:worleys-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org]
> Sent: Mon 11/9/2009 3:43 PM
> To: Vladislav Bolkhovitin
> Cc: Bart Van Assche; Arend Dittmer; Philip Pokorny; scst-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Vu Pham
> Subject: Re: SRPT and SCST
>  
> On Mon, Nov 9, 2009 at 1:26 PM, Vladislav Bolkhovitin <vst-d+Crzxg7Rs0@public.gmane.org> wrote:
>> Bart Van Assche, on 11/08/2009 12:49 PM wrote:
>>> On Fri, Nov 6, 2009 at 6:28 PM, Arend Dittmer
>>> <adittmer-pabcTyWEv4ZW60MLeMDbCVaTQe2KTcn/@public.gmane.org> wrote:
>>>> Please find attached the gzip'ed /var/log/messages.
>>> This log clearly show the login and logout actions from the different
>>> initiators. I couldn't find anything unusual in the posted log file
>>> however. Around which time did the initiator start complaining about
>>> aborted SCSI commands ? Does this issue also happen when using the SRP
>>> initiator included in a vanilla (non-OFED) Linux kernel ?
>> It looks painfully similar to what Chris Worley experienced some time ago
>> and somehow fixed/workarounded.
>>
>> Chris, can you comment on this?
> 
> The "thread=1" fixed the problem mostly, but I am working with another
> group that says they still get an abort, but haven't gotten around to
> providing me with the info I need to look at it.
> 
> Chris
>>> Bart.
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SRPT and SCST

[Bart Van Assche]

On Fri, Nov 6, 2009 at 3:54 AM, Philip Pokorny
<ppokorny-pabcTyWEv4ZW60MLeMDbCVaTQe2KTcn/@public.gmane.org> wrote:
>
> On Thu, Nov 5, 2009 at 2:27 PM, Vladislav Bolkhovitin <vst-d+Crzxg7Rs0@public.gmane.org> wrote:
> >
> > Philip Pokorny, on 11/05/2009 11:51 AM wrote:
> >>
> >> Chris Worley asked that we post this to scst and linux-rdma lists for discussion.
> >>
> >
> > Looks like ib_post_send() in srpt_perform_rdmas() returned ENOMEM and then srpt_xfer_data() "forgot" to unmapped corresponding SG with all related consequences.
>
> I would like to add the following to the above analysis:
> * The function srpt_xfer_data() calls srpt_perform_rdmas(). The last
> function calls ib_post_send() repeatedly. ib_post_send() can fail
> because the work queue on which the send request should be posted is
> full. When processing an sg list with multiple entries, it is possible
> that some entries get posted to the work queue and some not.
> * If this happens, the last send posted on the work queue won't
> trigger a completion event (this is a bug in srpt_perform_rdmas()).
> * If the srpt_perform_rdmas() call inside srpt_xfer_data() fails, it
> is possible that some of the sg list entries have been posted on the
> IB work queue and some not. So error recovery will be more complex
> than just a single call to ib_dma_unmap_sg().
>
> =======
> Is there something I could code up to sleep, pause or retry the calls to ib_post_send if they fail?

Proper error handling has to be implemented in srpt_xfer_data(). Also,
measures have to be taken such that no more work is posted on the IB
queue than the reserved size. Which value are you using for
SCST_MAX_TGT_DEV_COMMANDS (defined in scst/src/scst_priv.h) ?

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SRPT and SCST

[Bart Van Assche]

On Thu, Nov 5, 2009 at 9:51 AM, Philip Pokorny
<ppokorny-pabcTyWEv4ZW60MLeMDbCVaTQe2KTcn/@public.gmane.org> wrote:
>>
>> [ ... ]
>>
>> [ 8864.326702] <6>[0]: scst_queue_retry_cmd:1099:TGT QUEUE FULL:
>> incrementing retry_cmds 1
>> [ 8864.326709] <6>[0]: scst_queue_retry_cmd:1106:Some command(s) finished,
>> direct retry (finished_cmds=2023031, tgt->finished_cmds=2023137,
>> retry_cmds=0)
>> [ ... ]

(replying to an e-mail of one month ago)

The above issue should be fixed in SCST revision 1391, together with
the following changes:
* Support for OFED 1.5.
* The maximum supported value for the ib_srp kernel module parameter
srp_sg_tablesize has been increased from 58 to 128, at least when
ib_srpt is loaded with default parameters. Modifying the new ib_srpt
kernel module parameter srp_max_rdma_size makes it possible to support
even higher values of srp_sg_tablesize.
* Various small performance improvements.

The results I obtained with QDR HCA's, a RAM disk as target and a
vanilla 2.6.27.39 kernel + CFQ on the initiator system are:
* for asynchronous (buffered) I/O, writing speeds up to 1070 MB/s
(srp_sg_tablesize = 128; block size = 16 KB).
* for asynchronous (buffered) I/O, reading speeds up to 930 MB/s
(srp_sg_tablesize = 12; block size = 16 KB).
* for direct (non-buffered) I/O, writing speeds up to 1000 MB/s
(srp_sg_tablesize = 32; block size = 2 MB).
* for direct (non-buffered) I/O, reading speeds up to 1710 MB/s
(srp_sg_tablesize = 16; block size = 64 MB).

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SRPT and SCST

[Bart Van Assche]

(resending as plain text)

On Thu, Nov 5, 2009 at 10:51 AM, Philip Pokorny
<ppokorny-pabcTyWEv4ZW60MLeMDbCVaTQe2KTcn/@public.gmane.org> wrote:
>
> [ ... ]
>
> Performance (when it's working) is generally good at 800MB/sec aggregate, but we'd like to see better.  It appeared we were getting 1.3GB/s at one point.

(replying to an e-mail that was posted several months ago)

You might have hit a limitation of Linux' asynchronous I/O subsystem
(at the SRP initiator side). Asynchronous I/O performance has been
improved significantly in the 2.6.33 kernel. The throughput I measured
with fio, QDR HCA's and a 2.6.34 initiator is as follows:
* 940 MB/s for asynchronous writes and 1240 MB/s for asynchronous
reads and a block size of 64 KB.
* 2200 MB/s for direct writes and 2900 MB/s for direct reads and a
block size of 64000 KB.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html