Re: block: remove artifical max_hw_sectors cap

Re: block: remove artifical max_hw_sectors cap

[Christoph Hellwig]

On Mon, Dec 22, 2014 at 11:31:53AM +0100, Stefan Priebe - Profihost AG wrote:
> Hi,
> 
> since the below patch i've some problems with iscsi.
> 
> The LIO based iscsi Server is full of messages like this:
> SCSI OP 2ah with too big sectors 8344 exceeds fabric_max_sectors: 8192
> 
> And the client says:
> Buffer I/O error on device sdd, logical block 38861907
> lost page write due to I/O error on sdd
> 
> May be some code fixes for iscsi client is missing?

No, the problem seems that the the target is enforcing a size limit
without communicating it through the block limits EVPD page.

Nic, can you fix LIO to expose the proper max xfer size?

Re: block: remove artifical max_hw_sectors cap

[Stefan Priebe]

Hi Christoph,

Am 23.12.2014 um 09:28 schrieb Christoph Hellwig:
> On Mon, Dec 22, 2014 at 11:31:53AM +0100, Stefan Priebe - Profihost AG wrote:
>> Hi,
>>
>> since the below patch i've some problems with iscsi.
>>
>> The LIO based iscsi Server is full of messages like this:
>> SCSI OP 2ah with too big sectors 8344 exceeds fabric_max_sectors: 8192
>>
>> And the client says:
>> Buffer I/O error on device sdd, logical block 38861907
>> lost page write due to I/O error on sdd
>>
>> May be some code fixes for iscsi client is missing?
>
> No, the problem seems that the the target is enforcing a size limit
> without communicating it through the block limits EVPD page.
>
> Nic, can you fix LIO to expose the proper max xfer size?

some more problems while running this patch.

My crucial m500 and m550 ssds (i have around 60 in 30 different cases 
and motherboards but all attached as jbods at LSI HBAs) print those errors:

Dec 23 03:14:56 cloud2-1375 kernel: [585489.194158] Write(10): 2a 00 04 
f9 db f0 00 0c d8 00
Dec 23 03:14:56 cloud2-1375 kernel: [585489.194162] scsi target0:0:7: 
handle(0x000a), sas_address(0x4433221107000000), phy(7)
Dec 23 03:14:56 cloud2-1375 kernel: [585489.194163] scsi target0:0:7: 
enclosure_logical_id(0x5003048016aee700), slot(5)
Dec 23 03:14:56 cloud2-1375 kernel: [585489.194178] sd 0:0:7:0: task 
abort: SUCCESS scmd(ffff88018c6b4700)
Dec 23 03:14:56 cloud2-1375 kernel: [585489.194178] sd 0:0:7:0: 
attempting task abort! scmd(ffff8807a12e6800)
Dec 23 03:14:56 cloud2-1375 kernel: [585489.194179] sd 0:0:7:0: [sdh] CDB:
Dec 23 03:14:56 cloud2-1375 kernel: [585489.194180] Read(10): 28 00 15 
d4 f9 60 00 00 10 00
Dec 23 03:14:56 cloud2-1375 kernel: [585489.194183] scsi target0:0:7: 
handle(0x000a), sas_address(0x4433221107000000), phy(7)
Dec 23 03:14:56 cloud2-1375 kernel: [585489.194184] scsi target0:0:7: 
enclosure_logical_id(0x5003048016aee700), slot(5)
Dec 23 03:14:56 cloud2-1375 kernel: [585489.194199] sd 0:0:7:0: task 
abort: SUCCESS scmd(ffff8807a12e6800)
Dec 23 03:14:56 cloud2-1375 kernel: [585489.194199] sd 0:0:7:0: 
attempting task abort! scmd(ffff88078d989800)
Dec 23 03:14:56 cloud2-1375 kernel: [585489.194200] sd 0:0:7:0: [sdh] CDB:
Dec 23 03:14:56 cloud2-1375 kernel: [585489.194201] Read(10): 28 00 15 
aa d2 f9 00 00 20 00

Stefan

Re: block: remove artifical max_hw_sectors cap

[Christoph Hellwig]

On Sun, Dec 28, 2014 at 01:08:59PM +0100, Stefan Priebe wrote:
>> Nic, can you fix LIO to expose the proper max xfer size?
>
> some more problems while running this patch.
>
> My crucial m500 and m550 ssds (i have around 60 in 30 different cases and 
> motherboards but all attached as jbods at LSI HBAs) print those errors:

What is the max_sectors_kb value for them?

Re: block: remove artifical max_hw_sectors cap

[Stefan Priebe - Profihost AG]

Am 30.12.2014 um 12:32 schrieb Christoph Hellwig:
> On Sun, Dec 28, 2014 at 01:08:59PM +0100, Stefan Priebe wrote:
>>> Nic, can you fix LIO to expose the proper max xfer size?
>>
>> some more problems while running this patch.
>>
>> My crucial m500 and m550 ssds (i have around 60 in 30 different cases and 
>> motherboards but all attached as jbods at LSI HBAs) print those errors:
> 
> What is the max_sectors_kb value for them?
> 
# cat /sys/block/sdi/queue/max_sectors_kb
16383

Greets,
Stefan

Re: block: remove artifical max_hw_sectors cap

[Christoph Hellwig]

On Tue, Dec 30, 2014 at 03:15:26PM +0100, Stefan Priebe - Profihost AG wrote:
> > What is the max_sectors_kb value for them?
> > 
> # cat /sys/block/sdi/queue/max_sectors_kb
> 16383

That's odd, it's half of what ATA disks should support.  Is this
device attached to the HBA or an expander?

Re: block: remove artifical max_hw_sectors cap

[Stefan Priebe]

Am 05.01.2015 um 18:17 schrieb Christoph Hellwig:
> On Tue, Dec 30, 2014 at 03:15:26PM +0100, Stefan Priebe - Profihost AG wrote:
>>> What is the max_sectors_kb value for them?
>>>
>> # cat /sys/block/sdi/queue/max_sectors_kb
>> 16383
>
> That's odd, it's half of what ATA disks should support.  Is this
> device attached to the HBA or an expander?

No the Board has:
10x SATA3 (6Gbps) via C612 from Intel.

Stefan

Re: block: remove artifical max_hw_sectors cap

[Nicholas A. Bellinger]

On Tue, 2014-12-23 at 09:28 +0100, Christoph Hellwig wrote:
> On Mon, Dec 22, 2014 at 11:31:53AM +0100, Stefan Priebe - Profihost AG wrote:
> > Hi,
> > 
> > since the below patch i've some problems with iscsi.
> > 
> > The LIO based iscsi Server is full of messages like this:
> > SCSI OP 2ah with too big sectors 8344 exceeds fabric_max_sectors: 8192
> > 
> > And the client says:
> > Buffer I/O error on device sdd, logical block 38861907
> > lost page write due to I/O error on sdd
> > 
> > May be some code fixes for iscsi client is missing?
> 
> No, the problem seems that the the target is enforcing a size limit
> without communicating it through the block limits EVPD page.
> 
> Nic, can you fix LIO to expose the proper max xfer size?

The fabric_max_sectors=8192 value is already being exposed by target in
block limits EVPD as MAXIMUM TRANSFER LENGTH.

I'm guessing that since the host side support was not added until June
2014 in commit bcdb247c by MKP, Stefan is likely using an earlier
initiator that is not honoring this value.

However, this same issue has been reported elsewhere [1] with an MSFT FC
host sending I/Os of 8 MB that also doesn't appear to honor block limits
MAXIMUM TRANSFER LENGTH either.

So that said, I think it's about time to go ahead and drop the arbitrary
limitation of fabric_max_sectors, and simply enforce a sane limit (eg:
not UINT_MAX) for hw_max_sectors.

The only limitations for backends that I'm aware of is the FILEIO
FD_MAX_BYTES limit for the number of iovecs per vfs_[writev,readv] call,
and for IBLOCK what queue_max_hw_sectors() reports for raw block
make_request() based drivers that aren't able to 

I'll put together a patch soon to drop the fabric_max_sectors stuff, and
use the existing hw_max_sectors to enforce any backend specific I/O size
limitations.

--nab

[1] http://www.spinics.net/lists/target-devel/msg08024.html

Re: block: remove artifical max_hw_sectors cap

[Nicholas A. Bellinger]

On Tue, 2015-01-06 at 14:39 -0800, Nicholas A. Bellinger wrote:
> On Tue, 2014-12-23 at 09:28 +0100, Christoph Hellwig wrote:
> > On Mon, Dec 22, 2014 at 11:31:53AM +0100, Stefan Priebe - Profihost AG wrote:
> > > Hi,
> > > 
> > > since the below patch i've some problems with iscsi.
> > > 
> > > The LIO based iscsi Server is full of messages like this:
> > > SCSI OP 2ah with too big sectors 8344 exceeds fabric_max_sectors: 8192
> > > 
> > > And the client says:
> > > Buffer I/O error on device sdd, logical block 38861907
> > > lost page write due to I/O error on sdd
> > > 
> > > May be some code fixes for iscsi client is missing?
> > 
> > No, the problem seems that the the target is enforcing a size limit
> > without communicating it through the block limits EVPD page.
> > 
> > Nic, can you fix LIO to expose the proper max xfer size?
> 
> The fabric_max_sectors=8192 value is already being exposed by target in
> block limits EVPD as MAXIMUM TRANSFER LENGTH.
> 
> I'm guessing that since the host side support was not added until June
> 2014 in commit bcdb247c by MKP, Stefan is likely using an earlier
> initiator that is not honoring this value.
> 
> However, this same issue has been reported elsewhere [1] with an MSFT FC
> host sending I/Os of 8 MB that also doesn't appear to honor block limits
> MAXIMUM TRANSFER LENGTH either.
> 
> So that said, I think it's about time to go ahead and drop the arbitrary
> limitation of fabric_max_sectors, and simply enforce a sane limit (eg:
> not UINT_MAX) for hw_max_sectors.
> 
> The only limitations for backends that I'm aware of is the FILEIO
> FD_MAX_BYTES limit for the number of iovecs per vfs_[writev,readv] call,
> and for IBLOCK what queue_max_hw_sectors() reports for raw block
> make_request() based drivers that aren't able to 
> 

... split large I/O requests into smaller ones.

> I'll put together a patch soon to drop the fabric_max_sectors stuff, and
> use the existing hw_max_sectors to enforce any backend specific I/O size
> limitations.
> 
> --nab
> 
> [1] http://www.spinics.net/lists/target-devel/msg08024.html
> 
> --
> To unsubscribe from this list: send the line "unsubscribe target-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: block: remove artifical max_hw_sectors cap

[Christoph Hellwig]

On Tue, Jan 06, 2015 at 02:39:16PM -0800, Nicholas A. Bellinger wrote:
> The fabric_max_sectors=8192 value is already being exposed by target in
> block limits EVPD as MAXIMUM TRANSFER LENGTH.
> 
> I'm guessing that since the host side support was not added until June
> 2014 in commit bcdb247c by MKP, Stefan is likely using an earlier
> initiator that is not honoring this value.

If I understand Stefan correct he was using a recent 3.19-rc kernel,
which contains my commit in the subject that stops capping the
max_sectors used by the kernel below that which the hardware supports.

Given that LIO actually implements the block limits EVPD that suggests
we're not properly using that information.

Stefant, what does:

sg_vpd -p bl /dev/sdX

for the LIO device tell?

Re: block: remove artifical max_hw_sectors cap

[Stefan Priebe - Profihost AG]

Am 07.01.2015 um 15:31 schrieb Christoph Hellwig:
> On Tue, Jan 06, 2015 at 02:39:16PM -0800, Nicholas A. Bellinger wrote:
>> The fabric_max_sectors=8192 value is already being exposed by target in
>> block limits EVPD as MAXIMUM TRANSFER LENGTH.
>>
>> I'm guessing that since the host side support was not added until June
>> 2014 in commit bcdb247c by MKP, Stefan is likely using an earlier
>> initiator that is not honoring this value.
> 
> If I understand Stefan correct he was using a recent 3.19-rc kernel,
> which contains my commit in the subject that stops capping the
> max_sectors used by the kernel below that which the hardware supports.
> 
> Given that LIO actually implements the block limits EVPD that suggests
> we're not properly using that information.
> 
> Stefant, what does:
> 
> sg_vpd -p bl /dev/sdX
> 
> for the LIO device tell?

I'm sorry i'm using a 3.10.63 vanilla kernel. So i'm missing a commit?

Stefan

Re: block: remove artifical max_hw_sectors cap

[Christoph Hellwig]

On Wed, Jan 07, 2015 at 03:32:41PM +0100, Stefan Priebe - Profihost AG wrote:
> > If I understand Stefan correct he was using a recent 3.19-rc kernel,
> > which contains my commit in the subject that stops capping the
> > max_sectors used by the kernel below that which the hardware supports.
> > 
> > Given that LIO actually implements the block limits EVPD that suggests
> > we're not properly using that information.
> > 
> > Stefant, what does:
> > 
> > sg_vpd -p bl /dev/sdX
> > 
> > for the LIO device tell?
> 
> I'm sorry i'm using a 3.10.63 vanilla kernel. So i'm missing a commit?

You said "block: remove artifical max_hw_sectors cap" causes the I/O
errors for you, which should only be in a 3.19-ish kernel.