BIO request larger than our storage device supports in linux kernel 4.x

BIO request larger than our storage device supports in linux kernel 4.x

[Umesh Patel]

Hello,

We are registering BIO size of our storage device through linux kernel API =
blk_queue_max_hw_sectors worth of 104K. So max size of BIO that our block d=
evice can handle is 104 KB but kernel block layer is sending 256 KB worth o=
f BIO which is more than we are supporting.
This issue we are observing in 4.4 and also in latest 4.10.1 kernel release=
. Until this we had not seen this issue.

>From the kernel source code history I came to below things.
4.1 kernel version onwards=20
"nr_pages =3D min(sdio->pages_in_io, bio_get_nr_vecs(map_bh->b_bdev))"
has been removed which (bio_get_nr_vecs) was considering max_sectors_kb of =
queue which was 104 KB for our device.

Presently new code is something like
"nr_pages =3D min(sdio->pages_in_io, BIO_MAX_PAGES)"
which is sending 256 KB (BIO_MAX_PAGES) of BIO size to our device which we =
are not supporting.


So from some documentation I found out that 256 KB of bio size is the fix (=
from 4.x kernel) and that has to support by any drive.=20
Please let me know is there any way to change BIO size worth of 104 KB from=
 256 KB or any other way to register our BIO size with kernel or any other =
area to look in to ?.=20

-Umesh

Re: BIO request larger than our storage device supports in linux kernel 4.x

[Christoph Hellwig]

Can you send the code for the driver, please?

Re: BIO request larger than our storage device supports in linux kernel 4.x

[Jens Axboe]

On 03/08/2017 02:34 AM, Umesh Patel wrote:
> Hello,
> 
> We are registering BIO size of our storage device through linux kernel
> API blk_queue_max_hw_sectors worth of 104K. So max size of BIO that
> our block device can handle is 104 KB but kernel block layer is
> sending 256 KB worth of BIO which is more than we are supporting.
> This issue we are observing in 4.4 and also in latest 4.10.1 kernel
> release. Until this we had not seen this issue.
> 
> From the kernel source code history I came to below things.  4.1
> kernel version onwards "nr_pages = min(sdio->pages_in_io,
> bio_get_nr_vecs(map_bh->b_bdev))" has been removed which
> (bio_get_nr_vecs) was considering max_sectors_kb of queue which was
> 104 KB for our device.
> 
> Presently new code is something like "nr_pages =
> min(sdio->pages_in_io, BIO_MAX_PAGES)" which is sending 256 KB
> (BIO_MAX_PAGES) of BIO size to our device which we are not supporting.
> 
> 
> So from some documentation I found out that 256 KB of bio size is the
> fix (from 4.x kernel) and that has to support by any drive.  Please
> let me know is there any way to change BIO size worth of 104 KB from
> 256 KB or any other way to register our BIO size with kernel or any
> other area to look in to ?. 

I'm assuming this is some not-open driver, and I'm also assuming that
you are bypassing the proper API and using make_request_fn. In which
case you are missing a call to blk_queue_split().

-- 
Jens Axboe

RE: BIO request larger than our storage device supports in linux kernel 4.x

[Umesh Patel]

Yes. This is non-open driver for our Flash based PCI drive for enterprise s=
torage solution.

Some snippet of queue related code is below

struct request_queue *queue;

blk_queue_max_hw_sectors(dev->osdev.queue, (dev->aggr_max_size >> KERNEL_SE=
CTOR_SHIFT));
blk_queue_max_segments(dev->osdev.queue, BLK_MAX_SEGMENTS);
blk_queue_bounce_limit(dev->osdev.queue, BLK_BOUNCE_ANY);
blk_queue_logical_block_size(dev->osdev.queue, dev->hardsect_size);
blk_queue_physical_block_size(dev->osdev.queue, dev->hardsect_size);

dev->osdev.queue =3D blk_alloc_queue(GFP_KERNEL);

blk_queue_make_request(dev->osdev.queue, fmm_bdev_make_request_wrapper);

Here queue is part of our internal structure struct fmm_bdev *dev




-----Original Message-----
From: Jens Axboe [mailto:axboe@kernel.dk] =

Sent: Wednesday, March 8, 2017 8:35 PM
To: Umesh Patel; linux-block@vger.kernel.org
Subject: Re: BIO request larger than our storage device supports in linux k=
ernel 4.x

On 03/08/2017 02:34 AM, Umesh Patel wrote:
> Hello,
> =

> We are registering BIO size of our storage device through linux kernel =

> API blk_queue_max_hw_sectors worth of 104K. So max size of BIO that =

> our block device can handle is 104 KB but kernel block layer is =

> sending 256 KB worth of BIO which is more than we are supporting.
> This issue we are observing in 4.4 and also in latest 4.10.1 kernel =

> release. Until this we had not seen this issue.
> =

> From the kernel source code history I came to below things.  4.1 =

> kernel version onwards "nr_pages =3D min(sdio->pages_in_io, =

> bio_get_nr_vecs(map_bh->b_bdev))" has been removed which
> (bio_get_nr_vecs) was considering max_sectors_kb of queue which was
> 104 KB for our device.
> =

> Presently new code is something like "nr_pages =3D =

> min(sdio->pages_in_io, BIO_MAX_PAGES)" which is sending 256 KB
> (BIO_MAX_PAGES) of BIO size to our device which we are not supporting.
> =

> =

> So from some documentation I found out that 256 KB of bio size is the =

> fix (from 4.x kernel) and that has to support by any drive.  Please =

> let me know is there any way to change BIO size worth of 104 KB from
> 256 KB or any other way to register our BIO size with kernel or any =

> other area to look in to ?.

I'm assuming this is some not-open driver, and I'm also assuming that you a=
re bypassing the proper API and using make_request_fn. In which case you ar=
e missing a call to blk_queue_split().

--
Jens Axboe

Western Digital Corporation (and its subsidiaries) E-mail Confidentiality N=
otice & Disclaimer:

This e-mail and any files transmitted with it may contain confidential or l=
egally privileged information of WDC and/or its affiliates, and are intende=
d solely for the use of the individual or entity to which they are addresse=
d. If you are not the intended recipient, any disclosure, copying, distribu=
tion or any action taken or omitted to be taken in reliance on it, is prohi=
bited. If you have received this e-mail in error, please notify the sender =
immediately and delete the e-mail in its entirety from your system.

Re: BIO request larger than our storage device supports in linux kernel 4.x

[Jens Axboe]

On 03/08/2017 11:08 AM, Umesh Patel wrote:
> 
> Yes. This is non-open driver for our Flash based PCI drive for enterprise storage solution.

Please don't top post.

> Some snippet of queue related code is below
> 
> struct request_queue *queue;
> 
> blk_queue_max_hw_sectors(dev->osdev.queue, (dev->aggr_max_size >> KERNEL_SECTOR_SHIFT));
> blk_queue_max_segments(dev->osdev.queue, BLK_MAX_SEGMENTS);
> blk_queue_bounce_limit(dev->osdev.queue, BLK_BOUNCE_ANY);
> blk_queue_logical_block_size(dev->osdev.queue, dev->hardsect_size);
> blk_queue_physical_block_size(dev->osdev.queue, dev->hardsect_size);
> 
> dev->osdev.queue = blk_alloc_queue(GFP_KERNEL);
> 
> blk_queue_make_request(dev->osdev.queue, fmm_bdev_make_request_wrapper);
> 
> Here queue is part of our internal structure struct fmm_bdev *dev

You are doing it wrong, you should be writing this as a blk-mq driver.
Not only would that fix your issue, it would also solve a host of other
issues that I'm sure your driver has since it hasn't even been
reviewed.

Generally, I have very little interest in providing support for non
open drivers. Scratch that, no interest.

-- 
Jens Axboe

RE: BIO request larger than our storage device supports in linux kernel 4.x

[Umesh Patel]

You are doing it wrong, you should be writing this as a blk-mq driver.
Not only would that fix your issue, it would also solve a host of other iss=
ues that I'm sure your driver has since it hasn't even been reviewed.




[Umesh] : This has been working since 2.x kernel to till 3.18. It has this =
issue only in 4.x.



Thanks
Western Digital Corporation (and its subsidiaries) E-mail Confidentiality N=
otice & Disclaimer:

This e-mail and any files transmitted with it may contain confidential or l=
egally privileged information of WDC and/or its affiliates, and are intende=
d solely for the use of the individual or entity to which they are addresse=
d. If you are not the intended recipient, any disclosure, copying, distribu=
tion or any action taken or omitted to be taken in reliance on it, is prohi=
bited. If you have received this e-mail in error, please notify the sender =
immediately and delete the e-mail in its entirety from your system.

Re: BIO request larger than our storage device supports in linux kernel 4.x

[Jens Axboe]

On 03/08/2017 11:20 AM, Umesh Patel wrote:
>> You are doing it wrong, you should be writing this as a blk-mq driver.
>> Not only would that fix your issue, it would also solve a host of
>> other issues that I'm sure your driver has since it hasn't even been
>> reviewed.
> 
> [Umesh] : This has been working since 2.x kernel to till 3.18. It has
> this issue only in 4.x.

And had your driver been in the kernel, it would still be working. I already
told you what is wrong, go back and read the first reply.

-- 
Jens Axboe