Race in block/blk-mq-sched.c blk_mq_sched_dispatch_requests

Race in block/blk-mq-sched.c blk_mq_sched_dispatch_requests

[Gabriel Ryan]

We found a race in the block message queue for kernel v5.18-rc5 using
a race testing tool we are developing. We are reporting this race
because it appears to be potentially harmful. The race occurs in

block/blk-mq-sched.c:333 blk_mq_sched_dispatch_requests

    hctx->run++;

where multiple threads can schedule dispatch requests and increment
the request counter htctx->run simultaneously. This appears to lead to
undefined behavior where multiple conflicting updates to the hctx->run
 value could result in it not matching the number of requests that
have been scheduled with calls to blk_mq_sched_dispatch_requests.

Best,
Gabe

Re: Race in block/blk-mq-sched.c blk_mq_sched_dispatch_requests

[Jens Axboe]

On 1/17/24 1:16 PM, Gabriel Ryan wrote:
> We found a race in the block message queue for kernel v5.18-rc5 using
> a race testing tool we are developing. We are reporting this race
> because it appears to be potentially harmful. The race occurs in
> 
> block/blk-mq-sched.c:333 blk_mq_sched_dispatch_requests
> 
>     hctx->run++;
> 
> where multiple threads can schedule dispatch requests and increment
> the request counter htctx->run simultaneously. This appears to lead to
> undefined behavior where multiple conflicting updates to the hctx->run
>  value could result in it not matching the number of requests that
> have been scheduled with calls to blk_mq_sched_dispatch_requests.

I suggest you take a closer look at how that variable is actually
used.

-- 
Jens Axboe

Re: Race in block/blk-mq-sched.c blk_mq_sched_dispatch_requests

[Bart Van Assche]

On 1/17/24 12:17, Jens Axboe wrote:
> On 1/17/24 1:16 PM, Gabriel Ryan wrote:
>> We found a race in the block message queue for kernel v5.18-rc5 using
>> a race testing tool we are developing. We are reporting this race
>> because it appears to be potentially harmful. The race occurs in
>>
>> block/blk-mq-sched.c:333 blk_mq_sched_dispatch_requests
>>
>>      hctx->run++;
>>
>> where multiple threads can schedule dispatch requests and increment
>> the request counter htctx->run simultaneously. This appears to lead to
>> undefined behavior where multiple conflicting updates to the hctx->run
>> value could result in it not matching the number of requests that
>> have been scheduled with calls to blk_mq_sched_dispatch_requests.
> 
> I suggest you take a closer look at how that variable is actually
> used.

It's probably a good idea to explain this in a comment above the
code that increments hctx->runs because others may also be wondering
what the impact is of concurrent hctx->runs increments.

Thanks,

Bart.

Re: Race in block/blk-mq-sched.c blk_mq_sched_dispatch_requests

[Jens Axboe]

On 1/17/24 1:22 PM, Bart Van Assche wrote:
> On 1/17/24 12:17, Jens Axboe wrote:
>> On 1/17/24 1:16 PM, Gabriel Ryan wrote:
>>> We found a race in the block message queue for kernel v5.18-rc5 using
>>> a race testing tool we are developing. We are reporting this race
>>> because it appears to be potentially harmful. The race occurs in
>>>
>>> block/blk-mq-sched.c:333 blk_mq_sched_dispatch_requests
>>>
>>>      hctx->run++;
>>>
>>> where multiple threads can schedule dispatch requests and increment
>>> the request counter htctx->run simultaneously. This appears to lead to
>>> undefined behavior where multiple conflicting updates to the hctx->run
>>> value could result in it not matching the number of requests that
>>> have been scheduled with calls to blk_mq_sched_dispatch_requests.
>>
>> I suggest you take a closer look at how that variable is actually
>> used.
> 
> It's probably a good idea to explain this in a comment above the
> code that increments hctx->runs because others may also be wondering
> what the impact is of concurrent hctx->runs increments.

If you do a quick grep, you'll very quickly see that it's just used
for debugfs output. Being racy is not a problem. It should just get
removed, honestly, like I did with some of the other accounting some
time ago.

-- 
Jens Axboe

Re: Race in block/blk-mq-sched.c blk_mq_sched_dispatch_requests

[Gabriel Ryan]

Thank you for the response Jens, and apologies I did not realize that
the variable is only used for debug output before sending this report.

Best,
Gabe

On Wed, Jan 17, 2024 at 3:28 PM Jens Axboe <axboe@kernel.dk> wrote:
>
> On 1/17/24 1:22 PM, Bart Van Assche wrote:
> > On 1/17/24 12:17, Jens Axboe wrote:
> >> On 1/17/24 1:16 PM, Gabriel Ryan wrote:
> >>> We found a race in the block message queue for kernel v5.18-rc5 using
> >>> a race testing tool we are developing. We are reporting this race
> >>> because it appears to be potentially harmful. The race occurs in
> >>>
> >>> block/blk-mq-sched.c:333 blk_mq_sched_dispatch_requests
> >>>
> >>>      hctx->run++;
> >>>
> >>> where multiple threads can schedule dispatch requests and increment
> >>> the request counter htctx->run simultaneously. This appears to lead to
> >>> undefined behavior where multiple conflicting updates to the hctx->run
> >>> value could result in it not matching the number of requests that
> >>> have been scheduled with calls to blk_mq_sched_dispatch_requests.
> >>
> >> I suggest you take a closer look at how that variable is actually
> >> used.
> >
> > It's probably a good idea to explain this in a comment above the
> > code that increments hctx->runs because others may also be wondering
> > what the impact is of concurrent hctx->runs increments.
>
> If you do a quick grep, you'll very quickly see that it's just used
> for debugfs output. Being racy is not a problem. It should just get
> removed, honestly, like I did with some of the other accounting some
> time ago.
>
> --
> Jens Axboe
>
>

Re: Race in block/blk-mq-sched.c blk_mq_sched_dispatch_requests

[Jens Axboe]

On 1/17/24 1:36 PM, Gabriel Ryan wrote:
> Thank you for the response Jens, and apologies I did not realize that
> the variable is only used for debug output before sending this report.

No worries, but please do some due diligence on reports from a tool
before blindly sending them out. This will help reduce time waste on
both sides.

I'd also strongly recommend you upgrade to a recent kernel rather than
something that is 2 years old.

-- 
Jens Axboe