Re: [PATCH 3/4] block: move queue enter logic into blk_mq_submit_bio()

[Jens Axboe <axboe@kernel.dk>]

On 11/4/21 3:10 AM, Christoph Hellwig wrote:
> On Wed, Nov 03, 2021 at 12:32:21PM -0600, Jens Axboe wrote:
>> Retain the old logic for the fops based submit, but for our internal
>> blk_mq_submit_bio(), move the queue entering logic into the core
>> function itself.
> 
> Can you explain the why?  I guess you want to skip the extra reference
> for the cached requests now that they already have one.  But please
> state that, and explain why it is a fix, as to me it just seems like
> another little optimization.

It's just pointless to grab double references, and counter productive
too.

>> We need to be a bit careful if going into the scheduler, as a scheduler
>> or queue mappings can arbitrarily change before we have entered the queue.
>> Have the bio scheduler mapping do that separately, it's a very cheap
>> operation compared to actually doing merging locking and lookups.
> 
> So just don't do the merges for cache requets and side step this
> extra bio_queue_enter for that case?

I'd be fine with that, but it's a bit of a chicken and egg situation as
we don't know. I guess we could move the plugged request check earlier,
and just bypass merging there. Though that makes it a special case
thing, and it's generally useful now. Not sure that would be a good
idea.

>> -	if (unlikely(bio_queue_enter(bio) != 0))
>> -		return;
>> -
>>  	if (!submit_bio_checks(bio) || !blk_crypto_bio_prep(&bio))
>> -		goto queue_exit;
>> +		return;
> 
> This is broken, we really ant the submit checks under freeze
> protection to make sure the parameters can't be changed underneath
> us.

Which parameters are you worried about in submit_bio_checks()? I don't
immediately see anything that would make me worry about it.

>> +static inline bool blk_mq_queue_enter(struct request_queue *q, struct bio *bio)
>> +{
>> +	if (!blk_try_enter_queue(q, false) && bio_queue_enter(bio))
>> +		return false;
>> +	return true;
>> +}
> 
> This looks weird, as blk_try_enter_queue is already called by
> bio_queue_enter.

It's just for avoiding a pointless call into bio_queue_enter(), which
isn't needed it blk_try_enter_queue() is successful. The latter is short
and small and can be inlined, while bio_queue_enter() is a lot bigger.

>>  	} else {
>>  		struct blk_mq_alloc_data data = {
>>  			.q		= q,
>> @@ -2528,6 +2534,11 @@ void blk_mq_submit_bio(struct bio *bio)
>>  			.cmd_flags	= bio->bi_opf,
>>  		};
>>  
>> +		if (unlikely(!blk_mq_queue_enter(q, bio)))
>> +			return;
>> +
>> +		rq_qos_throttle(q, bio);
>> +
> 
> At some point the code in this !cached branch really needs to move
> into a helper..

Like in the next patch?

-- 
Jens Axboe