[PATCH 1/2] blk-flush: fix rq->flush.seq for post-flush requests

[PATCH 1/2] blk-flush: fix rq->flush.seq for post-flush requests

[chengming.zhou]

From: Chengming Zhou <zhouchengming@bytedance.com>

If the policy == (REQ_FSEQ_DATA | REQ_FSEQ_POSTFLUSH), it means that the
data sequence and post-flush sequence need to be done for this request.

The rq->flush.seq should record what sequences have been done (or don't
need to be done). So in this case, pre-flush doesn't need to be done,
we should init rq->flush.seq to REQ_FSEQ_PREFLUSH not REQ_FSEQ_POSTFLUSH.

Of course, this doesn't cause any problem in fact, since pre-flush and
post-flush sequence do the same thing for now.

But we'd better fix this value, and the next patch will depend on this
value to be correct.

Signed-off-by: Chengming Zhou <zhouchengming@bytedance.com>
---
 block/blk-flush.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/block/blk-flush.c b/block/blk-flush.c
index 4826d2d61a23..094a6adb2718 100644
--- a/block/blk-flush.c
+++ b/block/blk-flush.c
@@ -448,7 +448,7 @@ bool blk_insert_flush(struct request *rq)
 		 * the post flush, and then just pass the command on.
 		 */
 		blk_rq_init_flush(rq);
-		rq->flush.seq |= REQ_FSEQ_POSTFLUSH;
+		rq->flush.seq |= REQ_FSEQ_PREFLUSH;
 		spin_lock_irq(&fq->mq_flush_lock);
 		fq->flush_data_in_flight++;
 		spin_unlock_irq(&fq->mq_flush_lock);
-- 
2.41.0

[PATCH 2/2] blk-flush: don't need to end rq twice for non post-flush requests

[chengming.zhou]

From: Chengming Zhou <zhouchengming@bytedance.com>

Now we unconditionally blk_rq_init_flush() to replace rq->end_io to
make rq return twice back to the flush state machine for post-flush.

Obviously, non post-flush requests don't need it, they don't need to
end request twice, so they don't need to replace rq->end_io callback.
And the same for requests with the FUA bit on hardware with FUA support.

So we move blk_rq_init_flush() to REQ_FSEQ_DATA stage and only replace
rq->end_io if it needs post-flush. Otherwise, it can end like normal
request and doesn't need to return back to the flush state machine.

There are also some other good points:
1. all requests on hardware with FUA support won't have post-flush, so
   all of them don't need to end twice.

2. non post-flush requests won't have RQF_FLUSH_SEQ rq_flags set, so
   they can merge like normal requests.

3. we don't account non post-flush requests in flush_data_in_flight,
   since there is no point to defer pending flush for these requests.

Signed-off-by: Chengming Zhou <zhouchengming@bytedance.com>
---
 block/blk-flush.c | 22 ++++++++++++++++++----
 1 file changed, 18 insertions(+), 4 deletions(-)

diff --git a/block/blk-flush.c b/block/blk-flush.c
index 094a6adb2718..1b92654e8757 100644
--- a/block/blk-flush.c
+++ b/block/blk-flush.c
@@ -93,6 +93,7 @@ enum {
 
 static void blk_kick_flush(struct request_queue *q,
 			   struct blk_flush_queue *fq, blk_opf_t flags);
+static void blk_rq_init_flush(struct request *rq);
 
 static inline struct blk_flush_queue *
 blk_get_flush_queue(struct request_queue *q, struct blk_mq_ctx *ctx)
@@ -187,7 +188,15 @@ static void blk_flush_complete_seq(struct request *rq,
 		break;
 
 	case REQ_FSEQ_DATA:
-		fq->flush_data_in_flight++;
+		/*
+		 * Only for requests that need post-flush,
+		 * we need to do rq->end_io replacement trick
+		 * to return back to the flush state machine.
+		 */
+		if (!(rq->flush.seq & REQ_FSEQ_POSTFLUSH)) {
+			blk_rq_init_flush(rq);
+			fq->flush_data_in_flight++;
+		}
 		spin_lock(&q->requeue_lock);
 		list_move_tail(&rq->queuelist, &q->flush_list);
 		spin_unlock(&q->requeue_lock);
@@ -202,7 +211,13 @@ static void blk_flush_complete_seq(struct request *rq,
 		 * normal completion and end it.
 		 */
 		list_del_init(&rq->queuelist);
-		blk_flush_restore_request(rq);
+		/*
+		 * Only for requests that had rq->end_io replaced,
+		 * we need to restore rq->end_io and make it a normal
+		 * request before the second end.
+		 */
+		if (rq->rq_flags & RQF_FLUSH_SEQ)
+			blk_flush_restore_request(rq);
 		blk_mq_end_request(rq, error);
 		break;
 
@@ -389,7 +404,6 @@ static enum rq_end_io_ret mq_flush_data_end_io(struct request *rq,
 
 static void blk_rq_init_flush(struct request *rq)
 {
-	rq->flush.seq = 0;
 	rq->rq_flags |= RQF_FLUSH_SEQ;
 	rq->flush.saved_end_io = rq->end_io; /* Usually NULL */
 	rq->end_io = mq_flush_data_end_io;
@@ -424,6 +438,7 @@ bool blk_insert_flush(struct request *rq)
 	 * the request accounting.
 	 */
 	rq->cmd_flags |= REQ_SYNC;
+	rq->flush.seq = 0;
 
 	switch (policy) {
 	case 0:
@@ -458,7 +473,6 @@ bool blk_insert_flush(struct request *rq)
 		 * Mark the request as part of a flush sequence and submit it
 		 * for further processing to the flush state machine.
 		 */
-		blk_rq_init_flush(rq);
 		spin_lock_irq(&fq->mq_flush_lock);
 		blk_flush_complete_seq(rq, fq, REQ_FSEQ_ACTIONS & ~policy, 0);
 		spin_unlock_irq(&fq->mq_flush_lock);
-- 
2.41.0

Re: [PATCH 2/2] blk-flush: don't need to end rq twice for non post-flush requests

[Christoph Hellwig]

On Mon, Jul 10, 2023 at 02:47:05PM +0800, chengming.zhou@linux.dev wrote:
> From: Chengming Zhou <zhouchengming@bytedance.com>
> 
> Now we unconditionally blk_rq_init_flush() to replace rq->end_io to
> make rq return twice back to the flush state machine for post-flush.
> 
> Obviously, non post-flush requests don't need it, they don't need to
> end request twice, so they don't need to replace rq->end_io callback.
> And the same for requests with the FUA bit on hardware with FUA support.
> 
> So we move blk_rq_init_flush() to REQ_FSEQ_DATA stage and only replace
> rq->end_io if it needs post-flush. Otherwise, it can end like normal
> request and doesn't need to return back to the flush state machine.

I really like the idea behind this optimization, but I kinda hate
adding more magic to the already way too magic flush sequence.

I wonder if a better idea would be to kill the flush sequence entirely,
and just split the flush_queue into a preflush and a postflush queue.
This would remove a field from struct request and lead to more readable
code as well.

Re: [PATCH 2/2] blk-flush: don't need to end rq twice for non post-flush requests

[Chengming Zhou]

On 2023/7/10 21:33, Christoph Hellwig wrote:
> On Mon, Jul 10, 2023 at 02:47:05PM +0800, chengming.zhou@linux.dev wrote:
>> From: Chengming Zhou <zhouchengming@bytedance.com>
>>
>> Now we unconditionally blk_rq_init_flush() to replace rq->end_io to
>> make rq return twice back to the flush state machine for post-flush.
>>
>> Obviously, non post-flush requests don't need it, they don't need to
>> end request twice, so they don't need to replace rq->end_io callback.
>> And the same for requests with the FUA bit on hardware with FUA support.
>>
>> So we move blk_rq_init_flush() to REQ_FSEQ_DATA stage and only replace
>> rq->end_io if it needs post-flush. Otherwise, it can end like normal
>> request and doesn't need to return back to the flush state machine.
> 
> I really like the idea behind this optimization, but I kinda hate
> adding more magic to the already way too magic flush sequence.

Yes, agree.

> 
> I wonder if a better idea would be to kill the flush sequence entirely,
> and just split the flush_queue into a preflush and a postflush queue.
> This would remove a field from struct request and lead to more readable
> code as well.

I have thought about this for some time, it seems feasible. So I implement it
today and test it using blktests, it works.

I will send the patchset soon.

Thanks.

Re: [PATCH 1/2] blk-flush: fix rq->flush.seq for post-flush requests

[Christoph Hellwig]

On Mon, Jul 10, 2023 at 02:47:04PM +0800, chengming.zhou@linux.dev wrote:
> From: Chengming Zhou <zhouchengming@bytedance.com>
> 
> If the policy == (REQ_FSEQ_DATA | REQ_FSEQ_POSTFLUSH), it means that the
> data sequence and post-flush sequence need to be done for this request.
> 
> The rq->flush.seq should record what sequences have been done (or don't
> need to be done). So in this case, pre-flush doesn't need to be done,
> we should init rq->flush.seq to REQ_FSEQ_PREFLUSH not REQ_FSEQ_POSTFLUSH.
> 
> Of course, this doesn't cause any problem in fact, since pre-flush and
> post-flush sequence do the same thing for now.

I wonder if it really doesn't cause any problems, but the change for
sure looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

It should probably go before your other flush optimizations and maybe
grow a fixes tag.

Re: [PATCH 1/2] blk-flush: fix rq->flush.seq for post-flush requests

[Chengming Zhou]

On 2023/7/10 21:30, Christoph Hellwig wrote:
> On Mon, Jul 10, 2023 at 02:47:04PM +0800, chengming.zhou@linux.dev wrote:
>> From: Chengming Zhou <zhouchengming@bytedance.com>
>>
>> If the policy == (REQ_FSEQ_DATA | REQ_FSEQ_POSTFLUSH), it means that the
>> data sequence and post-flush sequence need to be done for this request.
>>
>> The rq->flush.seq should record what sequences have been done (or don't
>> need to be done). So in this case, pre-flush doesn't need to be done,
>> we should init rq->flush.seq to REQ_FSEQ_PREFLUSH not REQ_FSEQ_POSTFLUSH.
>>
>> Of course, this doesn't cause any problem in fact, since pre-flush and
>> post-flush sequence do the same thing for now.
> 
> I wonder if it really doesn't cause any problems, but the change for
> sure looks good:
> 
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> 
> It should probably go before your other flush optimizations and maybe
> grow a fixes tag.

Ok, will add a Fixes tag and send it as a separate patch since it's a bug fix.

Thanks.

Re: [External] Re: [PATCH 1/2] blk-flush: fix rq->flush.seq for post-flush requests

[Chengming Zhou]

On 2023/7/11 19:06, Chengming Zhou wrote:
> On 2023/7/10 21:30, Christoph Hellwig wrote:
>> On Mon, Jul 10, 2023 at 02:47:04PM +0800, chengming.zhou@linux.dev wrote:
>>> From: Chengming Zhou <zhouchengming@bytedance.com>
>>>
>>> If the policy == (REQ_FSEQ_DATA | REQ_FSEQ_POSTFLUSH), it means that the
>>> data sequence and post-flush sequence need to be done for this request.
>>>
>>> The rq->flush.seq should record what sequences have been done (or don't
>>> need to be done). So in this case, pre-flush doesn't need to be done,
>>> we should init rq->flush.seq to REQ_FSEQ_PREFLUSH not REQ_FSEQ_POSTFLUSH.
>>>
>>> Of course, this doesn't cause any problem in fact, since pre-flush and
>>> post-flush sequence do the same thing for now.
>>
>> I wonder if it really doesn't cause any problems, but the change for
>> sure looks good:
>>
>> Reviewed-by: Christoph Hellwig <hch@lst.de>
>>
>> It should probably go before your other flush optimizations and maybe
>> grow a fixes tag.
> 
> Ok, will add a Fixes tag and send it as a separate patch since it's a bug fix.
> 

Well, I should put it in that series before other flush optimizations instead.

Re: [PATCH 1/2] blk-flush: fix rq->flush.seq for post-flush requests

[Christoph Hellwig]

On Tue, Jul 11, 2023 at 07:06:20PM +0800, Chengming Zhou wrote:
> Ok, will add a Fixes tag and send it as a separate patch since it's a bug fix.

Btw, it's probably not worth resending patch 2 until we've figured out
and dealt with the SATA flush regression that Chuck reported.

Re: [PATCH 1/2] blk-flush: fix rq->flush.seq for post-flush requests

[Chengming Zhou]

On 2023/7/11 19:31, Christoph Hellwig wrote:
> On Tue, Jul 11, 2023 at 07:06:20PM +0800, Chengming Zhou wrote:
>> Ok, will add a Fixes tag and send it as a separate patch since it's a bug fix.
> 
> Btw, it's probably not worth resending patch 2 until we've figured out
> and dealt with the SATA flush regression that Chuck reported.

Ok, I will not resend patch 2. As for the patch 1, should I resend it as a separate patch
or just put it in that series [1] before other flush optimizations ?

I search on the block mail list, find the issue [2] you mentioned, will look into it too.

[1] https://lore.kernel.org/lkml/20230707093722.1338589-1-chengming.zhou@linux.dev/
[2] https://lore.kernel.org/linux-block/7A57C7AE-A51A-4254-888B-FE15CA21F9E9@oracle.com/

Re: [PATCH 1/2] blk-flush: fix rq->flush.seq for post-flush requests

[Christoph Hellwig]

On Tue, Jul 11, 2023 at 07:52:11PM +0800, Chengming Zhou wrote:
> On 2023/7/11 19:31, Christoph Hellwig wrote:
> > On Tue, Jul 11, 2023 at 07:06:20PM +0800, Chengming Zhou wrote:
> >> Ok, will add a Fixes tag and send it as a separate patch since it's a bug fix.
> > 
> > Btw, it's probably not worth resending patch 2 until we've figured out
> > and dealt with the SATA flush regression that Chuck reported.
> 
> Ok, I will not resend patch 2. As for the patch 1, should I resend it as a separate patch
> or just put it in that series [1] before other flush optimizations ?

I'd wait a bit for debugging the regression.  For the worst case we'll have
to revert the patch, which currently can be done cleanly, but can't be
with that patch.