[PATCH v2 2/2] blk-throttle: fix wrong initialization in case of dm device

[PATCH v2 2/2] blk-throttle: fix wrong initialization in case of dm device

[Joseph Qi]

From: Joseph Qi <qijiang.qj@alibaba-inc.com>

dm device set QUEUE_FLAG_NONROT in resume, which is after register
queue. That is to mean, the previous initialization in
blk_throtl_register_queue is wrong in this case.
Fix it by checking and then updating the info during root tg
initialization as we don't have a better choice.

Signed-off-by: Joseph Qi <qijiang.qj@alibaba-inc.com>
---
 block/blk-throttle.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index bf52035..6d6b220 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -541,6 +541,23 @@ static void throtl_pd_init(struct blkg_policy_data *pd)
 	if (cgroup_subsys_on_dfl(io_cgrp_subsys) && blkg->parent)
 		sq->parent_sq = &blkg_to_tg(blkg->parent)->service_queue;
 	tg->td = td;
+
+	/*
+	 * dm device set QUEUE_FLAG_NONROT in resume, which is after resister
+	 * queue, so the previous initialization is wrong in this case. Check
+	 * and update it here.
+	 */
+	if (blk_queue_nonrot(blkg->q) &&
+	    td->filtered_latency != LATENCY_FILTERED_SSD) {
+		int i;
+
+		td->throtl_slice = DFL_THROTL_SLICE_SSD;
+		td->filtered_latency = LATENCY_FILTERED_SSD;
+		for (i = 0; i < LATENCY_BUCKET_SIZE; i++) {
+			td->avg_buckets[READ][i].latency = 0;
+			td->avg_buckets[WRITE][i].latency = 0;
+		}
+	}
 }
 
 /*
-- 
1.9.4

Re: [PATCH v2 2/2] blk-throttle: fix wrong initialization in case of dm device

[Shaohua Li]

On Mon, Nov 20, 2017 at 10:00:27AM +0800, Joseph Qi wrote:
> From: Joseph Qi <qijiang.qj@alibaba-inc.com>
> 
> dm device set QUEUE_FLAG_NONROT in resume, which is after register
> queue. That is to mean, the previous initialization in
> blk_throtl_register_queue is wrong in this case.
> Fix it by checking and then updating the info during root tg
> initialization as we don't have a better choice.
> 
> Signed-off-by: Joseph Qi <qijiang.qj@alibaba-inc.com>
> ---
>  block/blk-throttle.c | 17 +++++++++++++++++
>  1 file changed, 17 insertions(+)
> 
> diff --git a/block/blk-throttle.c b/block/blk-throttle.c
> index bf52035..6d6b220 100644
> --- a/block/blk-throttle.c
> +++ b/block/blk-throttle.c
> @@ -541,6 +541,23 @@ static void throtl_pd_init(struct blkg_policy_data *pd)
>  	if (cgroup_subsys_on_dfl(io_cgrp_subsys) && blkg->parent)
>  		sq->parent_sq = &blkg_to_tg(blkg->parent)->service_queue;
>  	tg->td = td;
> +
> +	/*
> +	 * dm device set QUEUE_FLAG_NONROT in resume, which is after resister
> +	 * queue, so the previous initialization is wrong in this case. Check
> +	 * and update it here.
> +	 */
> +	if (blk_queue_nonrot(blkg->q) &&
> +	    td->filtered_latency != LATENCY_FILTERED_SSD) {
> +		int i;
> +
> +		td->throtl_slice = DFL_THROTL_SLICE_SSD;

if CONFIG_BLK_DEV_THROTTLING_LOW isn't not set, we use old slice, can you do
the same thing here? Otherwise,
Reviewed-by: Shaohua Li <shli@kernel.org>

> +		td->filtered_latency = LATENCY_FILTERED_SSD;
> +		for (i = 0; i < LATENCY_BUCKET_SIZE; i++) {
> +			td->avg_buckets[READ][i].latency = 0;
> +			td->avg_buckets[WRITE][i].latency = 0;
> +		}
> +	}
>  }
>  
>  /*
> -- 
> 1.9.4

Re: [PATCH v2 2/2] blk-throttle: fix wrong initialization in case of dm device

[Mike Snitzer]

On Sun, Nov 19, 2017 at 9:00 PM, Joseph Qi <jiangqi903@gmail.com> wrote:
> From: Joseph Qi <qijiang.qj@alibaba-inc.com>
>
> dm device set QUEUE_FLAG_NONROT in resume, which is after register
> queue. That is to mean, the previous initialization in
> blk_throtl_register_queue is wrong in this case.
> Fix it by checking and then updating the info during root tg
> initialization as we don't have a better choice.

Given DM motivated this change, curious why you didn't send this to dm-devel?

In any case, not sure why you need to reference "resume".  Very few
people will appreciate that detail.

Better to just say: DM device sets QUEUE_FLAG_NONROT after the queue
is registered.

> Signed-off-by: Joseph Qi <qijiang.qj@alibaba-inc.com>
> ---
>  block/blk-throttle.c | 17 +++++++++++++++++
>  1 file changed, 17 insertions(+)
>
> diff --git a/block/blk-throttle.c b/block/blk-throttle.c
> index bf52035..6d6b220 100644
> --- a/block/blk-throttle.c
> +++ b/block/blk-throttle.c
> @@ -541,6 +541,23 @@ static void throtl_pd_init(struct blkg_policy_data *pd)
>         if (cgroup_subsys_on_dfl(io_cgrp_subsys) && blkg->parent)
>                 sq->parent_sq = &blkg_to_tg(blkg->parent)->service_queue;
>         tg->td = td;
> +
> +       /*
> +        * dm device set QUEUE_FLAG_NONROT in resume, which is after resister
> +        * queue, so the previous initialization is wrong in this case. Check
> +        * and update it here.
> +        */

typo: s/resister/register/
But again, best to say:
DM device sets QUEUE_FLAG_NONROT after the queue is registered, ...

Also, an alternative to your patch would be to have DM's
dm_table_set_restrictions() call a blk-throttle function to setup
blk-throttle after it is done setting queue flags?
Saves all other drivers from having to run this 2 stage initialization
code.. just a thought.

Mike

Re: [PATCH v2 2/2] blk-throttle: fix wrong initialization in case of dm device

[Shaohua Li]

On Mon, Nov 20, 2017 at 04:02:03PM -0500, Mike Snitzer wrote:
> On Sun, Nov 19, 2017 at 9:00 PM, Joseph Qi <jiangqi903@gmail.com> wrote:
> > From: Joseph Qi <qijiang.qj@alibaba-inc.com>
> >
> > dm device set QUEUE_FLAG_NONROT in resume, which is after register
> > queue. That is to mean, the previous initialization in
> > blk_throtl_register_queue is wrong in this case.
> > Fix it by checking and then updating the info during root tg
> > initialization as we don't have a better choice.
> 
> Given DM motivated this change, curious why you didn't send this to dm-devel?
> 
> In any case, not sure why you need to reference "resume".  Very few
> people will appreciate that detail.
> 
> Better to just say: DM device sets QUEUE_FLAG_NONROT after the queue
> is registered.
> 
> > Signed-off-by: Joseph Qi <qijiang.qj@alibaba-inc.com>
> > ---
> >  block/blk-throttle.c | 17 +++++++++++++++++
> >  1 file changed, 17 insertions(+)
> >
> > diff --git a/block/blk-throttle.c b/block/blk-throttle.c
> > index bf52035..6d6b220 100644
> > --- a/block/blk-throttle.c
> > +++ b/block/blk-throttle.c
> > @@ -541,6 +541,23 @@ static void throtl_pd_init(struct blkg_policy_data *pd)
> >         if (cgroup_subsys_on_dfl(io_cgrp_subsys) && blkg->parent)
> >                 sq->parent_sq = &blkg_to_tg(blkg->parent)->service_queue;
> >         tg->td = td;
> > +
> > +       /*
> > +        * dm device set QUEUE_FLAG_NONROT in resume, which is after resister
> > +        * queue, so the previous initialization is wrong in this case. Check
> > +        * and update it here.
> > +        */
> 
> typo: s/resister/register/
> But again, best to say:
> DM device sets QUEUE_FLAG_NONROT after the queue is registered, ...
> 
> Also, an alternative to your patch would be to have DM's
> dm_table_set_restrictions() call a blk-throttle function to setup
> blk-throttle after it is done setting queue flags?
> Saves all other drivers from having to run this 2 stage initialization
> code.. just a thought.

Though the patch is a workaround, I'd prefer limiting the code into
blk-throttle itself. Populating it to drivers is a layer violation to me. The
initialization only runs once, so I think it's ok.

Thanks,
Shaohua

Re: [PATCH v2 2/2] blk-throttle: fix wrong initialization in case of dm device

[Joseph Qi]

Hi Mike,

On 17/11/21 05:02, Mike Snitzer wrote:
> On Sun, Nov 19, 2017 at 9:00 PM, Joseph Qi <jiangqi903@gmail.com> wrote:
>> From: Joseph Qi <qijiang.qj@alibaba-inc.com>
>>
>> dm device set QUEUE_FLAG_NONROT in resume, which is after register
>> queue. That is to mean, the previous initialization in
>> blk_throtl_register_queue is wrong in this case.
>> Fix it by checking and then updating the info during root tg
>> initialization as we don't have a better choice.
> 
> Given DM motivated this change, curious why you didn't send this to dm-devel?
> 
Since this is io.low specifically, I don't want the change in device
mapper layer. But I do agree with you that I have to cc dm-devel as
well. Sorry for missing that.

> In any case, not sure why you need to reference "resume".  Very few
> people will appreciate that detail.
> 
> Better to just say: DM device sets QUEUE_FLAG_NONROT after the queue
> is registered.
> 
Thanks for your advice. I will update and resend it.

Thanks,
Joseph

>> Signed-off-by: Joseph Qi <qijiang.qj@alibaba-inc.com>
>> ---
>>  block/blk-throttle.c | 17 +++++++++++++++++
>>  1 file changed, 17 insertions(+)
>>
>> diff --git a/block/blk-throttle.c b/block/blk-throttle.c
>> index bf52035..6d6b220 100644
>> --- a/block/blk-throttle.c
>> +++ b/block/blk-throttle.c
>> @@ -541,6 +541,23 @@ static void throtl_pd_init(struct blkg_policy_data *pd)
>>         if (cgroup_subsys_on_dfl(io_cgrp_subsys) && blkg->parent)
>>                 sq->parent_sq = &blkg_to_tg(blkg->parent)->service_queue;
>>         tg->td = td;
>> +
>> +       /*
>> +        * dm device set QUEUE_FLAG_NONROT in resume, which is after resister
>> +        * queue, so the previous initialization is wrong in this case. Check
>> +        * and update it here.
>> +        */
> 
> typo: s/resister/register/
> But again, best to say:
> DM device sets QUEUE_FLAG_NONROT after the queue is registered, ...
> 
> Also, an alternative to your patch would be to have DM's
> dm_table_set_restrictions() call a blk-throttle function to setup
> blk-throttle after it is done setting queue flags?
> Saves all other drivers from having to run this 2 stage initialization
> code.. just a thought.
> 
> Mike
> 

Re: [PATCH v2 2/2] blk-throttle: fix wrong initialization in case of dm device

[Joseph Qi]

On 17/11/21 01:15, Shaohua Li wrote:
> On Mon, Nov 20, 2017 at 10:00:27AM +0800, Joseph Qi wrote:
>> From: Joseph Qi <qijiang.qj@alibaba-inc.com>
>>
>> dm device set QUEUE_FLAG_NONROT in resume, which is after register
>> queue. That is to mean, the previous initialization in
>> blk_throtl_register_queue is wrong in this case.
>> Fix it by checking and then updating the info during root tg
>> initialization as we don't have a better choice.
>>
>> Signed-off-by: Joseph Qi <qijiang.qj@alibaba-inc.com>
>> ---
>>  block/blk-throttle.c | 17 +++++++++++++++++
>>  1 file changed, 17 insertions(+)
>>
>> diff --git a/block/blk-throttle.c b/block/blk-throttle.c
>> index bf52035..6d6b220 100644
>> --- a/block/blk-throttle.c
>> +++ b/block/blk-throttle.c
>> @@ -541,6 +541,23 @@ static void throtl_pd_init(struct blkg_policy_data *pd)
>>  	if (cgroup_subsys_on_dfl(io_cgrp_subsys) && blkg->parent)
>>  		sq->parent_sq = &blkg_to_tg(blkg->parent)->service_queue;
>>  	tg->td = td;
>> +
>> +	/*
>> +	 * dm device set QUEUE_FLAG_NONROT in resume, which is after resister
>> +	 * queue, so the previous initialization is wrong in this case. Check
>> +	 * and update it here.
>> +	 */
>> +	if (blk_queue_nonrot(blkg->q) &&
>> +	    td->filtered_latency != LATENCY_FILTERED_SSD) {
>> +		int i;
>> +
>> +		td->throtl_slice = DFL_THROTL_SLICE_SSD;
> 
> if CONFIG_BLK_DEV_THROTTLING_LOW isn't not set, we use old slice, can you do
> the same thing here? Otherwise,
> Reviewed-by: Shaohua Li <shli@kernel.org>
> 
Sure, I will update and resend it. Thanks for pointing this out.

Thanks,
Joseph

>> +		td->filtered_latency = LATENCY_FILTERED_SSD;
>> +		for (i = 0; i < LATENCY_BUCKET_SIZE; i++) {
>> +			td->avg_buckets[READ][i].latency = 0;
>> +			td->avg_buckets[WRITE][i].latency = 0;
>> +		}
>> +	}
>>  }
>>  
>>  /*
>> -- 
>> 1.9.4

Re: [PATCH v2 2/2] blk-throttle: fix wrong initialization in case of dm device

[Mike Snitzer]

On Mon, Nov 20 2017 at  4:23pm -0500,
Shaohua Li <shli@kernel.org> wrote:

> On Mon, Nov 20, 2017 at 04:02:03PM -0500, Mike Snitzer wrote:
> > On Sun, Nov 19, 2017 at 9:00 PM, Joseph Qi <jiangqi903@gmail.com> wrote:
> > > From: Joseph Qi <qijiang.qj@alibaba-inc.com>
> > >
> > > dm device set QUEUE_FLAG_NONROT in resume, which is after register
> > > queue. That is to mean, the previous initialization in
> > > blk_throtl_register_queue is wrong in this case.
> > > Fix it by checking and then updating the info during root tg
> > > initialization as we don't have a better choice.
> > 
> > Given DM motivated this change, curious why you didn't send this to dm-devel?
> > 
> > In any case, not sure why you need to reference "resume".  Very few
> > people will appreciate that detail.
> > 
> > Better to just say: DM device sets QUEUE_FLAG_NONROT after the queue
> > is registered.
> > 
> > > Signed-off-by: Joseph Qi <qijiang.qj@alibaba-inc.com>
> > > ---
> > >  block/blk-throttle.c | 17 +++++++++++++++++
> > >  1 file changed, 17 insertions(+)
> > >
> > > diff --git a/block/blk-throttle.c b/block/blk-throttle.c
> > > index bf52035..6d6b220 100644
> > > --- a/block/blk-throttle.c
> > > +++ b/block/blk-throttle.c
> > > @@ -541,6 +541,23 @@ static void throtl_pd_init(struct blkg_policy_data *pd)
> > >         if (cgroup_subsys_on_dfl(io_cgrp_subsys) && blkg->parent)
> > >                 sq->parent_sq = &blkg_to_tg(blkg->parent)->service_queue;
> > >         tg->td = td;
> > > +
> > > +       /*
> > > +        * dm device set QUEUE_FLAG_NONROT in resume, which is after resister
> > > +        * queue, so the previous initialization is wrong in this case. Check
> > > +        * and update it here.
> > > +        */
> > 
> > typo: s/resister/register/
> > But again, best to say:
> > DM device sets QUEUE_FLAG_NONROT after the queue is registered, ...
> > 
> > Also, an alternative to your patch would be to have DM's
> > dm_table_set_restrictions() call a blk-throttle function to setup
> > blk-throttle after it is done setting queue flags?
> > Saves all other drivers from having to run this 2 stage initialization
> > code.. just a thought.
> 
> Though the patch is a workaround, I'd prefer limiting the code into
> blk-throttle itself. Populating it to drivers is a layer violation to me. The
> initialization only runs once, so I think it's ok.

I fail to see how this is a "workaround".  Whatever you call it, the
blk-throttle code needs to cope with the fact that DM's request_queue
setup is different than other block devices.

If having DM call a function that blk-throttle provides (code that is in
block/blk-throttle.c) I fail to see the layer violation.  No more a
layer violation than any of the other block core code that block drivers
call to setup the request_queue's capabilities.

But I'm not well-versed on blk-throttle code (hence DM coming up lacking
relative to it).  SO I'll defer to you guys to sort it out how you think
best.

Thanks,
Mike