[GIT PULL] md-next 20230622

[GIT PULL] md-next 20230622

[Song Liu]

Hi Jens, 

Please consider pulling the following changes for md-next on top of your
for-6.5/block branch. The major changes are:

1. Deprecate bitmap file support, by Christoph Hellwig;
2. Fix deadlock with md sync thread, by Yu Kuai;
3. Refactor md io accounting, by Yu Kuai.

Thanks,
Song


The following changes since commit 648fa60fa7de3ca6f6303e1721591ad73def9cf0:

  block: don't return -EINVAL for not found names in devt_from_devname (2023-06-22 09:09:33 -0600)

are available in the Git repository at:

  https://git.kernel.org/pub/scm/linux/kernel/git/song/md.git tags/md-next-20230622

for you to fetch changes up to 41fb72ee7eeda723e619c6918dffaf05a55fc7dd:

  md/md-faulty: enable io accounting (2023-06-22 22:28:27 -0700)

----------------------------------------------------------------
Christoph Hellwig (11):
      md-bitmap: set BITMAP_WRITE_ERROR in write_sb_page
      md-bitmap: initialize variables at declaration time in md_bitmap_file_unmap
      md-bitmap: use %pD to print the file name in md_bitmap_file_kick
      md-bitmap: split file writes into a separate helper
      md-bitmap: rename read_page to read_file_page
      md-bitmap: refactor md_bitmap_init_from_disk
      md-bitmap: cleanup read_sb_page
      md-bitmap: account for mddev->bitmap_info.offset in read_sb_page
      md-bitmap: don't use ->index for pages backing the bitmap file
      md: make bitmap file support optional
      md: deprecate bitmap file support

Li Nan (1):
      md/raid10: fix the condition to call bio_end_io_acct()

Song Liu (1):
      md: use mddev->external to select holder in export_rdev()

Yu Kuai (17):
      md/raid1-10: fix casting from randomized structure in raid1_submit_write()
      Revert "md: unlock mddev before reap sync_thread in action_store"
      md: refactor action_store() for 'idle' and 'frozen'
      md: add a mutex to synchronize idle and frozen in action_store()
      md: refactor idle/frozen_sync_thread() to fix deadlock
      md: wake up 'resync_wait' at last in md_reap_sync_thread()
      md: enhance checking in md_check_recovery()
      md: fix 'delete_mutex' deadlock
      raid10: avoid spin_lock from fastpath from raid10_unplug()
      md: move initialization and destruction of 'io_acct_set' to md.c
      md: also clone new io if io accounting is disabled
      raid5: fix missing io accounting in raid5_align_endio()
      md/raid1: switch to use md_account_bio() for io accounting
      md/raid10: switch to use md_account_bio() for io accounting
      md/md-multipath: enable io accounting
      md/md-linear: enable io accounting
      md/md-faulty: enable io accounting

 drivers/md/Kconfig        |  10 +++++
 drivers/md/dm-raid.c      |   1 -
 drivers/md/md-bitmap.c    | 338 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------------------------------------------------------------------
 drivers/md/md-bitmap.h    |   1 +
 drivers/md/md-faulty.c    |   2 +
 drivers/md/md-linear.c    |   1 +
 drivers/md/md-multipath.c |   1 +
 drivers/md/md.c           | 245 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------------------------------------------------
 drivers/md/md.h           |  15 +++----
 drivers/md/raid0.c        |  16 +-------
 drivers/md/raid1-10.c     |   2 +-
 drivers/md/raid1.c        |  14 +++----
 drivers/md/raid1.h        |   1 -
 drivers/md/raid10.c       |  24 ++++++------
 drivers/md/raid10.h       |   1 -
 drivers/md/raid5.c        |  70 +++++++++------------------------
 16 files changed, 372 insertions(+), 370 deletions(-)

Re: [GIT PULL] md-next 20230622

[Jens Axboe]

On 6/22/23 11:48?PM, Song Liu wrote:
> Hi Jens, 
> 
> Please consider pulling the following changes for md-next on top of your
> for-6.5/block branch. The major changes are:
> 
> 1. Deprecate bitmap file support, by Christoph Hellwig;
> 2. Fix deadlock with md sync thread, by Yu Kuai;
> 3. Refactor md io accounting, by Yu Kuai.

This is quite a lot on the day that I prepare pull requests for the
merge window... I've said this many times before, but just to state this
in completeness, maybe it'll benefit others too:

1) Major changes for the next release should be sent to me _at least_ 1
   week prior to the merge window opening. That way it gets some decent
   soak time in linux-next before heading upstream.

2) Minor fixes, either for major pulls that already went into my next
   branch or just fixes in general, can be sent anytime and I'll shove
   them into the appropriate branch.

When bigger stuff gets sent this late, then I have two choices: reject
them and tell you to send it in for the next version, or setup a new
branch just for this so I can send it to Linus in a later pull in the
merge window. Neither of those two options are great - the first one
delays you by a release, the second one creates more churn and hassle
for me.

-- 
Jens Axboe

Re: [GIT PULL] md-next 20230622

[Song Liu]

Hi Jens, 

I am so sorry for this problem. 

> On Jun 23, 2023, at 7:12 AM, Jens Axboe <axboe@kernel.dk> wrote:
> 
> On 6/22/23 11:48?PM, Song Liu wrote:
>> Hi Jens, 
>> 
>> Please consider pulling the following changes for md-next on top of your
>> for-6.5/block branch. The major changes are:
>> 
>> 1. Deprecate bitmap file support, by Christoph Hellwig;
>> 2. Fix deadlock with md sync thread, by Yu Kuai;
>> 3. Refactor md io accounting, by Yu Kuai.
> 
> This is quite a lot on the day that I prepare pull requests for the
> merge window... I've said this many times before, but just to state this
> in completeness, maybe it'll benefit others too:
> 
> 1) Major changes for the next release should be sent to me _at least_ 1
>   week prior to the merge window opening. That way it gets some decent
>   soak time in linux-next before heading upstream.

I am aware of the rule. A couple reasons caused a late PR this time:

1. Set #1 and set #3 are relatively new, especially set #3, which was
   first sent earlier this week. Set #2 is older, but there was more
   discussions on it until recently. (It is still my fault not pushing
   on set #2 sooner). 

2. I wasn't very sure whether there will be a rc8. The announcement for
   rc7 didn't state it clearly. (Shall I assume there is no rc8 unless
   Linus states it clearly?)

3. I was hoping to group more patches into one PR. I guess this was the 
   biggest mistake, especially when it is close to the merge window.

> 2) Minor fixes, either for major pulls that already went into my next
>   branch or just fixes in general, can be sent anytime and I'll shove
>   them into the appropriate branch.
> 
> When bigger stuff gets sent this late, then I have two choices: reject
> them and tell you to send it in for the next version, or setup a new
> branch just for this so I can send it to Linus in a later pull in the
> merge window. Neither of those two options are great - the first one
> delays you by a release, the second one creates more churn and hassle
> for me.

I will prepare another PR with just fixes. 

Christoph, 

Please let me know if you need set #1 (deprecate file bitmap) to 
unblock other work. Otherwise, we will delay it until 6.6. 

Thanks,
Song

Re: [GIT PULL] md-next 20230622

[Jens Axboe]

On 6/23/23 9:08?AM, Song Liu wrote:
> Hi Jens, 
> 
> I am so sorry for this problem. 
> 
>> On Jun 23, 2023, at 7:12 AM, Jens Axboe <axboe@kernel.dk> wrote:
>>
>> On 6/22/23 11:48?PM, Song Liu wrote:
>>> Hi Jens, 
>>>
>>> Please consider pulling the following changes for md-next on top of your
>>> for-6.5/block branch. The major changes are:
>>>
>>> 1. Deprecate bitmap file support, by Christoph Hellwig;
>>> 2. Fix deadlock with md sync thread, by Yu Kuai;
>>> 3. Refactor md io accounting, by Yu Kuai.
>>
>> This is quite a lot on the day that I prepare pull requests for the
>> merge window... I've said this many times before, but just to state this
>> in completeness, maybe it'll benefit others too:
>>
>> 1) Major changes for the next release should be sent to me _at least_ 1
>>   week prior to the merge window opening. That way it gets some decent
>>   soak time in linux-next before heading upstream.
> 
> I am aware of the rule. A couple reasons caused a late PR this time:

Then please be explicit when sending out a pull request like this on why
it's late. Otherwise it just looks like a normal pull request, which it
is not. If your original pull request had any kind of explanation on why
so much is being sent so late, then we would not be having this followup
at all...

> 1. Set #1 and set #3 are relatively new, especially set #3, which was
>    first sent earlier this week. Set #2 is older, but there was more
>    discussions on it until recently. (It is still my fault not pushing
>    on set #2 sooner). 
> 
> 2. I wasn't very sure whether there will be a rc8. The announcement for
>    rc7 didn't state it clearly. (Shall I assume there is no rc8 unless
>    Linus states it clearly?)
> 
> 3. I was hoping to group more patches into one PR. I guess this was the 
>    biggest mistake, especially when it is close to the merge window.

Most of the time Linus will be explicit about _maybe_ doing an -rc8, but
it doesn't really change the rule that I should have bigger stuff in the
week between rc6 and rc7. When -rc7 is cut, my for-next branches should
be basically baked and done at that point, and then post -rc7 just fixes
for existing code. If people stick to that rc6-7 timing, then it'll
always work, regardless of an -rc8 happening or not.

Even when Linus has -rc8 musings in his later rc posts, it's not a given
that they will happen. 

>> 2) Minor fixes, either for major pulls that already went into my next
>>   branch or just fixes in general, can be sent anytime and I'll shove
>>   them into the appropriate branch.
>>
>> When bigger stuff gets sent this late, then I have two choices: reject
>> them and tell you to send it in for the next version, or setup a new
>> branch just for this so I can send it to Linus in a later pull in the
>> merge window. Neither of those two options are great - the first one
>> delays you by a release, the second one creates more churn and hassle
>> for me.
> 
> I will prepare another PR with just fixes. 
> 
> Christoph, 
> 
> Please let me know if you need set #1 (deprecate file bitmap) to 
> unblock other work. Otherwise, we will delay it until 6.6. 

I've done a for-6.5/block-late and put your stuff there, but it can be
dropped very easily. It doesn't really matter if Christoph's stuff can
get pushed, it's still a lot of commits late. So regardless of that, the
only real difference with a new PR would be that we'd drop some bits.
It'd still go into a late branch, as it is indeed late.

-- 
Jens Axboe

Re: [GIT PULL] md-next 20230622

[Jens Axboe]

On 6/23/23 9:16?AM, Jens Axboe wrote:
>> I will prepare another PR with just fixes. 
>>
>> Christoph, 
>>
>> Please let me know if you need set #1 (deprecate file bitmap) to 
>> unblock other work. Otherwise, we will delay it until 6.6. 
> 
> I've done a for-6.5/block-late and put your stuff there, but it can be
> dropped very easily. It doesn't really matter if Christoph's stuff can
> get pushed, it's still a lot of commits late. So regardless of that, the
> only real difference with a new PR would be that we'd drop some bits.
> It'd still go into a late branch, as it is indeed late.

On second thought, yes let's go your suggested route. Make a branch with
JUST the fixes, and send them my way. Anything else will have to wait
for 6.6 at this point. I'll drop my late branch for now.

-- 
Jens Axboe

Re: [GIT PULL] md-next 20230622

[Song Liu]

> On Jun 23, 2023, at 8:16 AM, Jens Axboe <axboe@kernel.dk> wrote:
> 
> On 6/23/23 9:08?AM, Song Liu wrote:
>> Hi Jens, 
>> 
>> I am so sorry for this problem. 
>> 
>>> On Jun 23, 2023, at 7:12 AM, Jens Axboe <axboe@kernel.dk> wrote:
>>> 
>>> On 6/22/23 11:48?PM, Song Liu wrote:
>>>> Hi Jens, 
>>>> 
>>>> Please consider pulling the following changes for md-next on top of your
>>>> for-6.5/block branch. The major changes are:
>>>> 
>>>> 1. Deprecate bitmap file support, by Christoph Hellwig;
>>>> 2. Fix deadlock with md sync thread, by Yu Kuai;
>>>> 3. Refactor md io accounting, by Yu Kuai.
>>> 
>>> This is quite a lot on the day that I prepare pull requests for the
>>> merge window... I've said this many times before, but just to state this
>>> in completeness, maybe it'll benefit others too:
>>> 
>>> 1) Major changes for the next release should be sent to me _at least_ 1
>>>  week prior to the merge window opening. That way it gets some decent
>>>  soak time in linux-next before heading upstream.
>> 
>> I am aware of the rule. A couple reasons caused a late PR this time:
> 
> Then please be explicit when sending out a pull request like this on why
> it's late. Otherwise it just looks like a normal pull request, which it
> is not. If your original pull request had any kind of explanation on why
> so much is being sent so late, then we would not be having this followup
> at all...

Indeed! I should have explained the case in the first place. 

> 
>> 1. Set #1 and set #3 are relatively new, especially set #3, which was
>>   first sent earlier this week. Set #2 is older, but there was more
>>   discussions on it until recently. (It is still my fault not pushing
>>   on set #2 sooner). 
>> 
>> 2. I wasn't very sure whether there will be a rc8. The announcement for
>>   rc7 didn't state it clearly. (Shall I assume there is no rc8 unless
>>   Linus states it clearly?)
>> 
>> 3. I was hoping to group more patches into one PR. I guess this was the 
>>   biggest mistake, especially when it is close to the merge window.
> 
> Most of the time Linus will be explicit about _maybe_ doing an -rc8, but
> it doesn't really change the rule that I should have bigger stuff in the
> week between rc6 and rc7. When -rc7 is cut, my for-next branches should
> be basically baked and done at that point, and then post -rc7 just fixes
> for existing code. If people stick to that rc6-7 timing, then it'll
> always work, regardless of an -rc8 happening or not.
> 
> Even when Linus has -rc8 musings in his later rc posts, it's not a given
> that they will happen. 

Thanks for the explanation. I will get big things ready before rc7 in 
the future. 

> 
>>> 2) Minor fixes, either for major pulls that already went into my next
>>>  branch or just fixes in general, can be sent anytime and I'll shove
>>>  them into the appropriate branch.
>>> 
>>> When bigger stuff gets sent this late, then I have two choices: reject
>>> them and tell you to send it in for the next version, or setup a new
>>> branch just for this so I can send it to Linus in a later pull in the
>>> merge window. Neither of those two options are great - the first one
>>> delays you by a release, the second one creates more churn and hassle
>>> for me.
>> 
>> I will prepare another PR with just fixes. 
>> 
>> Christoph, 
>> 
>> Please let me know if you need set #1 (deprecate file bitmap) to 
>> unblock other work. Otherwise, we will delay it until 6.6. 
> 
> I've done a for-6.5/block-late and put your stuff there, but it can be
> dropped very easily. It doesn't really matter if Christoph's stuff can
> get pushed, it's still a lot of commits late. So regardless of that, the
> only real difference with a new PR would be that we'd drop some bits.
> It'd still go into a late branch, as it is indeed late.

Thanks for taking them to the late branch. If it is dropped, shall I
1. Delay bigger sets until 6.6;
2. Send fixes in a separate PR (after 6.5-rc1)?

Thanks,
Song

Re: [GIT PULL] md-next 20230622

[Song Liu]

> On Jun 23, 2023, at 8:41 AM, Jens Axboe <axboe@kernel.dk> wrote:
> 
> On 6/23/23 9:16?AM, Jens Axboe wrote:
>>> I will prepare another PR with just fixes. 
>>> 
>>> Christoph, 
>>> 
>>> Please let me know if you need set #1 (deprecate file bitmap) to 
>>> unblock other work. Otherwise, we will delay it until 6.6. 
>> 
>> I've done a for-6.5/block-late and put your stuff there, but it can be
>> dropped very easily. It doesn't really matter if Christoph's stuff can
>> get pushed, it's still a lot of commits late. So regardless of that, the
>> only real difference with a new PR would be that we'd drop some bits.
>> It'd still go into a late branch, as it is indeed late.
> 
> On second thought, yes let's go your suggested route. Make a branch with
> JUST the fixes, and send them my way. Anything else will have to wait
> for 6.6 at this point. I'll drop my late branch for now.

Sounds good! Here is the updated pull request with only minor fixes. 
Three of the fixes are for patches in the for-6.5/block branch:

 - md: use mddev->external to select holder in export_rdev()
 - md/raid1-10: fix casting from randomized structure in raid1_submit_write()
 - md: fix 'delete_mutex' deadlock

Thanks,
Song



The following changes since commit fcaa174a9c995cf0af3967e55644a1543ea07e36:

  scsi/sg: don't grab scsi host module reference (2023-06-23 08:28:18 -0600)

are available in the Git repository at:

  https://git.kernel.org/pub/scm/linux/kernel/git/song/md.git tags/md-next-20230623

for you to fetch changes up to a8d5fdd4d2702d0c7ec125bd3bbce3fc589afa67:

  raid10: avoid spin_lock from fastpath from raid10_unplug() (2023-06-23 09:41:50 -0700)

----------------------------------------------------------------
Li Nan (1):
      md/raid10: fix the condition to call bio_end_io_acct()

Song Liu (1):
      md: use mddev->external to select holder in export_rdev()

Yu Kuai (3):
      md/raid1-10: fix casting from randomized structure in raid1_submit_write()
      md: fix 'delete_mutex' deadlock
      raid10: avoid spin_lock from fastpath from raid10_unplug()

 drivers/md/md.c       | 32 +++++++++++---------------------
 drivers/md/md.h       |  4 +---
 drivers/md/raid1-10.c |  2 +-
 drivers/md/raid10.c   |  6 +++---
 4 files changed, 16 insertions(+), 28 deletions(-)

Re: [GIT PULL] md-next 20230622

[Jens Axboe]

On 6/23/23 11:03?AM, Song Liu wrote:
> 
> 
>> On Jun 23, 2023, at 8:41 AM, Jens Axboe <axboe@kernel.dk> wrote:
>>
>> On 6/23/23 9:16?AM, Jens Axboe wrote:
>>>> I will prepare another PR with just fixes. 
>>>>
>>>> Christoph, 
>>>>
>>>> Please let me know if you need set #1 (deprecate file bitmap) to 
>>>> unblock other work. Otherwise, we will delay it until 6.6. 
>>>
>>> I've done a for-6.5/block-late and put your stuff there, but it can be
>>> dropped very easily. It doesn't really matter if Christoph's stuff can
>>> get pushed, it's still a lot of commits late. So regardless of that, the
>>> only real difference with a new PR would be that we'd drop some bits.
>>> It'd still go into a late branch, as it is indeed late.
>>
>> On second thought, yes let's go your suggested route. Make a branch with
>> JUST the fixes, and send them my way. Anything else will have to wait
>> for 6.6 at this point. I'll drop my late branch for now.
> 
> Sounds good! Here is the updated pull request with only minor fixes. 
> Three of the fixes are for patches in the for-6.5/block branch:
> 
>  - md: use mddev->external to select holder in export_rdev()
>  - md/raid1-10: fix casting from randomized structure in raid1_submit_write()
>  - md: fix 'delete_mutex' deadlock

Thanks, added to the for-6.5/block-late branch.

-- 
Jens Axboe

Re: [GIT PULL] md-next 20230622

[Christoph Hellwig]

On Fri, Jun 23, 2023 at 03:08:52PM +0000, Song Liu wrote:
> Please let me know if you need set #1 (deprecate file bitmap) to 
> unblock other work. Otherwise, we will delay it until 6.6. 

It was intended so that we don't have to make all of md depend on
the new config option for buffers heads I plan to introduce.  So it's
a bit of a pity that we won't have it for 6.5, but not a deal breaker.

Re: [GIT PULL] md-next 20230622

[Song Liu]

Hi Christoph,

On Mon, Jun 26, 2023 at 8:51 AM Christoph Hellwig <hch@lst.de> wrote:
>
> On Fri, Jun 23, 2023 at 03:08:52PM +0000, Song Liu wrote:
> > Please let me know if you need set #1 (deprecate file bitmap) to
> > unblock other work. Otherwise, we will delay it until 6.6.
>
> It was intended so that we don't have to make all of md depend on
> the new config option for buffers heads I plan to introduce.  So it's
> a bit of a pity that we won't have it for 6.5, but not a deal breaker.

Thanks for the confirmation!

Song