[git pull] device mapper changes for 4.19

[git pull] device mapper changes for 4.19

[Mike Snitzer]

Hi Linus,

The following changes since commit d72e90f33aa4709ebecc5005562f52335e106a60:

  Linux 4.18-rc6 (2018-07-22 14:12:20 -0700)

are available in the Git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git tags/for-4.19/dm-changes

for you to fetch changes up to 1e1132ea21da6d7be92a72195204379c819cb70b:

  dm writecache: fix a crash due to reading past end of dirty_bitmap (2018-08-16 13:43:01 -0400)

Please pull, thanks!
Mike

----------------------------------------------------------------
- A couple stable fixes for the DM writecache target.

- A stable fix for the DM cache target that fixes the potential for data
  corruption after an unclean shutdown of a cache device using writeback
  mode.

- Update DM integrity target to allow the metadata to be stored on a
  separate device from data.

- Fix DM kcopyd and the snapshot target to cond_resched() where
  appropriate and be more efficient with processing completed work.

- A few fixes and improvements for DM crypt.

- Add DM delay target feature to configure delay of flushes independent
  of writes.

- Update DM thin-provisioning target to include metadata_low_watermark
  threshold in pool status.

- Fix stale DM thin-provisioning Documentation.

----------------------------------------------------------------
Andy Grover (1):
      dm thin: include metadata_low_watermark threshold in pool status

David Jeffery (1):
      dm snapshot: improve performance by switching out_of_order_list to rbtree

Hou Tao (1):
      dm thin: stop no_space_timeout worker when switching to write-mode

Ilya Dryomov (1):
      dm cache metadata: set dirty on all cache blocks after a crash

John Pittman (2):
      dm cache: only allow a single io_mode cache feature to be requested
      dm kcopyd: avoid softlockup in run_complete_job

Kees Cook (1):
      dm crypt: convert essiv from ahash to shash

Mike Snitzer (4):
      dm thin: update stale "Status" Documentation
      dm kcopyd: return void from dm_kcopyd_copy()
      dm cache metadata: save in-core policy_hint_size to on-disk superblock
      dm snapshot: remove stale FIXME in snapshot_map()

Mikulas Patocka (15):
      dm delay: refactor repetitive code
      dm delay: add flush as a third class of IO
      dm integrity: change 'suspending' variable from bool to int
      dm integrity: decouple common code in dm_integrity_map_continue()
      dm integrity: implement fair range locks
      dm integrity: report provided data sectors in the status
      dm integrity: add ic->start in get_data_sector()
      dm integrity: allow separate metadata device
      dm integrity: use version 2 for separate metadata
      dm integrity: flush journal on suspend when using separate metadata device
      dm integrity: recalculate checksums on creation
      dm crypt: use wake_up_process() instead of a wait queue
      dm writecache: report start_sector in status line
      dm crypt: don't decrease device limits
      dm writecache: fix a crash due to reading past end of dirty_bitmap

 Documentation/device-mapper/delay.txt             |   3 +-
 Documentation/device-mapper/dm-integrity.txt      |   4 +
 Documentation/device-mapper/thin-provisioning.txt |  20 +-
 drivers/md/dm-cache-metadata.c                    |  13 +-
 drivers/md/dm-cache-target.c                      |  35 +-
 drivers/md/dm-crypt.c                             |  66 ++-
 drivers/md/dm-delay.c                             | 249 +++++------
 drivers/md/dm-integrity.c                         | 501 ++++++++++++++++++----
 drivers/md/dm-kcopyd.c                            |  18 +-
 drivers/md/dm-raid1.c                             |  17 +-
 drivers/md/dm-snap.c                              |  41 +-
 drivers/md/dm-thin.c                              |  31 +-
 drivers/md/dm-writecache.c                        |   8 +-
 drivers/md/dm-zoned-reclaim.c                     |   6 +-
 include/linux/dm-kcopyd.h                         |  12 +-
 15 files changed, 690 insertions(+), 334 deletions(-)

Re: [git pull] device mapper changes for 4.19

[Linus Torvalds]

This is only very weakly related to this pull request, but when I look
at the diffstat, I note that outside of the Documentation changes, it
looks like this:

On Thu, Aug 16, 2018 at 1:04 PM Mike Snitzer <snitzer@redhat.com> wrote:
>
>  drivers/md/dm-cache-metadata.c                    |  13 +-
>  drivers/md/dm-cache-target.c                      |  35 +-
>  drivers/md/dm-crypt.c                             |  66 ++-
>  drivers/md/dm-delay.c                             | 249 +++++------
>  drivers/md/dm-integrity.c                         | 501 ++++++++++++++++++----
>  drivers/md/dm-kcopyd.c                            |  18 +-
>  drivers/md/dm-raid1.c                             |  17 +-
>  drivers/md/dm-snap.c                              |  41 +-
>  drivers/md/dm-thin.c                              |  31 +-
>  drivers/md/dm-writecache.c                        |   8 +-
>  drivers/md/dm-zoned-reclaim.c                     |   6 +-
>  include/linux/dm-kcopyd.h                         |  12 +-

and what stands out to me is that "include/linux/" file.

It turns out that I don't think it should be in include/linux/ at all,
because it seems entirely internal to drivers/md.

A quick grep shows that nothing outside of drivers/md/ includes that
header file.

So I wonder if you would mind just moving that header file to
drivers/md/, and changing the

   #include <linux/dm-kcopyd.h>

into just

   #include "dm-kopyd.h"

instead?

The reason I care (weakly) is just that I end up looking at diffstats
a lot during the merge window, and it's always nice for me when I see
"this pull request only changes things internal to that particular
subsystem". Then I don't need to  worry as much about any interactions
with any other pulls.

And immediately when there are changes to the main include directory,
that changes, and I start going "ok, who else includes this that I
need to make sure wasn't impacted?"

Right now it's noticeable mainly because I happen to be on my laptop,
so a full "make allmodconfig" takes an hour. Instead, I ended up
checking manually whether rthat include file update could affect
anybody else, and just did the drivers/md/ subdirectory build.

Stupid reason, I know, but I thought I'd spend a couple of minutes
writing an email and maybe next time around I'll have one less thing
to check..

               Linus

Re: [git pull] device mapper changes for 4.19

[Mike Snitzer]

On Fri, Aug 17 2018 at  1:01pm -0400,
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> This is only very weakly related to this pull request, but when I look
> at the diffstat, I note that outside of the Documentation changes, it
> looks like this:
> 
> On Thu, Aug 16, 2018 at 1:04 PM Mike Snitzer <snitzer@redhat.com> wrote:
> >
> >  drivers/md/dm-cache-metadata.c                    |  13 +-
> >  drivers/md/dm-cache-target.c                      |  35 +-
> >  drivers/md/dm-crypt.c                             |  66 ++-
> >  drivers/md/dm-delay.c                             | 249 +++++------
> >  drivers/md/dm-integrity.c                         | 501 ++++++++++++++++++----
> >  drivers/md/dm-kcopyd.c                            |  18 +-
> >  drivers/md/dm-raid1.c                             |  17 +-
> >  drivers/md/dm-snap.c                              |  41 +-
> >  drivers/md/dm-thin.c                              |  31 +-
> >  drivers/md/dm-writecache.c                        |   8 +-
> >  drivers/md/dm-zoned-reclaim.c                     |   6 +-
> >  include/linux/dm-kcopyd.h                         |  12 +-
> 
> and what stands out to me is that "include/linux/" file.
> 
> It turns out that I don't think it should be in include/linux/ at all,
> because it seems entirely internal to drivers/md.
> 
> A quick grep shows that nothing outside of drivers/md/ includes that
> header file.
> 
> So I wonder if you would mind just moving that header file to
> drivers/md/, and changing the
> 
>    #include <linux/dm-kcopyd.h>
> 
> into just
> 
>    #include "dm-kopyd.h"
> 
> instead?

Sure.

> The reason I care (weakly) is just that I end up looking at diffstats
> a lot during the merge window, and it's always nice for me when I see
> "this pull request only changes things internal to that particular
> subsystem". Then I don't need to  worry as much about any interactions
> with any other pulls.

I'll review all DM headers in include/linux/ and move accordingly.

Only thing that made restrict some movement is concern for 3rd party DM
modules that aren't upstream (yet).  But I may just need to be the
bearer of bad news for those.

I should be able to get these tested changes to you early next week.
I'll defer to you on whether to take them for 4.19 or wait.

Thanks,
Mike

Re: [git pull] device mapper changes for 4.19

[Linus Torvalds]

On Fri, Aug 17, 2018 at 11:13 AM Mike Snitzer <snitzer@redhat.com> wrote:
>
> I should be able to get these tested changes to you early next week.
> I'll defer to you on whether to take them for 4.19 or wait.

There is absolutely no hurry about this - the comment really was more
about a "future development" than anything that needs to happen
immediately.

And only do it if it makes sense for your workflow, because it's
really not a big deal for me, it's just a detail that i tended to
notice.

            Linus

Re: [git pull] device mapper changes for 4.19

[Mikulas Patocka]

On Fri, 17 Aug 2018, Linus Torvalds wrote:

> This is only very weakly related to this pull request, but when I look
> at the diffstat, I note that outside of the Documentation changes, it
> looks like this:
> 
> On Thu, Aug 16, 2018 at 1:04 PM Mike Snitzer <snitzer@redhat.com> wrote:
> >
> >  drivers/md/dm-cache-metadata.c                    |  13 +-
> >  drivers/md/dm-cache-target.c                      |  35 +-
> >  drivers/md/dm-crypt.c                             |  66 ++-
> >  drivers/md/dm-delay.c                             | 249 +++++------
> >  drivers/md/dm-integrity.c                         | 501 ++++++++++++++++++----
> >  drivers/md/dm-kcopyd.c                            |  18 +-
> >  drivers/md/dm-raid1.c                             |  17 +-
> >  drivers/md/dm-snap.c                              |  41 +-
> >  drivers/md/dm-thin.c                              |  31 +-
> >  drivers/md/dm-writecache.c                        |   8 +-
> >  drivers/md/dm-zoned-reclaim.c                     |   6 +-
> >  include/linux/dm-kcopyd.h                         |  12 +-
> 
> and what stands out to me is that "include/linux/" file.
> 
> It turns out that I don't think it should be in include/linux/ at all,
> because it seems entirely internal to drivers/md.

dm-kcopyd.h (and dm-io.h and dm-bufio.h) are not restricted to device 
mapper. These subsystems were written by device mapper developers because 
they needed to reuse the same functionality in multiple dm targets, but 
these subsystems do not depend on device mapper functionality (with little 
changes, they could be compiled independently of dm) and could be used by 
any kernel code (for example filesystems).

I think that perhaps these files could be moved to some generic place like 
block/lib/, so that others could start using them too.

Mikulas