Re: [PATCH 0/6 v3] dax: Page invalidation fixes

Re: [PATCH 0/6 v3] dax: Page invalidation fixes

[Jan Kara]

On Mon 12-12-16 17:47:02, Jan Kara wrote:
> Hello,
> 
> this is the third revision of my fixes of races when invalidating hole pages in
> DAX mappings. See changelogs for details. The series is based on my patches to
> write-protect DAX PTEs which are currently carried in mm tree. This is a hard
> dependency because we really need to closely track dirtiness (and cleanness!)
> of radix tree entries in DAX mappings in order to avoid discarding valid dirty
> bits leading to missed cache flushes on fsync(2).
> 
> The tests have passed xfstests for xfs and ext4 in DAX and non-DAX mode.
> 
> Johannes, are you OK with patch 2/6 in its current form? I'd like to push these
> patches to some tree once DAX write-protection patches are merged.  I'm hoping
> to get at least first three patches merged for 4.10-rc2... Thanks!

OK, with the final ack from Johannes and since this is mostly DAX stuff,
can we take this through NVDIMM tree and push to Linus either late in the
merge window or for -rc2? These patches require my DAX patches sitting in mm
tree so they can be included in any git tree only once those patches land
in Linus' tree (which may happen only once Dave and Ted push out their
stuff - this is the most convoluted merge window I'd ever to deal with ;-)...
Dan?

								Honza

> 
> Changes since v2:
> * Added Reviewed-by tags
> * Fixed commit message of patch 3
> * Slightly simplified dax_iomap_pmd_fault()
> * Renamed truncation functions to express better what they do
> 
> Changes since v1:
> * Rebased on top of patches in mm tree
> * Added some Reviewed-by tags
> * renamed some functions based on review feedback
> 
> 								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: [PATCH 0/6 v3] dax: Page invalidation fixes

[Dan Williams]

On Tue, Dec 13, 2016 at 3:52 AM, Jan Kara <jack@suse.cz> wrote:
> On Mon 12-12-16 17:47:02, Jan Kara wrote:
>> Hello,
>>
>> this is the third revision of my fixes of races when invalidating hole pages in
>> DAX mappings. See changelogs for details. The series is based on my patches to
>> write-protect DAX PTEs which are currently carried in mm tree. This is a hard
>> dependency because we really need to closely track dirtiness (and cleanness!)
>> of radix tree entries in DAX mappings in order to avoid discarding valid dirty
>> bits leading to missed cache flushes on fsync(2).
>>
>> The tests have passed xfstests for xfs and ext4 in DAX and non-DAX mode.
>>
>> Johannes, are you OK with patch 2/6 in its current form? I'd like to push these
>> patches to some tree once DAX write-protection patches are merged.  I'm hoping
>> to get at least first three patches merged for 4.10-rc2... Thanks!
>
> OK, with the final ack from Johannes and since this is mostly DAX stuff,
> can we take this through NVDIMM tree and push to Linus either late in the
> merge window or for -rc2? These patches require my DAX patches sitting in mm
> tree so they can be included in any git tree only once those patches land
> in Linus' tree (which may happen only once Dave and Ted push out their
> stuff - this is the most convoluted merge window I'd ever to deal with ;-)...
> Dan?
>

I like the -rc2 plan better than sending a pull request based on some
random point in the middle of the merge window. I can give Linus a
heads up in my initial nvdimm pull request for -rc1 that for
coordination purposes we'll be sending this set of follow-on DAX
cleanups for -rc2.
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: [PATCH 0/6 v3] dax: Page invalidation fixes

[Dan Williams]

On Tue, Dec 13, 2016 at 10:57 AM, Dan Williams <dan.j.williams@intel.com> wrote:
> On Tue, Dec 13, 2016 at 3:52 AM, Jan Kara <jack@suse.cz> wrote:
>> On Mon 12-12-16 17:47:02, Jan Kara wrote:
>>> Hello,
>>>
>>> this is the third revision of my fixes of races when invalidating hole pages in
>>> DAX mappings. See changelogs for details. The series is based on my patches to
>>> write-protect DAX PTEs which are currently carried in mm tree. This is a hard
>>> dependency because we really need to closely track dirtiness (and cleanness!)
>>> of radix tree entries in DAX mappings in order to avoid discarding valid dirty
>>> bits leading to missed cache flushes on fsync(2).
>>>
>>> The tests have passed xfstests for xfs and ext4 in DAX and non-DAX mode.
>>>
>>> Johannes, are you OK with patch 2/6 in its current form? I'd like to push these
>>> patches to some tree once DAX write-protection patches are merged.  I'm hoping
>>> to get at least first three patches merged for 4.10-rc2... Thanks!
>>
>> OK, with the final ack from Johannes and since this is mostly DAX stuff,
>> can we take this through NVDIMM tree and push to Linus either late in the
>> merge window or for -rc2? These patches require my DAX patches sitting in mm
>> tree so they can be included in any git tree only once those patches land
>> in Linus' tree (which may happen only once Dave and Ted push out their
>> stuff - this is the most convoluted merge window I'd ever to deal with ;-)...
>> Dan?
>>
>
> I like the -rc2 plan better than sending a pull request based on some
> random point in the middle of the merge window. I can give Linus a
> heads up in my initial nvdimm pull request for -rc1 that for
> coordination purposes we'll be sending this set of follow-on DAX
> cleanups for -rc2.

So what's still pending for -rc2? I want to be explicit about what I'm
requesting Linus be prepared to receive after -rc1. The libnvdimm pull
request is very light this time around since I ended up deferring the
device-dax-subdivision topic until 4.11 and sub-section memory hotplug
didn't make the cutoff for -mm. We can spend some of that goodwill on
your patches ;-).

I can roll them into libnvdimm-for-next now for the integration
testing coverage, rebase to -rc1 when it's out, wait for your thumbs
up on the testing and send a pull request on the 23rd.
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: [PATCH 0/6 v3] dax: Page invalidation fixes

[Dan Williams]

On Fri, Dec 16, 2016 at 5:35 PM, Dan Williams <dan.j.williams@intel.com> wrote:
> On Tue, Dec 13, 2016 at 10:57 AM, Dan Williams <dan.j.williams@intel.com> wrote:
>> On Tue, Dec 13, 2016 at 3:52 AM, Jan Kara <jack@suse.cz> wrote:
>>> On Mon 12-12-16 17:47:02, Jan Kara wrote:
>>>> Hello,
>>>>
>>>> this is the third revision of my fixes of races when invalidating hole pages in
>>>> DAX mappings. See changelogs for details. The series is based on my patches to
>>>> write-protect DAX PTEs which are currently carried in mm tree. This is a hard
>>>> dependency because we really need to closely track dirtiness (and cleanness!)
>>>> of radix tree entries in DAX mappings in order to avoid discarding valid dirty
>>>> bits leading to missed cache flushes on fsync(2).
>>>>
>>>> The tests have passed xfstests for xfs and ext4 in DAX and non-DAX mode.
>>>>
>>>> Johannes, are you OK with patch 2/6 in its current form? I'd like to push these
>>>> patches to some tree once DAX write-protection patches are merged.  I'm hoping
>>>> to get at least first three patches merged for 4.10-rc2... Thanks!
>>>
>>> OK, with the final ack from Johannes and since this is mostly DAX stuff,
>>> can we take this through NVDIMM tree and push to Linus either late in the
>>> merge window or for -rc2? These patches require my DAX patches sitting in mm
>>> tree so they can be included in any git tree only once those patches land
>>> in Linus' tree (which may happen only once Dave and Ted push out their
>>> stuff - this is the most convoluted merge window I'd ever to deal with ;-)...
>>> Dan?
>>>
>>
>> I like the -rc2 plan better than sending a pull request based on some
>> random point in the middle of the merge window. I can give Linus a
>> heads up in my initial nvdimm pull request for -rc1 that for
>> coordination purposes we'll be sending this set of follow-on DAX
>> cleanups for -rc2.
>
> So what's still pending for -rc2? I want to be explicit about what I'm
> requesting Linus be prepared to receive after -rc1. The libnvdimm pull
> request is very light this time around since I ended up deferring the
> device-dax-subdivision topic until 4.11 and sub-section memory hotplug
> didn't make the cutoff for -mm. We can spend some of that goodwill on
> your patches ;-).
>
> I can roll them into libnvdimm-for-next now for the integration
> testing coverage, rebase to -rc1 when it's out, wait for your thumbs
> up on the testing and send a pull request on the 23rd.

Sorry, I meant the 30th of December.
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: [PATCH 0/6 v3] dax: Page invalidation fixes

[Jan Kara]

On Fri 16-12-16 17:35:35, Dan Williams wrote:
> On Tue, Dec 13, 2016 at 10:57 AM, Dan Williams <dan.j.williams@intel.com> wrote:
> > On Tue, Dec 13, 2016 at 3:52 AM, Jan Kara <jack@suse.cz> wrote:
> >> On Mon 12-12-16 17:47:02, Jan Kara wrote:
> >>> Hello,
> >>>
> >>> this is the third revision of my fixes of races when invalidating hole pages in
> >>> DAX mappings. See changelogs for details. The series is based on my patches to
> >>> write-protect DAX PTEs which are currently carried in mm tree. This is a hard
> >>> dependency because we really need to closely track dirtiness (and cleanness!)
> >>> of radix tree entries in DAX mappings in order to avoid discarding valid dirty
> >>> bits leading to missed cache flushes on fsync(2).
> >>>
> >>> The tests have passed xfstests for xfs and ext4 in DAX and non-DAX mode.
> >>>
> >>> Johannes, are you OK with patch 2/6 in its current form? I'd like to push these
> >>> patches to some tree once DAX write-protection patches are merged.  I'm hoping
> >>> to get at least first three patches merged for 4.10-rc2... Thanks!
> >>
> >> OK, with the final ack from Johannes and since this is mostly DAX stuff,
> >> can we take this through NVDIMM tree and push to Linus either late in the
> >> merge window or for -rc2? These patches require my DAX patches sitting in mm
> >> tree so they can be included in any git tree only once those patches land
> >> in Linus' tree (which may happen only once Dave and Ted push out their
> >> stuff - this is the most convoluted merge window I'd ever to deal with ;-)...
> >> Dan?
> >>
> >
> > I like the -rc2 plan better than sending a pull request based on some
> > random point in the middle of the merge window. I can give Linus a
> > heads up in my initial nvdimm pull request for -rc1 that for
> > coordination purposes we'll be sending this set of follow-on DAX
> > cleanups for -rc2.
> 
> So what's still pending for -rc2? I want to be explicit about what I'm
> requesting Linus be prepared to receive after -rc1. The libnvdimm pull
> request is very light this time around since I ended up deferring the
> device-dax-subdivision topic until 4.11 and sub-section memory hotplug
> didn't make the cutoff for -mm. We can spend some of that goodwill on
> your patches ;-).

;-) So I'd like all these 6 patches to go for rc2. The first three patches
fix invalidation of exceptional DAX entries (a bug which is there for a
long time) - without these patches data loss can occur on power failure
even though user called fsync(2). The other three patches change locking of
DAX faults so that ->iomap_begin() is called in a more relaxed locking
context and we are safe to start a transaction there for ext4.

> I can roll them into libnvdimm-for-next now for the integration
> testing coverage, rebase to -rc1 when it's out, wait for your thumbs
> up on the testing and send a pull request on the 23rd.

Yup, all prerequisites are merged now so you can pick these patches up.
Thanks! Note that I'll be on vacation on Dec 23 - Jan 1.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: [PATCH 0/6 v3] dax: Page invalidation fixes

[Dan Williams]

On Mon, Dec 19, 2016 at 1:56 AM, Jan Kara <jack@suse.cz> wrote:
> On Fri 16-12-16 17:35:35, Dan Williams wrote:
>> On Tue, Dec 13, 2016 at 10:57 AM, Dan Williams <dan.j.williams@intel.com> wrote:
>> > On Tue, Dec 13, 2016 at 3:52 AM, Jan Kara <jack@suse.cz> wrote:
>> >> On Mon 12-12-16 17:47:02, Jan Kara wrote:
>> >>> Hello,
>> >>>
>> >>> this is the third revision of my fixes of races when invalidating hole pages in
>> >>> DAX mappings. See changelogs for details. The series is based on my patches to
>> >>> write-protect DAX PTEs which are currently carried in mm tree. This is a hard
>> >>> dependency because we really need to closely track dirtiness (and cleanness!)
>> >>> of radix tree entries in DAX mappings in order to avoid discarding valid dirty
>> >>> bits leading to missed cache flushes on fsync(2).
>> >>>
>> >>> The tests have passed xfstests for xfs and ext4 in DAX and non-DAX mode.
>> >>>
>> >>> Johannes, are you OK with patch 2/6 in its current form? I'd like to push these
>> >>> patches to some tree once DAX write-protection patches are merged.  I'm hoping
>> >>> to get at least first three patches merged for 4.10-rc2... Thanks!
>> >>
>> >> OK, with the final ack from Johannes and since this is mostly DAX stuff,
>> >> can we take this through NVDIMM tree and push to Linus either late in the
>> >> merge window or for -rc2? These patches require my DAX patches sitting in mm
>> >> tree so they can be included in any git tree only once those patches land
>> >> in Linus' tree (which may happen only once Dave and Ted push out their
>> >> stuff - this is the most convoluted merge window I'd ever to deal with ;-)...
>> >> Dan?
>> >>
>> >
>> > I like the -rc2 plan better than sending a pull request based on some
>> > random point in the middle of the merge window. I can give Linus a
>> > heads up in my initial nvdimm pull request for -rc1 that for
>> > coordination purposes we'll be sending this set of follow-on DAX
>> > cleanups for -rc2.
>>
>> So what's still pending for -rc2? I want to be explicit about what I'm
>> requesting Linus be prepared to receive after -rc1. The libnvdimm pull
>> request is very light this time around since I ended up deferring the
>> device-dax-subdivision topic until 4.11 and sub-section memory hotplug
>> didn't make the cutoff for -mm. We can spend some of that goodwill on
>> your patches ;-).
>
> ;-) So I'd like all these 6 patches to go for rc2. The first three patches
> fix invalidation of exceptional DAX entries (a bug which is there for a
> long time) - without these patches data loss can occur on power failure
> even though user called fsync(2). The other three patches change locking of
> DAX faults so that ->iomap_begin() is called in a more relaxed locking
> context and we are safe to start a transaction there for ext4.
>
>> I can roll them into libnvdimm-for-next now for the integration
>> testing coverage, rebase to -rc1 when it's out, wait for your thumbs
>> up on the testing and send a pull request on the 23rd.
>
> Yup, all prerequisites are merged now so you can pick these patches up.
> Thanks! Note that I'll be on vacation on Dec 23 - Jan 1.

Sounds good, the contents are now out on libnvdimm-pending awaiting
0day-run before moving them over to libnvdimm-for-next, also it's down
to 5 patches since it seems that the "dax: Fix sleep in atomic contex
in grab_mapping_entry()" change went upstream already.
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: [PATCH 0/6 v3] dax: Page invalidation fixes

[Jan Kara]

On Mon 19-12-16 13:51:53, Dan Williams wrote:
> On Mon, Dec 19, 2016 at 1:56 AM, Jan Kara <jack@suse.cz> wrote:
> > On Fri 16-12-16 17:35:35, Dan Williams wrote:
> >> On Tue, Dec 13, 2016 at 10:57 AM, Dan Williams <dan.j.williams@intel.com> wrote:
> >> > On Tue, Dec 13, 2016 at 3:52 AM, Jan Kara <jack@suse.cz> wrote:
> >> >> On Mon 12-12-16 17:47:02, Jan Kara wrote:
> >> >>> Hello,
> >> >>>
> >> >>> this is the third revision of my fixes of races when invalidating hole pages in
> >> >>> DAX mappings. See changelogs for details. The series is based on my patches to
> >> >>> write-protect DAX PTEs which are currently carried in mm tree. This is a hard
> >> >>> dependency because we really need to closely track dirtiness (and cleanness!)
> >> >>> of radix tree entries in DAX mappings in order to avoid discarding valid dirty
> >> >>> bits leading to missed cache flushes on fsync(2).
> >> >>>
> >> >>> The tests have passed xfstests for xfs and ext4 in DAX and non-DAX mode.
> >> >>>
> >> >>> Johannes, are you OK with patch 2/6 in its current form? I'd like to push these
> >> >>> patches to some tree once DAX write-protection patches are merged.  I'm hoping
> >> >>> to get at least first three patches merged for 4.10-rc2... Thanks!
> >> >>
> >> >> OK, with the final ack from Johannes and since this is mostly DAX stuff,
> >> >> can we take this through NVDIMM tree and push to Linus either late in the
> >> >> merge window or for -rc2? These patches require my DAX patches sitting in mm
> >> >> tree so they can be included in any git tree only once those patches land
> >> >> in Linus' tree (which may happen only once Dave and Ted push out their
> >> >> stuff - this is the most convoluted merge window I'd ever to deal with ;-)...
> >> >> Dan?
> >> >>
> >> >
> >> > I like the -rc2 plan better than sending a pull request based on some
> >> > random point in the middle of the merge window. I can give Linus a
> >> > heads up in my initial nvdimm pull request for -rc1 that for
> >> > coordination purposes we'll be sending this set of follow-on DAX
> >> > cleanups for -rc2.
> >>
> >> So what's still pending for -rc2? I want to be explicit about what I'm
> >> requesting Linus be prepared to receive after -rc1. The libnvdimm pull
> >> request is very light this time around since I ended up deferring the
> >> device-dax-subdivision topic until 4.11 and sub-section memory hotplug
> >> didn't make the cutoff for -mm. We can spend some of that goodwill on
> >> your patches ;-).
> >
> > ;-) So I'd like all these 6 patches to go for rc2. The first three patches
> > fix invalidation of exceptional DAX entries (a bug which is there for a
> > long time) - without these patches data loss can occur on power failure
> > even though user called fsync(2). The other three patches change locking of
> > DAX faults so that ->iomap_begin() is called in a more relaxed locking
> > context and we are safe to start a transaction there for ext4.
> >
> >> I can roll them into libnvdimm-for-next now for the integration
> >> testing coverage, rebase to -rc1 when it's out, wait for your thumbs
> >> up on the testing and send a pull request on the 23rd.
> >
> > Yup, all prerequisites are merged now so you can pick these patches up.
> > Thanks! Note that I'll be on vacation on Dec 23 - Jan 1.
> 
> Sounds good, the contents are now out on libnvdimm-pending awaiting
> 0day-run before moving them over to libnvdimm-for-next, also it's down
> to 5 patches since it seems that the "dax: Fix sleep in atomic contex
> in grab_mapping_entry()" change went upstream already.

Yes, but I've accounted for that. Checking the libnvdimm-pending branch I
see you missed "ext2: Return BH_New buffers for zeroed blocks" which was
the first patch in the series. The subject is a slight misnomer since it is
about setting IOMAP_F_NEW flag instead these days but still it is needed...
Otherwise the DAX invalidation code would not propely invalidate zero pages
in the radix tree in response to writes for ext2.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: [PATCH 0/6 v3] dax: Page invalidation fixes

[Dan Williams]

On Mon, Dec 19, 2016 at 11:59 PM, Jan Kara <jack@suse.cz> wrote:
> On Mon 19-12-16 13:51:53, Dan Williams wrote:
[..]
> Yes, but I've accounted for that. Checking the libnvdimm-pending branch I
> see you missed "ext2: Return BH_New buffers for zeroed blocks" which was
> the first patch in the series. The subject is a slight misnomer since it is
> about setting IOMAP_F_NEW flag instead these days but still it is needed...
> Otherwise the DAX invalidation code would not propely invalidate zero pages
> in the radix tree in response to writes for ext2.

Ok, thanks. Updated libnvdimm-pending pushed out.
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: [PATCH 0/6 v3] dax: Page invalidation fixes

[Dave Chinner]

On Tue, Dec 13, 2016 at 12:52:09PM +0100, Jan Kara wrote:
> On Mon 12-12-16 17:47:02, Jan Kara wrote:
> > Hello,
> > 
> > this is the third revision of my fixes of races when invalidating hole pages in
> > DAX mappings. See changelogs for details. The series is based on my patches to
> > write-protect DAX PTEs which are currently carried in mm tree. This is a hard
> > dependency because we really need to closely track dirtiness (and cleanness!)
> > of radix tree entries in DAX mappings in order to avoid discarding valid dirty
> > bits leading to missed cache flushes on fsync(2).
> > 
> > The tests have passed xfstests for xfs and ext4 in DAX and non-DAX mode.
> > 
> > Johannes, are you OK with patch 2/6 in its current form? I'd like to push these
> > patches to some tree once DAX write-protection patches are merged.  I'm hoping
> > to get at least first three patches merged for 4.10-rc2... Thanks!
> 
> OK, with the final ack from Johannes and since this is mostly DAX stuff,
> can we take this through NVDIMM tree and push to Linus either late in the
> merge window or for -rc2? These patches require my DAX patches sitting in mm
> tree so they can be included in any git tree only once those patches land
> in Linus' tree (which may happen only once Dave and Ted push out their
> stuff - this is the most convoluted merge window I'd ever to deal with ;-)...

And I'm waiting on Jens and the block tree before I send Linus
a pulllreq for all the stuff I have queued because of the conflicts
in the iomap-direct IO patches I've also got in the XFS tree... :/

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/6 v3] dax: Page invalidation fixes

[Theodore Ts'o]

On Tue, Dec 13, 2016 at 12:52:09PM +0100, Jan Kara wrote:
> OK, with the final ack from Johannes and since this is mostly DAX stuff,
> can we take this through NVDIMM tree and push to Linus either late in the
> merge window or for -rc2? These patches require my DAX patches sitting in mm
> tree so they can be included in any git tree only once those patches land
> in Linus' tree (which may happen only once Dave and Ted push out their
> stuff - this is the most convoluted merge window I'd ever to deal with ;-)...
> Dan?

I've sent out the pull request for ext4.... which includes the
dax-4.0-iomap-pmd and fscrypt branch.  Yes, convoluted.  :-)

		      	      	       	    - Ted

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>