[RFC][PATCH v3 4/6] aio: Don't inherit aio ring memory at fork

[RFC][PATCH v3 4/6] aio: Don't inherit aio ring memory at fork

[KOSAKI Motohiro]

AIO folks, Am I missing anything?

===============
Subject: [RFC][PATCH] aio: Don't inherit aio ring memory at fork

Currently, mm_struct::ioctx_list member isn't copyed at fork. IOW aio context don't inherit at fork.
but only ring memory inherited. that's strange.

This patch mark DONTFORK to ring-memory too.
In addition, This patch has good side effect. it also fix "get_user_pages() vs fork" problem.

I think "man fork" also sould be changed. it only say

       *  The child does not inherit outstanding asynchronous I/O operations from
          its parent (aio_read(3), aio_write(3)).

but aio_context_t (return value of io_setup(2)) also don't inherit in current implementaion.


Cc: Jeff Moyer <jmoyer@redhat.com>
Cc: Zach Brown <zach.brown@oracle.com>
Cc: Jens Axboe <jens.axboe@oracle.com>
Cc: linux-fsdevel@vger.kernel.org
Cc: linux-api@vger.kernel.org,
Signed-off-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
---
 fs/aio.c |    8 ++++++++
 1 file changed, 8 insertions(+)

Index: b/fs/aio.c
===================================================================
--- a/fs/aio.c	2009-04-12 23:33:59.000000000 +0900
+++ b/fs/aio.c	2009-04-13 02:56:05.000000000 +0900
@@ -106,6 +106,7 @@ static int aio_setup_ring(struct kioctx 
 	unsigned nr_events = ctx->max_reqs;
 	unsigned long size;
 	int nr_pages;
+	int ret;
 
 	/* Compensate for the ring buffer's head/tail overlap entry */
 	nr_events += 2;	/* 1 is required, 2 for good luck */
@@ -140,6 +141,13 @@ static int aio_setup_ring(struct kioctx 
 		return -EAGAIN;
 	}
 
+	/*
+	 * aio context doesn't inherit while fork. (see mm_init())
+	 * Then, aio ring also mark DONTFORK.
+	 */
+	ret = sys_madvise(info->mmap_base, info->mmap_size, MADV_DONTFORK);
+	BUG_ON(ret);
+
 	dprintk("mmap address: 0x%08lx\n", info->mmap_base);
 	info->nr_pages = get_user_pages(current, ctx->mm,
 					info->mmap_base, nr_pages, 


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH v3 4/6] aio: Don't inherit aio ring memory at fork

[Andrea Arcangeli]

On Tue, Apr 14, 2009 at 03:20:20PM +0900, KOSAKI Motohiro wrote:
> In addition, This patch has good side effect. it also fix "get_user_pages() vs fork" problem.

Yes, patches like 3/6, 4/6, and 6/6 are the side effect of not fixing
the core race in gup and spreading the new rwsem around the gup users,
instead of sticking to a page-granular PG_flag touched at the same
time atomic_inc runs on page->_count.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH v3 4/6] aio: Don't inherit aio ring memory at fork

[Jeff Moyer]

KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com> writes:

> AIO folks, Am I missing anything?
>
> ===============
> Subject: [RFC][PATCH] aio: Don't inherit aio ring memory at fork
>
> Currently, mm_struct::ioctx_list member isn't copyed at fork. IOW aio context don't inherit at fork.
> but only ring memory inherited. that's strange.
>
> This patch mark DONTFORK to ring-memory too.

Well, given that clearly nobody relies on io contexts being copied to
the child, I think it's okay to make this change.  I think the current
behaviour violates the principal of least surprise, but I'm having a
hard time getting upset about that.  ;)

> In addition, This patch has good side effect. it also fix
> "get_user_pages() vs fork" problem.

Hmm, I don't follow you, here.  As I understand it, the get_user_pages
vs. fork problem has to do with the pages used for the actual I/O, not
the pages used to store the completion data.  So, could you elaborate a
bit on what you mean by the above statement?

> I think "man fork" also sould be changed. it only say
>
>        *  The child does not inherit outstanding asynchronous I/O operations from
>           its parent (aio_read(3), aio_write(3)).
> but aio_context_t (return value of io_setup(2)) also don't inherit in current implementaion.

I can certainly make that change, as I have other changes I need to push
to Michael, anyway.

Cheers,
Jeff

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH v3 4/6] aio: Don't inherit aio ring memory at fork

[KOSAKI Motohiro]

Hi!

> KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com> writes:
> 
> > AIO folks, Am I missing anything?
> >
> > ===============
> > Subject: [RFC][PATCH] aio: Don't inherit aio ring memory at fork
> >
> > Currently, mm_struct::ioctx_list member isn't copyed at fork. IOW aio context don't inherit at fork.
> > but only ring memory inherited. that's strange.
> >
> > This patch mark DONTFORK to ring-memory too.
> 
> Well, given that clearly nobody relies on io contexts being copied to
> the child, I think it's okay to make this change.  I think the current
> behaviour violates the principal of least surprise, but I'm having a
> hard time getting upset about that.  ;)

ok.
So, Can I get your Acked-by?

> > In addition, This patch has good side effect. it also fix
> > "get_user_pages() vs fork" problem.
> 
> Hmm, I don't follow you, here.  As I understand it, the get_user_pages
> vs. fork problem has to do with the pages used for the actual I/O, not
> the pages used to store the completion data.  So, could you elaborate a
> bit on what you mean by the above statement?

No.

The problem is, get_user_pages() increment page_count only.
but VM page-fault logic don't care page_count. (it only care page::_mapcount)
Then, fork and pagefault can change virtual-physical relationship although
get_user_pages() is called.

drawback worst aio scenario here
-----------------------------------------------------------------------
io_setup() and gup			inc page_count

fork					inc mapcount
					and make write-protect to pte

write ring from userland(*)		page fault and
					COW break.
					parent process get copyed page and
					child get original page owner-ship.

kmap and memcpy from kernel		change child page. (it mean data lost)

(*) Is this happend?

MADV_DONTFORK or down_read(mmap_sem) or down_read(mm_pinned_sem) 
or copy-at-fork mecanism(=Nick/Andrea patch) solve it.



> > I think "man fork" also sould be changed. it only say
> >
> >        *  The child does not inherit outstanding asynchronous I/O operations from
> >           its parent (aio_read(3), aio_write(3)).
> > but aio_context_t (return value of io_setup(2)) also don't inherit in current implementaion.
> 
> I can certainly make that change, as I have other changes I need to push
> to Michael, anyway.

thanks.

Re: [RFC][PATCH v3 4/6] aio: Don't inherit aio ring memory at fork

[Jeff Moyer]

KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com> writes:

> Hi!
>
>> KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com> writes:
>> 
>> > AIO folks, Am I missing anything?
>> >
>> > ===============
>> > Subject: [RFC][PATCH] aio: Don't inherit aio ring memory at fork
>> >
>> > Currently, mm_struct::ioctx_list member isn't copyed at fork. IOW aio context don't inherit at fork.
>> > but only ring memory inherited. that's strange.
>> >
>> > This patch mark DONTFORK to ring-memory too.
>> 
>> Well, given that clearly nobody relies on io contexts being copied to
>> the child, I think it's okay to make this change.  I think the current
>> behaviour violates the principal of least surprise, but I'm having a
>> hard time getting upset about that.  ;)
>
> ok.
> So, Can I get your Acked-by?

I have more comments below.

>> > In addition, This patch has good side effect. it also fix
>> > "get_user_pages() vs fork" problem.
>> 
>> Hmm, I don't follow you, here.  As I understand it, the get_user_pages
>> vs. fork problem has to do with the pages used for the actual I/O, not
>> the pages used to store the completion data.  So, could you elaborate a
>> bit on what you mean by the above statement?
>
> No.
>
> The problem is, get_user_pages() increment page_count only.
> but VM page-fault logic don't care page_count. (it only care page::_mapcount)
> Then, fork and pagefault can change virtual-physical relationship although
> get_user_pages() is called.
>
> drawback worst aio scenario here
> -----------------------------------------------------------------------
> io_setup() and gup			inc page_count
>
> fork					inc mapcount
> 					and make write-protect to pte
>
> write ring from userland(*)		page fault and
> 					COW break.
> 					parent process get copyed page and
> 					child get original page owner-ship.
>
> kmap and memcpy from kernel		change child page. (it mean data lost)
>
> (*) Is this happend?

I guess it's possible, but I don't know of any programs that do this.

> MADV_DONTFORK or down_read(mmap_sem) or down_read(mm_pinned_sem) 
> or copy-at-fork mecanism(=Nick/Andrea patch) solve it.

OK, thanks for the explanation.

+	/*
+	 * aio context doesn't inherit while fork. (see mm_init())
+	 * Then, aio ring also mark DONTFORK.
+	 */

Would you mind if I did some word-smithing on that comment?  Something
like:
	/*
	 * The io_context is not inherited by the child after fork()
         * (see mm_init).  Therefore, it makes little sense for the
         * completion ring to be inherited.
         */

+	ret = sys_madvise(info->mmap_base, info->mmap_size, MADV_DONTFORK);
+	BUG_ON(ret);
+

It appears there's no other way to set the VM_DONTCOPY flag, so I guess
calling sys_madvise is fine.  I'm not sure I agree with the BUG_ON(ret),
however, as EAGAIN may be feasible.

So, fix that up and you can add my reviewed-by.  I think you should push
this patch independent of the other patches in this series.

>> > I think "man fork" also sould be changed. it only say
>> >
>> >        *  The child does not inherit outstanding asynchronous I/O operations from
>> >           its parent (aio_read(3), aio_write(3)).
>> > but aio_context_t (return value of io_setup(2)) also don't inherit in current implementaion.
>> 
>> I can certainly make that change, as I have other changes I need to push
>> to Michael, anyway.
>
> thanks.

No problem.  As you know, I've already sent a patch for this.

Cheers,
Jeff

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC][PATCH v3 4/6] aio: Don't inherit aio ring memory at fork

[KOSAKI Motohiro]

Hi

> > drawback worst aio scenario here
> > -----------------------------------------------------------------------
> > io_setup() and gup			inc page_count
> >
> > fork					inc mapcount
> > 					and make write-protect to pte
> >
> > write ring from userland(*)		page fault and
> > 					COW break.
> > 					parent process get copyed page and
> > 					child get original page owner-ship.
> >
> > kmap and memcpy from kernel		change child page. (it mean data lost)
> >
> > (*) Is this happend?
> 
> I guess it's possible, but I don't know of any programs that do this.

Yup, I also think this isn't happen in real world.

> 
> > MADV_DONTFORK or down_read(mmap_sem) or down_read(mm_pinned_sem) 
> > or copy-at-fork mecanism(=Nick/Andrea patch) solve it.
> 
> OK, thanks for the explanation.
> 
> +	/*
> +	 * aio context doesn't inherit while fork. (see mm_init())
> +	 * Then, aio ring also mark DONTFORK.
> +	 */
> 
> Would you mind if I did some word-smithing on that comment?  Something
> like:
> 	/*
> 	 * The io_context is not inherited by the child after fork()
>          * (see mm_init).  Therefore, it makes little sense for the
>          * completion ring to be inherited.
>          */
> 
> +	ret = sys_madvise(info->mmap_base, info->mmap_size, MADV_DONTFORK);
> +	BUG_ON(ret);
> +
> 
> It appears there's no other way to set the VM_DONTCOPY flag, so I guess
> calling sys_madvise is fine.  I'm not sure I agree with the BUG_ON(ret),
> however, as EAGAIN may be feasible.
> 
> So, fix that up and you can add my reviewed-by.  I think you should push
> this patch independent of the other patches in this series.

Done :)




--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>