* [PATCH] ceph: fix corruption when using page_count 0 page in rbd
@ 2014-04-23 4:35 Chunwei Chen
2014-05-06 16:22 ` Ilya Dryomov
0 siblings, 1 reply; 6+ messages in thread
From: Chunwei Chen @ 2014-04-23 4:35 UTC (permalink / raw)
To: ceph-devel; +Cc: linux-kernel, Chunwei Chen, Sage Weil, Yehuda Sadeh, stable
It has been reported that using ZFSonLinux on rbd will result in memory
corruption. The bug report can be found here:
https://github.com/zfsonlinux/spl/issues/241
http://tracker.ceph.com/issues/7790
The reason is that ZFS will send pages with page_count 0 into rbd, which in
turns send them to tcp_sendpage. However, tcp_sendpage cannot deal with
page_count 0, as it will do get_page and put_page, and erroneously free the
page.
This type of issue has been noted before, and handled in iscsi, drbd,
etc. So, rbd should also handle this. This fix address this issue by fall back
to slower sendmsg when page_count 0 detected.
Cc: Sage Weil <sage@inktank.com>
Cc: Yehuda Sadeh <yehuda@inktank.com>
Cc: stable@vger.kernel.org
Signed-off-by: Chunwei Chen <tuxoko@gmail.com>
---
net/ceph/messenger.c | 20 +++++++++++++++++++-
1 file changed, 19 insertions(+), 1 deletion(-)
diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c
index 4f55f9c..9a964e7 100644
--- a/net/ceph/messenger.c
+++ b/net/ceph/messenger.c
@@ -557,7 +557,7 @@ static int ceph_tcp_sendmsg(struct socket *sock, struct kvec *iov,
return r;
}
-static int ceph_tcp_sendpage(struct socket *sock, struct page *page,
+static int __ceph_tcp_sendpage(struct socket *sock, struct page *page,
int offset, size_t size, bool more)
{
int flags = MSG_DONTWAIT | MSG_NOSIGNAL | (more ? MSG_MORE : MSG_EOR);
@@ -570,6 +570,24 @@ static int ceph_tcp_sendpage(struct socket *sock, struct page *page,
return ret;
}
+static int ceph_tcp_sendpage(struct socket *sock, struct page *page,
+ int offset, size_t size, bool more)
+{
+ int ret;
+ struct kvec iov;
+
+ /* sendpage cannot properly handle pages with page_count == 0,
+ * we need to fallback to sendmsg if that's the case */
+ if (page_count(page) >= 1)
+ return __ceph_tcp_sendpage(sock, page, offset, size, more);
+
+ iov.iov_base = kmap(page) + offset;
+ iov.iov_len = size;
+ ret = ceph_tcp_sendmsg(sock, &iov, 1, size, more);
+ kunmap(page);
+
+ return ret;
+}
/*
* Shutdown/close the socket for the given connection.
--
1.8.5.2
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [PATCH] ceph: fix corruption when using page_count 0 page in rbd
2014-04-23 4:35 [PATCH] ceph: fix corruption when using page_count 0 page in rbd Chunwei Chen
@ 2014-05-06 16:22 ` Ilya Dryomov
2014-05-06 16:31 ` Sage Weil
0 siblings, 1 reply; 6+ messages in thread
From: Ilya Dryomov @ 2014-05-06 16:22 UTC (permalink / raw)
To: Chunwei Chen; +Cc: Ceph Development, Linux Kernel Mailing List, Sage Weil
On Wed, Apr 23, 2014 at 8:35 AM, Chunwei Chen <tuxoko@gmail.com> wrote:
> It has been reported that using ZFSonLinux on rbd will result in memory
> corruption. The bug report can be found here:
>
> https://github.com/zfsonlinux/spl/issues/241
> http://tracker.ceph.com/issues/7790
>
> The reason is that ZFS will send pages with page_count 0 into rbd, which in
> turns send them to tcp_sendpage. However, tcp_sendpage cannot deal with
> page_count 0, as it will do get_page and put_page, and erroneously free the
> page.
>
> This type of issue has been noted before, and handled in iscsi, drbd,
> etc. So, rbd should also handle this. This fix address this issue by fall back
> to slower sendmsg when page_count 0 detected.
>
> Cc: Sage Weil <sage@inktank.com>
> Cc: Yehuda Sadeh <yehuda@inktank.com>
> Cc: stable@vger.kernel.org
> Signed-off-by: Chunwei Chen <tuxoko@gmail.com>
> ---
> net/ceph/messenger.c | 20 +++++++++++++++++++-
> 1 file changed, 19 insertions(+), 1 deletion(-)
>
> diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c
> index 4f55f9c..9a964e7 100644
> --- a/net/ceph/messenger.c
> +++ b/net/ceph/messenger.c
> @@ -557,7 +557,7 @@ static int ceph_tcp_sendmsg(struct socket *sock, struct kvec *iov,
> return r;
> }
>
> -static int ceph_tcp_sendpage(struct socket *sock, struct page *page,
> +static int __ceph_tcp_sendpage(struct socket *sock, struct page *page,
> int offset, size_t size, bool more)
> {
> int flags = MSG_DONTWAIT | MSG_NOSIGNAL | (more ? MSG_MORE : MSG_EOR);
> @@ -570,6 +570,24 @@ static int ceph_tcp_sendpage(struct socket *sock, struct page *page,
> return ret;
> }
>
> +static int ceph_tcp_sendpage(struct socket *sock, struct page *page,
> + int offset, size_t size, bool more)
> +{
> + int ret;
> + struct kvec iov;
> +
> + /* sendpage cannot properly handle pages with page_count == 0,
> + * we need to fallback to sendmsg if that's the case */
> + if (page_count(page) >= 1)
> + return __ceph_tcp_sendpage(sock, page, offset, size, more);
> +
> + iov.iov_base = kmap(page) + offset;
> + iov.iov_len = size;
> + ret = ceph_tcp_sendmsg(sock, &iov, 1, size, more);
> + kunmap(page);
> +
> + return ret;
> +}
Looks good to me. Have you tested it with pre "Fix crash when using
ZFS on Ceph rbd" ZFS?
Thanks,
Ilya
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH] ceph: fix corruption when using page_count 0 page in rbd
2014-05-06 16:22 ` Ilya Dryomov
@ 2014-05-06 16:31 ` Sage Weil
2014-05-06 16:34 ` Ilya Dryomov
0 siblings, 1 reply; 6+ messages in thread
From: Sage Weil @ 2014-05-06 16:31 UTC (permalink / raw)
To: Ilya Dryomov; +Cc: Chunwei Chen, Ceph Development, Linux Kernel Mailing List
On Tue, 6 May 2014, Ilya Dryomov wrote:
> On Wed, Apr 23, 2014 at 8:35 AM, Chunwei Chen <tuxoko@gmail.com> wrote:
> > It has been reported that using ZFSonLinux on rbd will result in memory
> > corruption. The bug report can be found here:
> >
> > https://github.com/zfsonlinux/spl/issues/241
> > http://tracker.ceph.com/issues/7790
> >
> > The reason is that ZFS will send pages with page_count 0 into rbd, which in
> > turns send them to tcp_sendpage. However, tcp_sendpage cannot deal with
> > page_count 0, as it will do get_page and put_page, and erroneously free the
> > page.
> >
> > This type of issue has been noted before, and handled in iscsi, drbd,
> > etc. So, rbd should also handle this. This fix address this issue by fall back
> > to slower sendmsg when page_count 0 detected.
> >
> > Cc: Sage Weil <sage@inktank.com>
> > Cc: Yehuda Sadeh <yehuda@inktank.com>
> > Cc: stable@vger.kernel.org
> > Signed-off-by: Chunwei Chen <tuxoko@gmail.com>
> > ---
> > net/ceph/messenger.c | 20 +++++++++++++++++++-
> > 1 file changed, 19 insertions(+), 1 deletion(-)
> >
> > diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c
> > index 4f55f9c..9a964e7 100644
> > --- a/net/ceph/messenger.c
> > +++ b/net/ceph/messenger.c
> > @@ -557,7 +557,7 @@ static int ceph_tcp_sendmsg(struct socket *sock, struct kvec *iov,
> > return r;
> > }
> >
> > -static int ceph_tcp_sendpage(struct socket *sock, struct page *page,
> > +static int __ceph_tcp_sendpage(struct socket *sock, struct page *page,
> > int offset, size_t size, bool more)
> > {
> > int flags = MSG_DONTWAIT | MSG_NOSIGNAL | (more ? MSG_MORE : MSG_EOR);
> > @@ -570,6 +570,24 @@ static int ceph_tcp_sendpage(struct socket *sock, struct page *page,
> > return ret;
> > }
> >
> > +static int ceph_tcp_sendpage(struct socket *sock, struct page *page,
> > + int offset, size_t size, bool more)
> > +{
> > + int ret;
> > + struct kvec iov;
> > +
> > + /* sendpage cannot properly handle pages with page_count == 0,
> > + * we need to fallback to sendmsg if that's the case */
> > + if (page_count(page) >= 1)
> > + return __ceph_tcp_sendpage(sock, page, offset, size, more);
> > +
> > + iov.iov_base = kmap(page) + offset;
> > + iov.iov_len = size;
> > + ret = ceph_tcp_sendmsg(sock, &iov, 1, size, more);
> > + kunmap(page);
> > +
> > + return ret;
> > +}
>
> Looks good to me. Have you tested it with pre "Fix crash when using
> ZFS on Ceph rbd" ZFS?
Once this looks ready, we should perhaps stick it in for-linus so that it
can go into 3.15.
sage
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH] ceph: fix corruption when using page_count 0 page in rbd
2014-05-06 16:31 ` Sage Weil
@ 2014-05-06 16:34 ` Ilya Dryomov
2014-05-07 4:16 ` tuxoko
0 siblings, 1 reply; 6+ messages in thread
From: Ilya Dryomov @ 2014-05-06 16:34 UTC (permalink / raw)
To: Sage Weil; +Cc: Chunwei Chen, Ceph Development, Linux Kernel Mailing List
On Tue, May 6, 2014 at 8:31 PM, Sage Weil <sage@inktank.com> wrote:
> On Tue, 6 May 2014, Ilya Dryomov wrote:
>> On Wed, Apr 23, 2014 at 8:35 AM, Chunwei Chen <tuxoko@gmail.com> wrote:
>> > It has been reported that using ZFSonLinux on rbd will result in memory
>> > corruption. The bug report can be found here:
>> >
>> > https://github.com/zfsonlinux/spl/issues/241
>> > http://tracker.ceph.com/issues/7790
>> >
>> > The reason is that ZFS will send pages with page_count 0 into rbd, which in
>> > turns send them to tcp_sendpage. However, tcp_sendpage cannot deal with
>> > page_count 0, as it will do get_page and put_page, and erroneously free the
>> > page.
>> >
>> > This type of issue has been noted before, and handled in iscsi, drbd,
>> > etc. So, rbd should also handle this. This fix address this issue by fall back
>> > to slower sendmsg when page_count 0 detected.
>> >
>> > Cc: Sage Weil <sage@inktank.com>
>> > Cc: Yehuda Sadeh <yehuda@inktank.com>
>> > Cc: stable@vger.kernel.org
>> > Signed-off-by: Chunwei Chen <tuxoko@gmail.com>
>> > ---
>> > net/ceph/messenger.c | 20 +++++++++++++++++++-
>> > 1 file changed, 19 insertions(+), 1 deletion(-)
>> >
>> > diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c
>> > index 4f55f9c..9a964e7 100644
>> > --- a/net/ceph/messenger.c
>> > +++ b/net/ceph/messenger.c
>> > @@ -557,7 +557,7 @@ static int ceph_tcp_sendmsg(struct socket *sock, struct kvec *iov,
>> > return r;
>> > }
>> >
>> > -static int ceph_tcp_sendpage(struct socket *sock, struct page *page,
>> > +static int __ceph_tcp_sendpage(struct socket *sock, struct page *page,
>> > int offset, size_t size, bool more)
>> > {
>> > int flags = MSG_DONTWAIT | MSG_NOSIGNAL | (more ? MSG_MORE : MSG_EOR);
>> > @@ -570,6 +570,24 @@ static int ceph_tcp_sendpage(struct socket *sock, struct page *page,
>> > return ret;
>> > }
>> >
>> > +static int ceph_tcp_sendpage(struct socket *sock, struct page *page,
>> > + int offset, size_t size, bool more)
>> > +{
>> > + int ret;
>> > + struct kvec iov;
>> > +
>> > + /* sendpage cannot properly handle pages with page_count == 0,
>> > + * we need to fallback to sendmsg if that's the case */
>> > + if (page_count(page) >= 1)
>> > + return __ceph_tcp_sendpage(sock, page, offset, size, more);
>> > +
>> > + iov.iov_base = kmap(page) + offset;
>> > + iov.iov_len = size;
>> > + ret = ceph_tcp_sendmsg(sock, &iov, 1, size, more);
>> > + kunmap(page);
>> > +
>> > + return ret;
>> > +}
>>
>> Looks good to me. Have you tested it with pre "Fix crash when using
>> ZFS on Ceph rbd" ZFS?
>
> Once this looks ready, we should perhaps stick it in for-linus so that it
> can go into 3.15.
That's the plan, I just wanted to confirm it's been tested, since ZFS
folks went ahead and fixed it by using compound pages.
Thanks,
Ilya
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH] ceph: fix corruption when using page_count 0 page in rbd
2014-05-06 16:34 ` Ilya Dryomov
@ 2014-05-07 4:16 ` tuxoko
2014-05-07 15:35 ` Ilya Dryomov
0 siblings, 1 reply; 6+ messages in thread
From: tuxoko @ 2014-05-07 4:16 UTC (permalink / raw)
To: Ilya Dryomov, Sage Weil; +Cc: Ceph Development, Linux Kernel Mailing List
On 西元2014年05月07日 00:34, Ilya Dryomov wrote:
> On Tue, May 6, 2014 at 8:31 PM, Sage Weil <sage@inktank.com> wrote:
>> On Tue, 6 May 2014, Ilya Dryomov wrote:
>>>
>>> Looks good to me. Have you tested it with pre "Fix crash when using
>>> ZFS on Ceph rbd" ZFS?
>>
>> Once this looks ready, we should perhaps stick it in for-linus so that it
>> can go into 3.15.
>
> That's the plan, I just wanted to confirm it's been tested, since ZFS
> folks went ahead and fixed it by using compound pages.
>
> Thanks,
>
> Ilya
>
Hi Ilya
Yes, I've tested before the ZFS fix and it worked for me.
Thanks,
Chunwei Chen
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] ceph: fix corruption when using page_count 0 page in rbd
2014-05-07 4:16 ` tuxoko
@ 2014-05-07 15:35 ` Ilya Dryomov
0 siblings, 0 replies; 6+ messages in thread
From: Ilya Dryomov @ 2014-05-07 15:35 UTC (permalink / raw)
To: tuxoko; +Cc: Sage Weil, Ceph Development, Linux Kernel Mailing List
On Wed, May 7, 2014 at 8:16 AM, tuxoko <tuxoko@gmail.com> wrote:
>
>
> On 西元2014年05月07日 00:34, Ilya Dryomov wrote:
>> On Tue, May 6, 2014 at 8:31 PM, Sage Weil <sage@inktank.com> wrote:
>>> On Tue, 6 May 2014, Ilya Dryomov wrote:
>>>>
>>>> Looks good to me. Have you tested it with pre "Fix crash when using
>>>> ZFS on Ceph rbd" ZFS?
>>>
>>> Once this looks ready, we should perhaps stick it in for-linus so that it
>>> can go into 3.15.
>>
>> That's the plan, I just wanted to confirm it's been tested, since ZFS
>> folks went ahead and fixed it by using compound pages.
>>
>> Thanks,
>>
>> Ilya
>>
>
> Hi Ilya
>
> Yes, I've tested before the ZFS fix and it worked for me.
Thanks, queued.
Ilya
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-05-07 15:35 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-04-23 4:35 [PATCH] ceph: fix corruption when using page_count 0 page in rbd Chunwei Chen
2014-05-06 16:22 ` Ilya Dryomov
2014-05-06 16:31 ` Sage Weil
2014-05-06 16:34 ` Ilya Dryomov
2014-05-07 4:16 ` tuxoko
2014-05-07 15:35 ` Ilya Dryomov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox