[Drbd-dev] Re: XFS on DRBD

[Drbd-dev] Re: XFS on DRBD

[Philipp Reisner]

On Wednesday 26 May 2004 09:36, you wrote:
> XFS does I/O on slab-allocated pages, which not a lot other code in
> the kernel does, and it seems drbd doesn't like that.  Do you have
> a pointer to the drbd code for Linux 2.6 so I can take a look?

Hi Christoph,

Drbd reached release status and finally everything works, but we have a
rather uggly workaround for XFS.

We usually use sendpage() to send the data blocks over the network.
But when XFS hands us its slab-allocated pages over, we may not
use sendpage. 

  If we call sendpage on such a page it will crash, since sendpage()
  calls put_page() after the job is done.

So currently we have this code in drbd:

	/* PARANOIA. if this ever triggers,
	 * something in the layers above us is really kaputt.
	 *one roundtrip later:
	 * doh. it triggered. so XFS _IS_ really kaputt ...
	 * oh well...
	 */
	if ( (page_count(page) < 1) || PageSlab(page) ) {
		/* e.g. XFS meta- & log-data is in slab pages, which have a
		 * page_count of 0 and/or have PageSlab() set...
		 * FIXME: This is a workaround.
		 */
		++fallback;
		sent =  _drbd_no_send_page(mdev, page, offset, size);
		if (likely(sent > 0)) len -= sent;
		goto out;
	}

Jul 27 09:58:32 mary kernel: drbd:sendpage fallback/total: 1/5381
Jul 27 09:59:32 mary kernel: drbd:sendpage fallback/total: 6/138083
Jul 27 10:00:32 mary kernel: drbd:sendpage fallback/total: 11/258761
Jul 27 10:01:32 mary kernel: drbd:sendpage fallback/total: 13/260085
Jul 27 10:02:32 mary kernel: drbd:sendpage fallback/total: 22/354874
Jul 27 10:03:32 mary kernel: drbd:sendpage fallback/total: 27/475558
Jul 27 10:04:37 mary kernel: drbd:sendpage fallback/total: 31/520124
Jul 27 10:05:37 mary kernel: drbd:sendpage fallback/total: 43/591427
Jul 27 10:06:37 mary kernel: drbd:sendpage fallback/total: 51/708371
Jul 27 10:07:42 mary kernel: drbd:sendpage fallback/total: 58/780179
Jul 27 10:08:42 mary kernel: drbd:sendpage fallback/total: 73/815288
Jul 27 10:09:42 mary kernel: drbd:sendpage fallback/total: 87/926600

Christoph,

IMHO XFS should only pass pages down the IO path which have a correct
page_count. 

Which is the right forum to discuss this issue ?

-Philipp

-- 
: Dipl-Ing Philipp Reisner                      Tel +43-1-8178292-50 :
: LINBIT Information Technologies GmbH          Fax +43-1-8178292-82 :
: Schönbrunnerstr 244, 1120 Vienna, Austria    http://www.linbit.com :

[Drbd-dev] Re: XFS on DRBD

[Lars Ellenberg]

/ 2004-07-27 10:15:53 +0200
\ Philipp Reisner:
> On Wednesday 26 May 2004 09:36, you wrote:
> > XFS does I/O on slab-allocated pages, which not a lot other code in
> > the kernel does, and it seems drbd doesn't like that.  Do you have
> > a pointer to the drbd code for Linux 2.6 so I can take a look?
> 
> Hi Christoph,
> 
> Drbd reached release status and finally everything works, but we have a
> rather uggly workaround for XFS.
> 
> We usually use sendpage() to send the data blocks over the network.
> But when XFS hands us its slab-allocated pages over, we may not
> use sendpage. 
> 
>   If we call sendpage on such a page it will crash, since sendpage()
... first does a get_page(), and then ...
>   calls put_page() after the job is done.
 --> but PageSlab(), so: BOOM.
this all happens _before_ any bio_endio is called on the associated bio.

> Christoph,
> 
> IMHO XFS should only pass pages down the IO path which have a correct
> page_count. 
> 
> Which is the right forum to discuss this issue ?

I suggest we either can solve this right here,
or we move this to lkml ...
maybe we should move to lkml anyways, since actually this
is a generic issue: how to pass slab pages around to other
sub systems, which do not know anything about that page,
but expect to be able to
  get_page(); queue_some_asynchronous_work_and_return();
      ... later ... put_page();

	Lars Ellenberg