All of lore.kernel.org
 help / color / mirror / Atom feed
From: Mike Snitzer <snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
To: Christoph Hellwig <hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
Cc: linux-lvm-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
	"ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org"
	<ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>,
	"ceph-users-Qp0mS5GaXlQ@public.gmane.org"
	<ceph-users-Qp0mS5GaXlQ@public.gmane.org>
Subject: Re: poor read performance on rbd+LVM, LVM overload
Date: Mon, 21 Oct 2013 11:06:05 -0400	[thread overview]
Message-ID: <20131021150605.GB28099@redhat.com> (raw)
In-Reply-To: <20131021150129.GA28099-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

On Mon, Oct 21 2013 at 11:01am -0400,
Mike Snitzer <snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> On Mon, Oct 21 2013 at 10:11am -0400,
> Christoph Hellwig <hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> wrote:
> 
> > On Sun, Oct 20, 2013 at 08:58:58PM -0700, Sage Weil wrote:
> > > It looks like without LVM we're getting 128KB requests (which IIRC is 
> > > typical), but with LVM it's only 4KB.  Unfortunately my memory is a bit 
> > > fuzzy here, but I seem to recall a property on the request_queue or device 
> > > that affected this.  RBD is currently doing
> > 
> > Unfortunately most device mapper modules still split all I/O into 4k
> > chunks before handling them.  They rely on the elevator to merge them
> > back together down the line, which isn't overly efficient but should at
> > least provide larger segments for the common cases.
> 
> It isn't DM that splits the IO into 4K chunks; it is the VM subsystem
> no?  Unless care is taken to assemble larger bios (higher up the IO
> stack, e.g. in XFS), all buffered IO will come to bio-based DM targets
> in $PAGE_SIZE granularity.
> 
> I would expect direct IO to before better here because it will make use
> of bio_add_page to build up larger IOs.

s/before/perform/ ;)
 
> Taking a step back, the rbd driver is exposing both the minimum_io_size
> and optimal_io_size as 4M.  This symmetry will cause XFS to _not_ detect
> the exposed limits as striping.  Therefore, AFAIK, XFS won't take steps
> to respect the limits when it assembles its bios (via bio_add_page).
> 
> Sage, any reason why you don't use traditional raid geomtry based IO
> limits?, e.g.:
> 
> minimum_io_size = raid chunk size
> optimal_io_size = raid chunk size * N stripes (aka full stripe)
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

WARNING: multiple messages have this Message-ID (diff)
From: Mike Snitzer <snitzer@redhat.com>
To: Christoph Hellwig <hch@infradead.org>
Cc: elder@inktank.com, Sage Weil <sage@inktank.com>,
	Ugis <ugis22@gmail.com>,
	linux-lvm@redhat.com,
	"ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>,
	"ceph-users@ceph.com" <ceph-users@ceph.com>
Subject: Re: [linux-lvm] poor read performance on rbd+LVM, LVM overload
Date: Mon, 21 Oct 2013 11:06:05 -0400	[thread overview]
Message-ID: <20131021150605.GB28099@redhat.com> (raw)
In-Reply-To: <20131021150129.GA28099@redhat.com>

On Mon, Oct 21 2013 at 11:01am -0400,
Mike Snitzer <snitzer@redhat.com> wrote:

> On Mon, Oct 21 2013 at 10:11am -0400,
> Christoph Hellwig <hch@infradead.org> wrote:
> 
> > On Sun, Oct 20, 2013 at 08:58:58PM -0700, Sage Weil wrote:
> > > It looks like without LVM we're getting 128KB requests (which IIRC is 
> > > typical), but with LVM it's only 4KB.  Unfortunately my memory is a bit 
> > > fuzzy here, but I seem to recall a property on the request_queue or device 
> > > that affected this.  RBD is currently doing
> > 
> > Unfortunately most device mapper modules still split all I/O into 4k
> > chunks before handling them.  They rely on the elevator to merge them
> > back together down the line, which isn't overly efficient but should at
> > least provide larger segments for the common cases.
> 
> It isn't DM that splits the IO into 4K chunks; it is the VM subsystem
> no?  Unless care is taken to assemble larger bios (higher up the IO
> stack, e.g. in XFS), all buffered IO will come to bio-based DM targets
> in $PAGE_SIZE granularity.
> 
> I would expect direct IO to before better here because it will make use
> of bio_add_page to build up larger IOs.

s/before/perform/ ;)
 
> Taking a step back, the rbd driver is exposing both the minimum_io_size
> and optimal_io_size as 4M.  This symmetry will cause XFS to _not_ detect
> the exposed limits as striping.  Therefore, AFAIK, XFS won't take steps
> to respect the limits when it assembles its bios (via bio_add_page).
> 
> Sage, any reason why you don't use traditional raid geomtry based IO
> limits?, e.g.:
> 
> minimum_io_size = raid chunk size
> optimal_io_size = raid chunk size * N stripes (aka full stripe)
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

  parent reply	other threads:[~2013-10-21 15:06 UTC|newest]

Thread overview: 36+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2013-10-16 14:46 poor read performance on rbd+LVM, LVM overload Ugis
2013-10-16 14:46 ` [linux-lvm] " Ugis
2013-10-16 16:16 ` Sage Weil
2013-10-16 16:16   ` [linux-lvm] " Sage Weil
     [not found]   ` <alpine.DEB.2.00.1310160914360.22271-vIokxiIdD2AQNTJnQDzGJqxOck334EZe@public.gmane.org>
2013-10-17  9:06     ` David McBride
2013-10-17  9:06       ` [linux-lvm] " David McBride
2013-10-17 15:18   ` Mike Snitzer
2013-10-17 15:18     ` [linux-lvm] " Mike Snitzer
     [not found]     ` <20131017151828.GB28859-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2013-10-18  7:56       ` Ugis
2013-10-18  7:56         ` [linux-lvm] " Ugis
2013-10-19  0:01         ` Sage Weil
2013-10-19  0:01           ` [linux-lvm] " Sage Weil
     [not found]           ` <alpine.DEB.2.00.1310181657151.19763-vIokxiIdD2AQNTJnQDzGJqxOck334EZe@public.gmane.org>
2013-10-20 15:18             ` Ugis
2013-10-20 15:18               ` [linux-lvm] " Ugis
     [not found]               ` <CAE63xUO4ZrzObMFeQ=FXGFnqpwWsjCiiDr2_VhOt91h=djofYw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-10-20 18:21                 ` Josh Durgin
2013-10-20 18:21                   ` [linux-lvm] [ceph-users] " Josh Durgin
2013-10-21  3:58                 ` Sage Weil
2013-10-21  3:58                   ` [linux-lvm] " Sage Weil
2013-10-21 14:11                   ` Christoph Hellwig
2013-10-21 14:11                     ` [linux-lvm] " Christoph Hellwig
     [not found]                     ` <20131021141147.GA30189-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2013-10-21 15:01                       ` Mike Snitzer
2013-10-21 15:01                         ` [linux-lvm] " Mike Snitzer
     [not found]                         ` <20131021150129.GA28099-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2013-10-21 15:06                           ` Mike Snitzer [this message]
2013-10-21 15:06                             ` Mike Snitzer
2013-10-21 16:02                         ` Sage Weil
2013-10-21 16:02                           ` [linux-lvm] " Sage Weil
     [not found]                           ` <alpine.DEB.2.00.1310210853140.29488-vIokxiIdD2AQNTJnQDzGJqxOck334EZe@public.gmane.org>
2013-10-21 17:48                             ` Mike Snitzer
2013-10-21 17:48                               ` [linux-lvm] " Mike Snitzer
     [not found]                               ` <20131021174850.GA29416-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2013-10-21 18:05                                 ` Sage Weil
2013-10-21 18:05                                   ` [linux-lvm] " Sage Weil
2013-10-21 18:06                         ` Christoph Hellwig
2013-10-21 18:06                           ` [linux-lvm] " Christoph Hellwig
     [not found]                           ` <20131021180616.GA7196-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2013-10-21 18:27                             ` Mike Snitzer
2013-10-21 18:27                               ` [linux-lvm] " Mike Snitzer
2013-10-30 14:53                               ` Ugis
2013-10-30 14:53                                 ` [linux-lvm] " Ugis

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20131021150605.GB28099@redhat.com \
    --to=snitzer-h+wxahxf7alqt0dzr+alfa@public.gmane.org \
    --cc=ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
    --cc=ceph-users-Qp0mS5GaXlQ@public.gmane.org \
    --cc=hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org \
    --cc=linux-lvm-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.