Re: [dm-devel] Re: dm-ioband: Test results.

[Vivek Goyal <vgoyal@redhat.com>]

On Mon, Apr 20, 2009 at 05:29:59PM +0900, Ryo Tsuruta wrote:
> Hi Vivek and Nauman,
> 
> > On Thu, Apr 16, 2009 at 7:11 AM, Vivek Goyal <vgoyal@redhat.com> wrote:
> > > On Thu, Apr 16, 2009 at 11:47:50AM +0900, Ryo Tsuruta wrote:
> > >> Hi Vivek,
> > >>
> > >> > General thoughts about dm-ioband
> > >> > ================================
> > >> > - Implementing control at second level has the advantage tha one does not
> > >> >   have to muck with IO scheduler code. But then it also has the
> > >> >   disadvantage that there is no communication with IO scheduler.
> > >> >
> > >> > - dm-ioband is buffering bio at higher layer and then doing FIFO release
> > >> >   of these bios. This FIFO release can lead to priority inversion problems
> > >> >   in certain cases where RT requests are way behind BE requests or
> > >> >   reader starvation where reader bios are getting hidden behind writer
> > >> >   bios etc. These are hard to notice issues in user space. I guess above
> > >> >   RT results do highlight the RT task problems. I am still working on
> > >> >   other test cases and see if i can show the probelm.
> > 
> > Ryo, I could not agree more with Vivek here. At Google, we have very
> > stringent requirement for latency of our RT requests. If RT requests
> > get queued in any higher layer (behind BE requests), all bets are off.
> > I don't find doing IO control at two layer for this particular reason.
> > The upper layer (dm-ioband in this case) would have to make sure that
> > RT requests are released immediately, irrespective of the state (FIFO
> > queuing and tokens held). And the lower layer (IO scheduling layer)
> > has to do the same. This requirement is not specific to us. I have
> > seen similar comments from filesystem folks here previously, in the
> > context of metadata updates being submitted as RT. Basically, the
> > semantics of RT class has to be preserved by any solution that is
> > build on top of CFQ scheduler.
> 
> I could see the priority inversion by running Vivek's script and I
> understand how RT requests has to be handled. I'll create a patch
> which makes dm-ioband cooperates with CFQ scheduler. However, do you
> think we need some kind of limitation on processes which belong to the
> RT class to prevent the processes from depleting bandwidth?

I think to begin with, we can keep the same behavior as CFQ. An RT task
can starve other tasks.

But we should provide two configurations and user can choose any one.
If RT task is in root group, it will starve other sibling tasks/groups. If
it is with-in a cgroup, then it will starve its sibling only with-in that
cgroup and will not impact other cgroups.

What I mean is following.

			root
			/  \
		      RT   group1

In above configuration RT task will starve everybody else.

			root
		        /   \
		   group1   group2
		   /  \
	         RT   BE

In above configuration  RT task will starve only sibling in group1 but
will not starve the tasks in group2 or in root.

Thanks
Vivek