Re: [Lsf-pc] [LSF/MM ATTEND] Filesystems -- Btrfs, cgroups, Storage topics from Facebook

[Jan Kara <jack@suse.cz>]

On Thu 02-01-14 19:11:44, Chris Mason wrote:
> On Thu, 2014-01-02 at 12:10 -0500, tj@kernel.org wrote:
> > Hey, Vivek.
> > 
> > On Thu, Jan 02, 2014 at 12:06:37PM -0500, Vivek Goyal wrote:
> > > So is this a separate configuration which can be done per bdi as opposed
> > > to per device? IOW throttling offered per per cgroup per bdi. This will
> > > help with the case of throttling over NFS too, which some people have
> > > been asking for.
> > 
> > Hah? No, bdi just being split per-cgroup on each device so that it can
> > properly propagate congestion upwards per-blkcg, just like how we
> > split request allocation per-cgroup in the block layer proper.
> > 
> 
> I'm not entirely sure how well this will fit with the filesystems (who
> already expect a single BDI), but it's worth trying.  I'm definitely
> worried about having too many blkcgs and over-committing the dirty memory
> limits.  BDI-per device setup we have now already has that problem, but
> I'm not sure its a good idea to make it worse.
  Well, we already have BDIs not tied to any particular device - e.g. in
NFS or in btrfs (I guess I don't have to tell you ;). In more complex
storage situations we rather seem to follow the rule one BDI per filesystem
instance as that's what makes sense for writeback and most of other stuff
in BDI. And we definitely want to split this "high-level" BDI because
that's the only thing which makes sense for writeback (you cannot really
tell whether e.g. dirty inode belongs to sda or sdb in RAID0 setting).

> > > So it sounds like re-implementing throttling infrastructure at bdi level
> > > now (Similar to what has been done at device level)? Of course use as
> > > much code as possible. But IIUC, proposal is that effectively there will
> > > can be two throttling controllers. One operating at bdi level and one
> > > operating below it at device level?
> > 
> > Not at all.  I was arguing explicitly against that.
> 
> Keep in mind that we do already have throttling at the BDI level.  I was
> definitely hoping we could consolidate some of that since it has grown
> some bits of an elevator already.  I'm not saying what we have in the
> bdi throttling isn't reasonable, but it is definitely replicating some
> of the infrastructure down below.
  What exactly are you talking about? Dirty throttling or something else?

> We'll (Josef, me, perhaps others at FB) will do some experiments before
> LSF.  The current plan is to throw some code at the wall and see what
> sticks.
  That would be great. People are talking about this on and off for at
least three years, there were some patches posted but so far noone was
persistent enough to get anything merged (to be fair at least we have
explored some dead ends where we decided that it's not worth the hassle
after we've seen the code / considered the API).

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR