automatic testing of cgroup writeback limiting (was: Re: Does XFS support cgroup writeback limiting?)

automatic testing of cgroup writeback limiting (was: Re: Does XFS support cgroup writeback limiting?)

[Martin Steigerwald]

I think it makes sense to include those that wrote the cgroup writeback 
limiting into the discussion on how to automatically test this feature. Thus 
add fsdevel and Tejun to CC.

Am Sonntag, 29. November 2015, 22:41:13 CET schrieb Lutz Vieweg:
> On 11/25/2015 10:35 PM, Dave Chinner wrote:
> >> 2) Create 3 different XFS filesystem instances on the block
> >> 
> >>     device, one for access by only the "good" processes,
> >>     on for access by only the "evil" processes, one for
> >>     shared access by at least two "good" and two "evil"
> >>     processes.
> > 
> > Why do you need multiple filesystems? The writeback throttling is
> > designed to work within a single filesystem...
> 
> Hmm. Previously, I thought that the limiting of buffered writes
> was realized by keeping track of the owners of dirty pages, and
> that filesystem support was just required to make sure that writing
> via a filesystem did not "anonymize" the dirty data. From what
> I had read in blkio-controller.txt it seemed evident that limitations
> would be accounted for "per block device", not "per filesystem", and
> options like
> 
> > echo "<major>:<minor>  <rate_bytes_per_second>" >
> > /cgrp/blkio.throttle.read_bps_device
> document how to configure limits per block device.
> 
> Now after reading through the new Writeback section of blkio-controller.txt
> again I am somewhat confused - the text states
> 
> > writeback operates on inode basis
> 
> and if that means inodes as in "file system inodes", this would
> indeed mean limits would be enforced "per filesystem" - and yet
> there are no options documented to specify limits for any specific
> filesystem.
> 
> Does this mean some process writing to a block device (not via filesystem)
> without "O_DIRECT" will dirty buffer pages, but those will not be limited
> (as they are neither synchronous nor via-filesystem writes)?
> That would mean VMs sharing some (physical or abstract) block device could
> not really be isolated regarding their asynchronous write I/O...
> 
> > Metadata IO not throttled - it is owned by the filesystem and hence
> > root cgroup.
> 
> Ouch. That kind of defeats the purpose of limiting evil processes'
> ability to DOS other processes.
> Wouldn't it be possible to assign some arbitrary cost to meta-data
> operations - like "account one page write for each meta-data change
> to the originating process of that change"? While certainly not
> allowing for limiting to byte-precise limits of write bandwidth,
> this would regain the ability to defend against DOS situations,
> and for well-behaved processes, the "cost" accounted for their
> not-so-frequent meta-data operations would probably not really hurt their
> writing
> speed.
> 
> >>  The test is successful if all "good processes" terminate successfully
> >>  
> >>    after a time not longer than it would take to write 10 times X MB to
> >>    the
> >>    rate-limited block device.
> > 
> > if we are rate limiting to 1MB/s, then a 10s test is not long enough
> > to reach steady state. Indeed, it's going to take at least 30s worth
> > of IO to guarantee that we getting writeback occurring for low
> > bandwidth streams....
> 
> Sure, the "X/100 MB per second" throttle to the scratch device
> was meant to result in a minimal test time of > 100s.
> 
> > i.e. the test needs to run for a period of time and then measure
> > the throughput of each stream, comparing it against the expected
> > throughput for the stream, rather than trying to write a fixed
> > bandwidth....
> 
> The reason why I thought it to be a good idea to have the "good" processes
> use only a limited write rate was to make sure that the actual write
> activity of those processes is spread out over enough time to make sure
> that they could, after all, feel some "pressure back" from the operating
> system that is applied only after the "bad" processes have filled up
> all RAM dedicated to dirty buffer cache.
> 
> Assume the test instance has lots of memory and would be willing to
> spend many Gigabytes of RAM for dirty buffer caches. Chances are that
> in such a situation the "good" processes might be done writing their
> limited amount of data almost instantaneously, because the data just
> went to RAM.
> 
> (I understand that if one used the absolute "blkio.throttle.write*" options
> pressure back could apply before the dirty buffer cache was maxed out,
> but in real-world scenarios people will almost always use the relative
> "blkio.weight" based limiting, after all, you usually don't want to throttle
> processes if there is plenty of bandwidth left no other process wants at
> the same time.)
> 
> 
> Regards,
> 
> Lutz Vieweg
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

Re: automatic testing of cgroup writeback limiting (was: Re: Does XFS support cgroup writeback limiting?)

[Tejun Heo]

Hello,

On Tue, Dec 01, 2015 at 09:38:03AM +0100, Martin Steigerwald wrote:
> > > echo "<major>:<minor>  <rate_bytes_per_second>" >
> > > /cgrp/blkio.throttle.read_bps_device
> > document how to configure limits per block device.
> > 
> > Now after reading through the new Writeback section of blkio-controller.txt
> > again I am somewhat confused - the text states
> > 
> > > writeback operates on inode basis

As opposed to pages.  cgroup ownership is tracked per inode, not per
page, so if multiple cgroups write to the same inode at the same time,
some IOs will be incorrectly attributed.

> > and if that means inodes as in "file system inodes", this would
> > indeed mean limits would be enforced "per filesystem" - and yet
> > there are no options documented to specify limits for any specific
> > filesystem.

cgroup ownership is per-inode.  IO throttling is per-device, so as
long as multiple filesystems map to the same device, they fall under
the same limit.

> > > Metadata IO not throttled - it is owned by the filesystem and hence
> > > root cgroup.
> > 
> > Ouch. That kind of defeats the purpose of limiting evil processes'
> > ability to DOS other processes.

cgroup isn't a security mechanism and has to make active tradeoffs
between isolation and overhead.  It doesn't provide protection against
malicious users and in general it's a pretty bad idea to depend on
cgroup for protection against hostile entities.  Although some
controller do better isolation than others, given how filesystems are
implemented, filesystem io control getting there will likely take a
while.

> > Wouldn't it be possible to assign some arbitrary cost to meta-data
> > operations - like "account one page write for each meta-data change
> > to the originating process of that change"? While certainly not
> > allowing for limiting to byte-precise limits of write bandwidth,
> > this would regain the ability to defend against DOS situations,
> > and for well-behaved processes, the "cost" accounted for their
> > not-so-frequent meta-data operations would probably not really hurt their
> > writing
> > speed.

For aggregate consumers, this sort of approaches does make sense -
measure total consumption by common operations and distribute the
charges afterwards; however, this will require quite a bit of work on
both io controller and filesystem sides.

> > (I understand that if one used the absolute "blkio.throttle.write*" options
> > pressure back could apply before the dirty buffer cache was maxed out,
> > but in real-world scenarios people will almost always use the relative
> > "blkio.weight" based limiting, after all, you usually don't want to throttle
> > processes if there is plenty of bandwidth left no other process wants at
> > the same time.)

I'd recommend configuring both memory.high and io.weight so that the
buffer area isn't crazy high compared to io bandwidth.  It should be
able to reach the configured ratio that way and also avoids two io
domains competing in the same io domain which can skew the results.

Thanks.

-- 
tejun

Re: automatic testing of cgroup writeback limiting

[Lutz Vieweg]

On 12/01/2015 05:38 PM, Tejun Heo wrote:
> As opposed to pages.  cgroup ownership is tracked per inode, not per
> page, so if multiple cgroups write to the same inode at the same time,
> some IOs will be incorrectly attributed.

I can't think of use cases where this could become a problem.
If more than one user/container/VM is allowed to write to the
same file at any one time, isolation is probably absent anyway ;-)

> cgroup ownership is per-inode.  IO throttling is per-device, so as
> long as multiple filesystems map to the same device, they fall under
> the same limit.

Good, that's why I assumed it useful to include a scenario with more
than one filesystem on the same device into the test scenario, just
to know whether there are unexpected issues if more than one filesystem
utilizes the same underlying device.

>>>> Metadata IO not throttled - it is owned by the filesystem and hence
>>>> root cgroup.
>>>
>>> Ouch. That kind of defeats the purpose of limiting evil processes'
>>> ability to DOS other processes.
>
> cgroup isn't a security mechanism and has to make active tradeoffs
> between isolation and overhead.  It doesn't provide protection against
> malicious users and in general it's a pretty bad idea to depend on
> cgroup for protection against hostile entities.

I wrote of "evil" processes for simplicity, but 99 out of 100 times
it's not intentional "evilness" that makes a process exhaust I/O
bandwidth of some device shared with other users/containers/VMs, it's
usually just bugs, inconsiderate programming or inappropriate use
that makes one process write like crazy, making other
users/containers/VMs suffer.

Whereever strict service level guarantees are relevant, and
applications require writing to storage, you currently cannot
consolidate two or more applications onto the same physical host,
even if they run under separate users/containers/VMs.

I understand there is no short or medium term solution that
would allow to isolate processes writing to the same filesytem
(because of the meta data writing), but is it correct to say
that at least VMs, which do not allow the virtual guest to
cause extensive meta data writes on the physical host, only
writes into pre-allocated image files, can be safely isolated
by the new "buffered write accounting"?

If so, we'd have stay away from user or container based isolation
of independently SLA'd applications, but could at least resort to VMs
using image files on a shared filesystem.

Regards,

Lutz Vieweg

Re: automatic testing of cgroup writeback limiting

[Tejun Heo]

Hello, Lutz.

On Thu, Dec 03, 2015 at 01:18:48AM +0100, Lutz Vieweg wrote:
> On 12/01/2015 05:38 PM, Tejun Heo wrote:
> >As opposed to pages.  cgroup ownership is tracked per inode, not per
> >page, so if multiple cgroups write to the same inode at the same time,
> >some IOs will be incorrectly attributed.
> 
> I can't think of use cases where this could become a problem.
> If more than one user/container/VM is allowed to write to the
> same file at any one time, isolation is probably absent anyway ;-)

Yeap, that's why the trade-off was made.

> >cgroup ownership is per-inode.  IO throttling is per-device, so as
> >long as multiple filesystems map to the same device, they fall under
> >the same limit.
> 
> Good, that's why I assumed it useful to include a scenario with more
> than one filesystem on the same device into the test scenario, just
> to know whether there are unexpected issues if more than one filesystem
> utilizes the same underlying device.

Sure, I'd recommend including multiple writers on a single filesystem
case too as that exposes entanglement in metadata handling.  That
should expose problems in more places.

> I wrote of "evil" processes for simplicity, but 99 out of 100 times
> it's not intentional "evilness" that makes a process exhaust I/O
> bandwidth of some device shared with other users/containers/VMs, it's
> usually just bugs, inconsiderate programming or inappropriate use
> that makes one process write like crazy, making other
> users/containers/VMs suffer.

Right now, what cgroup writeback can control is well-behaving
workloads which aren't dominated by metadata writeback.  We still have
ways to go but it still is a huge leap compared to what we had before.

> Whereever strict service level guarantees are relevant, and
> applications require writing to storage, you currently cannot
> consolidate two or more applications onto the same physical host,
> even if they run under separate users/containers/VMs.

You're right.  It can't do isolation well enough for things like
strict service level guarantee.

> I understand there is no short or medium term solution that
> would allow to isolate processes writing to the same filesytem
> (because of the meta data writing), but is it correct to say
> that at least VMs, which do not allow the virtual guest to
> cause extensive meta data writes on the physical host, only
> writes into pre-allocated image files, can be safely isolated
> by the new "buffered write accounting"?

Sure, that or loop mounts.  Pure data accesses should be fairly well
isolated.

Thanks.

-- 
tejun