Re: Local DoS through write heavy I/O on CFQ & Deadline

Re: Local DoS through write heavy I/O on CFQ & Deadline

[Jan Kara]

  Hello,

On Fri 12-10-12 17:29:50, Alex Bligh wrote:
> --On 12 October 2012 16:58:39 +0200 Michal Hocko <mhocko@suse.cz> wrote:
  Let me explain a couple of things...

> >Once dirty_ratio (resp. dirty_bytes) limit is hit then the process which
> >writes gets throttled. If this is not the case then there is a bug in
> >the throttling code.
> I believe that is the problem.
  So I believe the throttling works. What write throttling does is that it
throttles process when more than dirty_bytes (or dirty_ratio) memory would
become dirty. That clearly works as otherwise your testcase will drive the
machine out of memory. Now whenever some memory is cleaned by writeback,
your process is allowed to continue so there is always enough data to
write.

Actually, the throttling is somewhat more clever and doesn't allow a dirty
hog (like your dd test) to use all of the dirtiable limit. Instead the hog
is throttled somewhat earlier leaving some dirtiable memory to other
processes as well. Seeing that mysql process gets blocked during fsync(2)
(and not during write itself) this mechanism works right as well.

> >>Isn't the only thing that is going to change that it ends up
> >>triggering the writeback earlier?
> >
> >Set the limit lowe?
> 
> I think you mean 'lower'. If I do that, what I think will happen
> is that it will start the write-back earlier, but the writeback
> once started will not keep up with the generation of data, possibly
> because the throttling isn't going to work. Note that for
> instance using ionice to set priority or class to 'idle'
> has no effect. So, to test my hypothesis ...
  Yeah, ionice has its limitations. The problem is that all buffered
writes happen just into memory (so completely independently of ionice
settings). Subsequent writing of dirty memory to disk happens using flusher
thread which is a kernel process and it doesn't know anything about IO
priority set for task which created the file. If you wrote the file with
oflag=direct or oflag=sync you would see that ionice works as expected.

Now what *is* your problem is an ext4 behavior (proper list CCed) as David
Chinner correctly noted. Apparently journal thread is not able to commit
transaction for a long time. I've tried to reproduce your results
(admittedly with replacing myslq with a simplistic "dd if=/dev/zero of=file2
bs=1M count=1 conv=fsync") but I failed. fsync always returns in a couple
of seconds... What ext4 mount options do you use?

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: Local DoS through write heavy I/O on CFQ & Deadline

[Chris Friesen]

On 10/18/2012 03:28 PM, Jan Kara wrote:

>    Yeah, ionice has its limitations. The problem is that all buffered
> writes happen just into memory (so completely independently of ionice
> settings). Subsequent writing of dirty memory to disk happens using flusher
> thread which is a kernel process and it doesn't know anything about IO
> priority set for task which created the file. If you wrote the file with
> oflag=direct or oflag=sync you would see that ionice works as expected.

Has anyone looked at storing the ionice value with the buffered write 
request such that the actual writes to disk could be sorted by priority 
and done with the ionice level of the original caller?

Chris

Re: Local DoS through write heavy I/O on CFQ & Deadline

[Jan Kara]

On Thu 18-10-12 16:13:58, Chris Friesen wrote:
> On 10/18/2012 03:28 PM, Jan Kara wrote:
> 
> >   Yeah, ionice has its limitations. The problem is that all buffered
> >writes happen just into memory (so completely independently of ionice
> >settings). Subsequent writing of dirty memory to disk happens using flusher
> >thread which is a kernel process and it doesn't know anything about IO
> >priority set for task which created the file. If you wrote the file with
> >oflag=direct or oflag=sync you would see that ionice works as expected.
> 
> Has anyone looked at storing the ionice value with the buffered
> write request such that the actual writes to disk could be sorted by
> priority and done with the ionice level of the original caller?
  There's nothing as "buffered write request" in kernel. When buffered
write happens, data are just copied into page cache. We could attach a tag
to each modified page in the page cache but that would get really expensive.

Essentially the same problems happens with cgroups where buffered writes
are not accounted as well. There we considered to attach a tag to inodes
(which doesn't work well if processes from different cgroups / with
different IO priority write to the same inode but that's not that common)
which is reasonably cheap. But then you have to build smarts into flusher
thread to prioritize inodes according to tags (you cannot really let
flusher thread just submit IO with that priority because when it gets
blocked, it starves writeback with possible higher priority). Alternatively
you could have separate flusher thread per-cgroup / IO priority. That is
easier from code point of view but throughput suffers because of limited
merging of IO. So all in all the problem is known but hard to tackle.

							Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR