Heavy I/O causing slow interactivity

Heavy I/O causing slow interactivity

[Raymond Jennings]

I'm running gentoo and it's emerging llvm.  This I/O heavy process is
causing slowdowns when I attempt interactive stuff, including watching
a youtube video and accessing a chatroom.

Similar latency is induced during a heavy database application.

As an end user is there anything I can do to better support
interactive performance?

And as a potential kernel developer, is there anything I could tweak
in the kernel source to mitigate this behavior?

I've tried SCHED_IDLE and idle class with ionice, both to no avail.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Heavy I/O causing slow interactivity

[Michal Hocko]

On Mon 03-04-17 10:39:39, Raymond Jennings wrote:
> I'm running gentoo and it's emerging llvm.  This I/O heavy process is
> causing slowdowns when I attempt interactive stuff, including watching
> a youtube video and accessing a chatroom.
> 
> Similar latency is induced during a heavy database application.
> 
> As an end user is there anything I can do to better support
> interactive performance?
> 
> And as a potential kernel developer, is there anything I could tweak
> in the kernel source to mitigate this behavior?

How much memory do you have? What is your /proc/sys/vm/dirty_* setting?
What kind of storage do you use?
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Heavy I/O causing slow interactivity

[Mike Rapoport]

On Mon, Apr 03, 2017 at 10:39:39AM -0700, Raymond Jennings wrote:
> I'm running gentoo and it's emerging llvm.  This I/O heavy process is
> causing slowdowns when I attempt interactive stuff, including watching
> a youtube video and accessing a chatroom.

"emerge llvm" means "build LLVM compiler suite and install it", right?
This could be quite CPU and memory intensive as well.
Depending on your hardware and options you use it just may end up eating all
the system resources...
 
> Similar latency is induced during a heavy database application.
> 
> As an end user is there anything I can do to better support
> interactive performance?
> 
> And as a potential kernel developer, is there anything I could tweak
> in the kernel source to mitigate this behavior?
> 
> I've tried SCHED_IDLE and idle class with ionice, both to no avail.
 
--
Sincerely yours,
Mike.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Heavy I/O causing slow interactivity

[Raymond Jennings]

I have 32GiB of memory

Storage is an LVM volume group sitting on a pair of 2T western digital
drives, one WD Green, and the other WD Blue

My CPU is an i7, model 4790K.

What I'd like is some way for my system to fairly share the available
I/O bandwidth.  My youtube is sensitive to latency but doesn't chew up
a lot of throughput.  My I/O heavy stuff isn't really urgent and I
don't mind it yielding to the interactive stuff.

I remember a similiar concept being tried awhile ago with a scheduler
that "punished" processes that sucked up too much CPU and made sure
the short sporadic event driven interactive stuff got the scraps of
CPU when it needed them.

/proc/sys/vm/dirty is set up as follows

dirty_ratio 90
dirty_background_ratio 80
dirty_expire_centisecs 30000
dirty_writeback_centisecs 6000

It's gotten bad enough that my IRC client froze long enough to get
pinged off the server it was connected to, which means over 180
seconds of lag.

Since it was triggered by intense disk I/O I figured mm/vm was a good
place to ask for help

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Heavy I/O causing slow interactivity

[Michal Hocko]

On Wed 05-04-17 11:15:44, Raymond Jennings wrote:
> I have 32GiB of memory
> 
> Storage is an LVM volume group sitting on a pair of 2T western digital
> drives, one WD Green, and the other WD Blue
> 
> My CPU is an i7, model 4790K.
> 
> What I'd like is some way for my system to fairly share the available
> I/O bandwidth.  My youtube is sensitive to latency but doesn't chew up
> a lot of throughput.  My I/O heavy stuff isn't really urgent and I
> don't mind it yielding to the interactive stuff.
> 
> I remember a similiar concept being tried awhile ago with a scheduler
> that "punished" processes that sucked up too much CPU and made sure
> the short sporadic event driven interactive stuff got the scraps of
> CPU when it needed them.
> 
> /proc/sys/vm/dirty is set up as follows
> 
> dirty_ratio 90

So you allow 90% of your 32GB to be dirty and then get throttled which
will take quite some time until it gets synced to the disk. Even with a
fast storage. I would really recommend reducing dirty_ratio (and
background ratio as well) to something much more reasonable. E.g. start
the background IO at around 400MB and hard limit at 800MB. I am pretty
sure that the stalls you are seeing are related to the IO dirty
throttling.

-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Heavy I/O causing slow interactivity

[Raymond Jennings]

Would it make a difference if I cited that

My intent on upping the limits so high and pushing the dirty expiry so
far into the future was to *avoid* triggering background writeback.

In fact, dirty memory during one of these tests never actually rose a bunch.

Are you guys suggesting that if dirty memory gets high enough the
writeback turns into an OOM dodger that preempts foreground I/O?

What I was hoping for is for dirty writeback itself to be throttled
and stay out of the way of foreground I/O.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Heavy I/O causing slow interactivity

[Michal Hocko]

On Thu 13-04-17 11:13:41, Raymond Jennings wrote:
> Would it make a difference if I cited that
> 
> My intent on upping the limits so high and pushing the dirty expiry so
> far into the future was to *avoid* triggering background writeback.
> 
> In fact, dirty memory during one of these tests never actually rose a bunch.

How have you checked that?

> Are you guys suggesting that if dirty memory gets high enough the
> writeback turns into an OOM dodger that preempts foreground I/O?

Once you exceed the dirty limit, writers starts being throttled on new
writes. If things go especially bad than all writers get throttled
basically and that is where your stalls come from most probably.
 
> What I was hoping for is for dirty writeback itself to be throttled
> and stay out of the way of foreground I/O.

The amount of dirty data would just grow without any bounds if the
writers were not throttled...

-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>