btrfs scrub ioprio

btrfs scrub ioprio

[Jim Salter]

TL;DR scrub's ioprio argument isn't really helpful - a scrub murders 
system performance til it's done.

My system:

3.11 kernel (from Ubuntu Saucy)
btrfs-tools from 2013-07 (from Debian Sid)
Opteron 8-core CPU
32GB RAM
4 WD 1TB Black drives in a btrfs RAID10 (data and metadata).

iotop shows that the idle priority really is set on the processes:

 >> me@box:~$ sudo iotop -bn 1 | grep idle
 >> 28326 idle root       82.06 M/s    0.00 B/s  0.00 %  0.00 % btrfs 
scrub start -c3 /data
 >> 28327 idle root       89.30 M/s    0.00 B/s  0.00 %  0.00 % btrfs 
scrub start -c3 /data
 >> 28329 idle root       84.47 M/s    0.00 B/s  0.00 %  0.00 % btrfs 
scrub start -c3 /data
 >> 28331 idle root       79.64 M/s    0.00 B/s  0.00 %  0.00 % btrfs 
scrub start -c3 /data

But unfortunately, despite being on "idle" priority, the scrub is 
KILLING system performance. This is one of my eight CPU cores. Any core 
with a significant amount of non-idle time has three to four times as 
much wait time as it does user/system time. Brutal:

 >> Cpu4  :  4.4%us, 16.1%sy,  0.0%ni, 12.4%id, 67.2%wa, 0.0%hi,  
0.0%si,  0.0%st

Anybody got any ideas that might make the scrub go a little less... 
enthusiastically... and leave the system more usable?  Right now, with a 
scrub running, the system is so burdened that it may take as much as two 
to three seconds to so much as display a manpage. God help you if you 
want to, say, log into a VM running on the system. It'll GET there, but 
it'll look like it's iSCSI on TCP over carrier pigeon.

Re: btrfs scrub ioprio

[Austin S Hemmelgarn]

On 2013-11-24 22:45, Jim Salter wrote:
> TL;DR scrub's ioprio argument isn't really helpful - a scrub murders
> system performance til it's done.
> 
> My system:
> 
> 3.11 kernel (from Ubuntu Saucy)
> btrfs-tools from 2013-07 (from Debian Sid)
> Opteron 8-core CPU
> 32GB RAM
> 4 WD 1TB Black drives in a btrfs RAID10 (data and metadata).
> 
> iotop shows that the idle priority really is set on the processes:
> 
>>> me@box:~$ sudo iotop -bn 1 | grep idle
>>> 28326 idle root       82.06 M/s    0.00 B/s  0.00 %  0.00 % btrfs
> scrub start -c3 /data
>>> 28327 idle root       89.30 M/s    0.00 B/s  0.00 %  0.00 % btrfs
> scrub start -c3 /data
>>> 28329 idle root       84.47 M/s    0.00 B/s  0.00 %  0.00 % btrfs
> scrub start -c3 /data
>>> 28331 idle root       79.64 M/s    0.00 B/s  0.00 %  0.00 % btrfs
> scrub start -c3 /data
> 
> But unfortunately, despite being on "idle" priority, the scrub is
> KILLING system performance. This is one of my eight CPU cores. Any core
> with a significant amount of non-idle time has three to four times as
> much wait time as it does user/system time. Brutal:
> 
>>> Cpu4  :  4.4%us, 16.1%sy,  0.0%ni, 12.4%id, 67.2%wa, 0.0%hi, 
> 0.0%si,  0.0%st
> 
> Anybody got any ideas that might make the scrub go a little less...
> enthusiastically... and leave the system more usable?  Right now, with a
> scrub running, the system is so burdened that it may take as much as two
> to three seconds to so much as display a manpage. God help you if you
> want to, say, log into a VM running on the system. It'll GET there, but
> it'll look like it's iSCSI on TCP over carrier pigeon.
A large part of this might be that the idle I/O scheduling class does
not put a limit on bandwidth utilization, so even though it is using
'idle' bandwidth, it is actually using all of the bandwidth that is
'unused' by other tasks, which kills your disk latency.  As i see it,
the best option is probably either:
1. set up something to suspend the scrub if there are other pending I/O
requests (which would probably need kernel support)
2. allow the user to set a reasonable I/O bandwidth limit on the scrub
processes (you could already do this with the BIO cgroup, but it would
be nice to not need that to be compiled into the kernel to have this happen)

Re: btrfs scrub ioprio

[Jim Salter]

Can you elaborate on this please?  I'm not directly familiar with 
cgroups, I'd greatly appreciate a quick-and-dirty example of using BIO 
cgroup to limit I/O bandwidth.

Limiting bandwidth definitely would ameliorate the problem for me; I 
already use pv's bw-limiting feature to make btrfs send | btrfs receive 
tolerable.

On 11/25/2013 07:25 AM, Austin S Hemmelgarn wrote:
> 2. allow the user to set a reasonable I/O bandwidth limit on the scrub 
> processes (you could already do this with the BIO cgroup, but it would 
> be nice to not need that to be compiled into the kernel to have this 
> happen) 

Re: btrfs scrub ioprio

[Austin S Hemmelgarn]

On 2013-11-25 07:55, Jim Salter wrote:
> Can you elaborate on this please?  I'm not directly familiar with
> cgroups, I'd greatly appreciate a quick-and-dirty example of using BIO
> cgroup to limit I/O bandwidth.
> 
> Limiting bandwidth definitely would ameliorate the problem for me; I
> already use pv's bw-limiting feature to make btrfs send | btrfs receive
> tolerable.
> 
> On 11/25/2013 07:25 AM, Austin S Hemmelgarn wrote:
>> 2. allow the user to set a reasonable I/O bandwidth limit on the scrub
>> processes (you could already do this with the BIO cgroup, but it would
>> be nice to not need that to be compiled into the kernel to have this
>> happen) 
> 
I'm not very clear on the specifics, (the only cgroup I personally use
with any regularity is freezer), but I would suggest looking at
Documentation/cgroups/blkio-controller.txt in the kernel source tree
(cgroups are one of the better subsystems WRT documentation).  I'm
pretty certain that most distributions have the required kernel options
compiled in by default, and automatically mount the needed filesystems.

Re: btrfs scrub ioprio

[Holger Hoffstaette]

On Sun, 24 Nov 2013 22:45:59 -0500, Jim Salter wrote:

> TL;DR scrub's ioprio argument isn't really helpful - a scrub murders
> system performance til it's done.
> 
> My system:
> 
> 3.11 kernel (from Ubuntu Saucy)

I don't run Ubuntu, but *maybe* they use the deadline IO scheduler by
default, which does not handle IO priorities (by design). So maybe make
sure your devices don't say "deadline" in /sys/block/sdX/queue/scheduler,
and if they do, set them to cfq.

-h

Re: btrfs scrub ioprio

[Kai Krakow]

Holger Hoffstaette <holger.hoffstaette@googlemail.com> schrieb:

> On Sun, 24 Nov 2013 22:45:59 -0500, Jim Salter wrote:
> 
>> TL;DR scrub's ioprio argument isn't really helpful - a scrub murders
>> system performance til it's done.
>> 
>> My system:
>> 
>> 3.11 kernel (from Ubuntu Saucy)
> 
> I don't run Ubuntu, but *maybe* they use the deadline IO scheduler by
> default, which does not handle IO priorities (by design). So maybe make
> sure your devices don't say "deadline" in /sys/block/sdX/queue/scheduler,
> and if they do, set them to cfq.

OTOH I've switched from CFQ to Deadline in Gentoo and IO responsiveness has 
greatly improved since then (spinning rust). The system boots faster, is 
able to start programs faster during IO stress, but swapping seems to stress 
the system a lot more. But since I got 16 GB RAM this is not a big problem. 
Throughput is not as good as with CFQ - but this is a desktop system: 
latency counts more than throughput, and that improved a lot.

I've tried BFQ before switching to Deadline but BFQ just does not cut it. 
During high IO loads it simply does not feel interactive although it was 
designed to guarantee exactly that. The fact that it also made my kernel 
unstable (with bfq related backtraces in the kernel log) I've switched away 
from it.

But I have to admit that I'm using the following setting to reduce processes 
blocking on high IO loads, tho you should have enough RAM to use it:

$ cat /etc/local.d/nr_requests.start 
#!/bin/sh

for nr_requests in /sys/block/sd[abc]/queue/nr_requests; do
        echo -n 4096 >$nr_requests
done

My btrfs pool spans sd[abc].

I'm using a weekly scrubbing job and a daily rsync backup job and see no 
drawbacks from using deadline scheduler, rather it improved a lot.

@OP:

I suggest looking at the surrounding parameters also for trouble shooting: 
Maybe the SATA driver just doesn't cut it? Maybe a BIOS setting is wrong 
(AHCI mode?)... Does your system run low on RAM (with deadline this can 
become a heavy bottleneck)? Also, 3.12 kernel improved btrfs performance, 
you may want to try that first.

Did you check which IO scheduler is in use?

BTW: Yes, Ubuntu has switched from CFQ to Deadline by default for servers 
and desktops - with the same finding I had when trying. CFQ is more or less 
optimized for the general usecase and tries to behave like the old 
anticipatory scheduler, but it is designed with "fair" bandwidth 
distribution in mind which hurts interactivity a lot. But desktop systems 
feel fast by reducing latency, not by improving or fair-sharing bandwidth. 
Most servers need low latency for good service response times, throughput is 
not important. So deadline is not a bad choice for most use-cases (I know, 
"most use-cases" is subjective but I think for the edge-cases which benefit 
from CFQ the user already knows how to switch to the right scheduler). We've 
switched all our virtualized servers from CFQ to noop and they are running a 
lot better now with much decreased IO latency and lower IO wait. I think 
"fair bandwidth distribution" is just no longer that important with devices 
being able to deliver 100+ MB/s of bandwidth and low latency becomes more 
and more important...

But: YMMV... ;-)

Regards,
Kai