Re: Oddly slow read performance with near-full largish FS

[Austin S Hemmelgarn <ahferroin7@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 2805 bytes --]

On 2014-12-21 17:53, Charles Cazabon wrote:
> Hi, Robert,
>
> My performance issues with btrfs are more-or-less resolved now -- the
> performance under btrfs still seems quite variable compared to other
> filesystems -- my rsync speed is now varying between 40MB and ~90MB/s, with
> occasional intervals where it drops further, down into the 10-20MB/s range.
> Still no disk errors or SMART warnings that would indicate that problem is at
> the hardware level.
>
>> Do make sure you are regularly running the long "offline" test.
>
> Ok, I'll do that.
>
>> Otherwise SMART is just going to tell you the disk just died when it dies.
>
> Ya, I'm aware of how limited/useful the SMART diagnostics are.  I'm also
> paranoid enough to be using RAID 6...
>
>> The thing with "movablecore=" will not lead to an "out of memory"
>> condition or not, its a question of cache and buffer evictions.
>
> I'm fairly certain memory isn't the issue here.  For what it's worth:
>
> %Cpu(s):  2.1 us, 19.4 sy,  0.0 ni, 78.0 id,  0.2 wa,  0.3 hi,  0.0 si,  0.0 st
> KiB Mem:  16469880 total, 16301252 used,   168628 free,      720 buffers
> KiB Swap:  7811068 total,        0 used,  7811068 free, 15146580 cached
>
> Swappiness I've left at the default of 60, but I'm not seeing swapping going
> on regardless.
>
>>> Is this remaining difference (25 vs 100+ MB/s) simply due to btrfs not being
>>> tuned for performance yet
>
> I found the cause of this.  Stupidly enough, there was a bwlimit set up in a
> shell alias for rsync.
>
> So btrfs is not nearly as slow as I was seeing.  It's still slower than
> reading from an ext4 or XFS filesystem on these disks, but the absolute level
> of read speed seems reasonable enough given that btrfs has not been under
> heavy performance tuning to date.  My only remaining concern would be the
> variability I still see in the read speed.
This actually sounds kind of like the issues I have sometimes on my 
laptop using btrfs on an SSD, I've mostly resolved them by tuning IO 
scheduler parameters, as the default IO scheduler (the supposedly 
Completely Fair Queue, which was obviously named by a mathematician who 
had never actually run the algorithm) has some pretty brain-dead default 
settings.  The other thing I would suggest looking into regarding the 
variability is tuning the kernel's write-caching settings, with the 
defaults you're caching ~1.6G worth of writes before it forces 
write-back, which is a ridiculous amount;  I've that the highest value 
that is actually usable is about 256M, and that's only if you are doing 
mostly bursty IO and not the throughput focused stuff that rsync does, 
I'd say try setting /proc/sys/vm/dirty_background_bytes to 67108864 
(64M) and see if that helps things some.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2455 bytes --]