Re: Why is dedup inline, not delayed (as opposed to offline)? Explain like I'm five pls.

["Austin S. Hemmelgarn" <ahferroin7@gmail.com>]

On 2016-01-19 03:30, Duncan wrote:
> Austin S. Hemmelgarn posted on Mon, 18 Jan 2016 07:48:13 -0500 as
> excerpted:
>
>> On 2016-01-17 22:51, Duncan wrote:
>>>
>>> Checking a bit more my understanding, since you brought up the btrfs
>>> "commit=" mount option.
>>>
>>> I knew about the option previously, and obviously knew it worked in the
>>> same context as the page-cache stuff, but in my understanding the btrfs
>>> "commit=" mount option operates at the filesystem layer, not the
>>> general filesystem-vm layer controlled by /proc/sys/vm/dirty_*.  In my
>>> understanding, therefore, the two timeouts could effectively be added,
>>> yielding a maximum 1 minute (30 seconds btrfs default commit time plus
>>> 30 seconds vm expiry) commit time.
>>
>> In a way, yes, except the commit option controls when a transaction is
>> committed, and thus how often the log tree gets cleared.  It's
>> essentially saying 'ensure the filesystem is consistent without
>> replaying a log at least this often'.  AFAIUI, this doesn't guarantee
>> that you'll go that long without a transaction, but puts an upper bound
>> on it.  Looking at it another way, it pretty much says that you don't
>> care about losing the last n seconds of changes to the FS.
>
> Thanks.  That's the way I was treating it.
>
>> The sysctl values are a bit different, and control how long the kernel
>> will wait in the VFS layer to try and submit a larger batch of writes at
>> once, so that the block layer has more it can try to merge, and
>> hopefully things get written out faster as a result.  IOW, it's a knob
>> to control the VFS level write-back caching to try and tune for
>> performance.  This also ties in with
>> /proc/sys/vm/dirty_writeback_centisecs, which is how often after the
>> expiration hits that the kernel will flush a chunk of the cache, and
>> /proc/sys/vm/dirty_{background,}_{bytes,ratio} which puts an upper limit
>> on how much data will be buffered before trying to flush it out to
>> persistent storage.  You almost certainly want to change these, as they
>> defaults to 10% of system RAM, which is why it often takes a ridiculous
>> amount of time to unmount a flash drive that's been written to a lot.
>> dirty_{ratio,bytes} control the per-process limit, and
>> dirty_background_{ratio,bytes} control the system-wide limit.
>
> Got that too, and yes, I've been known to recommend to others changes to
> the now-days ridiculous 10% of system RAM buffer thing, as well. =:^)
> Random writes to spinning rust in particular may be 30 MiB/sec real-
> world, and 10% of 16 GiB is 1.6 GiB, 50-some seconds worth of writeback.
> When the timeout is 30 seconds and the backlog is nearly double that,
> something's wrong.  I set mine to 3% foreground (~ half a gig @ 16 GiB)
> and 1% (~160 MiB) background when I upgraded to 16 GiB RAM, tho now I
> have fast SSDs, but didn't see a need to boost it back up, as half a GiB
> is quite enough to have unsynced in case of a crash anyway.
Personally I usually just use small byte values (64MB for the system 
wide limit, and 4MB for the per-process limit).  I also do a decent 
amount of work with removable media (which takes longer to unmount the 
higher these are), and have good SSD's that do proper write-reordering 
and guarantee that writes will finish even if power dies in the middle, 
and don't care as much about write performance on my traditional disks 
(most of those are used as backing storage for VM's which can fit their 
entire working set in RAM, so having fast storage isn't as high priority 
for them).
>
> (Obviously once RAM goes above ~16 GiB, for systems not yet on fast SSD,
> the bytes values begin to make more sense to use than ratio, as 1% of RAM
> is simply no longer fine enough granularity.  But 1% of 16 GiB is ~163
> MiB, ~5 seconds worth @ 30 MiB/sec, so fine /enough/... barely.  The 3%
> foreground figure is then ~16 seconds worth of writeback, a bit
> uncomfortable if you're waiting on it, but comfortably below the 30
> second timeout and still at least tolerable in human terms, so not /too/
> bad.  And as I said, for me the system and /home are now on fast SSD, so
> in practice the only time I'm worrying about spinning rust transfer
> backlogs is on the media and backups drive, which is still spinning
> rust.  And it's tolerable there, so the ratio knobs continue to be fine,
> for my own use.)
>
>>> But that has always been an unverified on my part fuzzy assumption.
>>> The two times could be the same layer, with the btrfs mount option
>>> being a per-filesystem method of controlling the same thing that
>>> /proc/sys/vm/ dirty_expire_centisecs controls globally (as you seemed
>>> to imply above), or the two could be different layers but with the
>>> countdown times overlapping, both of which would result in a 30-second
>>> total timeout, instead of the 30+30=60 that I had assumed.
>>
>> The two timers do overlap.
>
> Good to have it verified. =:^)  The difference between 30 seconds and a
> minute's worth of work lost in a crash can be quite a lot, if one was
> copying a big set of small files at the time.
>
>>> And while we're at it, how does /proc/sys/vm/vfs_cache_pressure play
>>> into all this?  I know the dirty_* and how the dirty_*bytes vs.
>>> dirty_*ratio vs. dirty_*centisecs thing works, but don't quite
>>> understand how vfs_cache_pressure fits in with dirty_*.
>
>> vfs_cache_pressure controls how likely the kernel is to drop clean pages
>> (the documentation says just dentries and inodes, but I'm relatively
>> certain it's anything in the VFS cache) from the VFS cache to get memory
>> to allocate.  The higher this is, the more likely the VFS cache is to
>> get invalidated.  In general, you probably want to increase this on
>> systems that have fast storage (like SSD's or really good SAS RAID
>> arrays, 150 is usually a decent start), and decrease it if you have
>> really slow storage (Like a Raspberry Pi for example).  Setting this too
>> low (below about 50) however, will give you a very high chance of
>> getting an OOM condition.
>
> So vfs_cache_pressure only applies if you're out of "free" memory, and
> the kernel has to decide whether to dump cache or OOM, correct?  On
> systems with enough memory, and with stuff like the local package cache
> and/or multimedia on separate partitions that are mounted only when
> needed and unmounted when not, so actual system-and-apps plus buffers
> plus cache memory generally stays reasonably below total RAM, with
> reasonable ulimits and tmpfs maximum sizes set so apps can't go hog-wild,
> there's zero cache pressure so this setting doesn't apply at all...
> unless/until there's a bad kernel leak and/or several apps go somewhat
> wild, plus something's maximizing a few of those tmpfs, all at once, of
> course.
Kind of, it comes into play any time the kernel goes to reclaim memory, 
which is usually to complete higher order allocations in kernel space 
(like allocating big DMA buffers or similar stuff).  It's important to 
note that it's not usually a factor in dealing with an OOM condition 
(unless you lower it, in which case it can be a big contributing 
factor).  As an example, say you plug in a USB NIC, the kernel has to 
allocate a lot of different things to be able to work with it reliably, 
and /proc/sys/vfs_cache_pressure tells it how much to favor dropping 
bits of the VFS cache to satisfy those allocations as opposed to other 
methods (like memory compaction, which can be expensive on big systems).
>
> (As I write this system/app memory usage is ~2350 MiB, buffers 4 MiB,
> cache 7321 MiB, total usage ~9680 MiB, on a 16 GiB system.  That's with
> about three days uptime, after mounting the packages partition and
> remounting / rw and doing a bunch of builds, then umounting the pkgs
> partition, killing X and running a lib_users check to ensure no services
> are running on outdated deleted libs and need restarted, remounting / ro,
> and restarting X.  At some point I had the media partition mounted too,
> but now it's unmounted again, dropping that cache.  So in addition to
> cache memory which /could/ be dumped if I had to, I have 6+ GiB of
> entirely idle unused memory.  Nice as I don't have swap configured, so if
> I'm out of RAM, I'm out, but there's a lot of cache to dump first before
> it gets that bad.  Meanwhile, zero cache pressure, and 6+ GiB of spare
> RAM to use for apps/tmpfs/cache if I need it, before any cache dumps at
> all! =:^)
I wish I could get away with running without swap :)  My laptop only has 
8G of RAM, and I run Xen on my desktop, which means I have significantly 
less than the 32G of installed RAM to work with from my desktop VM 
there, and if I don't use swap, I often end up killing the machine 
trying to do some of the multimedia work I sometimes do.  OTOH, I've got 
swap on an SSD on both systems, which gets me ridiculous performance 
since I've got them configured to swap in and out pages in groups the 
size of an erase block on the SSD (which also means that it's not 
tearing up the SSD as much either).
>
>> Documentation/sysctl/vm.txt in the kernel sources covers them, although
>> the documentation is a bit sparse even there.
>
> I knew the kernel's proc documentation in Documentation/filesystems/
> proc.txt, plus whatever outside resource it was that originally got me
> looking into the whole thing in the first place, I had the /proc/sys/vm/
> dirty_* files and their usage covered.  But the sysctl/* doc files and
> the the vfs_cache_pressure proc file, not so much, and as I said I didn't
> understand how the btrfs commit= mount option fit into all of this.  So
> now I have a rather better understanding of how it all fits together. =:^)
Glad I could help.  The sysctl options are one of the places I would 
love to see better documented, I just don't have the time and enough 
knowledge of them to do so myself.  There's still a significant number 
that aren't documented there at all (lots of them in /proc/sys/kernel).