Re: Very various speed of grep operation on btrfs partition

[Austin S Hemmelgarn <ahferroin7@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 5345 bytes --]

On 2015-12-06 22:32, Duncan wrote:
> FWIW, I build kde without the semantic-desktop stuff even enabled at
> build-time (gentoo offers that option) here.  All the kdepim stuff (kmail,
> etc) uses it, so I dumped the several kdepim related apps (kmail,
> akregator, kaddressbook) I used here and found alternatives.  I don't
> normally need the indexing, which only takes space for the index and
> lowers performance, so it's all turned off at build-time.
Personally, I just avoid both KDE and GNOME.  For me efficiency is most 
important, and XFCE beats both at that (I've done testing between the 
three on my laptop with equivalent settings, and XFCE gives me about 
50-70% more battery life than either KDE or GNOME, and a much smaller 
memory footprint), followed closely by lack of lock-in (both GNOME and 
KDE have a very 'all or nothing' feel to them, and in the case of GNOME, 
it really is all or nothing).  I do have to say though, Okular is by far 
the best document viewer available for Linux.
>
>> A day later noticed that the effect of the cache is missing:
>> real 4m33.940s user 0m0.862s sys 0m1.711s
>
> That's probably due to something knocking it out of cache overnite.  If
> you have a cronjob running nitely to update the locate-variant database,
> for example, as many distros do by default, that'd do it, as that scans
> pretty much the entire filesystem, typically many times the size of RAM,
> thus trashing cache.
>
> The indexer could potentially wipe out cache too, particularly on lower
> memory machines, if it's actively indexing files, as that would normally
> pull what it's indexing into cache, throwing something else that hasn't
> been used for awhile away, unless the indexer is smart enough to do
> direct access and thus not disturb cache, since it's single-time access
> and caching it isn't going to do anything but force stuff from cache you
> use more frequently.
Somehow I doubt that the indexer is using O_DIRECT, while that bypasses 
cache, it also makes things really slow, which is directly counter to 
their intent to finish indexing as fast as possible (supposedly to 
minimize performance impact, but that's at odds with their behavior of 
trashing the cache).  That, and the slowest part of indexing is calling 
stat() on everything (stat is one of the slowest filesystem related 
system calls, and is in general usually near the top of the list of 
calls to avoid when doing real-time work, or for that matter anything 
that needs to be fast).
>
>> As I understand to solve my problem just need to do the cache is always
>> effective, even if memory occupied by other applications.
>>
>> Is possible to specify minimal size of disk cache?
>
> AFAIK, not directly.  What happens is that rather than leave the memory
> empty, the kernel caches stuff as it reads it.  If the memory is needed
> for apps, it's reclaimed from cache and used for apps.  So Linux systems
> tend to run close to zero really free memory, unless you just dropped
> caches or rebooted, or you just used some memory hog and it's done and
> just freed its memory, and you haven't read enough files since then to
> fill that memory back up with cache.
Yep, there's no direct way to force it to a fixed size.  Personally I 
would love to be able to reserve some fixed amount of RAM for disk 
caching, but I also usually run with _a lot_ of swap (usually on the 
order of 4 to 16 times physical RAM, I do work sometimes on really big 
images).
>
> However, if you're running swap, there's an adjustment, file
> /proc/sys/vm/swappiness, but would be set on most distros using the
> sysctrl config (/etc/sysctl.conf and/or /etc/sysctl.d/*), 0-100, that
> normally controls the balance preference between swapping apps out to
> keep cache (nearer 100) vs. dumping cache to keep more apps in RAM
> instead of swapped out (near 0). IIRC the default is 60.
You may also want to look into /proc/sys/vm/vfs_cache_pressure as well, 
the lower that is, the less likely it is that pages in the filesystem 
cache will be reclaimed.  This tends to have a bigger impact if all you 
care about is the filesystem cache.  Don't set it below about 50 though, 
otherwise it becomes very easy to run the system out of memory.
>
> Obviously if you're not running swap, all app memory must be kept in
> physical RAM as it can't be swapped out, and cache simply uses what's
> left.
>
>> Pity that I can't do 'echo 3 > /proc/sys/vm/drop_caches' on Windows
>> machine. It be interesting how fast grep would be work without cache.
>
> FWIW, I jumped off of MS when they started shipping malware[1] as part of
> the OS, with eXPrivacy.  So I've no idea if they've something similar,
> tho I'd be somewhat surprised if they didn't, at least as some obscure
> and possibly undocumented system call, so you'd have to call it from a
> program written for that purpose, instead of having it exposed such that
> any admin with suitable privs can do it with a single line command using
> only shell builtins, as Linux does.
Windows has no way that I know of outside of kernel mode to force the 
filesystem cache to be dropped.  I'm not sure if installing Cygwin and 
running sync from that works or not, but that's really the only thing I 
know of that might do it.



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]