Re: isolate_freepages_block and excessive CPU usage by OSD process

[Christian Marie <christian@ponies.io>]

[-- Attachment #1: Type: text/plain, Size: 1802 bytes --]

> Hello,
> 
> I had found recently that the OSD daemons under certain conditions
> (moderate vm pressure, moderate I/O, slightly altered vm settings) can
> go into loop involving isolate_freepages and effectively hit Ceph
> cluster performance.

Hi! I'm the creator of the server fault issue you reference:

http://serverfault.com/questions/642883/cause-of-page-fragmentation-on-large-server-with-xfs-20-disks-and-ceph

I'd like to get to the bottom of this very much, I'm seeing a very similar
pattern on 3.10.0-123.9.3.el7.x86_64, if this is fixed in later versions
perhaps we could backport something.

Here is some perf output:

http://ponies.io/raw/compaction.png

Looks pretty similar. I also have hundreds of MB logs and traces should we need
some specific question answered.

I've managed to reproduce many failed compactions with this:

https://gist.github.com/christian-marie/cde7e80c5edb889da541

I took some compaction stress test code and bolted on a little loop to mmap a
large sparse file and read every PAGE_SIZEth byte.

Run it once, compactions seem to do okay, run it again and they're really slow.
This seems to be because my little trick to fill up cache memory only seems to
work exactly half the time. Note that transhuge pages are only used to
introduce fragmentation/pressure here, turning transparent huge pages off
doesn't seem to make the slightest difference to the spinning-in-reclaim issue.

We are using Mellanox ipoib drivers which do not do scatter-gather, so I'm
currently working on adding support for that (the hardware supports it). Are
you also using ipoib or have something else doing high order allocations? It's
a bit concerning for me if you don't as it would suggest that cutting down on
those allocations won't help.

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]