Re: [Resend] Puzzling behaviour with multiple swap targets

[Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>]

Hi,
regarding another issue I was working together with Mel Gorman and I can 
now confirm that his patch https://lkml.org/lkml/2014/2/13/181
also fixes the issue discussed in this thread.

Thereby I want to encourage you to review and if appropriate pick Mels 
patch fixing not only the issue he described but also this one.

On 28/01/14 16:47, Christian Ehrhardt wrote:
> On 20/01/14 09:54, Christian Ehrhardt wrote:
>> On 20/01/14 02:05, Shaohua Li wrote:
>>> On Fri, Jan 17, 2014 at 01:39:43PM +0100, Christian Ehrhardt wrote:
> [...]
>>>
>>> Is the swap disk a SSD? If not, there is no point to partition the
>>> disk. Do you
>>> see any changes in iostat in the bad/good case, for example, request
>>> size,
>>> iodepth?
>>
>> Hi,
>> I use normal disks and SSDs or even the special s390 ramdisks - I agree
>> that partitioning makes no sense in a real case, but it doesn't matter
>> atm. I just partition to better show the effect that "more swap targets
>> -> less throughput" - and partitioning makes it easy for me to guarantee
>> that the HW ressources serving that I/O stay the same.
>>
>> IOstat and such things don't report very significant changes regarding
>> I/O depth. Sizes are more interesting with the bad case having slightly
>> more (16%) read I/Os and dropping average request size from 14.62 to
>> 11.89. Along with that goes a drop in read request merges of 28%.
>>
>> But I don't see how a workload that is random in memory would create
>> significantly better/worse chances for request merging depending on the
>> case if the disk is partitioned more or less often.
>> On the read path swap doesn't care about iterating disks, it just goes
>> by associated swap extends -> offsets to the disk.
>> And I thought in a random load that should be purely random and hit each
>> partition in e.g. the 4 partition case just by 25%.
>> I checked some blocktraces I had and can confirm as expected each got an
>> equal share.
>>
>>> There is one patch can avoid swapin reads more than swapout for random
>>> case,
>>> but still not in upstream yet. You can try it here:
>>> https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/mm/swap_state.c?id=5d19b04a2dae73382fb607f16e2acfb594d1c63f
>>>
>>>
>>
>> Great suggestion - it sounds very interesting to me, I'll give it a try
>> in a few days since I'm out Tue/Wed.
>
> I had already a patch prepared and successfully tested that allows to
> configure page cluster for read/write separately from userspace. That
> worked well but would require an admin to configure the "right" value
> for his system.
> Since that fails with so much kernel tunables and would also not be
> adaptive if the behaviour changes over time I very much prefer your
> solution to it.
> That is why I tried to verify your patch in my environment with at least
> some of the cases I used recently for swap analysis and improvement.
>
> The environment has 10G of real memory and drives a working set of 12.5Gb.
> So just a slight 1.25:1 overcommit (While s390 often runs in higher
> overcommits for most of the linux Swap issues so far 1.25:1 was enough
> to trigger and produced more reliable results).
> To swap I use 8x16G xpram devices which one can imagine as SSDs at main
> memory speed (good to make forecasts how ssds might behave in a few years).
>
> I compared a 3.10 kernel (I know a bit old already, but I knew that my
> env works fine with it) with and without the patch for swap readahead
> scaling.
>
> All memory is initially completely faulted in (memset) and thne warmed
> up with two full sweeps of the entire working set following the current
> workload configuration.
> The unit reported is MB/s the workload can achieve in its
> (overcommitted) memory being an average of 2 runs for 5 minutes each (+
> the init and warmup as described).
> (Noise is usually ~+/-5%, maybe a bit more in non exclusive runs like
> this when other things are on the machine)
>
> Memory Access is done via memcpy in either direction (R/W) with
> alternating sizes of:
> 5% 65536 bytes
> 5%  8192 bytes
> 90% 4096 bytes
>
> Further abbreviations
> PC = the currently configured page cluster size (0,3,5)
> M - Multi threaded (=32)
> S - Single threaded
> Seq/Rnd - Sequential/Random
>
>                        No Swap RA   With Swap RA     Diff
> PC0-M-Rnd        ~=     10732.97        9891.87   -7.84%
> PC0-M-Seq        ~=     10780.56       10587.76   -1.79%
> PC0-S-Rnd        ~=      2010.47        2067.51    2.84%
> PC0-S-Seq        ~=      1783.74        1834.28    2.83%
> PC3-M-Rnd        ~=     10745.19       10990.90    2.29%
> PC3-M-Seq        ~=     11792.67       11107.79   -5.81%
> PC3-S-Rnd        ~=      1301.28        2017.61   55.05%
> PC3-S-Seq        ~=      1664.40        1637.72   -1.60%
> PC5-M-Rnd        ~=      7568.56       10733.60   41.82%
> PC5-M-Seq        ~=          n/a       11208.40      n/a
> PC5-S-Rnd        ~=       608.48        2052.17  237.26%
> PC5-S-Seq        ~=      1604.97        1685.65    5.03%
> (for and PC5-M-Seq I ran out of time, but the remaining results are
> interesting enough already)
>
> I like what I see, there is nothing significantly out of the noise range
> which shouldn't be.
> The Page Cluster 0 cases didn't show an effect as expected.
> For page cluster 3 the multithreaded cases have hidden the impact to TP
> due to the fact that then just another thread can continue.
> But I checked sar data and see that PC3-M-Rnd has avoided about 50% of
> swapins while staying at equal throughput (1000k vs 500k pswpin/s).
> Other than that Random loads had the biggest improvements matching what
> I had with splitting up read/write page-cluster size.
> Eventually with page cluster 5 even the multi threaded cases start to
> show benefits of the readahead scaling code.
> In all that time sequential cases didn't change a lot.
>
> So I think that test worked fine. I see there were some discussion son
> the form of the implementation, but in terms of results I really like it
> as far as I had time to check it out.
>
>
>
> *** Context switch ***
>
> Now back to my original question about why swapping to multiple targets
> makes things slower.
> Your patch helps there a but as the Workload with the biggest issue was
> a random workload and I knew that with pagecluster set to zero the loss
> of efficiency with those multiple swap targets is stopped.
> But I consider that only a fix of the symptom and would love if one
> comes up with an idea actually *why* things get worse with more swap
> targets.
>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>