Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning

[Bob Liu <bob.liu@oracle.com>]

On 12/26/2013 04:42 PM, James Dingwall wrote:
> Bob Liu wrote:
>> On 12/20/2013 03:08 AM, James Dingwall wrote:
>>> Bob Liu wrote:
>>>> On 12/12/2013 12:30 AM, James Dingwall wrote:
>>>>> Bob Liu wrote:
>>>>>> On 12/10/2013 11:27 PM, Konrad Rzeszutek Wilk wrote:
>>>>>>> On Tue, Dec 10, 2013 at 02:52:40PM +0000, James Dingwall wrote:
>>>>>>>> Konrad Rzeszutek Wilk wrote:
>>>>>>>>> On Mon, Dec 09, 2013 at 05:50:29PM +0000, James Dingwall wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Since 3.11 I have noticed that the OOM killer quite frequently
>>>>>>>>>> triggers in my Xen guest domains which use ballooning to
>>>>>>>>>> increase/decrease their memory allocation according to their
>>>>>>>>>> requirements.  One example domain I have has a maximum memory
>>>>>>>>>> setting of ~1.5Gb but it usually idles at ~300Mb, it is also
>>>>>>>>>> configured with 2Gb swap which is almost 100% free.
>>>>>>>>>>
>>>>>>>>>> # free
>>>>>>>>>>                 total       used       free     shared    buffers
>>>>>>>>>> cached
>>>>>>>>>> Mem:        272080     248108      23972          0 1448     
>>>>>>>>>> 63064
>>>>>>>>>> -/+ buffers/cache:     183596      88484
>>>>>>>>>> Swap:      2097148          8    2097140
>>>>>>>>>>
>>>>>>>>>> There is plenty of available free memory in the hypervisor to
>>>>>>>>>> balloon to the maximum size:
>>>>>>>>>> # xl info | grep free_mem
>>>>>>>>>> free_memory            : 14923
>>>>>>>>>>
>>>>>>>>>> An example trace (they are always the same) from the oom
>>>>>>>>>> killer in
>>>>>>>>>> 3.12 is added below.  So far I have not been able to reproduce
>>>>>>>>>> this
>>>>>>>>>> at will so it is difficult to start bisecting it to see if a
>>>>>>>>>> particular change introduced this.  However it does seem that the
>>>>>>>>>> behaviour is wrong because a) ballooning could give the guest
>>>>>>>>>> more
>>>>>>>>>> memory, b) there is lots of swap available which could be used
>>>>>>>>>> as a
>>>>>>>>>> fallback.
>>>>>>> Keep in mind that swap with tmem is actually no more swap. Heh, that
>>>>>>> sounds odd -but basically pages that are destined for swap end up
>>>>>>> going in the tmem code which pipes them up to the hypervisor.
>>>>>>>
>>>>>>>>>> If other information could help or there are more tests that I
>>>>>>>>>> could
>>>>>>>>>> run then please let me know.
>>>>>>>>> I presume you have enabled 'tmem' both in the hypervisor and in
>>>>>>>>> the guest right?
>>>>>>>> Yes, domU and dom0 both have the tmem module loaded and  tmem
>>>>>>>> tmem_dedup=on tmem_compress=on is given on the xen command line.
>>>>>>> Excellent. The odd thing is that your swap is not used that much,
>>>>>>> but
>>>>>>> it should be (as that is part of what the self-balloon is suppose to
>>>>>>> do).
>>>>>>>
>>>>>>> Bob, you had a patch for the logic of how self-balloon is suppose
>>>>>>> to account for the slab - would this be relevant to this problem?
>>>>>>>
>>>>>> Perhaps, I have attached the patch.
>>>>>> James, could you please apply it and try your application again? You
>>>>>> have to rebuild the guest kernel.
>>>>>> Oh, and also take a look at whether frontswap is in use, you can
>>>>>> check
>>>>>> it by watching "cat /sys/kernel/debug/frontswap/*".
>>>>> I have tested this patch with a workload where I have previously seen
>>>>> failures and so far so good.  I'll try to keep a guest with it
>>>>> stressed
>>>>> to see if I do get any problems.  I don't know if it is expected but I
>>>> By the way, besides longer time of kswapd, is this patch work well
>>>> during your stress testing?
>>>>
>>>> Have you seen the OOM killer triggered quite frequently again?(with
>>>> selfshrink=true)
>>>>
>>>> Thanks,
>>>> -Bob
>>> It was looking good until today (selfshrink=true).  The trace below is
>>> during a compile of subversion, it looks like the memory has ballooned
>>> to almost the maximum permissible but even under pressure the swap disk
>>> has hardly come in to use.
>>>
>> So if without selfshrink the swap disk can be used a lot?
>>
>> If that's the case, I'm afraid the frontswap-selfshrink in
>> xen-selfballoon did something incorrect.
>>
>> Could you please try this patch which make the frontswap-selfshrink
>> slower and add a printk for debug.
>> Please still keep selfshrink=true in your test but can with or without
>> my previous patch.
>> Thanks a lot!
>>
> The oom trace below was triggered during a compile of gcc.  I have the
> full dmesg from boot which shows all the printks, please let me know if
> you would like to see that.
> 

Sorry for the later response.
Could you confirm that this problem doesn't exist if loading tmem with
selfshrinking=0 during compile gcc? It seems that you are compiling
difference packages during your testing.
This will help to figure out whether selfshrinking is the root cause.

Thanks,
-Bob