Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning

[Bob Liu <bob.liu@oracle.com>]

On 01/15/2014 04:49 PM, James Dingwall wrote:
> Bob Liu wrote:
>> On 01/07/2014 05:21 PM, James Dingwall wrote:
>>> Bob Liu wrote:
>>>> Could you confirm that this problem doesn't exist if loading tmem with
>>>> selfshrinking=0 during compile gcc? It seems that you are compiling
>>>> difference packages during your testing.
>>>> This will help to figure out whether selfshrinking is the root cause.
>>> Got an oom with selfshrinking=0, again during a gcc compile.
>>> Unfortunately I don't have a single test case which demonstrates the
>>> problem but as I mentioned before it will generally show up under
>>> compiles of large packages such as glibc, kdelibs, gcc etc.
>>>
>> So the root cause is not because enabled selfshrinking.
>> Then what I can think of is that the xen-selfballoon driver was too
>> aggressive, too many pages were ballooned out which causeed heavy memory
>> pressure to guest OS.
>> And kswapd started to reclaim page until most of pages were
>> unreclaimable(all_unreclaimable=yes for all zones), then OOM Killer was
>> triggered.
>> In theory the balloon driver should give back ballooned out pages to
>> guest OS, but I'm afraid this procedure is not fast enough.
>>
>> My suggestion is reserve a min memory for your guest OS so that the
>> xen-selfballoon won't be so aggressive.
>> You can do it through parameters selfballoon_reserved_mb or
>> selfballoon_min_usable_mb.
> I am still getting OOM errors with both of these set to 32 so I'll try
> another bump to 64.  I think that if I do find values which prevent it
> though then it is more of a work around than a fix because it still
> suggests that swap is not being used when ballooning is no longer

Yes, it's like a work around. But I don't think there is a better way.

>From the recent OOM log your reported:
[ 8212.940769] Free swap  = 1925576kB
[ 8212.940770] Total swap = 2097148kB

[504638.442136] Free swap  = 1868108kB
[504638.442137] Total swap = 2097148kB

171572KB and 229040KB data are swapped out to swap disk, I think there
are already significantly values for guest OS has only ~300M usable memory.
guest OS can't find out pages suitable for swap any more after so many
pages are swapped out, although at that time the swap device still have
enough space.

The OOM may not be triggered if the balloon driver can give back memory
to guest OS fast enough but I think it's unrealistic.
So the best way is reserve more memory to guest OS.

> capable of satisfying the request.  I've also got an Ubuntu Saucy (3.11
> kernel) guest running on the dom0 with tmem activated so I'm going to
> see if I can find a comparable workload to see if I get the same issue
> with a different kernel configuration.
> 
> James

-- 
Regards,
-Bob