Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning

[Bob Liu <bob.liu@oracle.com>]

On 01/29/2014 01:15 AM, James Dingwall wrote:
> Bob Liu wrote:
>>
>> I have made a patch by reserving extra 10% of original total memory, by
>> this way I think we can make the system much more reliably in all cases.
>> Could you please have a test? You don't need to set
>> selfballoon_reserved_mb by yourself any more.
> I have to say that with this patch the situation has definitely
> improved.  I have been running it with 3.12.[78] and 3.13 and pushing it
> quite hard for the last 10 days or so.  Unfortunately yesterday I got an

Good news!

> OOM during a compile (link) of webkit-gtk.  I think your patch is part
> of the solution but I'm not sure if the other bit is simply to be more
> generous with the guest memory allocation or something else.  Having
> tested with memory = 512  and no tmem I get an OOM with the same
> compile, with memory = 1024 and no tmem the compile completes ok (both
> cases without maxmem).  As my domains are usually started with memory =
> 512 and maxmem = 1024 it seems that there should be sufficient with my

But I think from the beginning tmem/balloon driver can't expand guest
memory from size 'memory' to 'maxmem' automatically.

> default parameters. Also for an experiment I set memory=1024 and removed
> maxmem and when tmem is activated I see "[ 3393.884105] xen:balloon:
> reserve_additional_memory: add_memory() failed: -17" printed many times
> in the guest kernel log.
> 

I'll take a look at it.

-- 
Regards,
-Bob

> Regards,
> James
> 
> [456770.748827] Mem-Info:
> [456770.748829] Node 0 DMA per-cpu:
> [456770.748833] CPU    0: hi:    0, btch:   1 usd:   0
> [456770.748835] CPU    1: hi:    0, btch:   1 usd:   0
> [456770.748836] Node 0 DMA32 per-cpu:
> [456770.748838] CPU    0: hi:  186, btch:  31 usd: 173
> [456770.748840] CPU    1: hi:  186, btch:  31 usd: 120
> [456770.748846] active_anon:91431 inactive_anon:96269 isolated_anon:0
>  active_file:13286 inactive_file:31256 isolated_file:0
>  unevictable:0 dirty:0 writeback:0 unstable:0
>  free:1155 slab_reclaimable:7001 slab_unreclaimable:3932
>  mapped:2300 shmem:88 pagetables:2576 bounce:0
>  free_cma:0 totalram:255578 balloontarget:327320
> [456770.748849] Node 0 DMA free:1956kB min:88kB low:108kB high:132kB
> active_anon:3128kB inactive_anon:3328kB active_file:1888kB
> inactive_file:2088kB unevictable:0kB isolated(anon):0kB
> isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB
> writeback:0kB mapped:32kB shmem:0kB slab_reclaimable:684kB
> slab_unreclaimable:720kB kernel_stack:72kB pagetables:488kB unstable:0kB
> bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:17841
> all_unreclaimable? yes
> [456770.748863] lowmem_reserve[]: 0 469 469 469
> [456770.748866] Node 0 DMA32 free:2664kB min:2728kB low:3408kB
> high:4092kB active_anon:362596kB inactive_anon:381748kB
> active_file:51256kB inactive_file:122936kB unevictable:0kB
> isolated(anon):0kB isolated(file):0kB present:1032192kB
> managed:1006400kB mlocked:0kB dirty:0kB writeback:0kB mapped:9168kB
> shmem:352kB slab_reclaimable:27320kB slab_unreclaimable:15008kB
> kernel_stack:1784kB pagetables:9816kB unstable:0kB bounce:0kB
> free_cma:0kB writeback_tmp:0kB pages_scanned:1382021 all_unreclaimable? yes
> [456770.748874] lowmem_reserve[]: 0 0 0 0
> [456770.748877] Node 0 DMA: 1*4kB (R) 0*8kB 0*16kB 5*32kB (R) 2*64kB (R)
> 1*128kB (R) 0*256kB 1*512kB (R) 1*1024kB (R) 0*2048kB 0*4096kB = 1956kB
> [456770.748890] Node 0 DMA32: 666*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB
> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2664kB
> [456770.748899] 48556 total pagecache pages
> [456770.748901] 35203 pages in swap cache
> [456770.748903] Swap cache stats: add 358621, delete 323418, find
> 206319/224002
> [456770.748904] Free swap  = 1671532kB
> [456770.748905] Total swap = 2097148kB
> [456770.748906] 262047 pages RAM
> [456770.748907] 0 pages HighMem/MovableOnly
> [456770.748908] 6448 pages reserved
> <snip process list>
> [456770.749070] Out of memory: Kill process 28271 (ld) score 110 or
> sacrifice child
> [456770.749073] Killed process 28271 (ld) total-vm:358488kB,
> anon-rss:324588kB, file-rss:1456kB
> 
>>
>>
>> xen_selfballoon_deaggressive.patch
>>
>>
>> diff --git a/drivers/xen/xen-selfballoon.c
>> b/drivers/xen/xen-selfballoon.c
>> index 21e18c1..8f33254 100644
>> --- a/drivers/xen/xen-selfballoon.c
>> +++ b/drivers/xen/xen-selfballoon.c
>> @@ -175,6 +175,7 @@ static void frontswap_selfshrink(void)
>>   #endif /* CONFIG_FRONTSWAP */
>>     #define MB2PAGES(mb)    ((mb) << (20 - PAGE_SHIFT))
>> +#define PAGES2MB(pages) ((pages) >> (20 - PAGE_SHIFT))
>>     /*
>>    * Use current balloon size, the goal (vm_committed_as), and hysteresis
>> @@ -525,6 +526,7 @@ EXPORT_SYMBOL(register_xen_selfballooning);
>>   int xen_selfballoon_init(bool use_selfballooning, bool
>> use_frontswap_selfshrink)
>>   {
>>       bool enable = false;
>> +    unsigned long reserve_pages;
>>         if (!xen_domain())
>>           return -ENODEV;
>> @@ -549,6 +551,26 @@ int xen_selfballoon_init(bool use_selfballooning,
>> bool use_frontswap_selfshrink)
>>       if (!enable)
>>           return -ENODEV;
>>   +    /*
>> +     * Give selfballoon_reserved_mb a default value(10% of total ram
>> pages)
>> +     * to make selfballoon not so aggressive.
>> +     *
>> +     * There are two reasons:
>> +     * 1) The goal_page doesn't contain some pages used by kernel space,
>> +     *    like slab cache and pages used by device drivers.
>> +     *
>> +     * 2) The balloon driver may not give back memory to guest OS fast
>> +     *    enough when the workload suddenly aquries a lot of memory.
>> +     *
>> +     * In both cases, the guest OS will suffer from memory pressure and
>> +     * OOM killer may be triggered.
>> +     * By reserving extra 10% of total ram pages, we can keep the system
>> +     * much more reliably and response faster in some cases.
>> +     */
>> +    if (!selfballoon_reserved_mb) {
>> +        reserve_pages = totalram_pages / 10;
>> +        selfballoon_reserved_mb = PAGES2MB(reserve_pages);
>> +    }
>>       schedule_delayed_work(&selfballoon_worker, selfballoon_interval
>> * HZ);
>>         return 0;
> 
>