Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning

[James Dingwall <james.dingwall@zynstra.com>]

Jan Beulich wrote:
>>>> On 09.12.13 at 18:50, James Dingwall <james.dingwall@zynstra.com> wrote:
>> Since 3.11 I have noticed that the OOM killer quite frequently triggers
>> in my Xen guest domains which use ballooning to increase/decrease their
>> memory allocation according to their requirements.  One example domain I
>> have has a maximum memory setting of ~1.5Gb but it usually idles at
>> ~300Mb, it is also configured with 2Gb swap which is almost 100% free.
>>
>> # free
>>                total       used       free     shared    buffers cached
>> Mem:        272080     248108      23972          0 1448      63064
>> -/+ buffers/cache:     183596      88484
>> Swap:      2097148          8    2097140
>>
>> There is plenty of available free memory in the hypervisor to balloon to
>> the maximum size:
>> # xl info | grep free_mem
>> free_memory            : 14923
> But you don't tell us how you trigger the process of re-filling
> memory. Yet that's the crucial point here; the fact that there is
> enough memory available in the hypervisor is only a necessary
> prerequisite.
My guest systems are Gentoo based and most often the problems happen 
while running emerge (Python script).  The example trace was taken from 
an emerge command launched via a cron script which runs emerge --sync ; 
emerge --deep --newuse -p -u world.  Almost all the other times I have 
seen the behaviour have been during emerge --deep --newuse -u world 
runs, most often during the build of large packages such as kdelibs, 
seamonkey or libreoffice.  However occasionally I have seen it with 
smaller builds or during the package merge phase where files are indexed 
and copied in to the live filesystem.

I have done some testing with the program below which successfully fills 
all memory and swap before being killed.  One thought I had was that 
perhaps there was some issue around a balloon down/up when the rate at 
which memory is being requested and released is high.  I will try 
another program with a random pattern of malloc/free to see if I can 
make a test case to help with a bisect.

Thanks,
James

#include <stdlib.h>

extern
int main(int argc, char *argv[])
{
         int leak_size = 1024 * 1024;

         while(1) {
                 malloc(leak_size);
         }

         return(0);
}

This is the output of vmstat 1 leading up to a kill of g++:

procs -----------memory---------- ---swap-- -----io---- -system-- 
----cpu----
  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy 
id wa
  2  0    836  42564     72  35912    0    0    56     0 14049 13585 17 
46 35  2
  2  0    836  42564     72  35916    0    0    56     0 14214 13747 17 
46 34  2
  2  0    836  42616     72  35840    0    0   112     0 14007 13564 17 
46 36  2
  2  0    836  42428     72  36024    0    0   236     0 12453 12025 15 
42 37  6
  2  0    796  41672     72  36104    0    0    44     8 14709 14243 18 
48 34  0
  2  0    796  41932     72  36288    0    0   148     0 13987 13484 17 
45 36  2
  2  0    796  42020     72  36196    0    0    60     0 13749 13288 17 
45 35  2
  3  0    796  41924     72  36284    0    0    72     0 14390 13868 17 
47 34  2
  2  0    796  42168     72  36032    0    0    40     0 11211 10819 14 
37 37 12
  2  0    760  41460     72  36116    0    0   108     8 14427 13946 18 
48 34  1
  2  0    760  35900     72  41600    0    0   108     0 13773 13362 18 
48 32  2
  2  0    760  35940     72  41576    0    0    56     0 14452 13978 18 
47 34  1
  2  0    760  35956     72  41540    0    0    88     0 14415 13975 17 
48 34  1
  2  0    760  35824     72  41648    0    0   104     0 14370 13925 17 
46 35  2
  3  0    724  35112     72  41776    0    0    96  1480 14271 13794 17 
46 35  2
  2  0    724  35012     72  41908    0    0   104     0 14457 13972 17 
47 34  1
  2  0    724  34884     72  42004    0    0   184     0 13801 13347 17 
45 37  2
  2  0    724  34888     72  42012    0    0    72    19 14590 14150 17 
48 34  1
  2  0    724  34720     72  42100    0    0   152    26 10802 10579 13 
35 38 13
  2  0    692  34044     72  42140    0    0    20    17 14573 14097 18 
47 34  1
  3  0    692  34116     72  42072    0    0    20     0 14542 14052 17 
47 35  1
  0  0    692  38540     72  42400    0    0   220     0 11981 11613 16 
42 42  1
  0  1    692  32224     72  43252    0    0  1460     0 4313 4377 13 13 
61 13
  5  0    692  31380     72  45084    0    0  1816     0 3241 3403 3  8 
28 61
  3  0    660  29552     72  47012    0    0  1060    14 14843 15905 22 
49 14 15
  2  0    660  17460     72  55592    0    0  7856    71 14869 16513 23 
51 15 12
  1  1    660  16732     72  56520    0    0   892    88 10888 17232 36 
53  6  6
  3  0    660  16964     72  57788    0    0   640     0 15741 18390 26 
57 11  6
  2  1    660  14448     72  60112    0    0  1556    40 14830 16736 22 
50 15 13
  3  0    628  10300     72  62424    0    0   996     0 15679 17393 24 
54 12 11
  3  0    628   9412     72  64216    0    0   504     0 17873 19182 27 
60  9  4
  3  0    628   7724     72  65480    0    0   332     0 16445 18588 25 
57 11  8
  1  1    628  12284     72  61088    0    0   212  5144 15099 17314 30 
62  5  3
  2  0    628  17264     72  53488    0    0    60     0 11906 17634 40 
53  2  5
  3  0    600  17440     72  53996    0    0   388    34 17648 20469 27 
63  6  5
  2  0    600  16684     72  54224    0    0   456   182 17288 20085 27 
62  8  3
  3  0    600  15112     72  56100    0    0   792     8 15689 17476 24 
55 11 10
  3  0    600  11752     72  58464    0    0   772     0 18287 20021 27 
62  7  5
  0  2    600   8628     72  60744    0    0  1536     0 15871 18785 24 
56 12  8
  3  0    568   8120     72  61820    0    0   820    38 10480 12030 17 
35 37 11
  2  0    568   4548     72  49460    0    0  5408  3156 7301 7390 7 27 
60  6
  1  0   6792   4592     72  41252    0    0  3172     0 6733 6601 12 38 
35 16
  1  0   1212 135860     72  41972    0    0  3192   136 5099 5011 10 34 
31 25