Re: [Xen-users] unexpected Out Of Memory (OOM)

[Olivier Bonvalet <xen.list@daevel.fr>]

Le mercredi 07 août 2013 à 11:16 +0100, Ian Campbell a écrit :
> On Wed, 2013-08-07 at 11:35 +0200, Olivier Bonvalet wrote:
> > I have a DomU which is currently running with 8GB of memory (confirmed
> > in /proc/meminfo), and only 6GB of memory is used, without process using
> > it :
> 
> Can we see the guest cfg file and the full guest console log please.
> 
> Ian.
> 
> 

Yes, I had to reboot the VM, but I can easily reproduce the problem with
a that kind of loop (the memory seem to not be released) :

daevel@to:~$ while [ 1 ] ; do free | grep Mem ; done
Mem:       8217004    1434084    6782920          0      11868     150632
Mem:       8217004    1434084    6782920          0      11868     150632
Mem:       8217004    1434140    6782864          0      11868     150632
Mem:       8217004    1434140    6782864          0      11868     150632
Mem:       8217004    1434196    6782808          0      11868     150632
Mem:       8217004    1434196    6782808          0      11868     150632
Mem:       8217004    1434252    6782752          0      11868     150632
Mem:       8217004    1434252    6782752          0      11868     150632
Mem:       8217004    1434308    6782696          0      11868     150632
Mem:       8217004    1434308    6782696          0      11868     150632
Mem:       8217004    1434308    6782696          0      11868     150632
Mem:       8217004    1434388    6782616          0      11868     150632


So, the cfg :

name		= 'reto'
vcpus		= 1
maxvcpus	= 8
memory		= 8192
vif		= [ 'mac=0e:00:00:00:8e:70,bridge=vlan' ]
disk		= [ 
                  '/dev/rbd/sas3copies/reto-root,,xvda,w',
                  '/dev/rbd/sas3copies/reto-home,,xvdb,w',
                  '/dev/rbd/sas3copies/reto-var,,xvdc,w',
                  '/dev/rbd/sas3copies/reto-mysql,,xvdd,w',
                  '/dev/rbd/sas3copies/reto-exim,,xvde,w',

		  '/dev/loop1,raw,xvdy,r',
		  '/dev/loop2,raw,xvdz,r' ]
kernel		= '/etc/xen/kernels/reto/vmlinuz'
ramdisk		= '/etc/xen/kernels/reto/initrd.img'
root		= '/dev/xvda ro rootfstype=ext4'
extra		= 'panic=60'



Then the console :

Parsing config from /etc/xen/reto.cfg
Daemon running with PID 20283
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 2.6.50-dae-xen (root@yiu) (gcc version 4.8.1 (Debian 4.8.1-8) ) #2 SMP Sun Aug 4 22:42:05 CEST 2013
[    0.000000] Command line: root=/dev/xvda ro rootfstype=ext4 panic=60
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
[    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[    0.000000] Xen: [mem 0x0000000000100000-0x00000001ffffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] MPS support code is not built-in.
[    0.000000] Using acpi=off or acpi=noirq or pci=noacpi may have problem
[    0.000000] DMI not present or invalid.
[    0.000000] e820: last_pfn = 0x200000 max_arch_pfn = 0x400000000
[    0.000000] e820: last_pfn = 0x100000 max_arch_pfn = 0x400000000
[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[    0.000000] init_memory_mapping: [mem 0x1ffe00000-0x1ffffffff]
[    0.000000] init_memory_mapping: [mem 0x1fc000000-0x1ffdfffff]
[    0.000000] init_memory_mapping: [mem 0x180000000-0x1fbffffff]
[    0.000000] init_memory_mapping: [mem 0x00100000-0x17fffffff]
[    0.000000] RAMDISK: [mem 0x01bbf000-0x02541fff]
[    0.000000] NUMA turned off
[    0.000000] Faking a node at [mem 0x0000000000000000-0x00000001ffffffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000-0x1ffffffff]
[    0.000000]   NODE_DATA [mem 0x1ff017000-0x1ff019fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
[    0.000000]   Normal   [mem 0x100000000-0x1ffffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00001000-0x0009ffff]
[    0.000000]   node   0: [mem 0x00100000-0x1ffffffff]
[    0.000000] smpboot: Allowing 8 CPUs, 0 hotplug CPUs
[    0.000000] No local APIC present
[    0.000000] APIC: disable apic facility
[    0.000000] APIC: switched to apic NOOP
[    0.000000] e820: cannot find a gap in the 32bit address range
[    0.000000] e820: PCI devices with unassigned 32bit BARs may break!
[    0.000000] e820: [mem 0x200100000-0x2004fffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 4.2.2 (preserve-AD)
[    0.000000] setup_percpu: NR_CPUS:32 nr_cpumask_bits:32 nr_cpu_ids:8 nr_node_ids:1
[    0.000000] PERCPU: Embedded 27 pages/cpu @ffff8801fea00000 s78400 r8192 d24000 u262144
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 2064266
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: root=/dev/xvda ro rootfstype=ext4 panic=60
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340
[    0.000000] Memory: 8205120k/8388608k available (3405k kernel code, 388k absent, 183100k reserved, 3194k data, 736k init)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	RCU restricting CPUs from NR_CPUS=32 to nr_cpu_ids=8.
[    0.000000] NR_IRQS:4352 nr_irqs:336 16
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [hvc0] enabled
[    0.000000] installing Xen timer for CPU 0
[    0.000000] tsc: Detected 3100.028 MHz processor
[    0.004000] Calibrating delay loop (skipped), value calculated using timer frequency.. 6200.05 BogoMIPS (lpj=12400112)
[    0.004000] pid_max: default: 32768 minimum: 301
[    0.004000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[    0.004000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
[    0.004000] Mount-cache hash table entries: 256
[    0.004202] Initializing cgroup subsys devices
[    0.004274] CPU: Physical Processor ID: 0
[    0.004280] CPU: Processor Core ID: 0
[    0.004289] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0
[    0.004289] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32
[    0.004289] tlb_flushall_shift: 5
[    0.012395] cpu 0 spinlock event irq 17
[    0.012443] Performance Events: unsupported p6 CPU model 45 no PMU driver, software events only.
[    0.012648] NMI watchdog: disabled (cpu0): hardware events not enabled
[    0.012709] installing Xen timer for CPU 1
[    0.012720] cpu 1 spinlock event irq 24
[    0.012762] SMP alternatives: switching to SMP code
[    0.020179] installing Xen timer for CPU 2
[    0.020196] cpu 2 spinlock event irq 31
[    0.020400] installing Xen timer for CPU 3
[    0.020410] cpu 3 spinlock event irq 38
[    0.020583] installing Xen timer for CPU 4
[    0.020595] cpu 4 spinlock event irq 45
[    0.020774] installing Xen timer for CPU 5
[    0.020784] cpu 5 spinlock event irq 52
[    0.020981] installing Xen timer for CPU 6
[    0.020992] cpu 6 spinlock event irq 59
[    0.021205] installing Xen timer for CPU 7
[    0.021219] cpu 7 spinlock event irq 66
[    0.021369] Brought up 8 CPUs
[    0.021440] devtmpfs: initialized
[    0.024352] Grant tables using version 2 layout.
[    0.024372] Grant table initialized
[    0.024424] NET: Registered protocol family 16
[    0.033098] dca service started, version 1.12.1
[    0.033624] PCI: setting up Xen PCI frontend stub
[    0.036205] bio: create slab <bio-0> at 0
[    0.036308] ACPI: Interpreter disabled.
[    0.036328] xen/balloon: Initialising balloon driver.
[    0.036328] xen-balloon: Initialising balloon driver.
[    0.040081] vgaarb: loaded
[    0.040146] SCSI subsystem initialized
[    0.040256] EDAC MC: Ver: 3.0.0
[    0.040308] PCI: System does not support PCI
[    0.040312] PCI: System does not support PCI
[    0.040433] Switching to clocksource xen
[    0.040497] pnp: PnP ACPI: disabled
[    0.041786] NET: Registered protocol family 2
[    0.042016] TCP established hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.042270] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.042388] TCP: Hash tables configured (established 65536 bind 65536)
[    0.042420] TCP: reno registered
[    0.042441] UDP hash table entries: 4096 (order: 5, 131072 bytes)
[    0.042476] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes)
[    0.042547] NET: Registered protocol family 1
[    0.042586] Unpacking initramfs...
[    0.049470] Freeing initrd memory: 9740k freed
[    0.052127] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.052914] sha1_ssse3: Using AVX optimized SHA-1 implementation
[    0.052962] sha256_ssse3: Using AVX optimized SHA-256 implementation
[    0.052994] sha512_ssse3: Using AVX optimized SHA-512 implementation
[    0.053303] Initialise module verification
[    0.053315] audit: initializing netlink socket (disabled)
[    0.053329] type=2000 audit(1375874154.717:1): initialized
[    0.055398] msgmni has been set to 16044
[    0.055821] Key type asymmetric registered
[    0.055827] Asymmetric key parser 'x509' registered
[    0.055871] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[    0.055878] io scheduler noop registered (default)
[    0.055881] io scheduler deadline registered
[    0.055911] io scheduler cfq registered
[    0.055965] ioatdma: Intel(R) QuickData Technology Driver 4.00
[    0.056082] Event-channel device installed.
[    0.206386] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    0.206624] Linux agpgart interface v0.103
[    0.640842] Loading iSCSI transport class v2.0-870.
[    0.640958] Initialising Xen virtual ethernet driver.
[    0.686248] blkfront: xvda: barrier or flush: disabled using persistent grants
[    0.690168]  xvda: unknown partition table
[    0.691395] i8042: PNP: No PS/2 controller found. Probing ports directly.
[    1.714687] i8042: No controller found
[    1.774442] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    1.774487] rtc_cmos: probe of rtc_cmos failed with error -38
[    1.774504] xen_wdt: Xen WatchDog Timer Driver v0.01
[    1.774545] xen_wdt: initialized (timeout=60s, nowayout=0)
[    1.774606] device-mapper: uevent: version 1.0.3
[    1.774659] device-mapper: ioctl: 4.24.0-ioctl (2013-01-15) initialised: dm-devel@redhat.com
[    1.774761] Netfilter messages via NETLINK v0.30.
[    1.774836] ip_tables: (C) 2000-2006 Netfilter Core Team
[    1.774941] TCP: cubic registered
[    1.774945] Initializing XFRM netlink socket
[    1.774951] NET: Registered protocol family 17
[    1.774962] NET: Registered protocol family 15
[    1.774971] Key type dns_resolver registered
[    1.775049] Loading module verification certificates
[    1.775717] MODSIGN: Loaded cert 'Magrathea: Glacier signing key: 6138b126c58f99141b59d11f9726480fc8670364'
[    1.775730] registered taskstats version 1
[    1.776382] blkfront: xvdb: barrier or flush: disabled using persistent grants
[    1.797557]  xvdb: unknown partition table
[    1.798526] blkfront: xvdc: barrier or flush: disabled using persistent grants
[    1.820815]  xvdc: unknown partition table
[    1.822259] blkfront: xvdd: barrier or flush: disabled using persistent grants
[    1.853766]  xvdd: unknown partition table
[    1.854749] blkfront: xvde: barrier or flush: disabled using persistent grants
[    1.857572]  xvde: unknown partition table
[    1.858010] Setting capacity to 2097152
[    1.858018] xvda: detected capacity change from 0 to 1073741824
[    1.858760] blkfront: xvdy: flush diskcache: enabled using persistent grants
[    1.860885]  xvdy: unknown partition table
[    1.861913] blkfront: xvdz: flush diskcache: enabled using persistent grants
[    1.863770]  xvdz: unknown partition table
[    1.864014] Setting capacity to 12582912
[    1.864022] xvdb: detected capacity change from 0 to 6442450944
[    1.904203] Setting capacity to 2097152
[    1.904221] xvdc: detected capacity change from 0 to 1073741824
[    1.904815] Setting capacity to 2097152
[    1.904824] xvdd: detected capacity change from 0 to 1073741824
[    1.905588] Setting capacity to 2097152
[    1.905595] xvde: detected capacity change from 0 to 1073741824
[    1.906034] Setting capacity to 8192
[    1.906040] xvdy: detected capacity change from 0 to 4194304
[    1.906344] Setting capacity to 34816
[    1.906351] xvdz: detected capacity change from 0 to 17825792
[    1.972186] drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
[    1.972672] Freeing unused kernel memory: 736k freed
[    1.972885] Write protecting the kernel read-only data: 6144k
[    1.975337] Freeing unused kernel memory: 680k freed
[    1.975809] Freeing unused kernel memory: 728k freed
Loading, please wait...
[    2.008230] udev[147]: starting version 164
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... done.
[    2.281423] EXT4-fs (xvda): INFO: recovery required on readonly filesystem
[    2.281436] EXT4-fs (xvda): write access will be enabled during recovery
[    2.789083] EXT4-fs (xvda): recovery complete
[    2.794546] EXT4-fs (xvda): mounted filesystem with ordered data mode. Opts: (null)
Begin: Running /scripts/local-bottom ... [    2.854763] EXT4-fs (xvdz): mounted filesystem without journal. Opts: (null)
[    2.896036] EXT4-fs (xvdy): mounted filesystem without journal. Opts: (null)
done.
done.
Begin: Running /scripts/init-bottom ... done.
INIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
Reading early boot files....
Starting the hotplug events dispatcher: udevd[    7.844815] udev[323]: starting version 164
.
Synthesizing the initial hotplug events...done.
Waiting for /dev to be fully populated...done.
Setting up LVM Volume Groups  Reading all physical volumes.  This may take a while...
  Found volume group "vg-reto-mysql" using metadata type lvm2
[    8.406895] bio: create slab <bio-1> at 1
  1 logical volume(s) in volume group "vg-reto-mysql" now active
.
Activating swap...done.
[    8.478687] EXT4-fs (xvda): re-mounted. Opts: (null)
Checking root file system...fsck from util-linux-ng 2.17.2
/dev/xvda: clean, 18649/65536 files, 138314/262144 blocks
done.
[    8.518678] EXT4-fs (xvda): re-mounted. Opts: errors=remount-ro
[    8.561167] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
Loading kernel modules...done.
Cleaning up ifupdown....
[    8.595343] zram: module is from the staging directory, the quality is unknown, you have been warned.
[    8.596003] zram: Created 1 device(s) ...
Setting up networking....
Setting up swapspace version 1, size = 262140 KiB
no label, UUID=06150089-664d-4e1b-ba3a-21e10901a7d3
[    9.610429] Adding 262140k swap on /dev/zram0.  Priority:1337 extents:1 across:262140k SSFS
Activating lvm and md swap...done.
Checking file systems...fsck from util-linux-ng 2.17.2
/dev/xvdc: recovering journal
/dev/xvdc: clean, 3131/65536 files, 61136/262144 blocks
/dev/xvdb: recovering journal
/dev/xvdb: clean, 44609/393216 files, 593209/1572864 blocks (check in 4 mounts)
/dev/mapper/vg--reto--mysql-data: recovering journal
/dev/mapper/vg--reto--mysql-data: clean, 334/55104 files, 17135/220160 blocks
done.
Mounting local filesystems...[   10.716114] EXT4-fs (xvdc): mounted filesystem with ordered data mode. Opts: (null)
[   10.870314] EXT4-fs (xvdb): mounted filesystem with ordered data mode. Opts: (null)
[   10.905646] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: data=ordered
[   10.952142] xor: automatically using best checksumming function:
[   10.992068]    avx       : 23162.000 MB/sec
[   11.060081] raid6: sse2x1    4307 MB/s
[   11.128068] raid6: sse2x2    6500 MB/s
[   11.196029] raid6: sse2x4    9198 MB/s
[   11.196039] raid6: using algorithm sse2x4 (9198 MB/s)
[   11.196043] raid6: using ssse3x2 recovery algorithm
[   11.202944] bio: create slab <bio-2> at 2
[   11.204124] Btrfs loaded
[   11.205468] device fsid 1eaac82a-191b-4255-8899-1ca628038ceb devid 1 transid 7432 /dev/xvde
[   11.206992] btrfs: not using ssd allocation scheme
[   11.207001] btrfs: use lzo compression
done.
Activating swapfile swap...done.
Cleaning up temporary files....
Configuring network interfaces...done.
Cleaning up temporary files....
Reading later boot files....
Setting kernel variables ...error: "net.ipv6.bindv6only" is an unknown key
failed.
INIT: Entering runlevel: 2
Using makefile-style concurrent boot in runlevel 2.
Starting enhanced syslogd: rsyslogd.
Setting sysfs variables....
[   14.427621] NET: Registered protocol family 10
[   14.433196] ip6_tables: (C) 2000-2006 Netfilter Core Team
Starting web server: apache2Warning: DocumentRoot [/home/rodolphe_me/http/www/] does not exist
Warning: DocumentRoot [/home/rodolphe_me/http/www/] does not exist
.
Starting deferred execution scheduler: atd.
Not starting internet superserver: no services enabled.
Starting NTP server: ntpd.
Starting periodic command scheduler: cron.
[   15.714524] sshd (1122): /proc/1122/oom_adj is deprecated, please use /proc/1122/oom_score_adj instead.
Starting OpenBSD Secure Shell server: sshd.
Starting recursive DNS server: unbound[1375874170] unbound[1121:0] warning: increased limit(open files) from 1024 to 4140
.
Starting Zabbix agent: zabbix_agentd.
Starting MySQL database server: mysqld.
Checking for corrupt, not cleanly closed and upgrade needing tables..
Starting ftp server: Running: /usr/sbin/pure-ftpd -f ftp -l pam -k 95 -0 -Z -D -c 500 -Y 1 -w -O clf:/var/log/pure-ftpd/transfer.log -A -8 UTF-8 -L 5000:5 -E -H -y 5:50 -p 40000:41000 -u 1000 -I 15 -C 10 -B
Starting MTA: exim4.

Debian GNU/Linux 6.0 to hvc0

to login: 



Thanks for your time.










_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel