Re: OOM when adding ipv6 route: How to make available more per-cpu memory?

Eric Dumazet
2010-11-05T18:06:06Z

Le vendredi 05 novembre 2010 à 10:19 -0700, Ben Greear a écrit :
> We are testing 500 mac-vlans with IPv6 addresses on a 32-bit kernel
> (2.6.36 + ubuntu patches + our patches) We have one
> routing table per interface. It seems that some of them cannot
> add routes due to lack of memory.
>
> root@lanforge-ubuntu:/home/lanforge# ip route show table 29
> root@lanforge-ubuntu:/home/lanforge# ./local/sbin/ip -6 route add default via 2001:98::1 dev eth12#15 table 29
> RTNETLINK answers: Cannot allocate memory
>
> I see errors such as this in the logs:
>
> [507107.846864] PERCPU: allocation failed, size=2048 align=4, failed to allocate new chunk
> [507107.846867] Pid: 3246, comm: ip Tainted: P 2.6.36-1-ct #7
> [507107.846869] Call Trace:
> [507107.846874] [<c05d4996>] ? printk+0x2d/0x2f
> [507107.846878] [<c021405e>] pcpu_alloc+0x32e/0x360
> [507107.846880] [<c02140bf>] __alloc_percpu+0xf/0x20
> [507107.846883] [<c0558d6d>] snmp_mib_init+0x3d/0x70
> [507107.846885] [<c0585692>] ipv6_add_dev+0x132/0x350
> [507107.846887] [<c0557f39>] ? inetdev_init+0xb9/0x180
> [507107.846889] [<c05895ff>] addrconf_notify+0x3f/0x490
> [507107.846891] [<c055884f>] ? inetdev_event+0x20f/0x280
> [507107.846894] [<c05daef3>] notifier_call_chain+0x43/0x60
> [507107.846897] [<c016df4f>] raw_notifier_call_chain+0x1f/0x30
> [507107.846899] [<c05008ec>] call_netdevice_notifiers+0x2c/0x60
> [507107.846901] [<c0502ecc>] register_netdevice+0x23c/0x380
> [507107.846907] [<f80aecf7>] macvlan_common_newlink+0x187/0x2a0 [macvlan]
> [507107.846910] [<f80ae540>] ? macvlan_setup+0x0/0x20 [macvlan]
> [507107.846912] [<f80aee37>] macvlan_newlink+0x27/0x30 [macvlan]
> [507107.846914] [<c0503060>] ? netif_rx+0x0/0x120
> [507107.846915] [<c05033c0>] ? dev_forward_skb+0x0/0xf0
> [507107.846917] [<f80aee10>] ? macvlan_newlink+0x0/0x30 [macvlan]
> [507107.846920] [<c050ea94>] rtnl_newlink+0x464/0x590
> [507107.846921] [<c050e7e6>] ? rtnl_newlink+0x1b6/0x590
> [507107.846927] [<c050cd2d>] rtnetlink_rcv_msg+0x13d/0x230
> [507107.846929] [<c05d721f>] ? _raw_spin_lock_irqsave+0x2f/0x50
> [507107.846931] [<c050e630>] ? rtnl_newlink+0x0/0x590
> [507107.846933] [<c050cbf0>] ? rtnetlink_rcv_msg+0x0/0x230
> [507107.846935] [<c0522276>] netlink_rcv_skb+0x86/0xb0
> [507107.846936] [<c050cbdc>] rtnetlink_rcv+0x1c/0x30
> [507107.846938] [<c0521f69>] netlink_unicast+0x259/0x280
> [507107.846940] [<c0522ba8>] netlink_sendmsg+0x1f8/0x300
> [507107.846943] [<c04f0b79>] sock_sendmsg+0xd9/0x100
> [507107.846945] [<c013e0dd>] ? finish_task_switch+0x3d/0xc0
> [507107.846947] [<c04f0b79>] ? sock_sendmsg+0xd9/0x100
> [507107.846950] [<c013361d>] ? kmap_atomic_prot+0xcd/0xf0
> [507107.846957] [<f88bd559>] ? h_d_revalidate+0xe9/0x240 [aufs]
> [507107.846958] [<c013361d>] ? kmap_atomic_prot+0xcd/0xf0
> [507107.846962] [<c0364c7d>] ? _copy_from_user+0x3d/0x130
> [507107.846965] [<c04f947a>] ? verify_iovec+0x5a/0xa0
> [507107.846966] [<c04f112d>] sys_sendmsg+0x15d/0x290
> [507107.846972] [<f88c05f1>] ? aufs_fault+0xf1/0x110 [aufs]
> [507107.846974] [<c013361d>] ? kmap_atomic_prot+0xcd/0xf0
> [507107.846977] [<c01f8726>] ? handle_mm_fault+0x146/0x400
> [507107.846979] [<c05dac0d>] ? do_page_fault+0x1cd/0x470
> [507107.846981] [<c023419d>] ? alloc_fd+0xbd/0xf0
> [507107.846983] [<c0364c7d>] ? _copy_from_user+0x3d/0x130
> [507107.846986] [<c04f183b>] sys_socketcall+0xeb/0x2a0
> [507107.846989] [<c0151d99>] ? irq_exit+0x39/0x70
> [507107.846991] [<c021bb2e>] ? sys_open+0x2e/0x40
> [507107.846993] [<c05d77a4>] syscall_call+0x7/0xb
>
>
> But, it appears to me that we have plenty of memory available.
>
> top - 10:09:13 up 6 days, 15:56, 9 users, load average: 0.37, 0.39, 0.45
> Tasks: 211 total, 2 running, 209 sleeping, 0 stopped, 0 zombie
> Cpu(s): 0.7%us, 2.1%sy, 0.0%ni, 97.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Mem: 3604332k total, 1510252k used, 2094080k free, 84928k buffers
> Swap: 154620k total, 0k used, 154620k free, 1045912k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
>
> root@lanforge-ubuntu:/home/lanforge# cat /proc/meminfo
> MemTotal: 3604332 kB
> MemFree: 2094568 kB
> Buffers: 84928 kB
> Cached: 1045920 kB
> SwapCached: 0 kB
> Active: 466784 kB
> Inactive: 805632 kB
> Active(anon): 148988 kB
> Inactive(anon): 6788 kB
> Active(file): 317796 kB
> Inactive(file): 798844 kB
> Unevictable: 0 kB
> Mlocked: 0 kB
> HighTotal: 2752008 kB
> HighFree: 1512380 kB
> LowTotal: 852324 kB
> LowFree: 582188 kB
> SwapTotal: 154620 kB
> SwapFree: 154620 kB
> Dirty: 492 kB
> Writeback: 0 kB
> AnonPages: 141316 kB
> Mapped: 39428 kB
> Shmem: 14208 kB
> Slab: 170788 kB
> SReclaimable: 81904 kB
> SUnreclaim: 88884 kB
> KernelStack: 2520 kB
> PageTables: 4516 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 1956784 kB
> Committed_AS: 582604 kB
> VmallocTotal: 122880 kB
> VmallocUsed: 48592 kB
> VmallocChunk: 24280 kB
> HardwareCorrupted: 0 kB
> HugePages_Total: 0
> HugePages_Free: 0
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 4096 kB
> DirectMap4k: 12280 kB
> DirectMap4M: 897024 kB
>
>
> Is there any way to tune the system so that it has more memory available
> for per-cpu data structures?

CC linux-kernel and Tejun Heo

How many possible cpus do you have ?
head -1 /proc/interrupts

and please post :
cat /proc/vmallocinfo

Thanks