From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eric Dumazet Subject: Re: OOM when adding ipv6 route: How to make available more per-cpu memory? Date: Fri, 05 Nov 2010 19:06:01 +0100 Message-ID: <1288980361.2882.1070.camel@edumazet-laptop> References: <4CD43C87.5040403@candelatech.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: NetDev , linux-kernel , Tejun Heo To: Ben Greear Return-path: Received: from mail-wy0-f174.google.com ([74.125.82.174]:40692 "EHLO mail-wy0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752573Ab0KESGG (ORCPT ); Fri, 5 Nov 2010 14:06:06 -0400 In-Reply-To: <4CD43C87.5040403@candelatech.com> Sender: netdev-owner@vger.kernel.org List-ID: Le vendredi 05 novembre 2010 =C3=A0 10:19 -0700, Ben Greear a =C3=A9cri= t : > We are testing 500 mac-vlans with IPv6 addresses on a 32-bit kernel > (2.6.36 + ubuntu patches + our patches) We have one > routing table per interface. It seems that some of them cannot > add routes due to lack of memory. >=20 > root@lanforge-ubuntu:/home/lanforge# ip route show table 29 > root@lanforge-ubuntu:/home/lanforge# ./local/sbin/ip -6 route add def= ault via 2001:98::1 dev eth12#15 table 29 > RTNETLINK answers: Cannot allocate memory >=20 > I see errors such as this in the logs: >=20 > [507107.846864] PERCPU: allocation failed, size=3D2048 align=3D4, fai= led to allocate new chunk > [507107.846867] Pid: 3246, comm: ip Tainted: P 2.6.36-1-ct= #7 > [507107.846869] Call Trace: > [507107.846874] [] ? printk+0x2d/0x2f > [507107.846878] [] pcpu_alloc+0x32e/0x360 > [507107.846880] [] __alloc_percpu+0xf/0x20 > [507107.846883] [] snmp_mib_init+0x3d/0x70 > [507107.846885] [] ipv6_add_dev+0x132/0x350 > [507107.846887] [] ? inetdev_init+0xb9/0x180 > [507107.846889] [] addrconf_notify+0x3f/0x490 > [507107.846891] [] ? inetdev_event+0x20f/0x280 > [507107.846894] [] notifier_call_chain+0x43/0x60 > [507107.846897] [] raw_notifier_call_chain+0x1f/0x30 > [507107.846899] [] call_netdevice_notifiers+0x2c/0x60 > [507107.846901] [] register_netdevice+0x23c/0x380 > [507107.846907] [] macvlan_common_newlink+0x187/0x2a0 [mac= vlan] > [507107.846910] [] ? macvlan_setup+0x0/0x20 [macvlan] > [507107.846912] [] macvlan_newlink+0x27/0x30 [macvlan] > [507107.846914] [] ? netif_rx+0x0/0x120 > [507107.846915] [] ? dev_forward_skb+0x0/0xf0 > [507107.846917] [] ? macvlan_newlink+0x0/0x30 [macvlan] > [507107.846920] [] rtnl_newlink+0x464/0x590 > [507107.846921] [] ? rtnl_newlink+0x1b6/0x590 > [507107.846927] [] rtnetlink_rcv_msg+0x13d/0x230 > [507107.846929] [] ? _raw_spin_lock_irqsave+0x2f/0x50 > [507107.846931] [] ? rtnl_newlink+0x0/0x590 > [507107.846933] [] ? rtnetlink_rcv_msg+0x0/0x230 > [507107.846935] [] netlink_rcv_skb+0x86/0xb0 > [507107.846936] [] rtnetlink_rcv+0x1c/0x30 > [507107.846938] [] netlink_unicast+0x259/0x280 > [507107.846940] [] netlink_sendmsg+0x1f8/0x300 > [507107.846943] [] sock_sendmsg+0xd9/0x100 > [507107.846945] [] ? finish_task_switch+0x3d/0xc0 > [507107.846947] [] ? sock_sendmsg+0xd9/0x100 > [507107.846950] [] ? kmap_atomic_prot+0xcd/0xf0 > [507107.846957] [] ? h_d_revalidate+0xe9/0x240 [aufs] > [507107.846958] [] ? kmap_atomic_prot+0xcd/0xf0 > [507107.846962] [] ? _copy_from_user+0x3d/0x130 > [507107.846965] [] ? verify_iovec+0x5a/0xa0 > [507107.846966] [] sys_sendmsg+0x15d/0x290 > [507107.846972] [] ? aufs_fault+0xf1/0x110 [aufs] > [507107.846974] [] ? kmap_atomic_prot+0xcd/0xf0 > [507107.846977] [] ? handle_mm_fault+0x146/0x400 > [507107.846979] [] ? do_page_fault+0x1cd/0x470 > [507107.846981] [] ? alloc_fd+0xbd/0xf0 > [507107.846983] [] ? _copy_from_user+0x3d/0x130 > [507107.846986] [] sys_socketcall+0xeb/0x2a0 > [507107.846989] [] ? irq_exit+0x39/0x70 > [507107.846991] [] ? sys_open+0x2e/0x40 > [507107.846993] [] syscall_call+0x7/0xb >=20 >=20 > But, it appears to me that we have plenty of memory available. >=20 > top - 10:09:13 up 6 days, 15:56, 9 users, load average: 0.37, 0.39,= 0.45 > Tasks: 211 total, 2 running, 209 sleeping, 0 stopped, 0 zombie > Cpu(s): 0.7%us, 2.1%sy, 0.0%ni, 97.2%id, 0.0%wa, 0.0%hi, 0.0%si= , 0.0%st > Mem: 3604332k total, 1510252k used, 2094080k free, 84928k buff= ers > Swap: 154620k total, 0k used, 154620k free, 1045912k cach= ed >=20 > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >=20 >=20 > root@lanforge-ubuntu:/home/lanforge# cat /proc/meminfo > MemTotal: 3604332 kB > MemFree: 2094568 kB > Buffers: 84928 kB > Cached: 1045920 kB > SwapCached: 0 kB > Active: 466784 kB > Inactive: 805632 kB > Active(anon): 148988 kB > Inactive(anon): 6788 kB > Active(file): 317796 kB > Inactive(file): 798844 kB > Unevictable: 0 kB > Mlocked: 0 kB > HighTotal: 2752008 kB > HighFree: 1512380 kB > LowTotal: 852324 kB > LowFree: 582188 kB > SwapTotal: 154620 kB > SwapFree: 154620 kB > Dirty: 492 kB > Writeback: 0 kB > AnonPages: 141316 kB > Mapped: 39428 kB > Shmem: 14208 kB > Slab: 170788 kB > SReclaimable: 81904 kB > SUnreclaim: 88884 kB > KernelStack: 2520 kB > PageTables: 4516 kB > NFS_Unstable: 0 kB > Bounce: 0 kB > WritebackTmp: 0 kB > CommitLimit: 1956784 kB > Committed_AS: 582604 kB > VmallocTotal: 122880 kB > VmallocUsed: 48592 kB > VmallocChunk: 24280 kB > HardwareCorrupted: 0 kB > HugePages_Total: 0 > HugePages_Free: 0 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 4096 kB > DirectMap4k: 12280 kB > DirectMap4M: 897024 kB >=20 >=20 > Is there any way to tune the system so that it has more memory availa= ble > for per-cpu data structures? CC linux-kernel and Tejun Heo How many possible cpus do you have ? head -1 /proc/interrupts and please post : cat /proc/vmallocinfo Thanks