* 3.5+, xfs and 32bit armhf - xfs_buf_get: failed to map pages
@ 2013-05-17 10:45 Paolo Pisati
2013-05-18 8:43 ` Jeff Liu
2013-05-19 1:13 ` Dave Chinner
0 siblings, 2 replies; 9+ messages in thread
From: Paolo Pisati @ 2013-05-17 10:45 UTC (permalink / raw)
To: xfs
While exercising swift on a single node 32bit armhf system running a 3.5 kernel,
i got this when i hit ~25% of fs space usage:
dmesg:
...
[ 3037.399406] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size.
[ 3037.399442] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size.
[ 3037.399469] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size.
[ 3037.399485] XFS (sda5): xfs_buf_get: failed to map pages
[ 3037.399485]
[ 3037.399501] XFS (sda5): Internal error xfs_trans_cancel at line 1466 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c. Caller 0xbf0235e0
[ 3037.399501]
[ 3037.413789] [<c00164cc>] (unwind_backtrace+0x0/0x104) from [<c04ed624>] (dump_stack+0x20/0x24)
[ 3037.413985] [<c04ed624>] (dump_stack+0x20/0x24) from [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs])
[ 3037.414321] [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs]) from [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs])
[ 3037.414654] [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs]) from [<bf0235e0>] (xfs_create+0x228/0x558 [xfs])
[ 3037.414953] [<bf0235e0>] (xfs_create+0x228/0x558 [xfs]) from [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs])
[ 3037.415239] [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs]) from [<bf01a8d0>] (xfs_vn_mkdir+0x20/0x24 [xfs])
[ 3037.415393] [<bf01a8d0>] (xfs_vn_mkdir+0x20/0x24 [xfs]) from [<c0135758>] (vfs_mkdir+0xc4/0x13c)
[ 3037.415410] [<c0135758>] (vfs_mkdir+0xc4/0x13c) from [<c013884c>] (sys_mkdirat+0xdc/0xe4)
[ 3037.415422] [<c013884c>] (sys_mkdirat+0xdc/0xe4) from [<c0138878>] (sys_mkdir+0x24/0x28)
[ 3037.415437] [<c0138878>] (sys_mkdir+0x24/0x28) from [<c000e320>] (ret_fast_syscall+0x0/0x30)
[ 3037.415452] XFS (sda5): xfs_do_force_shutdown(0x8) called from line 1467 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c. Return address = 0xbf06340c
[ 3037.416892] XFS (sda5): Corruption of in-memory data detected. Shutting down filesystem
[ 3037.425008] XFS (sda5): Please umount the filesystem and rectify the problem(s)
[ 3047.912480] XFS (sda5): xfs_log_force: error 5 returned.
flag@c13:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 225G 2.1G 212G 1% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 2.0G 4.0K 2.0G 1% /dev
tmpfs 405M 260K 404M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 2.0G 0 2.0G 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/sda1 228M 30M 186M 14% /boot
/dev/sda5 2.0G 569M 1.5G 28% /mnt/sdb1
flag@c13:~$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda2 14958592 74462 14884130 1% /
none 182027 1 182026 1% /sys/fs/cgroup
udev 177378 1361 176017 1% /dev
tmpfs 182027 807 181220 1% /run
none 182027 3 182024 1% /run/lock
none 182027 1 182026 1% /run/shm
none 182027 1 182026 1% /run/user
/dev/sda1 124496 35 124461 1% /boot
/dev/sda5 524288 237184 287104 46% /mnt/sdb1
the vmalloc space is ~256M usually on this box, so i enlarged it:
flag@c13:~$ dmesg | grep vmalloc
Kernel command line: console=ttyAMA0 nosplash vmalloc=512M
vmalloc : 0xdf800000 - 0xff000000 ( 504 MB)
and while i didn't hit the warning above, still after ~25% of usage, the storage
node died with:
May 17 06:26:00 c13 container-server ERROR __call__ error with PUT /sdb1/123172/AUTH_test/3b3d078015304a41b76b0ab083b7863a_5 : [Errno 28] No space
left on device: '/srv/1/node/sdb1/containers/123172' (txn: tx8ea3ce392ee94df096b16-00519605b0)
flag@c13:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 225G 3.9G 210G 2% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 2.0G 4.0K 2.0G 1% /dev
tmpfs 405M 260K 404M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 2.0G 0 2.0G 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/sda1 228M 25M 192M 12% /boot
/dev/sda5 2.0G 564M 1.5G 28% /mnt/sdb1
flag@c13:~$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda2 14958592 124409 14834183 1% /
none 114542 1 114541 1% /sys/fs/cgroup
udev 103895 1361 102534 2% /dev
tmpfs 114542 806 113736 1% /run
none 114542 3 114539 1% /run/lock
none 114542 1 114541 1% /run/shm
none 114542 1 114541 1% /run/user
/dev/sda1 124496 33 124463 1% /boot
/dev/sda5 524288 234880 289408 45% /mnt/sdb1
any idea what else shall i tune to workaround this? or is it a know problem that
involves 32bit arch and xfs?
--
bye,
p.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: 3.5+, xfs and 32bit armhf - xfs_buf_get: failed to map pages 2013-05-17 10:45 3.5+, xfs and 32bit armhf - xfs_buf_get: failed to map pages Paolo Pisati @ 2013-05-18 8:43 ` Jeff Liu 2013-05-19 1:13 ` Dave Chinner 1 sibling, 0 replies; 9+ messages in thread From: Jeff Liu @ 2013-05-18 8:43 UTC (permalink / raw) To: Paolo Pisati; +Cc: xfs On 05/17/2013 06:45 PM, Paolo Pisati wrote: > While exercising swift on a single node 32bit armhf system running a 3.5 kernel, > i got this when i hit ~25% of fs space usage: > > dmesg: > ... > [ 3037.399406] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. > [ 3037.399442] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. > [ 3037.399469] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. > [ 3037.399485] XFS (sda5): xfs_buf_get: failed to map pages > [ 3037.399485] > [ 3037.399501] XFS (sda5): Internal error xfs_trans_cancel at line 1466 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c. Caller 0xbf0235e0 > [ 3037.399501] > [ 3037.413789] [<c00164cc>] (unwind_backtrace+0x0/0x104) from [<c04ed624>] (dump_stack+0x20/0x24) > [ 3037.413985] [<c04ed624>] (dump_stack+0x20/0x24) from [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs]) > [ 3037.414321] [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs]) from [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs]) > [ 3037.414654] [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs]) from [<bf0235e0>] (xfs_create+0x228/0x558 [xfs]) > [ 3037.414953] [<bf0235e0>] (xfs_create+0x228/0x558 [xfs]) from [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs]) > [ 3037.415239] [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs]) from [<bf01a8d0>] (xfs_vn_mkdir+0x20/0x24 [xfs]) > [ 3037.415393] [<bf01a8d0>] (xfs_vn_mkdir+0x20/0x24 [xfs]) from [<c0135758>] (vfs_mkdir+0xc4/0x13c) > [ 3037.415410] [<c0135758>] (vfs_mkdir+0xc4/0x13c) from [<c013884c>] (sys_mkdirat+0xdc/0xe4) > [ 3037.415422] [<c013884c>] (sys_mkdirat+0xdc/0xe4) from [<c0138878>] (sys_mkdir+0x24/0x28) > [ 3037.415437] [<c0138878>] (sys_mkdir+0x24/0x28) from [<c000e320>] (ret_fast_syscall+0x0/0x30) > [ 3037.415452] XFS (sda5): xfs_do_force_shutdown(0x8) called from line 1467 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c. Return address = 0xbf06340c > [ 3037.416892] XFS (sda5): Corruption of in-memory data detected. Shutting down filesystem > [ 3037.425008] XFS (sda5): Please umount the filesystem and rectify the problem(s) > [ 3047.912480] XFS (sda5): xfs_log_force: error 5 returned. > > flag@c13:~$ df -h > Filesystem Size Used Avail Use% Mounted on > /dev/sda2 225G 2.1G 212G 1% / > none 4.0K 0 4.0K 0% /sys/fs/cgroup > udev 2.0G 4.0K 2.0G 1% /dev > tmpfs 405M 260K 404M 1% /run > none 5.0M 0 5.0M 0% /run/lock > none 2.0G 0 2.0G 0% /run/shm > none 100M 0 100M 0% /run/user > /dev/sda1 228M 30M 186M 14% /boot > /dev/sda5 2.0G 569M 1.5G 28% /mnt/sdb1 > > flag@c13:~$ df -i > Filesystem Inodes IUsed IFree IUse% Mounted on > /dev/sda2 14958592 74462 14884130 1% / > none 182027 1 182026 1% /sys/fs/cgroup > udev 177378 1361 176017 1% /dev > tmpfs 182027 807 181220 1% /run > none 182027 3 182024 1% /run/lock > none 182027 1 182026 1% /run/shm > none 182027 1 182026 1% /run/user > /dev/sda1 124496 35 124461 1% /boot > /dev/sda5 524288 237184 287104 46% /mnt/sdb1 > > the vmalloc space is ~256M usually on this box, so i enlarged it: > > flag@c13:~$ dmesg | grep vmalloc > Kernel command line: console=ttyAMA0 nosplash vmalloc=512M > vmalloc : 0xdf800000 - 0xff000000 ( 504 MB) > > and while i didn't hit the warning above, still after ~25% of usage, the storage > node died with: > > May 17 06:26:00 c13 container-server ERROR __call__ error with PUT /sdb1/123172/AUTH_test/3b3d078015304a41b76b0ab083b7863a_5 : [Errno 28] No space > left on device: '/srv/1/node/sdb1/containers/123172' (txn: tx8ea3ce392ee94df096b16-00519605b0) > > > flag@c13:~$ df -h > Filesystem Size Used Avail Use% Mounted on > /dev/sda2 225G 3.9G 210G 2% / > none 4.0K 0 4.0K 0% /sys/fs/cgroup > udev 2.0G 4.0K 2.0G 1% /dev > tmpfs 405M 260K 404M 1% /run > none 5.0M 0 5.0M 0% /run/lock > none 2.0G 0 2.0G 0% /run/shm > none 100M 0 100M 0% /run/user > /dev/sda1 228M 25M 192M 12% /boot > /dev/sda5 2.0G 564M 1.5G 28% /mnt/sdb1 > > flag@c13:~$ df -i > Filesystem Inodes IUsed IFree IUse% Mounted on > /dev/sda2 14958592 124409 14834183 1% / > none 114542 1 114541 1% /sys/fs/cgroup > udev 103895 1361 102534 2% /dev > tmpfs 114542 806 113736 1% /run > none 114542 3 114539 1% /run/lock > none 114542 1 114541 1% /run/shm > none 114542 1 114541 1% /run/user > /dev/sda1 124496 33 124463 1% /boot > /dev/sda5 524288 234880 289408 45% /mnt/sdb1 > > > any idea what else shall i tune to workaround this? or is it a know problem that > involves 32bit arch and xfs? I tried to reproduce this issue against the latest upstream tree on 32-bit system but no luck. Could you please supply the following info: 1) xfs_db -r "-c freesp -s" /dev/sda5 2) xfs_info /mnt/sdb1 Thanks, -Jeff _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 3.5+, xfs and 32bit armhf - xfs_buf_get: failed to map pages 2013-05-17 10:45 3.5+, xfs and 32bit armhf - xfs_buf_get: failed to map pages Paolo Pisati 2013-05-18 8:43 ` Jeff Liu @ 2013-05-19 1:13 ` Dave Chinner 2013-05-20 17:07 ` Paolo Pisati 1 sibling, 1 reply; 9+ messages in thread From: Dave Chinner @ 2013-05-19 1:13 UTC (permalink / raw) To: Paolo Pisati; +Cc: xfs On Fri, May 17, 2013 at 12:45:29PM +0200, Paolo Pisati wrote: > While exercising swift on a single node 32bit armhf system running a 3.5 kernel, > i got this when i hit ~25% of fs space usage: > > dmesg: > ... > [ 3037.399406] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. > [ 3037.399442] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. > [ 3037.399469] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. > [ 3037.399485] XFS (sda5): xfs_buf_get: failed to map pages > [ 3037.399485] > [ 3037.399501] XFS (sda5): Internal error xfs_trans_cancel at line 1466 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c. Caller 0xbf0235e0 > [ 3037.399501] > [ 3037.413789] [<c00164cc>] (unwind_backtrace+0x0/0x104) from [<c04ed624>] (dump_stack+0x20/0x24) > [ 3037.413985] [<c04ed624>] (dump_stack+0x20/0x24) from [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs]) > [ 3037.414321] [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs]) from [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs]) > [ 3037.414654] [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs]) from [<bf0235e0>] (xfs_create+0x228/0x558 [xfs]) > [ 3037.414953] [<bf0235e0>] (xfs_create+0x228/0x558 [xfs]) from [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs]) > [ 3037.415239] [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs]) from [<bf01a8d0>] (xfs_vn_mkdir+0x20/0x24 [xfs]) > [ 3037.415393] [<bf01a8d0>] (xfs_vn_mkdir+0x20/0x24 [xfs]) from [<c0135758>] (vfs_mkdir+0xc4/0x13c) > [ 3037.415410] [<c0135758>] (vfs_mkdir+0xc4/0x13c) from [<c013884c>] (sys_mkdirat+0xdc/0xe4) > [ 3037.415422] [<c013884c>] (sys_mkdirat+0xdc/0xe4) from [<c0138878>] (sys_mkdir+0x24/0x28) > [ 3037.415437] [<c0138878>] (sys_mkdir+0x24/0x28) from [<c000e320>] (ret_fast_syscall+0x0/0x30) > [ 3037.415452] XFS (sda5): xfs_do_force_shutdown(0x8) called from line 1467 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c. Return address = 0xbf06340c > [ 3037.416892] XFS (sda5): Corruption of in-memory data detected. Shutting down filesystem > [ 3037.425008] XFS (sda5): Please umount the filesystem and rectify the problem(s) > [ 3047.912480] XFS (sda5): xfs_log_force: error 5 returned. Hi Paolo, You've already contacted me off list about this and pointed me to this: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1176977 which contains information that everyone looking at the problem should know. Also, any progress on testing the backported fix mentioned in the bug? > and while i didn't hit the warning above, still after ~25% of > usage, the storage node died with: > > May 17 06:26:00 c13 container-server ERROR __call__ error with PUT /sdb1/123172/AUTH_test/3b3d078015304a41b76b0ab083b7863a_5 : [Errno 28] No space > left on device: '/srv/1/node/sdb1/containers/123172' (txn: tx8ea3ce392ee94df096b16-00519605b0) You're testing swift benchmark which is probably a small file workload with large attributes attached. It's a good chance that the workload is fragmenting free space because swift is doing bad things to allocation patterns. It's almost certainly exacerbated by the tiny filesystem you are using (1.5GB), but you can probably work around this problem for now with allocsize=4096. I've got a fix that I'm testing for the underlying cause of the problem I'm aware of with this workload, but I'll need more information about your storage/filesystem config to confirm it is the same root cause first. Can you include the info from here: http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F As well the freespace info that Jeff asked for? Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 3.5+, xfs and 32bit armhf - xfs_buf_get: failed to map pages 2013-05-19 1:13 ` Dave Chinner @ 2013-05-20 17:07 ` Paolo Pisati 2013-05-21 0:02 ` Dave Chinner 0 siblings, 1 reply; 9+ messages in thread From: Paolo Pisati @ 2013-05-20 17:07 UTC (permalink / raw) To: Dave Chinner; +Cc: xfs, Paolo Pisati On Sun, May 19, 2013 at 11:13:54AM +1000, Dave Chinner wrote: > > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1176977 > > which contains information that everyone looking at the problem > should know. Also, any progress on testing the backported fix > mentioned in the bug? the problem with the 'fix' is that it prevents xfs from erroring out, but swift-test fails regardless after ~25% of fs usage and i think having a bold 'xfs error' and a stack trace is more useful. > You're testing swift benchmark which is probably a small file > workload with large attributes attached. It's a good chance that > the workload is fragmenting free space because swift is doing bad > things to allocation patterns. It's almost certainly exacerbated by > the tiny filesystem you are using (1.5GB), but you can probably work > around this problem for now with allocsize=4096. ok, i repartitioned my disk but i can still reprodue it fairly easily: df -h: /dev/sda6 216G 573M 215G 1% /mnt/sdb1 df -i: /dev/sda6 56451072 235458 56215614 1% /mnt/sdb1 dmesg: ... [ 363.130877] XFS (sda6): Mounting Filesystem [ 363.146708] XFS (sda6): Ending clean mount [ 3055.520769] alloc_vmap_area: 18 callbacks suppressed [ 3055.520783] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. [ 3055.520817] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. [ 3055.520845] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. [ 3055.520861] XFS (sda6): xfs_buf_get: failed to map pages [ 3055.520861] [ 3055.520882] XFS (sda6): Internal error xfs_trans_cancel at line 1466 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c. Caller 0xbf0235e0 [ 3055.520882] [ 3055.535135] [<c00164cc>] (unwind_backtrace+0x0/0x104) from [<c04ed624>] (dump_stack+0x20/0x24) [ 3055.535345] [<c04ed624>] (dump_stack+0x20/0x24) from [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs]) [ 3055.535687] [<bf01091c>] (xfs_error_report+0x60/0x6c [xfs]) from [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs]) [ 3055.536023] [<bf0633f8>] (xfs_trans_cancel+0xfc/0x11c [xfs]) from [<bf0235e0>] (xfs_create+0x228/0x558 [xfs]) [ 3055.536314] [<bf0235e0>] (xfs_create+0x228/0x558 [xfs]) from [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs]) [ 3055.536589] [<bf01a7cc>] (xfs_vn_mknod+0x9c/0x180 [xfs]) from [<bf01a8f0>] (xfs_vn_create+0x1c/0x20 [xfs]) [ 3055.536741] [<bf01a8f0>] (xfs_vn_create+0x1c/0x20 [xfs]) from [<c01359d4>] (vfs_create+0xb4/0x120) [ 3055.536760] [<c01359d4>] (vfs_create+0xb4/0x120) from [<c0137c3c>] (do_last+0x860/0x9bc) [ 3055.536775] [<c0137c3c>] (do_last+0x860/0x9bc) from [<c0137fdc>] (path_openat+0xcc/0x428) [ 3055.536787] [<c0137fdc>] (path_openat+0xcc/0x428) from [<c0138458>] (do_filp_open+0x3c/0x90) [ 3055.536805] [<c0138458>] (do_filp_open+0x3c/0x90) from [<c0128248>] (do_sys_open+0xfc/0x1d0) [ 3055.536817] [<c0128248>] (do_sys_open+0xfc/0x1d0) from [<c0128348>] (sys_open+0x2c/0x30) [ 3055.536832] [<c0128348>] (sys_open+0x2c/0x30) from [<c000e320>] (ret_fast_syscall+0x0/0x30) [ 3055.536848] XFS (sda6): xfs_do_force_shutdown(0x8) called from line 1467 of file /build/buildd/linux-3.5.0/fs/xfs/xfs_trans.c. Return address = 0xbf06340c [ 3055.537327] XFS (sda6): Corruption of in-memory data detected. Shutting down filesystem [ 3055.545439] XFS (sda6): Please umount the filesystem and rectify the problem(s) [ 3070.301048] XFS (sda6): xfs_log_force: error 5 returned. [ 3100.381068] XFS (sda6): xfs_log_force: error 5 returned. [ 3130.461041] XFS (sda6): xfs_log_force: error 5 returned. [ 3160.541042] XFS (sda6): xfs_log_force: error 5 returned. [ 3190.621042] XFS (sda6): xfs_log_force: error 5 returned. [ 3220.701040] XFS (sda6): xfs_log_force: error 5 returned. [ 3250.781039] XFS (sda6): xfs_log_force: error 5 returned. [ 3280.861036] XFS (sda6): xfs_log_force: error 5 returned. [ 3310.941047] XFS (sda6): xfs_log_force: error 5 returned. [ 3341.021044] XFS (sda6): xfs_log_force: error 5 returned. [ 3371.101044] XFS (sda6): xfs_log_force: error 5 returned. [ 3401.181036] XFS (sda6): xfs_log_force: error 5 returned. [ 3431.261036] XFS (sda6): xfs_log_force: error 5 returned. [ 3461.341038] XFS (sda6): xfs_log_force: error 5 returned. [ 3491.421038] XFS (sda6): xfs_log_force: error 5 returned. [ 3521.501051] XFS (sda6): xfs_log_force: error 5 returned. [ 3551.581037] XFS (sda6): xfs_log_force: error 5 returned. [ 3581.661041] XFS (sda6): xfs_log_force: error 5 returned. > I've got a fix that I'm testing for the underlying cause of the > problem I'm aware of with this workload, but I'll need more > information about your storage/filesystem config to confirm it is > the same root cause first. Can you include the info from here: > > http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F flag@c13:~$ uname -a Linux c13 3.5.0-30-highbank #51-Ubuntu SMP Tue May 14 22:57:15 UTC 2013 armv7l armv7l armv7l GNU/Linux lag@c13:~$ xfs_repair -V xfs_repair version 3.1.7 armhf highbank node, 4 cores, 4GB mem flag@c13:~$ cat /proc/meminfo MemTotal: 4137004 kB MemFree: 2719752 kB Buffers: 39688 kB Cached: 580508 kB SwapCached: 0 kB Active: 631136 kB Inactive: 204552 kB Active(anon): 215520 kB Inactive(anon): 232 kB Active(file): 415616 kB Inactive(file): 204320 kB Unevictable: 0 kB Mlocked: 0 kB HighTotal: 3408896 kB HighFree: 2606516 kB LowTotal: 728108 kB LowFree: 113236 kB SwapTotal: 8378364 kB SwapFree: 8378364 kB Dirty: 4 kB Writeback: 0 kB AnonPages: 215516 kB Mapped: 8676 kB Shmem: 264 kB Slab: 317000 kB SReclaimable: 230392 kB SUnreclaim: 86608 kB KernelStack: 2192 kB PageTables: 2284 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 10446864 kB Committed_AS: 1049624 kB VmallocTotal: 245760 kB VmallocUsed: 2360 kB VmallocChunk: 241428 kB flag@c13:~$ cat /proc/mounts rootfs / rootfs rw 0 0 sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,relatime,size=2059248k,nr_inodes=177400,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,relatime,size=827404k,mode=755 0 0 /dev/disk/by-uuid/6594b183-3198-4dec-a97d-a3f834b98011 / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0 none /sys/fs/fuse/connections fusectl rw,relatime 0 0 none /sys/kernel/debug debugfs rw,relatime 0 0 none /sys/kernel/security securityfs rw,relatime 0 0 none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0 none /run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0 /dev/sda1 /boot ext2 rw,relatime,errors=continue 0 0 /dev/sda6 /mnt/sdb1 xfs rw,noatime,nodiratime,attr2,nobarrier,logbufs=8,noquota 0 0 flag@c13:~$ cat /proc/partitions major minor #blocks name 8 0 250059096 sda 8 1 248832 sda1 8 2 15625000 sda2 8 3 1 sda3 8 5 8378369 sda5 8 6 225804288 sda6 no RAID, no LVM common sata disk write cache on (unknown size) no BBWC AFAIK flag@c13:~$ xfs_info /mnt/sdb1/ meta-data=/dev/sda6 isize=1024 agcount=4, agsize=14112768 blks = sectsz=512 attr=2 data = bsize=4096 blocks=56451072, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal bsize=4096 blocks=27564, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 > > As well the freespace info that Jeff asked for? flag@c13:~$ sudo xfs_db -r "-c freesp -s" /dev/sda6 from to extents blocks pct 1 1 423 423 0.00 2 3 897 2615 0.01 4 7 136 915 0.00 8 15 24833 365797 0.86 8388608 14112768 3 41928421 99.13 total free extents 26292 total free blocks 42298171 average free extent size 1608.78 -- bye, p. _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 3.5+, xfs and 32bit armhf - xfs_buf_get: failed to map pages 2013-05-20 17:07 ` Paolo Pisati @ 2013-05-21 0:02 ` Dave Chinner 2013-05-23 14:34 ` Paolo Pisati 0 siblings, 1 reply; 9+ messages in thread From: Dave Chinner @ 2013-05-21 0:02 UTC (permalink / raw) To: Paolo Pisati; +Cc: xfs On Mon, May 20, 2013 at 07:07:10PM +0200, Paolo Pisati wrote: > On Sun, May 19, 2013 at 11:13:54AM +1000, Dave Chinner wrote: > > > > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1176977 > > > > which contains information that everyone looking at the problem > > should know. Also, any progress on testing the backported fix > > mentioned in the bug? > > the problem with the 'fix' is that it prevents xfs from erroring out, but > swift-test fails regardless after ~25% of fs usage and i think having a bold > 'xfs error' and a stack trace is more useful. I think your logic is misguided. There's a major difference between ENOSPC and a filesystem shutdown. After a shutdown you need to unmount, remount, and then work out what didn't make it to disk before you can restart. Not to mention that the ENOMEM that triggers the shutdown is highly system dependent - it will occur at different times on different machines and will be highly unpredictable. That's not a good thing. Compare that to a plain ENOSPC error: you can just remove files and keep going. > > You're testing swift benchmark which is probably a small file > > workload with large attributes attached. It's a good chance that > > the workload is fragmenting free space because swift is doing bad > > things to allocation patterns. It's almost certainly exacerbated by > > the tiny filesystem you are using (1.5GB), but you can probably work > > around this problem for now with allocsize=4096. > > ok, i repartitioned my disk but i can still reprodue it fairly easily: > > df -h: > /dev/sda6 216G 573M 215G 1% /mnt/sdb1 > > df -i: > /dev/sda6 56451072 235458 56215614 1% /mnt/sdb1 > > dmesg: > ... > [ 363.130877] XFS (sda6): Mounting Filesystem > [ 363.146708] XFS (sda6): Ending clean mount > [ 3055.520769] alloc_vmap_area: 18 callbacks suppressed > [ 3055.520783] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. > [ 3055.520817] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. > [ 3055.520845] vmap allocation for size 2097152 failed: use vmalloc=<size> to increase size. > [ 3055.520861] XFS (sda6): xfs_buf_get: failed to map pages Which is your ENOMEM error, not an ENOSPC error. So the larger filesystem meant you didn't hit the ENOSPC problem like I suspected it would.... > > I've got a fix that I'm testing for the underlying cause of the > > problem I'm aware of with this workload, but I'll need more > > information about your storage/filesystem config to confirm it is > > the same root cause first. Can you include the info from here: And that fix I mentioned will be useless if you don't apply the patch that avoids the vmap allocation problem.... > > http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F > > flag@c13:~$ uname -a > Linux c13 3.5.0-30-highbank #51-Ubuntu SMP Tue May 14 22:57:15 UTC 2013 armv7l armv7l armv7l GNU/Linux > > lag@c13:~$ xfs_repair -V > xfs_repair version 3.1.7 > > armhf highbank node, 4 cores, 4GB mem > > flag@c13:~$ cat /proc/meminfo > MemTotal: 4137004 kB > MemFree: 2719752 kB > Buffers: 39688 kB > Cached: 580508 kB > SwapCached: 0 kB > Active: 631136 kB > Inactive: 204552 kB > Active(anon): 215520 kB > Inactive(anon): 232 kB > Active(file): 415616 kB > Inactive(file): 204320 kB > Unevictable: 0 kB > Mlocked: 0 kB > HighTotal: 3408896 kB > HighFree: 2606516 kB > LowTotal: 728108 kB Oh, there's a likely cause of the vmalloc issue. You have 3.4GB of high memory, which means the kernel only has 700MB of low memory for slab caches, vmap regions, etc. An ia32 box has, by default 960MB of low memory which will be why you are seeing this more frequently than anyone using an ia32 machine. And an ia32 machine can be configured with 2G/2G or 3G/1G kernel/user address space splits, so most vmalloc problems can be worked around. > Slab: 317000 kB > SReclaimable: 230392 kB > SUnreclaim: 86608 kB And so you have 300MB in slab caches in low memory > KernelStack: 2192 kB > PageTables: 2284 kB > NFS_Unstable: 0 kB > Bounce: 0 kB > WritebackTmp: 0 kB > CommitLimit: 10446864 kB > Committed_AS: 1049624 kB > VmallocTotal: 245760 kB > VmallocUsed: 2360 kB > VmallocChunk: 241428 kB and 240MB in vmalloc space. so there's not much left of that 700MB of low memory space. So, you really need that vmap fix, and you need to configure your kernel with more low memory space. > > As well the freespace info that Jeff asked for? > > flag@c13:~$ sudo xfs_db -r "-c freesp -s" /dev/sda6 > from to extents blocks pct > 1 1 423 423 0.00 > 2 3 897 2615 0.01 > 4 7 136 915 0.00 > 8 15 24833 365797 0.86 > 8388608 14112768 3 41928421 99.13 > total free extents 26292 > total free blocks 42298171 > average free extent size 1608.78 We need this information after the ENOSPC error occurs, not soon after mkfs or after the ENOMEM error. If this is after ENOSPC, please unmount the filesystem, drop caches and rerun the freesp command... Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 3.5+, xfs and 32bit armhf - xfs_buf_get: failed to map pages 2013-05-21 0:02 ` Dave Chinner @ 2013-05-23 14:34 ` Paolo Pisati 2013-05-29 13:56 ` Paolo Pisati 2013-05-30 0:38 ` Dave Chinner 0 siblings, 2 replies; 9+ messages in thread From: Paolo Pisati @ 2013-05-23 14:34 UTC (permalink / raw) To: Dave Chinner; +Cc: xfs, Paolo Pisati On Tue, May 21, 2013 at 10:02:09AM +1000, Dave Chinner wrote: > > And that fix I mentioned will be useless if you don't apply the > patch that avoids the vmap allocation problem.... ok, so i recompiled a kernel+aforementioend fix, i repartitioned my disk and i ran the swift-bench for 2 days in a row until i got this: dmesg: ... [163596.605253] updatedb.mlocat: page allocation failure: order:0, mode:0x20 [163596.605299] [<c00164cc>] (unwind_backtrace+0x0/0x104) from [<c04edb20>] (dump_stack+0x20/0x24) [163596.605320] [<c04edb20>] (dump_stack+0x20/0x24) from [<c00e7780>] (warn_alloc_failed+0xd8/0x118) [163596.605335] [<c00e7780>] (warn_alloc_failed+0xd8/0x118) from [<c00e9b88>] (__alloc_pages_nodemask+0x524/0x708) [163596.605354] [<c00e9b88>] (__alloc_pages_nodemask+0x524/0x708) from [<c011b798>] (new_slab+0x22c/0x248) [163596.605370] [<c011b798>] (new_slab+0x22c/0x248) from [<c04f04f8>] (__slab_alloc.constprop.46+0x1a4/0x4c8) [163596.605383] [<c04f04f8>] (__slab_alloc.constprop.46+0x1a4/0x4c8) from [<c011ced4>] (kmem_cache_alloc+0x158/0x190) [163596.605402] [<c011ced4>] (kmem_cache_alloc+0x158/0x190) from [<c0332be0>] (scsi_pool_alloc_command+0x30/0x74) [163596.605417] [<c0332be0>] (scsi_pool_alloc_command+0x30/0x74) from [<c0332c80>] (scsi_host_alloc_command+0x24/0x78) [163596.605428] [<c0332c80>] (scsi_host_alloc_command+0x24/0x78) from [<c0332cf0>] (__scsi_get_command+0x1c/0xa0) [163596.605439] [<c0332cf0>] (__scsi_get_command+0x1c/0xa0) from [<c0332db0>] (scsi_get_command+0x3c/0xb0) [163596.605453] [<c0332db0>] (scsi_get_command+0x3c/0xb0) from [<c0338d44>] (scsi_get_cmd_from_req+0x50/0x60) [163596.605466] [<c0338d44>] (scsi_get_cmd_from_req+0x50/0x60) from [<c0339fd8>] (scsi_setup_fs_cmnd+0x4c/0xac) [163596.605482] [<c0339fd8>] (scsi_setup_fs_cmnd+0x4c/0xac) from [<c0343568>] (sd_prep_fn+0x114/0xaf4) [163596.605501] [<c0343568>] (sd_prep_fn+0x114/0xaf4) from [<c0299af4>] (blk_peek_request+0xc8/0x214) [163596.605514] [<c0299af4>] (blk_peek_request+0xc8/0x214) from [<c033a1b0>] (scsi_request_fn+0x40/0x504) [163596.605524] [<c033a1b0>] (scsi_request_fn+0x40/0x504) from [<c029a38c>] (blk_queue_bio+0x300/0x384) [163596.605536] [<c029a38c>] (blk_queue_bio+0x300/0x384) from [<c0298450>] (generic_make_request+0xb8/0xd8) [163596.605548] [<c0298450>] (generic_make_request+0xb8/0xd8) from [<c0298534>] (submit_bio+0xc4/0x17c) [163596.605756] [<c0298534>] (submit_bio+0xc4/0x17c) from [<bf00f1c4>] (_xfs_buf_ioapply+0x1bc/0x224 [xfs]) [163596.606002] [<bf00f1c4>] (_xfs_buf_ioapply+0x1bc/0x224 [xfs]) from [<bf00f314>] (xfs_buf_iorequest+0x4c/0x98 [xfs]) [163596.606241] [<bf00f314>] (xfs_buf_iorequest+0x4c/0x98 [xfs]) from [<bf00f868>] (_xfs_buf_read+0x34/0x50 [xfs]) [163596.606481] [<bf00f868>] (_xfs_buf_read+0x34/0x50 [xfs]) from [<bf00f964>] (xfs_buf_read+0xe0/0x108 [xfs]) [163596.606781] [<bf00f964>] (xfs_buf_read+0xe0/0x108 [xfs]) from [<bf06ba78>] (xfs_trans_read_buf+0x1e4/0x3e8 [xfs]) [163596.607115] [<bf06ba78>] (xfs_trans_read_buf+0x1e4/0x3e8 [xfs]) from [<bf053a9c>] (xfs_imap_to_bp+0x54/0x128 [xfs]) [163596.607432] [<bf053a9c>] (xfs_imap_to_bp+0x54/0x128 [xfs]) from [<bf057bc4>] (xfs_iread+0x6c/0x150 [xfs]) [163596.607719] [<bf057bc4>] (xfs_iread+0x6c/0x150 [xfs]) from [<bf015bfc>] (xfs_iget+0x210/0x72c [xfs]) [163596.607982] [<bf015bfc>] (xfs_iget+0x210/0x72c [xfs]) from [<bf0233b4>] (xfs_lookup+0xf4/0x114 [xfs]) [163596.608247] [<bf0233b4>] (xfs_lookup+0xf4/0x114 [xfs]) from [<bf01a5e8>] (xfs_vn_lookup+0x54/0x98 [xfs]) [163596.608387] [<bf01a5e8>] (xfs_vn_lookup+0x54/0x98 [xfs]) from [<c0134198>] (__lookup_hash+0x64/0xec) [163596.608402] [<c0134198>] (__lookup_hash+0x64/0xec) from [<c04f0d68>] (lookup_slow+0x50/0xac) [163596.608415] [<c04f0d68>] (lookup_slow+0x50/0xac) from [<c0136724>] (path_lookupat+0x730/0x794) [163596.608428] [<c0136724>] (path_lookupat+0x730/0x794) from [<c01367b4>] (do_path_lookup+0x2c/0xd0) [163596.608439] [<c01367b4>] (do_path_lookup+0x2c/0xd0) from [<c01385e0>] (user_path_at_empty+0x64/0x8c) [163596.608451] [<c01385e0>] (user_path_at_empty+0x64/0x8c) from [<c013862c>] (user_path_at+0x24/0x2c) [163596.608462] [<c013862c>] (user_path_at+0x24/0x2c) from [<c012dd3c>] (vfs_fstatat+0x40/0x78) [163596.608473] [<c012dd3c>] (vfs_fstatat+0x40/0x78) from [<c012dd9c>] (vfs_lstat+0x28/0x2c) [163596.608482] [<c012dd9c>] (vfs_lstat+0x28/0x2c) from [<c012e048>] (sys_lstat64+0x24/0x40) [163596.608495] [<c012e048>] (sys_lstat64+0x24/0x40) from [<c000e320>] (ret_fast_syscall+0x0/0x30) [163596.608503] Mem-info: [163596.608509] Normal per-cpu: [163596.608515] CPU 0: hi: 186, btch: 31 usd: 38 [163596.608521] CPU 1: hi: 186, btch: 31 usd: 218 [163596.608528] CPU 2: hi: 186, btch: 31 usd: 152 [163596.608533] CPU 3: hi: 186, btch: 31 usd: 171 [163596.608538] HighMem per-cpu: [163596.608544] CPU 0: hi: 186, btch: 31 usd: 46 [163596.608549] CPU 1: hi: 186, btch: 31 usd: 171 [163596.608555] CPU 2: hi: 186, btch: 31 usd: 168 [163596.608561] CPU 3: hi: 186, btch: 31 usd: 177 [163596.608574] active_anon:26367 inactive_anon:29153 isolated_anon:0 [163596.608574] active_file:396338 inactive_file:397959 isolated_file:0 [163596.608574] unevictable:0 dirty:0 writeback:5 unstable:0 [163596.608574] free:5145 slab_reclaimable:57625 slab_unreclaimable:7729 [163596.608574] mapped:1703 shmem:10 pagetables:581 bounce:0 [163596.608602] Normal free:15256kB min:3508kB low:4384kB high:5260kB active_anon:0kB inactive_anon:8kB active_file:848kB inactive_file:1560kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:772160kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:230500kB slab_unreclaimable:30916kB kernel_stack:2208kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [163596.608607] lowmem_reserve[]: 0 26423 26423 [163596.608628] HighMem free:5324kB min:512kB low:4352kB high:8192kB active_anon:105468kB inactive_anon:116604kB active_file:1584504kB inactive_file:1590276kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3382264kB mlocked:0kB dirty:0kB writeback:20kB mapped:6812kB shmem:40kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:2324kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [163596.608634] lowmem_reserve[]: 0 0 0 [163596.608643] Normal: 216*4kB 215*8kB 216*16kB 216*32kB 36*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 15256kB [163596.608668] HighMem: 233*4kB 67*8kB 141*16kB 22*32kB 8*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5324kB [163596.608692] 794329 total pagecache pages [163596.608697] 12 pages in swap cache [163596.608703] Swap cache stats: add 79, delete 67, find 9/11 [163596.608708] Free swap = 8378092kB [163596.608712] Total swap = 8378364kB [163596.670667] 1046784 pages of RAM [163596.670674] 6801 free pages [163596.670679] 12533 reserved pages [163596.670683] 36489 slab pages [163596.670687] 631668 pages shared [163596.670692] 12 pages swap cached [163596.670701] SLUB: Unable to allocate memory on node -1 (gfp=0x8020) [163596.670710] cache: kmalloc-192, object size: 192, buffer size: 192, default order: 0, min order: 0 [163596.670718] node 0: slabs: 2733, objs: 57393, free: 0 df -h: ... /dev/sda6 216G 53G 163G 25% /mnt/sdb1 df -i: ... /dev/sda6 56451072 19721920 36729152 35% /mnt/sdb1 flag@c13:~$ cat /proc/meminfo MemTotal: 4137004 kB MemFree: 1191096 kB Buffers: 23172 kB Cached: 2074116 kB SwapCached: 48 kB Active: 1301568 kB Inactive: 1016024 kB Active(anon): 103748 kB Inactive(anon): 116600 kB Active(file): 1197820 kB Inactive(file): 899424 kB Unevictable: 0 kB Mlocked: 0 kB HighTotal: 3408896 kB HighFree: 1108444 kB LowTotal: 728108 kB LowFree: 82652 kB SwapTotal: 8378364 kB SwapFree: 8378092 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 220284 kB Mapped: 7068 kB Shmem: 44 kB Slab: 263216 kB SReclaimable: 232212 kB SUnreclaim: 31004 kB KernelStack: 2192 kB PageTables: 2312 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 10446864 kB Committed_AS: 1051868 kB VmallocTotal: 245760 kB VmallocUsed: 2360 kB VmallocChunk: 241428 kB flag@c13:~$ sudo xfs_db -r "-c freesp -s" /dev/sda6 from to extents blocks pct 1 1 27058 27058 0.06 2 3 124367 358831 0.84 4 7 17656 121693 0.29 8 15 2856900 42122381 98.81 total free extents 3025981 total free blocks 42629963 average free extent size 14.088 -- bye, p. _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 3.5+, xfs and 32bit armhf - xfs_buf_get: failed to map pages 2013-05-23 14:34 ` Paolo Pisati @ 2013-05-29 13:56 ` Paolo Pisati 2013-05-30 0:42 ` Dave Chinner 2013-05-30 0:38 ` Dave Chinner 1 sibling, 1 reply; 9+ messages in thread From: Paolo Pisati @ 2013-05-29 13:56 UTC (permalink / raw) To: Dave Chinner; +Cc: xfs, Paolo Pisati On Thu, May 23, 2013 at 04:34:56PM +0200, Paolo Pisati wrote: > On Tue, May 21, 2013 at 10:02:09AM +1000, Dave Chinner wrote: > > > > And that fix I mentioned will be useless if you don't apply the > > patch that avoids the vmap allocation problem.... > > > ok, so i recompiled a kernel+aforementioend fix, i repartitioned my disk and i > ran the swift-bench for 2 days in a row until i got this: i'm testing a 3.5.y kernel plus those 3 patches: 549142a xfs: don't use speculative prealloc for small files f0843f4 xfs: limit speculative prealloc size on sparse files 454da09 xfs: inode allocation should use unmapped buffers. and i can confirm that: -using a small fs (2G) i cannot reproduce any -ENOSPC or vmalloc() problem anymore, the benchmark runs until running out of inodes -using a bigger fs (~250G), two days and my tests are still running good -- bye, p. _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 3.5+, xfs and 32bit armhf - xfs_buf_get: failed to map pages 2013-05-29 13:56 ` Paolo Pisati @ 2013-05-30 0:42 ` Dave Chinner 0 siblings, 0 replies; 9+ messages in thread From: Dave Chinner @ 2013-05-30 0:42 UTC (permalink / raw) To: Paolo Pisati; +Cc: xfs On Wed, May 29, 2013 at 03:56:41PM +0200, Paolo Pisati wrote: > On Thu, May 23, 2013 at 04:34:56PM +0200, Paolo Pisati wrote: > > On Tue, May 21, 2013 at 10:02:09AM +1000, Dave Chinner wrote: > > > > > > And that fix I mentioned will be useless if you don't apply the > > > patch that avoids the vmap allocation problem.... > > > > > > ok, so i recompiled a kernel+aforementioend fix, i repartitioned my disk and i > > ran the swift-bench for 2 days in a row until i got this: > > i'm testing a 3.5.y kernel plus those 3 patches: > > 549142a xfs: don't use speculative prealloc for small files > f0843f4 xfs: limit speculative prealloc size on sparse files > 454da09 xfs: inode allocation should use unmapped buffers. > > and i can confirm that: > > -using a small fs (2G) i cannot reproduce any -ENOSPC or vmalloc() problem > anymore, the benchmark runs until running out of inodes > > -using a bigger fs (~250G), two days and my tests are still running good Ok, good to know. The first patch you list there hasn't even been reviewed yet, so it might take some time before that is ready for -stable backport. Also, there are a bunch of fixes needed to the second patch you have there (f0843f4 xfs: limit speculative prealloc...) that would also be necessary for a -stable backport. i.e: e8108ce xfs: fix xfs_iomap_eof_prealloc_initial_size type e114b5f xfs: increase prealloc size to double that of the previous extent e78c420 xfs: fix potential infinite loop in xfs_iomap_prealloc_size() Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 3.5+, xfs and 32bit armhf - xfs_buf_get: failed to map pages 2013-05-23 14:34 ` Paolo Pisati 2013-05-29 13:56 ` Paolo Pisati @ 2013-05-30 0:38 ` Dave Chinner 1 sibling, 0 replies; 9+ messages in thread From: Dave Chinner @ 2013-05-30 0:38 UTC (permalink / raw) To: Paolo Pisati; +Cc: xfs On Thu, May 23, 2013 at 04:34:56PM +0200, Paolo Pisati wrote: > On Tue, May 21, 2013 at 10:02:09AM +1000, Dave Chinner wrote: > > > > And that fix I mentioned will be useless if you don't apply the > > patch that avoids the vmap allocation problem.... > > > ok, so i recompiled a kernel+aforementioend fix, i repartitioned my disk and i > ran the swift-bench for 2 days in a row until i got this: > > dmesg: > ... > [163596.605253] updatedb.mlocat: page allocation failure: order:0, mode:0x20 > [163596.605299] [<c00164cc>] (unwind_backtrace+0x0/0x104) from [<c04edb20>] (dump_stack+0x20/0x24) > [163596.605320] [<c04edb20>] (dump_stack+0x20/0x24) from [<c00e7780>] (warn_alloc_failed+0xd8/0x118) > [163596.605335] [<c00e7780>] (warn_alloc_failed+0xd8/0x118) from [<c00e9b88>] (__alloc_pages_nodemask+0x524/0x708) > [163596.605354] [<c00e9b88>] (__alloc_pages_nodemask+0x524/0x708) from [<c011b798>] (new_slab+0x22c/0x248) > [163596.605370] [<c011b798>] (new_slab+0x22c/0x248) from [<c04f04f8>] (__slab_alloc.constprop.46+0x1a4/0x4c8) > [163596.605383] [<c04f04f8>] (__slab_alloc.constprop.46+0x1a4/0x4c8) from [<c011ced4>] (kmem_cache_alloc+0x158/0x190) > [163596.605402] [<c011ced4>] (kmem_cache_alloc+0x158/0x190) from [<c0332be0>] (scsi_pool_alloc_command+0x30/0x74) > [163596.605417] [<c0332be0>] (scsi_pool_alloc_command+0x30/0x74) from [<c0332c80>] (scsi_host_alloc_command+0x24/0x78) > [163596.605428] [<c0332c80>] (scsi_host_alloc_command+0x24/0x78) from [<c0332cf0>] (__scsi_get_command+0x1c/0xa0) > [163596.605439] [<c0332cf0>] (__scsi_get_command+0x1c/0xa0) from [<c0332db0>] (scsi_get_command+0x3c/0xb0) > [163596.605453] [<c0332db0>] (scsi_get_command+0x3c/0xb0) from [<c0338d44>] (scsi_get_cmd_from_req+0x50/0x60) > [163596.605466] [<c0338d44>] (scsi_get_cmd_from_req+0x50/0x60) from [<c0339fd8>] (scsi_setup_fs_cmnd+0x4c/0xac) ENOMEM deep in the SCSI stack for an order 0 GFP_ATOMIC allocation. That's not an XFS problem - that's a SCSI stack issue. You should probably report that to the scsi list... > [163596.608574] active_anon:26367 inactive_anon:29153 isolated_anon:0 > [163596.608574] active_file:396338 inactive_file:397959 isolated_file:0 > [163596.608574] unevictable:0 dirty:0 writeback:5 unstable:0 > [163596.608574] free:5145 slab_reclaimable:57625 slab_unreclaimable:7729 > [163596.608574] mapped:1703 shmem:10 pagetables:581 bounce:0 > [163596.608602] Normal free:15256kB min:3508kB low:4384kB high:5260kB active_anon:0kB inactive_anon:8kB active_file:848kB inactive_file:1560kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:772160kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:230500kB slab_unreclaimable:30916kB kernel_stack:2208kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > [163596.608607] lowmem_reserve[]: 0 26423 26423 > [163596.608628] HighMem free:5324kB min:512kB low:4352kB high:8192kB active_anon:105468kB inactive_anon:116604kB active_file:1584504kB inactive_file:1590276kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3382264kB mlocked:0kB dirty:0kB writeback:20kB mapped:6812kB shmem:40kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:2324kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > [163596.608634] lowmem_reserve[]: 0 0 0 > [163596.608643] Normal: 216*4kB 215*8kB 216*16kB 216*32kB 36*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 15256kB > [163596.608668] HighMem: 233*4kB 67*8kB 141*16kB 22*32kB 8*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5324kB Though this says there is plenty of free order 0 pages in both low and high memory.... > [163596.608692] 794329 total pagecache pages > [163596.608697] 12 pages in swap cache > [163596.608703] Swap cache stats: add 79, delete 67, find 9/11 > [163596.608708] Free swap = 8378092kB > [163596.608712] Total swap = 8378364kB > [163596.670667] 1046784 pages of RAM > [163596.670674] 6801 free pages > [163596.670679] 12533 reserved pages > [163596.670683] 36489 slab pages > [163596.670687] 631668 pages shared > [163596.670692] 12 pages swap cached > [163596.670701] SLUB: Unable to allocate memory on node -1 (gfp=0x8020) > [163596.670710] cache: kmalloc-192, object size: 192, buffer size: 192, default order: 0, min order: 0 > [163596.670718] node 0: slabs: 2733, objs: 57393, free: 0 And it was slub that was unable to find a page when it should have been able to, so perhaps this is a VM problem? Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2013-05-30 0:42 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-05-17 10:45 3.5+, xfs and 32bit armhf - xfs_buf_get: failed to map pages Paolo Pisati 2013-05-18 8:43 ` Jeff Liu 2013-05-19 1:13 ` Dave Chinner 2013-05-20 17:07 ` Paolo Pisati 2013-05-21 0:02 ` Dave Chinner 2013-05-23 14:34 ` Paolo Pisati 2013-05-29 13:56 ` Paolo Pisati 2013-05-30 0:42 ` Dave Chinner 2013-05-30 0:38 ` Dave Chinner
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox