* ENOSPC but df and df -i show free space
@ 2011-06-19 21:50 Andy Isaacson
2011-06-19 22:18 ` Dave Chinner
0 siblings, 1 reply; 5+ messages in thread
From: Andy Isaacson @ 2011-06-19 21:50 UTC (permalink / raw)
To: xfs
% touch /d1/tmp/foo
touch: cannot touch `/d1/tmp/foo': No space left on device
% df /d1
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/vg0-d1 943616000 904690332 38925668 96% /d1
% df -i /d1
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/vg0-d1 167509008 11806336 155702672 8% /d1
% sudo xfs_growfs -n /d1
meta-data=/dev/mapper/vg0-d1 isize=256 agcount=18, agsize=13107200 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=235929600, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=25600, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
% grep d1 /proc/mounts
/dev/mapper/vg0-d1 /d1 xfs rw,relatime,attr2,noquota 0 0
Obviously I'm missing something, but what?
Nothing relevant in dmesg that I can see. The filesystem started out at
200 GB and has been xfs_growfs'd in 100GB increments up to its current
size of 900 GB.
x86_64 2.6.38-rc3-00019-gafe8a88 on a quad-core i7 with 12GB RAM, on DM
on AHCI SATA.
-andy
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: ENOSPC but df and df -i show free space 2011-06-19 21:50 ENOSPC but df and df -i show free space Andy Isaacson @ 2011-06-19 22:18 ` Dave Chinner 2011-06-19 22:58 ` Andy Isaacson 0 siblings, 1 reply; 5+ messages in thread From: Dave Chinner @ 2011-06-19 22:18 UTC (permalink / raw) To: Andy Isaacson; +Cc: xfs On Sun, Jun 19, 2011 at 02:50:39PM -0700, Andy Isaacson wrote: > % touch /d1/tmp/foo > touch: cannot touch `/d1/tmp/foo': No space left on device > % df /d1 > Filesystem 1K-blocks Used Available Use% Mounted on > /dev/mapper/vg0-d1 943616000 904690332 38925668 96% /d1 Problems like this will occur if you run your filesystem at > 85-90% full for extented periods.... > % df -i /d1 > Filesystem Inodes IUsed IFree IUse% Mounted on > /dev/mapper/vg0-d1 167509008 11806336 155702672 8% /d1 > % sudo xfs_growfs -n /d1 > meta-data=/dev/mapper/vg0-d1 isize=256 agcount=18, agsize=13107200 blks > = sectsz=512 attr=2 > data = bsize=4096 blocks=235929600, imaxpct=25 > = sunit=0 swidth=0 blks > naming =version 2 bsize=4096 ascii-ci=0 > log =internal bsize=4096 blocks=25600, version=2 > = sectsz=512 sunit=0 blks, lazy-count=1 > realtime =none extsz=4096 blocks=0, rtextents=0 > % grep d1 /proc/mounts > /dev/mapper/vg0-d1 /d1 xfs rw,relatime,attr2,noquota 0 0 > > Obviously I'm missing something, but what? Most likely is that you have no contiguous free space large enough to create a new inode chunk. using xfs_db to dump the freespace size histogram will tell you if this is the case or not. > Nothing relevant in dmesg that I can see. The filesystem started out at > 200 GB and has been xfs_growfs'd in 100GB increments up to its current > size of 900 GB. Ugh. Sounds like you've been running the filesystem near full for it's entire life.... Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: ENOSPC but df and df -i show free space 2011-06-19 22:18 ` Dave Chinner @ 2011-06-19 22:58 ` Andy Isaacson 2011-06-19 23:23 ` Dave Chinner 0 siblings, 1 reply; 5+ messages in thread From: Andy Isaacson @ 2011-06-19 22:58 UTC (permalink / raw) To: Dave Chinner; +Cc: xfs On Mon, Jun 20, 2011 at 08:18:52AM +1000, Dave Chinner wrote: > > % touch /d1/tmp/foo > > touch: cannot touch `/d1/tmp/foo': No space left on device > > % df /d1 > > Filesystem 1K-blocks Used Available Use% Mounted on > > /dev/mapper/vg0-d1 943616000 904690332 38925668 96% /d1 > > Problems like this will occur if you run your filesystem at > 85-90% > full for extented periods.... Ah, yes, that's definitely been the case. I grow the filesystem when it hits 95% utilization or thereabouts. Hadn't realized that's such an awful use case for xfs. > > % df -i /d1 > > Filesystem Inodes IUsed IFree IUse% Mounted on > > /dev/mapper/vg0-d1 167509008 11806336 155702672 8% /d1 > > % sudo xfs_growfs -n /d1 > > meta-data=/dev/mapper/vg0-d1 isize=256 agcount=18, agsize=13107200 blks > > = sectsz=512 attr=2 > > data = bsize=4096 blocks=235929600, imaxpct=25 > > = sunit=0 swidth=0 blks > > naming =version 2 bsize=4096 ascii-ci=0 > > log =internal bsize=4096 blocks=25600, version=2 > > = sectsz=512 sunit=0 blks, lazy-count=1 > > realtime =none extsz=4096 blocks=0, rtextents=0 > > % grep d1 /proc/mounts > > /dev/mapper/vg0-d1 /d1 xfs rw,relatime,attr2,noquota 0 0 > > > > Obviously I'm missing something, but what? > > Most likely is that you have no contiguous free space large enough > to create a new inode chunk. using xfs_db to dump the freespace > size histogram will tell you if this is the case or not. % sudo xfs_db -c freesp /dev/vg0/d1 from to extents blocks pct 1 1 168504 168504 1.71 2 3 446 1135 0.01 4 7 5550 37145 0.38 8 15 49159 524342 5.33 16 31 1383 29223 0.30 2097152 4194303 1 2931455 29.78 4194304 8388607 1 6150953 62.49 I don't really grok that output. -andy _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: ENOSPC but df and df -i show free space 2011-06-19 22:58 ` Andy Isaacson @ 2011-06-19 23:23 ` Dave Chinner 2011-06-19 23:55 ` Andy Isaacson 0 siblings, 1 reply; 5+ messages in thread From: Dave Chinner @ 2011-06-19 23:23 UTC (permalink / raw) To: Andy Isaacson; +Cc: xfs On Sun, Jun 19, 2011 at 03:58:34PM -0700, Andy Isaacson wrote: > On Mon, Jun 20, 2011 at 08:18:52AM +1000, Dave Chinner wrote: > > > % touch /d1/tmp/foo > > > touch: cannot touch `/d1/tmp/foo': No space left on device > > > % df /d1 > > > Filesystem 1K-blocks Used Available Use% Mounted on > > > /dev/mapper/vg0-d1 943616000 904690332 38925668 96% /d1 > > > > Problems like this will occur if you run your filesystem at > 85-90% > > full for extented periods.... > > Ah, yes, that's definitely been the case. I grow the filesystem when it > hits 95% utilization or thereabouts. Hadn't realized that's such an > awful use case for xfs. No allocation algorithm is perfect in all circumstances. The alogrithms in XFS tend to degrade when large contiguous freespace regions are not available, resulting in more fragmentation of data extents and subsequent freespace fragmentation when those files are removed or defragmented. The algorithms will recover if you free up enough space that large contiguous freespace extents re-form, but that can require removing a large amount of data.... > > > % df -i /d1 > > > Filesystem Inodes IUsed IFree IUse% Mounted on > > > /dev/mapper/vg0-d1 167509008 11806336 155702672 8% /d1 > > > % sudo xfs_growfs -n /d1 > > > meta-data=/dev/mapper/vg0-d1 isize=256 agcount=18, agsize=13107200 blks > > > = sectsz=512 attr=2 > > > data = bsize=4096 blocks=235929600, imaxpct=25 > > > = sunit=0 swidth=0 blks > > > naming =version 2 bsize=4096 ascii-ci=0 > > > log =internal bsize=4096 blocks=25600, version=2 > > > = sectsz=512 sunit=0 blks, lazy-count=1 > > > realtime =none extsz=4096 blocks=0, rtextents=0 > > > % grep d1 /proc/mounts > > > /dev/mapper/vg0-d1 /d1 xfs rw,relatime,attr2,noquota 0 0 > > > > > > Obviously I'm missing something, but what? > > > > Most likely is that you have no contiguous free space large enough > > to create a new inode chunk. using xfs_db to dump the freespace > > size histogram will tell you if this is the case or not. > > % sudo xfs_db -c freesp /dev/vg0/d1 > from to extents blocks pct > 1 1 168504 168504 1.71 > 2 3 446 1135 0.01 > 4 7 5550 37145 0.38 > 8 15 49159 524342 5.33 > 16 31 1383 29223 0.30 > 2097152 4194303 1 2931455 29.78 > 4194304 8388607 1 6150953 62.49 > > I don't really grok that output. It's the historgram of free space extent sizes. You have 168504 single free block regions (4k in size) in the filesystem, 446 between 8k and 12k (2-3 blocks), etc. Inode allocation requires aligned 16k allocations (64x256 byte inodes), so you need free extents in the 4-7 block range or larger, which you appear to have so it should not be failing. Did you dump this histogram while touch was giving ENOSPC errors? Also, it might be worthwhile dumping the per-ag histograms (use a for loop and the "freesp -a <x>" command) - it may be that certain AGs are out of contiguous freespace and that is causing the issue... FWIW, you shoul drun "echo 1 > /proc/sys/vm/drop_caches" before running the xfsdb comand so that it is not reading stale metadata from cache... Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: ENOSPC but df and df -i show free space 2011-06-19 23:23 ` Dave Chinner @ 2011-06-19 23:55 ` Andy Isaacson 0 siblings, 0 replies; 5+ messages in thread From: Andy Isaacson @ 2011-06-19 23:55 UTC (permalink / raw) To: Dave Chinner; +Cc: xfs On Mon, Jun 20, 2011 at 09:23:39AM +1000, Dave Chinner wrote: > No allocation algorithm is perfect in all circumstances. The > alogrithms in XFS tend to degrade when large contiguous freespace > regions are not available, resulting in more fragmentation of data > extents and subsequent freespace fragmentation when those files are > removed or defragmented. The algorithms will recover if you free up > enough space that large contiguous freespace extents re-form, but > that can require removing a large amount of data.... Thanks for the background explanation! > > > > % df -i /d1 > > > > Filesystem Inodes IUsed IFree IUse% Mounted on > > > > /dev/mapper/vg0-d1 167509008 11806336 155702672 8% /d1 > > > > % sudo xfs_growfs -n /d1 > > > > meta-data=/dev/mapper/vg0-d1 isize=256 agcount=18, agsize=13107200 blks > > > > = sectsz=512 attr=2 > > > > data = bsize=4096 blocks=235929600, imaxpct=25 > > > > = sunit=0 swidth=0 blks > > > > naming =version 2 bsize=4096 ascii-ci=0 > > > > log =internal bsize=4096 blocks=25600, version=2 > > > > = sectsz=512 sunit=0 blks, lazy-count=1 > > > > realtime =none extsz=4096 blocks=0, rtextents=0 > > > > % grep d1 /proc/mounts > > > > /dev/mapper/vg0-d1 /d1 xfs rw,relatime,attr2,noquota 0 0 > > > > > > > > Obviously I'm missing something, but what? > > > > > > Most likely is that you have no contiguous free space large enough > > > to create a new inode chunk. using xfs_db to dump the freespace > > > size histogram will tell you if this is the case or not. > > > > % sudo xfs_db -c freesp /dev/vg0/d1 > > from to extents blocks pct > > 1 1 168504 168504 1.71 > > 2 3 446 1135 0.01 > > 4 7 5550 37145 0.38 > > 8 15 49159 524342 5.33 > > 16 31 1383 29223 0.30 > > 2097152 4194303 1 2931455 29.78 > > 4194304 8388607 1 6150953 62.49 > > > > I don't really grok that output. > > It's the historgram of free space extent sizes. You have 168504 > single free block regions (4k in size) in the filesystem, 446 > between 8k and 12k (2-3 blocks), etc. Ah, OK! Now it makes sense. > Inode allocation requires aligned 16k allocations (64x256 byte > inodes), so you need free extents in the 4-7 block range or larger, > which you appear to have so it should not be failing. Did you dump > this histogram while touch was giving ENOSPC errors? Yes, that was before I grew the filesystem again to get back to a working state. I killed all the processes using the filesystem, unmounted it, and ran xfs_db. > Also, it might be worthwhile dumping the per-ag histograms (use a > for loop and the "freesp -a <x>" command) - it may be that certain > AGs are out of contiguous freespace and that is causing the issue... I've now grown the filesystem again to get it back into a working state; it's obviously not "production" per se given my janky configuration, but it is more convenient if I can create files. :) It's a shame that we lost the chance to do more debugging though. > FWIW, you shoul drun "echo 1 > /proc/sys/vm/drop_caches" before > running the xfsdb comand so that it is not reading stale metadata > from cache... I unmounted the filesystem before running xfs_db, so it should be fine. Didn't even occur to me that it might work to run xfs_db on a block device that's mounted and active... I did notice that the unmount command took a minute or two to complete. -andy _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2011-06-19 23:54 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-06-19 21:50 ENOSPC but df and df -i show free space Andy Isaacson 2011-06-19 22:18 ` Dave Chinner 2011-06-19 22:58 ` Andy Isaacson 2011-06-19 23:23 ` Dave Chinner 2011-06-19 23:55 ` Andy Isaacson
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox