* Slightly Urgent: XFS No Space Left On Device
@ 2015-03-30 18:19 Dave Hall
2015-03-30 19:45 ` Dave Chinner
0 siblings, 1 reply; 8+ messages in thread
From: Dave Hall @ 2015-03-30 18:19 UTC (permalink / raw)
To: xfs
[-- Attachment #1.1: Type: text/plain, Size: 2538 bytes --]
Hello,
I have an XFS file system that's getting 'No space left on device'
errors. xfs_fsr also complains of 'No space left'. The XFS Info is:
# xfs_info /data
meta-data=/dev/sdb1 isize=256 agcount=19,
agsize=268435440 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=4882431488, imaxpct=5
= sunit=16 swidth=160 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=16 blks,
lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 19T 12T 7.0T 62% /data
# df -ih .
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sdb1 3.7G 4.7M 3.7G 1% /data
xfs_db freesp shows that AG 0 seems to be full. I've included the
freesp for the first few AGs, but the rest seem pretty consistent with
AGs 1 - 4 that I've included below.
xfs_db> freesp -s -e 1000000000 -a 0
from to extents blocks pct
1 268435440 1930 3795 100.00
total free extents 1930
total free blocks 3795
average free extent size 1.96632
xfs_db> freesp -s -e 1000000000 -a 1
from to extents blocks pct
1 268435440 287006 173832255 100.00
total free extents 287006
total free blocks 173832255
average free extent size 605.675
xfs_db> freesp -s -e 1000000000 -a 2
from to extents blocks pct
1 268435440 272425 94291252 100.00
total free extents 272425
total free blocks 94291252
average free extent size 346.118
xfs_db> freesp -s -e 1000000000 -a 3
from to extents blocks pct
1 268435440 286421 110208404 100.00
total free extents 286421
total free blocks 110208404
average free extent size 384.778
xfs_db> freesp -s -e 1000000000 -a 4
from to extents blocks pct
1 268435440 277220 107118347 100.00
total free extents 277220
total free blocks 107118347
average free extent size 386.402
xfs_db>
Is there any way to figure out which files/directories are clogging up
AG 0? Any other ways to abate this issue? Any way to assure
proportional usage of AGs?
Thanks.
-Dave
[-- Attachment #1.2: Type: text/html, Size: 3907 bytes --]
[-- Attachment #2: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: Slightly Urgent: XFS No Space Left On Device 2015-03-30 18:19 Slightly Urgent: XFS No Space Left On Device Dave Hall @ 2015-03-30 19:45 ` Dave Chinner 2015-04-01 19:53 ` Dave Hall 0 siblings, 1 reply; 8+ messages in thread From: Dave Chinner @ 2015-03-30 19:45 UTC (permalink / raw) To: Dave Hall; +Cc: xfs On Mon, Mar 30, 2015 at 02:19:59PM -0400, Dave Hall wrote: > Hello, > > I have an XFS file system that's getting 'No space left on device' > errors. xfs_fsr also complains of 'No space left'. The XFS Info > is: > > # xfs_info /data > meta-data=/dev/sdb1 isize=256 agcount=19, > agsize=268435440 blks > = sectsz=512 attr=2 > data = bsize=4096 blocks=4882431488, imaxpct=5 > = sunit=16 swidth=160 blks > naming =version 2 bsize=4096 ascii-ci=0 > log =internal bsize=4096 blocks=521728, version=2 > = sectsz=512 sunit=16 blks, > lazy-count=1 > realtime =none extsz=4096 blocks=0, rtextents=0 > > # df -h . > Filesystem Size Used Avail Use% Mounted on > /dev/sdb1 19T 12T 7.0T 62% /data > # df -ih . > Filesystem Inodes IUsed IFree IUse% Mounted on > /dev/sdb1 3.7G 4.7M 3.7G 1% /data > > > xfs_db freesp shows that AG 0 seems to be full. I've included the > freesp for the first few AGs, but the rest seem pretty consistent > with AGs 1 - 4 that I've included below. > > xfs_db> freesp -s -e 1000000000 -a 0 Can you please drop the "-e 1000000" from these commands and post it again? The histogram of differing free space sizes is information we actually need to diagnose the problem... Also, kernel version, mount options and machine details are necessary to determine why AG0 might be full (e.g. allocation policy in use). Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Slightly Urgent: XFS No Space Left On Device 2015-03-30 19:45 ` Dave Chinner @ 2015-04-01 19:53 ` Dave Hall 2015-04-02 0:12 ` Dave Chinner 0 siblings, 1 reply; 8+ messages in thread From: Dave Hall @ 2015-04-01 19:53 UTC (permalink / raw) To: Dave Chinner; +Cc: xfs [-- Attachment #1.1: Type: text/plain, Size: 24357 bytes --] Please pardon the 'top-post', but here is the additional information requested: This is a Dell R720xd dual 8-core Xeon system with 128GB RAM. The RAID controller is Dell PERC H710 Mini with 12 2TB disks in RAID6. The OS is Debian 6 with kernel 3.2.0-0.bpo.4-amd64 #1 SMP Debian 3.2.65-1+deb7u2~bpo60+1 x86_64. From /proc/mounts: /dev/sdb1 /data xfs rw,noexec,noatime,attr2,delaylog,allocsize=64k,logbsize=64k,sunit=128,swidth=1280,usrquota,prjquota 0 0 Content-wise there are 7 first level directories. Four contain just a couple files. One of these has a 4.9TB file in it. The other 3 directories are multi-terabyte, but contain many hundreds of thousands of smaller files. There are nearly 5 million files in about 6500 directories, but less than 500 files are over 1GB in size, with only 200 over 20GB and less than 10 over 1TB. The output from xfs_info was previously included, but is repeated here: # xfs_info /data meta-data=/dev/sdb1 isize=256 agcount=19, agsize=268435440 blks = sectsz=512 attr=2 data = bsize=4096 blocks=4882431488, imaxpct=5 = sunit=16 swidth=160 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal bsize=4096 blocks=521728, version=2 = sectsz=512 sunit=16 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 # df -h . Filesystem Size Used Avail Use% Mounted on /dev/sdb1 19T 12T 7.0T 62% /data # df -ih . Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sdb1 3.7G 4.7M 3.7G 1% /data Here are the more extensive freesp outputs for each of the 19 AGs: # xfs_db -r /dev/sdb1 -c 'freesp -s -a0' from to extents blocks pct 1 1 747 747 19.68 2 3 1045 2496 65.77 4 7 138 552 14.55 total free extents 1930 total free blocks 3795 average free extent size 1.96632 (I don't recall the output from AG0 being so terse on Monday when I first posted, but the summary information is the same.) # xfs_db -r /dev/sdb1 -c 'freesp -s -a1' from to extents blocks pct 1 1 4494 4494 0.00 2 3 42096 91313 0.05 4 7 41096 232953 0.13 8 15 121930 1401067 0.81 16 31 44994 1067002 0.61 32 63 4 209 0.00 64 127 20 1888 0.00 128 255 38 7408 0.00 256 511 14728 5936038 3.41 512 1023 308 246748 0.14 1024 2047 14893 22978919 13.22 2048 4095 1229 4118315 2.37 4096 8191 778 4743029 2.73 8192 16383 329 3694322 2.13 16384 32767 51 1098154 0.63 32768 65535 3 98794 0.06 65536 131071 3 275197 0.16 131072 262143 4 957177 0.55 1048576 2097151 1 1968807 1.13 2097152 4194303 1 3085945 1.78 4194304 8388607 1 5131888 2.95 8388608 16777215 3 33124064 19.06 16777216 33554431 1 28684574 16.50 33554432 67108863 1 54883950 31.57 total free extents 287006 total free blocks 173832255 average free extent size 605.675 # xfs_db -r /dev/sdb1 -c 'freesp -s -a2' from to extents blocks pct 1 1 5695 5695 0.01 2 3 15405 39309 0.04 4 7 52230 296302 0.31 8 15 112686 1303036 1.38 16 31 967 20943 0.02 32 63 67 2983 0.00 64 127 343 31251 0.03 128 255 100 17428 0.02 256 511 76672 30821379 32.69 512 1023 4 2800 0.00 1024 2047 7537 11762194 12.47 2048 4095 326 1130975 1.20 4096 8191 251 1615591 1.71 8192 16383 105 1184516 1.26 16384 32767 14 274014 0.29 65536 131071 1 73535 0.08 131072 262143 1 234632 0.25 262144 524287 2 788639 0.84 524288 1048575 1 738305 0.78 1048576 2097151 17 34302421 36.38 8388608 16777215 1 9645304 10.23 total free extents 272425 total free blocks 94291252 # xfs_db -r /dev/sdb1 -c 'freesp -s -a3' from to extents blocks pct 1 1 5793 5793 0.01 2 3 30667 70359 0.06 4 7 53174 301241 0.27 8 15 129098 1480652 1.34 16 31 4875 116797 0.11 32 63 148 6755 0.01 64 127 13192 1200672 1.09 128 255 1754 286342 0.26 256 511 35132 14122824 12.81 512 1023 6 4349 0.00 1024 2047 11609 18155617 16.47 2048 4095 447 1557312 1.41 4096 8191 342 2120360 1.92 8192 16383 147 1685429 1.53 16384 32767 21 438149 0.40 32768 65535 5 221907 0.20 65536 131071 4 384869 0.35 131072 262143 3 576503 0.52 524288 1048575 1 524974 0.48 1048576 2097151 2 2718327 2.47 33554432 67108863 1 64229173 58.28 total free extents 286421 total free blocks 110208404 average free extent size 384.778 # xfs_db -r /dev/sdb1 -c 'freesp -s -a4' from to extents blocks pct 1 1 5399 5399 0.01 2 3 29098 67289 0.06 4 7 50889 287977 0.27 8 15 125018 1433485 1.34 16 31 4601 108565 0.10 32 63 86 3986 0.00 64 127 12587 1145709 1.07 128 255 1537 250472 0.23 256 511 35982 14464615 13.50 512 1023 2 1039 0.00 1024 2047 11074 17306417 16.16 2048 4095 428 1488906 1.39 4096 8191 343 2130436 1.99 8192 16383 141 1574556 1.47 16384 32767 22 437491 0.41 32768 65535 2 74530 0.07 65536 131071 2 198418 0.19 131072 262143 2 399680 0.37 262144 524287 1 278259 0.26 524288 1048575 1 858623 0.80 2097152 4194303 1 2357798 2.20 4194304 8388607 1 7007241 6.54 8388608 16777215 2 24665312 23.03 16777216 33554431 1 30572144 28.54 total free extents 277220 total free blocks 107118347 average free extent size 386.402 # xfs_db -r /dev/sdb1 -c 'freesp -s -a5' from to extents blocks pct 1 1 5623 5623 0.01 2 3 28053 65224 0.06 4 7 51000 288250 0.27 8 15 122593 1405739 1.32 16 31 4439 104165 0.10 32 63 107 4913 0.00 64 127 10904 992287 0.93 128 255 1458 237872 0.22 256 511 37480 15066766 14.19 512 1023 4 3298 0.00 1024 2047 11035 17206454 16.20 2048 4095 416 1447533 1.36 4096 8191 367 2264983 2.13 8192 16383 132 1507258 1.42 16384 32767 17 369018 0.35 32768 65535 5 252737 0.24 65536 131071 1 93292 0.09 131072 262143 2 369218 0.35 262144 524287 1 371390 0.35 8388608 16777215 1 11907027 11.21 16777216 33554431 1 17447945 16.43 33554432 67108863 1 34801264 32.77 total free extents 273640 total free blocks 106212256 average free extent size 388.146 # xfs_db -r /dev/sdb1 -c 'freesp -s -a6' from to extents blocks pct 1 1 5485 5485 0.01 2 3 28092 65622 0.06 4 7 51124 288408 0.27 8 15 122946 1411945 1.32 16 31 4295 99036 0.09 32 63 136 6165 0.01 64 127 10723 975901 0.91 128 255 1393 227148 0.21 256 511 37816 15202240 14.21 512 1023 9 6955 0.01 1024 2047 11001 17139027 16.02 2048 4095 452 1570875 1.47 4096 8191 310 1937437 1.81 8192 16383 140 1622878 1.52 16384 32767 22 432606 0.40 32768 65535 3 119928 0.11 65536 131071 2 201539 0.19 131072 262143 1 242792 0.23 524288 1048575 2 1642283 1.53 1048576 2097151 2 2522760 2.36 4194304 8388607 2 9405762 8.79 16777216 33554431 2 51878521 48.48 total free extents 273958 total free blocks 107005313 average free extent size 390.59 # xfs_db -r /dev/sdb1 -c 'freesp -s -a7' from to extents blocks pct 1 1 5728 5728 0.01 2 3 27342 63963 0.06 4 7 51098 288588 0.27 8 15 122083 1400413 1.29 16 31 4154 95945 0.09 32 63 125 5696 0.01 64 127 10490 954737 0.88 128 255 1377 225554 0.21 256 511 38215 15362799 14.12 512 1023 5 3014 0.00 1024 2047 11138 17383490 15.98 2048 4095 446 1547400 1.42 4096 8191 314 1940099 1.78 8192 16383 138 1553781 1.43 16384 32767 26 526808 0.48 32768 65535 5 198738 0.18 65536 131071 3 306072 0.28 131072 262143 1 204457 0.19 524288 1048575 1 675084 0.62 4194304 8388607 1 6256240 5.75 8388608 16777215 1 16700425 15.35 16777216 33554431 2 43106323 39.62 total free extents 272693 total free blocks 108805354 average free extent size 399.003 # xfs_db -r /dev/sdb1 -c 'freesp -s -a8' from to extents blocks pct 1 1 5545 5545 0.01 2 3 27537 64379 0.06 4 7 50486 284834 0.28 8 15 121719 1398087 1.35 16 31 4169 96146 0.09 32 63 140 6404 0.01 64 127 10168 925246 0.90 128 255 1347 219934 0.21 256 511 38396 15435162 14.96 512 1023 9 6657 0.01 1024 2047 11038 17234155 16.70 2048 4095 411 1427988 1.38 4096 8191 337 2110360 2.04 8192 16383 134 1540661 1.49 16384 32767 29 608663 0.59 32768 65535 4 194772 0.19 65536 131071 1 103722 0.10 131072 262143 1 204540 0.20 1048576 2097151 1 1177573 1.14 16777216 33554431 1 19036961 18.45 33554432 67108863 1 41120777 39.84 total free extents 271474 total free blocks 103202566 average free extent size 380.156 # xfs_db -r /dev/sdb1 -c 'freesp -s -a9' from to extents blocks pct 1 1 5614 5614 0.01 2 3 27343 63817 0.06 4 7 50789 286921 0.26 8 15 122085 1402116 1.28 16 31 4116 95310 0.09 32 63 152 6954 0.01 64 127 10679 971872 0.89 128 255 1315 215145 0.20 256 511 38557 15499672 14.19 512 1023 6 4435 0.00 1024 2047 11119 17330956 15.86 2048 4095 428 1485414 1.36 4096 8191 313 1932235 1.77 8192 16383 158 1823615 1.67 16384 32767 20 427607 0.39 32768 65535 4 162954 0.15 65536 131071 1 74125 0.07 262144 524287 2 782823 0.72 524288 1048575 1 979230 0.90 4194304 8388607 1 6064549 5.55 33554432 67108863 1 59625070 54.58 total free extents 272704 total free blocks 109240434 average free extent size 400.582 # xfs_db -r /dev/sdb1 -c 'freesp -s -a10' from to extents blocks pct 1 1 5451 5451 0.01 2 3 27619 64469 0.06 4 7 50888 287306 0.27 8 15 122129 1401775 1.30 16 31 4156 96465 0.09 32 63 112 5115 0.00 64 127 10378 944415 0.87 128 255 1336 218180 0.20 256 511 38056 15298154 14.15 512 1023 6 3630 0.00 1024 2047 10908 17025890 15.75 2048 4095 443 1541035 1.43 4096 8191 326 2036141 1.88 8192 16383 150 1670607 1.55 16384 32767 23 497495 0.46 32768 65535 6 259503 0.24 65536 131071 1 80765 0.07 131072 262143 2 466041 0.43 8388608 16777215 2 24552174 22.72 16777216 33554431 2 41626231 38.51 total free extents 271994 total free blocks 108080842 average free extent size 397.365 # xfs_db -r /dev/sdb1 -c 'freesp -s -a11' from to extents blocks pct 1 1 5753 5753 0.01 2 3 28506 66164 0.06 4 7 51222 289018 0.27 8 15 122115 1400237 1.31 16 31 4325 100622 0.09 32 63 121 5515 0.01 64 127 11218 1020941 0.95 128 255 1419 231469 0.22 256 511 37233 14967258 13.96 512 1023 13 10433 0.01 1024 2047 11040 17243570 16.08 2048 4095 438 1528105 1.42 4096 8191 313 1948122 1.82 8192 16383 137 1545209 1.44 16384 32767 17 340315 0.32 32768 65535 3 135239 0.13 524288 1048575 1 806510 0.75 1048576 2097151 1 1670160 1.56 2097152 4194303 1 3359120 3.13 4194304 8388607 1 4927086 4.59 8388608 16777215 2 26372734 24.59 16777216 33554431 1 29269614 27.29 total free extents 273880 total free blocks 107243194 average free extent size 391.57 # xfs_db -r /dev/sdb1 -c 'freesp -s -a12' from to extents blocks pct 1 1 5373 5373 0.01 2 3 27530 64216 0.06 4 7 50788 286603 0.27 8 15 121652 1396720 1.31 16 31 4188 97008 0.09 32 63 71 3299 0.00 64 127 10446 950836 0.89 128 255 1349 220210 0.21 256 511 37835 15209592 14.28 512 1023 1 918 0.00 1024 2047 10950 17087135 16.04 2048 4095 416 1445170 1.36 4096 8191 341 2103801 1.98 8192 16383 146 1643458 1.54 16384 32767 27 551354 0.52 32768 65535 5 173876 0.16 65536 131071 3 273193 0.26 262144 524287 2 695714 0.65 524288 1048575 2 1740580 1.63 16777216 33554431 1 22797321 21.40 33554432 67108863 1 39761770 37.33 total free extents 271127 total free blocks 106508147 average free extent size 392.835 # xfs_db -r /dev/sdb1 -c 'freesp -s -a13' from to extents blocks pct 1 1 5756 5756 0.01 2 3 27074 63268 0.06 4 7 50796 287174 0.26 8 15 121675 1397015 1.28 16 31 4260 98417 0.09 32 63 136 6191 0.01 64 127 10324 939549 0.86 128 255 1315 215314 0.20 256 511 39195 15756002 14.40 512 1023 8 5675 0.01 1024 2047 11129 17335479 15.84 2048 4095 419 1457554 1.33 4096 8191 321 2012733 1.84 8192 16383 143 1666063 1.52 16384 32767 23 460740 0.42 32768 65535 2 103286 0.09 65536 131071 2 193585 0.18 262144 524287 1 356370 0.33 33554432 67108863 1 67081225 61.29 total free extents 272580 total free blocks 109441396 average free extent size 401.502 # xfs_db -r /dev/sdb1 -c 'freesp -s -a14' from to extents blocks pct 1 1 5585 5585 0.01 2 3 26740 62793 0.06 4 7 50781 286750 0.27 8 15 120804 1388061 1.30 16 31 4186 96930 0.09 32 63 160 7192 0.01 64 127 9898 900897 0.84 128 255 1318 215049 0.20 256 511 38427 15446911 14.48 512 1023 7 4130 0.00 1024 2047 11116 17330340 16.25 2048 4095 390 1356701 1.27 4096 8191 307 1917633 1.80 8192 16383 150 1679866 1.57 16384 32767 24 490742 0.46 32768 65535 3 156921 0.15 65536 131071 3 290496 0.27 524288 1048575 1 715032 0.67 1048576 2097151 1 1570472 1.47 33554432 67108863 1 62750353 58.83 total free extents 269902 total free blocks 106672854 average free extent size 395.228 # xfs_db -r /dev/sdb1 -c 'freesp -s -a15' from to extents blocks pct 1 1 5734 5734 0.01 2 3 15777 40616 0.05 4 7 51372 290289 0.39 8 15 121640 1396823 1.88 16 31 3105 69153 0.09 32 63 14 700 0.00 64 127 19 1760 0.00 128 255 3157 530350 0.71 256 511 18 7797 0.01 512 1023 7 4504 0.01 1024 2047 44155 71890115 96.61 2048 4095 5 13601 0.02 4096 8191 4 20168 0.03 8192 16383 3 24586 0.03 16384 32767 4 80524 0.11 32768 65535 1 37430 0.05 total free extents 245015 total free blocks 74414150 average free extent size 303.713 # xfs_db -r /dev/sdb1 -c 'freesp -s -a16' from to extents blocks pct 1 1 5458 5458 0.01 2 3 29896 69017 0.07 4 7 50646 286147 0.28 8 15 123250 1414603 1.37 16 31 4257 99155 0.10 32 63 112 5139 0.00 64 127 13228 1203844 1.17 128 255 1363 222544 0.22 256 511 31264 12567433 12.17 512 1023 8 5911 0.01 1024 2047 11091 17306832 16.76 2048 4095 452 1573760 1.52 4096 8191 356 2239416 2.17 8192 16383 135 1522673 1.47 16384 32767 17 350269 0.34 32768 65535 3 122543 0.12 65536 131071 3 374987 0.36 131072 262143 5 1169749 1.13 524288 1048575 2 1884165 1.83 1048576 2097151 1 1237015 1.20 8388608 16777215 1 9194667 8.91 16777216 33554431 2 50384042 48.80 total free extents 271550 total free blocks 103239369 average free extent size 380.185 # xfs_db -r /dev/sdb1 -c 'freesp -s -a17' from to extents blocks pct 1 1 5788 5788 0.01 2 3 26404 61921 0.06 4 7 50904 287563 0.27 8 15 120710 1385219 1.30 16 31 4204 97175 0.09 32 63 76 3490 0.00 64 127 10045 914186 0.86 128 255 1392 228552 0.21 256 511 36867 14820192 13.90 512 1023 7 4938 0.00 1024 2047 11286 17637792 16.54 2048 4095 441 1532071 1.44 4096 8191 334 2078958 1.95 8192 16383 123 1419610 1.33 16384 32767 19 396082 0.37 32768 65535 5 224967 0.21 65536 131071 4 362807 0.34 131072 262143 1 155224 0.15 262144 524287 2 866414 0.81 524288 1048575 1 999449 0.94 1048576 2097151 1 1158766 1.09 2097152 4194303 1 2528878 2.37 8388608 16777215 3 39151859 36.72 16777216 33554431 1 20313097 19.05 total free extents 268619 total free blocks 106634998 average free extent size 396.975 # xfs_db -r /dev/sdb1 -c 'freesp -s -a18' from to extents blocks pct 1 1 5588 5588 0.03 2 3 24900 58887 0.32 4 7 50929 287739 1.56 8 15 120592 1386142 7.52 16 31 4089 93924 0.51 32 63 141 6372 0.03 64 127 8468 770640 4.18 128 255 1339 218783 1.19 256 511 22 8582 0.05 512 1023 4 2711 0.01 1024 2047 9719 15333235 83.15 2048 4095 4 10960 0.06 4096 8191 1 4097 0.02 8192 16383 1 8769 0.05 16384 32767 2 32791 0.18 32768 65535 5 210714 1.14 total free extents 225804 total free blocks 18439934 average free extent size 81.6635 Dave Hall Binghamton University kdhall@binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office) On 03/30/2015 03:45 PM, Dave Chinner wrote: > On Mon, Mar 30, 2015 at 02:19:59PM -0400, Dave Hall wrote: > >> Hello, >> >> I have an XFS file system that's getting 'No space left on device' >> errors. xfs_fsr also complains of 'No space left'. The XFS Info >> is: >> >> # xfs_info /data >> meta-data=/dev/sdb1 isize=256 agcount=19, >> agsize=268435440 blks >> = sectsz=512 attr=2 >> data = bsize=4096 blocks=4882431488, imaxpct=5 >> = sunit=16 swidth=160 blks >> naming =version 2 bsize=4096 ascii-ci=0 >> log =internal bsize=4096 blocks=521728, version=2 >> = sectsz=512 sunit=16 blks, >> lazy-count=1 >> realtime =none extsz=4096 blocks=0, rtextents=0 >> >> # df -h . >> Filesystem Size Used Avail Use% Mounted on >> /dev/sdb1 19T 12T 7.0T 62% /data >> # df -ih . >> Filesystem Inodes IUsed IFree IUse% Mounted on >> /dev/sdb1 3.7G 4.7M 3.7G 1% /data >> >> >> xfs_db freesp shows that AG 0 seems to be full. I've included the >> freesp for the first few AGs, but the rest seem pretty consistent >> with AGs 1 - 4 that I've included below. >> >> xfs_db> freesp -s -e 1000000000 -a 0 >> > Can you please drop the "-e 1000000" from these commands and post it > again? The histogram of differing free space sizes is information > we actually need to diagnose the problem... > > Also, kernel version, mount options and machine details are > necessary to determine why AG0 might be full (e.g. allocation policy > in use). > > Cheers, > > Dave. > [-- Attachment #1.2: Type: text/html, Size: 49665 bytes --] [-- Attachment #2: Type: text/plain, Size: 121 bytes --] _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Slightly Urgent: XFS No Space Left On Device 2015-04-01 19:53 ` Dave Hall @ 2015-04-02 0:12 ` Dave Chinner 2015-04-02 14:32 ` Dave Hall 0 siblings, 1 reply; 8+ messages in thread From: Dave Chinner @ 2015-04-02 0:12 UTC (permalink / raw) To: Dave Hall; +Cc: xfs On Wed, Apr 01, 2015 at 03:53:28PM -0400, Dave Hall wrote: > Please pardon the 'top-post', but here is the additional information > requested: > > This is a Dell R720xd dual 8-core Xeon system with 128GB RAM. The > RAID controller is Dell PERC H710 Mini with 12 2TB disks in RAID6. > > The OS is Debian 6 with kernel 3.2.0-0.bpo.4-amd64 #1 SMP Debian > 3.2.65-1+deb7u2~bpo60+1 x86_64. So defaults to inode32 allocation.... > From /proc/mounts: > > /dev/sdb1 /data xfs > rw,noexec,noatime,attr2,delaylog,allocsize=64k,logbsize=64k,sunit=128,swidth=1280,usrquota,prjquota > 0 0 ... and inode64 is not in the mount options..... > The output from xfs_info was previously included, but is repeated here: > > # xfs_info /data > meta-data=/dev/sdb1 isize=256 agcount=19,agsize=268435440 blks Inode allocation requires contiguous free space of 16k aligned to 8k boundaries to allocate new inode chunks. Also, 1TB AGs, so with inode32, inodes can only be allocated in AG 0. > Here are the more extensive freesp outputs for each of the 19 AGs: > > # xfs_db -r /dev/sdb1 -c 'freesp -s -a0' > from to extents blocks pct > 1 1 747 747 19.68 > 2 3 1045 2496 65.77 > 4 7 138 552 14.55 > total free extents 1930 > total free blocks 3795 > average free extent size 1.96632 And that says you have no correctly aligned free 16k extents that can be allocated in AG 0. i.e. no more inodes can be allocated, and that's where the ENOSPC is coming from. Unmount, add the inode64 mount option, and you'll be able to allocate inodes again as they will be allowed to be allocated in any AG, not just AG 0. Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Slightly Urgent: XFS No Space Left On Device 2015-04-02 0:12 ` Dave Chinner @ 2015-04-02 14:32 ` Dave Hall 2015-04-02 14:36 ` Grozdan 2015-04-02 22:36 ` Dave Chinner 0 siblings, 2 replies; 8+ messages in thread From: Dave Hall @ 2015-04-02 14:32 UTC (permalink / raw) To: Dave Chinner; +Cc: xfs Thanks for the help. Rookie error. I didn't set these mount options, but I see that this option is set for all of the other XFS volumes I have. I am wondering why XFS would default this way though. Seems like heuristically you could assume that a large volume on a 64-bit OS would need 64-bit inodes. At least perhaps put out a message from mkfs.xfs suggesting the use of inode64 on the mount command? Thanks. -Dave Dave Hall Binghamton University kdhall@binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office) On 04/01/2015 08:12 PM, Dave Chinner wrote: > On Wed, Apr 01, 2015 at 03:53:28PM -0400, Dave Hall wrote: > >> Please pardon the 'top-post', but here is the additional information >> requested: >> >> This is a Dell R720xd dual 8-core Xeon system with 128GB RAM. The >> RAID controller is Dell PERC H710 Mini with 12 2TB disks in RAID6. >> >> The OS is Debian 6 with kernel 3.2.0-0.bpo.4-amd64 #1 SMP Debian >> 3.2.65-1+deb7u2~bpo60+1 x86_64. >> > So defaults to inode32 allocation.... > > >> From /proc/mounts: >> >> /dev/sdb1 /data xfs >> rw,noexec,noatime,attr2,delaylog,allocsize=64k,logbsize=64k,sunit=128,swidth=1280,usrquota,prjquota >> 0 0 >> > ... and inode64 is not in the mount options..... > > >> The output from xfs_info was previously included, but is repeated here: >> >> # xfs_info /data >> meta-data=/dev/sdb1 isize=256 agcount=19,agsize=268435440 blks >> > Inode allocation requires contiguous free space of 16k aligned to 8k > boundaries to allocate new inode chunks. Also, 1TB AGs, so with > inode32, inodes can only be allocated in AG 0. > > >> Here are the more extensive freesp outputs for each of the 19 AGs: >> >> # xfs_db -r /dev/sdb1 -c 'freesp -s -a0' >> from to extents blocks pct >> 1 1 747 747 19.68 >> 2 3 1045 2496 65.77 >> 4 7 138 552 14.55 >> total free extents 1930 >> total free blocks 3795 >> average free extent size 1.96632 >> > And that says you have no correctly aligned free 16k extents that > can be allocated in AG 0. i.e. no more inodes can be allocated, and > that's where the ENOSPC is coming from. > > Unmount, add the inode64 mount option, and you'll be able to > allocate inodes again as they will be allowed to be allocated in > any AG, not just AG 0. > > Cheers, > > Dave. > _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Slightly Urgent: XFS No Space Left On Device 2015-04-02 14:32 ` Dave Hall @ 2015-04-02 14:36 ` Grozdan 2015-04-02 14:41 ` Grozdan 2015-04-02 22:36 ` Dave Chinner 1 sibling, 1 reply; 8+ messages in thread From: Grozdan @ 2015-04-02 14:36 UTC (permalink / raw) To: Dave Hall; +Cc: Xfs On Thu, Apr 2, 2015 at 4:32 PM, Dave Hall <kdhall@binghamton.edu> wrote: > Thanks for the help. Rookie error. I didn't set these mount options, but I > see that this option is set for all of the other XFS volumes I have. > > I am wondering why XFS would default this way though. Seems like > heuristically you could assume that a large volume on a 64-bit OS would need > 64-bit inodes. At least perhaps put out a message from mkfs.xfs suggesting > the use of inode64 on the mount command? inode64 has been made default, even for 32-bit systems, by recent versions of xfsprogs so I'd suggest to upgrade your xfsprogs > > Thanks. > > -Dave > > Dave Hall > Binghamton University > kdhall@binghamton.edu > 607-760-2328 (Cell) > 607-777-4641 (Office) > > > On 04/01/2015 08:12 PM, Dave Chinner wrote: >> >> On Wed, Apr 01, 2015 at 03:53:28PM -0400, Dave Hall wrote: >> >>> >>> Please pardon the 'top-post', but here is the additional information >>> requested: >>> >>> This is a Dell R720xd dual 8-core Xeon system with 128GB RAM. The >>> RAID controller is Dell PERC H710 Mini with 12 2TB disks in RAID6. >>> >>> The OS is Debian 6 with kernel 3.2.0-0.bpo.4-amd64 #1 SMP Debian >>> 3.2.65-1+deb7u2~bpo60+1 x86_64. >>> >> >> So defaults to inode32 allocation.... >> >> >>> >>> From /proc/mounts: >>> >>> /dev/sdb1 /data xfs >>> >>> rw,noexec,noatime,attr2,delaylog,allocsize=64k,logbsize=64k,sunit=128,swidth=1280,usrquota,prjquota >>> 0 0 >>> >> >> ... and inode64 is not in the mount options..... >> >> >>> >>> The output from xfs_info was previously included, but is repeated here: >>> >>> # xfs_info /data >>> meta-data=/dev/sdb1 isize=256 agcount=19,agsize=268435440 >>> blks >>> >> >> Inode allocation requires contiguous free space of 16k aligned to 8k >> boundaries to allocate new inode chunks. Also, 1TB AGs, so with >> inode32, inodes can only be allocated in AG 0. >> >> >>> >>> Here are the more extensive freesp outputs for each of the 19 AGs: >>> >>> # xfs_db -r /dev/sdb1 -c 'freesp -s -a0' >>> from to extents blocks pct >>> 1 1 747 747 19.68 >>> 2 3 1045 2496 65.77 >>> 4 7 138 552 14.55 >>> total free extents 1930 >>> total free blocks 3795 >>> average free extent size 1.96632 >>> >> >> And that says you have no correctly aligned free 16k extents that >> can be allocated in AG 0. i.e. no more inodes can be allocated, and >> that's where the ENOSPC is coming from. >> >> Unmount, add the inode64 mount option, and you'll be able to >> allocate inodes again as they will be allowed to be allocated in >> any AG, not just AG 0. >> >> Cheers, >> >> Dave. >> > > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs -- Yours truly _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Slightly Urgent: XFS No Space Left On Device 2015-04-02 14:36 ` Grozdan @ 2015-04-02 14:41 ` Grozdan 0 siblings, 0 replies; 8+ messages in thread From: Grozdan @ 2015-04-02 14:41 UTC (permalink / raw) To: Dave Hall; +Cc: Xfs On Thu, Apr 2, 2015 at 4:36 PM, Grozdan <neutrino8@gmail.com> wrote: > On Thu, Apr 2, 2015 at 4:32 PM, Dave Hall <kdhall@binghamton.edu> wrote: >> Thanks for the help. Rookie error. I didn't set these mount options, but I >> see that this option is set for all of the other XFS volumes I have. >> >> I am wondering why XFS would default this way though. Seems like >> heuristically you could assume that a large volume on a 64-bit OS would need >> 64-bit inodes. At least perhaps put out a message from mkfs.xfs suggesting >> the use of inode64 on the mount command? > > > inode64 has been made default, even for 32-bit systems, by recent > versions of xfsprogs so I'd suggest to upgrade your xfsprogs sorry, I was thinking of the crc flag. XFS uses by default inode64 from kernel versions 3.7 and up > >> >> Thanks. >> >> -Dave >> >> Dave Hall >> Binghamton University >> kdhall@binghamton.edu >> 607-760-2328 (Cell) >> 607-777-4641 (Office) >> >> >> On 04/01/2015 08:12 PM, Dave Chinner wrote: >>> >>> On Wed, Apr 01, 2015 at 03:53:28PM -0400, Dave Hall wrote: >>> >>>> >>>> Please pardon the 'top-post', but here is the additional information >>>> requested: >>>> >>>> This is a Dell R720xd dual 8-core Xeon system with 128GB RAM. The >>>> RAID controller is Dell PERC H710 Mini with 12 2TB disks in RAID6. >>>> >>>> The OS is Debian 6 with kernel 3.2.0-0.bpo.4-amd64 #1 SMP Debian >>>> 3.2.65-1+deb7u2~bpo60+1 x86_64. >>>> >>> >>> So defaults to inode32 allocation.... >>> >>> >>>> >>>> From /proc/mounts: >>>> >>>> /dev/sdb1 /data xfs >>>> >>>> rw,noexec,noatime,attr2,delaylog,allocsize=64k,logbsize=64k,sunit=128,swidth=1280,usrquota,prjquota >>>> 0 0 >>>> >>> >>> ... and inode64 is not in the mount options..... >>> >>> >>>> >>>> The output from xfs_info was previously included, but is repeated here: >>>> >>>> # xfs_info /data >>>> meta-data=/dev/sdb1 isize=256 agcount=19,agsize=268435440 >>>> blks >>>> >>> >>> Inode allocation requires contiguous free space of 16k aligned to 8k >>> boundaries to allocate new inode chunks. Also, 1TB AGs, so with >>> inode32, inodes can only be allocated in AG 0. >>> >>> >>>> >>>> Here are the more extensive freesp outputs for each of the 19 AGs: >>>> >>>> # xfs_db -r /dev/sdb1 -c 'freesp -s -a0' >>>> from to extents blocks pct >>>> 1 1 747 747 19.68 >>>> 2 3 1045 2496 65.77 >>>> 4 7 138 552 14.55 >>>> total free extents 1930 >>>> total free blocks 3795 >>>> average free extent size 1.96632 >>>> >>> >>> And that says you have no correctly aligned free 16k extents that >>> can be allocated in AG 0. i.e. no more inodes can be allocated, and >>> that's where the ENOSPC is coming from. >>> >>> Unmount, add the inode64 mount option, and you'll be able to >>> allocate inodes again as they will be allowed to be allocated in >>> any AG, not just AG 0. >>> >>> Cheers, >>> >>> Dave. >>> >> >> >> _______________________________________________ >> xfs mailing list >> xfs@oss.sgi.com >> http://oss.sgi.com/mailman/listinfo/xfs > > > > -- > Yours truly -- Yours truly _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Slightly Urgent: XFS No Space Left On Device 2015-04-02 14:32 ` Dave Hall 2015-04-02 14:36 ` Grozdan @ 2015-04-02 22:36 ` Dave Chinner 1 sibling, 0 replies; 8+ messages in thread From: Dave Chinner @ 2015-04-02 22:36 UTC (permalink / raw) To: Dave Hall; +Cc: xfs On Thu, Apr 02, 2015 at 10:32:54AM -0400, Dave Hall wrote: > Thanks for the help. Rookie error. I didn't set these mount > options, but I see that this option is set for all of the other XFS > volumes I have. > > I am wondering why XFS would default this way though. Seems like > heuristically you could assume that a large volume on a 64-bit OS > would need 64-bit inodes. The historical argument against inode64 on 64 bit machines was NFS export and 32 bit clients, which could then break if we used 64 bit inodes. The Linux NFS client handles this just fine on 32bit machines (has for a few years), so there's no reason for us to care very much about this anymore.... > At least perhaps put out a message from > mkfs.xfs suggesting the use of inode64 on the mount command? inode64 is now the default for upstream kernels (has been for a couple of years), but older distros are not going to get those kernel updates, or anything we might have put in xfsprogs... Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2015-04-02 22:36 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2015-03-30 18:19 Slightly Urgent: XFS No Space Left On Device Dave Hall 2015-03-30 19:45 ` Dave Chinner 2015-04-01 19:53 ` Dave Hall 2015-04-02 0:12 ` Dave Chinner 2015-04-02 14:32 ` Dave Hall 2015-04-02 14:36 ` Grozdan 2015-04-02 14:41 ` Grozdan 2015-04-02 22:36 ` Dave Chinner
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox