* [Ocfs2-devel] disk full when it shouldn't @ 2010-11-08 15:14 Massimo Cetra 2010-11-09 1:06 ` Tao Ma 0 siblings, 1 reply; 10+ messages in thread From: Massimo Cetra @ 2010-11-08 15:14 UTC (permalink / raw) To: ocfs2-devel Hi all, i've been using OCFS2 for a while (2 years or so, now) but i still have a big problem. on each filesystem that i have, it happens that, even if df reports that the disk has plenty of free space avalaible, no one can write any file. This behaviour has happened for several usages of OCFS2 (webserver and mail storage) so both with big files and small files. It happens more frequently on the mail partitions that i manage. I have tried to search for some documentation about this problems and it seems that i'm not the only one experiencing this behaviour. The kernels showing this behaviour are both 2.6.32 and 2.6.31. This happens with OCFS2-tools 1.4.2 and 1.4.3 In the last weeks i have reformatted some partitions from scratch using 1.4.3 but the problem persists. I have found somewhere a bash script to analyze the sysdir and output useful informations for the debugging. The output of this script during the problem is attached. Any hint about this ? Thanks, Max -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: debugfs.txt Url: http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20101108/6a59bcae/attachment-0001.txt ^ permalink raw reply [flat|nested] 10+ messages in thread
* [Ocfs2-devel] disk full when it shouldn't 2010-11-08 15:14 [Ocfs2-devel] disk full when it shouldn't Massimo Cetra @ 2010-11-09 1:06 ` Tao Ma 2010-12-02 17:20 ` Massimo Cetra 0 siblings, 1 reply; 10+ messages in thread From: Tao Ma @ 2010-11-09 1:06 UTC (permalink / raw) To: ocfs2-devel Hi Massimo, On 11/08/2010 11:14 PM, Massimo Cetra wrote: > Hi all, > > i've been using OCFS2 for a while (2 years or so, now) but i still have > a big problem. > > on each filesystem that i have, it happens that, even if df reports that > the disk has plenty of free space avalaible, no one can write any file. > This behaviour has happened for several usages of OCFS2 (webserver and > mail storage) so both with big files and small files. > > It happens more frequently on the mail partitions that i manage. > I have tried to search for some documentation about this problems and it > seems that i'm not the only one experiencing this behaviour. > > The kernels showing this behaviour are both 2.6.32 and 2.6.31. > This happens with OCFS2-tools 1.4.2 and 1.4.3 > > In the last weeks i have reformatted some partitions from scratch using > 1.4.3 but the problem persists. > > I have found somewhere a bash script to analyze the sysdir and output > useful informations for the debugging. > The output of this script during the problem is attached. From the output, it seems that you shouldn't meet with ENOSPC problem at that time. So when you meet with ENOSPC, could you please do 'sync' first(that will sync the metadata to the disk so that stat_sysdir can output the real metadata info) and then do stat_sysdir? btw, you can also try to 'touch' one new file in your ocfs2 volume. So if it succeeds, it means that the inode_alloc is ok. Otherwise, your inode_alloc has no space. Please also try this test when you meet with ENOSPC. btw, Regards, Tao ^ permalink raw reply [flat|nested] 10+ messages in thread
* [Ocfs2-devel] disk full when it shouldn't 2010-11-09 1:06 ` Tao Ma @ 2010-12-02 17:20 ` Massimo Cetra 2010-12-03 1:22 ` Tao Ma 0 siblings, 1 reply; 10+ messages in thread From: Massimo Cetra @ 2010-12-02 17:20 UTC (permalink / raw) To: ocfs2-devel Il 09/11/2010 02:06, Tao Ma ha scritto: > Hi Massimo, > > From the output, it seems that you shouldn't meet with ENOSPC problem > at that time. So when you meet with ENOSPC, could you please do 'sync' > first(that will sync the metadata to the disk so that stat_sysdir can > output the real metadata info) and then do stat_sysdir? > > btw, you can also try to 'touch' one new file in your ocfs2 volume. So > if it succeeds, it means that the inode_alloc is ok. Otherwise, your > inode_alloc has no space. Please also try this test when you meet with > ENOSPC. Hi Tao, As i supposed, the problem arose another time. Attached you may find the stat_sysdir output. For the sake of completeness, have to say that: - The output of df was the following: # df -i Filesystem Inode IUsati ILib. IUso% Montato su [CUT] /dev/drbd2 9174751 4695156 4479595 52% /var/mail # df Filesystem blocchi di 1K Usati Disponib. Uso% Montato su [CUT] /dev/drbd2 36699004 18780656 17918348 52% /var/mail The filesystem was not full: sheet3-1:/var/mail/virtual# dd if=/dev/zero of=TEST bs=1024 count=1024 dd: scrittura di `TEST': No space left on device 1009+0 records in 1008+0 records out 1032192 bytes (1,0 MB) copied, 0,454305 s, 2,3 MB/s sheet3-1:/var/mail/virtual# mailq from postfix had the following errors: (maildir delivery failed: error writing message: No space left on device) (maildir delivery failed: error writing message: No space left on device) (maildir delivery failed: error writing message: No space left on device) A bunch of emails coundn't be delivered. I temporarily fixed it by removing old emails and the system recovered. I had another problem on a couple of different servers today and i'm investigating to send more informations. Max -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: stat_sysdir.txt Url: http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20101202/6e3f39b6/attachment-0001.txt ^ permalink raw reply [flat|nested] 10+ messages in thread
* [Ocfs2-devel] disk full when it shouldn't 2010-12-02 17:20 ` Massimo Cetra @ 2010-12-03 1:22 ` Tao Ma 2010-12-03 2:16 ` Tao Ma 2010-12-27 14:31 ` Massimo Cetra 0 siblings, 2 replies; 10+ messages in thread From: Tao Ma @ 2010-12-03 1:22 UTC (permalink / raw) To: ocfs2-devel Hi Massimo, Massimo Cetra wrote: > Il 09/11/2010 02:06, Tao Ma ha scritto: >> Hi Massimo, >> >> From the output, it seems that you shouldn't meet with ENOSPC problem >> at that time. So when you meet with ENOSPC, could you please do 'sync' >> first(that will sync the metadata to the disk so that stat_sysdir can >> output the real metadata info) and then do stat_sysdir? >> >> btw, you can also try to 'touch' one new file in your ocfs2 volume. So >> if it succeeds, it means that the inode_alloc is ok. Otherwise, your >> inode_alloc has no space. Please also try this test when you meet with >> ENOSPC. > > Hi Tao, > > As i supposed, the problem arose another time. > > Attached you may find the stat_sysdir output. Got it. > > > For the sake of completeness, have to say that: > > - The output of df was the following: > > # df -i > Filesystem Inode IUsati ILib. IUso% Montato su > [CUT] > /dev/drbd2 9174751 4695156 4479595 52% /var/mail > > # df > Filesystem blocchi di 1K Usati Disponib. Uso% Montato su > [CUT] > /dev/drbd2 36699004 18780656 17918348 52% /var/mail I have to say that df is only an approximate way to get the real info of an ocfs2 volume. df -i use statfs(2). And since there is no easy way to calculate all the inodes without locking every node's inode allocator, it just searchs the global_bitmap and gives a rough number. > > The filesystem was not full: > > sheet3-1:/var/mail/virtual# dd if=/dev/zero of=TEST bs=1024 count=1024 > dd: scrittura di `TEST': No space left on device > 1009+0 records in > 1008+0 records out > 1032192 bytes (1,0 MB) copied, 0,454305 s, 2,3 MB/s > sheet3-1:/var/mail/virtual# So this is node2? From the stat_sysdir output you attached, inode_alloc:0001 is full. It has no free inodes and the creation will fail. So you are meeting with the problem of fragmentation. Discontig block group should fix it. Regards, Tao ^ permalink raw reply [flat|nested] 10+ messages in thread
* [Ocfs2-devel] disk full when it shouldn't 2010-12-03 1:22 ` Tao Ma @ 2010-12-03 2:16 ` Tao Ma 2010-12-27 14:31 ` Massimo Cetra 1 sibling, 0 replies; 10+ messages in thread From: Tao Ma @ 2010-12-03 2:16 UTC (permalink / raw) To: ocfs2-devel Hi Massimo, I just noticed anther thing. The inode_alloc:0000 isn't full, so node1 should steal inode from node0, why it can't? You delete some files in node0? Regards, Tao On 12/03/2010 09:22 AM, Tao Ma wrote: > Hi Massimo, > Massimo Cetra wrote: >> Il 09/11/2010 02:06, Tao Ma ha scritto: >>> Hi Massimo, >>> >>> From the output, it seems that you shouldn't meet with ENOSPC problem >>> at that time. So when you meet with ENOSPC, could you please do 'sync' >>> first(that will sync the metadata to the disk so that stat_sysdir can >>> output the real metadata info) and then do stat_sysdir? >>> >>> btw, you can also try to 'touch' one new file in your ocfs2 volume. So >>> if it succeeds, it means that the inode_alloc is ok. Otherwise, your >>> inode_alloc has no space. Please also try this test when you meet with >>> ENOSPC. >> >> Hi Tao, >> >> As i supposed, the problem arose another time. >> >> Attached you may find the stat_sysdir output. > Got it. >> >> >> For the sake of completeness, have to say that: >> >> - The output of df was the following: >> >> # df -i >> Filesystem Inode IUsati ILib. IUso% Montato su >> [CUT] >> /dev/drbd2 9174751 4695156 4479595 52% /var/mail >> >> # df >> Filesystem blocchi di 1K Usati Disponib. Uso% Montato su >> [CUT] >> /dev/drbd2 36699004 18780656 17918348 52% /var/mail > I have to say that df is only an approximate way to get the real info of > an ocfs2 volume. > df -i use statfs(2). And since there is no easy way to calculate all the > inodes without > locking every node's inode allocator, it just searchs the global_bitmap > and gives a rough number. > >> >> The filesystem was not full: >> >> sheet3-1:/var/mail/virtual# dd if=/dev/zero of=TEST bs=1024 count=1024 >> dd: scrittura di `TEST': No space left on device >> 1009+0 records in >> 1008+0 records out >> 1032192 bytes (1,0 MB) copied, 0,454305 s, 2,3 MB/s >> sheet3-1:/var/mail/virtual# > So this is node2? From the stat_sysdir output you attached, > inode_alloc:0001 is full. > It has no free inodes and the creation will fail. > > So you are meeting with the problem of fragmentation. Discontig block > group should fix it. > > Regards, > Tao > > _______________________________________________ > Ocfs2-devel mailing list > Ocfs2-devel at oss.oracle.com > http://oss.oracle.com/mailman/listinfo/ocfs2-devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* [Ocfs2-devel] disk full when it shouldn't 2010-12-03 1:22 ` Tao Ma 2010-12-03 2:16 ` Tao Ma @ 2010-12-27 14:31 ` Massimo Cetra 2010-12-27 15:48 ` Tao Ma 1 sibling, 1 reply; 10+ messages in thread From: Massimo Cetra @ 2010-12-27 14:31 UTC (permalink / raw) To: ocfs2-devel Il 03/12/2010 02:22, Tao Ma ha scritto: > I have to say that df is only an approximate way to get the real info > of an ocfs2 volume. > df -i use statfs(2). And since there is no easy way to calculate all > the inodes without > locking every node's inode allocator, it just searchs the > global_bitmap and gives a rough number. > >> >> The filesystem was not full: >> >> sheet3-1:/var/mail/virtual# dd if=/dev/zero of=TEST bs=1024 count=1024 >> dd: scrittura di `TEST': No space left on device >> 1009+0 records in >> 1008+0 records out >> 1032192 bytes (1,0 MB) copied, 0,454305 s, 2,3 MB/s >> sheet3-1:/var/mail/virtual# > So this is node2? From the stat_sysdir output you attached, > inode_alloc:0001 is full. > It has no free inodes and the creation will fail. > > So you are meeting with the problem of fragmentation. Discontig block > group should fix it. > Hi Tao, back again. I have decided to install 2.6.36.1 on all clusters to see if the fragmentation problem has gone. Unfortunately the problem persists, even if with different sympthoms. Last week i had the same enospace problem but this time it was consistent on both nodes (on both of them i couldn't even "touch" a single file". Previously i was able to touch it and sometimes to write several Mb to a single file. This time df was reporting about 49% of disk usage. I'm attaching the stat_sysdir output for both nodes. Kernel is a vanilla 2.6.36.1. Am i missing something ? Is this fragmentation ? Did i hit a bug or rhe problem is between the keyboard and the chair ? Thanks for your help. Massimo P.S.: if you could explain how to interpret the stat_sysdir output i could be less tedius and write less mails! -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 23dic-OCFS-DRBD2-s2.txt Url: http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20101227/fc6941e2/attachment-0002.txt -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 23dic-OCFS-DRBD2-s1.txt Url: http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20101227/fc6941e2/attachment-0003.txt ^ permalink raw reply [flat|nested] 10+ messages in thread
* [Ocfs2-devel] disk full when it shouldn't 2010-12-27 14:31 ` Massimo Cetra @ 2010-12-27 15:48 ` Tao Ma 2010-12-27 16:52 ` Sunil Mushran 2010-12-28 11:38 ` Massimo Cetra 0 siblings, 2 replies; 10+ messages in thread From: Tao Ma @ 2010-12-27 15:48 UTC (permalink / raw) To: ocfs2-devel Hi Massimo, On 12/27/2010 10:31 PM, Massimo Cetra wrote: > > Il 03/12/2010 02:22, Tao Ma ha scritto: >> I have to say that df is only an approximate way to get the real info >> of an ocfs2 volume. >> df -i use statfs(2). And since there is no easy way to calculate all >> the inodes without >> locking every node's inode allocator, it just searchs the >> global_bitmap and gives a rough number. >> >>> >>> The filesystem was not full: >>> >>> sheet3-1:/var/mail/virtual# dd if=/dev/zero of=TEST bs=1024 count=1024 >>> dd: scrittura di `TEST': No space left on device >>> 1009+0 records in >>> 1008+0 records out >>> 1032192 bytes (1,0 MB) copied, 0,454305 s, 2,3 MB/s >>> sheet3-1:/var/mail/virtual# >> So this is node2? From the stat_sysdir output you attached, >> inode_alloc:0001 is full. >> It has no free inodes and the creation will fail. >> >> So you are meeting with the problem of fragmentation. Discontig block >> group should fix it. >> > > Hi Tao, > > back again. > > I have decided to install 2.6.36.1 on all clusters to see if the > fragmentation problem has gone. > > Unfortunately the problem persists, even if with different sympthoms. > > Last week i had the same enospace problem but this time it was > consistent on both nodes (on both of them i couldn't even "touch" a > single file". > Previously i was able to touch it and sometimes to write several Mb to a > single file. > > This time df was reporting about 49% of disk usage. > > I'm attaching the stat_sysdir output for both nodes. Kernel is a vanilla > 2.6.36.1. > > Am i missing something ? > Is this fragmentation ? yes, it is, I have checked your attached statsysdir output. > Did i hit a bug or rhe problem is between the keyboard and the chair ? oh, it isn't that easy by just upgrading the kernel. ;) So you have to enable that feature by tunefs.ocfs2. I am afraid Oracle didn't release a new ocfs2-tools now(I had left Oracle and I don't know the release detail now), so you need to download the latest source and compile the ocfs2-tools by yourself. git clone git://oss.oracle.com/git/ocfs2-tools.git and after you compile it, just do tunefs.ocfs2 --fs-features=discontig-bg /dev/sdx. And you will enable this feature(it will require you to umount the volume among all the nodes) and enjoy it.:) > > Thanks for your help. > > Massimo > > P.S.: if you could explain how to interpret the stat_sysdir output i > could be less tedius and write less mails! ok. So you can check inode_alloc:000x which is used to create inodes. And in your case, both inode_alloc:0000 and inode_alloc:0001 is full(check the line "Bitmap" for these inode_allocs), so no new file can be created now. Regards, Tao ^ permalink raw reply [flat|nested] 10+ messages in thread
* [Ocfs2-devel] disk full when it shouldn't 2010-12-27 15:48 ` Tao Ma @ 2010-12-27 16:52 ` Sunil Mushran 2010-12-28 11:38 ` Massimo Cetra 1 sibling, 0 replies; 10+ messages in thread From: Sunil Mushran @ 2010-12-27 16:52 UTC (permalink / raw) To: ocfs2-devel On 12/27/2010 07:48 AM, Tao Ma wrote: > So you have to enable that feature by tunefs.ocfs2. I am afraid Oracle > didn't release a new ocfs2-tools now(I had left Oracle and I don't know > the release detail now), so you need to download the latest source and > compile the ocfs2-tools by yourself. > git clone git://oss.oracle.com/git/ocfs2-tools.git > and after you compile it, just do > tunefs.ocfs2 --fs-features=discontig-bg /dev/sdx. > And you will enable this feature(it will require you to umount the > volume among all the nodes) and enjoy it.:) The source tarball is available here. http://oss.oracle.com/projects/ocfs2-tools/files/source/v1.6/ Ping your distro for the packages. ^ permalink raw reply [flat|nested] 10+ messages in thread
* [Ocfs2-devel] disk full when it shouldn't 2010-12-27 15:48 ` Tao Ma 2010-12-27 16:52 ` Sunil Mushran @ 2010-12-28 11:38 ` Massimo Cetra 2010-12-28 12:06 ` Massimo Cetra 1 sibling, 1 reply; 10+ messages in thread From: Massimo Cetra @ 2010-12-28 11:38 UTC (permalink / raw) To: ocfs2-devel Il 27/12/2010 16:48, Tao Ma ha scritto: > oh, it isn't that easy by just upgrading the kernel. ;) > So you have to enable that feature by tunefs.ocfs2. I am afraid Oracle > didn't release a new ocfs2-tools now(I had left Oracle and I don't > know the release detail now), so you need to download the latest > source and compile the ocfs2-tools by yourself. > git clone git://oss.oracle.com/git/ocfs2-tools.git > and after you compile it, just do > tunefs.ocfs2 --fs-features=discontig-bg /dev/sdx. > And you will enable this feature(it will require you to umount the > volume among all the nodes) and enjoy it.:) Oh perfect. Thanks, i really didn't realize that fs_discontig was a flag. The old tools didn't report it in the docs so i didn't find it anywhere. Now: sorry to be so annoying but i have googled without success ( i only found the patch that adds the following error). I cannot find nor the reason for this error: tunefs.ocfs2: Chain allocator is corrupt while opening device "/dev/drbd2" Nor the cause, anywhere. fsck doesn't seems to find the problem. Thanks again. I hope thise questions will help other people as well. Massimo ---------------------------------------------- sheet3-1:~# tunefs.ocfs2 --fs-features=discontig-bg /dev/drbd2 tunefs.ocfs2: Chain allocator is corrupt while opening device "/dev/drbd2" sheet3-1:~# fsck.ocfs2 /dev/drbd2 fsck.ocfs2 1.6.3 Checking OCFS2 filesystem in /dev/drbd2: Label: mail-storage UUID: 7BB7636E26A7457192657E6521A5AFDC Number of blocks: 9174751 Block size: 4096 Number of clusters: 9174751 Cluster size: 4096 Number of slots: 4 /dev/drbd2 is clean. It will be checked after 20 additional mounts. ^ permalink raw reply [flat|nested] 10+ messages in thread
* [Ocfs2-devel] disk full when it shouldn't 2010-12-28 11:38 ` Massimo Cetra @ 2010-12-28 12:06 ` Massimo Cetra 0 siblings, 0 replies; 10+ messages in thread From: Massimo Cetra @ 2010-12-28 12:06 UTC (permalink / raw) To: ocfs2-devel Il 28/12/2010 12:38, Massimo Cetra ha scritto: > Oh perfect. > Thanks, i really didn't realize that fs_discontig was a flag. > The old tools didn't report it in the docs so i didn't find it anywhere. > > Now: sorry to be so annoying but i have googled without success ( i > only found the patch that adds the following error). > > I cannot find nor the reason for this error: > > tunefs.ocfs2: Chain allocator is corrupt while opening device > "/dev/drbd2" > > Nor the cause, anywhere. > fsck doesn't seems to find the problem. > Ok, silly me. fsck.ocfs2 -f device solved the problem. Thanks for your help and i hope that the proboblem won't show again. Massimo ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2010-12-28 12:06 UTC | newest] Thread overview: 10+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2010-11-08 15:14 [Ocfs2-devel] disk full when it shouldn't Massimo Cetra 2010-11-09 1:06 ` Tao Ma 2010-12-02 17:20 ` Massimo Cetra 2010-12-03 1:22 ` Tao Ma 2010-12-03 2:16 ` Tao Ma 2010-12-27 14:31 ` Massimo Cetra 2010-12-27 15:48 ` Tao Ma 2010-12-27 16:52 ` Sunil Mushran 2010-12-28 11:38 ` Massimo Cetra 2010-12-28 12:06 ` Massimo Cetra
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).