[Ocfs2-devel] disk full when it shouldn't

[Ocfs2-devel] disk full when it shouldn't

[Massimo Cetra]

Hi all,

i've been using OCFS2 for a while (2 years or so, now) but i still have 
a big problem.

on each filesystem that i have, it happens that, even if df reports that 
the disk has plenty of free space avalaible, no one can write any file.
This behaviour has happened for several usages of OCFS2 (webserver and 
mail storage) so both with big files and small files.

It happens more frequently on the mail partitions that i manage.
I have tried to search for some documentation about this problems and it 
seems that i'm not the only one experiencing this behaviour.

The kernels showing this behaviour are both 2.6.32 and 2.6.31.
This happens with OCFS2-tools 1.4.2 and 1.4.3

In the last weeks i have reformatted some partitions from scratch using 
1.4.3 but the problem persists.

I have found somewhere a bash script to analyze the sysdir and output 
useful informations for the debugging.
The output of this script during the problem is attached.

Any hint about this ?

Thanks,

Max


-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: debugfs.txt
Url: http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20101108/6a59bcae/attachment-0001.txt 

[Ocfs2-devel] disk full when it shouldn't

[Tao Ma]

Hi Massimo,

On 11/08/2010 11:14 PM, Massimo Cetra wrote:
> Hi all,
>
> i've been using OCFS2 for a while (2 years or so, now) but i still have
> a big problem.
>
> on each filesystem that i have, it happens that, even if df reports that
> the disk has plenty of free space avalaible, no one can write any file.
> This behaviour has happened for several usages of OCFS2 (webserver and
> mail storage) so both with big files and small files.
>
> It happens more frequently on the mail partitions that i manage.
> I have tried to search for some documentation about this problems and it
> seems that i'm not the only one experiencing this behaviour.
>
> The kernels showing this behaviour are both 2.6.32 and 2.6.31.
> This happens with OCFS2-tools 1.4.2 and 1.4.3
>
> In the last weeks i have reformatted some partitions from scratch using
> 1.4.3 but the problem persists.
>
> I have found somewhere a bash script to analyze the sysdir and output
> useful informations for the debugging.
> The output of this script during the problem is attached.
 From the output, it seems that you shouldn't meet with ENOSPC problem 
at that time. So when you meet with ENOSPC, could you please do 'sync' 
first(that will sync the metadata to the disk so that stat_sysdir can 
output the real metadata info) and then do stat_sysdir?

btw, you can also try to 'touch' one new file in your ocfs2 volume. So 
if it succeeds, it means that the inode_alloc is ok. Otherwise, your 
inode_alloc has no space. Please also try this test when you meet with 
ENOSPC.

btw,

Regards,
Tao

[Ocfs2-devel] disk full when it shouldn't

[Massimo Cetra]

Il 09/11/2010 02:06, Tao Ma ha scritto:
> Hi Massimo,
>
> From the output, it seems that you shouldn't meet with ENOSPC problem
> at that time. So when you meet with ENOSPC, could you please do 'sync'
> first(that will sync the metadata to the disk so that stat_sysdir can
> output the real metadata info) and then do stat_sysdir?
>
> btw, you can also try to 'touch' one new file in your ocfs2 volume. So
> if it succeeds, it means that the inode_alloc is ok. Otherwise, your
> inode_alloc has no space. Please also try this test when you meet with
> ENOSPC.

Hi Tao,

As i supposed, the problem arose another time.

Attached you may find the stat_sysdir output.

For the sake of completeness, have to say that:

- The output of df was the following:

# df -i
Filesystem             Inode   IUsati  ILib. IUso% Montato su
[CUT]
/dev/drbd2           9174751 4695156 4479595   52% /var/mail

# df
Filesystem        blocchi di   1K   Usati Disponib. Uso% Montato su
[CUT]
/dev/drbd2            36699004  18780656  17918348  52% /var/mail

The filesystem was not full:

sheet3-1:/var/mail/virtual# dd if=/dev/zero of=TEST bs=1024 count=1024
dd: scrittura di `TEST': No space left on device
1009+0 records in
1008+0 records out
1032192 bytes (1,0 MB) copied, 0,454305 s, 2,3 MB/s
sheet3-1:/var/mail/virtual#


mailq from postfix had the following errors:

       (maildir delivery failed: error writing message: No space left on 
device)
       (maildir delivery failed: error writing message: No space left on 
device)
       (maildir delivery failed: error writing message: No space left on 
device)

A bunch of emails coundn't be delivered.

I temporarily fixed it by removing old emails and the system recovered.

I had another problem on a couple of different servers today and i'm 
investigating to send more informations.

Max




-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: stat_sysdir.txt
Url: http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20101202/6e3f39b6/attachment-0001.txt 

[Ocfs2-devel] disk full when it shouldn't

[Tao Ma]

Hi Massimo,
Massimo Cetra wrote:
> Il 09/11/2010 02:06, Tao Ma ha scritto:
>> Hi Massimo,
>>
>> From the output, it seems that you shouldn't meet with ENOSPC problem
>> at that time. So when you meet with ENOSPC, could you please do 'sync'
>> first(that will sync the metadata to the disk so that stat_sysdir can
>> output the real metadata info) and then do stat_sysdir?
>>
>> btw, you can also try to 'touch' one new file in your ocfs2 volume. So
>> if it succeeds, it means that the inode_alloc is ok. Otherwise, your
>> inode_alloc has no space. Please also try this test when you meet with
>> ENOSPC.
>
> Hi Tao,
>
> As i supposed, the problem arose another time.
>
> Attached you may find the stat_sysdir output.
Got it.
>
>
> For the sake of completeness, have to say that:
>
> - The output of df was the following:
>
> # df -i
> Filesystem             Inode   IUsati  ILib. IUso% Montato su
> [CUT]
> /dev/drbd2           9174751 4695156 4479595   52% /var/mail
>
> # df
> Filesystem        blocchi di   1K   Usati Disponib. Uso% Montato su
> [CUT]
> /dev/drbd2            36699004  18780656  17918348  52% /var/mail
I have to say that df is only an approximate way to get the real info of 
an ocfs2 volume.
df -i use statfs(2). And since there is no easy way to calculate all the 
inodes without
locking every node's inode allocator, it just searchs the global_bitmap 
and gives a rough number.

>
> The filesystem was not full:
>
> sheet3-1:/var/mail/virtual# dd if=/dev/zero of=TEST bs=1024 count=1024
> dd: scrittura di `TEST': No space left on device
> 1009+0 records in
> 1008+0 records out
> 1032192 bytes (1,0 MB) copied, 0,454305 s, 2,3 MB/s
> sheet3-1:/var/mail/virtual#
So this is node2? From the stat_sysdir output you attached, 
inode_alloc:0001 is full.
It has no free inodes and the creation will fail.

So you are meeting with the problem of fragmentation. Discontig block 
group should fix it.

Regards,
Tao

[Ocfs2-devel] disk full when it shouldn't

[Tao Ma]

Hi Massimo,

I just noticed anther thing.
The inode_alloc:0000 isn't full, so node1 should steal inode from node0, 
why it can't? You delete some files in node0?

Regards,
Tao

On 12/03/2010 09:22 AM, Tao Ma wrote:
> Hi Massimo,
> Massimo Cetra wrote:
>> Il 09/11/2010 02:06, Tao Ma ha scritto:
>>> Hi Massimo,
>>>
>>>  From the output, it seems that you shouldn't meet with ENOSPC problem
>>> at that time. So when you meet with ENOSPC, could you please do 'sync'
>>> first(that will sync the metadata to the disk so that stat_sysdir can
>>> output the real metadata info) and then do stat_sysdir?
>>>
>>> btw, you can also try to 'touch' one new file in your ocfs2 volume. So
>>> if it succeeds, it means that the inode_alloc is ok. Otherwise, your
>>> inode_alloc has no space. Please also try this test when you meet with
>>> ENOSPC.
>>
>> Hi Tao,
>>
>> As i supposed, the problem arose another time.
>>
>> Attached you may find the stat_sysdir output.
> Got it.
>>
>>
>> For the sake of completeness, have to say that:
>>
>> - The output of df was the following:
>>
>> # df -i
>> Filesystem             Inode   IUsati  ILib. IUso% Montato su
>> [CUT]
>> /dev/drbd2           9174751 4695156 4479595   52% /var/mail
>>
>> # df
>> Filesystem        blocchi di   1K   Usati Disponib. Uso% Montato su
>> [CUT]
>> /dev/drbd2            36699004  18780656  17918348  52% /var/mail
> I have to say that df is only an approximate way to get the real info of
> an ocfs2 volume.
> df -i use statfs(2). And since there is no easy way to calculate all the
> inodes without
> locking every node's inode allocator, it just searchs the global_bitmap
> and gives a rough number.
>
>>
>> The filesystem was not full:
>>
>> sheet3-1:/var/mail/virtual# dd if=/dev/zero of=TEST bs=1024 count=1024
>> dd: scrittura di `TEST': No space left on device
>> 1009+0 records in
>> 1008+0 records out
>> 1032192 bytes (1,0 MB) copied, 0,454305 s, 2,3 MB/s
>> sheet3-1:/var/mail/virtual#
> So this is node2? From the stat_sysdir output you attached,
> inode_alloc:0001 is full.
> It has no free inodes and the creation will fail.
>
> So you are meeting with the problem of fragmentation. Discontig block
> group should fix it.
>
> Regards,
> Tao
>
> _______________________________________________
> Ocfs2-devel mailing list
> Ocfs2-devel at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-devel

[Ocfs2-devel] disk full when it shouldn't

[Massimo Cetra]

Il 03/12/2010 02:22, Tao Ma ha scritto:
> I have to say that df is only an approximate way to get the real info
> of an ocfs2 volume.
> df -i use statfs(2). And since there is no easy way to calculate all
> the inodes without
> locking every node's inode allocator, it just searchs the
> global_bitmap and gives a rough number.
>
>>
>> The filesystem was not full:
>>
>> sheet3-1:/var/mail/virtual# dd if=/dev/zero of=TEST bs=1024 count=1024
>> dd: scrittura di `TEST': No space left on device
>> 1009+0 records in
>> 1008+0 records out
>> 1032192 bytes (1,0 MB) copied, 0,454305 s, 2,3 MB/s
>> sheet3-1:/var/mail/virtual#
> So this is node2? From the stat_sysdir output you attached,
> inode_alloc:0001 is full.
> It has no free inodes and the creation will fail.
>
> So you are meeting with the problem of fragmentation. Discontig block
> group should fix it.
>

Hi Tao,

back again.

I have decided to install 2.6.36.1 on all clusters to see if the 
fragmentation problem has gone.

Unfortunately the problem persists, even if with different sympthoms.

Last week i had the same enospace problem but this time it was 
consistent on both nodes (on both of them i couldn't even "touch" a 
single file".
Previously i was able to touch it and sometimes to write several Mb to a 
single file.

This time df was reporting about 49% of disk usage.

I'm attaching the stat_sysdir output for both nodes. Kernel is a vanilla 
2.6.36.1.

Am i missing something ?
Is this fragmentation ?
Did i hit a bug or rhe problem is between the keyboard and the chair ?

Thanks for your help.

Massimo

P.S.: if you could explain how to interpret the stat_sysdir output i 
could be less tedius and write less mails!


-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 23dic-OCFS-DRBD2-s2.txt
Url: http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20101227/fc6941e2/attachment-0002.txt 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 23dic-OCFS-DRBD2-s1.txt
Url: http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20101227/fc6941e2/attachment-0003.txt 

[Ocfs2-devel] disk full when it shouldn't

[Tao Ma]

Hi Massimo,
On 12/27/2010 10:31 PM, Massimo Cetra wrote:
>
> Il 03/12/2010 02:22, Tao Ma ha scritto:
>> I have to say that df is only an approximate way to get the real info
>> of an ocfs2 volume.
>> df -i use statfs(2). And since there is no easy way to calculate all
>> the inodes without
>> locking every node's inode allocator, it just searchs the
>> global_bitmap and gives a rough number.
>>
>>>
>>> The filesystem was not full:
>>>
>>> sheet3-1:/var/mail/virtual# dd if=/dev/zero of=TEST bs=1024 count=1024
>>> dd: scrittura di `TEST': No space left on device
>>> 1009+0 records in
>>> 1008+0 records out
>>> 1032192 bytes (1,0 MB) copied, 0,454305 s, 2,3 MB/s
>>> sheet3-1:/var/mail/virtual#
>> So this is node2? From the stat_sysdir output you attached,
>> inode_alloc:0001 is full.
>> It has no free inodes and the creation will fail.
>>
>> So you are meeting with the problem of fragmentation. Discontig block
>> group should fix it.
>>
>
> Hi Tao,
>
> back again.
>
  > I have decided to install 2.6.36.1 on all clusters to see if the
> fragmentation problem has gone.
>
> Unfortunately the problem persists, even if with different sympthoms.
>
> Last week i had the same enospace problem but this time it was
> consistent on both nodes (on both of them i couldn't even "touch" a
> single file".
> Previously i was able to touch it and sometimes to write several Mb to a
> single file.
>
> This time df was reporting about 49% of disk usage.
>
> I'm attaching the stat_sysdir output for both nodes. Kernel is a vanilla
> 2.6.36.1.
>
> Am i missing something ?
> Is this fragmentation ?
yes, it is, I have checked your attached statsysdir output.
> Did i hit a bug or rhe problem is between the keyboard and the chair ?
oh, it isn't that easy by just upgrading the kernel. ;)
So you have to enable that feature by tunefs.ocfs2. I am afraid Oracle 
didn't release a new ocfs2-tools now(I had left Oracle and I don't know 
the release detail now), so you need to download the latest source and 
compile the ocfs2-tools by yourself.
git clone git://oss.oracle.com/git/ocfs2-tools.git
and after you compile it, just do
tunefs.ocfs2 --fs-features=discontig-bg /dev/sdx.
And you will enable this feature(it will require you to umount the 
volume among all the nodes) and enjoy it.:)
>
> Thanks for your help.
>
> Massimo
>
> P.S.: if you could explain how to interpret the stat_sysdir output i
> could be less tedius and write less mails!
ok. So you can check inode_alloc:000x which is used to create inodes. 
And in your case, both inode_alloc:0000 and inode_alloc:0001 is 
full(check the line "Bitmap" for these inode_allocs), so no new file can 
be created now.

Regards,
Tao

[Ocfs2-devel] disk full when it shouldn't

[Sunil Mushran]

On 12/27/2010 07:48 AM, Tao Ma wrote:
> So you have to enable that feature by tunefs.ocfs2. I am afraid Oracle
> didn't release a new ocfs2-tools now(I had left Oracle and I don't know
> the release detail now), so you need to download the latest source and
> compile the ocfs2-tools by yourself.
> git clone git://oss.oracle.com/git/ocfs2-tools.git
> and after you compile it, just do
> tunefs.ocfs2 --fs-features=discontig-bg /dev/sdx.
> And you will enable this feature(it will require you to umount the
> volume among all the nodes) and enjoy it.:)

The source tarball is available here.

http://oss.oracle.com/projects/ocfs2-tools/files/source/v1.6/

Ping your distro for the packages.

[Ocfs2-devel] disk full when it shouldn't

[Massimo Cetra]

Il 27/12/2010 16:48, Tao Ma ha scritto:
> oh, it isn't that easy by just upgrading the kernel. ;)
> So you have to enable that feature by tunefs.ocfs2. I am afraid Oracle 
> didn't release a new ocfs2-tools now(I had left Oracle and I don't 
> know the release detail now), so you need to download the latest 
> source and compile the ocfs2-tools by yourself.
> git clone git://oss.oracle.com/git/ocfs2-tools.git
> and after you compile it, just do
> tunefs.ocfs2 --fs-features=discontig-bg /dev/sdx.
> And you will enable this feature(it will require you to umount the 
> volume among all the nodes) and enjoy it.:)

Oh perfect.
Thanks, i really didn't realize that fs_discontig was a flag.
The old tools didn't report it in the docs so i didn't find it anywhere.

Now: sorry to be so annoying but i have googled without success ( i only 
found the patch that adds the following error).

I cannot find nor the reason for this error:

tunefs.ocfs2: Chain allocator is corrupt while opening device "/dev/drbd2"

Nor the cause, anywhere.
fsck doesn't seems to find the problem.

Thanks again. I hope thise questions will help other people as well.

Massimo


----------------------------------------------

sheet3-1:~# tunefs.ocfs2 --fs-features=discontig-bg /dev/drbd2
tunefs.ocfs2: Chain allocator is corrupt while opening device "/dev/drbd2"

sheet3-1:~# fsck.ocfs2 /dev/drbd2
fsck.ocfs2 1.6.3
Checking OCFS2 filesystem in /dev/drbd2:
   Label:              mail-storage
   UUID:               7BB7636E26A7457192657E6521A5AFDC
   Number of blocks:   9174751
   Block size:         4096
   Number of clusters: 9174751
   Cluster size:       4096
   Number of slots:    4

/dev/drbd2 is clean.  It will be checked after 20 additional mounts.

[Ocfs2-devel] disk full when it shouldn't

[Massimo Cetra]

Il 28/12/2010 12:38, Massimo Cetra ha scritto:

> Oh perfect.
> Thanks, i really didn't realize that fs_discontig was a flag.
> The old tools didn't report it in the docs so i didn't find it anywhere.
>
> Now: sorry to be so annoying but i have googled without success ( i
> only found the patch that adds the following error).
>
> I cannot find nor the reason for this error:
>
> tunefs.ocfs2: Chain allocator is corrupt while opening device
> "/dev/drbd2"
>
> Nor the cause, anywhere.
> fsck doesn't seems to find the problem.
>
Ok, silly me.

fsck.ocfs2 -f device solved the problem.

Thanks for your help and i hope that the proboblem won't show again.

Massimo