disk runs full

disk runs full

[Alexander Gruber]

Hi together,

since several weeks the root partition (of course reiserfs, v3.6)  of 
our webserver runs slowly full although there are not enough files on 
the disk to fill up all the space. After a reboot everything is fine again!
I already searched google and the mail archive but the only hint i found 
was about open files but lsof shows not even one opened file on the root 
partition. Of course /var /home etc. are on different partitions.

Please help me!

Regards

Alex

Re: disk runs full

[Vladimir Saveliev]

Hello

On Thu, 2005-05-12 at 02:15, Alexander Gruber wrote:
> Hi together,
> 
> since several weeks the root partition (of course reiserfs, v3.6)  of 
> our webserver runs slowly full although there are not enough files on 
> the disk to fill up all the space. After a reboot everything is fine again!

Do I understand corectly that after some time of webserver work free
space on root filesystem runs out and comes back on reboot?
How did you check that there are not enough files to fill up all the
space?

> I already searched google and the mail archive but the only hint i found 
> was about open files but lsof shows not even one opened file on the root 
> partition. Of course /var /home etc. are on different partitions.
> 
> Please help me!
> 
> Regards
> 
> Alex
> 

Re: disk runs full

[Alexander Gruber]

I checked it with du -sh * on the root partition and the result was much 
smaller than the used space reported by df.

Vladimir Saveliev wrote:

>Hello
>
>On Thu, 2005-05-12 at 02:15, Alexander Gruber wrote:
>  
>
>>Hi together,
>>
>>since several weeks the root partition (of course reiserfs, v3.6)  of 
>>our webserver runs slowly full although there are not enough files on 
>>the disk to fill up all the space. After a reboot everything is fine again!
>>    
>>
>
>Do I understand corectly that after some time of webserver work free
>space on root filesystem runs out and comes back on reboot?
>How did you check that there are not enough files to fill up all the
>space?
>
>  
>
>>I already searched google and the mail archive but the only hint i found 
>>was about open files but lsof shows not even one opened file on the root 
>>partition. Of course /var /home etc. are on different partitions.
>>
>>Please help me!
>>
>>Regards
>>
>>Alex
>>
>>    
>>
>
>  
>

Re: disk runs full

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1350 bytes --]

On Thu, 12 May 2005 13:50:31 +0200, Alexander Gruber said:
> I checked it with du -sh * on the root partition and the result was much 
> smaller than the used space reported by df.

Note that temporary files are often creat()ed and then unlink()ed, leaving
the open file descriptor as the last reference.  You should probably run
'lsof' or similar tool.  On my laptop at the moment:

lsof -n | grep dele
cardmgr    2207       root    3u   CHR      254,0                5556 /dev/cm-2123-2 (deleted)
cardmgr    2207       root    4u   CHR      254,1                5559 /dev/cm-2123-5 (deleted)
cardmgr    2207       root    5u   CHR      254,2                5562 /dev/cm-2123-8 (deleted)
exmh       7142     valdis   10u   REG       0,16        0      74035 /tmp/tclfG25oV (deleted)
gconfd-2   7805     valdis   13wW  REG       0,16      641      44735 /tmp/gconfd-valdis/lock/0t1115905590ut151063u967p7805r252866408k3219173544 (deleted)
aspell     9481     valdis    2u   REG       0,16        0      74035 /tmp/tclfG25oV (deleted)

So there's 2 open but unlinked files on /tmp, and du and df will show up different
values. (Note that exmh did an open() of a file, unlinked it, and then passed
the open file descriptor to aspell as stdin - so that space will be reclaimed
once *both* of those processes have done a close() on the file descriptor).

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: disk runs full

[Alexander Gruber]

[-- Attachment #1: Type: text/plain, Size: 1683 bytes --]

I already thought of this too but the output doesn´t look like as if 
there are over 3GB not closed yet. Before the reboot i did a few minutes 
ago only /dev/null was opened several times. I would say about 30-40 
instances.


Valdis.Kletnieks@vt.edu wrote:

>On Thu, 12 May 2005 13:50:31 +0200, Alexander Gruber said:
>  
>
>>I checked it with du -sh * on the root partition and the result was much 
>>smaller than the used space reported by df.
>>    
>>
>
>Note that temporary files are often creat()ed and then unlink()ed, leaving
>the open file descriptor as the last reference.  You should probably run
>'lsof' or similar tool.  On my laptop at the moment:
>
>lsof -n | grep dele
>cardmgr    2207       root    3u   CHR      254,0                5556 /dev/cm-2123-2 (deleted)
>cardmgr    2207       root    4u   CHR      254,1                5559 /dev/cm-2123-5 (deleted)
>cardmgr    2207       root    5u   CHR      254,2                5562 /dev/cm-2123-8 (deleted)
>exmh       7142     valdis   10u   REG       0,16        0      74035 /tmp/tclfG25oV (deleted)
>gconfd-2   7805     valdis   13wW  REG       0,16      641      44735 /tmp/gconfd-valdis/lock/0t1115905590ut151063u967p7805r252866408k3219173544 (deleted)
>aspell     9481     valdis    2u   REG       0,16        0      74035 /tmp/tclfG25oV (deleted)
>
>So there's 2 open but unlinked files on /tmp, and du and df will show up different
>values. (Note that exmh did an open() of a file, unlinked it, and then passed
>the open file descriptor to aspell as stdin - so that space will be reclaimed
>once *both* of those processes have done a close() on the file descriptor).
>  
>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 3158 bytes --]