[linux-lvm] difference in space used

[linux-lvm] difference in space used

[Greg Hanson]

Hi all,

Very new to LVM, running into the following problem: We have a logical 
volume /dev/mapper/vg0-lv2 mounted on /log. While the disk usage for the 
/log shows only about 23 megs, running  df for the logical volume is 
showing over 1 gig. 

The logical volume was previously set to 6 gigs, and when the /log mount 
point only had 123 megs, the LV was full.

We redirected the output from /log to another location, unmounted 
/dev/mapper/vg0-lv2, ran fsck, extended the LV to 12 gigs and remounted.

But now the system is filling up again quickly as shown below.
 
 [root@newbox1 ~]# df -T
Filesystem    Type   1K-blocks      Used Available Use% Mounted on
/dev/sda1     ext3    12389324   7195644   4564336  62% /
none         tmpfs     1037404         0   1037404   0% /dev/shm
/dev/mapper/vg0-lv0
              ext3    15118728   1890900  12459828  14% /var/lib
/dev/mapper/vg0-lv1
              ext3     5039616    728048   4055568  16% /home
10.202.2.100:/share1
               nfs   352982176 125698112 226931088  36% /snap
/dev/mapper/vg0-lv4
              ext3    71523868  25955308  41935408  39% /backup
/dev/mapper/vg0-lv2
              ext3    12385456   1411556  10470584  12% /log


[root@newbox1 ~]# df /log/
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/vg0-lv2   12385456   1391740  10490400  12% /log

[root@newbox1 ~]# du /log/
16      /log/lost+found
23696   /log/httpd
23716   /log/

[root@newbox1 ~]# vgdisplay
  --- Volume group ---
  VG Name               vg0
  System ID            
  Format                lvm2
  Metadata Areas        10
  Metadata Sequence No  18
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                4
  Open LV               4
  Max PV                0
  Cur PV                10
  Act PV                10
  VG Size               107.51 GB
  PE Size               4.00 MB
  Total PE              27522
  Alloc PE / Size       25812 / 100.83 GB
  Free  PE / Size       1710 / 6.68 GB
  VG UUID               2WzghL-C7Nf-17oy-ISA9-hZu4-Bhtc-WvHF8Z

[root@newbox1 ~]# lvdisplay /dev/vg0/lv2
  --- Logical volume ---
  LV Name                /dev/vg0/lv2
  VG Name                vg0
  LV UUID                DHb2dZ-zYpx-bipG-rjmL-CxBu-q0At-1oOBJB
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                12.00 GB
  Current LE             3072
  Segments               2
  Allocation             inherit
  Read ahead sectors     0
  Block device           253:2

Any ideas as to what could be going on?

Thank you,

Greg

-- 
Greg Hanson
1506 Gilbert Ave
Coeur d'Alene, ID 83815

email		greg@valuemedia.com
Phone		208-667-2442
Toll Free	800-949-1889
Fax		775-256-2231

Re: [linux-lvm] difference in space used

[Ross Vandegrift]

On Fri, May 18, 2007 at 03:04:35PM -0700, Greg Hanson wrote:
> [root@newbox1 ~]# du /log/
> 16      /log/lost+found
> 23696   /log/httpd
> 23716   /log/
> 
> Any ideas as to what could be going on?

An unlinked file isn't actually deleted until all file handles to it
are closed.  Since this is your log partition, you probably have
broken log rotation script.  I bet its moving the files to a new name,
but the logging process isn't closing and reopening the logfile.

These files that are unlinked but still have open handles to them
continue to consume their usual amount of disk space.  FWIW, I have
seen this issue particularly noticably with MySQL when the query log
is enabled, but that's a wild guess...

-- 
Ross Vandegrift
ross@kallisti.us

"The good Christian should beware of mathematicians, and all those who
make empty prophecies. The danger already exists that the mathematicians
have made a covenant with the devil to darken the spirit and to confine
man in the bonds of Hell."
	--St. Augustine, De Genesi ad Litteram, Book II, xviii, 37

Re: [linux-lvm] difference in space used

[Greg Hanson]

[-- Attachment #1: Type: text/plain, Size: 1565 bytes --]

Ross Vandegrift wrote:
> On Fri, May 18, 2007 at 03:04:35PM -0700, Greg Hanson wrote:
>   
>> [root@newbox1 ~]# du /log/
>> 16      /log/lost+found
>> 23696   /log/httpd
>> 23716   /log/
>>
>> Any ideas as to what could be going on?
>>     
>
> An unlinked file isn't actually deleted until all file handles to it
> are closed.  Since this is your log partition, you probably have
> broken log rotation script.  I bet its moving the files to a new name,
> but the logging process isn't closing and reopening the logfile.
>
> These files that are unlinked but still have open handles to them
> continue to consume their usual amount of disk space.  FWIW, I have
> seen this issue particularly noticably with MySQL when the query log
> is enabled, but that's a wild guess...
>
>   
Thanks for the input Ross.  You are exactly on target.  I found another
post on another forum with regards to this,

http://lists.debian.org/debian-isp/2002/05/msg00046.html

and started examining the log rotate scripts. There was an error in the
path in the HUP line to restart apache, and there were many deleted log
files laying around.

this command

lsof | grep deleted

confirmed that there were several occurrences of error and access logs hanging in there.  
Did an apachectl stop and start and things went back to normal.

Thank you so much for the response!


Greg

-- 
Greg Hanson
Interchange Consulting
Perusion
1506 E Gilbert Ave
Coeur d'Alene, ID 83815

Email		greg@perusion.com
Phone		208-667-2442
Toll Free	800-949-1889
Fax		775-256-2231
Web		http://www.perusion.com



[-- Attachment #2: Type: text/html, Size: 2281 bytes --]