free space of root partition decreases unaccountably by some 1024 blocks on every umount+linux shutdown

free space of root partition decreases unaccountably by some 1024 blocks on every umount+linux shutdown

[Roland Eggner]

[-- Attachment #1.1: Type: text/plain, Size: 4192 bytes --]

History which lead to actual problem
------------------------------------
On July 4th I switched from kernel 2.6.29.5 to 2.6.29.6.

On July 18th I noticed the first time this unaccountable decrease of free space of my root partition /dev/hda7:
For at least several boot-shutdown-cycles it has decreased on every cycle by some 1020 … 1030 blocks from originally above 100 MB to 96 MB.
Expected change at most ±1 block.  Neither xfs_check nor xfs_repair -dn could detect any flaws.
I booted a sidux image with kernel 2.6.27, mounted /dev/hda7 readonly (actually sidux flavour “/dev/sda7”), and compared reported free space ➜ decrease seems to occur on umount + system shutdown.  Neither xfs_check nor “xfs_repair -dn” running under sidux kernel 2.6.27 could detect any flaws.
Only one other flaw is noteworthy (seperate bugreport scheduled):
Since one of the 2.6.28.? kernels mount procedures require a time randomly dithering between less than 1 second and more than 30 seconds with almost no harddisk i/o activity, with
NO difference between “mount -a” during boot and manual mounts,
NO difference between mounts of plain and loop-aes-encrypted partitions,
NO difference between kernels 2.6.29.[1-6] and 2.6.30.4.
Only loop mounts of vfat images happen without remarkable delay.
Just after xfs_check and “xfs_repair -dn” have verified a particular loop-aes-encrypted filesystem errorfree, kernel complains every time “superblock invalid” and mounts it successfully.  Kernel 2.6.30.4 shows the same crazy behavior on mounting of the same and additionally another particular loop-aes-encrypted filesystem.  Cannot say, if this extreme slow mounts are XFS specific, because I do not use any other filesystem on this linux system.

On August 10th — after reading “kernel.org/ChangeLog-2.6.30.bz2:  xfs: fix bad_features2 fixups for the root filesystem …” — I switched to kernel 2.6.30.4, which left the problem UNMODIFIED.

On August 12th free space has decreased to 48 MB, so I am forced to take actions.


System details
--------------
System based on Debian testing.
Kernel:  from kernel.org tainted by NVIDIA video driver.
Applied patches:  one from loop-aes-source and another one to fs/namespace.c setting MNT_STRICTATIME as default mount option.
smartmontools short selftest reports are errorfree.
“hdparm -W0 /dev/hda” performed by a bootscript and external encrypted logdevices are precautions to ensure consistent loop-aes-encrypted xfs filesystems — during 1 year of usage I encountered a few power failures, which caused “dirty” shutdowns of my notebook but never any filesystem related problems :)


#  /usr/sbin/xfs_info /
meta-data=/dev/root              isize=256    agcount=4, agsize=748776 blks
         =                       sectsz=512   attr=2
data     =                       bsize=1024   blocks=2995102, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=1024   ascii-ci=0
log      =internal               bsize=1024   blocks=10240, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0


$  grep /dev/root /proc/mounts
/dev/root / xfs rw,strictatime,attr2,nobarrier,noquota 0 0


At /var /tmp /home are other partitions mounted, therefore the only known write activities on root partition apart from atime updates and temporary lockfiles are concerning at most 6 blocks:
#  ( export LANGUAGE=en_GB ; find / -ctime -1 -not -type d -print0 | xargs -0 -- du -bc )
16      /etc/network/run/ifstate
44      /etc/adjtime~
44      /etc/adjtime
23      /etc/resolv.conf
1970    /etc/mtab
2097    total
No known write activities on system startup prior to mount or on shutdown after umount in directories on root partition, which are hidden by mounts when system is running.


xfs_metadump output from 2 consecutive linux sessions, lzma compressed, provided via http-server on request from known XFS developers (preferable gpg signed).
Should I perform a xfsdump|mkfs.xfs|xfsrestore-cycle?
Thanks!

-- 
Roland Eggner

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: free space of root partition decreases unaccountably by some 1024 blocks on every umount+linux shutdown

[Eric Sandeen]

Roland Eggner wrote:
> History which lead to actual problem
> ------------------------------------
> On July 4th I switched from kernel 2.6.29.5 to 2.6.29.6.
> 
> On July 18th I noticed the first time this unaccountable decrease of free space of my root partition /dev/hda7:
> For at least several boot-shutdown-cycles it has decreased on every cycle by some 1020 … 1030 blocks from originally above 100 MB to 96 MB.
> Expected change at most ±1 block.  Neither xfs_check nor xfs_repair -dn could detect any flaws.

Maybe I missed it in the email, but how have you ruled out the
possibility that files are simply growing, thereby using the space?

Have you compared

# ls -laR /

and/or

# du -hcx /

between a couple of boots?

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

free space of root partition decreases unaccountably by some 1024 blocks on every umount+linux shutdown : additional informations

[Roland Eggner]

[-- Attachment #1.1: Type: text/plain, Size: 3999 bytes --]

On Thursday August 13th 05:16:40 2009 Eric Sandeen wrote:
> Roland Eggner wrote:
> > On July 18th I noticed the first time this unaccountable decrease of free space of my root partition /dev/hda7:
> > For at least several boot-shutdown-cycles it has decreased on every cycle by some 1020 … 1030 blocks from originally above 100 MB to 96 MB.
> > Expected change at most ±1 block.  Neither xfs_check nor xfs_repair -dn could detect any flaws.
> Maybe I missed it in the email, but how have you ruled out the
> possibility that files are simply growing, thereby using the space?
Look at the last but one paragraph in my mail from August 12th
http://oss.sgi.com/archives/xfs/2009-08/msg00113.html
(obviously the find argument “-mount” got lost on copy+paste, sorry).



Facts summarized:
----------------

(a)  Free space decreases unaccountably even if I circumvent all shutdown scripts und shutdown by SysReq R S U B.  Note: no SIGTERM, no SIGKILL.

(b)  Unaccountable free space decrease is triggered EXACTLY, ALWAYS and EXCLUSIVELY by this “remount,ro”.  Note: NOT by any other write activities.

(c)  Binary comparison of images written with dd exhibited NO unexpected writes to me (I do not have deeper knowledge of xfs internals):  in a particular case free space difference reported by df has been 1026 blocks = 1050624 byte, whereas count of differing bytes reported by “cmp -b -l” has been only 135189.  “cmp -b -l” DID exhibit expected writes e.g. /etc/mtab.  (Eric Sandeen got details via private mail and offered kindly to have a look at xfs_metadump extraction from this images — thanks!).

(d)  Until now I could NOT detect this problem at any other partition, it seems that ONLY the root partition is affected.

(e)  I performed xfsdump | mkfs.xfs | xfsrestore and got back some 200 Mbyte free space, but only temporarily — at subsequent linux shutdowns free space CONTINUES to decrease, just starting from a new offset.

(f)  I booted a sidux image and reset the lazy-counter attribute by “xfs_admin -c 0” ➔ Apart from “XFS: correcting sb_features alignment problem” message at next mount, EXACTLY the same result as after measures (e).  (WARNING:  Never use “xfs_admin -c 0” unless you have a current backup of your valuable data!!)

(g)  Kernel 2.6.30.4 shows the problem too.  (And introduces some other flaws, “show stopping” for me — therefore I stay at kernel 2.6.29.6).

(h)  Apart from following single incident, I got never any error messages from this filesystem, neither from xfs_check nor from xfs_repair:
On July 17th a run of xfs_repair yielded following report — beeing busy on that day, I ignored the message apart from saving it for later analysis:
#  xfs_repair -dn /dev/hda7
bad nblocks 1 for free inode 9722
bad nlink 1 for free inode 9722
bad mode 0100644 for free inode 9722
link count mismatch for inode 9722 (name ?), nlink 0, counted 1
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.



If I can provide any additional debugging info, let me know.

-- 
Roland Eggner

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs