Segmentation fault in xfs_db frag, xfsprogs-3.1.8

Segmentation fault in xfs_db frag, xfsprogs-3.1.8

[Richard Ems]

Hi list,

I have a cron script running all weekends reporting me the fragmentation
of one of our backup systems.
It ran fine up to last weekend when it started reporting "Segmentation
fault".

This is on an openSUSE 12.1 64bit, running kernel 3.3.6.

I tried with xfsprogs3.1.6 and the self compiled 3.1.8 version. Both end
with segmentation fault.

# df -h /dev/sdc
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc         80T   57T   23T  72% /backup/ES-6664

# mount | grep sdc
/dev/sdc on /backup/ES-6664 type xfs
(rw,noatime,attr2,nobarrier,inode64,logbufs=8,logbsize=256k,sunit=128,swidth=3712,noquota,_netdev)
/dev/sdc on /backup/ES-6664 type xfs
(ro,relatime,attr2,nobarrier,inode64,logbufs=8,logbsize=256k,sunit=128,swidth=3712,noquota,_netdev)

The FS gets first mounted rw and then bind mounted ro on top of it.

# xfs_info /dev/sdc
meta-data=/dev/sdc               isize=512    agcount=80,
agsize=268435440 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=21240231040, imaxpct=1
         =                       sunit=16     swidth=464 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

# echo frag | xfs_db -r /dev/sdc
Segmentation fault

# echo frag | /opt/xfsprogs-3.1.8/sbin/xfs_db -r /dev/sdc
Segmentation fault


Any clues?

Thanks,
Richard


-- 
Richard Ems       mail: Richard.Ems@Cape-Horn-Eng.com

Cape Horn Engineering S.L.
C/ Dr. J.J. Dómine 1, 5º piso
46011 Valencia
Tel : +34 96 3242923 / Fax 924
http://www.cape-horn-eng.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Segmentation fault in xfs_db frag, xfsprogs-3.1.8

[Richard Ems]

On 06/06/2012 01:45 AM, Dave Chinner wrote:
> No surprise if you have a large filesystem and the filesystem is
> changing while xfs_db is running. xfs_db is not coherent with
> mounted filesytems, and it is not recommended that you use it that
> way.s xfs-db is a debugging tool, not a filesystem state reporting
> tool.

Ok, thanks, didn't know that.
I would like to monitor the fragmentation value for all my mounted XFS.
I think I read in previous list messages that also other people are
using xfs_db this way. Or is there another way to get the fragmentation
value?


I found this segmentation fault error very strange, since I have been
using "echo frag | xfs_db -r" for months already on other big
filesystems - 2 x 17 TB, 1 x 25 TB, 1 x 20 TB - and NEVER got a
segmentation fault. All 4 filessystems are mounted read-write and are in
heavy IO use, nevertheless "echo frag | xfs_db -r" runs as expected (by
me), no one xfs_db run crashed with a segmentation fault at all for
months, running every weekend.
It also ran several times without giving any errors on this 80 TB XFS,
but then started to throw segmentation faults some weeks ago.


>> # mount | grep sdc
>> /dev/sdc on /backup/ES-6664 type xfs
>> (rw,noatime,attr2,nobarrier,inode64,logbufs=8,logbsize=256k,sunit=128,swidth=3712,noquota,_netdev)
>> /dev/sdc on /backup/ES-6664 type xfs
>> (ro,relatime,attr2,nobarrier,inode64,logbufs=8,logbsize=256k,sunit=128,swidth=3712,noquota,_netdev)
>>
>> The FS gets first mounted rw and then bind mounted ro on top of it.
> 
> That doesn't mean the filesystem is mount read only. The only way to
> ensure that is to remount it read only, because the bind mount
> doesn't force file descriptors that are already open to magically
> diasappear or become read-only.

Yes, thanks! I was just trying to show how the XFS was actually mounted,
and not stating that it was mounted read-only at all. Perhaps I should
have explained it better.


>> # echo frag | /opt/xfsprogs-3.1.8/sbin/xfs_db -r /dev/sdc
>> Segmentation fault
> 
> and if you clear caches before running xfs_db again?

I will try this. I unmounted the filesystem now and I am running frag,
but I will later try to reproduce the segmentation fault and then see if
clearing caches helps.

> 
>> Any clues?
> 
> Not really, but then again you are doing something that really isn't
> guaranteed to work properly in all cases. Build xfs_db from the
> latest tree and run it without stripping the binary (i.e. run it
> from the build tree), capture the core file and get a backtrace of
> where it failed from xfs_db. more than likely it will point to
> xfs_db having encountered something that is not entirely metadata
> anymore...


Many thanks Dave for all clarifications!



-- 
Richard Ems       mail: Richard.Ems@Cape-Horn-Eng.com

Cape Horn Engineering S.L.
C/ Dr. J.J. Dómine 1, 5º piso
46011 Valencia
Tel : +34 96 3242923 / Fax 924
http://www.cape-horn-eng.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Segmentation fault in xfs_db frag, xfsprogs-3.1.8

[Dave Chinner]

On Wed, Jun 06, 2012 at 12:06:36PM +0200, Richard Ems wrote:
> On 06/06/2012 01:45 AM, Dave Chinner wrote:
> > No surprise if you have a large filesystem and the filesystem is
> > changing while xfs_db is running. xfs_db is not coherent with
> > mounted filesytems, and it is not recommended that you use it that
> > way.s xfs-db is a debugging tool, not a filesystem state reporting
> > tool.
> 
> Ok, thanks, didn't know that.
> I would like to monitor the fragmentation value for all my mounted XFS.
> I think I read in previous list messages that also other people are
> using xfs_db this way. Or is there another way to get the fragmentation
> value?

No, and what xfs_db reports is mostly useless. You don't have a
fragmentation problem unless you are noticing performance problems,
and no fragmentation number will ever tell you that....

> I found this segmentation fault error very strange, since I have been
> using "echo frag | xfs_db -r" for months already on other big
> filesystems - 2 x 17 TB, 1 x 25 TB, 1 x 20 TB - and NEVER got a
> segmentation fault. All 4 filessystems are mounted read-write and are in
> heavy IO use, nevertheless "echo frag | xfs_db -r" runs as expected (by
> me), no one xfs_db run crashed with a segmentation fault at all for
> months, running every weekend.
> It also ran several times without giving any errors on this 80 TB XFS,
> but then started to throw segmentation faults some weeks ago.

It's entirely possible it is running out of memory, failing to
capture the failure and dereferencing a null pointer....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Segmentation fault in xfs_db frag, xfsprogs-3.1.8

[Eric Sandeen]

On 6/6/12 5:06 AM, Richard Ems wrote:
> On 06/06/2012 01:45 AM, Dave Chinner wrote:
>> No surprise if you have a large filesystem and the filesystem is
>> changing while xfs_db is running. xfs_db is not coherent with
>> mounted filesytems, and it is not recommended that you use it that
>> way.s xfs-db is a debugging tool, not a filesystem state reporting
>> tool.
> 
> Ok, thanks, didn't know that.
> I would like to monitor the fragmentation value for all my mounted XFS.
> I think I read in previous list messages that also other people are
> using xfs_db this way. Or is there another way to get the fragmentation
> value?

As Dave said, this just isn't that useful.

http://xfs.org/index.php/XFS_FAQ#Q:_The_xfs_db_.22frag.22_command_says_I.27m_over_50.25.__Is_that_bad.3F

explains why.

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs