* XFS, empty files after a crash
@ 2012-02-21 12:01 kfx
2012-02-21 16:25 ` Peter Grandi
` (2 more replies)
0 siblings, 3 replies; 11+ messages in thread
From: kfx @ 2012-02-21 12:01 UTC (permalink / raw)
To: xfs
Hi,
After a crash, a lot of files on a xfs file system report an empty size
with "ls -a" but not with "du".
xfs_check and xfs_repair don't report any problem to repair (although an
internal log is used) :(
So if I understand correctly, those empty files actually use space on
the disk but are incorrectly reported as empty by the system.
Is there a way to copy back the used blocks to recover the files ?
Example for a previously not empty file:
# ls -l myfile
-rw-r--r-- 1 me me 0 2012-01-24 15:29 myfile
# du myfile
27460 myfile
# du --apparent-size myfile
0 myfile
# xfs_bmap myfile
myfile: no extents
System: Ubuntu 11.04 2.6.38-8-server x86_64
xfs_info /dev/sdc1
meta-data=/dev/sdc1 isize=256 agcount=10,
agsize=268435328 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=2441215488, imaxpct=5
= sunit=128 swidth=640 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Thank you
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: XFS, empty files after a crash
2012-02-21 12:01 XFS, empty files after a crash kfx
@ 2012-02-21 16:25 ` Peter Grandi
2012-02-22 3:18 ` Eric Sandeen
2012-02-21 17:39 ` Stan Hoeppner
2012-02-27 1:04 ` Dave Chinner
2 siblings, 1 reply; 11+ messages in thread
From: Peter Grandi @ 2012-02-21 16:25 UTC (permalink / raw)
To: Linux fs XFS
> Hi, After a crash, a lot of files on a xfs file system report
> an empty size with "ls -a" but not with "du". [ ... ]
It is a FAQ:
http://xfs.org/index.php/XFS_FAQ#Q:_Why_do_I_see_binary_NULLS_in_some_files_after_recovery_when_I_unplugged_the_power.3F
Usually it is a good idea to do a web search before reporting a
problem.
> Is there a way to copy back the used blocks to recover the
> files ?
No, because your applications told XFS that the content of those
blocks could be thrown away in case of crash.
It is your responsibility to ensure that your applications use
'fsync' or at least 'fsyncdata' when data should be stored
permanently before a crash occurs.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: XFS, empty files after a crash
2012-02-21 16:25 ` Peter Grandi
@ 2012-02-22 3:18 ` Eric Sandeen
2012-02-23 19:38 ` Nathaniel W. Turner
0 siblings, 1 reply; 11+ messages in thread
From: Eric Sandeen @ 2012-02-22 3:18 UTC (permalink / raw)
To: Peter Grandi; +Cc: Linux fs XFS
On 2/21/12 10:25 AM, Peter Grandi wrote:
>> Hi, After a crash, a lot of files on a xfs file system report
>> an empty size with "ls -a" but not with "du". [ ... ]
>
> It is a FAQ:
>
> http://xfs.org/index.php/XFS_FAQ#Q:_Why_do_I_see_binary_NULLS_in_some_files_after_recovery_when_I_unplugged_the_power.3F
>
> Usually it is a good idea to do a web search before reporting a
> problem.
He seems to be reporting a different problem.
In the FAQ case above, one would get a file but "it has is a size but no extents " - i.e. ls -l would show 1M, but du would show 0, and no extents are allocated.
In this case, he has no size, blocks are reported used, but no extents are allocated. That actually seems like a new/odd case.
>> Is there a way to copy back the used blocks to recover the
>> files ?
>
> No, because your applications told XFS that the content of those
> blocks could be thrown away in case of crash.
> It is your responsibility to ensure that your applications use
> 'fsync' or at least 'fsyncdata' when data should be stored
> permanently before a crash occurs.
I'd agree that there is likely nothing to recover; there are no extents allocated. I'm not sure why du is reporting space used though.
-Eric
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: XFS, empty files after a crash
2012-02-22 3:18 ` Eric Sandeen
@ 2012-02-23 19:38 ` Nathaniel W. Turner
2012-02-23 20:07 ` kadafax
0 siblings, 1 reply; 11+ messages in thread
From: Nathaniel W. Turner @ 2012-02-23 19:38 UTC (permalink / raw)
To: xfs
On 02/21/2012 10:18 PM, Eric Sandeen wrote:
> I'd agree that there is likely nothing to recover; there are no extents
> allocated. I'm not sure why du is reporting space used though.
I have noticed that xfs_bmap (which uses the ioctl bmap interface) will not
report extents after EOF. There probably *are* extents allocated to this file,
but they are not being reported by xfs_bmap.
kfx, try getting the inode number of the file (via stat or ls -i) and then
doing something like this:
xfs_db -r $DEV -c "inode $INO" -c "bmap"
This will dump the entire extent map, and I suspect you'll see at least one extent.
nate
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: XFS, empty files after a crash
2012-02-23 19:38 ` Nathaniel W. Turner
@ 2012-02-23 20:07 ` kadafax
2012-02-23 22:15 ` Nathaniel W. Turner
0 siblings, 1 reply; 11+ messages in thread
From: kadafax @ 2012-02-23 20:07 UTC (permalink / raw)
To: xfs
> kfx, try getting the inode number of the file (via stat or ls -i) and
> then doing something like this:
>
> xfs_db -r $DEV -c "inode $INO" -c "bmap"
# xfs_db -r /dev/sdc1 -c "inode 114748" -c "bmap"
data offset 0 startblock 1881705728 (7/2657536) count 6460 flag 0
# xfs_db -r /dev/sdc1 -c "inode 114754" -c "bmap"
data offset 0 startblock 1077794560 (4/4052736) count 6582 flag 0
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: XFS, empty files after a crash
2012-02-23 20:07 ` kadafax
@ 2012-02-23 22:15 ` Nathaniel W. Turner
2012-02-24 10:21 ` kadafax
0 siblings, 1 reply; 11+ messages in thread
From: Nathaniel W. Turner @ 2012-02-23 22:15 UTC (permalink / raw)
To: xfs
On 02/23/2012 03:07 PM, kadafax@gmail.com wrote:
>
>> kfx, try getting the inode number of the file (via stat or ls -i) and then
>> doing something like this:
>>
>> xfs_db -r $DEV -c "inode $INO" -c "bmap"
>
>
> # xfs_db -r /dev/sdc1 -c "inode 114748" -c "bmap"
> data offset 0 startblock 1881705728 (7/2657536) count 6460 flag 0
>
> # xfs_db -r /dev/sdc1 -c "inode 114754" -c "bmap"
> data offset 0 startblock 1077794560 (4/4052736) count 6582 flag 0
If you want to see what's behind those data extents (which are probably
partially written), you could do something along these lines:
# Determine the AG size
agblocks=$(xfs_db -r /dev/sdc1 -c sb -c p | grep ^agblocks | sed 's/.* = //')
# Copy the extent in the first file, which consists of 6460 blocks (~26MB)
# in AG 7 starting at AG-relative block 2657536:
dd if=/dev/sdc1 bs=4096 skip=$(($agblocks * 7 + 2657536)) count=6460 of=./blob
# examine ./blob
...
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: XFS, empty files after a crash
2012-02-23 22:15 ` Nathaniel W. Turner
@ 2012-02-24 10:21 ` kadafax
0 siblings, 0 replies; 11+ messages in thread
From: kadafax @ 2012-02-24 10:21 UTC (permalink / raw)
To: xfs
Le 23/02/12 23:15, Nathaniel W. Turner a écrit :
> On 02/23/2012 03:07 PM, kadafax@gmail.com wrote:
>>
>>> kfx, try getting the inode number of the file (via stat or ls -i)
>>> and then doing something like this:
>>>
>>> xfs_db -r $DEV -c "inode $INO" -c "bmap"
>>
>>
>> # xfs_db -r /dev/sdc1 -c "inode 114748" -c "bmap"
>> data offset 0 startblock 1881705728 (7/2657536) count 6460 flag 0
>>
>> # xfs_db -r /dev/sdc1 -c "inode 114754" -c "bmap"
>> data offset 0 startblock 1077794560 (4/4052736) count 6582 flag 0
>
> If you want to see what's behind those data extents (which are
> probably partially written), you could do something along these lines:
>
> # Determine the AG size
> agblocks=$(xfs_db -r /dev/sdc1 -c sb -c p | grep ^agblocks | sed 's/.*
> = //')
>
> # Copy the extent in the first file, which consists of 6460 blocks
> (~26MB)
> # in AG 7 starting at AG-relative block 2657536:
> dd if=/dev/sdc1 bs=4096 skip=$(($agblocks * 7 + 2657536)) count=6460
> of=./blob
> # examine ./blob
Awesome, it's working for each file with non-null data extents.
I guess that for inode for which the command "xfs_db -r $DEV -c "inode
$INO" -c "bmap"" returns a null result there is no hope ?
Anyway, thanks a lot Nathaniel !
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: XFS, empty files after a crash
2012-02-21 12:01 XFS, empty files after a crash kfx
2012-02-21 16:25 ` Peter Grandi
@ 2012-02-21 17:39 ` Stan Hoeppner
2012-02-27 1:04 ` Dave Chinner
2 siblings, 0 replies; 11+ messages in thread
From: Stan Hoeppner @ 2012-02-21 17:39 UTC (permalink / raw)
To: kadafax; +Cc: xfs
On 2/21/2012 6:01 AM, kfx wrote:
> After a crash, a lot of files on a xfs file system report an empty size
> with "ls -a" but not with "du".
> xfs_check and xfs_repair don't report any problem to repair (although an
> internal log is used) :(
> So if I understand correctly, those empty files actually use space on
> the disk but are incorrectly reported as empty by the system.
> Is there a way to copy back the used blocks to recover the files ?
What was the nature of the crash? Kernel panic? Power failure? Other?
Do you have have backups of previously existing files that are now shown
as zero size? If so delete and restore, maybe just overwrite.
For new files that were in the process of being created you're simply
out of luck. The metadata may have hit the journal but the file data in
buffer cache was lost.
In memory write buffering/ordering techniques are what give modern
filesystems their write throughput performance. Unfortunately this
comes at a cost, when systems crash. As Peter mentioned, one way to
mitigate this is coding apps to sync their writes. This obviously has a
performance impact. Another is possibly to mount filesystems with the
sync option, which will obviously severely impact write performance.
Both will prevent or at least lessen the carnage you've seen as a result
of this crash. You may not be able to live the performance of the
latter, though.
--
Stan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: XFS, empty files after a crash
2012-02-21 12:01 XFS, empty files after a crash kfx
2012-02-21 16:25 ` Peter Grandi
2012-02-21 17:39 ` Stan Hoeppner
@ 2012-02-27 1:04 ` Dave Chinner
2012-02-27 8:43 ` kadafax
2 siblings, 1 reply; 11+ messages in thread
From: Dave Chinner @ 2012-02-27 1:04 UTC (permalink / raw)
To: kfx; +Cc: xfs
On Tue, Feb 21, 2012 at 01:01:43PM +0100, kfx wrote:
> Hi,
>
> After a crash, a lot of files on a xfs file system report an empty
> size with "ls -a" but not with "du".
> xfs_check and xfs_repair don't report any problem to repair
> (although an internal log is used) :(
> So if I understand correctly, those empty files actually use space
> on the disk but are incorrectly reported as empty by the system.
> Is there a way to copy back the used blocks to recover the files ?
>
> Example for a previously not empty file:
> # ls -l myfile
> -rw-r--r-- 1 me me 0 2012-01-24 15:29 myfile
>
> # du myfile
> 27460 myfile
>
> # du --apparent-size myfile
> 0 myfile
>
> # xfs_bmap myfile
> myfile: no extents
That doesn't seem right:
$ ls -l foobar
-rw------- 1 root root 0 Feb 27 11:54 foobar
$ du foobar
1024 foobar
$ du --apparent-size foobar
0 foobar
$ xfs_bmap foobar
foobar:
0: [0..2047]: 255169872..255171919
So xfs_bmap can and does report extents beyond EOF.
du uses the fstat(2) syscall to get the block count from the inode,
so it's seeing an inode with a block count but no extents. Can you
dump the inode via xfs_db like so:
$ ls -i foobar
604424233 foobar
$ sudo xfs_db -r -c "inode 604424233" -c p /dev/md0
core.magic = 0x494e
core.mode = 0100600
core.version = 2
.....
And post the output?
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: XFS, empty files after a crash
2012-02-27 1:04 ` Dave Chinner
@ 2012-02-27 8:43 ` kadafax
2012-02-28 1:39 ` Dave Chinner
0 siblings, 1 reply; 11+ messages in thread
From: kadafax @ 2012-02-27 8:43 UTC (permalink / raw)
To: xfs
Le 27/02/12 02:04, Dave Chinner a écrit :
> On Tue, Feb 21, 2012 at 01:01:43PM +0100, kfx wrote:
>> # du myfile
>> 27460 myfile
>>
>> # du --apparent-size myfile
>> 0 myfile
>>
>> # xfs_bmap myfile
>> myfile: no extents
> That doesn't seem right:
>
> $ ls -l foobar
> -rw------- 1 root root 0 Feb 27 11:54 foobar
> $ du foobar
> 1024 foobar
> $ du --apparent-size foobar
> 0 foobar
> $ xfs_bmap foobar
> foobar:
> 0: [0..2047]: 255169872..255171919
>
> So xfs_bmap can and does report extents beyond EOF.
>
> du uses the fstat(2) syscall to get the block count from the inode,
> so it's seeing an inode with a block count but no extents. Can you
> dump the inode via xfs_db like so:
>
> $ ls -i foobar
> 604424233 foobar
> $ sudo xfs_db -r -c "inode 604424233" -c p /dev/md0
> core.magic = 0x494e
> core.mode = 0100600
> core.version = 2
> .....
>
> And post the output?
[root@server 1_out]# du foobar
25840 foobar
[root@server 1_out]# du --apparent-size foobar
0 foobar
[root@server 1_out]# xfs_bmap foobar
foobar: no extents
[root@server 1_out]# ls -i foobar
114748 foobar
[root@server 1_out]# xfs_db -r -c "inode 114748" -c p /dev/sdc1
core.magic = 0x494e
core.mode = 0100644
core.version = 2
core.format = 2 (extents)
core.nlinkv2 = 1
core.onlink = 0
core.projid = 0
core.uid = 12488
core.gid = 12488
core.flushiter = 0
core.atime.sec = Tue Jan 24 14:50:52 2012
core.atime.nsec = 609667096
core.mtime.sec = Tue Jan 24 14:50:53 2012
core.mtime.nsec = 579672930
core.ctime.sec = Tue Jan 24 14:50:53 2012
core.ctime.nsec = 579672930
core.size = 0
core.nblocks = 6460
core.extsize = 0
core.nextents = 1
core.naextents = 0
core.forkoff = 0
core.aformat = 2 (extents)
core.dmevmask = 0
core.dmstate = 0
core.newrtbm = 0
core.prealloc = 0
core.realtime = 0
core.immutable = 0
core.append = 0
core.sync = 0
core.noatime = 0
core.nodump = 0
core.rtinherit = 0
core.projinherit = 0
core.nosymlinks = 0
core.extsz = 0
core.extszinherit = 0
core.nodefrag = 0
core.filestream = 0
core.gen = 1924597489
next_unlinked = null
u.bmx[0] = [startoff,startblock,blockcount,extentflag]
0:[0,1881705728,6460,0]
After using the Nathaniel's advices and dd'ing the file:
[root@server recover]# ls -alh foobar
-rw-r--r-- 1 root root 26M Feb 24 10:39 /recover/foobar
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: XFS, empty files after a crash
2012-02-27 8:43 ` kadafax
@ 2012-02-28 1:39 ` Dave Chinner
0 siblings, 0 replies; 11+ messages in thread
From: Dave Chinner @ 2012-02-28 1:39 UTC (permalink / raw)
To: kadafax; +Cc: xfs
On Mon, Feb 27, 2012 at 09:43:42AM +0100, kadafax@gmail.com wrote:
> Le 27/02/12 02:04, Dave Chinner a écrit :
> >On Tue, Feb 21, 2012 at 01:01:43PM +0100, kfx wrote:
> >># du myfile
> >>27460 myfile
> >>
> >># du --apparent-size myfile
> >>0 myfile
> >>
> >># xfs_bmap myfile
> >>myfile: no extents
> >That doesn't seem right:
> >
> >$ ls -l foobar
> >-rw------- 1 root root 0 Feb 27 11:54 foobar
> >$ du foobar
> >1024 foobar
> >$ du --apparent-size foobar
> >0 foobar
> >$ xfs_bmap foobar
> >foobar:
> > 0: [0..2047]: 255169872..255171919
> >
> >So xfs_bmap can and does report extents beyond EOF.
> >
> >du uses the fstat(2) syscall to get the block count from the inode,
> >so it's seeing an inode with a block count but no extents. Can you
> >dump the inode via xfs_db like so:
.....
> [root@server 1_out]# xfs_db -r -c "inode 114748" -c p /dev/sdc1
> core.magic = 0x494e
> core.mode = 0100644
> core.version = 2
> core.format = 2 (extents)
.....
> core.size = 0
> core.nblocks = 6460
> core.extsize = 0
> core.nextents = 1
.....
> core.prealloc = 0
....
> core.extsz = 0
> core.extszinherit = 0
....
> u.bmx[0] = [startoff,startblock,blockcount,extentflag]
> 0:[0,1881705728,6460,0]
Ok, so there's an extent there, but xfs_bmap is not showing it
because it isn't a preallocation but rather a normal extent. What
this indicates is that the inode size was not updated on disk after
the data write had occurred.
Given you are running 2.6.38 (the unbuntu kernel), this is likely
caused by a known bug introduced in 2.6.37 by VFS changes outside
XFS to prevent sync livelocks. That has been fixed in current 3.0.x
stable kernels and more recent 3.2+ kernels. I'd suggest upgrading
your kernel.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2012-02-28 1:39 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-02-21 12:01 XFS, empty files after a crash kfx
2012-02-21 16:25 ` Peter Grandi
2012-02-22 3:18 ` Eric Sandeen
2012-02-23 19:38 ` Nathaniel W. Turner
2012-02-23 20:07 ` kadafax
2012-02-23 22:15 ` Nathaniel W. Turner
2012-02-24 10:21 ` kadafax
2012-02-21 17:39 ` Stan Hoeppner
2012-02-27 1:04 ` Dave Chinner
2012-02-27 8:43 ` kadafax
2012-02-28 1:39 ` Dave Chinner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox