* xfs_bmbt_rec_64 leading to wrong blocks @ 2014-08-06 10:12 Felipe Monteiro de Carvalho 2014-08-06 11:34 ` Brian Foster 0 siblings, 1 reply; 6+ messages in thread From: Felipe Monteiro de Carvalho @ 2014-08-06 10:12 UTC (permalink / raw) To: xfs Hello, I am writing a software which reads xfs partitions and it is working well so far, but, at in a particular folder with 10k files, only 7k files appear, the other ones don't. I read an inode which has di_core.di_format = XFS_DINODE_FMT_BTREE No problem here, the btree is trivial, it has only 1 element inside it, which leads to a list of 82 extends. Extends 0..52 aprox. are all great, and work fine. Extent 53 (and other ones after that) point to very wierd memory areas which don't match the pattern that I saw previously =( Here is the hex data of the extents, the selected extent in nr 53: http://magnifier.sourceforge.net/temp/xfs/extent_with_xfs_bmbt_rec_64.png And here is debug information showing which values I extracted from this table: i=0 FLocalExtent.StartOff=0 FLocalExtent.StartBlock=66748 FLocalExtent.BlockCount=1 lStartBlock=65212 i=1 FLocalExtent.StartOff=1 FLocalExtent.StartBlock=66758 FLocalExtent.BlockCount=2 lStartBlock=65222 i=2 FLocalExtent.StartOff=3 FLocalExtent.StartBlock=66772 FLocalExtent.BlockCount=1 lStartBlock=65236 .... i=51 FLocalExtent.StartOff=77 FLocalExtent.StartBlock=67468 FLocalExtent.BlockCount=1 lStartBlock=65932 i=52 FLocalExtent.StartOff=78 FLocalExtent.StartBlock=67479 FLocalExtent.BlockCount=1 lStartBlock=65943 i=53 FLocalExtent.StartOff=8388608 FLocalExtent.StartBlock=66749 FLocalExtent.BlockCount=1 lStartBlock=65213 i=54 FLocalExtent.StartOff=8388609 FLocalExtent.StartBlock=66783 FLocalExtent.BlockCount=2 lStartBlock=65247 ... Note how StartOff is suddenly so big! But I manually checked the bits comparing to xfs_bmbt_rec_64 and the value written is that one. But what sense does it make for StartOff to jump like that? And why are the blocks pointing to a non-XD2D block? Here is the block to which one of the valid extents point to: http://magnifier.sourceforge.net/temp/xfs/correct_block_xd2d.png And here is a block to which one of the extents that I cannot interpretate point to: http://magnifier.sourceforge.net/temp/xfs/wierd_block_xd2d.png and another one: http://magnifier.sourceforge.net/temp/xfs/wierd_xfs_dir2_data_entry.png Any ideas??? I already worked so much on this, but I cannot figure out =( thanks, -- Felipe Monteiro de Carvalho _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: xfs_bmbt_rec_64 leading to wrong blocks 2014-08-06 10:12 xfs_bmbt_rec_64 leading to wrong blocks Felipe Monteiro de Carvalho @ 2014-08-06 11:34 ` Brian Foster 2014-08-06 12:31 ` Felipe Monteiro de Carvalho 0 siblings, 1 reply; 6+ messages in thread From: Brian Foster @ 2014-08-06 11:34 UTC (permalink / raw) To: Felipe Monteiro de Carvalho; +Cc: xfs On Wed, Aug 06, 2014 at 12:12:12PM +0200, Felipe Monteiro de Carvalho wrote: > Hello, > > I am writing a software which reads xfs partitions and it is working > well so far, but, at in a particular folder with 10k files, only 7k > files appear, the other ones don't. > > I read an inode which has di_core.di_format = XFS_DINODE_FMT_BTREE > > No problem here, the btree is trivial, it has only 1 element inside > it, which leads to a list of 82 extends. > > Extends 0..52 aprox. are all great, and work fine. > > Extent 53 (and other ones after that) point to very wierd memory areas > which don't match the pattern that I saw previously =( > > Here is the hex data of the extents, the selected extent in nr 53: > > http://magnifier.sourceforge.net/temp/xfs/extent_with_xfs_bmbt_rec_64.png > > And here is debug information showing which values I extracted from this table: > > i=0 FLocalExtent.StartOff=0 FLocalExtent.StartBlock=66748 > FLocalExtent.BlockCount=1 lStartBlock=65212 > i=1 FLocalExtent.StartOff=1 FLocalExtent.StartBlock=66758 > FLocalExtent.BlockCount=2 lStartBlock=65222 > i=2 FLocalExtent.StartOff=3 FLocalExtent.StartBlock=66772 > FLocalExtent.BlockCount=1 lStartBlock=65236 > .... > > i=51 FLocalExtent.StartOff=77 FLocalExtent.StartBlock=67468 > FLocalExtent.BlockCount=1 lStartBlock=65932 > i=52 FLocalExtent.StartOff=78 FLocalExtent.StartBlock=67479 > FLocalExtent.BlockCount=1 lStartBlock=65943 > i=53 FLocalExtent.StartOff=8388608 FLocalExtent.StartBlock=66749 > FLocalExtent.BlockCount=1 lStartBlock=65213 > i=54 FLocalExtent.StartOff=8388609 FLocalExtent.StartBlock=66783 > FLocalExtent.BlockCount=2 lStartBlock=65247 > ... > > Note how StartOff is suddenly so big! But I manually checked the bits > comparing to xfs_bmbt_rec_64 and the value written is that one. But > what sense does it make for StartOff to jump like that? > Assuming 4k blocks, 8388608 is 32G, which is the offset interval used for some internal sections of directories on XFS. If this is a directory, it could be a node/leaf or freelist block. Looking at xfs_da_format.h, it looks like XFS_DIR2_LEAF_OFFSET points to this offset. This block basically points to a btree of hashed entry names for lookup. See here for basics: http://xfs.org/docs/xfsdocs-xml-dev/XFS_Filesystem_Structure//tmp/en-US/html/Directories.html ... and here for a diagram: http://xfs.org/docs/xfsdocs-xml-dev/XFS_Filesystem_Structure//tmp/en-US/html/Node_Directories.html > And why are the blocks pointing to a non-XD2D block? > > Here is the block to which one of the valid extents point to: > > http://magnifier.sourceforge.net/temp/xfs/correct_block_xd2d.png > > And here is a block to which one of the extents that I cannot > interpretate point to: > > http://magnifier.sourceforge.net/temp/xfs/wierd_block_xd2d.png > and another one: > http://magnifier.sourceforge.net/temp/xfs/wierd_xfs_dir2_data_entry.png > The 0xFEBE magic seems to confirm the second one as a node block. Brian > Any ideas??? > > I already worked so much on this, but I cannot figure out =( > > thanks, > -- > Felipe Monteiro de Carvalho > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: xfs_bmbt_rec_64 leading to wrong blocks 2014-08-06 11:34 ` Brian Foster @ 2014-08-06 12:31 ` Felipe Monteiro de Carvalho 2014-08-06 14:23 ` Brian Foster 0 siblings, 1 reply; 6+ messages in thread From: Felipe Monteiro de Carvalho @ 2014-08-06 12:31 UTC (permalink / raw) To: Brian Foster; +Cc: xfs Thanks! Indeed that's it =) One strange thing however is that in my xfs_dir2_data.h there is: #define XFS_DIR2_DATA_ALIGN_LOG 3 /* i.e., 8 bytes */ #define XFS_DIR2_SPACE_SIZE (1ULL << (32 + XFS_DIR2_DATA_ALIGN_LOG)) #define XFS_DIR2_LEAF_SPACE 1 #define XFS_DIR2_LEAF_OFFSET (XFS_DIR2_LEAF_SPACE * XFS_DIR2_SPACE_SIZE) That doesn't match the number that I am seeing here for the offset. Here I see 1 << 23, but according to the headers it should be 1 << 35 -- Felipe Monteiro de Carvalho On Wed, Aug 6, 2014 at 1:34 PM, Brian Foster <bfoster@redhat.com> wrote: > On Wed, Aug 06, 2014 at 12:12:12PM +0200, Felipe Monteiro de Carvalho wrote: >> Hello, >> >> I am writing a software which reads xfs partitions and it is working >> well so far, but, at in a particular folder with 10k files, only 7k >> files appear, the other ones don't. >> >> I read an inode which has di_core.di_format = XFS_DINODE_FMT_BTREE >> >> No problem here, the btree is trivial, it has only 1 element inside >> it, which leads to a list of 82 extends. >> >> Extends 0..52 aprox. are all great, and work fine. >> >> Extent 53 (and other ones after that) point to very wierd memory areas >> which don't match the pattern that I saw previously =( >> >> Here is the hex data of the extents, the selected extent in nr 53: >> >> http://magnifier.sourceforge.net/temp/xfs/extent_with_xfs_bmbt_rec_64.png >> >> And here is debug information showing which values I extracted from this table: >> >> i=0 FLocalExtent.StartOff=0 FLocalExtent.StartBlock=66748 >> FLocalExtent.BlockCount=1 lStartBlock=65212 >> i=1 FLocalExtent.StartOff=1 FLocalExtent.StartBlock=66758 >> FLocalExtent.BlockCount=2 lStartBlock=65222 >> i=2 FLocalExtent.StartOff=3 FLocalExtent.StartBlock=66772 >> FLocalExtent.BlockCount=1 lStartBlock=65236 >> .... >> >> i=51 FLocalExtent.StartOff=77 FLocalExtent.StartBlock=67468 >> FLocalExtent.BlockCount=1 lStartBlock=65932 >> i=52 FLocalExtent.StartOff=78 FLocalExtent.StartBlock=67479 >> FLocalExtent.BlockCount=1 lStartBlock=65943 >> i=53 FLocalExtent.StartOff=8388608 FLocalExtent.StartBlock=66749 >> FLocalExtent.BlockCount=1 lStartBlock=65213 >> i=54 FLocalExtent.StartOff=8388609 FLocalExtent.StartBlock=66783 >> FLocalExtent.BlockCount=2 lStartBlock=65247 >> ... >> >> Note how StartOff is suddenly so big! But I manually checked the bits >> comparing to xfs_bmbt_rec_64 and the value written is that one. But >> what sense does it make for StartOff to jump like that? >> > > Assuming 4k blocks, 8388608 is 32G, which is the offset interval used > for some internal sections of directories on XFS. If this is a > directory, it could be a node/leaf or freelist block. Looking at > xfs_da_format.h, it looks like XFS_DIR2_LEAF_OFFSET points to this > offset. This block basically points to a btree of hashed entry names for > lookup. See here for basics: > > http://xfs.org/docs/xfsdocs-xml-dev/XFS_Filesystem_Structure//tmp/en-US/html/Directories.html > > ... and here for a diagram: > > http://xfs.org/docs/xfsdocs-xml-dev/XFS_Filesystem_Structure//tmp/en-US/html/Node_Directories.html > >> And why are the blocks pointing to a non-XD2D block? >> >> Here is the block to which one of the valid extents point to: >> >> http://magnifier.sourceforge.net/temp/xfs/correct_block_xd2d.png >> >> And here is a block to which one of the extents that I cannot >> interpretate point to: >> >> http://magnifier.sourceforge.net/temp/xfs/wierd_block_xd2d.png >> and another one: >> http://magnifier.sourceforge.net/temp/xfs/wierd_xfs_dir2_data_entry.png >> > > The 0xFEBE magic seems to confirm the second one as a node block. > > Brian > >> Any ideas??? >> >> I already worked so much on this, but I cannot figure out =( >> >> thanks, >> -- >> Felipe Monteiro de Carvalho >> >> _______________________________________________ >> xfs mailing list >> xfs@oss.sgi.com >> http://oss.sgi.com/mailman/listinfo/xfs -- Felipe Monteiro de Carvalho _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: xfs_bmbt_rec_64 leading to wrong blocks 2014-08-06 12:31 ` Felipe Monteiro de Carvalho @ 2014-08-06 14:23 ` Brian Foster 2014-08-07 11:51 ` Felipe Monteiro de Carvalho 0 siblings, 1 reply; 6+ messages in thread From: Brian Foster @ 2014-08-06 14:23 UTC (permalink / raw) To: Felipe Monteiro de Carvalho; +Cc: xfs On Wed, Aug 06, 2014 at 02:31:11PM +0200, Felipe Monteiro de Carvalho wrote: > Thanks! Indeed that's it =) > > One strange thing however is that in my xfs_dir2_data.h there is: > > #define XFS_DIR2_DATA_ALIGN_LOG 3 /* i.e., 8 bytes */ > #define XFS_DIR2_SPACE_SIZE (1ULL << (32 + XFS_DIR2_DATA_ALIGN_LOG)) > #define XFS_DIR2_LEAF_SPACE 1 > #define XFS_DIR2_LEAF_OFFSET (XFS_DIR2_LEAF_SPACE * XFS_DIR2_SPACE_SIZE) > > That doesn't match the number that I am seeing here for the offset. > Here I see 1 << 23, but according to the headers it should be 1 << 35 > We're talking about extent map data here right? If so, the offset on disk is in filesystem block units. E.g., block 8388608 * 4096 == 32G == (1 << 35). Brian > -- > Felipe Monteiro de Carvalho > > > On Wed, Aug 6, 2014 at 1:34 PM, Brian Foster <bfoster@redhat.com> wrote: > > On Wed, Aug 06, 2014 at 12:12:12PM +0200, Felipe Monteiro de Carvalho wrote: > >> Hello, > >> > >> I am writing a software which reads xfs partitions and it is working > >> well so far, but, at in a particular folder with 10k files, only 7k > >> files appear, the other ones don't. > >> > >> I read an inode which has di_core.di_format = XFS_DINODE_FMT_BTREE > >> > >> No problem here, the btree is trivial, it has only 1 element inside > >> it, which leads to a list of 82 extends. > >> > >> Extends 0..52 aprox. are all great, and work fine. > >> > >> Extent 53 (and other ones after that) point to very wierd memory areas > >> which don't match the pattern that I saw previously =( > >> > >> Here is the hex data of the extents, the selected extent in nr 53: > >> > >> http://magnifier.sourceforge.net/temp/xfs/extent_with_xfs_bmbt_rec_64.png > >> > >> And here is debug information showing which values I extracted from this table: > >> > >> i=0 FLocalExtent.StartOff=0 FLocalExtent.StartBlock=66748 > >> FLocalExtent.BlockCount=1 lStartBlock=65212 > >> i=1 FLocalExtent.StartOff=1 FLocalExtent.StartBlock=66758 > >> FLocalExtent.BlockCount=2 lStartBlock=65222 > >> i=2 FLocalExtent.StartOff=3 FLocalExtent.StartBlock=66772 > >> FLocalExtent.BlockCount=1 lStartBlock=65236 > >> .... > >> > >> i=51 FLocalExtent.StartOff=77 FLocalExtent.StartBlock=67468 > >> FLocalExtent.BlockCount=1 lStartBlock=65932 > >> i=52 FLocalExtent.StartOff=78 FLocalExtent.StartBlock=67479 > >> FLocalExtent.BlockCount=1 lStartBlock=65943 > >> i=53 FLocalExtent.StartOff=8388608 FLocalExtent.StartBlock=66749 > >> FLocalExtent.BlockCount=1 lStartBlock=65213 > >> i=54 FLocalExtent.StartOff=8388609 FLocalExtent.StartBlock=66783 > >> FLocalExtent.BlockCount=2 lStartBlock=65247 > >> ... > >> > >> Note how StartOff is suddenly so big! But I manually checked the bits > >> comparing to xfs_bmbt_rec_64 and the value written is that one. But > >> what sense does it make for StartOff to jump like that? > >> > > > > Assuming 4k blocks, 8388608 is 32G, which is the offset interval used > > for some internal sections of directories on XFS. If this is a > > directory, it could be a node/leaf or freelist block. Looking at > > xfs_da_format.h, it looks like XFS_DIR2_LEAF_OFFSET points to this > > offset. This block basically points to a btree of hashed entry names for > > lookup. See here for basics: > > > > http://xfs.org/docs/xfsdocs-xml-dev/XFS_Filesystem_Structure//tmp/en-US/html/Directories.html > > > > ... and here for a diagram: > > > > http://xfs.org/docs/xfsdocs-xml-dev/XFS_Filesystem_Structure//tmp/en-US/html/Node_Directories.html > > > >> And why are the blocks pointing to a non-XD2D block? > >> > >> Here is the block to which one of the valid extents point to: > >> > >> http://magnifier.sourceforge.net/temp/xfs/correct_block_xd2d.png > >> > >> And here is a block to which one of the extents that I cannot > >> interpretate point to: > >> > >> http://magnifier.sourceforge.net/temp/xfs/wierd_block_xd2d.png > >> and another one: > >> http://magnifier.sourceforge.net/temp/xfs/wierd_xfs_dir2_data_entry.png > >> > > > > The 0xFEBE magic seems to confirm the second one as a node block. > > > > Brian > > > >> Any ideas??? > >> > >> I already worked so much on this, but I cannot figure out =( > >> > >> thanks, > >> -- > >> Felipe Monteiro de Carvalho > >> > >> _______________________________________________ > >> xfs mailing list > >> xfs@oss.sgi.com > >> http://oss.sgi.com/mailman/listinfo/xfs > > > > -- > Felipe Monteiro de Carvalho > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: xfs_bmbt_rec_64 leading to wrong blocks 2014-08-06 14:23 ` Brian Foster @ 2014-08-07 11:51 ` Felipe Monteiro de Carvalho 2014-08-07 12:56 ` Brian Foster 0 siblings, 1 reply; 6+ messages in thread From: Felipe Monteiro de Carvalho @ 2014-08-07 11:51 UTC (permalink / raw) To: Brian Foster; +Cc: xfs Hello, Thanks, that worked perfectly =) My actual idea is to not only read the partition, but to also recover deleted/corrupted files. With the information that you provided I advanced a lot in this direction, and I can now recover the file information in that memory area completely. I can also obtain the inode nr (well 4 bytes of it are preserved, but its better then nothing), but my current problem is with the inode itself. In a deleted file the following changes are done by the linux driver in the inode: xfs_dinode_core_t.di_mode was changed to zero xfs_dinode_core_t.di_format was changed from 1 (local) to 2 (extent) But di_mode in particular is a key element as I am using it to differentiate files from directories. Is there anything else what I can use to differentiate files from directories in case di_mode was corrupted? I am searching here but so far I didn't find anything... thanks, -- Felipe Monteiro de Carvalho _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: xfs_bmbt_rec_64 leading to wrong blocks 2014-08-07 11:51 ` Felipe Monteiro de Carvalho @ 2014-08-07 12:56 ` Brian Foster 0 siblings, 0 replies; 6+ messages in thread From: Brian Foster @ 2014-08-07 12:56 UTC (permalink / raw) To: Felipe Monteiro de Carvalho; +Cc: xfs On Thu, Aug 07, 2014 at 01:51:01PM +0200, Felipe Monteiro de Carvalho wrote: > Hello, > > Thanks, that worked perfectly =) > > My actual idea is to not only read the partition, but to also recover > deleted/corrupted files. With the information that you provided I > advanced a lot in this direction, and I can now recover the file > information in that memory area completely. I can also obtain the > inode nr (well 4 bytes of it are preserved, but its better then > nothing), but my current problem is with the inode itself. In a > deleted file the following changes are done by the linux driver in the > inode: > > xfs_dinode_core_t.di_mode was changed to zero > xfs_dinode_core_t.di_format was changed from 1 (local) to 2 (extent) > These are set in xfs_ifree() when the inode is pulled from the unlinked list and ultimately freed. Resetting these values certainly makes sense, but I'm not aware of the reason why di_format is set as such. > But di_mode in particular is a key element as I am using it to > differentiate files from directories. > In general you can't rely on on-disk data once the inode has been freed. Perhaps you should start a new thread with some kind of write up about what you're trying to accomplish and how you're going about it. Brian > Is there anything else what I can use to differentiate files from > directories in case di_mode was corrupted? I am searching here but so > far I didn't find anything... > > thanks, > -- > Felipe Monteiro de Carvalho > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-08-07 12:56 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-08-06 10:12 xfs_bmbt_rec_64 leading to wrong blocks Felipe Monteiro de Carvalho 2014-08-06 11:34 ` Brian Foster 2014-08-06 12:31 ` Felipe Monteiro de Carvalho 2014-08-06 14:23 ` Brian Foster 2014-08-07 11:51 ` Felipe Monteiro de Carvalho 2014-08-07 12:56 ` Brian Foster
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
