Bug (?) : cumulative xfsrestore does not restore files and folders in a directory which was renamed

Bug (?) : cumulative xfsrestore does not restore files and folders in a directory which was renamed

[Damien Gombault]

[-- Attachment #1.1: Type: text/plain, Size: 2121 bytes --]

Hi.

I have reported few weeks ago a bug on the Red Hat Bugzilla for CentOS 7 : https://bugzilla.redhat.com/show_bug.cgi?id=1341126
I had to restore a level0+level1 dump for my ownCloud server, but during the level1 restoration, xfsrestore showed many warnings and some directories/files were not restored.
I recently tried to restore a more recent level0+level1 dump and I get the same results with other directories/files.
I tried to figure out why and now I am able to reproduce the problem with a minimal test case.

Here is the test case :

Create a new XFS filesystem, mount it as "/mnt/test"
Then in "/mnt/test"
Create a directory "dira/"
Create a file "filea" in directory "dira/"
You should get :

.
└── dira
     └── filea

Make a level 0 dump :
xfsdump -l 0 -f /root/test0.dump /mnt/test

Rename "dira/" to "dirA/"
Create a directory "dirb/" in "dirA/"
Create a file "fileb" in "dirb/"
You should get :

.
└── dirA
     ├── dirb
     │   └── fileb
     └── filea

Make a level 1 dump :
xfsdump -l 1 -f /root/test1.dump /mnt/test

Create a new XFS filesystem, mount it as "/mnt/restore"

Restore (cumulative mode) level 0 dump :
xfsrestore -r -f /root/test0.dump /mnt/restore/

Restore (cumulative mode) level 1 dump :
xfsrestore -r -f /root/test1.dump /mnt/restore/

You get a WARNING :
xfsrestore: WARNING: open of dirA/dirb/fileb failed: Aucun fichier ou dossier de ce type: discarding ino 526337 (No such file or directory in English)
Directory "dirb" and file "fileb" is not restored :

.
├── dirA
│   └── filea
└── xfsrestorehousekeepingdir
     ├── dirattr
     ├── dirextattr
     ├── namreg
     ├── state
     └── tree

I attached test0.dump (level 0 dump), test1.dump (level 1 dump), debug0.txt (xfsrestore level 0 debug trace), debug1.txt (xfsrestore level 1 debug trace).

I can reproduce it on CentOS 7 and Fedora 24.
On Fedora 24, I used :
xfsprogs-4.5.0-1.fc24.x86_64
xfsdump-3.1.6-2.fc24.x86_64
kernel-4.5.7-300.fc24.x86_64

Is there a bug in xfsdump/xfsrestore tools and cumulative restoration feature ?


[-- Attachment #1.2: Type: text/html, Size: 2468 bytes --]

[-- Attachment #2: debug0.txt --]
[-- Type: text/plain, Size: 3414 bytes --]

xfsrestore: raising stack size soft limit from 0x800000 to 0x2000000
xfsrestore: getpagesize( ) returns 4096
xfsrestore: parent tid is 140547709941632
xfsrestore: using file dump (drive_simple) strategy
xfsrestore: version 3.1.4 (dump format 3.0)
xfsrestore: thread 140547587008256 created for stream 0
xfsrestore: Media_create
xfsrestore: checking and validating command line dump id/label
xfsrestore: searching media for dump
xfsrestore: Media_mfile_next: purp==0 pos==0
xfsrestore: examining media file 0
xfsrestore: file 0 in object 0 of stream 0
xfsrestore: file 0 in stream, file 0 of dump 0 on object
xfsrestore: dump found: checking
xfsrestore: dump description: 
xfsrestore: hostname: hostrecia1.recia.fr
xfsrestore: mount point: /mnt/test
xfsrestore: volume: /dev/mapper/centos_hostrecia1-test
xfsrestore: session time: Wed Jun 22 12:19:26 2016
xfsrestore: level: 0
xfsrestore: session label: "test"
xfsrestore: media label: "test"
xfsrestore: file system id: a81fa86b-92dd-41ea-862a-dc2a3e45d0b8
xfsrestore: session id: 817f2f21-bc06-46d5-b6a4-f49d1a059efe
xfsrestore: media id: f93e8f03-7606-4069-b754-e18b2972322c
xfsrestore: using online session inventory
xfsrestore: searching media for directory dump
xfsrestore: Media_mfile_next: purp==1 pos==1
xfsrestore: dump session label: "test"
xfsrestore: dump session id: 817f2f21-bc06-46d5-b6a4-f49d1a059efe
xfsrestore: stream 0, object 0, file 0
xfsrestore: initializing directory attributes registry
xfsrestore: initializing directory entry name registry
xfsrestore: initializing directory hierarchy image
xfsrestore: node_init: vmsz = 4611686018427359231 (0x3fffffffffff8fff) segsz = 58720256 (0x3800000) nodesperseg = 1048576 (0x100000) winmapmax = 4096 (0x1000)
xfsrestore: pre-growing new node array segment at 32768 size 58720256
xfsrestore: reading directories
xfsrestore: reading the ino map
xfsrestore: reading the directories 
xfsrestore: directory 2048 0 (0): updating
xfsrestore: dirent dira 2051 0: adding (new)
xfsrestore: directory 2051 0 (0): upgrading to dir
xfsrestore: dirent filea 2052 0: adding (new)
xfsrestore: 2 directories and 2 entries processed
xfsrestore: number of mmap calls for windows = 1
xfsrestore: Media_atnondir
xfsrestore: directory post-processing
xfsrestore: adjusting dirent ref flags
xfsrestore: applying subtree selections
xfsrestore: making new directories
xfsrestore: mkdir dira
xfsrestore: getting next media file for non-dir restore
xfsrestore: Media_mfile_next: purp==2 pos==3
xfsrestore: dump session label: "test"
xfsrestore: dump session id: 817f2f21-bc06-46d5-b6a4-f49d1a059efe
xfsrestore: stream 0, object 0, file 0
xfsrestore: restoring non-directory files
xfsrestore: media file 0 in object 0 of stream 0
xfsrestore: file 0 in stream, file 0 in dump 0 on object
xfsrestore: restoring dira/filea (2052 0)
xfsrestore: restoring regular file ino 2052 dira/filea
xfsrestore: Media_end: pos==3
xfsrestore: getting next media file for non-dir restore
xfsrestore: Media_mfile_next: purp==2 pos==0
xfsrestore: tree finalize
xfsrestore: content.c: 2605: mlog_exit called: exit_code: SUCCESS return: OK (success)
xfsrestore: all children have exited
xfsrestore: restore complete: 0 seconds elapsed
xfsrestore: main.c: 871: mlog_exit called: exit_code: SUCCESS return: UNKNOWN (unknown error)
xfsrestore: Restore Summary:
xfsrestore:   stream 0 /root/test0.dump OK (success)
xfsrestore: Restore Status: SUCCESS

[-- Attachment #3: debug1.txt --]
[-- Type: text/plain, Size: 3699 bytes --]

xfsrestore: raising stack size soft limit from 0x800000 to 0x2000000
xfsrestore: getpagesize( ) returns 4096
xfsrestore: parent tid is 139893610231680
xfsrestore: using file dump (drive_simple) strategy
xfsrestore: version 3.1.4 (dump format 3.0)
xfsrestore: thread 139893487298304 created for stream 0
xfsrestore: Media_create
xfsrestore: checking and validating command line dump id/label
xfsrestore: searching media for dump
xfsrestore: Media_mfile_next: purp==0 pos==0
xfsrestore: examining media file 0
xfsrestore: file 0 in object 0 of stream 0
xfsrestore: file 0 in stream, file 0 of dump 0 on object
xfsrestore: dump found: checking
xfsrestore: dump description: 
xfsrestore: hostname: hostrecia1.recia.fr
xfsrestore: mount point: /mnt/test
xfsrestore: volume: /dev/mapper/centos_hostrecia1-test
xfsrestore: session time: Wed Jun 22 12:20:42 2016
xfsrestore: level: 1
xfsrestore: session label: "test"
xfsrestore: media label: "test"
xfsrestore: file system id: a81fa86b-92dd-41ea-862a-dc2a3e45d0b8
xfsrestore: session id: 27606c57-7b3b-43ff-83b9-69158e75f612
xfsrestore: media id: 1fcb7c9c-44b2-427d-becc-d10816e2c430
xfsrestore: using online session inventory
xfsrestore: searching media for directory dump
xfsrestore: Media_mfile_next: purp==1 pos==1
xfsrestore: dump session label: "test"
xfsrestore: dump session id: 27606c57-7b3b-43ff-83b9-69158e75f612
xfsrestore: stream 0, object 0, file 0
xfsrestore: initializing directory attributes registry
xfsrestore: reading directories
xfsrestore: reading the ino map
xfsrestore: reading the directories 
xfsrestore: directory 2048 0 (0): updating
xfsrestore: dirent dirA 2051 0: renaming (name)
xfsrestore: directory 2051 0 (0): updating
xfsrestore: dirent filea 2052 0: retaining (nondir)
xfsrestore: dirent dirb 526336 1283006502: adding (new)
xfsrestore: directory 526336 1283006502 (1283006502): upgrading to dir
xfsrestore: dirent fileb 526337 1283006502: adding (new)
xfsrestore: 3 directories and 4 entries processed
xfsrestore: number of mmap calls for windows = 1
xfsrestore: Media_atnondir
xfsrestore: directory post-processing
xfsrestore: adjusting dirent ref flags
xfsrestore: applying subtree selections
xfsrestore: eliminating unreferenced directory entries
xfsrestore: rename dir dira to orphanage/2051.0
xfsrestore: making new directories
xfsrestore: performing directory renames
xfsrestore: rename dir orphanage/2051.0 to dirA
xfsrestore: processing hard links
xfsrestore: processing hardlinks to 2052 0
xfsrestore: processing hardlinks to 526337 1283006502
xfsrestore: getting next media file for non-dir restore
xfsrestore: Media_mfile_next: purp==2 pos==3
xfsrestore: dump session label: "test"
xfsrestore: dump session id: 27606c57-7b3b-43ff-83b9-69158e75f612
xfsrestore: stream 0, object 0, file 0
xfsrestore: restoring non-directory files
xfsrestore: media file 0 in object 0 of stream 0
xfsrestore: file 0 in stream, file 0 in dump 0 on object
xfsrestore: restoring dirA/dirb/fileb (526337 1283006502)
xfsrestore: restoring regular file ino 526337 dirA/dirb/fileb
xfsrestore: WARNING: open of dirA/dirb/fileb failed: Aucun fichier ou dossier de ce type: discarding ino 526337
xfsrestore: Media_end: pos==3
xfsrestore: getting next media file for non-dir restore
xfsrestore: Media_mfile_next: purp==2 pos==0
xfsrestore: tree finalize
xfsrestore: content.c: 2605: mlog_exit called: exit_code: SUCCESS return: OK (success)
xfsrestore: all children have exited
xfsrestore: restore complete: 0 seconds elapsed
xfsrestore: main.c: 871: mlog_exit called: exit_code: SUCCESS return: UNKNOWN (unknown error)
xfsrestore: Restore Summary:
xfsrestore:   stream 0 /root/test1.dump OK (success)
xfsrestore: Restore Status: SUCCESS

[-- Attachment #4: test0.dump --]
[-- Type: application/octet-stream, Size: 21632 bytes --]

[-- Attachment #5: test1.dump --]
[-- Type: application/octet-stream, Size: 21960 bytes --]

[-- Attachment #6: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Bug (?) : cumulative xfsrestore does not restore files and folders in a directory which was renamed

[Dave Chinner]

[please line wrap your email at 68-72 columns]

On Wed, Jun 22, 2016 at 04:45:19PM +0200, Damien Gombault wrote:
> Hi.
> 
> I have reported few weeks ago a bug on the Red Hat Bugzilla for
> CentOS 7 : https://bugzilla.redhat.com/show_bug.cgi?id=1341126 I
> had to restore a level0+level1 dump for my ownCloud server, but
> during the level1 restoration, xfsrestore showed many warnings and
> some directories/files were not restored.  I recently tried to
> restore a more recent level0+level1 dump and I get the same
> results with other directories/files.  I tried to figure out why
> and now I am able to reproduce the problem with a minimal test
> case.
> 
> Here is the test case :

<snip test case>

It's hard to understand xfsrestore problems without the xfsdump
output that generated the dumps being restored.

What is the (verbose) output of xfsdump under this test?

....
> xfsrestore: level: 0
> xfsrestore: session label: "test"
> xfsrestore: media label: "test"
...
> xfsrestore: directory 2048 0 (0): updating
> xfsrestore: dirent dira 2051 0: adding (new)
> xfsrestore: directory 2051 0 (0): upgrading to dir
> xfsrestore: dirent filea 2052 0: adding (new)
> xfsrestore: 2 directories and 2 entries processed

Which is correct.

....

> xfsrestore: level: 1
> xfsrestore: session label: "test"
> xfsrestore: media label: "test"
....
> xfsrestore: directory 2048 0 (0): updating
> xfsrestore: dirent dirA 2051 0: renaming (name)
> xfsrestore: directory 2051 0 (0): updating
> xfsrestore: dirent filea 2052 0: retaining (nondir)
> xfsrestore: dirent dirb 526336 1283006502: adding (new)
> xfsrestore: directory 526336 1283006502 (1283006502): upgrading to dir
> xfsrestore: dirent fileb 526337 1283006502: adding (new)
> xfsrestore: 3 directories and 4 entries processed

So the inventory looks ok....

> xfsrestore: rename dir dira to orphanage/2051.0
> xfsrestore: making new directories
> xfsrestore: performing directory renames
> xfsrestore: rename dir orphanage/2051.0 to dirA

And the rename occurs...

> xfsrestore: processing hard links
> xfsrestore: processing hardlinks to 2052 0
> xfsrestore: processing hardlinks to 526337 1283006502
> xfsrestore: getting next media file for non-dir restore
> xfsrestore: Media_mfile_next: purp==2 pos==3
> xfsrestore: dump session label: "test"
> xfsrestore: dump session id: 27606c57-7b3b-43ff-83b9-69158e75f612
> xfsrestore: stream 0, object 0, file 0
> xfsrestore: restoring non-directory files
> xfsrestore: media file 0 in object 0 of stream 0
> xfsrestore: file 0 in stream, file 0 in dump 0 on object
> xfsrestore: restoring dirA/dirb/fileb (526337 1283006502)
> xfsrestore: restoring regular file ino 526337 dirA/dirb/fileb
> xfsrestore: WARNING: open of dirA/dirb/fileb failed: Aucun fichier ou dossier de ce type: discarding ino 526337

But it seems that dirb hasn't been created before the file in it
is being restored. THis can happen because the inventory is not
correct, whichmay in fact be a problem with dump rather than
restore...

I'll have a bit of a play around here, see if I can reproduce it.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Bug (?) : cumulative xfsrestore does not restore files and folders in a directory which was renamed

[Dave Chinner]

On Thu, Jun 23, 2016 at 08:09:59AM +1000, Dave Chinner wrote:
> But it seems that dirb hasn't been created before the file in it
> is being restored. THis can happen because the inventory is not
> correct, whichmay in fact be a problem with dump rather than
> restore...
> 
> I'll have a bit of a play around here, see if I can reproduce it.

Yes, i can reproduce it, so I'll have a deeper look.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Bug (?) : cumulative xfsrestore does not restore files and folders in a directory which was renamed

[Dave Chinner]

On Thu, Jun 23, 2016 at 08:23:28AM +1000, Dave Chinner wrote:
> On Thu, Jun 23, 2016 at 08:09:59AM +1000, Dave Chinner wrote:
> > But it seems that dirb hasn't been created before the file in it
> > is being restored. THis can happen because the inventory is not
> > correct, whichmay in fact be a problem with dump rather than
> > restore...
> > 
> > I'll have a bit of a play around here, see if I can reproduce it.
> 
> Yes, i can reproduce it, so I'll have a deeper look.

Can you try this patch?

-Dave
-- 
Dave Chinner
david@fromorbit.com

restore: make new directories after renames

From: Dave Chinner <dchinner@redhat.com>

Damien Gombault reported that restores of cumulative dumps with
renamed directories were throwing an error and were incomplete:

xfsrestore: file 0 in stream, file 0 in dump 0 on object
xfsrestore: restoring dirA/dirb/fileb (526337 1283006502)
xfsrestore: restoring regular file ino 526337 dirA/dirb/fileb
xfsrestore: WARNING: open of dirA/dirb/fileb failed: Aucun fichier ou dossier de
ce type: discarding ino 526337

This was triggered by a level 1 dump containing a directory rename
and a new directory being created inside the renamed directory. i.e:

$ mv dira dirA
$ mkdir dirA/dirb
$ echo foo > dirA/dirb/fileb

xfs_restore handles directory renames by first moving the old
directory to the orphanage, then renaming it from the orphanage to
it's new location. This, in itself is fine.

The problem is that restore creates the new directories between
these two steps. Hence any new directory created in a renamed
directory cannot be restored from a cumulative dump because when
restore tries to create the new directory neither the old directory
path nor the new directory path exists.  Hence it silently drops the
new directory, resulting in subsequent errors tryin gto restore
files within that new directory.

The simple fix - just change the order of operations
in tree_post() so that new directories are created after all the
renames are processed - is not useful. All that does is break the
case of renames into newly created directories.

However, because the making of directories that already exist or
can't be made silently fails, and the create process does not modify
the internal directory tree, we can run the directory creation
function multiple times. Hence we can run directory creation both
before and after the directory rename step, hence ensuring both
new parents and new child directories are created appropriately.

This still may not be sufficient for complex directory
reorganisations, but it does address the reported problem in a
manner that is unlikely to cause regresssions. This is important,
because this code has not changed at all since it was first publicly
released in early 2001. Hence the minimum change we can make to fix
the reported problem is the least risky approach we can take.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 restore/tree.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 51 insertions(+), 10 deletions(-)

diff --git a/restore/tree.c b/restore/tree.c
index 0336e77..8e6fab1 100644
--- a/restore/tree.c
+++ b/restore/tree.c
@@ -1163,15 +1163,35 @@ tree_subtree_parse( bool_t sensepr, char *path )
 	return BOOL_TRUE;
 }
 
-/* tree_post - called after the dirdump has been applied.
- * first phase is to eliminate all unreferenced dirents.
- * done by recursive depth-wise descent of the tree. on the way
- * up, unlink or orphan unreferenced nondirs, unlink unreferenced
- * dirs, orphan dirs to be renamed. if a dir is unreferenced, but
+/* tree_post - called after the dirdump has been applied.  first phase is to
+ * eliminate all unreferenced dirents.  done by recursive depth-wise descent of
+ * the tree. on the way up, unlink or orphan unreferenced nondirs, unlink
+ * unreferenced dirs, orphan dirs to be renamed. if a dir is unreferenced, but
  * contains referenced dirents, orphan those dirents. orphan unreferenced
  * nondirs if they are the last link to an inode referenced but not real
  * somewhere else in the tree. next, make new directories. then rename
  * directories. finally, create hardlinks from orphanage.
+ *
+ * Note: the way renamed directories are handled by first orphaning them leads
+ * to a chicken and egg problem - the directory does not exist when we try to
+ * make a new directory inside the renamed directory destination. This fails
+ * silently, leaving us with ENOENT errors when trying to restore files within
+ * that new directory.
+ *
+ * To prevent this from happening, we need to create new subdirectories *after*
+ * we have processed all the renamed directories. However, so that the renames
+ * succeed, we also have to create any new parent directories that the renames
+ * depend on.
+ *
+ * Hence we have to do two mkdir passes: one before the renames to create new
+ * ancestors for rename destinations and one after the rename to create new
+ * children in rename destinations.
+ *
+ * NOTE: a simple before/after creation as done below may be too simple for
+ * complex directory structure manipulations. e.g. rename into a new child in
+ * the destination of another rename. This may have to become an iterative loop
+ * that runs until all renames and mkdirs are resolved. We'll cross that bridge
+ * when we need to, not now.
  */
 static bool_t noref_elim_recurse( nh_t parh,
 				  nh_t cldh,
@@ -1216,7 +1236,14 @@ tree_post( char *path1, char *path2 )
 		}
 	}
 
-	/* make new directories
+#ifdef TREE_CHK
+	assert( tree_chk( ));
+#endif /* TREE_CHK */
+
+	/*
+	 * make new directories to ensure rename destination ancestors are
+	 * present before attempting the renames. This will silently skip all
+	 * the creations that are in rename destinations.
 	 */
 	mlog( MLOG_DEBUG | MLOG_TREE,
 	      "making new directories\n" );
@@ -1228,10 +1255,6 @@ tree_post( char *path1, char *path2 )
 		return BOOL_FALSE;
 	}
 
-#ifdef TREE_CHK
-	assert( tree_chk( ));
-#endif /* TREE_CHK */
-
 	/* rename directories
 	 */
 	if ( ! persp->p_fullpr ) {
@@ -1250,6 +1273,24 @@ tree_post( char *path1, char *path2 )
 	assert( tree_chk( ));
 #endif /* TREE_CHK */
 
+	/*
+	 * Repeat making new directories to create directories in rename
+	 * destinations that were skipped in the first pass.
+	 */
+	mlog( MLOG_DEBUG | MLOG_TREE,
+	      "making new directories in renamed ancestors\n" );
+	rootp = Node_map( persp->p_rooth );
+	cldh = rootp->n_cldh;
+	Node_unmap( persp->p_rooth, &rootp );
+	ok = mkdirs_recurse( persp->p_rooth, cldh, path1 );
+	if ( ! ok ) {
+		return BOOL_FALSE;
+	}
+
+#ifdef TREE_CHK
+	assert( tree_chk( ));
+#endif /* TREE_CHK */
+
 	/* process hard links
 	 */
 	if ( ! persp->p_fullpr ) {

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Bug (?) : cumulative xfsrestore does not restore files and folders in a directory which was renamed

[Bill O'Donnell]

On Thu, Jun 23, 2016 at 11:42:17AM +1000, Dave Chinner wrote:
> On Thu, Jun 23, 2016 at 08:23:28AM +1000, Dave Chinner wrote:
> > On Thu, Jun 23, 2016 at 08:09:59AM +1000, Dave Chinner wrote:
> > > But it seems that dirb hasn't been created before the file in it
> > > is being restored. THis can happen because the inventory is not
> > > correct, whichmay in fact be a problem with dump rather than
> > > restore...
> > > 
> > > I'll have a bit of a play around here, see if I can reproduce it.
> > 
> > Yes, i can reproduce it, so I'll have a deeper look.
> 
> Can you try this patch?

FWIW, I tried it on a RHEL7 box and it works fine with Damien's test case.

Thanks-
Bill

> 
> -Dave
> -- 
> Dave Chinner
> david@fromorbit.com
> 
> restore: make new directories after renames
> 
> From: Dave Chinner <dchinner@redhat.com>
> 
> Damien Gombault reported that restores of cumulative dumps with
> renamed directories were throwing an error and were incomplete:
> 
> xfsrestore: file 0 in stream, file 0 in dump 0 on object
> xfsrestore: restoring dirA/dirb/fileb (526337 1283006502)
> xfsrestore: restoring regular file ino 526337 dirA/dirb/fileb
> xfsrestore: WARNING: open of dirA/dirb/fileb failed: Aucun fichier ou dossier de
> ce type: discarding ino 526337
> 
> This was triggered by a level 1 dump containing a directory rename
> and a new directory being created inside the renamed directory. i.e:
> 
> $ mv dira dirA
> $ mkdir dirA/dirb
> $ echo foo > dirA/dirb/fileb
> 
> xfs_restore handles directory renames by first moving the old
> directory to the orphanage, then renaming it from the orphanage to
> it's new location. This, in itself is fine.
> 
> The problem is that restore creates the new directories between
> these two steps. Hence any new directory created in a renamed
> directory cannot be restored from a cumulative dump because when
> restore tries to create the new directory neither the old directory
> path nor the new directory path exists.  Hence it silently drops the
> new directory, resulting in subsequent errors tryin gto restore
> files within that new directory.
> 
> The simple fix - just change the order of operations
> in tree_post() so that new directories are created after all the
> renames are processed - is not useful. All that does is break the
> case of renames into newly created directories.
> 
> However, because the making of directories that already exist or
> can't be made silently fails, and the create process does not modify
> the internal directory tree, we can run the directory creation
> function multiple times. Hence we can run directory creation both
> before and after the directory rename step, hence ensuring both
> new parents and new child directories are created appropriately.
> 
> This still may not be sufficient for complex directory
> reorganisations, but it does address the reported problem in a
> manner that is unlikely to cause regresssions. This is important,
> because this code has not changed at all since it was first publicly
> released in early 2001. Hence the minimum change we can make to fix
> the reported problem is the least risky approach we can take.
> 
> Signed-off-by: Dave Chinner <dchinner@redhat.com>
> ---
>  restore/tree.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++++++----------
>  1 file changed, 51 insertions(+), 10 deletions(-)
> 
> diff --git a/restore/tree.c b/restore/tree.c
> index 0336e77..8e6fab1 100644
> --- a/restore/tree.c
> +++ b/restore/tree.c
> @@ -1163,15 +1163,35 @@ tree_subtree_parse( bool_t sensepr, char *path )
>  	return BOOL_TRUE;
>  }
>  
> -/* tree_post - called after the dirdump has been applied.
> - * first phase is to eliminate all unreferenced dirents.
> - * done by recursive depth-wise descent of the tree. on the way
> - * up, unlink or orphan unreferenced nondirs, unlink unreferenced
> - * dirs, orphan dirs to be renamed. if a dir is unreferenced, but
> +/* tree_post - called after the dirdump has been applied.  first phase is to
> + * eliminate all unreferenced dirents.  done by recursive depth-wise descent of
> + * the tree. on the way up, unlink or orphan unreferenced nondirs, unlink
> + * unreferenced dirs, orphan dirs to be renamed. if a dir is unreferenced, but
>   * contains referenced dirents, orphan those dirents. orphan unreferenced
>   * nondirs if they are the last link to an inode referenced but not real
>   * somewhere else in the tree. next, make new directories. then rename
>   * directories. finally, create hardlinks from orphanage.
> + *
> + * Note: the way renamed directories are handled by first orphaning them leads
> + * to a chicken and egg problem - the directory does not exist when we try to
> + * make a new directory inside the renamed directory destination. This fails
> + * silently, leaving us with ENOENT errors when trying to restore files within
> + * that new directory.
> + *
> + * To prevent this from happening, we need to create new subdirectories *after*
> + * we have processed all the renamed directories. However, so that the renames
> + * succeed, we also have to create any new parent directories that the renames
> + * depend on.
> + *
> + * Hence we have to do two mkdir passes: one before the renames to create new
> + * ancestors for rename destinations and one after the rename to create new
> + * children in rename destinations.
> + *
> + * NOTE: a simple before/after creation as done below may be too simple for
> + * complex directory structure manipulations. e.g. rename into a new child in
> + * the destination of another rename. This may have to become an iterative loop
> + * that runs until all renames and mkdirs are resolved. We'll cross that bridge
> + * when we need to, not now.
>   */
>  static bool_t noref_elim_recurse( nh_t parh,
>  				  nh_t cldh,
> @@ -1216,7 +1236,14 @@ tree_post( char *path1, char *path2 )
>  		}
>  	}
>  
> -	/* make new directories
> +#ifdef TREE_CHK
> +	assert( tree_chk( ));
> +#endif /* TREE_CHK */
> +
> +	/*
> +	 * make new directories to ensure rename destination ancestors are
> +	 * present before attempting the renames. This will silently skip all
> +	 * the creations that are in rename destinations.
>  	 */
>  	mlog( MLOG_DEBUG | MLOG_TREE,
>  	      "making new directories\n" );
> @@ -1228,10 +1255,6 @@ tree_post( char *path1, char *path2 )
>  		return BOOL_FALSE;
>  	}
>  
> -#ifdef TREE_CHK
> -	assert( tree_chk( ));
> -#endif /* TREE_CHK */
> -
>  	/* rename directories
>  	 */
>  	if ( ! persp->p_fullpr ) {
> @@ -1250,6 +1273,24 @@ tree_post( char *path1, char *path2 )
>  	assert( tree_chk( ));
>  #endif /* TREE_CHK */
>  
> +	/*
> +	 * Repeat making new directories to create directories in rename
> +	 * destinations that were skipped in the first pass.
> +	 */
> +	mlog( MLOG_DEBUG | MLOG_TREE,
> +	      "making new directories in renamed ancestors\n" );
> +	rootp = Node_map( persp->p_rooth );
> +	cldh = rootp->n_cldh;
> +	Node_unmap( persp->p_rooth, &rootp );
> +	ok = mkdirs_recurse( persp->p_rooth, cldh, path1 );
> +	if ( ! ok ) {
> +		return BOOL_FALSE;
> +	}
> +
> +#ifdef TREE_CHK
> +	assert( tree_chk( ));
> +#endif /* TREE_CHK */
> +
>  	/* process hard links
>  	 */
>  	if ( ! persp->p_fullpr ) {
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Bug (?) : cumulative xfsrestore does not restore files and folders in a directory which was renamed

[Damien Gombault]

[-- Attachment #1: Type: text/plain, Size: 3168 bytes --]

Hi.

Thank you for the patch.

The patch fixes simple cases (like my reproducer) but it does not fix
restoration of more complex directory structures : I have tested the 
patch on my
real data, it fixes some but not all warnings (a user renamed/moved a 
lots of
files and directory between the 2 dumps).

Here is another testcase which fails (with the patch) :

mkdir dira
mkdir dira/dirc
touch dira/dirc/filea
mkdir dirb

.
├── dira
│   └── dirc
│       └── filea
└── dirb

Make a level 0 dump.

mv dirb dira/dirB
mv dira/dirc/ dira/dirB/dirC
touch dira/dirB/dirC/fileb

.
└── dira
     └── dirB
         └── dirC
             ├── filea
             └── fileb

Make a level 1 dump.

Restore level 0 then level 1 dump, you will get :

xfsrestore: directory post-processing
xfsrestore: WARNING: unable to set secure extended attribute for 
dira/dirB: No
such file or directory (2)
xfsrestore: restoring non-directory files
xfsrestore: WARNING: open of dira/dirB/dirC/fileb failed: No such file or
directory: discarding ino 524386
xfsrestore: WARNING: unable to set secure extended attribute for
dira/dirB/dirC/fileb: No such file or directory (2)
xfsrestore: WARNING: path_to_handle of dira/dirB/dirC failed:No such file or
directory
xfsrestore: could not set access and modification times of 
dira/dirB/dirC: No
such file or directory
xfsrestore: chown (uid=0, gid=0) dira/dirB/dirC failed: No such file or
directory
xfsrestore: chmod dira/dirB/dirC failed: No such file or directory
xfsrestore: WARNING: attempt to set extended attributes (xflags 0x80000000,
extsize = 0x0, projid = 0x0) of dira/dirB/dirC failed: Bad file descriptor
xfsrestore: WARNING: path_to_handle of dira/dirB failed:No such file or
directory
xfsrestore: could not set access and modification times of dira/dirB: No 
such
file or directory
xfsrestore: chown (uid=0, gid=0) dira/dirB failed: No such file or directory
xfsrestore: chmod dira/dirB failed: No such file or directory
xfsrestore: WARNING: attempt to set extended attributes (xflags 0x80000000,
extsize = 0x0, projid = 0x0) of dira/dirB failed: Bad file descriptor
xfsrestore: WARNING: unable to rmdir /mnt/test//orphanage: Directory not 
empty

Directory dirB is placed in orphanage folder and fileb is not restored :

.
├── dira
├── orphanage
│   └── 1069152.734839917
│       └── dirC
│           └── filea
└── xfsrestorehousekeepingdir
     ├── dirattr
     ├── dirextattr
     ├── namreg
     ├── state
     └── tree

--
Damien Gombault

Le 23/06/2016 à 03:42, Dave Chinner a écrit :
> On Thu, Jun 23, 2016 at 08:23:28AM +1000, Dave Chinner wrote:
>> On Thu, Jun 23, 2016 at 08:09:59AM +1000, Dave Chinner wrote:
>>> But it seems that dirb hasn't been created before the file in it
>>> is being restored. THis can happen because the inventory is not
>>> correct, whichmay in fact be a problem with dump rather than
>>> restore...
>>>
>>> I'll have a bit of a play around here, see if I can reproduce it.
>> Yes, i can reproduce it, so I'll have a deeper look.
> Can you try this patch?
>
> -Dave


[-- Attachment #2: damien_gombault.vcf --]
[-- Type: text/x-vcard, Size: 437 bytes --]

begin:vcard
fn:Damien Gombault
n:Gombault;Damien
org;quoted-printable:GIP Recia =E2=88=92 Centre de ressources r=C3=A9gional du num=C3=A9rique
adr;quoted-printable;dom:Parc d'Activit=C3=A9s les Aulnaies;;151, Rue de la Juine;Olivet;;45160
email;internet:damien.gombault@recia.fr
title;quoted-printable:Responsable des syst=C3=A8mes informatiques
tel;work:02 38 42 14 71
tel;cell:06 42 62 78 19
url:http://recia.fr
version:2.1
end:vcard


[-- Attachment #3: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Bug (?) : cumulative xfsrestore does not restore files and folders in a directory which was renamed

[Dave Chinner]

On Fri, Jun 24, 2016 at 03:51:57PM +0200, Damien Gombault wrote:
> Hi.
> 
> Thank you for the patch.
> 
> The patch fixes simple cases (like my reproducer) but it does not fix
> restoration of more complex directory structures : I have tested the
> patch on my
> real data, it fixes some but not all warnings (a user renamed/moved
> a lots of
> files and directory between the 2 dumps).
> 
> Here is another testcase which fails (with the patch) :
> 
> mkdir dira
> mkdir dira/dirc
> touch dira/dirc/filea
> mkdir dirb
> 
> .
> ├── dira
> │   └── dirc
> │       └── filea
> └── dirb
> 
> Make a level 0 dump.
> 
> mv dirb dira/dirB
> mv dira/dirc/ dira/dirB/dirC
> touch dira/dirB/dirC/fileb
> 
> .
> └── dira
>     └── dirB
>         └── dirC
>             ├── filea
>             └── fileb

Well, yes. I specifically said in the patch description that it
doesn't fix those nested dependency cases and that much deeper
surgery is needed to handle those cases.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs