Unable to xfsdump/xfsrestore filesystem

Unable to xfsdump/xfsrestore filesystem

[David Brown]

I am using xfsdump with multiple levels to backup my main system
(Fedora f31, xfs root and home on LVM).  Kernel is
5.5.6-201.fc31.x86_64.

When doing a test restore of my backup, when I reach my level 2
backup, I get the following warnings:

    xfsrestore: directory post-processing
    xfsrestore: WARNING: unable to rename dir orphanage/422178422.2232121414 to dir <<path>>/b-cloud: No such file or directory

In addition, I get hundreds of these.  They seem to all be related to
that same directory.

    xfsrestore: restoring non-directory files
    xfsrestore: WARNING: open of orphanage/422178422.2232121414/modules/atmel/cmake_install.cmake failed: No such file or directory: discarding ino 2833282
    xfsrestore: WARNING: open of orphanage/422178422.2232121414/modules/atmel/asf/common/cmake_install.cmake failed: No such file or directory: discarding ino 2833283

When I verify the restore (using https://github.com/d3zd3z/rsure), I
indeed find that <<path>>/b-cloud is missing, but another directory
next to it is still present that should be removed.

Restores of subsequent print similar messages (and don't restore the
directory), until one of them dies with this:

    xfsrestore: WARNING: unable to rename dir orphanage/422178422.2232121414 to dir <<path>>/b-cloud: No such file or directory
    xfsrestore: node.c:539: node_map: Assertion `nh != NH_NULL' failed.

Any ideas on how to debug this?

Thanks,
David

Re: Unable to xfsdump/xfsrestore filesystem

[Eric Sandeen]

On 3/3/20 8:50 AM, David Brown wrote:
> I am using xfsdump with multiple levels to backup my main system
> (Fedora f31, xfs root and home on LVM).  Kernel is
> 5.5.6-201.fc31.x86_64.
> 
> When doing a test restore of my backup, when I reach my level 2
> backup, I get the following warnings:
> 
>     xfsrestore: directory post-processing
>     xfsrestore: WARNING: unable to rename dir orphanage/422178422.2232121414 to dir <<path>>/b-cloud: No such file or directory
> 
> In addition, I get hundreds of these.  They seem to all be related to
> that same directory.
> 
>     xfsrestore: restoring non-directory files
>     xfsrestore: WARNING: open of orphanage/422178422.2232121414/modules/atmel/cmake_install.cmake failed: No such file or directory: discarding ino 2833282
>     xfsrestore: WARNING: open of orphanage/422178422.2232121414/modules/atmel/asf/common/cmake_install.cmake failed: No such file or directory: discarding ino 2833283
> 
> When I verify the restore (using https://github.com/d3zd3z/rsure), I
> indeed find that <<path>>/b-cloud is missing, but another directory
> next to it is still present that should be removed.
> 
> Restores of subsequent print similar messages (and don't restore the
> directory), until one of them dies with this:
> 
>     xfsrestore: WARNING: unable to rename dir orphanage/422178422.2232121414 to dir <<path>>/b-cloud: No such file or directory
>     xfsrestore: node.c:539: node_map: Assertion `nh != NH_NULL' failed.
> 
> Any ideas on how to debug this?

xfsdump/restore is a beast to debug.  Are you using the latest version of xfsdump package?

Can you put the entire restore session logs somewhere, possibly with verbosity increased?

(Any chance you have the dump logs, still?)

-Eric

Re: Unable to xfsdump/xfsrestore filesystem

[Dave Chinner]

On Tue, Mar 03, 2020 at 09:50:00AM -0700, David Brown wrote:
> I am using xfsdump with multiple levels to backup my main system
> (Fedora f31, xfs root and home on LVM).  Kernel is
> 5.5.6-201.fc31.x86_64.
> 
> When doing a test restore of my backup, when I reach my level 2
> backup, I get the following warnings:
> 
>     xfsrestore: directory post-processing
>     xfsrestore: WARNING: unable to rename dir orphanage/422178422.2232121414 to dir <<path>>/b-cloud: No such file or directory

The orphanage is a special xfsrestore thing. files and directories
without any tree links at the time they are restored from the media
file are placed into the orphanage. When a link to that object in
the orphanage is found, it moves it into place.

However, directory post processing also removes stale tree entries.
Hence you can get files in the orphanage that have no directories
that point to them at all, and they then get discarded because there
is nowhere to restore them to.

This process can be recursive, as directory post processing can move
entries to the orphanage and vice versa, meaning that files that
were skipped in the restore because they had not tree linkage now
have their parent directory in place and so can now be restored.
There's a chicken-and-egg problem in this algorithm, too, which can
lead to an unresolvable recursion loop, so restore may not be able
to resolve everything correctly form such a media file....

> In addition, I get hundreds of these.  They seem to all be related to
> that same directory.
> 
>     xfsrestore: restoring non-directory files
>     xfsrestore: WARNING: open of orphanage/422178422.2232121414/modules/atmel/cmake_install.cmake failed: No such file or directory: discarding ino 2833282
>     xfsrestore: WARNING: open of orphanage/422178422.2232121414/modules/atmel/asf/common/cmake_install.cmake failed: No such file or directory: discarding ino 2833283

Yep, this is typical of a dump taken on a live, changing filesystem.
i.e. This is not an xfsrestore problem but a result of changing
filesystem structure while xfsdump is running.

That is, xfsdump has scanned the filesystem, found all the
files/dirs that need to be backed up, then gone and read all the
data into the media file. It does a separate directory traversal to
write the directory heirarchy into the dump inventory. As a result
of the time time difference between the file data backup vs the
directory struct backup, the dump media file contains files that
don't exist in the directory tree and the directory tree can contain
files that don't exist in the dump.

Hence the backup data in the media file can be inconsistent as it is
not a point in time snapshot of the live filesystem, and the only
time you are going to see that inconsistency is whey you go to
restore the media file.

> When I verify the restore (using https://github.com/d3zd3z/rsure), I
> indeed find that <<path>>/b-cloud is missing, but another directory
> next to it is still present that should be removed.

Yup, that can happen on a live filesystem backup. If you want
a guaranteed consistent backup, then you need to run xfsdump on a
static snapshot of the filesystem and not the live filesystem
itself.

> Restores of subsequent print similar messages (and don't restore the
> directory), until one of them dies with this:
> 
>     xfsrestore: WARNING: unable to rename dir orphanage/422178422.2232121414 to dir <<path>>/b-cloud: No such file or directory
>     xfsrestore: node.c:539: node_map: Assertion `nh != NH_NULL' failed.
> 
> Any ideas on how to debug this?

The node freeing code has been passed a null pointer, which
indicates there's an inconsistency in the internal directory tree
that restore is building. Smells like a double-free is occuring.

The usual is in order: turn the debugging logging all the way up,
run it under gdb to find the path that leads to the crash, make your
eyeballs bleed by reading the xfsdump code...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Unable to xfsdump/xfsrestore filesystem

[David Brown]

On Wed, Mar 04, 2020 at 09:18:21AM +1100, Dave Chinner wrote:

> > In addition, I get hundreds of these.  They seem to all be related to
> > that same directory.
> > 
> >     xfsrestore: restoring non-directory files
> >     xfsrestore: WARNING: open of orphanage/422178422.2232121414/modules/atmel/cmake_install.cmake failed: No such file or directory: discarding ino 2833282
> >     xfsrestore: WARNING: open of orphanage/422178422.2232121414/modules/atmel/asf/common/cmake_install.cmake failed: No such file or directory: discarding ino 2833283
> 
> Yep, this is typical of a dump taken on a live, changing filesystem.
> i.e. This is not an xfsrestore problem but a result of changing
> filesystem structure while xfsdump is running.

This is not the case.  I am creating lvm snapsnots and dumping those
with XFS dump.  Nothing should be changing the filesystem during the
xfsdump.

David