dump/restore not supported for ext4

dump/restore not supported for ext4

[Justin Piszcz]

Hi,

I should have researched it more:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=511651
https://lists.ubuntu.com/archives/universe-bugs/2009-June/098729.html

Looks like it's broken, I agree with the reporters, dump should abort if 
it is dealing with an ext4 filesystem, since you cannot restore data from 
an ext4 dump.

Justin.

Re: dump/restore not supported for ext4

[Eric Sandeen]

Justin Piszcz wrote:
> Hi,
> 
> I should have researched it more:
> 
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=511651
> https://lists.ubuntu.com/archives/universe-bugs/2009-June/098729.html
> 
> Looks like it's broken, I agree with the reporters, dump should abort if
> it is dealing with an ext4 filesystem, since you cannot restore data
> from an ext4 dump.

does dump still read the mounted block device directly?  I'm not familiar
with the internals (need to look, I guess) but if so, I wonder how that
+ delalloc get along...

-Eric

> Justin.
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: dump/restore not supported for ext4

[tytso]

On Thu, Mar 04, 2010 at 05:26:22PM -0600, Eric Sandeen wrote:
> Justin Piszcz wrote:
> > Hi,
> > 
> > I should have researched it more:
> > 
> > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=511651
> > https://lists.ubuntu.com/archives/universe-bugs/2009-June/098729.html
> > 
> > Looks like it's broken, I agree with the reporters, dump should abort if
> > it is dealing with an ext4 filesystem, since you cannot restore data
> > from an ext4 dump.
> 
> does dump still read the mounted block device directly?  I'm not familiar
> with the internals (need to look, I guess) but if so, I wonder how that
> + delalloc get along...

It does read the mounted block device directly, and so it's certainly
not a _recommended_ way to back up your ext4 filesystem.  It should
work, though, since it just uses the high-level libext2fs functions
--- and a while back, I think I did a quick test and found that it
really did work.  So I'm not sure what broke, but it might not be that
hard to fix.  That being said, it may not be worth it to fix it, since
with delayed allocation, backups using dump will be even more
unreliable than they were before.

						- Ted

Re: dump/restore not supported for ext4

[Gertjan van Wingerde]

On 03/04/10 23:05, Justin Piszcz wrote:
> Hi,
> 
> I should have researched it more:
> 
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=511651
> https://lists.ubuntu.com/archives/universe-bugs/2009-June/098729.html
> 
> Looks like it's broken, I agree with the reporters, dump should abort if
> it is dealing with an ext4 filesystem, since you cannot restore data
> from an ext4 dump.
> 
> Justin.
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Hi Justin,

It seems you are using an outdated version of dump.
The latest version of dump, 0.4b42 does contain support for ext4.
I've been using it for the last 8 months for my backups of ext4 without
any problems and restoring is not issue at all with this version.

---
Gertjan.

Re: dump/restore not supported for ext4

[Justin Piszcz]

On Fri, 5 Mar 2010, Gertjan van Wingerde wrote:

> On 03/04/10 23:05, Justin Piszcz wrote:

[ .. ]

> Hi Justin,
>
> It seems you are using an outdated version of dump.
> The latest version of dump, 0.4b42 does contain support for ext4.
> I've been using it for the last 8 months for my backups of ext4 without
> any problems and restoring is not issue at all with this version.

Hello,

I tried the following package in sid:
http://http.us.debian.org/debian/pool/main/d/dump/dump_0.4b42-2_amd64.deb

Per the changelog:

Changes between versions 0.4b41 and 0.4b42 (released June 18, 2009)
===================================================================
18. Add (preliminary) ext4 support - thanks to libext2fs which does
   all the job for us. Thanks to Gertjan van Wingerde
   <gwingerde@gmail.com> for the patch.

Why Debian does not include this in testing is beyond me..??

BTW: The man page needs to be fixed in dump as well then (in the new version)

Here:
        dump - ext2/3 filesystem backup
Here:
        Dump examines files on an ext2/3 filesystem and determines which  files
Here:
        ext2/3 filesystems.  Specifically, it does not work with  FAT  filesys-
Here:
        disk block and sector number and the ext2/3 logical  block  number.  It


--

OTHER/TESTING:

SIZES:

-rw-r--r-- 1 root root 11900416000 2010-03-05 05:31 root-without-z.ext4dump
-rw-r--r-- 1 root root 4670305765 2010-03-05 05:54 root_with-j=1.ext4dump
-rw-r--r-- 1 root root 4801444167 2010-03-05 04:54 root_with-z=1.ext4dump
-rw-r--r-- 1 root root 4759018091 2010-03-05 04:58 root_with-z=2.ext4dump
-rw-r--r-- 1 root root 4732663941 2010-03-05 05:02 root_with-z=3.ext4dump
-rw-r--r-- 1 root root 4644973699 2010-03-05 05:06 root_with-z=4.ext4dump
-rw-r--r-- 1 root root 4598494695 2010-03-05 05:09 root_with-z=5.ext4dump
-rw-r--r-- 1 root root 4585625035 2010-03-05 05:13 root_with-z=6.ext4dump
-rw-r--r-- 1 root root 4579357817 2010-03-05 05:18 root_with-z=7.ext4dump
-rw-r--r-- 1 root root 4574024703 2010-03-05 05:22 root_with-z=8.ext4dump
-rw-r--r-- 1 root root 4573194841 2010-03-05 05:27 root_with-z=9.ext4dump

TIMES:

-no compression
     dump:   4.87user 34.60system 3:36.72elapsed 18%CPU

-zlib
dump -z1: 241.27user 60.82system 3:35.08elapsed 140%CPU
dump -z2: 248.17user 60.99system 3:41.85elapsed 139%CPU
dump -z3: 257.37user 60.77system 3:47.90elapsed 139%CPU
dump -z4: 326.30user 63.92system 3:54.26elapsed 166%CPU
dump -z5: 346.77user 60.61system 3:50.25elapsed 176%CPU
dump -z6: 367.04user 61.06system 4:02.27elapsed 176%CPU
dump -z7: 388.87user 62.35system 4:10.59elapsed 180%CPU
dump -z8: 454.96user 60.92system 4:28.70elapsed 191%CPU
dump -z9: 539.88user 64.73system 5:06.22elapsed 197%CPU

-bzlib (further testing not needed after -j1, took a long time, got bigger)
dump -j1: 3052.01user 112.34system 20:07.96elapsed 261%CPU

Suffice to say..

The 7z took about 45-60minutes, bzip2 after that and gzip after that.
It appears dump w/-z9 is the best option pertaining to time/space (VERY FAST)

-rw-r--r-- 1 root root  4670305765 2010-03-05 05:54 root_with-j=1.ext4dump
-rw-r--r-- 1 root root 11900416000 2010-03-05 05:31 root-without-z.ext4dump
-rw-r--r-- 1 root root  3032284032 2010-03-05 07:19 root-without-z.ext4dump.7z
-rw-r--r-- 1 root root  3700128170 2010-03-05 06:41 root-without-z.ext4dump.bz2
-rw-r--r-- 1 root root  4079857439 2010-03-05 06:29 root-without-z.ext4dump.gz
-rw-r--r-- 1 root root  4801444167 2010-03-05 04:54 root_with-z=1.ext4dump
-rw-r--r-- 1 root root  4759018091 2010-03-05 04:58 root_with-z=2.ext4dump
-rw-r--r-- 1 root root  4732663941 2010-03-05 05:02 root_with-z=3.ext4dump
-rw-r--r-- 1 root root  4644973699 2010-03-05 05:06 root_with-z=4.ext4dump
-rw-r--r-- 1 root root  4598494695 2010-03-05 05:09 root_with-z=5.ext4dump
-rw-r--r-- 1 root root  4585625035 2010-03-05 05:13 root_with-z=6.ext4dump
-rw-r--r-- 1 root root  4579357817 2010-03-05 05:18 root_with-z=7.ext4dump
-rw-r--r-- 1 root root  4574024703 2010-03-05 05:22 root_with-z=8.ext4dump
-rw-r--r-- 1 root root  4573194841 2010-03-05 05:27 root_with-z=9.ext4dump

And the most important part!

Restoring from backup from two different systems, 12GB and ~30GB:

SYSTEM_1

p34:/x2/bup/p34/2010-03-05/restore# restore -f ../p34-root-2010-03-05.ext4dump -r 
Dump tape is compressed.
./home/user/.local/share/applications/defaults.list: (inode 5898660) not found on tape
expected next file 5770835, got 5770833
expected next file 5898665, got 5898661
p34:/x2/bup/p34/2010-03-05/restore#

For the most part, it worked, obviously this was on a live system so I believe
this is to be expected?

SYSTEM_2 (EXT4 DUMP OVER SSH)

ssh -l root l1 "dump -0f - /boot -z9 -L $today" > "$destdir/$bfile"
ssh -l root l1 "dump -0f - / -z9 -L $today" > "$destdir/$rfile"

p34:/x2/bup/l1/2010-03-05/restore# /usr/bin/time restore -f ../l1-root-2010-03-05.ext4dump  -r
Dump tape is compressed.
112.97user 27.86system 6:18.10elapsed 37%CPU (0avgtext+0avgdata 152432maxresident)k
0inputs+0outputs (5major+12756minor)pagefaults 0swaps
p34:/x2/bup/l1/2010-03-05/restore#

Quite nice!

Thanks,

Justin.

Re: dump/restore not supported for ext4

[Bdale Garbee]

[-- Attachment #1: Type: text/plain, Size: 1528 bytes --]

On Fri, 5 Mar 2010 09:27:52 -0500 (EST), Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
> I tried the following package in sid:
> http://http.us.debian.org/debian/pool/main/d/dump/dump_0.4b42-2_amd64.deb
> 
> Why Debian does not include this in testing is beyond me..??

I just looked and it appears that this version of dump has never been
successfully built on mips or mipsel, which blocks it from being
promoted to testing. Checking the autobuilder build logs, it looks like
the root cause may be a toolchain problem, though it's an error message
I've never seen before. 

https://buildd.debian.org/fetch.cgi?&pkg=dump&ver=0.4b42-2&arch=mips&stamp=1259085131&file=log

Chasing down that error message, it looks like this is indeed believed
to a toolchain issue, fixed by:

http://gcc.gnu.org/ml/gcc-patches/2009-09/msg02070.html

I don't know, offhand, if this patch is reflected in the current Debian
gcc packages.  If so, then perhaps I just need to request that the mips
and mipsel autobuilders retry the builds?  CC'ing the debian-gcc team
for advice.

> BTW: The man page needs to be fixed in dump as well then (in the new
> version)

Thanks for noticing those!  I've made those changes in my git repo for
the next Debian package upload.  Hopefully Stelian will also fix those
in his next release, I see you've copied him on this.

> Quite nice!

Thanks for testing this so thoroughly.  I myself back up two servers
with ext4 file systems using amanda and dump quite happily.

Bdale

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: dump/restore not supported for ext4

[Stelian Pop]

On Fri, Mar 05, 2010 at 09:21:31AM -0700, Bdale Garbee wrote:

> Thanks for noticing those!  I've made those changes in my git repo for
> the next Debian package upload.  Hopefully Stelian will also fix those
> in his next release, I see you've copied him on this.

Done !

Stelian.
-- 
Stelian Pop <stelian@popies.net>