[PATCH 0/4] Add handy fs move features

[PATCH 0/4] Add handy fs move features

[Phillip Susi]

The following patches add some handy features to e2image related
to moving the filesystem.  They can be useful for converting between
disks and images where one is partitioned and the other is not.
It will also allow gparted to more efficiently move ext[234]
partitions by skipping unused blocks.


Phillip Susi (4):
  e2image: add offset switches
  e2image: perform in-place move
  e2image: add progress indicator
  e2image: handle SIGINT safely

 misc/e2image.8.in |  45 +++++++++++++-
 misc/e2image.c    | 182 ++++++++++++++++++++++++++++++++++++++++++++++++++----
 2 files changed, 213 insertions(+), 14 deletions(-)

-- 
1.8.1.2

[PATCH 1/4] e2image: add offset switches

[Phillip Susi]

Add -o and -O switches to specify the offset where the source
and destination filesystems start.  This is useful if you have
an image of a partitioned disk or wish to create one.

Signed-off-by: Phillip Susi <psusi@ubuntu.com>
---
 misc/e2image.8.in | 36 ++++++++++++++++++++++++++++++++++++
 misc/e2image.c    | 24 +++++++++++++++++++-----
 2 files changed, 55 insertions(+), 5 deletions(-)

diff --git a/misc/e2image.8.in b/misc/e2image.8.in
index 84b9729..86d3dfc 100644
--- a/misc/e2image.8.in
+++ b/misc/e2image.8.in
@@ -10,6 +10,14 @@ e2image \- Save critical ext2/ext3/ext4 filesystem metadata to a file
 [
 .B \-rsIQa
 ]
+[
+.B \-o
+.I source_offset
+]
+[
+.B \-O
+.I dest_offset
+]
 .I device
 .I image-file
 .SH DESCRIPTION
@@ -191,6 +199,34 @@ give an image that is suitable to use to clone the entire FS or
 for backup purposes.  Note that this option only works with the
 raw or QCOW2 formats.
 .PP
+.SH OFFSETS
+Normally a filesystem starts at the beginning of a partition, and
+.B e2image
+is run on the partition.  When working with image files, you don't
+have the option of using the partition device, so you can specify
+the offset where the filesystem starts directly with the
+.B \-o
+option.  Similarly the
+.B \-O
+option specifies the offset that should be seeked to in the destination
+before writing the filesystem.
+.PP
+For example, if you have a
+.B dd
+image of a whole hard drive that contains an ext2 fs in a partition
+starting at 1 MiB, you can clone that fs with:
+.PP
+.br
+\	\fBe2image \-aro 1048576 img /dev/sda1\fR
+.br
+.PP
+Or you can clone a fs into an image file, leaving room in the first
+MiB for a partition table with:
+.PP
+.br
+\	\fBe2image -arO 1048576 /dev/sda1 img\fR
+.br
+.PP
 .SH AUTHOR
 .B e2image
 was written by Theodore Ts'o (tytso@mit.edu).
diff --git a/misc/e2image.c b/misc/e2image.c
index 885a794..cf486c0 100644
--- a/misc/e2image.c
+++ b/misc/e2image.c
@@ -55,6 +55,7 @@ char * device_name = NULL;
 char all_data;
 char output_is_blk;
 /* writing to blk device: don't skip zeroed blocks */
+blk64_t source_offset, dest_offset;
 
 static void lseek_error_and_exit(int errnum)
 {
@@ -87,7 +88,7 @@ static int get_bits_from_size(size_t size)
 
 static void usage(void)
 {
-	fprintf(stderr, _("Usage: %s [-rsIQa] device image_file\n"),
+	fprintf(stderr, _("Usage: %s [-rsIQa] [-o source_offset] [-O dest_offset] device image_file\n"),
 		program_name);
 	exit (1);
 }
@@ -1269,7 +1270,7 @@ int main (int argc, char ** argv)
 	if (argc && *argv)
 		program_name = *argv;
 	add_error_table(&et_ext2_error_table);
-	while ((c = getopt(argc, argv, "rsIQa")) != EOF)
+	while ((c = getopt(argc, argv, "rsIQao:O:")) != EOF)
 		switch (c) {
 		case 'I':
 			flags |= E2IMAGE_INSTALL_FLAG;
@@ -1290,6 +1291,12 @@ int main (int argc, char ** argv)
 		case 'a':
 			all_data = 1;
 			break;
+		case 'o':
+			source_offset = strtoull(optarg, NULL, 0);
+			break;
+		case 'O':
+			dest_offset = strtoull(optarg, NULL, 0);
+			break;
 		default:
 			usage();
 		}
@@ -1317,9 +1324,11 @@ int main (int argc, char ** argv)
 			goto skip_device;
 		}
 	}
-
-	retval = ext2fs_open (device_name, open_flag, 0, 0,
-			      unix_io_manager, &fs);
+	char *options;
+	asprintf (&options, "offset=%llu", source_offset);
+	retval = ext2fs_open2 (device_name, options, open_flag, 0, 0,
+			       unix_io_manager, &fs);
+	free (options);
         if (retval) {
 		com_err (program_name, retval, _("while trying to open %s"),
 			 device_name);
@@ -1338,6 +1347,11 @@ skip_device:
 			exit(1);
 		}
 	}
+	if (dest_offset)
+		if (ext2fs_llseek (fd, dest_offset, SEEK_SET) < 0) {
+			perror("ext2fs_llseek");
+			exit(1);
+		}
 
 	if ((img_type & E2IMAGE_QCOW2) && (fd == 1)) {
 		com_err(program_name, 0, "QCOW2 image can not be written to "
-- 
1.8.1.2

Re: [PATCH 1/4] e2image: add offset switches

[Theodore Ts'o]

On Tue, Oct 08, 2013 at 11:56:29AM -0400, Phillip Susi wrote:
> Add -o and -O switches to specify the offset where the source
> and destination filesystems start.  This is useful if you have
> an image of a partitioned disk or wish to create one.
> 
> Signed-off-by: Phillip Susi <psusi@ubuntu.com>

Specifying a dest_offset only makes sense in raw mode, no?

	     		      	    	  - Ted

Re: [PATCH 1/4] e2image: add offset switches

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/15/2013 11:32 PM, Theodore Ts'o wrote:
> On Tue, Oct 08, 2013 at 11:56:29AM -0400, Phillip Susi wrote:
>> Add -o and -O switches to specify the offset where the source and
>> destination filesystems start.  This is useful if you have an
>> image of a partitioned disk or wish to create one.
>> 
>> Signed-off-by: Phillip Susi <psusi@ubuntu.com>
> 
> Specifying a dest_offset only makes sense in raw mode, no?

I think it makes sense in qcow2 mode, but it seems it doesn't work and
isn't trivial to fix, so I'll disable it for now.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSryqNAAoJEI5FoCIzSKrwJi0H/jA0887Dku5kMrpga3fTJCf+
rjGe93qHoqf6DOprj+2XI4cTJOY572729QrxKqbtlicY5F7lbB/9JKXn4VNcFJEk
6+1iQHh5mmFMP9g2hWxQPgfm93UAUYlC/pWyQvwTeUjKBp+EPYmoUtrDIZYpc79S
eFIsFt2TwduQ3zcwwLLOx2knQNyic9e3mwPF0NeIXbByfAW6hhPHLh1eg4zbTcJH
GYQqNyjk0vS0qmW+CBfAxyH4eYk6JgXtmg8c+8eePwlQLpbIoYW84cs6+yWbAu1y
1tVeqi2YTuYk+kU0t5DEj5sHoYzSXTZCukO8G/cThrhEMVnG1pwNRH8egjoyyYo=
=Ty8d
-----END PGP SIGNATURE-----

[PATCH 1/4] e2image: add offset switches

[Phillip Susi]

Add -o and -O switches to specify the offset where the source
and destination filesystems start.  This is useful if you have
an image of a partitioned disk or wish to create one.

Signed-off-by: Phillip Susi <psusi@ubuntu.com>
---
 misc/e2image.8.in | 36 ++++++++++++++++++++++++++++++++++++
 misc/e2image.c    | 30 ++++++++++++++++++++++++------
 2 files changed, 60 insertions(+), 6 deletions(-)

diff --git a/misc/e2image.8.in b/misc/e2image.8.in
index 84b9729..86d3dfc 100644
--- a/misc/e2image.8.in
+++ b/misc/e2image.8.in
@@ -10,6 +10,14 @@ e2image \- Save critical ext2/ext3/ext4 filesystem metadata to a file
 [
 .B \-rsIQa
 ]
+[
+.B \-o
+.I source_offset
+]
+[
+.B \-O
+.I dest_offset
+]
 .I device
 .I image-file
 .SH DESCRIPTION
@@ -191,6 +199,34 @@ give an image that is suitable to use to clone the entire FS or
 for backup purposes.  Note that this option only works with the
 raw or QCOW2 formats.
 .PP
+.SH OFFSETS
+Normally a filesystem starts at the beginning of a partition, and
+.B e2image
+is run on the partition.  When working with image files, you don't
+have the option of using the partition device, so you can specify
+the offset where the filesystem starts directly with the
+.B \-o
+option.  Similarly the
+.B \-O
+option specifies the offset that should be seeked to in the destination
+before writing the filesystem.
+.PP
+For example, if you have a
+.B dd
+image of a whole hard drive that contains an ext2 fs in a partition
+starting at 1 MiB, you can clone that fs with:
+.PP
+.br
+\	\fBe2image \-aro 1048576 img /dev/sda1\fR
+.br
+.PP
+Or you can clone a fs into an image file, leaving room in the first
+MiB for a partition table with:
+.PP
+.br
+\	\fBe2image -arO 1048576 /dev/sda1 img\fR
+.br
+.PP
 .SH AUTHOR
 .B e2image
 was written by Theodore Ts'o (tytso@mit.edu).
diff --git a/misc/e2image.c b/misc/e2image.c
index 885a794..f7f2e90 100644
--- a/misc/e2image.c
+++ b/misc/e2image.c
@@ -55,6 +55,7 @@ char * device_name = NULL;
 char all_data;
 char output_is_blk;
 /* writing to blk device: don't skip zeroed blocks */
+blk64_t source_offset, dest_offset;
 
 static void lseek_error_and_exit(int errnum)
 {
@@ -87,7 +88,7 @@ static int get_bits_from_size(size_t size)
 
 static void usage(void)
 {
-	fprintf(stderr, _("Usage: %s [-rsIQa] device image_file\n"),
+	fprintf(stderr, _("Usage: %s [-rsIQa] [-o source_offset] [-O dest_offset] device image_file\n"),
 		program_name);
 	exit (1);
 }
@@ -1269,7 +1270,7 @@ int main (int argc, char ** argv)
 	if (argc && *argv)
 		program_name = *argv;
 	add_error_table(&et_ext2_error_table);
-	while ((c = getopt(argc, argv, "rsIQa")) != EOF)
+	while ((c = getopt(argc, argv, "rsIQao:O:")) != EOF)
 		switch (c) {
 		case 'I':
 			flags |= E2IMAGE_INSTALL_FLAG;
@@ -1290,6 +1291,12 @@ int main (int argc, char ** argv)
 		case 'a':
 			all_data = 1;
 			break;
+		case 'o':
+			source_offset = strtoull(optarg, NULL, 0);
+			break;
+		case 'O':
+			dest_offset = strtoull(optarg, NULL, 0);
+			break;
 		default:
 			usage();
 		}
@@ -1301,7 +1308,11 @@ int main (int argc, char ** argv)
 					 "with raw or QCOW2 images.");
 		exit(1);
 	}
-
+	if ((source_offset || dest_offset) && img_type != E2IMAGE_RAW) {
+		com_err(program_name, 0,
+			"Offsets are only allowed with raw images.");
+		exit(1);
+	}
 	device_name = argv[optind];
 	image_fn = argv[optind+1];
 
@@ -1317,9 +1328,11 @@ int main (int argc, char ** argv)
 			goto skip_device;
 		}
 	}
-
-	retval = ext2fs_open (device_name, open_flag, 0, 0,
-			      unix_io_manager, &fs);
+	char *options;
+	asprintf (&options, "offset=%llu", source_offset);
+	retval = ext2fs_open2 (device_name, options, open_flag, 0, 0,
+			       unix_io_manager, &fs);
+	free (options);
         if (retval) {
 		com_err (program_name, retval, _("while trying to open %s"),
 			 device_name);
@@ -1338,6 +1351,11 @@ skip_device:
 			exit(1);
 		}
 	}
+	if (dest_offset)
+		if (ext2fs_llseek (fd, dest_offset, SEEK_SET) < 0) {
+			perror("ext2fs_llseek");
+			exit(1);
+		}
 
 	if ((img_type & E2IMAGE_QCOW2) && (fd == 1)) {
 		com_err(program_name, 0, "QCOW2 image can not be written to "
-- 
1.8.3.2

Re: [PATCH 1/4] e2image: add offset switches

[Theodore Ts'o]

On Mon, Dec 16, 2013 at 11:32:39AM -0500, Phillip Susi wrote:
> Add -o and -O switches to specify the offset where the source
> and destination filesystems start.  This is useful if you have
> an image of a partitioned disk or wish to create one.
> 
> Signed-off-by: Phillip Susi <psusi@ubuntu.com>

Thanks, applied.

					- Ted

[PATCH 2/4] e2image: perform in-place move

[Phillip Susi]

If given at least one offset and only one file, assume source
and dest are the same, and do an in place move.

Signed-off-by: Phillip Susi <psusi@ubuntu.com>
---
 misc/e2image.8.in |  3 +++
 misc/e2image.c    | 57 ++++++++++++++++++++++++++++++++++++++++++++++++-------
 2 files changed, 53 insertions(+), 7 deletions(-)

diff --git a/misc/e2image.8.in b/misc/e2image.8.in
index 86d3dfc..3d3aa18 100644
--- a/misc/e2image.8.in
+++ b/misc/e2image.8.in
@@ -227,6 +227,9 @@ MiB for a partition table with:
 \	\fBe2image -arO 1048576 /dev/sda1 img\fR
 .br
 .PP
+If you specify at least one offset, and only one file, an in-place
+move will be performed, allowing you to safely move the filesystem
+from one offset to another.
 .SH AUTHOR
 .B e2image
 was written by Theodore Ts'o (tytso@mit.edu).
diff --git a/misc/e2image.c b/misc/e2image.c
index cf486c0..2557eef 100644
--- a/misc/e2image.c
+++ b/misc/e2image.c
@@ -56,6 +56,7 @@ char all_data;
 char output_is_blk;
 /* writing to blk device: don't skip zeroed blocks */
 blk64_t source_offset, dest_offset;
+char move_mode;
 
 static void lseek_error_and_exit(int errnum)
 {
@@ -488,6 +489,9 @@ static void output_meta_data_blocks(ext2_filsys fs, int fd)
 	blk64_t		blk;
 	char		*buf, *zero_buf;
 	int		sparse = 0;
+	blk64_t		start = 0;
+	blk64_t		distance = 0;
+	blk64_t		end = ext2fs_blocks_count(fs->super);
 
 	retval = ext2fs_get_mem(fs->blocksize, &buf);
 	if (retval) {
@@ -499,7 +503,19 @@ static void output_meta_data_blocks(ext2_filsys fs, int fd)
 		com_err(program_name, retval, "while allocating buffer");
 		exit(1);
 	}
-	for (blk = 0; blk < ext2fs_blocks_count(fs->super); blk++) {
+	/* when doing an in place move to the right, you can't start
+	   at the beginning or you will overwrite data, so instead
+	   divide the fs up into distance size chunks and write them
+	   in reverse. */
+	if (move_mode && dest_offset > source_offset) {
+		distance = (dest_offset - source_offset) / fs->blocksize;
+		if (distance < ext2fs_blocks_count(fs->super))
+			start = ext2fs_blocks_count(fs->super) - distance;
+	}
+more_blocks:
+	if (distance)
+		ext2fs_llseek (fd, (start * fs->blocksize) + dest_offset, SEEK_SET);
+	for (blk = start; blk < end; blk++) {
 		if ((blk >= fs->super->s_first_data_block) &&
 		    ext2fs_test_block_bitmap2(meta_block_map, blk)) {
 			retval = io_channel_read_blk64(fs->io, blk, 1, buf);
@@ -528,9 +544,31 @@ static void output_meta_data_blocks(ext2_filsys fs, int fd)
 			}
 		}
 	}
+	if (distance && start) {
+		if (start < distance) {
+			end = start;
+			start = 0;
+		} else {
+			end -= distance;
+			start -= distance;
+			if (end < distance) {
+				/* past overlap, do rest in one go */
+				end = start;
+				start = 0;
+			}
+		}
+		sparse = 0;
+		goto more_blocks;
+	}
 #ifdef HAVE_FTRUNCATE64
 	if (sparse) {
-		ext2_loff_t offset = ext2fs_llseek(fd, sparse, SEEK_CUR);
+		ext2_loff_t offset;
+		if (distance)
+			offset = ext2fs_llseek(
+				fd,
+				fs->blocksize * ext2fs_blocks_count(fs->super) + dest_offset,
+				SEEK_SET);
+		else offset = ext2fs_llseek(fd, sparse, SEEK_CUR);
 
 		if (offset < 0)
 			lseek_error_and_exit(errno);
@@ -538,7 +576,7 @@ static void output_meta_data_blocks(ext2_filsys fs, int fd)
 			write_block(fd, zero_buf, -1, 1, -1);
 	}
 #else
-	if (sparse)
+	if (sparse && !distance)
 		write_block(fd, zero_buf, sparse-1, 1, -1);
 #endif
 	ext2fs_free_mem(&zero_buf);
@@ -1300,7 +1338,10 @@ int main (int argc, char ** argv)
 		default:
 			usage();
 		}
-	if (optind != argc - 2 )
+	if (optind == argc - 1 &&
+	    (source_offset || dest_offset))
+		    move_mode = 1;
+	else if (optind != argc - 2 )
 		usage();
 
 	if (all_data && !img_type) {
@@ -1310,7 +1351,9 @@ int main (int argc, char ** argv)
 	}
 
 	device_name = argv[optind];
-	image_fn = argv[optind+1];
+	if (move_mode)
+		image_fn = device_name;
+	else image_fn = argv[optind+1];
 
 	if (flags & E2IMAGE_INSTALL_FLAG) {
 		install_image(device_name, image_fn, img_type);
@@ -1340,10 +1383,10 @@ skip_device:
 	if (strcmp(image_fn, "-") == 0)
 		fd = 1;
 	else {
-		fd = ext2fs_open_file(image_fn, O_CREAT|O_TRUNC|O_WRONLY, 0600);
+		fd = ext2fs_open_file(image_fn, O_CREAT|O_WRONLY, 0600);
 		if (fd < 0) {
 			com_err(program_name, errno,
-				_("while trying to open %s"), argv[optind+1]);
+				_("while trying to open %s"), image_fn);
 			exit(1);
 		}
 	}
-- 
1.8.1.2

Re: [PATCH 2/4] e2image: perform in-place move

[Theodore Ts'o]

On Tue, Oct 08, 2013 at 11:56:30AM -0400, Phillip Susi wrote:
> If given at least one offset and only one file, assume source
> and dest are the same, and do an in place move.
> 
> Signed-off-by: Phillip Susi <psusi@ubuntu.com>

You **really** only want to allow move_mode if you are in raw mode.
In particular, the safe move isn't even implemented in the other mods
(nor does it make any sense), and if the user simply forgot to specify
-r, it would be a shame if it ended up destroying the user's file
system.  :-(

						- Ted

Re: [PATCH 2/4] e2image: perform in-place move

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/15/2013 11:33 PM, Theodore Ts'o wrote:
> On Tue, Oct 08, 2013 at 11:56:30AM -0400, Phillip Susi wrote:
>> If given at least one offset and only one file, assume source and
>> dest are the same, and do an in place move.
>> 
>> Signed-off-by: Phillip Susi <psusi@ubuntu.com>
> 
> You **really** only want to allow move_mode if you are in raw
> mode. In particular, the safe move isn't even implemented in the
> other mods (nor does it make any sense), and if the user simply
> forgot to specify -r, it would be a shame if it ended up destroying
> the user's file system.  :-(

Good point, also only with -a.  Fixed.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSryrAAAoJEI5FoCIzSKrwOicH/25obbcwApT1fMCZx3ZESYyg
q2zOgJafZCZfe4cx6Knn8TDZ+gk66229IslTMi3oXDTXkPXQXdc5oAtcuAs8iKNN
/bIgT+JKrwhvUsP8rL4F4oIfq1gzx9//YgS5awgyKV7653qGSmtXUdp5TZhSD8KU
f8dz66pWz/jnapv072qdqJOIXRndIGTSNWfR7M/GC0DVAKn+vRJVjXYRQ1Woov2m
tkzM5glZIFW5PUVngmckkUHLwxbdVHSQPQJE+Jib44ZOLmqYtcAfxfsrUxjbc/Ur
jWFUZOHBoDlNEmeHafDhudw3dC3yyutW+uTFKrg9dy85g5tobZ9NICZK4XgiJfw=
=/KYA
-----END PGP SIGNATURE-----

[PATCH 2/4] e2image: perform in-place move

[Phillip Susi]

If given at least one offset and only one file, assume source
and dest are the same, and do an in place move.

Signed-off-by: Phillip Susi <psusi@ubuntu.com>
---
 misc/e2image.8.in |  3 +++
 misc/e2image.c    | 67 +++++++++++++++++++++++++++++++++++++++++++++++++------
 2 files changed, 63 insertions(+), 7 deletions(-)

diff --git a/misc/e2image.8.in b/misc/e2image.8.in
index 86d3dfc..3d3aa18 100644
--- a/misc/e2image.8.in
+++ b/misc/e2image.8.in
@@ -227,6 +227,9 @@ MiB for a partition table with:
 \	\fBe2image -arO 1048576 /dev/sda1 img\fR
 .br
 .PP
+If you specify at least one offset, and only one file, an in-place
+move will be performed, allowing you to safely move the filesystem
+from one offset to another.
 .SH AUTHOR
 .B e2image
 was written by Theodore Ts'o (tytso@mit.edu).
diff --git a/misc/e2image.c b/misc/e2image.c
index f7f2e90..2dcdb2e 100644
--- a/misc/e2image.c
+++ b/misc/e2image.c
@@ -56,6 +56,7 @@ char all_data;
 char output_is_blk;
 /* writing to blk device: don't skip zeroed blocks */
 blk64_t source_offset, dest_offset;
+char move_mode;
 
 static void lseek_error_and_exit(int errnum)
 {
@@ -488,6 +489,9 @@ static void output_meta_data_blocks(ext2_filsys fs, int fd)
 	blk64_t		blk;
 	char		*buf, *zero_buf;
 	int		sparse = 0;
+	blk64_t		start = 0;
+	blk64_t		distance = 0;
+	blk64_t		end = ext2fs_blocks_count(fs->super);
 
 	retval = ext2fs_get_mem(fs->blocksize, &buf);
 	if (retval) {
@@ -499,7 +503,19 @@ static void output_meta_data_blocks(ext2_filsys fs, int fd)
 		com_err(program_name, retval, "while allocating buffer");
 		exit(1);
 	}
-	for (blk = 0; blk < ext2fs_blocks_count(fs->super); blk++) {
+	/* when doing an in place move to the right, you can't start
+	   at the beginning or you will overwrite data, so instead
+	   divide the fs up into distance size chunks and write them
+	   in reverse. */
+	if (move_mode && dest_offset > source_offset) {
+		distance = (dest_offset - source_offset) / fs->blocksize;
+		if (distance < ext2fs_blocks_count(fs->super))
+			start = ext2fs_blocks_count(fs->super) - distance;
+	}
+more_blocks:
+	if (distance)
+		ext2fs_llseek (fd, (start * fs->blocksize) + dest_offset, SEEK_SET);
+	for (blk = start; blk < end; blk++) {
 		if ((blk >= fs->super->s_first_data_block) &&
 		    ext2fs_test_block_bitmap2(meta_block_map, blk)) {
 			retval = io_channel_read_blk64(fs->io, blk, 1, buf);
@@ -528,9 +544,31 @@ static void output_meta_data_blocks(ext2_filsys fs, int fd)
 			}
 		}
 	}
+	if (distance && start) {
+		if (start < distance) {
+			end = start;
+			start = 0;
+		} else {
+			end -= distance;
+			start -= distance;
+			if (end < distance) {
+				/* past overlap, do rest in one go */
+				end = start;
+				start = 0;
+			}
+		}
+		sparse = 0;
+		goto more_blocks;
+	}
 #ifdef HAVE_FTRUNCATE64
 	if (sparse) {
-		ext2_loff_t offset = ext2fs_llseek(fd, sparse, SEEK_CUR);
+		ext2_loff_t offset;
+		if (distance)
+			offset = ext2fs_llseek(
+				fd,
+				fs->blocksize * ext2fs_blocks_count(fs->super) + dest_offset,
+				SEEK_SET);
+		else offset = ext2fs_llseek(fd, sparse, SEEK_CUR);
 
 		if (offset < 0)
 			lseek_error_and_exit(errno);
@@ -538,7 +576,7 @@ static void output_meta_data_blocks(ext2_filsys fs, int fd)
 			write_block(fd, zero_buf, -1, 1, -1);
 	}
 #else
-	if (sparse)
+	if (sparse && !distance)
 		write_block(fd, zero_buf, sparse-1, 1, -1);
 #endif
 	ext2fs_free_mem(&zero_buf);
@@ -1300,7 +1338,10 @@ int main (int argc, char ** argv)
 		default:
 			usage();
 		}
-	if (optind != argc - 2 )
+	if (optind == argc - 1 &&
+	    (source_offset || dest_offset))
+		    move_mode = 1;
+	else if (optind != argc - 2 )
 		usage();
 
 	if (all_data && !img_type) {
@@ -1313,8 +1354,20 @@ int main (int argc, char ** argv)
 			"Offsets are only allowed with raw images.");
 		exit(1);
 	}
+	if (move_mode && img_type != E2IMAGE_RAW) {
+		com_err(program_name, 0,
+			"Move mode is only allowed with raw images.");
+		exit(1);
+	}
+	if (move_mode && !all_data) {
+		com_err(program_name, 0,
+			"Move mode requires all data mode.");
+		exit(1);
+	}
 	device_name = argv[optind];
-	image_fn = argv[optind+1];
+	if (move_mode)
+		image_fn = device_name;
+	else image_fn = argv[optind+1];
 
 	if (flags & E2IMAGE_INSTALL_FLAG) {
 		install_image(device_name, image_fn, img_type);
@@ -1344,10 +1397,10 @@ skip_device:
 	if (strcmp(image_fn, "-") == 0)
 		fd = 1;
 	else {
-		fd = ext2fs_open_file(image_fn, O_CREAT|O_TRUNC|O_WRONLY, 0600);
+		fd = ext2fs_open_file(image_fn, O_CREAT|O_WRONLY, 0600);
 		if (fd < 0) {
 			com_err(program_name, errno,
-				_("while trying to open %s"), argv[optind+1]);
+				_("while trying to open %s"), image_fn);
 			exit(1);
 		}
 	}
-- 
1.8.3.2

Re: [PATCH 2/4] e2image: perform in-place move

[Theodore Ts'o]

On Mon, Dec 16, 2013 at 11:33:19AM -0500, Phillip Susi wrote:
> If given at least one offset and only one file, assume source
> and dest are the same, and do an in place move.
> 
> Signed-off-by: Phillip Susi <psusi@ubuntu.com>

Thanks, applied.

				- Ted

Re: [PATCH 2/4] e2image: perform in-place move

[Theodore Ts'o]

On Tue, Dec 24, 2013 at 11:58:26PM -0500, Theodore Ts'o wrote:
> On Mon, Dec 16, 2013 at 11:33:19AM -0500, Phillip Susi wrote:
> > If given at least one offset and only one file, assume source
> > and dest are the same, and do an in place move.
> > 
> > Signed-off-by: Phillip Susi <psusi@ubuntu.com>
> 
> Thanks, applied.

Sigh, I've had to revert this change since it is causing test
failures.  **Please** run the regression test suite before you submit
patches.

					- Ted

Contents of i_e2image.failed:

e2image -r i_e2image/image1024.orig _image.raw
e2image 1.42.8 (20-Jun-2013)
e2image -Q i_e2image/image1024.orig _image.qcow2
e2image 1.42.8 (20-Jun-2013)
e2image -r _image.qcow2 _image.qcow2.raw
e2image 1.42.8 (20-Jun-2013)
e2image -r i_e2image/image2048.orig _image.raw
e2image 1.42.8 (20-Jun-2013)
e2image -Q i_e2image/image2048.orig _image.qcow2
e2image 1.42.8 (20-Jun-2013)
e2image -r _image.qcow2 _image.qcow2.raw
e2image 1.42.8 (20-Jun-2013)
e2image -r i_e2image/image4096.orig _image.raw
e2image 1.42.8 (20-Jun-2013)
e2image -Q i_e2image/image4096.orig _image.qcow2
e2image 1.42.8 (20-Jun-2013)
e2image -r _image.qcow2 _image.qcow2.raw
e2image 1.42.8 (20-Jun-2013)
md5sums:
i_e2image/image1024.orig
d34914e0da07bdae80ab02288118fae2  image1024.orig
bbef4e50d7237546c7d9c521d3df5b68  _image.raw
1d4eb39452bed097dcd2c5bcd57180e6  _image.qcow2
bbef4e50d7237546c7d9c521d3df5b68  _image.qcow2.raw
i_e2image/image2048.orig
aa9f702de181188f2a6d2c5158686c09  image2048.orig
e097e571c809ec5418f060e2a2ec1737  _image.raw
52dba396b0fc338b99defcb086058d80  _image.qcow2
e097e571c809ec5418f060e2a2ec1737  _image.qcow2.raw
i_e2image/image4096.orig
1d3e7f15b2ce9ca07aa23c32951c5176  image4096.orig
ebaddd9f5a62610d789adbbec4ee7f46  _image.raw
a72165eaffb6900889108fbd05002822  _image.qcow2
ebaddd9f5a62610d789adbbec4ee7f46  _image.qcow2.raw

8,10c8,10
< e6f8410d0690ef551bee0c2c0c642d8c  _image.raw
< dbbd9aa97c6c946b9122586bbd2a325a  _image.qcow2
< e6f8410d0690ef551bee0c2c0c642d8c  _image.qcow2.raw
---
> e097e571c809ec5418f060e2a2ec1737  _image.raw
> 52dba396b0fc338b99defcb086058d80  _image.qcow2
> e097e571c809ec5418f060e2a2ec1737  _image.qcow2.raw
13,15c13,15
< 734119dd8f240a33704139f8cdd8127c  _image.raw
< 85fdbf5a8451b24b36ab82a02196deb9  _image.qcow2
< 734119dd8f240a33704139f8cdd8127c  _image.qcow2.raw
---
> ebaddd9f5a62610d789adbbec4ee7f46  _image.raw
> a72165eaffb6900889108fbd05002822  _image.qcow2
> ebaddd9f5a62610d789adbbec4ee7f46  _image.qcow2.raw

Re: [PATCH 2/4] e2image: perform in-place move

[Theodore Ts'o]

On Wed, Dec 25, 2013 at 11:48:57PM -0500, Theodore Ts'o wrote:
> Sigh, I've had to revert this change since it is causing test
> failures.  **Please** run the regression test suite before you submit
> patches.

I figured out the problem.  The issue is that you changed e2image to
no longer truncate the output file (i.e., you removed O_TRUNC).

The test failure was caused by the fact that previously written data
blocks that were in unallocated blocks weren't getting over written,
and so this would cause the md5 checksums to break.  I can work around
this in the test by deleting the output files, but I think I'll also
modify your comment to avoid the truncate only when trying to create a
raw image dump.  For qcow2 and normal e2image files, it's better to
keep the existing behavior of truncating the output when we first open
it.

This change of your also turned up a bug which I had to fix up.  It
could happen today when people used e2image to copy a file system to
another block device:

commit aa2c743331f392e49d5b168b5e3d8c8a8084b8b4
Author: Theodore Ts'o <tytso@mit.edu>
Date:   Wed Dec 25 16:33:52 2013 -0500

    e2image: only skip writing zero blocks when writing to a new file
    
    The e2image progam was originally intended to create image files.
    However, some people have started using e2image to copy a file system
    from one block device to another, since it is more efficient than
    using dd because it only copies the blocks which are in use.  If we
    are doing this, however, we must not skip writing blocks which are all
    zero in the source device, since they may not be zero in the
    destination device.
    
    Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
    Cc: Phillip Susi <psusi@ubuntu.com>

						- Ted

[PATCH 3/4] e2image: add progress indicator

[Phillip Susi]

When given the -p switch, print progress information, including
block counts, percentage complete, estimated time remaining, and
throughput.

Signed-off-by: Phillip Susi <psusi@ubuntu.com>
---
 misc/e2image.8.in |  6 +++--
 misc/e2image.c    | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 68 insertions(+), 4 deletions(-)

diff --git a/misc/e2image.8.in b/misc/e2image.8.in
index 3d3aa18..e5f2f04 100644
--- a/misc/e2image.8.in
+++ b/misc/e2image.8.in
@@ -8,7 +8,7 @@ e2image \- Save critical ext2/ext3/ext4 filesystem metadata to a file
 .SH SYNOPSIS
 .B e2image
 [
-.B \-rsIQa
+.B \-rsIQap
 ]
 [
 .B \-o
@@ -197,7 +197,9 @@ only includes fs metadata, not regular file data.  The
 option can be specified to include all data.  This will
 give an image that is suitable to use to clone the entire FS or
 for backup purposes.  Note that this option only works with the
-raw or QCOW2 formats.
+raw or QCOW2 formats.  The
+.B \-p
+switch may be given to show progress.
 .PP
 .SH OFFSETS
 Normally a filesystem starts at the beginning of a partition, and
diff --git a/misc/e2image.c b/misc/e2image.c
index 2557eef..27be2b1 100644
--- a/misc/e2image.c
+++ b/misc/e2image.c
@@ -57,6 +57,7 @@ char output_is_blk;
 /* writing to blk device: don't skip zeroed blocks */
 blk64_t source_offset, dest_offset;
 char move_mode;
+char show_progress;
 
 static void lseek_error_and_exit(int errnum)
 {
@@ -89,7 +90,7 @@ static int get_bits_from_size(size_t size)
 
 static void usage(void)
 {
-	fprintf(stderr, _("Usage: %s [-rsIQa] [-o source_offset] [-O dest_offset] device image_file\n"),
+	fprintf(stderr, _("Usage: %s [-rsIQap] [-o source_offset] [-O dest_offset] device image_file\n"),
 		program_name);
 	exit (1);
 }
@@ -492,6 +493,10 @@ static void output_meta_data_blocks(ext2_filsys fs, int fd)
 	blk64_t		start = 0;
 	blk64_t		distance = 0;
 	blk64_t		end = ext2fs_blocks_count(fs->super);
+	time_t		last_update;
+	time_t		start_time;
+	blk64_t		total_written = 0;
+	int		bscount;
 
 	retval = ext2fs_get_mem(fs->blocksize, &buf);
 	if (retval) {
@@ -503,6 +508,16 @@ static void output_meta_data_blocks(ext2_filsys fs, int fd)
 		com_err(program_name, retval, "while allocating buffer");
 		exit(1);
 	}
+	if (show_progress) {
+		printf("Copying ");
+		bscount = printf("%llu / %llu blocks (%llu%%)",
+				 total_written,
+				 meta_blocks_count,
+				 (total_written + 50) / ((meta_blocks_count + 50) / 100));
+		fflush(stdout);
+		last_update = time(NULL);
+		start_time = time(NULL);
+	}
 	/* when doing an in place move to the right, you can't start
 	   at the beginning or you will overwrite data, so instead
 	   divide the fs up into distance size chunks and write them
@@ -516,6 +531,30 @@ more_blocks:
 	if (distance)
 		ext2fs_llseek (fd, (start * fs->blocksize) + dest_offset, SEEK_SET);
 	for (blk = start; blk < end; blk++) {
+		if (show_progress && last_update != time(NULL)) {
+			last_update = time(NULL);
+			while (bscount--)
+				printf("\b");
+			bscount = printf("%llu / %llu blocks (%llu%%)",
+					 total_written,
+					 meta_blocks_count,
+					 (total_written + 50) /
+					 ((meta_blocks_count + 50) / 100));
+			time_t duration = time(NULL) - start_time;
+			if (duration > 5) {
+				time_t est = (duration *
+					      meta_blocks_count / total_written) -
+					(duration);
+				char buff[30];
+				strftime(buff, 30, "%T", gmtime(&est));
+				bscount += printf(" %s remaining at %.2f MB/s",
+						  buff,
+						  ((float)total_written /
+						   ((1024 * 1024) / fs->blocksize)) /
+						  duration);
+			}
+			fflush (stdout);
+		}
 		if ((blk >= fs->super->s_first_data_block) &&
 		    ext2fs_test_block_bitmap2(meta_block_map, blk)) {
 			retval = io_channel_read_blk64(fs->io, blk, 1, buf);
@@ -523,6 +562,7 @@ more_blocks:
 				com_err(program_name, retval,
 					"error reading block %llu", blk);
 			}
+			total_written++;
 			if (scramble_block_map &&
 			    ext2fs_test_block_bitmap2(scramble_block_map, blk))
 				scramble_dir_block(fs, blk, buf);
@@ -560,6 +600,23 @@ more_blocks:
 		sparse = 0;
 		goto more_blocks;
 	}
+	if (show_progress) {
+		while (bscount--)
+			printf("\b");
+		time_t duration = time(NULL) - start_time;
+		char buff[30];
+		strftime(buff, 30, "%T", gmtime(&duration));
+		printf("\b\b\b\b\b\b\b\bCopied %llu / %llu blocks (%llu%%) in "
+		       "%s at %.2f MB/s       \n",
+		       total_written,
+		       meta_blocks_count,
+		       (total_written + 50) / ((meta_blocks_count + 50) / 100),
+		       buff,
+		       ((float)total_written /
+			((1024 * 1024) / fs->blocksize)) /
+		       duration);
+
+	}
 #ifdef HAVE_FTRUNCATE64
 	if (sparse) {
 		ext2_loff_t offset;
@@ -1124,6 +1181,8 @@ static void write_raw_image_file(ext2_filsys fs, int fd, int type, int flags)
 	}
 
 	mark_table_blocks(fs);
+	if (show_progress)
+		printf("Scanning inodes...\n");
 
 	retval = ext2fs_open_inode_scan(fs, 0, &scan);
 	if (retval) {
@@ -1308,7 +1367,7 @@ int main (int argc, char ** argv)
 	if (argc && *argv)
 		program_name = *argv;
 	add_error_table(&et_ext2_error_table);
-	while ((c = getopt(argc, argv, "rsIQao:O:")) != EOF)
+	while ((c = getopt(argc, argv, "rsIQao:O:p")) != EOF)
 		switch (c) {
 		case 'I':
 			flags |= E2IMAGE_INSTALL_FLAG;
@@ -1335,6 +1394,9 @@ int main (int argc, char ** argv)
 		case 'O':
 			dest_offset = strtoull(optarg, NULL, 0);
 			break;
+		case 'p':
+			show_progress = 1;
+			break;
 		default:
 			usage();
 		}
-- 
1.8.1.2

Re: [PATCH 3/4] e2image: add progress indicator

[Theodore Ts'o]

On Tue, Oct 08, 2013 at 11:56:31AM -0400, Phillip Susi wrote:
> When given the -p switch, print progress information, including
> block counts, percentage complete, estimated time remaining, and
> throughput.
> 
> Signed-off-by: Phillip Susi <psusi@ubuntu.com>

Thanks, applied.

					- Ted

[PATCH 4/4] e2image: handle SIGINT safely

[Phillip Susi]

When doing an in place move, interrupting it past the
point of no return will destroy the filesystem since
parts of it have been overwritten.  Catch SIGINT the
first time and issue a warning if this is the case.

Signed-off-by: Phillip Susi <psusi@ubuntu.com>
---
 misc/e2image.c | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/misc/e2image.c b/misc/e2image.c
index 27be2b1..2340f15 100644
--- a/misc/e2image.c
+++ b/misc/e2image.c
@@ -35,6 +35,7 @@ extern int optind;
 #include <sys/stat.h>
 #include <sys/types.h>
 #include <assert.h>
+#include <signal.h>
 
 #include "ext2fs/ext2_fs.h"
 #include "ext2fs/ext2fs.h"
@@ -484,6 +485,14 @@ static void scramble_dir_block(ext2_filsys fs, blk64_t blk, char *buf)
 	}
 }
 
+static char got_sigint;
+
+static void sigint_handler(int unsused)
+{
+	got_sigint = 1;
+	signal (SIGINT, SIG_DFL);
+}
+
 static void output_meta_data_blocks(ext2_filsys fs, int fd)
 {
 	errcode_t	retval;
@@ -527,10 +536,39 @@ static void output_meta_data_blocks(ext2_filsys fs, int fd)
 		if (distance < ext2fs_blocks_count(fs->super))
 			start = ext2fs_blocks_count(fs->super) - distance;
 	}
+	if (move_mode)
+		signal (SIGINT, sigint_handler);
 more_blocks:
 	if (distance)
 		ext2fs_llseek (fd, (start * fs->blocksize) + dest_offset, SEEK_SET);
 	for (blk = start; blk < end; blk++) {
+		if (got_sigint) {
+			if (distance) {
+				/* moving to the right */
+				if (distance >= ext2fs_blocks_count(fs->super) ||
+				    start == ext2fs_blocks_count(fs->super) - distance)
+					kill (getpid(), SIGINT);
+			} else {
+				/* moving to the left */
+				if (blk < (source_offset - dest_offset) / fs->blocksize)
+					kill (getpid(), SIGINT);
+			}
+			if (show_progress)
+				printf ("\r");
+			printf ("Stopping now will destroy the filesystem, "
+				"interrupt again if you are sure\n");
+			if (show_progress) {
+				printf("Copying ");
+				bscount = printf("%llu / %llu blocks (%llu%%)",
+						 total_written,
+						 meta_blocks_count,
+						 (total_written + 50) / ((meta_blocks_count + 50)
+									 / 100));
+				fflush(stdout);
+			}
+
+			got_sigint = 0;
+		}
 		if (show_progress && last_update != time(NULL)) {
 			last_update = time(NULL);
 			while (bscount--)
@@ -600,6 +638,7 @@ more_blocks:
 		sparse = 0;
 		goto more_blocks;
 	}
+	signal (SIGINT, SIG_DFL);
 	if (show_progress) {
 		while (bscount--)
 			printf("\b");
-- 
1.8.1.2

Re: [PATCH 4/4] e2image: handle SIGINT safely

[Theodore Ts'o]

On Tue, Oct 08, 2013 at 11:56:32AM -0400, Phillip Susi wrote:
> When doing an in place move, interrupting it past the
> point of no return will destroy the filesystem since
> parts of it have been overwritten.  Catch SIGINT the
> first time and issue a warning if this is the case.
> 
> Signed-off-by: Phillip Susi <psusi@ubuntu.com>

Thanks, applied.

					- Ted