Re: [RFH] Partition table recovery

Re: [RFH] Partition table recovery

[Al Boldi]

Theodore Tso wrote:
> On Sat, Jul 21, 2007 at 07:54:14PM +0200, Rene Herman wrote:
> > sfdisk -d already works most of the time. Not as a verbatim tool (I
> > actually semi-frequently use a "sfdisk -d /dev/hda | sfdisk" invocation
> > as a way to _rewrite_ the CHS fields to other values after changing
> > machines around on a disk) but something you'd backup on the FS level
> > should, in my opinion, need to be less fragile than would be possible
> > with just 512 bytes available.
>
> *IF* you remember to store the sfdisk -d somewhere useful.  In my "How
> To Recover From Hard Drive Catastrophies" classes, I tell them to
> print out a copy of "sfdisk -l /dev/hda ; sfdisk -d /dev/hda" and tape
> it to the side of the computer.  I also tell them do regular backups.
> What to make a guess how many them actually follow this good advice?
> Far fewer than I would like, I suspect...
>
> What I'm suggesting is the equivalent of sfdisk -d, except we'd be
> doing it automatically without requiring the user to take any kind of
> explicit action.  Is it perfect?  No, although the edge conditions are
> quite rare these days and generally involve users using legacy systems
> and/or doing Weird Shit such that They Really Should Know To Do Their
> Own Explicit Backups.  But for the novice users, it should work Just
> Fine.

Sounds great, but it may be advisable to hook this into the partition 
modification routines instead of mkfs/fsck.  Which would mean that the 
partition manager could ask the kernel to instruct its fs subsystem to 
update the backup partition table for each known fs-type that supports such 
a feature.


Thanks!

--
Al

Re: [RFH] Partition table recovery

[Theodore Tso]

On Sun, Jul 22, 2007 at 07:10:31AM +0300, Al Boldi wrote:
> Sounds great, but it may be advisable to hook this into the partition 
> modification routines instead of mkfs/fsck.  Which would mean that the 
> partition manager could ask the kernel to instruct its fs subsystem to 
> update the backup partition table for each known fs-type that supports such 
> a feature.

Well, let's think about this a bit.  What are the requirements?

1) The partition manager should be able explicitly request that a new
backup of the partition tables be stashed in each filesystem that has
room for such a backup.  That way, when the user affirmatively makes a
partition table change, it can get backed up in all of the right
places automatically.

2) The fsck program should *only* stash a backup of the partition
table if there currently isn't one in the filesystem.  It may be that
the partition table has been corrupted, and so merely doing an fsck
should not transfer a current copy of the partition table to the
filesystem-secpfic backup area.  It could be that the partition table
was only partially recovered, and we don't want to overwrite the
previously existing backups except on an explicit request from the
system administrator.

3) The mkfs program should automatically create a backup of the
current partition table layout.  That way we get a backup in the newly
created filesystem as soon as it is created.

4) The exact location of the backup may vary from filesystem to
filesystem.  For ext2/3/4, bytes 512-1023 are always unused, and don't
interfere with the boot sector at bytes 0-511, so that's the obvious
location.  Other filesystems may have that location in use, and some
other location might be a better place to store it.  Ideally it will
be a well-known location, that isn't dependent on finding an inode
table, or some such, but that may not be possible for all filesystems.

OK, so how about this as a solution that meets the above requirements?

/sbin/partbackup <device> [<fspart>]

	Will scan <device> (i.e., /dev/hda, /dev/sdb, etc.) and create
	a 512 byte partition backup, using the format I've previously
	described.  If <fspart> is specified on the command line, it
	will use the blkid library to determine the filesystem type of
	<fspart>, and then attempt to execute
	/dev/partbackupfs.<fstype> to write the partition backup to
	<fspart>.  If <fspart> is '-', then it will write the 512 byte
	partition table to stdout.  If <fspart> is not specified on
	the command line, /sbin/partbackup will iterate over all
	partitions in <device>, use the blkid library to attempt to
	determine the correct filesystem type, and then execute 
	/sbin/partbackupfs.<fstype> if such a backup program exists.

/sbin/partbackupfs.<fstype> <fspart>

	... is a filesystem specific program for filesystem type
	<fstype>.  It will assure that <fspart> (i.e., /dev/hda1,
	/dev/sdb3) is of an appropriate filesystem type, and then read
	512 bytes from stdin and write it out to <fspart> to an
	appropriate place for that filesystem.

Partition managers will be encouraged to check to see if
/sbin/partbackup exists, and if so, after the partition table is
written, will check to see if /sbin/partbackup exists, and if so, to
call it with just one argument (i.e., /sbin/partbackup /dev/hdb).
They SHOULD provide an option for the user to suppress the backup from
happening, but the backup should be the default behavior.

An /etc/mkfs.<fstype> program is encouraged to run /sbin/partbackup
with two arguments (i.e., /sbin/partbackup /dev/hdb /dev/hdb3) when
creating a filesystem.

An /etc/fsck.<fstype> program is encouraged to check to see if a
partition backup exists (assuming the filesystem supports it), and if
not, call /sbin/partbackup with two arguments.

A filesystem utility package for a particular filesystem type is
encouraged to make the above changes to its mkfs and fsck programs, as
well as provide an /sbin/partbackupfs.<fstype> program.

I would do this all in userspace, though.  Is there any reason to get
the kernel involved?  I don't think so.

						- Ted

Re: [RFH] Partition table recovery

[Al Boldi]

Theodore Tso wrote:
> On Sun, Jul 22, 2007 at 07:10:31AM +0300, Al Boldi wrote:
> > Sounds great, but it may be advisable to hook this into the partition
> > modification routines instead of mkfs/fsck.  Which would mean that the
> > partition manager could ask the kernel to instruct its fs subsystem to
> > update the backup partition table for each known fs-type that supports
> > such a feature.
>
> Well, let's think about this a bit.  What are the requirements?
>
> 1) The partition manager should be able explicitly request that a new
> backup of the partition tables be stashed in each filesystem that has
> room for such a backup.  That way, when the user affirmatively makes a
> partition table change, it can get backed up in all of the right
> places automatically.
>
> 2) The fsck program should *only* stash a backup of the partition
> table if there currently isn't one in the filesystem.  It may be that
> the partition table has been corrupted, and so merely doing an fsck
> should not transfer a current copy of the partition table to the
> filesystem-secpfic backup area.  It could be that the partition table
> was only partially recovered, and we don't want to overwrite the
> previously existing backups except on an explicit request from the
> system administrator.
>
> 3) The mkfs program should automatically create a backup of the
> current partition table layout.  That way we get a backup in the newly
> created filesystem as soon as it is created.
>
> 4) The exact location of the backup may vary from filesystem to
> filesystem.  For ext2/3/4, bytes 512-1023 are always unused, and don't
> interfere with the boot sector at bytes 0-511, so that's the obvious
> location.  Other filesystems may have that location in use, and some
> other location might be a better place to store it.  Ideally it will
> be a well-known location, that isn't dependent on finding an inode
> table, or some such, but that may not be possible for all filesystems.
>
> OK, so how about this as a solution that meets the above requirements?
>
> /sbin/partbackup <device> [<fspart>]
>
> 	Will scan <device> (i.e., /dev/hda, /dev/sdb, etc.) and create
> 	a 512 byte partition backup, using the format I've previously
> 	described.  If <fspart> is specified on the command line, it
> 	will use the blkid library to determine the filesystem type of
> 	<fspart>, and then attempt to execute
> 	/dev/partbackupfs.<fstype> to write the partition backup to
> 	<fspart>.  If <fspart> is '-', then it will write the 512 byte
> 	partition table to stdout.  If <fspart> is not specified on
> 	the command line, /sbin/partbackup will iterate over all
> 	partitions in <device>, use the blkid library to attempt to
> 	determine the correct filesystem type, and then execute
> 	/sbin/partbackupfs.<fstype> if such a backup program exists.
>
> /sbin/partbackupfs.<fstype> <fspart>
>
> 	... is a filesystem specific program for filesystem type
> 	<fstype>.  It will assure that <fspart> (i.e., /dev/hda1,
> 	/dev/sdb3) is of an appropriate filesystem type, and then read
> 	512 bytes from stdin and write it out to <fspart> to an
> 	appropriate place for that filesystem.
>
> Partition managers will be encouraged to check to see if
> /sbin/partbackup exists, and if so, after the partition table is
> written, will check to see if /sbin/partbackup exists, and if so, to
> call it with just one argument (i.e., /sbin/partbackup /dev/hdb).
> They SHOULD provide an option for the user to suppress the backup from
> happening, but the backup should be the default behavior.
>
> An /etc/mkfs.<fstype> program is encouraged to run /sbin/partbackup
> with two arguments (i.e., /sbin/partbackup /dev/hdb /dev/hdb3) when
> creating a filesystem.
>
> An /etc/fsck.<fstype> program is encouraged to check to see if a
> partition backup exists (assuming the filesystem supports it), and if
> not, call /sbin/partbackup with two arguments.
>
> A filesystem utility package for a particular filesystem type is
> encouraged to make the above changes to its mkfs and fsck programs, as
> well as provide an /sbin/partbackupfs.<fstype> program.

Great!

> I would do this all in userspace, though.  Is there any reason to get
> the kernel involved?  I don't think so.

Yes, doing things in userspace, when possible, is much better.  But, a change 
in the partition table has to be relayed to the kernel, and when that change 
happens to be on a mounted disk, then the partition manager complains of not 
being able to update the kernel's view.  So how can this be addressed?


Thanks!

--
Al

Re: [RFH] Partition table recovery

[Indan Zupancic]

On Sun, July 22, 2007 18:28, Theodore Tso wrote:
> On Sun, Jul 22, 2007 at 07:10:31AM +0300, Al Boldi wrote:
>> Sounds great, but it may be advisable to hook this into the partition
>> modification routines instead of mkfs/fsck.  Which would mean that the
>> partition manager could ask the kernel to instruct its fs subsystem to
>> update the backup partition table for each known fs-type that supports such
>> a feature.
>
> Well, let's think about this a bit.  What are the requirements?
>
> 1) The partition manager should be able explicitly request that a new
> backup of the partition tables be stashed in each filesystem that has
> room for such a backup.  That way, when the user affirmatively makes a
> partition table change, it can get backed up in all of the right
> places automatically.
>
> 2) The fsck program should *only* stash a backup of the partition
> table if there currently isn't one in the filesystem.  It may be that
> the partition table has been corrupted, and so merely doing an fsck
> should not transfer a current copy of the partition table to the
> filesystem-secpfic backup area.  It could be that the partition table
> was only partially recovered, and we don't want to overwrite the
> previously existing backups except on an explicit request from the
> system administrator.
>
> 3) The mkfs program should automatically create a backup of the
> current partition table layout.  That way we get a backup in the newly
> created filesystem as soon as it is created.
>
> 4) The exact location of the backup may vary from filesystem to
> filesystem.  For ext2/3/4, bytes 512-1023 are always unused, and don't
> interfere with the boot sector at bytes 0-511, so that's the obvious
> location.  Other filesystems may have that location in use, and some
> other location might be a better place to store it.  Ideally it will
> be a well-known location, that isn't dependent on finding an inode
> table, or some such, but that may not be possible for all filesystems.

To be on the safe side, maybe also add a checksum, timestamp and
something identifying the disk the filesystem was created on.

Regards,

Indan

Re: [RFH] Partition table recovery

[Rene Herman]

[-- Attachment #1: Type: text/plain, Size: 4314 bytes --]

On 07/22/2007 06:28 PM, Theodore Tso wrote:

[ Al -- don't drop CCs please ]

> Well, let's think about this a bit.  What are the requirements?
> 
> 1) The partition manager should be able explicitly request that a new
> backup of the partition tables be stashed in each filesystem that has
> room for such a backup.  That way, when the user affirmatively makes a
> partition table change, it can get backed up in all of the right
> places automatically.

D-Bus? ;-)

> 2) The fsck program should *only* stash a backup of the partition
> table if there currently isn't one in the filesystem.  It may be that
> the partition table has been corrupted, and so merely doing an fsck
> should not transfer a current copy of the partition table to the
> filesystem-secpfic backup area.  It could be that the partition table
> was only partially recovered, and we don't want to overwrite the
> previously existing backups except on an explicit request from the
> system administrator.
> 
> 3) The mkfs program should automatically create a backup of the
> current partition table layout.  That way we get a backup in the newly
> created filesystem as soon as it is created.

On an integrated system like this, do you consider it acceptable to only do 
the MS-DOS partitions and not the other types that may be present _inside_ 
those partitions? (MINIX subpartitions, BSD slices, ...). I believe those 
should really also be done, but this would require keeping more information 
again.

> 4) The exact location of the backup may vary from filesystem to
> filesystem.  For ext2/3/4, bytes 512-1023 are always unused, and don't
> interfere with the boot sector at bytes 0-511, so that's the obvious
> location.  Other filesystems may have that location in use, and some
> other location might be a better place to store it.  Ideally it will
> be a well-known location, that isn't dependent on finding an inode
> table, or some such, but that may not be possible for all filesystems.
> 
> OK, so how about this as a solution that meets the above requirements?
> 
> /sbin/partbackup <device> [<fspart>]
> 
> 	Will scan <device> (i.e., /dev/hda, /dev/sdb, etc.) and create
> 	a 512 byte partition backup, using the format I've previously
> 	described.  If <fspart> is specified on the command line, it
> 	will use the blkid library to determine the filesystem type of
> 	<fspart>, and then attempt to execute
> 	/dev/partbackupfs.<fstype> to write the partition backup to
> 	<fspart>.  If <fspart> is '-', then it will write the 512 byte
> 	partition table to stdout.  If <fspart> is not specified on
> 	the command line, /sbin/partbackup will iterate over all
> 	partitions in <device>, use the blkid library to attempt to
> 	determine the correct filesystem type, and then execute 
> 	/sbin/partbackupfs.<fstype> if such a backup program exists.

I've cleaned up what I posted yesterday a bit and made it into the type of 
standalone-by-design program you suggest here (the version from yesterday 
required a -DTEST to be so). Not with the blkid bits though. Just dumps the 
sector to stdout (and a textual version to stderr if compiled with -DDEBUG).

I (very) briefly looked at blkid but unless I'm mistaken blkid needs device 
names? The documentation seems to be missing. When scanning the device for 
the partition table, we've built a list of partitions with offsets into the 
device and it would be nice if we could hand the fd and the offset off to 
something directly. If the program has to construct device names itself 
there's another truckload of pitfalls right there.

It wouldn't be hard to log minors as such, but you'd also need to be very 
sure you'd always do this in the same order as the kernel so that what I 
consider to be "/dev/sda2" is the same the kernel considers it to be. This 
is again rather fragile.

It might in fact make sense to just ask the kernel for the partitions on a 
device and not bother with scanning anything ourselves. Ie, just walk sysfs. 
Would you agree? This siginificantly reduces the risk of things getting out 
of sync, both scanning order and implementation.

The kernel doesn't currently store/export everything you'd want to store in 
a backup (as far as I'm aware, that is) but that could conceivably change. 
It would make things significantly less fragile.

Rene.

[-- Attachment #2: partbackup.c --]
[-- Type: text/plain, Size: 4583 bytes --]

/*
 * Public Domain 2007, Rene Herman
 */

#define _LARGEFILE64_SOURCE

#include <stdlib.h>
#include <stdio.h>
#include <stdint.h>
#include <string.h>

#include <sys/types.h>
#include <sys/stat.h>
#include <sys/ioctl.h>
#include <unistd.h>
#include <fcntl.h>

enum {
	DOS_EXTENDED   = 0x05,
	WIN98_EXTENDED = 0x0f,
	LINUX_EXTENDED = 0x85,
};

struct partition {
        uint8_t boot_ind;
	uint8_t head;
	uint8_t sector;
	uint8_t cyl;
        uint8_t sys_ind;
	uint8_t end_head;
	uint8_t end_sector;
	uint8_t end_cyl;
        uint32_t start;
        uint32_t size;
} __attribute__((packed));

struct entry {
	uint8_t flags;
	uint8_t type;
	uint16_t __1;
	uint64_t start;
	uint32_t size;
} __attribute__((packed));

enum {
	ENTRY_FLAG_PRIMARY  = 0x01,
	ENTRY_FLAG_BOOTABLE = 0x80,
};

struct backup {
	uint8_t signature[8];
	uint16_t type;
	uint8_t heads;
	uint8_t sectors;
	uint8_t count;
	uint8_t __1[3];
	struct entry table[31];
} __attribute__((packed));

#define BACKUP_SIGNATURE "PARTBAK1"

enum {
	BACKUP_TYPE_MBR = 1,
};

struct backup backup = {
	.signature = BACKUP_SIGNATURE,
	.type	   = BACKUP_TYPE_MBR,
};

#define ARRAY_SIZE(arr) (sizeof arr / sizeof arr[0])

int is_extended(struct partition *partition)
{
	int ret = 0;

	switch (partition->sys_ind) {
	case DOS_EXTENDED:
	case WIN98_EXTENDED:
	case LINUX_EXTENDED:
		ret = 1;
	}
	return ret;
}

unsigned char *get_sector(int fd, uint64_t n)
{
	unsigned char *sector;

	if (lseek64(fd, n << 9, SEEK_SET) < 0) {
		perror("lseek64");
		return NULL;
	}
	sector = malloc(512);
	if (!sector) {
		fprintf(stderr, "malloc: out of memory\n");
		return NULL;
	}
	if (read(fd, sector, 512) != 512) {
		perror("read");
		free(sector);
		return NULL;
	}
	return sector;
}

void put_sector(unsigned char *sector)
{
	free(sector);
}

#define TABLE_OFFSET (512 - 2 - 4 * sizeof(struct partition))

inline struct partition *table(unsigned char *sector)
{
	return (struct partition *)(sector + TABLE_OFFSET);
}

int do_sector(int fd, uint32_t offset, uint32_t start)
{
	unsigned char *sector;
	struct partition *cur;
	struct partition *ext = NULL;
	int ret = 0;

	sector = get_sector(fd, offset + start);
	if (!sector)
		return -1;

	if (sector[510] != 0x55 || sector[511] != 0xaa) {
		ret = -1;
		goto out;
	}
	for (cur = table(sector); cur < table(sector) + 4; cur++) {
		struct entry *entry;

		if (!cur->size)
			continue;

		cur->end_head += 1;
		if (backup.heads < cur->end_head)
			backup.heads = cur->end_head;

		cur->end_sector &= 0x3f;
		if (backup.sectors < cur->end_sector) 
			backup.sectors = cur->end_sector;

		if (is_extended(cur)) {
			if (!offset) {
				ret = do_sector(fd, cur->start, 0);
				if (ret < 0)
					goto out;
			} else if (!ext)
				ext = cur;
			continue;
		}
		if (backup.count == ARRAY_SIZE(backup.table)) {
			fprintf(stderr, "do_sector: out of space!\n");
			ret = -1;
			goto out;
		}
		entry = backup.table + backup.count++;

		entry->flags = cur->boot_ind;
		if (!offset)
			entry->flags |= ENTRY_FLAG_PRIMARY;

		entry->type  = cur->sys_ind;
		entry->start = cur->start + start;
		entry->size  = cur->size;
	}
	if (ext)
		ret = do_sector(fd, offset, ext->start);
  out:
	put_sector(sector);
	return ret;
}

void show_backup(void)
{
#ifdef DEBUG

	int i;

	fprintf(stderr, "signature: ");
	for (i = 0; i < 8; i++)
		fprintf(stderr, "%c", backup.signature[i]);
	fprintf(stderr, "\n");

	fprintf(stderr, "     type: %d\n", backup.type);
	fprintf(stderr, "    heads: %d\n", backup.heads);
	fprintf(stderr, "  sectors: %d\n", backup.sectors);
	fprintf(stderr, "    count: %d\n", backup.count);

	for (i = 0; i < backup.count; i++) {
		fprintf(stderr, "\n");
		fprintf(stderr, "%2d: flags: %02x\n", i, backup.table[i].flags);
		fprintf(stderr, "%2d:  type: %02x\n", i, backup.table[i].type);
		fprintf(stderr, "%2d: start: %llu\n", i, backup.table[i].start);
		fprintf(stderr, "%2d:  size: %u\n", i, backup.table[i].size);
	}
#endif
}

int main(int argc, char *argv[])
{
	int fd;
	struct stat buf;

	if (argc != 2) {
		printf("%s <blkdev>\n", argv[0]);
		return EXIT_FAILURE;
	}
	fd = open(argv[1], O_RDONLY);
	if (fd < 0) {
		perror("open");
		return EXIT_FAILURE;
	}
	if (fstat(fd, &buf) < 0) {
		perror("stat");
		return EXIT_FAILURE;
	}
	if (!S_ISBLK(buf.st_mode))
		fprintf(stderr, "warning: not a block device\n");
	
	if (do_sector(fd, 0, 0) < 0)
		return EXIT_FAILURE;

	if (close(fd) < 0) {
		perror("close");
		return EXIT_FAILURE;
	}
	if (write(STDOUT_FILENO, &backup, sizeof backup) != sizeof backup) {
		perror("write");
		return EXIT_FAILURE;
	}

	show_backup();

	return EXIT_SUCCESS;
}

Re: [RFH] Partition table recovery

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 5916 bytes --]

On Mon, 2007-07-23 10:15:21 +0200, Rene Herman <rene.herman@gmail.com> wrote:
> /*
>  * Public Domain 2007, Rene Herman
>  */
> 
> #define _LARGEFILE64_SOURCE
> 
> #include <stdlib.h>
> #include <stdio.h>
> #include <stdint.h>
> #include <string.h>
> 
> #include <sys/types.h>
> #include <sys/stat.h>
> #include <sys/ioctl.h>
> #include <unistd.h>
> #include <fcntl.h>
> 
> enum {
> 	DOS_EXTENDED   = 0x05,
> 	WIN98_EXTENDED = 0x0f,
> 	LINUX_EXTENDED = 0x85,
> };
> 
> struct partition {
>         uint8_t boot_ind;
> 	uint8_t head;

Different indention.

> 	uint8_t sector;
> 	uint8_t cyl;
>         uint8_t sys_ind;
> 	uint8_t end_head;
> 	uint8_t end_sector;
> 	uint8_t end_cyl;
>         uint32_t start;
>         uint32_t size;

As multibyte on-disk variables, these will need LE/BE conversion.

> } __attribute__((packed));
> 
> struct entry {
> 	uint8_t flags;
> 	uint8_t type;
> 	uint16_t __1;
> 	uint64_t start;
> 	uint32_t size;

Dito.

> } __attribute__((packed));
> 
> enum {
> 	ENTRY_FLAG_PRIMARY  = 0x01,
> 	ENTRY_FLAG_BOOTABLE = 0x80,
> };
> 
> struct backup {
> 	uint8_t signature[8];
> 	uint16_t type;
> 	uint8_t heads;
> 	uint8_t sectors;
> 	uint8_t count;
> 	uint8_t __1[3];
> 	struct entry table[31];
> } __attribute__((packed));
> 
> #define BACKUP_SIGNATURE "PARTBAK1"
> 
> enum {
> 	BACKUP_TYPE_MBR = 1,
> };
> 
> struct backup backup = {
> 	.signature = BACKUP_SIGNATURE,
> 	.type	   = BACKUP_TYPE_MBR,
> };
> 
> #define ARRAY_SIZE(arr) (sizeof arr / sizeof arr[0])
> 
> int is_extended(struct partition *partition)
> {
> 	int ret = 0;
> 
> 	switch (partition->sys_ind) {
> 	case DOS_EXTENDED:
> 	case WIN98_EXTENDED:
> 	case LINUX_EXTENDED:
> 		ret = 1;
> 	}
> 	return ret;
> }
> 
> unsigned char *get_sector(int fd, uint64_t n)
> {
> 	unsigned char *sector;
> 
> 	if (lseek64(fd, n << 9, SEEK_SET) < 0) {
> 		perror("lseek64");
> 		return NULL;
> 	}
> 	sector = malloc(512);
> 	if (!sector) {
> 		fprintf(stderr, "malloc: out of memory\n");
> 		return NULL;
> 	}
> 	if (read(fd, sector, 512) != 512) {
> 		perror("read");
> 		free(sector);
> 		return NULL;
> 	}
> 	return sector;
> }
> 
> void put_sector(unsigned char *sector)
> {
> 	free(sector);
> }
> 
> #define TABLE_OFFSET (512 - 2 - 4 * sizeof(struct partition))
> 
> inline struct partition *table(unsigned char *sector)
> {
> 	return (struct partition *)(sector + TABLE_OFFSET);
> }
> 
> int do_sector(int fd, uint32_t offset, uint32_t start)
> {
> 	unsigned char *sector;
> 	struct partition *cur;
> 	struct partition *ext = NULL;
> 	int ret = 0;
> 
> 	sector = get_sector(fd, offset + start);
> 	if (!sector)
> 		return -1;
> 
> 	if (sector[510] != 0x55 || sector[511] != 0xaa) {
> 		ret = -1;
> 		goto out;
> 	}
> 	for (cur = table(sector); cur < table(sector) + 4; cur++) {
> 		struct entry *entry;
> 
> 		if (!cur->size)
> 			continue;
> 
> 		cur->end_head += 1;
> 		if (backup.heads < cur->end_head)
> 			backup.heads = cur->end_head;
> 
> 		cur->end_sector &= 0x3f;
> 		if (backup.sectors < cur->end_sector) 
> 			backup.sectors = cur->end_sector;
> 
> 		if (is_extended(cur)) {
> 			if (!offset) {
> 				ret = do_sector(fd, cur->start, 0);
> 				if (ret < 0)
> 					goto out;
> 			} else if (!ext)
> 				ext = cur;
> 			continue;
> 		}
> 		if (backup.count == ARRAY_SIZE(backup.table)) {
> 			fprintf(stderr, "do_sector: out of space!\n");
> 			ret = -1;
> 			goto out;
> 		}
> 		entry = backup.table + backup.count++;
> 
> 		entry->flags = cur->boot_ind;
> 		if (!offset)
> 			entry->flags |= ENTRY_FLAG_PRIMARY;
> 
> 		entry->type  = cur->sys_ind;
> 		entry->start = cur->start + start;
> 		entry->size  = cur->size;

LE/BE issues here...

> 	}
> 	if (ext)
> 		ret = do_sector(fd, offset, ext->start);
>   out:
> 	put_sector(sector);
> 	return ret;
> }
> 
> void show_backup(void)
> {
> #ifdef DEBUG
> 
> 	int i;
> 
> 	fprintf(stderr, "signature: ");
> 	for (i = 0; i < 8; i++)
> 		fprintf(stderr, "%c", backup.signature[i]);
> 	fprintf(stderr, "\n");
> 
> 	fprintf(stderr, "     type: %d\n", backup.type);
> 	fprintf(stderr, "    heads: %d\n", backup.heads);
> 	fprintf(stderr, "  sectors: %d\n", backup.sectors);
> 	fprintf(stderr, "    count: %d\n", backup.count);
> 
> 	for (i = 0; i < backup.count; i++) {
> 		fprintf(stderr, "\n");
> 		fprintf(stderr, "%2d: flags: %02x\n", i, backup.table[i].flags);
> 		fprintf(stderr, "%2d:  type: %02x\n", i, backup.table[i].type);
> 		fprintf(stderr, "%2d: start: %llu\n", i, backup.table[i].start);
> 		fprintf(stderr, "%2d:  size: %u\n", i, backup.table[i].size);

You'll output wrong values here, depending on your host system.

> 	}
> #endif
> }
> 
> int main(int argc, char *argv[])
> {
> 	int fd;
> 	struct stat buf;
> 
> 	if (argc != 2) {
> 		printf("%s <blkdev>\n", argv[0]);
> 		return EXIT_FAILURE;
> 	}
> 	fd = open(argv[1], O_RDONLY);
> 	if (fd < 0) {
> 		perror("open");
> 		return EXIT_FAILURE;
> 	}
> 	if (fstat(fd, &buf) < 0) {
> 		perror("stat");
> 		return EXIT_FAILURE;
> 	}
> 	if (!S_ISBLK(buf.st_mode))
> 		fprintf(stderr, "warning: not a block device\n");
> 	
> 	if (do_sector(fd, 0, 0) < 0)
> 		return EXIT_FAILURE;
> 
> 	if (close(fd) < 0) {
> 		perror("close");
> 		return EXIT_FAILURE;
> 	}
> 	if (write(STDOUT_FILENO, &backup, sizeof backup) != sizeof backup) {
> 		perror("write");
> 		return EXIT_FAILURE;
> 	}
> 
> 	show_backup();
> 
> 	return EXIT_SUCCESS;
> }

Looks like a useful program, but you'd definively fix the LE/BE
issues. If you do that, it'll be able to even run on BE machines, too.

MfG, JBG

-- 
      Jan-Benedict Glaw      jbglaw@lug-owl.de              +49-172-7608481
Signature of:               Träume nicht von Dein Leben: Lebe Deinen Traum!
the second  :

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [RFH] Partition table recovery

[Rene Herman]

[-- Attachment #1: Type: text/plain, Size: 1206 bytes --]

On 07/23/2007 10:41 AM, Jan-Benedict Glaw wrote:

> As multibyte on-disk variables, these will need LE/BE conversion.

Indeed, thanks -- has been updated in the version that is attached.

Also fixes a bug that snuck in (failed to add offset to entry->start).

>> struct entry {
>> 	uint8_t flags;
>> 	uint8_t type;
>> 	uint16_t __1;
>> 	uint64_t start;
>> 	uint32_t size;
> 
> Dito.

This can stay for now. The partition table backup would indeed need some 
defined byte-order but it might be "whatever order the filesystem it's 
backed up onto uses". Since it's not directly written to any filesystem for 
now, host order will do currently.

> Looks like a useful program, but you'd definively fix the LE/BE issues.
> If you do that, it'll be able to even run on BE machines, too.

I might disagree with the usefulness. I believe that preferably a backup 
should be made with help from the kernel (if only by walking sysfs) to avoid 
things getting out of sync between kernel and backup program.

(this program largely does the same as the kernel does but even now there's 
already a difference in so far that I didn't bother to de-garbage the 3rd 
and 4th entries in the second level extendeds).

Rene.


[-- Attachment #2: partbackup.c --]
[-- Type: text/plain, Size: 4748 bytes --]

/*
 * Public Domain 2007, Rene Herman
 */

#define _LARGEFILE64_SOURCE

#include <stdlib.h>
#include <stdio.h>
#include <stdint.h>
#include <string.h>

#include <sys/types.h>
#include <sys/stat.h>
#include <sys/ioctl.h>
#include <unistd.h>
#include <fcntl.h>

#include <endian.h>
#include <byteswap.h>

static inline uint32_t le_32(uint32_t n)
{
#ifdef __LITTLE_ENDIAN
	return n;
#else
	return bswap_32(n);
#endif
}

enum {
	DOS_EXTENDED   = 0x05,
	WIN98_EXTENDED = 0x0f,
	LINUX_EXTENDED = 0x85,
};

struct partition {
	uint8_t boot_ind;
	uint8_t head;
	uint8_t sector;
	uint8_t cyl;
	uint8_t sys_ind;
	uint8_t end_head;
	uint8_t end_sector;
	uint8_t end_cyl;
	uint32_t start;
	uint32_t size;
} __attribute__((packed));

struct entry {
	uint8_t flags;
	uint8_t type;
	uint16_t __1;
	uint64_t start;
	uint32_t size;
} __attribute__((packed));

enum {
	ENTRY_FLAG_PRIMARY  = 0x01,
	ENTRY_FLAG_BOOTABLE = 0x80,
};

struct backup {
	uint8_t signature[8];
	uint16_t type;
	uint8_t heads;
	uint8_t sectors;
	uint8_t count;
	uint8_t __1[3];
	struct entry table[31];
} __attribute__((packed));

#define BACKUP_SIGNATURE "PARTBAK1"

enum {
	BACKUP_TYPE_MBR = 1,
};

struct backup backup = {
	.signature = BACKUP_SIGNATURE,
	.type	   = BACKUP_TYPE_MBR,
};

#define ARRAY_SIZE(arr) (sizeof arr / sizeof arr[0])

int is_extended(struct partition *partition)
{
	int ret = 0;

	switch (partition->sys_ind) {
	case DOS_EXTENDED:
	case WIN98_EXTENDED:
	case LINUX_EXTENDED:
		ret = 1;
	}
	return ret;
}

unsigned char *get_sector(int fd, uint64_t n)
{
	unsigned char *sector;

	if (lseek64(fd, n << 9, SEEK_SET) < 0) {
		perror("lseek64");
		return NULL;
	}
	sector = malloc(512);
	if (!sector) {
		fprintf(stderr, "malloc: out of memory\n");
		return NULL;
	}
	if (read(fd, sector, 512) != 512) {
		perror("read");
		free(sector);
		return NULL;
	}
	return sector;
}

void put_sector(unsigned char *sector)
{
	free(sector);
}

#define TABLE_OFFSET (512 - 2 - 4 * sizeof(struct partition))

inline struct partition *table(unsigned char *sector)
{
	return (struct partition *)(sector + TABLE_OFFSET);
}

int do_sector(int fd, uint32_t offset, uint32_t start)
{
	unsigned char *sector;
	struct partition *cur;
	struct partition *ext = NULL;
	int ret = 0;

	sector = get_sector(fd, offset + start);
	if (!sector)
		return -1;

	if (sector[510] != 0x55 || sector[511] != 0xaa) {
		ret = -1;
		goto out;
	}
	for (cur = table(sector); cur < table(sector) + 4; cur++) {
		struct entry *entry;

		if (!cur->size)
			continue;

		cur->end_head += 1;
		if (backup.heads < cur->end_head)
			backup.heads = cur->end_head;

		cur->end_sector &= 0x3f;
		if (backup.sectors < cur->end_sector) 
			backup.sectors = cur->end_sector;

		if (is_extended(cur)) {
			if (!offset) {
				ret = do_sector(fd, le_32(cur->start), 0);
				if (ret < 0)
					goto out;
			} else if (!ext)
				ext = cur;
			continue;
		}
		if (backup.count == ARRAY_SIZE(backup.table)) {
			fprintf(stderr, "do_sector: out of space!\n");
			ret = -1;
			goto out;
		}
		entry = backup.table + backup.count++;

		entry->flags = cur->boot_ind;
		if (!offset)
			entry->flags |= ENTRY_FLAG_PRIMARY;

		entry->type  = cur->sys_ind;
		entry->start = le_32(cur->start) + offset + start;
		entry->size  = le_32(cur->size);
	}
	if (ext)
		ret = do_sector(fd, offset, le_32(ext->start));
  out:
	put_sector(sector);
	return ret;
}

void show_backup(void)
{
#ifdef DEBUG
	int i;

	fprintf(stderr, "signature: ");
	for (i = 0; i < 8; i++)
		fprintf(stderr, "%c", backup.signature[i]);
	fprintf(stderr, "\n");

	fprintf(stderr, "     type: %d\n", backup.type);
	fprintf(stderr, "    heads: %d\n", backup.heads);
	fprintf(stderr, "  sectors: %d\n", backup.sectors);
	fprintf(stderr, "    count: %d\n", backup.count);

	for (i = 0; i < backup.count; i++) {
		fprintf(stderr, "\n");
		fprintf(stderr, "%2d: flags: %02x\n", i, backup.table[i].flags);
		fprintf(stderr, "%2d:  type: %02x\n", i, backup.table[i].type);
		fprintf(stderr, "%2d: start: %llu\n", i, backup.table[i].start);
		fprintf(stderr, "%2d:  size: %u\n", i, backup.table[i].size);
	}
#endif
}

int main(int argc, char *argv[])
{
	int fd;
	struct stat buf;

	if (argc != 2) {
		printf("%s <blkdev>\n", argv[0]);
		return EXIT_FAILURE;
	}
	fd = open(argv[1], O_RDONLY);
	if (fd < 0) {
		perror("open");
		return EXIT_FAILURE;
	}
	if (fstat(fd, &buf) < 0) {
		perror("stat");
		return EXIT_FAILURE;
	}
	if (!S_ISBLK(buf.st_mode))
		fprintf(stderr, "warning: not a block device\n");
	
	if (do_sector(fd, 0, 0) < 0)
		return EXIT_FAILURE;

	if (close(fd) < 0) {
		perror("close");
		return EXIT_FAILURE;
	}
	if (write(STDOUT_FILENO, &backup, sizeof backup) != sizeof backup) {
		perror("write");
		return EXIT_FAILURE;
	}

	show_backup();

	return EXIT_SUCCESS;
}

Re: [RFH] Partition table recovery

[Rene Herman]

On 07/23/2007 12:54 PM, Rene Herman wrote:

> static inline uint32_t le_32(uint32_t n)
> {
> #ifdef __LITTLE_ENDIAN
> 	return n;
> #else
> 	return bswap_32(n);
> #endif
> }

#if __BYTE_ORDER == __LITTLE_ENDIAN, that is. sigh.

Rene.

Re: [RFH] Partition table recovery

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 564 bytes --]

On Mon, 2007-07-23 14:39:11 +0200, Rene Herman <rene.herman@gmail.com> wrote:
> On 07/23/2007 12:54 PM, Rene Herman wrote:
> 
> > static inline uint32_t le_32(uint32_t n)
> > {
> > #ifdef __LITTLE_ENDIAN
> >	return n;
> > #else
> > 	return bswap_32(n);
> > #endif
> > }
> 
> #if __BYTE_ORDER == __LITTLE_ENDIAN, that is. sigh.

Don't forget PDP11 byteorder :-)

MfG, JBG

-- 
      Jan-Benedict Glaw      jbglaw@lug-owl.de              +49-172-7608481
  Signature of:                           Wenn ich wach bin, träume ich.
  the second  :

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [RFH] Partition table recovery

[Rene Herman]

On 07/23/2007 03:15 PM, Jan-Benedict Glaw wrote:

>>> static inline uint32_t le_32(uint32_t n)
>>> {
>>> #ifdef __LITTLE_ENDIAN
>>> 	return n;
>>> #else
>>> 	return bswap_32(n);
>>> #endif
>>> }
>> #if __BYTE_ORDER == __LITTLE_ENDIAN, that is. sigh.
> 
> Don't forget PDP11 byteorder :-)

How could I? It cracks me up every time...

Rene.

Re: [RFH] Partition table recovery

[Bill Davidsen]

Rene Herman wrote:
> On 07/23/2007 10:41 AM, Jan-Benedict Glaw wrote:
>
>> As multibyte on-disk variables, these will need LE/BE conversion.
>
> Indeed, thanks -- has been updated in the version that is attached.
>
> Also fixes a bug that snuck in (failed to add offset to entry->start).
>
>>> struct entry {
>>>     uint8_t flags;
>>>     uint8_t type;
>>>     uint16_t __1;
>>>     uint64_t start;
>>>     uint32_t size;
>>
>> Dito.
>
> This can stay for now. The partition table backup would indeed need 
> some defined byte-order but it might be "whatever order the filesystem 
> it's backed up onto uses". Since it's not directly written to any 
> filesystem for now, host order will do currently.
>
>> Looks like a useful program, but you'd definively fix the LE/BE issues.
>> If you do that, it'll be able to even run on BE machines, too.
>
> I might disagree with the usefulness. I believe that preferably a 
> backup should be made with help from the kernel (if only by walking 
> sysfs) to avoid things getting out of sync between kernel and backup 
> program.
>
> (this program largely does the same as the kernel does but even now 
> there's already a difference in so far that I didn't bother to 
> de-garbage the 3rd and 4th entries in the second level extendeds).
>
How can I politely say this code really needs comments?
To quote the late R. W. Benway, "If it was hard to write it should be 
hard to understand."
(regarding code in FORTRAN II on punched cards, ca 1965)
> Rene.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: [RFH] Partition table recovery

[Theodore Tso]

On Mon, Jul 23, 2007 at 10:15:21AM +0200, Rene Herman wrote:
> On an integrated system like this, do you consider it acceptable to only do 
> the MS-DOS partitions and not the other types that may be present _inside_ 
> those partitions? (MINIX subpartitions, BSD slices, ...). I believe those 
> should really also be done, but this would require keeping more information 
> again.

Well, I'm considering this to be a MBR backup scheme, so Minix and BSD
slices are legacy systems which are out of scope.  If they are busted
in the same way as MBR in terms of not having redundant backups of
critical data, when they have a lot fewer excuses that MBR, and they
can address that issue in their own way.  The number of Linux users
that also have Minix and BSD partitions are a vanishingly small number
in any case.

> I (very) briefly looked at blkid but unless I'm mistaken blkid needs device 
> names? The documentation seems to be missing. When scanning the device for 
> the partition table, we've built a list of partitions with offsets into the 
> device and it would be nice if we could hand the fd and the offset off to 
> something directly. If the program has to construct device names itself 
> there's another truckload of pitfalls right there.

Yeah, good point, I'd have to add that support into blkid.  It's been
on my todo list, but I just haven't gotten around to it yet.

> It might in fact make sense to just ask the kernel for the partitions on a 
> device and not bother with scanning anything ourselves. Ie, just walk 
> sysfs. Would you agree? This siginificantly reduces the risk of things 
> getting out of sync, both scanning order and implementation.

My concern of sysfs is that #1, it won't work on older kernels since
you would need to add new fields to backup what we want, and #2, I'm
still fundamentally distrustful of sysfs because there isn't a bright
line between what is an exported interface that will never change, and
something which is considered an "internal implementation detail" that
can change whenever some kernel hacker feels like it.  (Or when some
kernel hacker is careless...)  So as far as I'm concerned sysfs is a
terrible, TERRIBLE way to export a published interface where we
promise stability to userspace.

So I'd just as soon do this in userspace; after all, the entire
partition manager (and there are multiple ones; fdisk, sfdisk, gpart,
etc.) all in userspace, and that needs to be in synch with the kernel
partition reading code anyway.  So one more userspace implementation
is in my mind much cleaner than trying to push the needed
functionality into sysfs, and then hoping against hope that it doesn't
accidentally change in the future.

						- Ted

Re: [RFH] Partition table recovery

[Rene Herman]

On 07/23/2007 03:58 PM, Theodore Tso wrote:

> Well, I'm considering this to be a MBR backup scheme, so Minix and BSD 
> slices are legacy systems which are out of scope.  If they are busted in
> the same way as MBR in terms of not having redundant backups of critical
> data, when they have a lot fewer excuses that MBR, and they can address
> that issue in their own way.  The number of Linux users that also have
> Minix and BSD partitions are a vanishingly small number in any case.

I'd in fact expect quite a few people to have a FreeBSD partition around. 
And MINIX if they are in university and in an operating systems course...

But "they should take whatever precautions they want themselves" is a valid 
argument.

[ blkid ]

> Yeah, good point, I'd have to add that support into blkid.  It's been
> on my todo list, but I just haven't gotten around to it yet.

I'll for now stop updating the partbackup thingy as posted. Given that Linux 
  only follows the first extended in the list of extendeds (which sort of 
destroys the nice recursion anyway) it might want to be iterative instead of 
recursive if the thing has a future -- not very important though.

> My concern of sysfs is that #1, it won't work on older kernels since
> you would need to add new fields to backup what we want,

I'd be okay with that.

> and #2, I'm still fundamentally distrustful of sysfs because there isn't
> a bright line between what is an exported interface that will never
> change, and something which is considered an "internal implementation
> detail" that can change whenever some kernel hacker feels like it.  (Or
> when some kernel hacker is careless...)  So as far as I'm concerned sysfs
> is a terrible, TERRIBLE way to export a published interface where we 
> promise stability to userspace.

Oh come on, that's going overboard a bit, it's not all _that_ bad! Finding 
say "sda" will be possible without breaking too many times. Admittedly, the 
kernel's partittion scanning order is also not likely to change as it would 
certainly break userspace, but code duplication, with the possiblity of bugs 
slipping in at least userspace-ways (considering the kernel the reference no 
matter what it does) is a concern. Somewhat. A little.

> So I'd just as soon do this in userspace; after all, the entire partition
> manager (and there are multiple ones; fdisk, sfdisk, gpart, etc.) all in
> userspace, and that needs to be in synch with the kernel partition
> reading code anyway.  So one more userspace implementation is in my mind
> much cleaner than trying to push the needed functionality into sysfs, and
> then hoping against hope that it doesn't accidentally change in the
> future.

* rene envisions /lib/libpart.so...

Not to mention my Grand Visions of a totally new Linux native partitioning 
scheme probably modelled after BSD slices (as also mentioned in a previous 
message just now). Or perhaps LVM already fills that role comfortably. It's 
certainly what I hear everyone talking about these days.

Rene.