endianness of Linux kernel RAID

endianness of Linux kernel RAID

[Gregory Seidman]

It turns out that if one uses the kernel (2.4.x-2.6.x) RAID support (RAID5,
anyway, since that's all I've tested), the RAID'd disks cannot be moved to
another system with a different endianness. I don't know how hard that
would be to fix, but it did mean that when my ancient Mac clone I was using
as a server died I couldn't bring the RAID over to my spare x86 Linux box;
I had to dedicate other hardware (my dual G4 that I had been happily using
with OS X) to the task instead.

Unfortunately, I haven't gotten into kernel development and, therefore, do
not have the necessary expertise to fix it. Is anyone here interested in
this issue?

--Greg

Re: endianness of Linux kernel RAID

[Paul Clements]

Gregory Seidman wrote:
> It turns out that if one uses the kernel (2.4.x-2.6.x) RAID support (RAID5,
> anyway, since that's all I've tested), the RAID'd disks cannot be moved to
> another system with a different endianness. I don't know how hard that

> Unfortunately, I haven't gotten into kernel development and, therefore, do
> not have the necessary expertise to fix it. Is anyone here interested in
> this issue?

Version 1 superblock support, which was recently added to the 2.6 kernel 
and mdadm tool, fixes this issue. Metadata is always stored in little 
endian on disk.

--
Paul

Re: endianness of Linux kernel RAID

[Gregory Seidman]

On Wed, Aug 03, 2005 at 09:05:27AM -0400, Paul Clements wrote:
} Gregory Seidman wrote:
} >It turns out that if one uses the kernel (2.4.x-2.6.x) RAID support (RAID5,
} >anyway, since that's all I've tested), the RAID'd disks cannot be moved to
} >another system with a different endianness. I don't know how hard that
} 
} >Unfortunately, I haven't gotten into kernel development and, therefore, do
} >not have the necessary expertise to fix it. Is anyone here interested in
} >this issue?
} 
} Version 1 superblock support, which was recently added to the 2.6 kernel 
} and mdadm tool, fixes this issue. Metadata is always stored in little 
} endian on disk.

Great! A few questions:

1) Does this mean that the fix will be in 2.6.13?
2) Does the version 1 you refer to have to do with pre-2.6 RAID support?
3) My existing RAID was original set up under 2.4.18 on PPC, but I'm
   running 2.6.11 now and it works fine... for now. If metadata was
   previously stored in host endian, does that mean that my existing RAID
   that was set up on PPC is going to break, or be upgraded, or what?

} Paul
--Greg

Re: endianness of Linux kernel RAID

[Paul Clements]

Gregory Seidman wrote:

> 1) Does this mean that the fix will be in 2.6.13?

Yes, version 1 superblock support is in 2.6.13. You need the latest 
mdadm as well.

> 2) Does the version 1 you refer to have to do with pre-2.6 RAID support?

No. 2.4 kernels do not understand version 1 superblocks. They only 
support 0.90 superblocks (which is what you've got).

> 3) My existing RAID was original set up under 2.4.18 on PPC, but I'm
>    running 2.6.11 now and it works fine... for now. If metadata was
>    previously stored in host endian, does that mean that my existing RAID
>    that was set up on PPC is going to break, or be upgraded, or what?

You'll have to re-create the array using the version 1 superblock 
format. The existing 0.90 superblocks will continue to be used until 
you've done that.

--
Paul

Re: endianness of Linux kernel RAID

[Dan Stromberg]

[-- Attachment #1: Type: text/plain, Size: 2337 bytes --]


I don't have time to fix this by a long stretch, but I do have a web
page that talks about endianness issues, including ways of avoiding
painting yourself into a corner with them:

http://dcs.nac.uci.edu/~strombrg/converting-binary.html

The section at the top about architecture-neutral formats is probably
most relevant to this discussion - more so than the conversion programs.

As far as fixing up the mess, it'd probably be best to have some sort of
RAID usage option that would allow:

1) byte swap all endian-dependent integers on reads and writes - EG
little to big, or big to little (the operation is typically its own
inverse)

2) use a canonical format (best for use at creation time) that will
always store byte-sex dependent integers the same way, irrespective of
what sort of platform one is on

It might also not hurt to, at the same time,  make sure that integer
size (say, 32 bits vs 64 bits) isn't an issue.

I'm actually a bit discouraged that more projects aren't just using some
sort of canonical, arbitrary-precision integer format, to eliminate
these issues for good.  Granted, disk space isn't growing fast enough at
current speed to exceed a 128 bit integer anytime soon, but you never
know if there's going to be a big leap in disk capacity someday.  IMO,
this is the ideal long-term solution to this sort of problem.

On Wed, 2005-08-03 at 08:07 -0400, Gregory Seidman wrote:
> It turns out that if one uses the kernel (2.4.x-2.6.x) RAID support (RAID5,
> anyway, since that's all I've tested), the RAID'd disks cannot be moved to
> another system with a different endianness. I don't know how hard that
> would be to fix, but it did mean that when my ancient Mac clone I was using
> as a server died I couldn't bring the RAID over to my spare x86 Linux box;
> I had to dedicate other hardware (my dual G4 that I had been happily using
> with OS X) to the task instead.
> 
> Unfortunately, I haven't gotten into kernel development and, therefore, do
> not have the necessary expertise to fix it. Is anyone here interested in
> this issue?
> 
> --Greg
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: endianness of Linux kernel RAID

[Neil Brown]

[-- Attachment #1: message body text --]
[-- Type: text/plain, Size: 261 bytes --]


Oopps....
I meant to send the following in reply to a recent message from
Gregory Seidman, but inadvertently send it in reply to an earlier
message from Aaron Botsis (which I must have missed..)

Still, the offer is open to either, or anyone else.

NeilBrown


[-- Attachment #2: forwarded message --]
[-- Type: message/rfc822, Size: 2069 bytes --]

From: Neil Brown <neilb@cse.unsw.edu.au>
To: Aaron Botsis <aaron@blight.com>
Cc: linux-raid@vger.kernel.org
Subject: Re: raid5 endian question....
Date: Thu, 4 Aug 2005 15:09:05 +1000
Message-ID: <17137.41713.860387.102586@cse.unsw.edu.au>

On Sunday June 19, aaron@blight.com wrote:
> Hello,
> 
> I'm attempting to move a raid5 array from a big endian ppc machine to  
> a little endian x86.
> Upon trying to import the array, I get:
> mdadm: No super block found on /dev/hde1 (Expected magic a92b4efc,  
> got fc4e2ba9)
> 
> I assume this to be an endianess issue...
> Is this a bug? Is just the magic number whacked? or are the  
> superblock data structures too?
> 
> what's the best way to fix this?

As has been said, the original raid superblock (still the default)
uses host-endian data and so cannot be moved between machines.
The new version uses little-endian, but isn't in wide use yet.

It would not be too hard to get mdadm to re-write the superblocks when
assembling an array on a different machine.  However I am not in a
position to test it very well.

If you would be willing to help test some changes over the next
several days, I'd be happy to add this functionality to mdadm.
The risk of data-loss for you would be very remote.  However there
would be some risk of extended down-time while you wait for me to fix
some silly bug.

The change would involve:
  1/ Teaching "mdadm --examine" to recognise a wrong-endian superblock
     and display it with suitable swaps.
  2/ Teaching "mdadm --assemble" to accept "--update=endian" which
     would byte-swap all the superblocks and write them out again.

NeilBrown

 

Re: endianness of Linux kernel RAID

[Neil Brown]

On Thursday August 4, neilb@cse.unsw.edu.au wrote:
> 
> Oopps....
> I meant to send the following in reply to a recent message from
> Gregory Seidman, but inadvertently send it in reply to an earlier
> message from Aaron Botsis (which I must have missed..)
> 
> Still, the offer is open to either, or anyone else.


I decided to try it anyway...

The following patch, when applied to mdadm-2.0-devel-3 (Recently
released) should allow:

  mdadm --examine --metadata=0.swap /dev/sda1

which will show the superblock with bytes swapped.  If that looks right for
all devices, then

  mdadm --assemble /dev/mdX --update=byteorder /dev/sda1 /dev/sdb1 ...


will assemble the array after swapping the byte order on all devices.
Once it has been assembled this way, the superblocks will have the
correct byte order, and in future the array can be assembled in the
normal way.

Feedback would be very welcome.

Thanks,
NeilBrown

Signed-off-by: Neil Brown <neilb@cse.unsw.edu.au>

### Diffstat output
 ./Assemble.c     |    3 +++
 ./mdadm.c        |   14 ++++++++++++++
 ./super0.c       |   40 +++++++++++++++++++++++++++++++++++++++-

diff ./Assemble.c~current~ ./Assemble.c
--- ./Assemble.c~current~	2005-08-01 10:51:45.000000000 +1000
+++ ./Assemble.c	2005-08-04 16:40:12.000000000 +1000
@@ -334,6 +334,9 @@ int Assemble(struct supertype *st, char 
 		free(super);
 	super = NULL;
 
+	if (update && strcmp(update, "byteorder")==0)
+		st->minor_version = 90;
+
 	if (devcnt == 0) {
 		fprintf(stderr, Name ": no devices found for %s\n",
 			mddev);

diff ./mdadm.c~current~ ./mdadm.c
--- ./mdadm.c~current~	2005-07-07 09:19:53.000000000 +1000
+++ ./mdadm.c	2005-08-04 16:39:36.000000000 +1000
@@ -547,6 +547,20 @@ int main(int argc, char *argv[])
 				continue;
 			if (strcmp(update, "resync")==0)
 				continue;
+			if (strcmp(update, "byteorder")==0) {
+				if (ss) {
+					fprintf(stderr, Name ": must not set metadata type with --update=byteorder.\n");
+					exit(2);
+				}
+				for(i=0; !ss && superlist[i]; i++) 
+					ss = superlist[i]->match_metadata_desc("0.swap");
+				if (!ss) {
+					fprintf(stderr, Name ": INTERNAL ERROR cannot find 0.swap\n");
+					exit(2);
+				}
+
+				continue;
+			}
 			fprintf(stderr, Name ": '--update %s' invalid.  Only 'sparc2.2', 'super-minor', 'resync' or 'summaries' supported\n",update);
 			exit(2);
 

diff ./super0.c~current~ ./super0.c
--- ./super0.c~current~	2005-08-02 16:37:09.000000000 +1000
+++ ./super0.c	2005-08-04 16:43:10.000000000 +1000
@@ -52,7 +52,39 @@ static unsigned long calc_sb0_csum(mdp_s
 	return newcsum;
 }
 
+
+void super0_swap_endian(struct mdp_superblock_s *sb)
+{
+	/* as super0 superblocks are host-endian, it is sometimes
+	 * useful to be able to swap the endianness 
+	 * as (almost) everything is u32's we byte-swap every 4byte
+	 * number.
+	 * We then also have to swap the events_hi and events_lo
+	 */
+	char *sbc = (char *)sb;
+	__u32 t32;
+	int i;
+
+	for (i=0; i < MD_SB_BYTES ; i+=4) {
+		char t = sbc[i];
+		sbc[i] = sbc[i+3];
+		sbc[i+3] = t;
+		t=sbc[i+1];
+		sbc[i+1]=sbc[i+2];
+		sbc[i+2]=t;
+	}
+	t32 = sb->events_hi;
+	sb->events_hi = sb->events_lo;
+	sb->events_lo = t32;
+
+	t32 = sb->cp_events_hi;
+	sb->cp_events_hi = sb->cp_events_lo;
+	sb->cp_events_lo = t32;
+
+}	
+
 #ifndef MDASSEMBLE
+
 static void examine_super0(void *sbv)
 {
 	mdp_super_t *sb = sbv;
@@ -572,6 +604,9 @@ static int load_super0(struct supertype 
 		return 1;
 	}
 
+	if (st->ss && st->minor_version == 9)
+		super0_swap_endian(super);
+
 	if (super->md_magic != MD_SB_MAGIC) {
 		if (devname)
 			fprintf(stderr, Name ": No super block found on %s (Expected magic %08x, got %08x)\n",
@@ -611,6 +646,10 @@ static struct supertype *match_metadata_
 		)
 		return st;
 
+	st->minor_version = 9; /* flag for 'byte-swapped' */
+	if (strcmp(arg, "0.swap")==0)
+		return st;
+
 	free(st);
 	return NULL;
 }
@@ -742,7 +781,6 @@ int write_bitmap0(struct supertype *st, 
 
 	return rv;
 }
-	
 
 struct superswitch super0 = {
 #ifndef MDASSEMBLE

Re: endianness of Linux kernel RAID

[Daniel Pittman]

Neil Brown <neilb@cse.unsw.edu.au> writes:
> On Thursday August 4, neilb@cse.unsw.edu.au wrote:
>> 
>> Oopps....
>> I meant to send the following in reply to a recent message from
>> Gregory Seidman, but inadvertently send it in reply to an earlier
>> message from Aaron Botsis (which I must have missed..)
>> 
>> Still, the offer is open to either, or anyone else.
>
> I decided to try it anyway...
>
> The following patch, when applied to mdadm-2.0-devel-3 (Recently
> released) should allow:
>
>   mdadm --examine --metadata=0.swap /dev/sda1
>
> which will show the superblock with bytes swapped.  If that looks right for
> all devices, then
>
>   mdadm --assemble /dev/mdX --update=byteorder /dev/sda1 /dev/sdb1 ...
>
> will assemble the array after swapping the byte order on all devices.
> Once it has been assembled this way, the superblocks will have the
> correct byte order, and in future the array can be assembled in the
> normal way.
>
> Feedback would be very welcome.

If I read your notes correctly, it should be possible to determine that
this is (or isn't) a byte-swapped v0 superblock by checking for the
byte-swapped magic number, right?

In that case, I think it would be good to have --update=byteorder refuse
to work if that magic number check either wasn't the byte-swapped
variant.

Adding a 'force' or something to bypass the check if you *really* want
to seems reasonable-ish.

That would make it just that little bit harder to accidentally byte-swap
a superblock when you shouldn't have done so.

  Daniel

Re: endianness of Linux kernel RAID

[Gregory Seidman]

On Thu, Aug 04, 2005 at 03:14:07PM +1000, Neil Brown wrote:
[...]
} If you would be willing to help test some changes over the next
} several days, I'd be happy to add this functionality to mdadm.
} The risk of data-loss for you would be very remote.  However there
} would be some risk of extended down-time while you wait for me to fix
} some silly bug.
[...]

I am planning on building a new RAID soon (from external FW/USB2 drives),
and I'll be happy to test on that. Unfortunately, the timeframe for that is
before the end of the year, and I can't risk downtime on my current RAID
due to the imminent threat of murder by my wife if her email and music are
not available.

I greatly appreciate that this is progressing, however.

} NeilBrown
--Greg

Re: endianness of Linux kernel RAID

[Brent Walsh]

Neil Brown wrote:

>I decided to try it anyway...
>
>The following patch, when applied to
>mdadm-2.0-devel-3 (Recently
>released) should allow:
>
>  mdadm --examine --metadata=0.swap /dev/sda1
>
>which will show the superblock with bytes swapped.
>If that looks right for all devices, then
>
>  mdadm --assemble /dev/mdX
>--update=byteorder /dev/sda1 /dev/sdb1 ...
>
>
>will assemble the array after swapping the byte
>order on all devices. Once it has been assembled
>this way, the superblocks will have the correct
>byte order, and in future the array can be
>assembled in the normal way.

I have a PowerPC based NAS device that was damaged by
a recent a brownout.  The drives in the array are
fine, but the device's firmware was corrupted.  My
first attempt to assemble the array on my x86 system
failed due to the endian differences.  After applying
this patch, the mdadm examine command worked
perfectly.  As of yet, I have not attempted to update
the superblocks.  I would like to leave the original
superblocks intact so that I can reinstall the drives
in my NAS device when it is repaired.  Is it possible
to assemble the array without overwriting the
superblocks?

-Brent



		
____________________________________________________
Start your day with Yahoo! - make it your home page 
http://www.yahoo.com/r/hs 
 

Re: endianness of Linux kernel RAID

[Neil Brown]

On Monday August 8, blinux_adm@yahoo.com wrote:
> 
> I have a PowerPC based NAS device that was damaged by
> a recent a brownout.  The drives in the array are
> fine, but the device's firmware was corrupted.  My
> first attempt to assemble the array on my x86 system
> failed due to the endian differences.  After applying
> this patch, the mdadm examine command worked
> perfectly.  As of yet, I have not attempted to update
> the superblocks.  I would like to leave the original
> superblocks intact so that I can reinstall the drives
> in my NAS device when it is repaired.  Is it possible
> to assemble the array without overwriting the
> superblocks?

Not really....  The kernel reads the superblocks to find out the
details of the array.  If it cannot read them, it won't assemble the
array.

Depending on the type of array, you could --build the array instead.
This allows for arrays that deliberately don't have superblocks.
It should work for raid0, might work for linear, could work for raid1,
but no for anything else.

If you want to try it, send me the --examine output. and I'll suggest
a command.

NeilBrown

Re: endianness of Linux kernel RAID

[Brent Walsh]

--- Neil Brown <neilb@cse.unsw.edu.au> wrote:

> Not really....  The kernel reads the superblocks to
> find out the
> details of the array.  If it cannot read them, it
> won't assemble the
> array.
> 
> Depending on the type of array, you could --build
> the array instead.
> This allows for arrays that deliberately don't have
> superblocks.
> It should work for raid0, might work for linear,
> could work for raid1,
> but no for anything else.
> 
> If you want to try it, send me the --examine output.
> and I'll suggest
> a command.
> 

My array is raid5, so I decided to overwrite the
big-endian superblocks and assemble the array.  Your
patch worked as expected and the array is functioning
properly.  Thanks for the help.

Is there a simple way convert the superblocks back to
big-endian while the drives are connected to the
little-endian machine?

-Brent

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: endianness of Linux kernel RAID

[Neil Brown]

On Thursday August 11, blinux_adm@yahoo.com wrote:
> 
> My array is raid5, so I decided to overwrite the
> big-endian superblocks and assemble the array.  Your
> patch worked as expected and the array is functioning
> properly.  Thanks for the help.
> 
> Is there a simple way convert the superblocks back to
> big-endian while the drives are connected to the
> little-endian machine?

Not with mdadm... I could but I don't think I really want to.

However I wrote the following little program to test the byteswapping
code in mdadm.  It does a full byteswap of the superblock on the
device that you pass as an argument.

NeilBrown


-----------cut--here--------
#include <unistd.h>
#include <stdlib.h>
#include <fcntl.h>
#include <stdio.h>
#include <sys/mount.h>
/*
 * This is a tiny test program to endian-swap
 * the superblock on a given device.
 * We simply read 4k from where the superblock should be
 * do the swap, and write it back
 * Don't use this on a real array, use mdadm.
 */

#define MD_RESERVED_BYTES		(64 * 1024)
#define MD_RESERVED_SECTORS		(MD_RESERVED_BYTES / 512)

#define MD_NEW_SIZE_SECTORS(x)		((x & ~(MD_RESERVED_SECTORS - 1)) - MD_RESERVED_SECTORS)

main(int argc, char *argv[])
{
	int fd, i;
	unsigned long size;
	unsigned long long offset;
	char super[4096];
	if (argc != 2) {
		fprintf(stderr, "Usage: swap_super device\n");
		exit(1);
	}
	fd = open(argv[1], O_RDWR);
	if (fd<0) {
		perror(argv[1]);
		exit(1);
	}
	if (ioctl(fd, BLKGETSIZE, &size)) {
		perror("BLKGETSIZE");
		exit(1);
	}
	offset = MD_NEW_SIZE_SECTORS(size) * 512LL;
	if (lseek64(fd, offset, 0) < 0LL) {
		perror("lseek64");
		exit(1);
	}
	if (read(fd, super, 4096) != 4096) {
		perror("read");
		exit(1);
	}


	for (i=0; i < 4096 ; i+=4) {
		char t = super[i];
		super[i] = super[i+3];
		super[i+3] = t;
		t=super[i+1];
		super[i+1]=super[i+2];
		super[i+2]=t;
	}
	/* swap the u64 events counters */
	for (i=0; i<4; i++) {
		/* events_hi and events_lo */
		char t=super[32*4+7*4 +i];
		super[32*4+7*4 +i] = super[32*4+8*4 +i];
		super[32*4+8*4 +i] = t;

		/* cp_events_hi and cp_events_lo */
		t=super[32*4+9*4 +i];
		super[32*4+9*4 +i] = super[32*4+10*4 +i];
		super[32*4+10*4 +i] = t;
	}


	if (lseek64(fd, offset, 0) < 0LL) {
		perror("lseek64");
		exit(1);
	}
	if (write(fd, super, 4096) != 4096) {
		perror("write");
		exit(1);
	}
	exit(0);

}