From: Brian Candler <B.Candler@pobox.com>
To: Sebastian Riemer <sebastian.riemer@profitbricks.com>
Cc: linux-raid@vger.kernel.org
Subject: Re: Assembly failure
Date: Tue, 10 Jul 2012 18:06:46 +0100 [thread overview]
Message-ID: <20120710170646.GA29962@nsrc.org> (raw)
In-Reply-To: <4FFC5CC0.8050808@profitbricks.com>
On Tue, Jul 10, 2012 at 06:48:00PM +0200, Sebastian Riemer wrote:
> 1. Are you crazy to do so? The kernel 3.2 is the stable for Ubuntu 12.04.
Possibly crazy :-) Specifically I had been testing out whether the
direct-io-enable option for glusterfs would be helpful (it wasn't) - this
required FUSE support for O_DIRECT which is not in 3.2.0.
> 2. Please provide the complete Ubuntu version number of your kernel so
> that we can look for the commits in the Ubuntu Git.
brian@dev-storage1:~$ uname -a
Linux dev-storage1.example.com 3.4.0-030400-generic #201205210521 SMP Mon May 21 09:22:02 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
Packages:
ii linux-headers-3.4.0-030400 3.4.0-030400.201205210521 Header files related to Linux kernel version 3.4.0
ii linux-headers-3.4.0-030400-generic 3.4.0-030400.201205210521 Linux kernel headers for version 3.4.0 on 32 bit x86 SMP
ii linux-image-3.4.0-030400-generic 3.4.0-030400.201205210521 Linux kernel image for version 3.4.0 on 32 bit x86 SMP
It was downloaded from here:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.4-precise/
There doesn't seem to be any newer 3.4.x for precise.
> Should be
> git://kernel.ubuntu.com/ppisati/ubuntu-quantal.git.
> There were some nasty bugs in 3.4.0 mainline - I don't know if the fixes
> for them are in your kernel.
I have no problem rolling back into 3.2.0, but I'm also very happy to do any
diagnostics which may be helpful before I do so.
Regards,
Brian.
> Cheers,
> Sebastian
>
>
> On 10.07.2012 18:33, Brian Candler wrote:
> > An odd one here.
> >
> > Ubuntu 12.04 system, updated to 3.4.0 kernel from the mainline-ppa. Machine
> > has a boot disk plus 12 other disks in a RAID10 far2 array.
> >
> > System was working fine, but after most recent reboot mdraid failed to
> > assemble.
> >
> > root@dev-storage1:~# cat /proc/mdstat
> > Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
> > md127 : inactive sdk[9](S) sdl[10](S) sdj[8](S) sdg[5](S) sdc[1](S) sdm[11](S) sdf[4](S) sde[3](S) sdi[7](S) sdd[2](S) sdb[0](S) sdh[6](S)
> > 35163186720 blocks super 1.2
> >
> > unused devices: <none>
> >
> > dmesg shows periodic "export_rdev" messages:
> >
> > root@dev-storage1:~# dmesg | grep md:
> > [ 953.986401] md: export_rdev(sdo)
> > [ 953.988515] md: export_rdev(sdo)
> > [ 960.237392] md: export_rdev(sdp)
> > [ 960.241928] md: export_rdev(sdp)
> > [ 960.965132] md: export_rdev(sdr)
> > [ 960.967265] md: export_rdev(sdr)
> > [ 1012.573415] md: export_rdev(sdo)
> > [ 1012.575650] md: export_rdev(sdo)
> > [ 1012.829690] md: export_rdev(sdp)
> > [ 1012.831493] md: export_rdev(sdp)
> > ...
> > [19378.332473] md: export_rdev(sds)
> > [19378.333764] md: export_rdev(sds)
> > [19417.220171] md: export_rdev(sdr)
> > [19417.221748] md: export_rdev(sdr)
> > [23739.824227] md: export_rdev(sdr)
> > [23739.825554] md: export_rdev(sdr)
> > [23740.568940] md: export_rdev(sds)
> > [23740.570079] md: export_rdev(sds)
> >
> > metadata (see below) suggests that some drives think members 1/3/4 are
> > missing, but those drives think the array is fine. The "Events" counts are
> > different on some members though.
> >
> > What's the best thing to do here - attempt to force assembly? Any ideas how
> > it got into this state?
> >
> > The machine was rebooted a couple of times but in what should have been a
> > clean way, i.e. sudo reboot or sudo halt -p.
> >
> > Many thanks,
> >
> > Brian.
> >
> >
> > root@dev-storage1:~# for i in /dev/sd{j..u}; do echo "=== $i ==="; mdadm --examine $i; done
> > === /dev/sdj ===
> > /dev/sdj:
> > Magic : a92b4efc
> > Version : 1.2
> > Feature Map : 0x1
> > Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
> > Name : storage1:storage1
> > Creation Time : Thu Jun 7 13:51:21 2012
> > Raid Level : raid10
> > Raid Devices : 12
> >
> > Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
> > Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
> > Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
> > Data Offset : 2048 sectors
> > Super Offset : 8 sectors
> > State : clean
> > Device UUID : e96965d5:bf8986b7:fa83b813:e27aa17f
> >
> > Internal Bitmap : 8 sectors from superblock
> > Update Time : Tue Jul 10 10:46:19 2012
> > Checksum : 5673ac95 - correct
> > Events : 29374
> >
> > Layout : far=2
> > Chunk Size : 1024K
> >
> > Device Role : Active device 8
> > Array State : A.A..AAAAAAA ('A' == active, '.' == missing)
> > === /dev/sdk ===
> > /dev/sdk:
> > Magic : a92b4efc
> > Version : 1.2
> > Feature Map : 0x1
> > Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
> > Name : storage1:storage1
> > Creation Time : Thu Jun 7 13:51:21 2012
> > Raid Level : raid10
> > Raid Devices : 12
> >
> > Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
> > Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
> > Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
> > Data Offset : 2048 sectors
> > Super Offset : 8 sectors
> > State : clean
> > Device UUID : f15bb47f:85ca59b1:cad42dec:f8b1b63c
> >
> > Internal Bitmap : 8 sectors from superblock
> > Update Time : Tue Jul 10 10:46:19 2012
> > Checksum : 9020674a - correct
> > Events : 29374
> >
> > Layout : far=2
> > Chunk Size : 1024K
> >
> > Device Role : Active device 9
> > Array State : A.A..AAAAAAA ('A' == active, '.' == missing)
> > === /dev/sdl ===
> > /dev/sdl:
> > Magic : a92b4efc
> > Version : 1.2
> > Feature Map : 0x1
> > Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
> > Name : storage1:storage1
> > Creation Time : Thu Jun 7 13:51:21 2012
> > Raid Level : raid10
> > Raid Devices : 12
> >
> > Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
> > Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
> > Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
> > Data Offset : 2048 sectors
> > Super Offset : 8 sectors
> > State : clean
> > Device UUID : e2d82a4c:8409d883:cf2d9b7c:83829aad
> >
> > Internal Bitmap : 8 sectors from superblock
> > Update Time : Tue Jul 10 10:46:19 2012
> > Checksum : 30664ccb - correct
> > Events : 29374
> >
> > Layout : far=2
> > Chunk Size : 1024K
> >
> > Device Role : Active device 10
> > Array State : A.A..AAAAAAA ('A' == active, '.' == missing)
> > === /dev/sdm ===
> > /dev/sdm:
> > Magic : a92b4efc
> > Version : 1.2
> > Feature Map : 0x1
> > Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
> > Name : storage1:storage1
> > Creation Time : Thu Jun 7 13:51:21 2012
> > Raid Level : raid10
> > Raid Devices : 12
> >
> > Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
> > Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
> > Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
> > Data Offset : 2048 sectors
> > Super Offset : 8 sectors
> > State : clean
> > Device UUID : e3603570:8e767487:63f3131b:afe358ea
> >
> > Internal Bitmap : 8 sectors from superblock
> > Update Time : Tue Jul 10 10:46:19 2012
> > Checksum : 33446897 - correct
> > Events : 29374
> >
> > Layout : far=2
> > Chunk Size : 1024K
> >
> > Device Role : Active device 11
> > Array State : A.A..AAAAAAA ('A' == active, '.' == missing)
> > === /dev/sdn ===
> > /dev/sdn:
> > Magic : a92b4efc
> > Version : 1.2
> > Feature Map : 0x1
> > Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
> > Name : storage1:storage1
> > Creation Time : Thu Jun 7 13:51:21 2012
> > Raid Level : raid10
> > Raid Devices : 12
> >
> > Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
> > Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
> > Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
> > Data Offset : 2048 sectors
> > Super Offset : 8 sectors
> > State : clean
> > Device UUID : 549a5230:005cedf8:b37a0d7e:36648ff0
> >
> > Internal Bitmap : 8 sectors from superblock
> > Update Time : Tue Jul 10 10:46:19 2012
> > Checksum : ce0a8f46 - correct
> > Events : 29374
> >
> > Layout : far=2
> > Chunk Size : 1024K
> >
> > Device Role : Active device 0
> > Array State : A.A..AAAAAAA ('A' == active, '.' == missing)
> > === /dev/sdo ===
> > /dev/sdo:
> > Magic : a92b4efc
> > Version : 1.2
> > Feature Map : 0x1
> > Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
> > Name : storage1:storage1
> > Creation Time : Thu Jun 7 13:51:21 2012
> > Raid Level : raid10
> > Raid Devices : 12
> >
> > Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
> > Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
> > Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
> > Data Offset : 2048 sectors
> > Super Offset : 8 sectors
> > State : active
> > Device UUID : b09f3869:09ce8a89:31ed7097:d3621064
> >
> > Internal Bitmap : 8 sectors from superblock
> > Update Time : Sat Jul 7 14:02:07 2012
> > Checksum : 246eb119 - correct
> > Events : 29355
> >
> > Layout : far=2
> > Chunk Size : 1024K
> >
> > Device Role : Active device 2
> > Array State : A.A..AAAAAAA ('A' == active, '.' == missing)
> > === /dev/sdp ===
> > /dev/sdp:
> > Magic : a92b4efc
> > Version : 1.2
> > Feature Map : 0x1
> > Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
> > Name : storage1:storage1
> > Creation Time : Thu Jun 7 13:51:21 2012
> > Raid Level : raid10
> > Raid Devices : 12
> >
> > Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
> > Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
> > Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
> > Data Offset : 2048 sectors
> > Super Offset : 8 sectors
> > State : active
> > Device UUID : 71494ee2:1504b35a:00a1d927:543db7c6
> >
> > Internal Bitmap : 8 sectors from superblock
> > Update Time : Sat Jul 7 14:00:55 2012
> > Checksum : 61c11eeb - correct
> > Events : 29352
> >
> > Layout : far=2
> > Chunk Size : 1024K
> >
> > Device Role : Active device 3
> > Array State : A.AA.AAAAAAA ('A' == active, '.' == missing)
> > === /dev/sdq ===
> > /dev/sdq:
> > Magic : a92b4efc
> > Version : 1.2
> > Feature Map : 0x1
> > Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
> > Name : storage1:storage1
> > Creation Time : Thu Jun 7 13:51:21 2012
> > Raid Level : raid10
> > Raid Devices : 12
> >
> > Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
> > Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
> > Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
> > Data Offset : 2048 sectors
> > Super Offset : 8 sectors
> > State : clean
> > Device UUID : 17fcd5b8:97fc715f:0877d022:3770d08b
> >
> > Internal Bitmap : 8 sectors from superblock
> > Update Time : Tue Jul 10 10:46:19 2012
> > Checksum : fd1699fc - correct
> > Events : 29374
> >
> > Layout : far=2
> > Chunk Size : 1024K
> >
> > Device Role : Active device 7
> > Array State : A.A..AAAAAAA ('A' == active, '.' == missing)
> > === /dev/sdr ===
> > /dev/sdr:
> > Magic : a92b4efc
> > Version : 1.2
> > Feature Map : 0x1
> > Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
> > Name : storage1:storage1
> > Creation Time : Thu Jun 7 13:51:21 2012
> > Raid Level : raid10
> > Raid Devices : 12
> >
> > Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
> > Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
> > Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
> > Data Offset : 2048 sectors
> > Super Offset : 8 sectors
> > State : clean
> > Device UUID : e0565412:a68cf236:9da9a141:e89f935a
> >
> > Internal Bitmap : 8 sectors from superblock
> > Update Time : Tue Jul 3 04:21:50 2012
> > Checksum : 1ba72ec1 - correct
> > Events : 20228
> >
> > Layout : far=2
> > Chunk Size : 1024K
> >
> > Device Role : Active device 1
> > Array State : AAAAAAAAAAAA ('A' == active, '.' == missing)
> > === /dev/sds ===
> > /dev/sds:
> > Magic : a92b4efc
> > Version : 1.2
> > Feature Map : 0x1
> > Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
> > Name : storage1:storage1
> > Creation Time : Thu Jun 7 13:51:21 2012
> > Raid Level : raid10
> > Raid Devices : 12
> >
> > Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
> > Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
> > Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
> > Data Offset : 2048 sectors
> > Super Offset : 8 sectors
> > State : active
> > Device UUID : c0f3bbab:70f3e69e:1314cdca:072d5ad9
> >
> > Internal Bitmap : 8 sectors from superblock
> > Update Time : Tue Jul 3 06:37:33 2012
> > Checksum : 24f36d4c - correct
> > Events : 29312
> >
> > Layout : far=2
> > Chunk Size : 1024K
> >
> > Device Role : Active device 5
> > Array State : A.AA.AAAAAAA ('A' == active, '.' == missing)
> > === /dev/sdt ===
> > /dev/sdt:
> > Magic : a92b4efc
> > Version : 1.2
> > Feature Map : 0x1
> > Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
> > Name : storage1:storage1
> > Creation Time : Thu Jun 7 13:51:21 2012
> > Raid Level : raid10
> > Raid Devices : 12
> >
> > Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
> > Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
> > Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
> > Data Offset : 2048 sectors
> > Super Offset : 8 sectors
> > State : clean
> > Device UUID : 6b4f9595:4de46aa8:fa695fe8:59b797e7
> >
> > Internal Bitmap : 8 sectors from superblock
> > Update Time : Tue Jul 10 10:46:19 2012
> > Checksum : 44250f1b - correct
> > Events : 29374
> >
> > Layout : far=2
> > Chunk Size : 1024K
> >
> > Device Role : Active device 6
> > Array State : A.A..AAAAAAA ('A' == active, '.' == missing)
> > === /dev/sdu ===
> > /dev/sdu:
> > Magic : a92b4efc
> > Version : 1.2
> > Feature Map : 0x1
> > Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
> > Name : storage1:storage1
> > Creation Time : Thu Jun 7 13:51:21 2012
> > Raid Level : raid10
> > Raid Devices : 12
> >
> > Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
> > Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
> > Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
> > Data Offset : 2048 sectors
> > Super Offset : 8 sectors
> > State : active
> > Device UUID : 46f03854:de80bec8:b44b062c:dd265ba3
> >
> > Internal Bitmap : 8 sectors from superblock
> > Update Time : Tue Jul 3 06:36:18 2012
> > Checksum : 22812848 - correct
> > Events : 29272
> >
> > Layout : far=2
> > Chunk Size : 1024K
> >
> > Device Role : Active device 4
> > Array State : A.AAAAAAAAAA ('A' == active, '.' == missing)
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2012-07-10 17:06 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-07-10 16:33 Assembly failure Brian Candler
2012-07-10 16:48 ` Sebastian Riemer
2012-07-10 17:06 ` Brian Candler [this message]
2012-07-10 17:38 ` Sebastian Riemer
2012-07-10 18:59 ` Brian Candler
2012-07-11 2:43 ` NeilBrown
2012-07-11 7:58 ` Brian Candler
2012-07-11 8:27 ` Christian Balzer
2012-07-11 9:09 ` Brian Candler
2012-07-11 10:32 ` Mikael Abrahamsson
2012-07-11 10:47 ` Brian Candler
2012-07-11 10:44 ` Roman Mamedov
2012-07-11 17:21 ` Christian Balzer
2012-07-13 18:52 ` Brian Candler
2012-07-10 17:05 ` pants
-- strict thread matches above, loose matches on Subject: below --
2012-07-13 20:34 Richard Scobie
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20120710170646.GA29962@nsrc.org \
--to=b.candler@pobox.com \
--cc=linux-raid@vger.kernel.org \
--cc=sebastian.riemer@profitbricks.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.