* force raid reassembly [not found] <DB4PR07MB409B56E9B9173E33079902ED58A0@DB4PR07MB409.eurprd07.prod.outlook.com> @ 2017-08-08 22:27 ` Bryan Hepworth 2017-08-08 22:44 ` Joe Landman 2017-08-09 10:34 ` Joe Landman 0 siblings, 2 replies; 6+ messages in thread From: Bryan Hepworth @ 2017-08-08 22:27 UTC (permalink / raw) To: linux-raid@vger.kernel.org Hi all I'd be grateful of another pair of eyes just to check and verify what I'm thinking. I've had an array fail and I've checked the components with the mdadm --examine command. Five out of the six event counts are correct and I'm hoping I can get it to assemble with the force option: - "mdadm --assemble --force /dev/mdX <list of devices>" mdadm --assemble --force /dev/md0 /dev/sdd1 /dev/sdg1 /dev/sdf1 /dev/sde1 /dev/sdb1 /dev/sdc1 Output from --examine is as follows: - [root@inscribe4 dev]# mdadm --examine /dev/sde1 /dev/sde1: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 Name : intermediate0 Creation Time : Tue Oct 15 21:03:50 2013 Raid Level : raid5 Raid Devices : 6 Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) Array Size : 9766901760 (9314.44 GiB 10001.31 GB) Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) Data Offset : 262144 sectors Super Offset : 8 sectors Unused Space : before=262064 sectors, after=1154 sectors State : clean Device UUID : 8d0b6c8e:0c526265:6ec0a71b:52a636ba Internal Bitmap : 8 sectors from superblock Update Time : Tue Aug 8 14:16:53 2017 Checksum : 44b7b461 - correct Events : 119894 Layout : left-symmetric Chunk Size : 512K Device Role : spare Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) [root@inscribe4 dev]# mdadm --examine /dev/sdg1 /dev/sdg1: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 Name : intermediate0 Creation Time : Tue Oct 15 21:03:50 2013 Raid Level : raid5 Raid Devices : 6 Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) Array Size : 9766901760 (9314.44 GiB 10001.31 GB) Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) Data Offset : 262144 sectors Super Offset : 8 sectors Unused Space : before=262064 sectors, after=1154 sectors State : clean Device UUID : e238e74b:d40efc43:d34d876d:ca0700b0 Internal Bitmap : 8 sectors from superblock Update Time : Tue Aug 8 14:16:53 2017 Checksum : 28758d54 - correct Events : 119894 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 1 Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) [root@inscribe4 dev]# mdadm --examine /dev/sdd1 /dev/sdd1: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 Name : intermediate0 Creation Time : Tue Oct 15 21:03:50 2013 Raid Level : raid5 Raid Devices : 6 Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) Array Size : 9766901760 (9314.44 GiB 10001.31 GB) Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) Data Offset : 262144 sectors Super Offset : 8 sectors Unused Space : before=262064 sectors, after=1154 sectors State : clean Device UUID : 1732f121:f6f0df09:fafd2411:7c78ce00 Internal Bitmap : 8 sectors from superblock Update Time : Tue Aug 8 14:16:53 2017 Checksum : b8cf8981 - correct Events : 119894 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 0 Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) [root@inscribe4 dev]# mdadm --examine /dev/sdf1 /dev/sdf1: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 Name : intermediate0 Creation Time : Tue Oct 15 21:03:50 2013 Raid Level : raid5 Raid Devices : 6 Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) Array Size : 9766901760 (9314.44 GiB 10001.31 GB) Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) Data Offset : 262144 sectors Super Offset : 8 sectors Unused Space : before=262064 sectors, after=1154 sectors State : clean Device UUID : 3d414ad2:4310c1f9:98f50214:3e9b1e33 Internal Bitmap : 8 sectors from superblock Update Time : Tue Aug 8 14:16:53 2017 Checksum : 8e37d25d - correct Events : 119894 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 2 Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) [root@inscribe4 dev]# mdadm --examine /dev/sdb1 /dev/sdb1: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 Name : intermediate0 Creation Time : Tue Oct 15 21:03:50 2013 Raid Level : raid5 Raid Devices : 6 Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) Array Size : 9766901760 (9314.44 GiB 10001.31 GB) Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) Data Offset : 262144 sectors Super Offset : 8 sectors Unused Space : before=262064 sectors, after=1154 sectors State : clean Device UUID : a84263cd:75ce20b9:a3a72a1c:b18b2457 Internal Bitmap : 8 sectors from superblock Update Time : Mon Jul 31 22:50:37 2017 Checksum : 74d320d6 - correct Events : 119879 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 5 Array State : AAA.AA ('A' == active, '.' == missing, 'R' == replacing) [root@inscribe4 dev]# mdadm --examine /dev/sdc1 /dev/sdc1: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 Name : intermediate0 Creation Time : Tue Oct 15 21:03:50 2013 Raid Level : raid5 Raid Devices : 6 Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) Array Size : 9766901760 (9314.44 GiB 10001.31 GB) Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) Data Offset : 262144 sectors Super Offset : 8 sectors Unused Space : before=262064 sectors, after=1154 sectors State : clean Device UUID : be5ec20e:249645e5:22aeca25:a60c7599 Internal Bitmap : 8 sectors from superblock Update Time : Tue Aug 8 14:16:53 2017 Checksum : 2e529fae - correct Events : 119894 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 4 Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) Thanks Bryan ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: force raid reassembly 2017-08-08 22:27 ` force raid reassembly Bryan Hepworth @ 2017-08-08 22:44 ` Joe Landman [not found] ` <DB4PR07MB40970AB64E677A452895432D58B0@DB4PR07MB409.eurprd07.prod.outlook.com> 2017-08-09 10:34 ` Joe Landman 1 sibling, 1 reply; 6+ messages in thread From: Joe Landman @ 2017-08-08 22:44 UTC (permalink / raw) To: Bryan Hepworth, linux-raid@vger.kernel.org On 8/8/17 3:27 PM, Bryan Hepworth wrote: > Hi all > > I'd be grateful of another pair of eyes just to check and verify what I'm thinking. I've had an array fail and I've checked the components with the mdadm --examine command. Five out of the six event counts are correct and I'm hoping I can get it to assemble with the force option: - > > "mdadm --assemble --force /dev/mdX <list of devices>" > > mdadm --assemble --force /dev/md0 /dev/sdd1 /dev/sdg1 /dev/sdf1 /dev/sde1 /dev/sdb1 /dev/sdc1 I don't think you need to explicitly list the devices (/dev/sd(..)) in the force You could do something like this: mdadm --examine --scan | grep md0 > /tmp/md0.conf mdadm -As -c /tmp/md0.conf if this doesn't work, then the force *may* be an option, but be very careful with this. > > Output from --examine is as follows: - > > > [root@inscribe4 dev]# mdadm --examine /dev/sde1 > /dev/sde1: > Magic : a92b4efc > Version : 1.2 > Feature Map : 0x1 > Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 > Name : intermediate0 > Creation Time : Tue Oct 15 21:03:50 2013 > Raid Level : raid5 > Raid Devices : 6 > > Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) > Array Size : 9766901760 (9314.44 GiB 10001.31 GB) > Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) > Data Offset : 262144 sectors > Super Offset : 8 sectors > Unused Space : before=262064 sectors, after=1154 sectors > State : clean > Device UUID : 8d0b6c8e:0c526265:6ec0a71b:52a636ba > > Internal Bitmap : 8 sectors from superblock > Update Time : Tue Aug 8 14:16:53 2017 > Checksum : 44b7b461 - correct > Events : 119894 > > Layout : left-symmetric > Chunk Size : 512K > > Device Role : spare > Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) > > [root@inscribe4 dev]# mdadm --examine /dev/sdg1 > /dev/sdg1: > Magic : a92b4efc > Version : 1.2 > Feature Map : 0x1 > Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 > Name : intermediate0 > Creation Time : Tue Oct 15 21:03:50 2013 > Raid Level : raid5 > Raid Devices : 6 > > Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) > Array Size : 9766901760 (9314.44 GiB 10001.31 GB) > Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) > Data Offset : 262144 sectors > Super Offset : 8 sectors > Unused Space : before=262064 sectors, after=1154 sectors > State : clean > Device UUID : e238e74b:d40efc43:d34d876d:ca0700b0 > > Internal Bitmap : 8 sectors from superblock > Update Time : Tue Aug 8 14:16:53 2017 > Checksum : 28758d54 - correct > Events : 119894 > > Layout : left-symmetric > Chunk Size : 512K > > Device Role : Active device 1 > Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) > > [root@inscribe4 dev]# mdadm --examine /dev/sdd1 > /dev/sdd1: > Magic : a92b4efc > Version : 1.2 > Feature Map : 0x1 > Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 > Name : intermediate0 > Creation Time : Tue Oct 15 21:03:50 2013 > Raid Level : raid5 > Raid Devices : 6 > > Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) > Array Size : 9766901760 (9314.44 GiB 10001.31 GB) > Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) > Data Offset : 262144 sectors > Super Offset : 8 sectors > Unused Space : before=262064 sectors, after=1154 sectors > State : clean > Device UUID : 1732f121:f6f0df09:fafd2411:7c78ce00 > > Internal Bitmap : 8 sectors from superblock > Update Time : Tue Aug 8 14:16:53 2017 > Checksum : b8cf8981 - correct > Events : 119894 > > Layout : left-symmetric > Chunk Size : 512K > > Device Role : Active device 0 > Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) > > [root@inscribe4 dev]# mdadm --examine /dev/sdf1 > /dev/sdf1: > Magic : a92b4efc > Version : 1.2 > Feature Map : 0x1 > Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 > Name : intermediate0 > Creation Time : Tue Oct 15 21:03:50 2013 > Raid Level : raid5 > Raid Devices : 6 > > Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) > Array Size : 9766901760 (9314.44 GiB 10001.31 GB) > Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) > Data Offset : 262144 sectors > Super Offset : 8 sectors > Unused Space : before=262064 sectors, after=1154 sectors > State : clean > Device UUID : 3d414ad2:4310c1f9:98f50214:3e9b1e33 > > Internal Bitmap : 8 sectors from superblock > Update Time : Tue Aug 8 14:16:53 2017 > Checksum : 8e37d25d - correct > Events : 119894 > > Layout : left-symmetric > Chunk Size : 512K > > Device Role : Active device 2 > Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) > > [root@inscribe4 dev]# mdadm --examine /dev/sdb1 > /dev/sdb1: > Magic : a92b4efc > Version : 1.2 > Feature Map : 0x1 > Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 > Name : intermediate0 > Creation Time : Tue Oct 15 21:03:50 2013 > Raid Level : raid5 > Raid Devices : 6 > > Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) > Array Size : 9766901760 (9314.44 GiB 10001.31 GB) > Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) > Data Offset : 262144 sectors > Super Offset : 8 sectors > Unused Space : before=262064 sectors, after=1154 sectors > State : clean > Device UUID : a84263cd:75ce20b9:a3a72a1c:b18b2457 > > Internal Bitmap : 8 sectors from superblock > Update Time : Mon Jul 31 22:50:37 2017 > Checksum : 74d320d6 - correct > Events : 119879 > > Layout : left-symmetric > Chunk Size : 512K > > Device Role : Active device 5 > Array State : AAA.AA ('A' == active, '.' == missing, 'R' == replacing) > > [root@inscribe4 dev]# mdadm --examine /dev/sdc1 > /dev/sdc1: > Magic : a92b4efc > Version : 1.2 > Feature Map : 0x1 > Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 > Name : intermediate0 > Creation Time : Tue Oct 15 21:03:50 2013 > Raid Level : raid5 > Raid Devices : 6 > > Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) > Array Size : 9766901760 (9314.44 GiB 10001.31 GB) > Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) > Data Offset : 262144 sectors > Super Offset : 8 sectors > Unused Space : before=262064 sectors, after=1154 sectors > State : clean > Device UUID : be5ec20e:249645e5:22aeca25:a60c7599 > > Internal Bitmap : 8 sectors from superblock > Update Time : Tue Aug 8 14:16:53 2017 > Checksum : 2e529fae - correct > Events : 119894 > > Layout : left-symmetric > Chunk Size : 512K > > Device Role : Active device 4 > Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) > > Thanks > > Bryan > -- > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Joe Landman e: joe.landman@gmail.com t: @hpcjoe c: +1 734 612 4615 w: https://scalability.org g: https://github.com/joelandman l: https://www.linkedin.com/in/joelandman ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <DB4PR07MB40970AB64E677A452895432D58B0@DB4PR07MB409.eurprd07.prod.outlook.com>]
[parent not found: <WM!c7e5d36941a99d370e99c9ecff040c34199c6ab48c7c1548590a2e9075a62b0e4a8a44d8c43afabb53f77be4c9108e21!@mailhub-mx3.ncl.ac.uk>]
* RE: force raid reassembly [not found] ` <WM!c7e5d36941a99d370e99c9ecff040c34199c6ab48c7c1548590a2e9075a62b0e4a8a44d8c43afabb53f77be4c9108e21!@mailhub-mx3.ncl.ac.uk> @ 2017-08-09 7:40 ` Bryan Hepworth 0 siblings, 0 replies; 6+ messages in thread From: Bryan Hepworth @ 2017-08-09 7:40 UTC (permalink / raw) To: Joe Landman, linux-raid@vger.kernel.org >-----Original Message----- >From: Joe Landman [mailto:joe.landman@gmail.com] >Sent: 08 August 2017 23:44 >To: Bryan Hepworth; linux-raid@vger.kernel.org >Subject: Re: force raid reassembly > > > >On 8/8/17 3:27 PM, Bryan Hepworth wrote: >> Hi all >> >> I'd be grateful of another pair of eyes just to check and verify what I'm >thinking. I've had an array fail and I've checked the components with the >mdadm --examine command. Five out of the six event counts are correct and >I'm hoping I can get it to assemble with the force option: - >> >> "mdadm --assemble --force /dev/mdX <list of devices>" >> >> mdadm --assemble --force /dev/md0 /dev/sdd1 /dev/sdg1 /dev/sdf1 >/dev/sde1 /dev/sdb1 /dev/sdc1 > >I don't think you need to explicitly list the devices (/dev/sd(..)) in >the force > >You could do something like this: > >mdadm --examine --scan | grep md0 > /tmp/md0.conf >mdadm -As -c /tmp/md0.conf > >if this doesn't work, then the force *may* be an option, but be very >careful with this. >> Hi Joe Thanks for the reply, unfortunately that came back as follows: - [root@inscribe4 ~]# mdadm --examine --scan ARRAY /dev/md/intermediate1 metadata=1.2 UUID=64a34aa3:3703e1b0:37d3173a:bec0c874 name=intermediate1 ARRAY /dev/md/intermediate0 metadata=1.2 UUID=ace00fb5:8771d55a:cf36728b:ddc3ce89 name=intermediate0 spares=1 ARRAY /dev/md/128 metadata=1.2 UUID=d59b4fac:ade4fdbf:13ebe850:5e70ba42 name=igmimager.ncl.ac.uk:128 [root@inscribe4 md]# mdadm --examine --scan | grep intermediate0 > /tmp/intermediate0.conf [root@inscribe4 md]# cat /tmp/intermediate0.conf ARRAY /dev/md/intermediate0 metadata=1.2 UUID=ace00fb5:8771d55a:cf36728b:ddc3ce89 name=intermediate0 [root@inscribe4 md]# mdadm --assemble --scan -c /tmp/intermediate0.conf mdadm: /dev/md/intermediate0 assembled from 4 drives and 1 spare - not enough to start the array. [root@inscribe4 md]# Looks like I'll have to use the force option unless you can think of a better way Thanks Bryan ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: force raid reassembly 2017-08-08 22:27 ` force raid reassembly Bryan Hepworth 2017-08-08 22:44 ` Joe Landman @ 2017-08-09 10:34 ` Joe Landman 2017-08-12 17:44 ` Anthony Youngman 1 sibling, 1 reply; 6+ messages in thread From: Joe Landman @ 2017-08-09 10:34 UTC (permalink / raw) To: Bryan Hepworth, linux-raid Ok, you are missing 2 drives in a 6 drive raid. This is generally bad news. Are you sure all your drives are accessible? Can you read partition tables from each? If not, f2f hen a force won't likely do anything. Can you give context and logs entries on what happend to cause this? Also do you have backups of the data... raid is not a backup itself ... if need be, can you recover from a recent backup? On August 8, 2017 3:39:46 PM Bryan Hepworth <bryan.hepworth@newcastle.ac.uk> wrote: > Hi all > > I'd be grateful of another pair of eyes just to check and verify what I'm > thinking. I've had an array fail and I've checked the components with the > mdadm --examine command. Five out of the six event counts are correct and > I'm hoping I can get it to assemble with the force option: - > > "mdadm --assemble --force /dev/mdX <list of devices>" > > mdadm --assemble --force /dev/md0 /dev/sdd1 /dev/sdg1 /dev/sdf1 /dev/sde1 > /dev/sdb1 /dev/sdc1 > > Output from --examine is as follows: - > > > [root@inscribe4 dev]# mdadm --examine /dev/sde1 > /dev/sde1: > Magic : a92b4efc > Version : 1.2 > Feature Map : 0x1 > Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 > Name : intermediate0 > Creation Time : Tue Oct 15 21:03:50 2013 > Raid Level : raid5 > Raid Devices : 6 > > Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) > Array Size : 9766901760 (9314.44 GiB 10001.31 GB) > Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) > Data Offset : 262144 sectors > Super Offset : 8 sectors > Unused Space : before=262064 sectors, after=1154 sectors > State : clean > Device UUID : 8d0b6c8e:0c526265:6ec0a71b:52a636ba > > Internal Bitmap : 8 sectors from superblock > Update Time : Tue Aug 8 14:16:53 2017 > Checksum : 44b7b461 - correct > Events : 119894 > > Layout : left-symmetric > Chunk Size : 512K > > Device Role : spare > Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) > > [root@inscribe4 dev]# mdadm --examine /dev/sdg1 > /dev/sdg1: > Magic : a92b4efc > Version : 1.2 > Feature Map : 0x1 > Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 > Name : intermediate0 > Creation Time : Tue Oct 15 21:03:50 2013 > Raid Level : raid5 > Raid Devices : 6 > > Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) > Array Size : 9766901760 (9314.44 GiB 10001.31 GB) > Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) > Data Offset : 262144 sectors > Super Offset : 8 sectors > Unused Space : before=262064 sectors, after=1154 sectors > State : clean > Device UUID : e238e74b:d40efc43:d34d876d:ca0700b0 > > Internal Bitmap : 8 sectors from superblock > Update Time : Tue Aug 8 14:16:53 2017 > Checksum : 28758d54 - correct > Events : 119894 > > Layout : left-symmetric > Chunk Size : 512K > > Device Role : Active device 1 > Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) > > [root@inscribe4 dev]# mdadm --examine /dev/sdd1 > /dev/sdd1: > Magic : a92b4efc > Version : 1.2 > Feature Map : 0x1 > Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 > Name : intermediate0 > Creation Time : Tue Oct 15 21:03:50 2013 > Raid Level : raid5 > Raid Devices : 6 > > Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) > Array Size : 9766901760 (9314.44 GiB 10001.31 GB) > Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) > Data Offset : 262144 sectors > Super Offset : 8 sectors > Unused Space : before=262064 sectors, after=1154 sectors > State : clean > Device UUID : 1732f121:f6f0df09:fafd2411:7c78ce00 > > Internal Bitmap : 8 sectors from superblock > Update Time : Tue Aug 8 14:16:53 2017 > Checksum : b8cf8981 - correct > Events : 119894 > > Layout : left-symmetric > Chunk Size : 512K > > Device Role : Active device 0 > Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) > > [root@inscribe4 dev]# mdadm --examine /dev/sdf1 > /dev/sdf1: > Magic : a92b4efc > Version : 1.2 > Feature Map : 0x1 > Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 > Name : intermediate0 > Creation Time : Tue Oct 15 21:03:50 2013 > Raid Level : raid5 > Raid Devices : 6 > > Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) > Array Size : 9766901760 (9314.44 GiB 10001.31 GB) > Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) > Data Offset : 262144 sectors > Super Offset : 8 sectors > Unused Space : before=262064 sectors, after=1154 sectors > State : clean > Device UUID : 3d414ad2:4310c1f9:98f50214:3e9b1e33 > > Internal Bitmap : 8 sectors from superblock > Update Time : Tue Aug 8 14:16:53 2017 > Checksum : 8e37d25d - correct > Events : 119894 > > Layout : left-symmetric > Chunk Size : 512K > > Device Role : Active device 2 > Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) > > [root@inscribe4 dev]# mdadm --examine /dev/sdb1 > /dev/sdb1: > Magic : a92b4efc > Version : 1.2 > Feature Map : 0x1 > Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 > Name : intermediate0 > Creation Time : Tue Oct 15 21:03:50 2013 > Raid Level : raid5 > Raid Devices : 6 > > Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) > Array Size : 9766901760 (9314.44 GiB 10001.31 GB) > Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) > Data Offset : 262144 sectors > Super Offset : 8 sectors > Unused Space : before=262064 sectors, after=1154 sectors > State : clean > Device UUID : a84263cd:75ce20b9:a3a72a1c:b18b2457 > > Internal Bitmap : 8 sectors from superblock > Update Time : Mon Jul 31 22:50:37 2017 > Checksum : 74d320d6 - correct > Events : 119879 > > Layout : left-symmetric > Chunk Size : 512K > > Device Role : Active device 5 > Array State : AAA.AA ('A' == active, '.' == missing, 'R' == replacing) > > [root@inscribe4 dev]# mdadm --examine /dev/sdc1 > /dev/sdc1: > Magic : a92b4efc > Version : 1.2 > Feature Map : 0x1 > Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 > Name : intermediate0 > Creation Time : Tue Oct 15 21:03:50 2013 > Raid Level : raid5 > Raid Devices : 6 > > Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) > Array Size : 9766901760 (9314.44 GiB 10001.31 GB) > Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) > Data Offset : 262144 sectors > Super Offset : 8 sectors > Unused Space : before=262064 sectors, after=1154 sectors > State : clean > Device UUID : be5ec20e:249645e5:22aeca25:a60c7599 > > Internal Bitmap : 8 sectors from superblock > Update Time : Tue Aug 8 14:16:53 2017 > Checksum : 2e529fae - correct > Events : 119894 > > Layout : left-symmetric > Chunk Size : 512K > > Device Role : Active device 4 > Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) > > Thanks > > Bryan > -- > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: force raid reassembly 2017-08-09 10:34 ` Joe Landman @ 2017-08-12 17:44 ` Anthony Youngman [not found] ` <DBXPR07MB415CB58355ECBF4DB0D6632D59B0@DBXPR07MB415.eurprd07.prod.outlook.com> 0 siblings, 1 reply; 6+ messages in thread From: Anthony Youngman @ 2017-08-12 17:44 UTC (permalink / raw) To: Joe Landman, Bryan Hepworth, linux-raid Only thing is, sdb1 says "AAA.AA", which is a bit weird, unless sdb1 is the very last one, which is the last dot in AAA.A. of all the others. If five drives all have almost identical event counts, I'd be inclined to ddrescue the dodgy one (probably sdb1) and hope it all works out. The alternative is to ddrescue both dodgy drives, and write the program that marks blocks as dodgy. Then you can hope that a scan can rebuild everything. Cheers, Wol On 09/08/17 11:34, Joe Landman wrote: > Ok, you are missing 2 drives in a 6 drive raid. This is generally bad > news. Are you sure all your drives are accessible? Can you read > partition tables from each? If not, f2f hen a force won't likely do > anything. > > Can you give context and logs entries on what happend to cause this? > Also do you have backups of the data... raid is not a backup itself ... > if need be, can you recover from a recent backup? > > > On August 8, 2017 3:39:46 PM Bryan Hepworth > <bryan.hepworth@newcastle.ac.uk> wrote: > >> Hi all >> >> I'd be grateful of another pair of eyes just to check and verify what >> I'm thinking. I've had an array fail and I've checked the components >> with the mdadm --examine command. Five out of the six event counts are >> correct and I'm hoping I can get it to assemble with the force option: - >> >> "mdadm --assemble --force /dev/mdX <list of devices>" >> >> mdadm --assemble --force /dev/md0 /dev/sdd1 /dev/sdg1 /dev/sdf1 >> /dev/sde1 /dev/sdb1 /dev/sdc1 >> >> Output from --examine is as follows: - >> >> >> [root@inscribe4 dev]# mdadm --examine /dev/sde1 >> /dev/sde1: >> Magic : a92b4efc >> Version : 1.2 >> Feature Map : 0x1 >> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 >> Name : intermediate0 >> Creation Time : Tue Oct 15 21:03:50 2013 >> Raid Level : raid5 >> Raid Devices : 6 >> >> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) >> Array Size : 9766901760 (9314.44 GiB 10001.31 GB) >> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) >> Data Offset : 262144 sectors >> Super Offset : 8 sectors >> Unused Space : before=262064 sectors, after=1154 sectors >> State : clean >> Device UUID : 8d0b6c8e:0c526265:6ec0a71b:52a636ba >> >> Internal Bitmap : 8 sectors from superblock >> Update Time : Tue Aug 8 14:16:53 2017 >> Checksum : 44b7b461 - correct >> Events : 119894 >> >> Layout : left-symmetric >> Chunk Size : 512K >> >> Device Role : spare >> Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) >> >> [root@inscribe4 dev]# mdadm --examine /dev/sdg1 >> /dev/sdg1: >> Magic : a92b4efc >> Version : 1.2 >> Feature Map : 0x1 >> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 >> Name : intermediate0 >> Creation Time : Tue Oct 15 21:03:50 2013 >> Raid Level : raid5 >> Raid Devices : 6 >> >> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) >> Array Size : 9766901760 (9314.44 GiB 10001.31 GB) >> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) >> Data Offset : 262144 sectors >> Super Offset : 8 sectors >> Unused Space : before=262064 sectors, after=1154 sectors >> State : clean >> Device UUID : e238e74b:d40efc43:d34d876d:ca0700b0 >> >> Internal Bitmap : 8 sectors from superblock >> Update Time : Tue Aug 8 14:16:53 2017 >> Checksum : 28758d54 - correct >> Events : 119894 >> >> Layout : left-symmetric >> Chunk Size : 512K >> >> Device Role : Active device 1 >> Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) >> >> [root@inscribe4 dev]# mdadm --examine /dev/sdd1 >> /dev/sdd1: >> Magic : a92b4efc >> Version : 1.2 >> Feature Map : 0x1 >> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 >> Name : intermediate0 >> Creation Time : Tue Oct 15 21:03:50 2013 >> Raid Level : raid5 >> Raid Devices : 6 >> >> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) >> Array Size : 9766901760 (9314.44 GiB 10001.31 GB) >> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) >> Data Offset : 262144 sectors >> Super Offset : 8 sectors >> Unused Space : before=262064 sectors, after=1154 sectors >> State : clean >> Device UUID : 1732f121:f6f0df09:fafd2411:7c78ce00 >> >> Internal Bitmap : 8 sectors from superblock >> Update Time : Tue Aug 8 14:16:53 2017 >> Checksum : b8cf8981 - correct >> Events : 119894 >> >> Layout : left-symmetric >> Chunk Size : 512K >> >> Device Role : Active device 0 >> Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) >> >> [root@inscribe4 dev]# mdadm --examine /dev/sdf1 >> /dev/sdf1: >> Magic : a92b4efc >> Version : 1.2 >> Feature Map : 0x1 >> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 >> Name : intermediate0 >> Creation Time : Tue Oct 15 21:03:50 2013 >> Raid Level : raid5 >> Raid Devices : 6 >> >> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) >> Array Size : 9766901760 (9314.44 GiB 10001.31 GB) >> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) >> Data Offset : 262144 sectors >> Super Offset : 8 sectors >> Unused Space : before=262064 sectors, after=1154 sectors >> State : clean >> Device UUID : 3d414ad2:4310c1f9:98f50214:3e9b1e33 >> >> Internal Bitmap : 8 sectors from superblock >> Update Time : Tue Aug 8 14:16:53 2017 >> Checksum : 8e37d25d - correct >> Events : 119894 >> >> Layout : left-symmetric >> Chunk Size : 512K >> >> Device Role : Active device 2 >> Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) >> >> [root@inscribe4 dev]# mdadm --examine /dev/sdb1 >> /dev/sdb1: >> Magic : a92b4efc >> Version : 1.2 >> Feature Map : 0x1 >> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 >> Name : intermediate0 >> Creation Time : Tue Oct 15 21:03:50 2013 >> Raid Level : raid5 >> Raid Devices : 6 >> >> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) >> Array Size : 9766901760 (9314.44 GiB 10001.31 GB) >> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) >> Data Offset : 262144 sectors >> Super Offset : 8 sectors >> Unused Space : before=262064 sectors, after=1154 sectors >> State : clean >> Device UUID : a84263cd:75ce20b9:a3a72a1c:b18b2457 >> >> Internal Bitmap : 8 sectors from superblock >> Update Time : Mon Jul 31 22:50:37 2017 >> Checksum : 74d320d6 - correct >> Events : 119879 >> >> Layout : left-symmetric >> Chunk Size : 512K >> >> Device Role : Active device 5 >> Array State : AAA.AA ('A' == active, '.' == missing, 'R' == replacing) >> >> [root@inscribe4 dev]# mdadm --examine /dev/sdc1 >> /dev/sdc1: >> Magic : a92b4efc >> Version : 1.2 >> Feature Map : 0x1 >> Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 >> Name : intermediate0 >> Creation Time : Tue Oct 15 21:03:50 2013 >> Raid Level : raid5 >> Raid Devices : 6 >> >> Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) >> Array Size : 9766901760 (9314.44 GiB 10001.31 GB) >> Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) >> Data Offset : 262144 sectors >> Super Offset : 8 sectors >> Unused Space : before=262064 sectors, after=1154 sectors >> State : clean >> Device UUID : be5ec20e:249645e5:22aeca25:a60c7599 >> >> Internal Bitmap : 8 sectors from superblock >> Update Time : Tue Aug 8 14:16:53 2017 >> Checksum : 2e529fae - correct >> Events : 119894 >> >> Layout : left-symmetric >> Chunk Size : 512K >> >> Device Role : Active device 4 >> Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) >> >> Thanks >> >> Bryan >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <DBXPR07MB415CB58355ECBF4DB0D6632D59B0@DBXPR07MB415.eurprd07.prod.outlook.com>]
[parent not found: <WM!13c5dc8d433950e8d178eb39ac57b28caef2d362792181894459f3aa56870cedd0dcd8b5aa75be7d1b060d3021a4afe9!@mailhub-mx3.ncl.ac.uk>]
* RE: force raid reassembly [not found] ` <WM!13c5dc8d433950e8d178eb39ac57b28caef2d362792181894459f3aa56870cedd0dcd8b5aa75be7d1b060d3021a4afe9!@mailhub-mx3.ncl.ac.uk> @ 2017-08-25 8:03 ` Bryan Hepworth 0 siblings, 0 replies; 6+ messages in thread From: Bryan Hepworth @ 2017-08-25 8:03 UTC (permalink / raw) To: Anthony Youngman, Joe Landman, linux-raid@vger.kernel.org >-----Original Message----- >From: Anthony Youngman [mailto:antlists@youngman.org.uk] >Sent: 12 August 2017 18:44 >To: Joe Landman; Bryan Hepworth; linux-raid@vger.kernel.org >Subject: Re: force raid reassembly > >Only thing is, sdb1 says "AAA.AA", which is a bit weird, unless sdb1 is >the very last one, which is the last dot in AAA.A. of all the others. > >If five drives all have almost identical event counts, I'd be inclined >to ddrescue the dodgy one (probably sdb1) and hope it all works out. > >The alternative is to ddrescue both dodgy drives, and write the program >that marks blocks as dodgy. Then you can hope that a scan can rebuild >everything. > >Cheers, >Wol Hi Anthony I've got another quick question, hope you can see what you reckon... The disks are in an md1200 storage shelf and all seem to have settled down now. Could you take a quick look at the output and see if you think the same advice stands. Thanks so much. Bryan [root@inscribe4 ~]# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md125 : active (auto-read-only) raid5 sdh1[4] sdg1[6] sdi1[3] sdk1[1] sdl1[0] sdj1[7] 9767552000 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6] [UUUUUU] bitmap: 0/15 pages [0KB], 65536KB chunk md126 : inactive sdb1[4] sde1[7] sdc1[0] sdf1[1] sdd1[8](S) 9766904645 blocks super 1.2 md127 : active (auto-read-only) raid6 sdo[2] sdq[4] sdm[0] sdw[10] sdt[12] sds[6] sdn[1] sdr[5] sdu[8] sdp[3] sdx[11] sdv[9] 19535129600 blocks super 1.2 level 6, 512k chunk, algorithm 2 [12/12] [UUUUUUUUUUUU] bitmap: 0/15 pages [0KB], 65536KB chunk unused devices: <none> [root@inscribe4 ~]# mdadm --examine /dev/sda1 /dev/sda1: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 Name : intermediate0 Creation Time : Tue Oct 15 21:03:50 2013 Raid Level : raid5 Raid Devices : 6 Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) Array Size : 9766901760 (9314.44 GiB 10001.31 GB) Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) Data Offset : 262144 sectors Super Offset : 8 sectors Unused Space : before=262064 sectors, after=1154 sectors State : clean Device UUID : a84263cd:75ce20b9:a3a72a1c:b18b2457 Internal Bitmap : 8 sectors from superblock Update Time : Mon Jul 31 22:50:37 2017 Checksum : 74d320d6 - correct Events : 119879 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 5 Array State : AAA.AA ('A' == active, '.' == missing, 'R' == replacing) [root@inscribe4 ~]# mdadm --examine /dev/sdb1 /dev/sdb1: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 Name : intermediate0 Creation Time : Tue Oct 15 21:03:50 2013 Raid Level : raid5 Raid Devices : 6 Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) Array Size : 9766901760 (9314.44 GiB 10001.31 GB) Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) Data Offset : 262144 sectors Super Offset : 8 sectors Unused Space : before=262064 sectors, after=1154 sectors State : clean Device UUID : be5ec20e:249645e5:22aeca25:a60c7599 Internal Bitmap : 8 sectors from superblock Update Time : Tue Aug 8 14:16:53 2017 Checksum : 2e529fae - correct Events : 119894 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 4 Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) [root@inscribe4 ~]# mdadm --examine /dev/sdc1 /dev/sdc1: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 Name : intermediate0 Creation Time : Tue Oct 15 21:03:50 2013 Raid Level : raid5 Raid Devices : 6 Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) Array Size : 9766901760 (9314.44 GiB 10001.31 GB) Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) Data Offset : 262144 sectors Super Offset : 8 sectors Unused Space : before=262064 sectors, after=1154 sectors State : clean Device UUID : 1732f121:f6f0df09:fafd2411:7c78ce00 Internal Bitmap : 8 sectors from superblock Update Time : Tue Aug 8 14:16:53 2017 Checksum : b8cf8981 - correct Events : 119894 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 0 Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) [root@inscribe4 ~]# mdadm --examine /dev/sdd1 /dev/sdd1: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 Name : intermediate0 Creation Time : Tue Oct 15 21:03:50 2013 Raid Level : raid5 Raid Devices : 6 Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) Array Size : 9766901760 (9314.44 GiB 10001.31 GB) Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) Data Offset : 262144 sectors Super Offset : 8 sectors Unused Space : before=262064 sectors, after=1154 sectors State : clean Device UUID : 8d0b6c8e:0c526265:6ec0a71b:52a636ba Internal Bitmap : 8 sectors from superblock Update Time : Tue Aug 8 14:16:53 2017 Checksum : 44b7b461 - correct Events : 119894 Layout : left-symmetric Chunk Size : 512K Device Role : spare Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) [root@inscribe4 ~]# mdadm --examine /dev/sde1 /dev/sde1: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 Name : intermediate0 Creation Time : Tue Oct 15 21:03:50 2013 Raid Level : raid5 Raid Devices : 6 Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) Array Size : 9766901760 (9314.44 GiB 10001.31 GB) Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) Data Offset : 262144 sectors Super Offset : 8 sectors Unused Space : before=262064 sectors, after=1154 sectors State : clean Device UUID : 3d414ad2:4310c1f9:98f50214:3e9b1e33 Internal Bitmap : 8 sectors from superblock Update Time : Tue Aug 8 14:16:53 2017 Checksum : 8e37d25d - correct Events : 119894 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 2 Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) [root@inscribe4 ~]# mdadm --examine /dev/sdf1 /dev/sdf1: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : ace00fb5:8771d55a:cf36728b:ddc3ce89 Name : intermediate0 Creation Time : Tue Oct 15 21:03:50 2013 Raid Level : raid5 Raid Devices : 6 Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB) Array Size : 9766901760 (9314.44 GiB 10001.31 GB) Used Dev Size : 3906760704 (1862.89 GiB 2000.26 GB) Data Offset : 262144 sectors Super Offset : 8 sectors Unused Space : before=262064 sectors, after=1154 sectors State : clean Device UUID : e238e74b:d40efc43:d34d876d:ca0700b0 Internal Bitmap : 8 sectors from superblock Update Time : Tue Aug 8 14:16:53 2017 Checksum : 28758d54 - correct Events : 119894 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 1 Array State : AAA.A. ('A' == active, '.' == missing, 'R' == replacing) [root@inscribe4 ~]# ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-08-25 8:03 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <DB4PR07MB409B56E9B9173E33079902ED58A0@DB4PR07MB409.eurprd07.prod.outlook.com>
2017-08-08 22:27 ` force raid reassembly Bryan Hepworth
2017-08-08 22:44 ` Joe Landman
[not found] ` <DB4PR07MB40970AB64E677A452895432D58B0@DB4PR07MB409.eurprd07.prod.outlook.com>
[not found] ` <WM!c7e5d36941a99d370e99c9ecff040c34199c6ab48c7c1548590a2e9075a62b0e4a8a44d8c43afabb53f77be4c9108e21!@mailhub-mx3.ncl.ac.uk>
2017-08-09 7:40 ` Bryan Hepworth
2017-08-09 10:34 ` Joe Landman
2017-08-12 17:44 ` Anthony Youngman
[not found] ` <DBXPR07MB415CB58355ECBF4DB0D6632D59B0@DBXPR07MB415.eurprd07.prod.outlook.com>
[not found] ` <WM!13c5dc8d433950e8d178eb39ac57b28caef2d362792181894459f3aa56870cedd0dcd8b5aa75be7d1b060d3021a4afe9!@mailhub-mx3.ncl.ac.uk>
2017-08-25 8:03 ` Bryan Hepworth
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox