Odd --examine output

Odd --examine output

[Vanhorn, Mike]

These two disks are (or, at least, were) part of my failed level 6 array.
What does this output indicate?

# mdadm --examine /dev/sdg
/dev/sdg:
   MBR Magic : aa55
Partition[0] :   3907024002 sectors at           63 (type fd)
# mdadm --examine /dev/sdh
/dev/sdh:
   MBR Magic : aa55
Partition[0] :   3907024002 sectors at           63 (type fd)

Why doesn't it instead give the same sort of metadata that the other disks
show? It's as if there is no raid superblock on these two disks, but how
is this possible if the disks have been part of a working array?


---
Mike VanHorn
Senior Computer Systems Administrator
College of Engineering and Computer Science
Wright State University
265 Russ Engineering Center
937-775-5157
michael.vanhorn@wright.edu
http://www.cecs.wright.edu/~mvanhorn/

Re: Odd --examine output

[Vanhorn, Mike]

>Why doesn't it instead give the same sort of metadata that the other disks
>show? It's as if there is no raid superblock on these two disks, but how
>is this possible if the disks have been part of a working array?

Update: by chance, I discovered that if I

 mdadm -e /dev/sdg1

then I get the output of the superblock. That is, the superblock is found
on the partition (apparently) but not on the disk device. This also works
for disk sdh, for which I couldn't find a superblock, either. However, for
the rest of the disks from the array (sd[cdefi]), there is no such device
as "sdc1", for example.

I have also discovered that for the drives where mdadm -e /dev/sdc (just
the disk) works, the version of the metadata is 0.90.00, where as those
other two where the superblock is on the partition (i.e. sdg1), the
version is 1.2. This makes more sense, since I create the array originally
(back in December) with the following command:

 mdadm --create --verbose /dev/md0 --metadata=1.2 --level=6
--raid-devices=7 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1
/dev/sdh1 /dev/sdi1 --spare-devices=1 /dev/sdj1


So, if I created the array using version 1.2 of the metadata and the
devices specified are sd[cdefghij]1, then why am I not able to see that
metadata now?

I am really confused.

---
Mike VanHorn
Senior Computer Systems Administrator
College of Engineering and Computer Science
Wright State University
265 Russ Engineering Center
937-775-5157
michael.vanhorn@wright.edu
http://www.cecs.wright.edu/~mvanhorn/

Re: Odd --examine output

[Phil Turmel]

On 04/11/2013 04:31 PM, Vanhorn, Mike wrote:

> So, if I created the array using version 1.2 of the metadata and the
> devices specified are sd[cdefghij]1, then why am I not able to see that
> metadata now?
> 
> I am really confused.

Version 0.90 metadata, which lives at the end of the disk or partition,
doesn't include a length or other piece of partition geometry that
clearly identifies the beginning of the array.  So v0.90 metadata that
belongs to the last partition of a disk can be mistaken for metadata
belonging to that whole disk, and vice versa.

v1.x metadata has more device info, so this can't happen to v1.0
metadata, even though it is at the end of the device as well.  v1.1 and
v1.2 metadata are at the beginning, therefore won't be found with the
wrong starting sector.

I suspect what happened to your system:

1) Old array running w/ v0.90 metadata on /dev/sd[]1 is taken out of
service.
2) New array w/ v1.2 metdata is created, but for some reason, the v0.90
metadata doesn't get destroyed.
3) Partition tables on certain disks are wiped, hiding the v1.2 metadata.

I'm not sure if any particular version of mdadm would do #2, but it is
plausible.

#3 is quite concerning, as it suggest the presence of a rogue program
with admin rights, or a rogue administrator, or a *really* questionable
device driver.

Anyways, if you copy the partition table from a properly working drive
to one of the oddballs, it might expose the original partition data,
including the v1.2 metadata.  (Although, you seem to have partitions
starting at sector 63--a real performance killer for modern drives.)

There's other data I suggested you collect in my response to your
original inquiry.  There'd be much less guesswork in my replies if you
supplied it.

Phil

Re: Odd --examine output

[Vanhorn, Mike]

On 4/12/13 10:52 AM, "Phil Turmel" <philip@turmel.org> wrote:

[snip]

>As noted above, the partition tables aren't wiped.  Just the device
>nodes are missing.  You could try a "blockdev --rereadpt /dev/sdX" on
>affected drives to see if it is a transient issue.

That did it! I was able to run blockdev for all of the drives that had
missing devices for the partitions, and then was able to

 mdadm --assemble --force /dev/md0 /dev/sd[cdefghi]1

and it assembled using all of the disks except, for some reason, sde1 and
sdf1. I think sde1 got left out because it had been dropped before the
raid actually stopped, and I think I could have added it back in with

 mdadm /dev/md0 --re-add /dev/sde1

(since /dev/sde actually seems to be fine). However, once I got the
filesystem mounted, my first priority was to get the data off, so I didn't
try to re-add that disk.

I don't know why sdf1 got left out.
 
[snip]

>If the partition is *not* aligned, each large chunk written will have at
>least two R-M-W cycles.

I snipped most of that explanation, but thank you for it; it really helps
me understand what was going on with my partitions.

>I guess "lsdrv" didn't work for you.  I'm naturally curious how it
>failed....

I don't have an lsdrv command, so I did the 'ls -l' that you suggested.

>Anyways, your detailed smartctl reports show big problems:
>
>1)  You have multiple drives with many dozens of pending relocations.
>This suggests that your regular scrubs are not happening on schedule.  A
>"check" scrub turns pending relocations into either real relocations, or
>no error at all (successful rewrite).  Typically the latter.

I've got a raid-check script that runs from cron.weekly. I really did
think it was working, because every week I would check and the array was
re-building.

>2) All of your self-test log entries show "short offline".  That isn't
>rigorous enough.  You need "long offline" self-tests occasionally, too.
> Or just use the long self-test every time.

I will take this into account, and being using the long test.

>3) You have a drive that entirely failed its SMART assessment
>{WD-WMAUR0381532 ==> /dev/sdj} due to excessive actual relocations.
>Replace this drive immediately.

I will. I have a spare disk on the shelf ready to go, once I feel safe
that the data is copied.

[snip]

>NOT a guess.  Back up what you can, while you can, and start over.  Use
>"fdisk -u" so you can ensure partitions start on multiples of eight (8)
>sectors.  (Modern fdisk uses 1MB alignment by default.  Highly
>recommended.)

That is exactly what I'm going to do. I feel like an idiot that there
seems to have been so many things wrong and I didn't realize it. Now,
thanks to your help, and I am much more enlightened.

Thanks!

---
Mike VanHorn
Senior Computer Systems Administrator
College of Engineering and Computer Science
Wright State University
265 Russ Engineering Center
937-775-5157
michael.vanhorn@wright.edu
http://www.cecs.wright.edu/~mvanhorn/

Re: Odd --examine output

[Phil Turmel]

On 04/12/2013 12:47 PM, Vanhorn, Mike wrote:
> On 4/12/13 10:52 AM, "Phil Turmel" <philip@turmel.org> wrote:

[snip /]

>> I guess "lsdrv" didn't work for you.  I'm naturally curious how it
>> failed....
> 
> I don't have an lsdrv command, so I did the 'ls -l' that you suggested.

Ah.  You missed the footnote in the original suggestion.  "lsdrv" is a
script I published on github based on encouragement from others on
linux-raid.  It summarizes a system's storage layout, annotated with
drive, subsystem, and filesystem serial numbers wherever possible.

http://github.com/pturmel/lsdrv

I occasionally poke at it to keep it usable on a variety of distros,
including older ones.  Bug reports welcome.

Phil

Re: Odd --examine output

[Vanhorn, Mike]

On 4/12/13 10:52 AM, "Phil Turmel" <philip@turmel.org> wrote:

>NOT a guess.  Back up what you can, while you can, and start over.  Use
>"fdisk -u" so you can ensure partitions start on multiples of eight (8)
>sectors.  (Modern fdisk uses 1MB alignment by default.  Highly
>recommended.)
>
>

So, if I start the partition at sector 64 (rather than 63), that's better,
right (since 64 is a multiple of 8)? Or is there more math to do and I'm
still not getting it?

---
Mike VanHorn
Senior Computer Systems Administrator
College of Engineering and Computer Science
Wright State University
265 Russ Engineering Center
937-775-5157
michael.vanhorn@wright.edu
http://www.cecs.wright.edu/~mvanhorn/

Re: Odd --examine output

[Phil Turmel]

On 04/15/2013 09:46 AM, Vanhorn, Mike wrote:
> On 4/12/13 10:52 AM, "Phil Turmel" <philip@turmel.org> wrote:
> 
>> NOT a guess.  Back up what you can, while you can, and start over.  Use
>> "fdisk -u" so you can ensure partitions start on multiples of eight (8)
>> sectors.  (Modern fdisk uses 1MB alignment by default.  Highly
>> recommended.)
>>
>>
> 
> So, if I start the partition at sector 64 (rather than 63), that's better,
> right (since 64 is a multiple of 8)? Or is there more math to do and I'm
> still not getting it?

No, you're right.  Sector 64 is a reasonable choice for spinning rust
disks.  The more modern default of sector 2048 (1MB) is intended to
cover future advances, including solid state drives' erase blocks.
(Although it seems that the best SSDs handle that factor on their own.)

Phil

Re: Odd --examine output

[Oliver Schinagl]

On 15-04-13 22:13, John Stoffel wrote:
>>>>>> "Roy" == Roy Sigurd Karlsbakk <roy@karlsbakk.net> writes:
>>>> NOT a guess. Back up what you can, while you can, and start over. Use
>>>> "fdisk -u" so you can ensure partitions start on multiples of eight
>>>> (8)
>>>> sectors. (Modern fdisk uses 1MB alignment by default. Highly
>>>> recommended.)
>>>>
>>>>
>>> So, if I start the partition at sector 64 (rather than 63), that's
>>> better,
>>> right (since 64 is a multiple of 8)? Or is there more math to do and
>>> I'm still not getting it?
> Roy> I still don't understand why people use partitions for RAID when
> Roy> the whole drive is used anyway. Partitions were invented to
> Roy> partition things up and are of no use if you want to spend the
> Roy> whole drive's space for RAID use (or otherwise).
>
> Because if I take a 2tb disk a I put a partition on there which is a
> bit smaller than the full disk, if I then add a new 2tb (or any other
> size) disk which says it's 2tb, but it's really a bit smaller, then
> I'm not screwed.  I've had it happen.
Or if you use them for your OS, and have several raids for /, /usr, /var 
etc.
Or you buy 2 or 3 batches of disks, all varying sizes (1TB != 1TB) and 
having partitions allows you to at least align them all to the same size.
And its even possible to buy a replacement disk, that's actually larger 
but cheaper as prices came down.
A strange use case could be that 1 disk split into 2 partitions could be 
a hot-spare for 2 arrays.

When using 1MiB offsets for partitions, I don't think there's any 
performance loss at all (due to alignment) nor will it slow down 
anything because md is on a partition rather then a disk. Wasting 1 MiB 
per disk is really not significant.

And when using raid1/10 it allows you in theory to wipe the superblock 
and make the partition point tot he real data;   though there's very 
little use for that imo.

I personally, don't see why you want to use the entire disk, is there 
any advantage?
>
> John
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Odd --examine output

[Roy Sigurd Karlsbakk]

> >NOT a guess. Back up what you can, while you can, and start over. Use
> >"fdisk -u" so you can ensure partitions start on multiples of eight
> >(8)
> >sectors. (Modern fdisk uses 1MB alignment by default. Highly
> >recommended.)
> >
> >
> 
> So, if I start the partition at sector 64 (rather than 63), that's
> better,
> right (since 64 is a multiple of 8)? Or is there more math to do and
> I'm still not getting it?

I still don't understand why people use partitions for RAID when the whole drive is used anyway. Partitions were invented to partition things up and are of no use if you want to spend the whole drive's space for RAID use (or otherwise).

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Odd --examine output

[John Stoffel]

>>>>> "Roy" == Roy Sigurd Karlsbakk <roy@karlsbakk.net> writes:

>> >NOT a guess. Back up what you can, while you can, and start over. Use
>> >"fdisk -u" so you can ensure partitions start on multiples of eight
>> >(8)
>> >sectors. (Modern fdisk uses 1MB alignment by default. Highly
>> >recommended.)
>> >
>> >
>> 
>> So, if I start the partition at sector 64 (rather than 63), that's
>> better,
>> right (since 64 is a multiple of 8)? Or is there more math to do and
>> I'm still not getting it?

Roy> I still don't understand why people use partitions for RAID when
Roy> the whole drive is used anyway. Partitions were invented to
Roy> partition things up and are of no use if you want to spend the
Roy> whole drive's space for RAID use (or otherwise).

Because if I take a 2tb disk a I put a partition on there which is a
bit smaller than the full disk, if I then add a new 2tb (or any other
size) disk which says it's 2tb, but it's really a bit smaller, then
I'm not screwed.  I've had it happen.

John

Re: Odd --examine output

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 1693 bytes --]

On Mon Apr 15, 2013 at 04:13:35PM -0400, John Stoffel wrote:

> >>>>> "Roy" == Roy Sigurd Karlsbakk <roy@karlsbakk.net> writes:
> 
> >> >NOT a guess. Back up what you can, while you can, and start over. Use
> >> >"fdisk -u" so you can ensure partitions start on multiples of eight
> >> >(8)
> >> >sectors. (Modern fdisk uses 1MB alignment by default. Highly
> >> >recommended.)
> >> >
> >> >
> >> 
> >> So, if I start the partition at sector 64 (rather than 63), that's
> >> better,
> >> right (since 64 is a multiple of 8)? Or is there more math to do and
> >> I'm still not getting it?
> 
> Roy> I still don't understand why people use partitions for RAID when
> Roy> the whole drive is used anyway. Partitions were invented to
> Roy> partition things up and are of no use if you want to spend the
> Roy> whole drive's space for RAID use (or otherwise).
> 
> Because if I take a 2tb disk a I put a partition on there which is a
> bit smaller than the full disk, if I then add a new 2tb (or any other
> size) disk which says it's 2tb, but it's really a bit smaller, then
> I'm not screwed.  I've had it happen.
> 
Recently? AFAIK, all modern drives (everything over 320G-ish IIRC) use
standardised sizes (i.e. any 2TB disk will present exactly the same
number of sectors). I recall reading that all the manufacturers agreed
to do this to prevent just this sort of issue, though I can't find a
reference to it now.

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Odd --examine output

[Sam Bingner]

On Apr 15, 2013, at 8:43 AM, "Roy Sigurd Karlsbakk" <roy@karlsbakk.net> wrote:

>>> NOT a guess. Back up what you can, while you can, and start over. Use
>>> "fdisk -u" so you can ensure partitions start on multiples of eight
>>> (8)
>>> sectors. (Modern fdisk uses 1MB alignment by default. Highly
>>> recommended.)
>> 
>> So, if I start the partition at sector 64 (rather than 63), that's
>> better,
>> right (since 64 is a multiple of 8)? Or is there more math to do and
>> I'm still not getting it?
> 
> I still don't understand why people use partitions for RAID when the whole drive is used anyway. Partitions were invented to partition things up and are of no use if you want to spend the whole drive's space for RAID use (or otherwise).
> 
I partition devices because it makes it much simpler to know what is on that device.  Granted there are other ways to do this but if I put a partition on it, during some random installer process it won't assume the drive is empty and happily write a partition table or otherwise screw over the device without my knowledge...

I tried going partition-less at one point, but it wasn't worth it to save 1MB of space while considering the previous issue and removing the flexibility to utilize any additional space due to uneven sized drives to, for example, hold a "boot" array that is on the same drives but is a raid1 instead of a raid6... 

Sam

Re: Odd --examine output

[Roy Sigurd Karlsbakk]

> > Roy> I still don't understand why people use partitions for RAID
> > when
> > Roy> the whole drive is used anyway. Partitions were invented to
> > Roy> partition things up and are of no use if you want to spend the
> > Roy> whole drive's space for RAID use (or otherwise).
> >
> > Because if I take a 2tb disk a I put a partition on there which is a
> > bit smaller than the full disk, if I then add a new 2tb (or any
> > other
> > size) disk which says it's 2tb, but it's really a bit smaller, then
> > I'm not screwed. I've had it happen.
> >
> Recently? AFAIK, all modern drives (everything over 320G-ish IIRC) use
> standardised sizes (i.e. any 2TB disk will present exactly the same
> number of sectors). I recall reading that all the manufacturers agreed
> to do this to prevent just this sort of issue, though I can't find a
> reference to it now.

Indeed - you have to go back to smaller drives than those in current production to find varying sizes for equally labelled drive sizes.

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Odd --examine output

[Roy Sigurd Karlsbakk]

> > Roy> I still don't understand why people use partitions for RAID
> > when
> > Roy> the whole drive is used anyway. Partitions were invented to
> > Roy> partition things up and are of no use if you want to spend the
> > Roy> whole drive's space for RAID use (or otherwise).
> >
> > Because if I take a 2tb disk a I put a partition on there which is a
> > bit smaller than the full disk, if I then add a new 2tb (or any
> > other
> > size) disk which says it's 2tb, but it's really a bit smaller, then
> > I'm not screwed. I've had it happen.
> >
> Recently? AFAIK, all modern drives (everything over 320G-ish IIRC) use
> standardised sizes (i.e. any 2TB disk will present exactly the same
> number of sectors). I recall reading that all the manufacturers agreed
> to do this to prevent just this sort of issue, though I can't find a
> reference to it now.

Oh, and btw, last time I had an issue with varying drive sizes, I had two drives in a MD mirror on a Sun^WOracle box (X2100 M2), and the new drive that arrived from Sun^WOracle, was *smaller* than the other one, even though one of the argument I've heard for buying overpriced drives from Sun^WOracle, was that they tuned their firmware to cap their drives to always expose the same amount of sectors. I guess that was just marketing BS. I ended up reinstalling the box…

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Odd --examine output

[John Stoffel]

>> > Roy> I still don't understand why people use partitions for RAID
>> > when
>> > Roy> the whole drive is used anyway. Partitions were invented to
>> > Roy> partition things up and are of no use if you want to spend the
>> > Roy> whole drive's space for RAID use (or otherwise).
>> >
>> > Because if I take a 2tb disk a I put a partition on there which is a
>> > bit smaller than the full disk, if I then add a new 2tb (or any
>> > other
>> > size) disk which says it's 2tb, but it's really a bit smaller, then
>> > I'm not screwed. I've had it happen.
>> >
>> Recently? AFAIK, all modern drives (everything over 320G-ish IIRC) use
>> standardised sizes (i.e. any 2TB disk will present exactly the same
>> number of sectors). I recall reading that all the manufacturers agreed
>> to do this to prevent just this sort of issue, though I can't find a
>> reference to it now.

Roy> Indeed - you have to go back to smaller drives than those in
Roy> current production to find varying sizes for equally labelled
Roy> drive sizes.

Who says it might never happen again, that drive makers go back to
varying block counts for disks?  Or SSDs?  Better safe than sorry, and
a gig of wasted space is cheap insurance.  Plus, it gives you a place
to put boot blocks for GRUB, etc.  

John

Re: Odd --examine output

[Roy Sigurd Karlsbakk]

> Roy> Indeed - you have to go back to smaller drives than those in
> Roy> current production to find varying sizes for equally labelled
> Roy> drive sizes.
> 
> Who says it might never happen again, that drive makers go back to
> varying block counts for disks? Or SSDs? Better safe than sorry, and
> a gig of wasted space is cheap insurance. Plus, it gives you a place
> to put boot blocks for GRUB, etc.

Well, do as you please. When disk vendors standardise, they tend to keep those standards. As I said, the new drive was *smaller* than the older one, so even if I had used partitions, problems would arise. If you're paranoid, though, cut the drive size by 1-5%. Still, all the [124]TB drives I've seen so far have had the same amount of sectors, from Hitachi, Seagate and WD.

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html