mount(2) system call failed: Structure needs cleaning?

mount(2) system call failed: Structure needs cleaning?

[David C. Rankin]

All,

  After my success yesterday in starting an array from a failed system to get
a few of the reaming hylafax/avantfax files from the drive, I decided to start
the array again and make sure there was nothing else I needed to recover from
the drive. I can't get it to mount today, mount says:

# mount /dev/md126 /mnt/af/
mount: /mnt/af: mount(2) system call failed: Structure needs cleaning.

  Huh? Yesterday, when I was done with the fax file recovery, I unplugged the
usb cable attaching the drive to the computer, but did so before I had stopped
the drive. So I plugged it back in and stopped it, then unplugged it again.
All seemed OK.

  However, when I tried to create/start the array as I did yesterday, I
noticed this time mdadm created the array with Version=1.2 instead of the
original Version=1.0. It also didn't list the bitmap as internal. Strange.

  So rather then messing with it further, I stopped the array, and went back
to the original mdadm -D information to create the array as it was before.
This is the original array detail:

# mdadm -D /dev/md126
/dev/md126:
        Version : 1.0
  Creation Time : Thu Aug 21 01:43:22 2008
     Raid Level : raid1
     Array Size : 20972752 (20.00 GiB 21.48 GB)
  Used Dev Size : 20972752 (20.00 GiB 21.48 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Fri Oct 20 15:55:58 2017
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : 1
           UUID : e45cfbeb:77c2b93b:43d3d214:390d0f25
         Events : 19154344

    Number   Major   Minor   RaidDevice State
       0       8       69        0      active sync   /dev/sde5
       -       0        0        1      removed

  So to create the array again I used:

# mdadm --verbose --create /dev/md126 --level=1 --raid-devices=2 \
--metadata=1.0 --bitmap=internal -o \
--uuid=e45cfbeb:77c2b93b:43d3d214:390d0f25 /dev/sdf5 missing

  After creating the array, things look OK, but there is a 16-bytes difference
in the array size and logically, the 'Creation Time' has changed, as well as
the 'Name' field:

# mdadm -D /dev/md126
/dev/md126:
        Version : 1.0
  Creation Time : Thu Oct 26 19:07:26 2017
     Raid Level : raid1
     Array Size : 20972736 (20.00 GiB 21.48 GB)
  Used Dev Size : 20972736 (20.00 GiB 21.48 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Thu Oct 26 19:07:26 2017
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : valkyrie:126  (local to host valkyrie)
           UUID : e45cfbeb:77c2b93b:43d3d214:390d0f25
         Events : 0

    Number   Major   Minor   RaidDevice State
       0       8       85        0      active sync   /dev/sdf5
       -       0        0        1      removed


  So, now I really am stumped (and I didn't solve it during the time I was
writing this e-mail -- this time :). Did something break when I unplugged it
without first stopping it yesterday? Is the 16-byte difference in size what it
considers dirty?

  The logs from trying to mount it say there is an overlap with the superblock:

Oct 26 19:26:04 valkyrie kernel: EXT4-fs (md126): mounting ext3 file system
using the ext4 subsystem
Oct 26 19:26:04 valkyrie kernel: EXT4-fs (md126): ext4_check_descriptors:
Block bitmap for group 0 overlaps superblock
Oct 26 19:26:04 valkyrie kernel: EXT4-fs (md126): ext4_check_descriptors:
Inode bitmap for group 0 overlaps superblock
Oct 26 19:26:04 valkyrie kernel: EXT4-fs (md126): ext4_check_descriptors:
Inode table for group 0 overlaps superblock
Oct 26 19:26:04 valkyrie kernel: EXT4-fs (md126): ext4_check_descriptors:
Block bitmap for group 1 overlaps superblock
Oct 26 19:26:04 valkyrie kernel: EXT4-fs (md126): ext4_check_descriptors:
Block bitmap for group 1 not in group (block 0)!
Oct 26 19:26:04 valkyrie kernel: EXT4-fs (md126): group descriptors corrupted!

  So I'm a bit stuck. Did I corrupt everything when I created the array
without specifying the version 1.0? (I still have the other drive in the array
-- untouched), but as for this array, is there anything I can do to fix what
the log messages are complaining about and mount it again?

  When I get the motherboard back from getting a new set of shiny capacitors,
I had planned on just leaving this drive disconnected, failing it in the
array, and then attempting a re-add (or at this point, is there another way I
should try and pair it back with the saved drive to make sure it doesn't
corrupt the good one?

  Sigh... I knew that usb cable rig was just asking for trouble...

-- 
David C. Rankin, J.D.,P.E.

Re: mount(2) system call failed: Structure needs cleaning?

[David C. Rankin]

On 10/26/2017 07:32 PM, David C. Rankin wrote:
> All,
> 
<snip>
> This is the original array detail:
> 
> # mdadm -D /dev/md126
> /dev/md126:
>         Version : 1.0
>   Creation Time : Thu Aug 21 01:43:22 2008
>      Raid Level : raid1
>      Array Size : 20972752 (20.00 GiB 21.48 GB)
>   Used Dev Size : 20972752 (20.00 GiB 21.48 GB)
>    Raid Devices : 2
>   Total Devices : 1
>     Persistence : Superblock is persistent
> 
>   Intent Bitmap : Internal
> 
>     Update Time : Fri Oct 20 15:55:58 2017
>           State : clean, degraded
>  Active Devices : 1
> Working Devices : 1
>  Failed Devices : 0
>   Spare Devices : 0
> 
>            Name : 1
>            UUID : e45cfbeb:77c2b93b:43d3d214:390d0f25
>          Events : 19154344
> 
>     Number   Major   Minor   RaidDevice State
>        0       8       69        0      active sync   /dev/sde5
>        -       0        0        1      removed
> 
<snip>
>   Sigh... I knew that usb cable rig was just asking for trouble...
> 

Yes, it looks like something on the array was corrupted. When I attempt to set
the --size to the exact size of the array, it says I have 4 less bytes than
the original array size:

# mdadm --verbose --create /dev/md126 --level=1 --raid-devices=2
--metadata=1.0 --bitmap=internal --size=20972752 --readonly
--uuid=e45cfbeb:77c2b93b:43d3d214:390d0f25 /dev/sdf5 missing
mdadm: /dev/sdf5 is smaller than given size. 20972748K < 20972752K + metadata
mdadm: create aborted

(this is an old ATA drive, so I wonder if the 4096 block size is messing
things up?)

Is there a way out of this conundrum?

-- 
David C. Rankin, J.D.,P.E.

Re: mount(2) system call failed: Structure needs cleaning?

[Wols Lists]

On 27/10/17 01:59, David C. Rankin wrote:
> Yes, it looks like something on the array was corrupted. When I attempt to set
> the --size to the exact size of the array, it says I have 4 less bytes than
> the original array size:
> 
> # mdadm --verbose --create /dev/md126 --level=1 --raid-devices=2
> --metadata=1.0 --bitmap=internal --size=20972752 --readonly
> --uuid=e45cfbeb:77c2b93b:43d3d214:390d0f25 /dev/sdf5 missing
> mdadm: /dev/sdf5 is smaller than given size. 20972748K < 20972752K + metadata
> mdadm: create aborted
> 
> (this is an old ATA drive, so I wonder if the 4096 block size is messing
> things up?)
> 
> Is there a way out of this conundrum?

Quite likely, but there's a good chance you'll need a disk hex editor :-(

You say you ran "mdadm --create" over the array? Oh dear oh dear oh
dear! :-(

If the array was originally 1.0, and the re-create created it as 1.2, it
just stomped all over your data.

And now it's probably having problems re-assembling because it's finding
two superblocks and doesn't know what to do with them.

Bottom line, yes I suspect your array is recoverable. But your
filesystem is ALSO damaged, and trying to fix two problems has probably
squared the workload ... :-(

The only good news I can see is it's a mirror. Your best bet is probably
to destroy both superblocks, and try and mount the partition directly
not as a mirror. That way, your only problem will be a damaged
filesystem. Don't try this until you've had better advice than mine, though!

Cheers,
Wol (the bringer of bad news :-(

Re: mount(2) system call failed: Structure needs cleaning?

[David C. Rankin]

On 10/27/2017 04:45 AM, Wols Lists wrote:
> You say you ran "mdadm --create" over the array? Oh dear oh dear oh
> dear! :-(

Yes, that's what I fear... If mdadm wasn't so darn reliable, I'd have to mess
with it more than I do :)
> 
> If the array was originally 1.0, and the re-create created it as 1.2, it
> just stomped all over your data.
> 

That explains it nicely.

> And now it's probably having problems re-assembling because it's finding
> two superblocks and doesn't know what to do with them.
> 
> Bottom line, yes I suspect your array is recoverable. But your
> filesystem is ALSO damaged, and trying to fix two problems has probably
> squared the workload ... :-(
> 
> The only good news I can see is it's a mirror. Your best bet is probably
> to destroy both superblocks, and try and mount the partition directly
> not as a mirror. That way, your only problem will be a damaged
> filesystem. Don't try this until you've had better advice than mine, though!
> 

Then the best bet is probably to just get rid of this array and wait until I
get it installed back in the box with the good disk, leave this unplugged,
boot in degraded mode on the good disk and re-add this one and let it re-sync.
(I suppose?)

How best to clear this array completely so that it doesn't cause mdadm any
problem on the rebuild from the good array?

> Cheers,
> Wol (the bringer of bad news :-(

Oh, the irony...

-- 
David C. Rankin, J.D.,P.E.

Re: mount(2) system call failed: Structure needs cleaning?

[Wols Lists]

On 27/10/17 11:02, David C. Rankin wrote:
> How best to clear this array completely so that it doesn't cause mdadm any
> problem on the rebuild from the good array?

My favourite ... "dd if=/dev/zero of=/dev/sdX". Takes a
LLoooonnnnnngggggggg time.

But there is an option to mdadm that will wipe the superblock. I don't
know whether it will wipe both at once, I'd try it three times and if it
comes back with "can't find superblock" the third time you know it's
done :-) I've never used it so I can't say how it works.

Oh - and when you add it back to the good disk, use --add, and NOT
--re-add. That implies to mdadm that it's a new disk needing re-config.

Cheers,
Wol