Re: MD array keeps resyncing after rebooting

[NeilBrown <neilb@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 4173 bytes --]

On Thu, 01 Aug 2013 20:15:31 +0200 Martin Wilck <mwilck@arcor.de> wrote:

> Hello Francis,
> > 
> > As you noticed the state is "Not Consistent". In my understanding it
> > becomes "Consistent" when  the array is stopped.
> 
> Correct.
> 
> > I checked during the shudown process that the array is correctly
> > stopped since at that point I got:
> > 
> > # mdadm -E /dev/sda | egrep "state"
> >         state[0] : Optimal, Consistent
> >    init state[0] : Fully Initialised
> 
> This looks as it should, actually. This looks as if md is doing what
> it's supposed to.
> 
> > After rebooting, it appears that the BIOS changed a part of VD
> > GUID[0]. I'm not sure if that can confuse the kernel and if it's the
> > reason why the kernel shows:
> > 
> >     [  832.944623] md/raid1:md126: not clean -- starting background
> > reconstruction
> 
> The BIOS obviously changes the meta data. The GUID itself shouldn't be
> the problem as long as it's consistently changed everywhere, but it's
> certainly strange to change it - it's meant to be constant and unique
> for this array.
> 
> It would be important to see the state of the meta data after md
> shutdown and immediately after boot (before md actually starts), so that
> we can exactly see what the BIOS has done.
> 
> > but this is obviously where a resync is triggered during each reboot
> > whatever the initial state of the array. The kernel message is
> > actually issued by drivers/md/raid1.c, in particular:
> > 
> >         if (mddev->recovery_cp != MaxSector)
> >                 printk(KERN_NOTICE "md/raid1:%s: not clean"
> >                        " -- starting background reconstruction\n",
> >                        mdname(mddev));
> > 
> > I don't understand the condition and how a resync can be triggered there.
> 
> The kernel is just reacting to something it has been told by
> mdadm/mdmon. mdadm, in turn, just reads the meta data. It is highly
> likely that the meta data indicated that the array was not, or only
> partially, initialized. In this case mdadm will always start a full
> reconstruction.
> 
> > Oh, this is with kernel 3.4.54.
> > 
> > Can you (or anyone else) spot something wrong with these information ?
> 
> Well, obviously the BIOS made a change to the meta data. Why so? We can
> only guess; perhaps something that mdadm wrote to the meta data didn't
> please the BIOS code, and it "thought" it needed to do something
> differently.
> 
> mdadm -E may not be enough here. We need to inspect the raw meta data
> 1. after BIOS created the array and before mdadm started
> 2. after mdadm shutdown
> 3. after BIOS reboot, before mdadm started.
> 4. (if you have a windows driver, it might be interesting to see how the
> meta data looks after windows has shut down after step 1.)
> 
> You can dump the metadata with mdadm --dump, but the result is difficult
> to handle because it's a sparse image the size of your disk.
> Unless all your tools handle sparse files well, you will get stuck.
> 
> Here is a slightly more type-intensive but safer method:
> 
> Use "sg_readcap /dev/sda" to print the LBA of the last block.
> Using this number, run
> 
>  dd if=/dev/sda of=/tmp/sda bs=1b skip=$LBA
> 
> then do hexdump -C /tmp/sda. You see your "DDF anchor" structure. At
> offsets 0x0060 and 0x0068, you find the LBAs of the primary and
> secondary header, in big endian. Use the smaller number of the two
> (usually the secondary header at 0x0068). In may case the hexdump line reads
> 
> 00000060  00 00 00 00 3a 37 c0 40  00 00 00 00 3a 37 20 50
> 
> The primary LBA is 0x3a37c040, the secondary 3a372050, which is less.
> Next, using the smaller number, run
> 
> dd if=/dev/sda bs=1b skip=$((0x3a372050)) | gzip -c /tmp/sda-ddf.gz
> 

For future reference, with mdadm 3.3 you can
 mkdir /tmp/md
 mdadm --dump /tmp/md /dev/sd*
 tar czSvf /tmp/md.tgz /tmp/md

The files in /tmp/md are sparse (lots of zeros). 
The 'S' flag to tar tells it to handle these well.
So

 tar xvf /tmp/md.tgz

will recover the sparse files which will have the metadata and nothing else.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]