Re: [Drbd-dev] [GIT PULL] drbd-8.4 for mainline

[Lutz Vieweg <lvml@5t9.de>]

On 09/23/2011 04:18 PM, Lars Ellenberg wrote:
> On Fri, Sep 23, 2011 at 11:48:17AM +0200, Lutz Vieweg wrote:
>> On 08/24/2011 04:41 PM, Philipp Reisner wrote:
>>>    This release brings a new meta-data format. Forward (8.3 ->   8.4)
>>>    conversion happens complete seamless.
>>
>> Well, it may happen seamless unless you reboot the machine before
>> it has completed the resyncing with the other (still 8.3 running)
>> machine.
>>
>> I happened to require yet some other kernel setting (unrelated to DRBD),
>> installed the changed kernel, while one DRBD resource was still syncing
>> (for the first time after upgrading to 8.4), and rebooted (clean shutdown/restart).
>>
>> After the reboot, DRBD complained about not finding valid meta data
>> on the device, and I had to write new metadata and perform a full sync
>> to rectify the situation.
>
> You probably had 8.4 userland already, but 8.3 module loaded still?

Sort of... the sequence was:
- installed CentOS 6 (which comes with some ancient standard kernel)
- installed drbd-8.4 user-land tools
- reboot 1
- installed linux-3.0.3 + drbd-kernel-module
- reboot 2
- re-installed linux-3.0.3 + drbd-kernel-module
- reboot 3 (while DRBD-sync still in progress)
=== at this point the drbd-meta data was said to be corrupt

> If you then unload the 8.3 module (no more /proc/drbd),
> then do "drbdadm apply-al all".  That should "fix" it.

Too late for that now :-)

> Otherwise, please tell us how to reproduce, and we will figure out what
> needs to be improved during the upgrade/downgrade paths.

Since this was happening just the one time during the upgrade,
I wouldn't dare to say this is highly reproducable.

But I wanted to warn those who might plan for upgrading both
machines of a DRBD-cluster at the same time... this could have
resulted in a loss of data, if the above happened on both sides.

Regards,

Lutz Vieweg