[Drbd-dev] [PATCH] drbd-utils: Fix the default node id initialization when upgrade v8 to v9

[Drbd-dev] [PATCH] drbd-utils: Fix the default node id initialization when upgrade v8 to v9

[Nick Wang]

Node id will be initialized as "0" in md_convert_08_to_09() when upgrading to drbd9 
(still using two nodes without changing configuration like expliciting set the node id). 
However, "0" is legal as a node id, which may cause an "ambiguous node id" error on certain 
node when starting the resource. It is more appropriate to use "-1" as the default node id 
when initializing meta data.
This issue can be reporduced on the drbd-utils-8.9.10.

Error happened when starting service after creating md in drbd9, command are as follow:
upgrade-2:~ # drbdadm create-md single
You want me to create a v09 style flexible-size internal meta data block.
There appears to be a v08 flexible-size internal meta data block
already in place on /dev/vdb5 at byte offset 1073737728

Valid v08 meta-data found, convert to v09?
[need to type 'yes' to confirm] yes

Writing meta data...
New drbd meta data block successfully created.

upgrade-2:~ # drbdadm up single
3: Failure: (119) No valid meta-data signature found.

    ==> Use 'drbdadm create-md res' to initialize meta-data area. <==

Command 'drbdsetup attach 3 /dev/vdb5 /dev/vdb5 internal --on-io-error=pass_on' terminated with exit code 10

upgrade-2:~ # drbdadm status single
single role:Secondary
  disk:Diskless
  upgrade-1 connection:StandAlone

Error log messages in node "upgrade-2":
drbd single/0 drbd3: ambiguous node id: meta-data: 0, config: 1

Signed-off-by: Nick Wang <nwang@suse.com>
CC: Lars Ellenberg <lars.ellenberg@linbit.com>
CC: Roland Kammerer <roland.kammerer@linbit.com>
CC: drbd-dev@lists.linbit.com 

---
 user/shared/drbdmeta.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/user/shared/drbdmeta.c b/user/shared/drbdmeta.c
index 9a28984..e2a9606 100644
--- a/user/shared/drbdmeta.c
+++ b/user/shared/drbdmeta.c
@@ -3868,6 +3868,7 @@ void md_convert_08_to_09(struct format *cfg)
 
 	cfg->md.flags &= ~(MDF_CONNECTED_IND | MDF_FULL_SYNC | MDF_PEER_OUT_DATED);
 
+	cfg->md.node_id = -1;
 	cfg->md.magic = DRBD_MD_MAGIC_09;
 	re_initialize_md_offsets(cfg);
 
-- 
1.8.5.6

Re: [Drbd-dev] [PATCH] drbd-utils: Fix the default node id initialization when upgrade v8 to v9

[Roland Kammerer]

On Fri, Jan 13, 2017 at 02:15:58PM +0800, Nick Wang wrote:
> Signed-off-by: Nick Wang <nwang@suse.com>
> CC: Lars Ellenberg <lars.ellenberg@linbit.com>
> CC: Roland Kammerer <roland.kammerer@linbit.com>
> CC: drbd-dev@lists.linbit.com 
> 
> ---
>  user/shared/drbdmeta.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/user/shared/drbdmeta.c b/user/shared/drbdmeta.c
> index 9a28984..e2a9606 100644
> --- a/user/shared/drbdmeta.c
> +++ b/user/shared/drbdmeta.c
> @@ -3868,6 +3868,7 @@ void md_convert_08_to_09(struct format *cfg)
>  
>  	cfg->md.flags &= ~(MDF_CONNECTED_IND | MDF_FULL_SYNC | MDF_PEER_OUT_DATED);
>  
> +	cfg->md.node_id = -1;
>  	cfg->md.magic = DRBD_MD_MAGIC_09;
>  	re_initialize_md_offsets(cfg);
>  
> -- 
> 1.8.5.6

Looks good to me, we do the same if we initialize v9 meta data, but so
far we did not do it when we converted.

Thanks, rck

Re: [Drbd-dev] [PATCH] drbd-utils: Fix the default node id initialization when upgrade v8 to v9

[Lars Ellenberg]

On Fri, Jan 13, 2017 at 02:15:58PM +0800, Nick Wang wrote:
> Node id will be initialized as "0" in md_convert_08_to_09() when upgrading to drbd9 
> (still using two nodes without changing configuration like expliciting set the node id). 

With DRBD 9, you *have* to set the node id in the configuration.
If there is a way around that, we need to fix it.

Also, IMO, in virtually all 8 -> 9 upgrade scenarios, you are better off
creating new DRBD 9 meta data instead of trying to "convert".

Still; your suggestion seems sensible enough.

Thanks,
    Lars

> diff --git a/user/shared/drbdmeta.c b/user/shared/drbdmeta.c
> index 9a28984..e2a9606 100644
> --- a/user/shared/drbdmeta.c
> +++ b/user/shared/drbdmeta.c
> @@ -3868,6 +3868,7 @@ void md_convert_08_to_09(struct format *cfg)
>  
>  	cfg->md.flags &= ~(MDF_CONNECTED_IND | MDF_FULL_SYNC | MDF_PEER_OUT_DATED);
>  
> +	cfg->md.node_id = -1;
>  	cfg->md.magic = DRBD_MD_MAGIC_09;
>  	re_initialize_md_offsets(cfg);

Re: [Drbd-dev] [PATCH] drbd-utils: Fix the default node id initialization when upgrade v8 to v9

[Nick Wang]

>>> On 2017-1-13 at 19:03, in message <20170113110345.GA9172@soda.linbit>, Lars
Ellenberg <lars.ellenberg@linbit.com> wrote:
> With DRBD 9, you *have* to set the node id in the configuration.
> If there is a way around that, we need to fix it.
> 
I can see error messages when lacking of node-id or connection sections in multiple nodes scenario.

> Also, IMO, in virtually all 8 -> 9 upgrade scenarios, you are better off
> creating new DRBD 9 meta data instead of trying to "convert".
> 

Thanks for the information. Will always do so when upgrading from v8 to v9.

Best regards,
Nick 

Re: [Drbd-dev] [PATCH] drbd-utils: Fix the default node id initialization when upgrade v8 to v9

[Lars Ellenberg]

On Fri, Jan 13, 2017 at 10:01:50AM -0900, David M. Syzdek wrote:
> 
> 
> > On Jan 13, 2017, at 2:03 AM, Lars Ellenberg <lars.ellenberg@linbit.com> wrote:
> > 
> > On Fri, Jan 13, 2017 at 02:15:58PM +0800, Nick Wang wrote:
> >> Node id will be initialized as "0" in md_convert_08_to_09() when upgrading to drbd9 
> >> (still using two nodes without changing configuration like expliciting set the node id). 
> > 
> > With DRBD 9, you *have* to set the node id in the configuration.
> > If there is a way around that, we need to fix it.
> 
> If by configuration, you are referring the .res resource file, then
> this may be the intention, however this is not currently the case.  I
> am able to initialize a new resource on two peers using the following
> config without using the ’node-id’ directives:

Yes, I forgot, if we find a two-node-only 8 compat config,
we sort the IP addresses (or something),
and use that to implicitly assign node-id.

works for most "simple" setups,
may break for more interesting ones
(IPs look different depending on node)

> > Also, IMO, in virtually all 8 -> 9 upgrade scenarios, you are better off
> > creating new DRBD 9 meta data instead of trying to "convert”.
> 
> Is there a technical reason

There are two basic scenarios, "rolling" upgrade,
and "stop, replace, and restart the world". 

"stop the world" obviously means downtime.
But for the "rolling" upgrade,
you need more steps and at least a switchover,
which typically also implies a downtime.

My preferred upgrade procedure would be "replace the world"
 * make sure all data is "stable" and in sync
 * stop all services (yes, that means a short down time)
 * upgrade modules
 * recreate meta data, and prepare to skip the initial sync,
   because we know everything is in sync (see first step)
   potentially allocating more bitmap slots in preparation for N > 2.
 * bring it all online.

If that is not acceptable, or you think the rolling upgrade incurs less
service interruption, you go with the "upgrade meta data in place"
procedure, and hope we got that right :-)

-- 
: Lars Ellenberg
: LINBIT | Keeping the Digital World Running
: DRBD -- Heartbeat -- Corosync -- Pacemaker
: R&D, Integration, Ops, Consulting, Support

DRBD® and LINBIT® are registered trademarks of LINBIT

Re: [Drbd-dev] [PATCH] drbd-utils: Fix the default node id initialization when upgrade v8 to v9

[Nick Wang]

>>> On 1/16/2017 at  6:35 下午, in message <20170116103502.GF9172@soda.linbit>,
Lars Ellenberg <lars.ellenberg@linbit.com> wrote:
> > > Also, IMO, in virtually all 8 -> 9 upgrade scenarios, you are better off
> > > creating new DRBD 9 meta data instead of trying to "convert”.
> > 
> > Is there a technical reason
> 
> There are two basic scenarios, "rolling" upgrade,
> and "stop, replace, and restart the world". 
> 
> "stop the world" obviously means downtime.
> But for the "rolling" upgrade,
> you need more steps and at least a switchover,
> which typically also implies a downtime.
> 
> My preferred upgrade procedure would be "replace the world"
>  * make sure all data is "stable" and in sync
>  * stop all services (yes, that means a short down time)
>  * upgrade modules
>  * recreate meta data, and prepare to skip the initial sync,
>    because we know everything is in sync (see first step)
>    potentially allocating more bitmap slots in preparation for N > 2.
>  * bring it all online.
> 
> If that is not acceptable, or you think the rolling upgrade incurs less
> service interruption, you go with the "upgrade meta data in place"
> procedure, and hope we got that right :-)
> 
The rolling upgrade as long as the network protocol support can 
shorten the downtime to the switch-over time. If all data is in sync, 
i think the md(activity log and bitmaps) is safe to be replaced.

So far, for all the scenarios i tried, both "convert meta data" for still 
2 nodes and "replace meta data with a new one" for adding more nodes
works for a rolling upgrade on secondary node when all data is in sync:)

Best regards,
Nick