[Drbd-dev] ASSERT( drbd_md_ss(device->ldev) == device->ldev->md.md_offset )

[Drbd-dev] ASSERT( drbd_md_ss(device->ldev) == device->ldev->md.md_offset )

[Eric Wheeler]

Hello all,

We noticed after resizing one backing device and not the other, that the 
side with the larger device issued the following assertion:

ASSERT( drbd_md_ss(device->ldev) == device->ldev->md.md_offset ) in drbd/obj/default/drbd_main.c:3257

We were issuing a drbdadm resize --assume-clean, but for "reasons", the 
far-end did not resize. Below you can see the trace of the side issuing 
assertions. Is this a bug that should be hanadled intelligent way?

What does the failed assertion indicate? It appears to assert shortly 
after role change to-or-from Primary-or-Secondary.

We are using DRBD 8.4.7-2 from git.

[576057.341024] block drbd8033: logical block size of local backend does not match (drbd:512, backend:4096); was this a late attach?
[576057.343299] block drbd8033: drbd_bm_resize called with capacity == 21854808
[576057.344366] block drbd8033: resync bitmap: bits=2731851 words=42686 pages=84
[576057.345395] block drbd8033: size = 10 GB (10927404 KB)
[576057.381838] block drbd8033: Writing the whole bitmap, size changed
[576057.382865] block drbd8033: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
[576057.384064] block drbd8033: Resync of new storage suppressed with --assume-clean
[576061.509044] block drbd8033: logical block size of local backend does not match (drbd:512, backend:4096); was this a late attach?
[576911.559399] block drbd8033: role( Primary -> Secondary ) 
[576911.560677] block drbd8033: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
[662435.411830] block drbd8033: role( Secondary -> Primary ) 
[663204.542351] block drbd8033: role( Primary -> Secondary ) 
[663204.543515] block drbd8033: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
[748834.185982] block drbd8033: role( Secondary -> Primary ) 
[750155.916752] block drbd8033: role( Primary -> Secondary ) 
[750155.918007] block drbd8033: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
[750155.918768] block drbd8033: ASSERT( drbd_md_ss(device->ldev) == device->ldev->md.md_offset ) in /root/rpmbuild/BUILD/drbd-8.4.7-2-e4242d818e66301920ef28733f533053e924717f/obj/default/drbd_main.c:3257
[835228.859747] block drbd8033: role( Secondary -> Primary ) 
[835228.860996] block drbd8033: ASSERT( drbd_md_ss(device->ldev) == device->ldev->md.md_offset ) in /root/rpmbuild/BUILD/drbd-8.4.7-2-e4242d818e66301920ef28733f533053e924717f/obj/default/drbd_main.c:3257
[836287.928694] block drbd8033: role( Primary -> Secondary ) 
[836287.929886] block drbd8033: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
[836287.930695] block drbd8033: ASSERT( drbd_md_ss(device->ldev) == device->ldev->md.md_offset ) in /root/rpmbuild/BUILD/drbd-8.4.7-2-e4242d818e66301920ef28733f533053e924717f/obj/default/drbd_main.c:3257
[921626.879416] block drbd8033: role( Secondary -> Primary ) 
[921626.880644] block drbd8033: ASSERT( drbd_md_ss(device->ldev) == device->ldev->md.md_offset ) in /root/rpmbuild/BUILD/drbd-8.4.7-2-e4242d818e66301920ef28733f533053e924717f/obj/default/drbd_main.c:3257
[922755.643770] block drbd8033: role( Primary -> Secondary ) 
[922755.645472] block drbd8033: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
[922755.646186] block drbd8033: ASSERT( drbd_md_ss(device->ldev) == device->ldev->md.md_offset ) in /root/rpmbuild/BUILD/drbd-8.4.7-2-e4242d818e66301920ef28733f533053e924717f/obj/default/drbd_main.c:3257
[1008022.668036] block drbd8033: role( Secondary -> Primary ) 
[1008022.669397] block drbd8033: ASSERT( drbd_md_ss(device->ldev) == device->ldev->md.md_offset ) in /root/rpmbuild/BUILD/drbd-8.4.7-2-e4242d818e66301920ef28733f533053e924717f/obj/default/drbd_main.c:3257
[1009802.518711] block drbd8033: role( Primary -> Secondary ) 
[1009802.519884] block drbd8033: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
[1009802.520536] block drbd8033: ASSERT( drbd_md_ss(device->ldev) == device->ldev->md.md_offset ) in /root/rpmbuild/BUILD/drbd-8.4.7-2-e4242d818e66301920ef28733f533053e924717f/obj/default/drbd_main.c:3257
[1024556.860980] block drbd8033: role( Secondary -> Primary ) 
[1024556.863208] block drbd8033: ASSERT( drbd_md_ss(device->ldev) == device->ldev->md.md_offset ) in /root/rpmbuild/BUILD/drbd-8.4.7-2-e4242d818e66301920ef28733f533053e924717f/obj/default/drbd_main.c:3257
[1024614.269377] block drbd8033: role( Primary -> Secondary ) 
[1024614.271469] block drbd8033: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
[1024614.272483] block drbd8033: ASSERT( drbd_md_ss(device->ldev) == device->ldev->md.md_offset ) in /root/rpmbuild/BUILD/drbd-8.4.7-2-e4242d818e66301920ef28733f533053e924717f/obj/default/drbd_main.c:3257


--
Eric Wheeler

Re: [Drbd-dev] ASSERT( drbd_md_ss(device->ldev) == device->ldev->md.md_offset )

[Lars Ellenberg]

On Mon, Sep 19, 2016 at 12:09:12PM -0700, Eric Wheeler wrote:
> Hello all,
> 
> We noticed after resizing one backing device and not the other, that the 
> side with the larger device issued the following assertion:
> 
> ASSERT( drbd_md_ss(device->ldev) == device->ldev->md.md_offset ) in drbd/obj/default/drbd_main.c:3257
>
> We were issuing a drbdadm resize --assume-clean, but for "reasons", the 
> far-end did not resize. Below you can see the trace of the side issuing 
> assertions. Is this a bug that should be hanadled intelligent way?
> 
> What does the failed assertion indicate? It appears to assert shortly 
> after role change to-or-from Primary-or-Secondary.

DRBD is configured for "internal" meta data,
for some reason does some meta data IO, and realizes that someone
resized the backing device under it, without telling it to.

Should not do further harm.

On the box that is logging these asserts,
do a "drbdadm check-resize".

Preferably, you should follow a backend resize immediately
with a drbdadm check-resize (or resize).

> We are using DRBD 8.4.7-2 from git.
> 
> [576057.341024] block drbd8033: logical block size of local backend does not match (drbd:512, backend:4096); was this a late attach?
> [576057.343299] block drbd8033: drbd_bm_resize called with capacity == 21854808
> [576057.344366] block drbd8033: resync bitmap: bits=2731851 words=42686 pages=84
> [576057.345395] block drbd8033: size = 10 GB (10927404 KB)
> [576057.381838] block drbd8033: Writing the whole bitmap, size changed
> [576057.382865] block drbd8033: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
> [576057.384064] block drbd8033: Resync of new storage suppressed with --assume-clean
> [576061.509044] block drbd8033: logical block size of local backend does not match (drbd:512, backend:4096); was this a late attach?
> [576911.559399] block drbd8033: role( Primary -> Secondary ) 
> [576911.560677] block drbd8033: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
> [662435.411830] block drbd8033: role( Secondary -> Primary ) 
> [663204.542351] block drbd8033: role( Primary -> Secondary ) 
> [663204.543515] block drbd8033: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
> [748834.185982] block drbd8033: role( Secondary -> Primary ) 
> [750155.916752] block drbd8033: role( Primary -> Secondary ) 
> [750155.918007] block drbd8033: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
> [750155.918768] block drbd8033: ASSERT( drbd_md_ss(device->ldev) == device->ldev->md.md_offset ) in /root/rpmbuild/BUILD/drbd-8.4.7-2-e4242d818e66301920ef28733f533053e924717f/obj/default/drbd_main.c:3257
> [835228.859747] block drbd8033: role( Secondary -> Primary ) 
> [835228.860996] block drbd8033: ASSERT( drbd_md_ss(device->ldev) == device->ldev->md.md_offset ) in /root/rpmbuild/BUILD/drbd-8.4.7-2-e4242d818e66301920ef28733f533053e924717f/obj/default/drbd_main.c:3257

-- 
: Lars Ellenberg
: LINBIT | Keeping the Digital World Running
: DRBD -- Heartbeat -- Corosync -- Pacemaker
: R&D, Integration, Ops, Consulting, Support

DRBD® and LINBIT® are registered trademarks of LINBIT

Re: [Drbd-dev] ASSERT( drbd_md_ss(device->ldev) == device->ldev->md.md_offset )

[Eric Wheeler]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3663 bytes --]

On Tue, 20 Sep 2016, Lars Ellenberg wrote:

> On Mon, Sep 19, 2016 at 12:09:12PM -0700, Eric Wheeler wrote:
> > Hello all,
> > 
> > We noticed after resizing one backing device and not the other, that the 
> > side with the larger device issued the following assertion:
> > 
> > ASSERT( drbd_md_ss(device->ldev) == device->ldev->md.md_offset ) in drbd/obj/default/drbd_main.c:3257
> >
> > We were issuing a drbdadm resize --assume-clean, but for "reasons", the 
> > far-end did not resize. Below you can see the trace of the side issuing 
> > assertions. Is this a bug that should be hanadled intelligent way?
> > 
> > What does the failed assertion indicate? It appears to assert shortly 
> > after role change to-or-from Primary-or-Secondary.
> 
> DRBD is configured for "internal" meta data,
> for some reason does some meta data IO, and realizes that someone
> resized the backing device under it, without telling it to.
> 
> Should not do further harm.

"Further" ?  Was there any harm (corruption) in the first place?

> On the box that is logging these asserts,
> do a "drbdadm check-resize".
> 
> Preferably, you should follow a backend resize immediately
> with a drbdadm check-resize (or resize).

Is `drbdadm check-resize` a node-local operation?  Does it move metadata 
to the end on both sides, or only on the side invoking check-resize?

Thanks for your help!

--
Eric Wheeler


> 
> > We are using DRBD 8.4.7-2 from git.
> > 
> > [576057.341024] block drbd8033: logical block size of local backend does not match (drbd:512, backend:4096); was this a late attach?
> > [576057.343299] block drbd8033: drbd_bm_resize called with capacity == 21854808
> > [576057.344366] block drbd8033: resync bitmap: bits=2731851 words=42686 pages=84
> > [576057.345395] block drbd8033: size = 10 GB (10927404 KB)
> > [576057.381838] block drbd8033: Writing the whole bitmap, size changed
> > [576057.382865] block drbd8033: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
> > [576057.384064] block drbd8033: Resync of new storage suppressed with --assume-clean
> > [576061.509044] block drbd8033: logical block size of local backend does not match (drbd:512, backend:4096); was this a late attach?
> > [576911.559399] block drbd8033: role( Primary -> Secondary ) 
> > [576911.560677] block drbd8033: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
> > [662435.411830] block drbd8033: role( Secondary -> Primary ) 
> > [663204.542351] block drbd8033: role( Primary -> Secondary ) 
> > [663204.543515] block drbd8033: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
> > [748834.185982] block drbd8033: role( Secondary -> Primary ) 
> > [750155.916752] block drbd8033: role( Primary -> Secondary ) 
> > [750155.918007] block drbd8033: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
> > [750155.918768] block drbd8033: ASSERT( drbd_md_ss(device->ldev) == device->ldev->md.md_offset ) in /root/rpmbuild/BUILD/drbd-8.4.7-2-e4242d818e66301920ef28733f533053e924717f/obj/default/drbd_main.c:3257
> > [835228.859747] block drbd8033: role( Secondary -> Primary ) 
> > [835228.860996] block drbd8033: ASSERT( drbd_md_ss(device->ldev) == device->ldev->md.md_offset ) in /root/rpmbuild/BUILD/drbd-8.4.7-2-e4242d818e66301920ef28733f533053e924717f/obj/default/drbd_main.c:3257
> 
> -- 
> : Lars Ellenberg
> : LINBIT | Keeping the Digital World Running
> : DRBD -- Heartbeat -- Corosync -- Pacemaker
> : R&D, Integration, Ops, Consulting, Support
> 
> DRBD® and LINBIT® are registered trademarks of LINBIT
> _______________________________________________
> drbd-dev mailing list
> drbd-dev@lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-dev
> 

Re: [Drbd-dev] ASSERT( drbd_md_ss(device->ldev) == device->ldev->md.md_offset )

[Lars Ellenberg]

On Fri, Sep 23, 2016 at 03:01:11PM -0700, Eric Wheeler wrote:
> On Tue, 20 Sep 2016, Lars Ellenberg wrote:
> 
> > On Mon, Sep 19, 2016 at 12:09:12PM -0700, Eric Wheeler wrote:
> > > Hello all,
> > > 
> > > We noticed after resizing one backing device and not the other, that the 
> > > side with the larger device issued the following assertion:
> > > 
> > > ASSERT( drbd_md_ss(device->ldev) == device->ldev->md.md_offset ) in drbd/obj/default/drbd_main.c:3257
> > >
> > > We were issuing a drbdadm resize --assume-clean, but for "reasons", the 
> > > far-end did not resize. Below you can see the trace of the side issuing 
> > > assertions. Is this a bug that should be hanadled intelligent way?
> > > 
> > > What does the failed assertion indicate? It appears to assert shortly 
> > > after role change to-or-from Primary-or-Secondary.
> > 
> > DRBD is configured for "internal" meta data,
> > for some reason does some meta data IO, and realizes that someone
> > resized the backing device under it, without telling it to.
> > 
> > Should not do further harm.
> 
> "Further" ?  Was there any harm (corruption) in the first place?

Spamming your logs ;)

> > On the box that is logging these asserts,
> > do a "drbdadm check-resize".
> > 
> > Preferably, you should follow a backend resize immediately
> > with a drbdadm check-resize (or resize).
> 
> Is `drbdadm check-resize` a node-local operation?  Does it move metadata 
> to the end on both sides, or only on the side invoking check-resize?

Depends on whether or not you have a user set size limit configured.

"check-resize" tells DRBD to check for possible backend resize,
and to move its meta data to where it would expect it
if it would attach now.

After the meta data was moved,
we send our new backend size, current exposed size,
and user capped size if any, to the peer.
Which will take the hint and itself double-check
if it's backend has been resized,
if so, do the same (see above).

Also, whenever we receive such size tuples, we check if it would allow
us to agree on a new size, and if so, do it right there.

-- 
: Lars Ellenberg
: LINBIT | Keeping the Digital World Running
: DRBD -- Heartbeat -- Corosync -- Pacemaker
: R&D, Integration, Ops, Consulting, Support

DRBD® and LINBIT® are registered trademarks of LINBIT