All of lore.kernel.org

* Re: Clone data inconsistency in hammer
From: Bartłomiej Święcki @ 2016-12-22 12:33 UTC (permalink / raw)
  To: dillaman-H+wXaHxf7aLQT0dZR+AlfA; +Cc: ceph-devel, ceph-users
In-Reply-To: <20161222101009.32e14565-Q8zpDM2kqfNz3UrvHfEF4cXErentcBXJSavXxUIaH7I@public.gmane.org>

Hi,

I have problems runnign Kraken tools on Hammer/Jewel cluster (official 11.1.0 debs),
it asserts:

/build/ceph-11.1.0/src/mon/MonMap.cc: In function 'void MonMap::sanitize_mons(std::map<std::basic_string<char>, entity_addr_t>&)' thread 7fffd37fe700 time 2016-12-22 12:26:23.457058
/build/ceph-11.1.0/src/mon/MonMap.cc: 70: FAILED assert(mon_info.count(p.first))

I tried to debug it a bit and it looks like mon_info has temporary mon names:

(gdb) p mon_info
$1 = std::map with 3 elements = {["noname-a"] = {name = "noname-a", .....

while it checks for a correct one:

(gdb) p p
$2 = {first = "mon-01-690d38c0-2567-447b-bdfb-0edd137183db", ....

Anyway, I was thinking about the missing image problem - maybe it would be easier
to recreate removed image? Would restoring rbd_header object be enough?

P.S. Adding ceph-devel

On Thu, 22 Dec 2016 10:10:09 +0100
Bartłomiej Święcki <bartlomiej.swiecki@corp.ovh.com> wrote:

> Hi Jason,
> 
> I'll test kraken tools since it happened on production, everything works there
> since the clone is flattened after being created and the production equivalent
> of "test" user can access the image only after it has been flattened.
> 
> The issue happened when someone accidentally removed not-yet-flattened image
> using the user with weaker permissions. Good to hear this has been spotted
> already.
> 
> Thanks for help,
> Bartek
> 
> 
> 
> On Wed, 21 Dec 2016 11:53:57 -0500
> Jason Dillaman <jdillama@redhat.com> wrote:
> 
> > You are unfortunately the second person today to hit an issue where
> > "rbd remove" incorrectly proceeds when it hits a corner-case error.
> > 
> > First things first, when you configure your new user, you needed to
> > give it "rx" permissions to the parent image's pool. If you attempted
> > the clone operation using the "test" user, the clone would have
> > immediately failed due to this issue.
> > 
> > Second, unless this is a test cluster where you can delete the
> > "rbd_children" object in the "rbd" pool (i.e. you don't have any
> > additional clones in the rbd pool) via the rados CLI, you will need to
> > use the Kraken release candidate (or master branch) version of the
> > rados CLI to manually manipulate the "rbd_children" object to remove
> > the dangling reference to the deleted image.
> > 
> > On Wed, Dec 21, 2016 at 6:57 AM, Bartłomiej Święcki
> > <bartlomiej.swiecki@corp.ovh.com> wrote:
> > > Hi,
> > >
> > > I'm currently investigating a case where Ceph cluster ended up with inconsistent clone information.
> > >
> > > Here's a what I did to quickly reproduce:
> > > * Created new cluster (tested in hammer 0.94.6 and jewel 10.2.3)
> > > * Created two pools: test and rbd
> > > * Created base image in pool test, created snapshot, protected it and created clone of this snapshot in pool rbd:
> > >         # rbd -p test create --size 10 --image-format 2 base
> > >         # rbd -p test snap create base@base
> > >         # rbd -p test snap protect base@base
> > >         # rbd clone test/base@base rbd/destination
> > > * Created new user called "test" with rwx permissions to rbd pool only:
> > >         caps: [mon] allow r
> > >         caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=rbd
> > > * Using this newly creted user I removed the cloned image in rbd pool, had errors but finally removed the image:
> > >         # rbd --id test -p rbd rm destination
> > >         2016-12-21 11:50:03.758221 7f32b7459700 -1 librbd::image::OpenRequest: failed to retreive name: (1) Operation not permitted
> > >         2016-12-21 11:50:03.758288 7f32b6c58700 -1 librbd::image::RefreshParentRequest: failed to open parent image: (1) Operation not permitted
> > >         2016-12-21 11:50:03.758312 7f32b6c58700 -1 librbd::image::RefreshRequest: failed to refresh parent image: (1) Operation not permitted
> > >         2016-12-21 11:50:03.758333 7f32b6c58700 -1 librbd::image::OpenRequest: failed to refresh image: (1) Operation not permitted
> > >         2016-12-21 11:50:03.759366 7f32b6c58700 -1 librbd::ImageState: failed to open image: (1) Operation not permitted
> > >         Removing image: 100% complete...done.
> > >
> > > At this point there's no cloned image but the original snapshot still has reference to it:
> > >
> > > # rbd -p test snap unprotect base@base
> > > 2016-12-21 11:53:47.359060 7fee037fe700 -1 librbd::SnapshotUnprotectRequest: cannot unprotect: at least 1 child(ren) [29b0238e1f29] in pool 'rbd'
> > > 2016-12-21 11:53:47.359678 7fee037fe700 -1 librbd::SnapshotUnprotectRequest: encountered error: (16) Device or resource busy
> > > 2016-12-21 11:53:47.359691 7fee037fe700 -1 librbd::SnapshotUnprotectRequest: 0x7fee39ae9340 should_complete_error: ret_val=-16
> > > 2016-12-21 11:53:47.360627 7fee037fe700 -1 librbd::SnapshotUnprotectRequest: 0x7fee39ae9340 should_complete_error: ret_val=-16
> > > rbd: unprotecting snap failed: (16) Device or resource busy
> > >
> > > # rbd -p test children base@base
> > > rbd: listing children failed: (2) No such file or directory2016-12-21
> > > 11:53:08.716987 7ff2b2eaad80 -1 librbd: Error looking up name for image
> > > id 29b0238e1f29 in pool rbd
> > >
> > >
> > > Any ideas on how this could be fixed?
> > >
> > >
> > > Thanks,
> > > Bartek
> > > _______________________________________________
> > > ceph-users mailing list
> > > ceph-users@lists.ceph.com
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > 
> > 
> > 
> > -- 
> > Jason
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

^ permalink raw reply