Re: Data still in OSD directories after removing

[Olivier Bonvalet <ceph.list-PaEMFeTk6C1QFI55V6+gNQ@public.gmane.org>]

Le mercredi 21 mai 2014 à 18:20 -0700, Josh Durgin a écrit :
> On 05/21/2014 03:03 PM, Olivier Bonvalet wrote:
> > Le mercredi 21 mai 2014 à 08:20 -0700, Sage Weil a écrit :
> >> You're certain that that is the correct prefix for the rbd image you
> >> removed?  Do you see the objects lists when you do 'rados -p rbd ls - |
> >> grep <prefix>'?
> >
> > I'm pretty sure yes : since I didn't see a lot of space freed by the
> > "rbd snap purge" command, I looked at the RBD prefix before to do the
> > "rbd rm" (it's not the first time I see that problem, but previous time
> > without the RBD prefix I was not able to check).
> >
> > So :
> > - "rados -p sas3copies ls - | grep rb.0.14bfb5a.238e1f29" return nothing
> > at all
> > - # rados stat -p sas3copies rb.0.14bfb5a.238e1f29.00000002f026
> >   error stat-ing sas3copies/rb.0.14bfb5a.238e1f29.00000002f026: No such
> > file or directory
> > - # rados stat -p sas3copies rb.0.14bfb5a.238e1f29.000000000000
> >   error stat-ing sas3copies/rb.0.14bfb5a.238e1f29.000000000000: No such
> > file or directory
> > - # ls -al /var/lib/ceph/osd/ceph-67/current/9.1fe_head/DIR_E/DIR_F/DIR_1/DIR_7/rb.0.14bfb5a.238e1f29.00000002f026__a252_E68871FE__9
> > -rw-r--r-- 1 root root 4194304 oct.   8  2013 /var/lib/ceph/osd/ceph-67/current/9.1fe_head/DIR_E/DIR_F/DIR_1/DIR_7/rb.0.14bfb5a.238e1f29.00000002f026__a252_E68871FE__9
> >
> >
> >> If the objects really are orphaned, teh way to clean them up is via 'rados
> >> -p rbd rm <objectname>'.  I'd like to get to the bottom of how they ended
> >> up that way first, though!
> >
> > I suppose the problem came from me, by doing CTRL+C while "rbd snap
> > purge $IMG".
> > "rados rm -p sas3copies rb.0.14bfb5a.238e1f29.00000002f026" don't remove
> > thoses files, and just answer with a "No such file or directory".
> 
> Those files are all for snapshots, which are removed by the osds
> asynchronously in a process called 'snap trimming'. There's no
> way to directly remove them via rados.
> 
> Since you stopped 'rbd snap purge' partway through, it may
> have removed the reference to the snapshot before removing
> the snapshot itself.
> 
> You can get a list of snapshot ids for the remaining objects
> via the 'rados listsnaps' command, and use
> rados_ioctx_selfmanaged_snap_remove() (no convenient wrapper
> unfortunately) on each of those snapshot ids to be sure they are all
> scheduled for asynchronous deletion.
> 
> Josh
> 

Great : "rados listsnaps" see it :
        # rados listsnaps -p sas3copies rb.0.14bfb5a.238e1f29.00000002f026
        rb.0.14bfb5a.238e1f29.00000002f026:
        cloneid	snaps	size	overlap
        41554	35746	4194304	[]

So, I have to write&compile a wrapper to
rados_ioctx_selfmanaged_snap_remove(), and find a way to obtain a list
of all "orphan" objects ?

I also try to recreate the object (rados put) then remove it (rados rm),
but snapshots still here.

Olivier

_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com