Re: Data still in OSD directories after removing

[Josh Durgin <josh.durgin-4GqslpFJ+cxBDgjK7y7TUQ@public.gmane.org>]

On 05/21/2014 03:03 PM, Olivier Bonvalet wrote:
> Le mercredi 21 mai 2014 à 08:20 -0700, Sage Weil a écrit :
>> You're certain that that is the correct prefix for the rbd image you
>> removed?  Do you see the objects lists when you do 'rados -p rbd ls - |
>> grep <prefix>'?
>
> I'm pretty sure yes : since I didn't see a lot of space freed by the
> "rbd snap purge" command, I looked at the RBD prefix before to do the
> "rbd rm" (it's not the first time I see that problem, but previous time
> without the RBD prefix I was not able to check).
>
> So :
> - "rados -p sas3copies ls - | grep rb.0.14bfb5a.238e1f29" return nothing
> at all
> - # rados stat -p sas3copies rb.0.14bfb5a.238e1f29.00000002f026
>   error stat-ing sas3copies/rb.0.14bfb5a.238e1f29.00000002f026: No such
> file or directory
> - # rados stat -p sas3copies rb.0.14bfb5a.238e1f29.000000000000
>   error stat-ing sas3copies/rb.0.14bfb5a.238e1f29.000000000000: No such
> file or directory
> - # ls -al /var/lib/ceph/osd/ceph-67/current/9.1fe_head/DIR_E/DIR_F/DIR_1/DIR_7/rb.0.14bfb5a.238e1f29.00000002f026__a252_E68871FE__9
> -rw-r--r-- 1 root root 4194304 oct.   8  2013 /var/lib/ceph/osd/ceph-67/current/9.1fe_head/DIR_E/DIR_F/DIR_1/DIR_7/rb.0.14bfb5a.238e1f29.00000002f026__a252_E68871FE__9
>
>
>> If the objects really are orphaned, teh way to clean them up is via 'rados
>> -p rbd rm <objectname>'.  I'd like to get to the bottom of how they ended
>> up that way first, though!
>
> I suppose the problem came from me, by doing CTRL+C while "rbd snap
> purge $IMG".
> "rados rm -p sas3copies rb.0.14bfb5a.238e1f29.00000002f026" don't remove
> thoses files, and just answer with a "No such file or directory".

Those files are all for snapshots, which are removed by the osds
asynchronously in a process called 'snap trimming'. There's no
way to directly remove them via rados.

Since you stopped 'rbd snap purge' partway through, it may
have removed the reference to the snapshot before removing
the snapshot itself.

You can get a list of snapshot ids for the remaining objects
via the 'rados listsnaps' command, and use
rados_ioctx_selfmanaged_snap_remove() (no convenient wrapper
unfortunately) on each of those snapshot ids to be sure they are all
scheduled for asynchronous deletion.

Josh
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com