* Re: Data still in OSD directories after removing [not found] <1400578369.11397.9.camel@localhost> @ 2014-05-21 10:17 ` Olivier Bonvalet 2014-05-21 15:20 ` Sage Weil 0 siblings, 1 reply; 6+ messages in thread From: Olivier Bonvalet @ 2014-05-21 10:17 UTC (permalink / raw) To: ceph-users-idqoXFIVOFJgJs9I8MT0rw; +Cc: Ceph-devel Hi, I have a lot of space wasted by this problem (about 10GB per OSD, just for this RBD image). If OSDs can't detect orphans files, should I manually detect them, then remove them ? This command can do the job, at least for this image prefix : find /var/lib/ceph/osd/ -name 'rb.0.14bfb5a.238e1f29.*' -delete Thanks for any advice, Olivier PS : not sure if this kind of problem is for the user or dev mailing list. Le mardi 20 mai 2014 à 11:32 +0200, Olivier Bonvalet a écrit : > Hi, > > short : I removed a 1TB RBD image, but I still see files about it on > OSD. > > > long : > 1) I did : "rbd snap purge $pool/$img" > but since it overload the cluster, I stopped it (CTRL+C) > 2) latter, "rbd snap purge $pool/$img" > 3) then, "rbd rm $pool/$img" > > now, on the disk I can found files of this v1 RBD image (prefix was > rb.0.14bfb5a.238e1f29) : > > # find /var/lib/ceph/osd/ceph-64/ -name 'rb.0.14bfb5a.238e1f29.*' > /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_3/rb.0.14bfb5a.238e1f29.000000021431__snapdir_C96635C1__9 > /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_3/rb.0.14bfb5a.238e1f29.000000005622__a252_32F435C1__9 > /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_3/rb.0.14bfb5a.238e1f29.000000021431__a252_C96635C1__9 > /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_3/rb.0.14bfb5a.238e1f29.000000005622__snapdir_32F435C1__9 > /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_9/rb.0.14bfb5a.238e1f29.000000011e08__a172_594495C1__9 > /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_9/rb.0.14bfb5a.238e1f29.000000011e08__snapdir_594495C1__9 > /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_A/rb.0.14bfb5a.238e1f29.000000021620__a252_779FA5C1__9 > ... > > > So, is there a way to force OSD to detect if files are orphans, then > remove them ? > > Thanks, > Olivier > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Data still in OSD directories after removing 2014-05-21 10:17 ` Data still in OSD directories after removing Olivier Bonvalet @ 2014-05-21 15:20 ` Sage Weil [not found] ` <alpine.DEB.2.00.1405210818560.1689-vIokxiIdD2AQNTJnQDzGJqxOck334EZe@public.gmane.org> 0 siblings, 1 reply; 6+ messages in thread From: Sage Weil @ 2014-05-21 15:20 UTC (permalink / raw) To: Olivier Bonvalet; +Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw, Ceph-devel Hi Olivier, On Wed, 21 May 2014, Olivier Bonvalet wrote: > Hi, > > I have a lot of space wasted by this problem (about 10GB per OSD, just > for this RBD image). > If OSDs can't detect orphans files, should I manually detect them, then > remove them ? > > This command can do the job, at least for this image prefix : > find /var/lib/ceph/osd/ -name 'rb.0.14bfb5a.238e1f29.*' -delete You should definitely not do this! :) You're certain that that is the correct prefix for the rbd image you removed? Do you see the objects lists when you do 'rados -p rbd ls - | grep <prefix>'? If the objects really are orphaned, teh way to clean them up is via 'rados -p rbd rm <objectname>'. I'd like to get to the bottom of how they ended up that way first, though! sage > > Thanks for any advice, > Olivier > > PS : not sure if this kind of problem is for the user or dev mailing > list. > > Le mardi 20 mai 2014 ? 11:32 +0200, Olivier Bonvalet a ?crit : > > Hi, > > > > short : I removed a 1TB RBD image, but I still see files about it on > > OSD. > > > > > > long : > > 1) I did : "rbd snap purge $pool/$img" > > but since it overload the cluster, I stopped it (CTRL+C) > > 2) latter, "rbd snap purge $pool/$img" > > 3) then, "rbd rm $pool/$img" > > > > now, on the disk I can found files of this v1 RBD image (prefix was > > rb.0.14bfb5a.238e1f29) : > > > > # find /var/lib/ceph/osd/ceph-64/ -name 'rb.0.14bfb5a.238e1f29.*' > > /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_3/rb.0.14bfb5a.238e1f29.000000021431__snapdir_C96635C1__9 > > /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_3/rb.0.14bfb5a.238e1f29.000000005622__a252_32F435C1__9 > > /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_3/rb.0.14bfb5a.238e1f29.000000021431__a252_C96635C1__9 > > /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_3/rb.0.14bfb5a.238e1f29.000000005622__snapdir_32F435C1__9 > > /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_9/rb.0.14bfb5a.238e1f29.000000011e08__a172_594495C1__9 > > /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_9/rb.0.14bfb5a.238e1f29.000000011e08__snapdir_594495C1__9 > > /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_A/rb.0.14bfb5a.238e1f29.000000021620__a252_779FA5C1__9 > > ... > > > > > > So, is there a way to force OSD to detect if files are orphans, then > > remove them ? > > > > Thanks, > > Olivier > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <alpine.DEB.2.00.1405210818560.1689-vIokxiIdD2AQNTJnQDzGJqxOck334EZe@public.gmane.org>]
* Re: Data still in OSD directories after removing [not found] ` <alpine.DEB.2.00.1405210818560.1689-vIokxiIdD2AQNTJnQDzGJqxOck334EZe@public.gmane.org> @ 2014-05-21 22:03 ` Olivier Bonvalet 2014-05-22 1:20 ` Josh Durgin 0 siblings, 1 reply; 6+ messages in thread From: Olivier Bonvalet @ 2014-05-21 22:03 UTC (permalink / raw) To: Sage Weil; +Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw, Ceph-devel Le mercredi 21 mai 2014 à 08:20 -0700, Sage Weil a écrit : > > You should definitely not do this! :) Of course ;) > > You're certain that that is the correct prefix for the rbd image you > removed? Do you see the objects lists when you do 'rados -p rbd ls - | > grep <prefix>'? I'm pretty sure yes : since I didn't see a lot of space freed by the "rbd snap purge" command, I looked at the RBD prefix before to do the "rbd rm" (it's not the first time I see that problem, but previous time without the RBD prefix I was not able to check). So : - "rados -p sas3copies ls - | grep rb.0.14bfb5a.238e1f29" return nothing at all - # rados stat -p sas3copies rb.0.14bfb5a.238e1f29.00000002f026 error stat-ing sas3copies/rb.0.14bfb5a.238e1f29.00000002f026: No such file or directory - # rados stat -p sas3copies rb.0.14bfb5a.238e1f29.000000000000 error stat-ing sas3copies/rb.0.14bfb5a.238e1f29.000000000000: No such file or directory - # ls -al /var/lib/ceph/osd/ceph-67/current/9.1fe_head/DIR_E/DIR_F/DIR_1/DIR_7/rb.0.14bfb5a.238e1f29.00000002f026__a252_E68871FE__9 -rw-r--r-- 1 root root 4194304 oct. 8 2013 /var/lib/ceph/osd/ceph-67/current/9.1fe_head/DIR_E/DIR_F/DIR_1/DIR_7/rb.0.14bfb5a.238e1f29.00000002f026__a252_E68871FE__9 > If the objects really are orphaned, teh way to clean them up is via 'rados > -p rbd rm <objectname>'. I'd like to get to the bottom of how they ended > up that way first, though! I suppose the problem came from me, by doing CTRL+C while "rbd snap purge $IMG". "rados rm -p sas3copies rb.0.14bfb5a.238e1f29.00000002f026" don't remove thoses files, and just answer with a "No such file or directory". Thanks, Olivier _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Data still in OSD directories after removing 2014-05-21 22:03 ` Olivier Bonvalet @ 2014-05-22 1:20 ` Josh Durgin [not found] ` <537D50EA.9020901-4GqslpFJ+cxBDgjK7y7TUQ@public.gmane.org> 0 siblings, 1 reply; 6+ messages in thread From: Josh Durgin @ 2014-05-22 1:20 UTC (permalink / raw) To: Olivier Bonvalet, Sage Weil; +Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw, Ceph-devel On 05/21/2014 03:03 PM, Olivier Bonvalet wrote: > Le mercredi 21 mai 2014 à 08:20 -0700, Sage Weil a écrit : >> You're certain that that is the correct prefix for the rbd image you >> removed? Do you see the objects lists when you do 'rados -p rbd ls - | >> grep <prefix>'? > > I'm pretty sure yes : since I didn't see a lot of space freed by the > "rbd snap purge" command, I looked at the RBD prefix before to do the > "rbd rm" (it's not the first time I see that problem, but previous time > without the RBD prefix I was not able to check). > > So : > - "rados -p sas3copies ls - | grep rb.0.14bfb5a.238e1f29" return nothing > at all > - # rados stat -p sas3copies rb.0.14bfb5a.238e1f29.00000002f026 > error stat-ing sas3copies/rb.0.14bfb5a.238e1f29.00000002f026: No such > file or directory > - # rados stat -p sas3copies rb.0.14bfb5a.238e1f29.000000000000 > error stat-ing sas3copies/rb.0.14bfb5a.238e1f29.000000000000: No such > file or directory > - # ls -al /var/lib/ceph/osd/ceph-67/current/9.1fe_head/DIR_E/DIR_F/DIR_1/DIR_7/rb.0.14bfb5a.238e1f29.00000002f026__a252_E68871FE__9 > -rw-r--r-- 1 root root 4194304 oct. 8 2013 /var/lib/ceph/osd/ceph-67/current/9.1fe_head/DIR_E/DIR_F/DIR_1/DIR_7/rb.0.14bfb5a.238e1f29.00000002f026__a252_E68871FE__9 > > >> If the objects really are orphaned, teh way to clean them up is via 'rados >> -p rbd rm <objectname>'. I'd like to get to the bottom of how they ended >> up that way first, though! > > I suppose the problem came from me, by doing CTRL+C while "rbd snap > purge $IMG". > "rados rm -p sas3copies rb.0.14bfb5a.238e1f29.00000002f026" don't remove > thoses files, and just answer with a "No such file or directory". Those files are all for snapshots, which are removed by the osds asynchronously in a process called 'snap trimming'. There's no way to directly remove them via rados. Since you stopped 'rbd snap purge' partway through, it may have removed the reference to the snapshot before removing the snapshot itself. You can get a list of snapshot ids for the remaining objects via the 'rados listsnaps' command, and use rados_ioctx_selfmanaged_snap_remove() (no convenient wrapper unfortunately) on each of those snapshot ids to be sure they are all scheduled for asynchronous deletion. Josh _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <537D50EA.9020901-4GqslpFJ+cxBDgjK7y7TUQ@public.gmane.org>]
* Re: Data still in OSD directories after removing [not found] ` <537D50EA.9020901-4GqslpFJ+cxBDgjK7y7TUQ@public.gmane.org> @ 2014-05-22 8:56 ` Olivier Bonvalet 2016-04-29 12:09 ` [ceph-users] " Andrey Korolyov 0 siblings, 1 reply; 6+ messages in thread From: Olivier Bonvalet @ 2014-05-22 8:56 UTC (permalink / raw) To: Josh Durgin; +Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw, Ceph-devel Le mercredi 21 mai 2014 à 18:20 -0700, Josh Durgin a écrit : > On 05/21/2014 03:03 PM, Olivier Bonvalet wrote: > > Le mercredi 21 mai 2014 à 08:20 -0700, Sage Weil a écrit : > >> You're certain that that is the correct prefix for the rbd image you > >> removed? Do you see the objects lists when you do 'rados -p rbd ls - | > >> grep <prefix>'? > > > > I'm pretty sure yes : since I didn't see a lot of space freed by the > > "rbd snap purge" command, I looked at the RBD prefix before to do the > > "rbd rm" (it's not the first time I see that problem, but previous time > > without the RBD prefix I was not able to check). > > > > So : > > - "rados -p sas3copies ls - | grep rb.0.14bfb5a.238e1f29" return nothing > > at all > > - # rados stat -p sas3copies rb.0.14bfb5a.238e1f29.00000002f026 > > error stat-ing sas3copies/rb.0.14bfb5a.238e1f29.00000002f026: No such > > file or directory > > - # rados stat -p sas3copies rb.0.14bfb5a.238e1f29.000000000000 > > error stat-ing sas3copies/rb.0.14bfb5a.238e1f29.000000000000: No such > > file or directory > > - # ls -al /var/lib/ceph/osd/ceph-67/current/9.1fe_head/DIR_E/DIR_F/DIR_1/DIR_7/rb.0.14bfb5a.238e1f29.00000002f026__a252_E68871FE__9 > > -rw-r--r-- 1 root root 4194304 oct. 8 2013 /var/lib/ceph/osd/ceph-67/current/9.1fe_head/DIR_E/DIR_F/DIR_1/DIR_7/rb.0.14bfb5a.238e1f29.00000002f026__a252_E68871FE__9 > > > > > >> If the objects really are orphaned, teh way to clean them up is via 'rados > >> -p rbd rm <objectname>'. I'd like to get to the bottom of how they ended > >> up that way first, though! > > > > I suppose the problem came from me, by doing CTRL+C while "rbd snap > > purge $IMG". > > "rados rm -p sas3copies rb.0.14bfb5a.238e1f29.00000002f026" don't remove > > thoses files, and just answer with a "No such file or directory". > > Those files are all for snapshots, which are removed by the osds > asynchronously in a process called 'snap trimming'. There's no > way to directly remove them via rados. > > Since you stopped 'rbd snap purge' partway through, it may > have removed the reference to the snapshot before removing > the snapshot itself. > > You can get a list of snapshot ids for the remaining objects > via the 'rados listsnaps' command, and use > rados_ioctx_selfmanaged_snap_remove() (no convenient wrapper > unfortunately) on each of those snapshot ids to be sure they are all > scheduled for asynchronous deletion. > > Josh > Great : "rados listsnaps" see it : # rados listsnaps -p sas3copies rb.0.14bfb5a.238e1f29.00000002f026 rb.0.14bfb5a.238e1f29.00000002f026: cloneid snaps size overlap 41554 35746 4194304 [] So, I have to write&compile a wrapper to rados_ioctx_selfmanaged_snap_remove(), and find a way to obtain a list of all "orphan" objects ? I also try to recreate the object (rados put) then remove it (rados rm), but snapshots still here. Olivier _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [ceph-users] Data still in OSD directories after removing 2014-05-22 8:56 ` Olivier Bonvalet @ 2016-04-29 12:09 ` Andrey Korolyov 0 siblings, 0 replies; 6+ messages in thread From: Andrey Korolyov @ 2016-04-29 12:09 UTC (permalink / raw) To: Olivier Bonvalet Cc: ceph-users@lists.ceph.com, Ceph-devel, Sage Weil, Josh Durgin On Thu, May 22, 2014 at 12:56 PM, Olivier Bonvalet <ceph.list@daevel.fr> wrote: > > Le mercredi 21 mai 2014 à 18:20 -0700, Josh Durgin a écrit : >> On 05/21/2014 03:03 PM, Olivier Bonvalet wrote: >> > Le mercredi 21 mai 2014 à 08:20 -0700, Sage Weil a écrit : >> >> You're certain that that is the correct prefix for the rbd image you >> >> removed? Do you see the objects lists when you do 'rados -p rbd ls - | >> >> grep <prefix>'? >> > >> > I'm pretty sure yes : since I didn't see a lot of space freed by the >> > "rbd snap purge" command, I looked at the RBD prefix before to do the >> > "rbd rm" (it's not the first time I see that problem, but previous time >> > without the RBD prefix I was not able to check). >> > >> > So : >> > - "rados -p sas3copies ls - | grep rb.0.14bfb5a.238e1f29" return nothing >> > at all >> > - # rados stat -p sas3copies rb.0.14bfb5a.238e1f29.00000002f026 >> > error stat-ing sas3copies/rb.0.14bfb5a.238e1f29.00000002f026: No such >> > file or directory >> > - # rados stat -p sas3copies rb.0.14bfb5a.238e1f29.000000000000 >> > error stat-ing sas3copies/rb.0.14bfb5a.238e1f29.000000000000: No such >> > file or directory >> > - # ls -al /var/lib/ceph/osd/ceph-67/current/9.1fe_head/DIR_E/DIR_F/DIR_1/DIR_7/rb.0.14bfb5a.238e1f29.00000002f026__a252_E68871FE__9 >> > -rw-r--r-- 1 root root 4194304 oct. 8 2013 /var/lib/ceph/osd/ceph-67/current/9.1fe_head/DIR_E/DIR_F/DIR_1/DIR_7/rb.0.14bfb5a.238e1f29.00000002f026__a252_E68871FE__9 >> > >> > >> >> If the objects really are orphaned, teh way to clean them up is via 'rados >> >> -p rbd rm <objectname>'. I'd like to get to the bottom of how they ended >> >> up that way first, though! >> > >> > I suppose the problem came from me, by doing CTRL+C while "rbd snap >> > purge $IMG". >> > "rados rm -p sas3copies rb.0.14bfb5a.238e1f29.00000002f026" don't remove >> > thoses files, and just answer with a "No such file or directory". >> >> Those files are all for snapshots, which are removed by the osds >> asynchronously in a process called 'snap trimming'. There's no >> way to directly remove them via rados. >> >> Since you stopped 'rbd snap purge' partway through, it may >> have removed the reference to the snapshot before removing >> the snapshot itself. >> >> You can get a list of snapshot ids for the remaining objects >> via the 'rados listsnaps' command, and use >> rados_ioctx_selfmanaged_snap_remove() (no convenient wrapper >> unfortunately) on each of those snapshot ids to be sure they are all >> scheduled for asynchronous deletion. >> >> Josh >> > > Great : "rados listsnaps" see it : > # rados listsnaps -p sas3copies rb.0.14bfb5a.238e1f29.00000002f026 > rb.0.14bfb5a.238e1f29.00000002f026: > cloneid snaps size overlap > 41554 35746 4194304 [] > > So, I have to write&compile a wrapper to > rados_ioctx_selfmanaged_snap_remove(), and find a way to obtain a list > of all "orphan" objects ? > > I also try to recreate the object (rados put) then remove it (rados rm), > but snapshots still here. > > Olivier Hi, there is a certainly an issue with (at least) old FileStore and snapshot chunks as they are getting completely unreferenced even for listsnaps example from above but are presented in omap and on filesystem after complete image and snapshot removal. Given the fact that the control flow has not been interrupted ever, e.g. snap deletion command was always been successful on exit either was image removal itself, what could be done for those poor data chunks? In fact this leakage on a long-scale runs like eight months in a given case could be quite problematic to handle, as orphans do consume almost as much data as the 'active' rest of the storage on selected OSDs. Since the chunks are still referenced in omap by some reason, they must not be deleted directly, so my question could be narrowed down to a possible existing workaround for this. Thanks! 3.1b0_head$ find . -type f -name '*64ba14d3dd381*' -mtime +90 ./DIR_0/DIR_B/DIR_1/DIR_1/rbd\udata.64ba14d3dd381.0000000000020dd7__23116_25FB11B0__3 ./DIR_0/DIR_B/DIR_1/DIR_1/rbd\udata.64ba14d3dd381.0000000000020dd7__241e9_25FB11B0__3 ./DIR_0/DIR_B/DIR_1/DIR_1/rbd\udata.64ba14d3dd381.0000000000020dd7__2507f_25FB11B0__3 ./DIR_0/DIR_B/DIR_1/DIR_1/rbd\udata.64ba14d3dd381.0000000000020dd7__25dfd_25FB11B0__3 find . -type f -name '*64ba14d3dd381*snap*' ./DIR_0/DIR_B/DIR_1/DIR_1/rbd\udata.64ba14d3dd381.0000000000020dd7__snapdir_25FB11B0__3 ./DIR_0/DIR_B/DIR_1/DIR_2/DIR_3/rbd\udata.64ba14d3dd381.0000000000010eb3__snapdir_2B8321B0__3 ./DIR_0/DIR_B/DIR_1/DIR_4/DIR_6/rbd\udata.64ba14d3dd381.000000000001c715__snapdir_F5D641B0__3 ./DIR_0/DIR_B/DIR_1/DIR_4/DIR_9/rbd\udata.64ba14d3dd381.000000000002b694__snapdir_CC4941B0__3 ./DIR_0/DIR_B/DIR_1/DIR_5/DIR_9/rbd\udata.64ba14d3dd381.000000000001b6f7__snapdir_08B951B0__3 -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-04-29 12:09 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <1400578369.11397.9.camel@localhost>
2014-05-21 10:17 ` Data still in OSD directories after removing Olivier Bonvalet
2014-05-21 15:20 ` Sage Weil
[not found] ` <alpine.DEB.2.00.1405210818560.1689-vIokxiIdD2AQNTJnQDzGJqxOck334EZe@public.gmane.org>
2014-05-21 22:03 ` Olivier Bonvalet
2014-05-22 1:20 ` Josh Durgin
[not found] ` <537D50EA.9020901-4GqslpFJ+cxBDgjK7y7TUQ@public.gmane.org>
2014-05-22 8:56 ` Olivier Bonvalet
2016-04-29 12:09 ` [ceph-users] " Andrey Korolyov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).