deleting objects from a pool

deleting objects from a pool

[Deneau, Tom]

I have benchmarking situations where I want to leave a pool around but
delete a lot of objects from the pool.  Is there any really fast way to do that?
I noticed rados rmpool is fast but I don't want to remove the pool.

I have been spawning multiple threads, each deleting a subset of the objects
(which I believe is what rados bench write does) but even that can be very slow.

-- Tom Deneau

Re: deleting objects from a pool

[Dałek, Piotr]

> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> owner@vger.kernel.org] On Behalf Of Deneau, Tom
> Sent: Wednesday, June 24, 2015 6:44 PM
> 
> I have benchmarking situations where I want to leave a pool around but
> delete a lot of objects from the pool.  Is there any really fast way to do that?
> I noticed rados rmpool is fast but I don't want to remove the pool.
> 
> I have been spawning multiple threads, each deleting a subset of the objects
> (which I believe is what rados bench write does) but even that can be very
> slow.

For now, apart from "rados -p <poolname> cleanup" (which doesn't purge the pool, but merely removes objects written during last benchmark run), the only option is by brute force:

for i in $(rados -p <poolname> ls); do (rados -p <poolname> rm $i &>/dev/null &); done;

There's no "purge pool" command in rados -- not yet, at least. I was thinking about one, but never really had time to implement one.

With best regards / Pozdrawiam
Piotr Dałek
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: deleting objects from a pool

[Deneau, Tom]

I've noticed that deleting objects from a basic k=2 m=1 erasure pool is much much slower than deleting a similar number of objects from a replicated size 3 pool (so the same number of files to be deleted).   It looked like the ec pool object deletion was almost 20x slower.  Is there a lot more work to be done to delete an ec pool object?

-- Tom



> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> owner@vger.kernel.org] On Behalf Of Dalek, Piotr
> Sent: Wednesday, June 24, 2015 11:56 AM
> To: ceph-devel
> Subject: Re: deleting objects from a pool
> 
> > -----Original Message-----
> > From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> > owner@vger.kernel.org] On Behalf Of Deneau, Tom
> > Sent: Wednesday, June 24, 2015 6:44 PM
> >
> > I have benchmarking situations where I want to leave a pool around but
> > delete a lot of objects from the pool.  Is there any really fast way to do
> that?
> > I noticed rados rmpool is fast but I don't want to remove the pool.
> >
> > I have been spawning multiple threads, each deleting a subset of the
> objects
> > (which I believe is what rados bench write does) but even that can be very
> > slow.
> 
> For now, apart from "rados -p <poolname> cleanup" (which doesn't purge the
> pool, but merely removes objects written during last benchmark run), the only
> option is by brute force:
> 
> for i in $(rados -p <poolname> ls); do (rados -p <poolname> rm $i &>/dev/null
> &); done;
> 
> There's no "purge pool" command in rados -- not yet, at least. I was thinking
> about one, but never really had time to implement one.
> 
> With best regards / Pozdrawiam
> Piotr Dałek
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: deleting objects from a pool

[Podoski, Igor]

Hi,

It appears, that cleanup can be used as a purge:

rados -p <poolname> cleanup  --prefix ""

Regards,
Igor.


-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Deneau, Tom
Sent: Wednesday, June 24, 2015 10:22 PM
To: Dałek, Piotr; ceph-devel
Subject: RE: deleting objects from a pool

I've noticed that deleting objects from a basic k=2 m=1 erasure pool is much much slower than deleting a similar number of objects from a replicated size 3 pool (so the same number of files to be deleted).   It looked like the ec pool object deletion was almost 20x slower.  Is there a lot more work to be done to delete an ec pool object?

-- Tom



> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel- 
> owner@vger.kernel.org] On Behalf Of Dalek, Piotr
> Sent: Wednesday, June 24, 2015 11:56 AM
> To: ceph-devel
> Subject: Re: deleting objects from a pool
> 
> > -----Original Message-----
> > From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel- 
> > owner@vger.kernel.org] On Behalf Of Deneau, Tom
> > Sent: Wednesday, June 24, 2015 6:44 PM
> >
> > I have benchmarking situations where I want to leave a pool around 
> > but delete a lot of objects from the pool.  Is there any really fast 
> > way to do
> that?
> > I noticed rados rmpool is fast but I don't want to remove the pool.
> >
> > I have been spawning multiple threads, each deleting a subset of the
> objects
> > (which I believe is what rados bench write does) but even that can 
> > be very slow.
> 
> For now, apart from "rados -p <poolname> cleanup" (which doesn't purge 
> the pool, but merely removes objects written during last benchmark 
> run), the only option is by brute force:
> 
> for i in $(rados -p <poolname> ls); do (rados -p <poolname> rm $i 
> &>/dev/null &); done;
> 
> There's no "purge pool" command in rados -- not yet, at least. I was 
> thinking about one, but never really had time to implement one.
> 
> With best regards / Pozdrawiam
> Piotr Dałek
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: deleting objects from a pool

[Deneau, Tom]

Igor --

Good command to know, but this is still very slow on an erasure pool.
For example, on my cluster it took 10 seconds with rados bench to write 10,000 40K size objects to an ecpool.
And it took almost 6 minutes to delete them using the command below.

-- Tom


> -----Original Message-----
> From: Podoski, Igor [mailto:Igor.Podoski@ts.fujitsu.com]
> Sent: Thursday, June 25, 2015 1:06 AM
> To: Deneau, Tom; Dałek, Piotr; ceph-devel
> Subject: RE: deleting objects from a pool
> 
> Hi,
> 
> It appears, that cleanup can be used as a purge:
> 
> rados -p <poolname> cleanup  --prefix ""
> 
> Regards,
> Igor.
> 
> 
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> owner@vger.kernel.org] On Behalf Of Deneau, Tom
> Sent: Wednesday, June 24, 2015 10:22 PM
> To: Dałek, Piotr; ceph-devel
> Subject: RE: deleting objects from a pool
> 
> I've noticed that deleting objects from a basic k=2 m=1 erasure pool is much
> much slower than deleting a similar number of objects from a replicated size
> 3 pool (so the same number of files to be deleted).   It looked like the ec
> pool object deletion was almost 20x slower.  Is there a lot more work to be
> done to delete an ec pool object?
> 
> -- Tom
> 
> 
> 
> > -----Original Message-----
> > From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> > owner@vger.kernel.org] On Behalf Of Dalek, Piotr
> > Sent: Wednesday, June 24, 2015 11:56 AM
> > To: ceph-devel
> > Subject: Re: deleting objects from a pool
> >
> > > -----Original Message-----
> > > From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> > > owner@vger.kernel.org] On Behalf Of Deneau, Tom
> > > Sent: Wednesday, June 24, 2015 6:44 PM
> > >
> > > I have benchmarking situations where I want to leave a pool around
> > > but delete a lot of objects from the pool.  Is there any really fast
> > > way to do
> > that?
> > > I noticed rados rmpool is fast but I don't want to remove the pool.
> > >
> > > I have been spawning multiple threads, each deleting a subset of the
> > objects
> > > (which I believe is what rados bench write does) but even that can
> > > be very slow.
> >
> > For now, apart from "rados -p <poolname> cleanup" (which doesn't purge
> > the pool, but merely removes objects written during last benchmark
> > run), the only option is by brute force:
> >
> > for i in $(rados -p <poolname> ls); do (rados -p <poolname> rm $i
> > &>/dev/null &); done;
> >
> > There's no "purge pool" command in rados -- not yet, at least. I was
> > thinking about one, but never really had time to implement one.
> >
> > With best regards / Pozdrawiam
> > Piotr Dałek
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More majordomo
> > info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the
> body of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: deleting objects from a pool

[David Zafman]

If you have rados bench data around, you'll need to run cleanup a second 
time because the first time the "benchmark_last_metadata" object
will be consulted to find what objects to remove.

Also, using cleanup this way will only remove objects from the default 
namespace unless a namespace is specified with the -N option.

rados -p <poolname> -N <namespace> cleanup --prefix ""

David

On 6/24/15 11:06 PM, Podoski, Igor wrote:
> Hi,
>
> It appears, that cleanup can be used as a purge:
>
> rados -p <poolname> cleanup  --prefix ""
>
> Regards,
> Igor.
>
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Deneau, Tom
> Sent: Wednesday, June 24, 2015 10:22 PM
> To: Dałek, Piotr; ceph-devel
> Subject: RE: deleting objects from a pool
>
> I've noticed that deleting objects from a basic k=2 m=1 erasure pool is much much slower than deleting a similar number of objects from a replicated size 3 pool (so the same number of files to be deleted).   It looked like the ec pool object deletion was almost 20x slower.  Is there a lot more work to be done to delete an ec pool object?
>
> -- Tom
>
>
>
>> -----Original Message-----
>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
>> owner@vger.kernel.org] On Behalf Of Dalek, Piotr
>> Sent: Wednesday, June 24, 2015 11:56 AM
>> To: ceph-devel
>> Subject: Re: deleting objects from a pool
>>
>>> -----Original Message-----
>>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
>>> owner@vger.kernel.org] On Behalf Of Deneau, Tom
>>> Sent: Wednesday, June 24, 2015 6:44 PM
>>>
>>> I have benchmarking situations where I want to leave a pool around
>>> but delete a lot of objects from the pool.  Is there any really fast
>>> way to do
>> that?
>>> I noticed rados rmpool is fast but I don't want to remove the pool.
>>>
>>> I have been spawning multiple threads, each deleting a subset of the
>> objects
>>> (which I believe is what rados bench write does) but even that can
>>> be very slow.
>> For now, apart from "rados -p <poolname> cleanup" (which doesn't purge
>> the pool, but merely removes objects written during last benchmark
>> run), the only option is by brute force:
>>
>> for i in $(rados -p <poolname> ls); do (rados -p <poolname> rm $i
>> &>/dev/null &); done;
>>
>> There's no "purge pool" command in rados -- not yet, at least. I was
>> thinking about one, but never really had time to implement one.
>>
>> With best regards / Pozdrawiam
>> Piotr Dałek
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in the body of a message to majordomo@vger.kernel.org More majordomo
>> info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: deleting objects from a pool

[Podoski, Igor]

Hi David,

You're right, now I see adding --run-name "" will clean all benchmark data from specified namespace, so you can run command only once.

rados -p <poolname> -N <namespace> cleanup --prefix "" --run-name ""

Regards,
Igor.


-----Original Message-----
From: David Zafman [mailto:dzafman@redhat.com] 
Sent: Friday, June 26, 2015 3:46 AM
To: Podoski, Igor; Deneau, Tom; Dałek, Piotr; ceph-devel
Subject: Re: deleting objects from a pool


If you have rados bench data around, you'll need to run cleanup a second time because the first time the "benchmark_last_metadata" object will be consulted to find what objects to remove.

Also, using cleanup this way will only remove objects from the default namespace unless a namespace is specified with the -N option.

rados -p <poolname> -N <namespace> cleanup --prefix ""

David

On 6/24/15 11:06 PM, Podoski, Igor wrote:
> Hi,
>
> It appears, that cleanup can be used as a purge:
>
> rados -p <poolname> cleanup  --prefix ""
>
> Regards,
> Igor.
>
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org 
> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Deneau, Tom
> Sent: Wednesday, June 24, 2015 10:22 PM
> To: Dałek, Piotr; ceph-devel
> Subject: RE: deleting objects from a pool
>
> I've noticed that deleting objects from a basic k=2 m=1 erasure pool is much much slower than deleting a similar number of objects from a replicated size 3 pool (so the same number of files to be deleted).   It looked like the ec pool object deletion was almost 20x slower.  Is there a lot more work to be done to delete an ec pool object?
>
> -- Tom
>
>
>
>> -----Original Message-----
>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel- 
>> owner@vger.kernel.org] On Behalf Of Dalek, Piotr
>> Sent: Wednesday, June 24, 2015 11:56 AM
>> To: ceph-devel
>> Subject: Re: deleting objects from a pool
>>
>>> -----Original Message-----
>>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel- 
>>> owner@vger.kernel.org] On Behalf Of Deneau, Tom
>>> Sent: Wednesday, June 24, 2015 6:44 PM
>>>
>>> I have benchmarking situations where I want to leave a pool around 
>>> but delete a lot of objects from the pool.  Is there any really fast 
>>> way to do
>> that?
>>> I noticed rados rmpool is fast but I don't want to remove the pool.
>>>
>>> I have been spawning multiple threads, each deleting a subset of the
>> objects
>>> (which I believe is what rados bench write does) but even that can 
>>> be very slow.
>> For now, apart from "rados -p <poolname> cleanup" (which doesn't 
>> purge the pool, but merely removes objects written during last 
>> benchmark run), the only option is by brute force:
>>
>> for i in $(rados -p <poolname> ls); do (rados -p <poolname> rm $i 
>> &>/dev/null &); done;
>>
>> There's no "purge pool" command in rados -- not yet, at least. I was 
>> thinking about one, but never really had time to implement one.
>>
>> With best regards / Pozdrawiam
>> Piotr Dałek
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in the body of a message to majordomo@vger.kernel.org More majordomo 
>> info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: deleting objects from a pool

[David Zafman]

This is a dangerous command because it can remove all your objects. At 
least it can only do one namespace at a time.  It was intended to 
cleanup rados bench runs, and is dangerous because it doesn't require 
extra hoops like rados rmpool does.

I'm tempted to disallow a usage this way with empty --prefix/--run-name 
arguments.

David

On 6/25/15 10:40 PM, Podoski, Igor wrote:
> Hi David,
>
> You're right, now I see adding --run-name "" will clean all benchmark data from specified namespace, so you can run command only once.
>
> rados -p <poolname> -N <namespace> cleanup --prefix "" --run-name ""
>
> Regards,
> Igor.
>
>
> -----Original Message-----
> From: David Zafman [mailto:dzafman@redhat.com]
> Sent: Friday, June 26, 2015 3:46 AM
> To: Podoski, Igor; Deneau, Tom; Dałek, Piotr; ceph-devel
> Subject: Re: deleting objects from a pool
>
>
> If you have rados bench data around, you'll need to run cleanup a second time because the first time the "benchmark_last_metadata" object will be consulted to find what objects to remove.
>
> Also, using cleanup this way will only remove objects from the default namespace unless a namespace is specified with the -N option.
>
> rados -p <poolname> -N <namespace> cleanup --prefix ""
>
> David
>
> On 6/24/15 11:06 PM, Podoski, Igor wrote:
>> Hi,
>>
>> It appears, that cleanup can be used as a purge:
>>
>> rados -p <poolname> cleanup  --prefix ""
>>
>> Regards,
>> Igor.
>>
>>
>> -----Original Message-----
>> From: ceph-devel-owner@vger.kernel.org
>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Deneau, Tom
>> Sent: Wednesday, June 24, 2015 10:22 PM
>> To: Dałek, Piotr; ceph-devel
>> Subject: RE: deleting objects from a pool
>>
>> I've noticed that deleting objects from a basic k=2 m=1 erasure pool is much much slower than deleting a similar number of objects from a replicated size 3 pool (so the same number of files to be deleted).   It looked like the ec pool object deletion was almost 20x slower.  Is there a lot more work to be done to delete an ec pool object?
>>
>> -- Tom
>>
>>
>>
>>> -----Original Message-----
>>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
>>> owner@vger.kernel.org] On Behalf Of Dalek, Piotr
>>> Sent: Wednesday, June 24, 2015 11:56 AM
>>> To: ceph-devel
>>> Subject: Re: deleting objects from a pool
>>>
>>>> -----Original Message-----
>>>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
>>>> owner@vger.kernel.org] On Behalf Of Deneau, Tom
>>>> Sent: Wednesday, June 24, 2015 6:44 PM
>>>>
>>>> I have benchmarking situations where I want to leave a pool around
>>>> but delete a lot of objects from the pool.  Is there any really fast
>>>> way to do
>>> that?
>>>> I noticed rados rmpool is fast but I don't want to remove the pool.
>>>>
>>>> I have been spawning multiple threads, each deleting a subset of the
>>> objects
>>>> (which I believe is what rados bench write does) but even that can
>>>> be very slow.
>>> For now, apart from "rados -p <poolname> cleanup" (which doesn't
>>> purge the pool, but merely removes objects written during last
>>> benchmark run), the only option is by brute force:
>>>
>>> for i in $(rados -p <poolname> ls); do (rados -p <poolname> rm $i
>>> &>/dev/null &); done;
>>>
>>> There's no "purge pool" command in rados -- not yet, at least. I was
>>> thinking about one, but never really had time to implement one.
>>>
>>> With best regards / Pozdrawiam
>>> Piotr Dałek
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>>> in the body of a message to majordomo@vger.kernel.org More majordomo
>>> info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in the body of a message to majordomo@vger.kernel.org More majordomo
>> info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in the body of a message to majordomo@vger.kernel.org More majordomo
>> info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: deleting objects from a pool

[Dałek, Piotr]

> -----Original Message-----
> From: David Zafman [mailto:dzafman@redhat.com]
> Sent: Friday, June 26, 2015 5:21 PM
> 
> This is a dangerous command because it can remove all your objects.

Well, that's the point of it, isn't it?

> At least
> it can only do one namespace at a time.  It was intended to cleanup rados
> bench runs,

And sometimes you need to clean up failed or more than one run (when "cleanup" doesn't work).

> and is dangerous because it doesn't require extra hoops like
> rados rmpool does.

Actually, it does, because you need to know that you can specify empty prefix or run name, and that's not that common knowledge.
 
> I'm tempted to disallow a usage this way with empty --prefix/--run-name
> arguments.

One solution would be to have "purge" command with syntax similar to "rmpool".
Disallowing any kind of "purge" without replacement would be a serious PITA for many users, including developers.

With best regards / Pozdrawiam
Piotr Dałek


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html