how to free space from rados bench comman?

how to free space from rados bench comman?

[Stefan Priebe - Profihost AG]

Hi,

every rados bench write uses disk space and my space fills up. How to
free this space again?

Used command?
rados -p data bench 60 write -t 16

Stefan

Re: how to free space from rados bench comman?

[Wido den Hollander]

Hi,

On 24-05-12 09:01, Stefan Priebe - Profihost AG wrote:
> Hi,
>
> every rados bench write uses disk space and my space fills up. How to
> free this space again?
>
> Used command?
> rados -p data bench 60 write -t 16

What does this show:

$ rados -p data ls|wc -l

If that shows something greater than 0 it means you still have objects 
in that pool which are using up space.

Try removing those objects manually. Be cautious not to remove any other 
objects!

To be safe I'd recommend running benchmark commands in a separate pool.

Also note that when you remove objects it will take some time before the 
OSD's have removed them and you see the usage go down with "ceph -s".

Wido

>
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: how to free space from rados bench comman?

[Stefan Priebe - Profihost AG]

Am 24.05.2012 09:28, schrieb Wido den Hollander:
> Hi,
> 
> On 24-05-12 09:01, Stefan Priebe - Profihost AG wrote:
>> Hi,
>>
>> every rados bench write uses disk space and my space fills up. How to
>> free this space again?
>>
>> Used command?
>> rados -p data bench 60 write -t 16
> 
> What does this show:
> 
> $ rados -p data ls|wc -l

~# rados -p data ls|wc -l
46631

I do not use the data pool so it is seperate ;-) i only use the rbd pool
for block devices.

So i will free the space with:
for i in `rados -p data ls`; do echo $i; rados -p data rm $i; done

Thanks!

Stefan

Re: how to free space from rados bench comman?

[Wido den Hollander]

On 24-05-12 09:38, Stefan Priebe - Profihost AG wrote:
> Am 24.05.2012 09:28, schrieb Wido den Hollander:
>> Hi,
>>
>> On 24-05-12 09:01, Stefan Priebe - Profihost AG wrote:
>>> Hi,
>>>
>>> every rados bench write uses disk space and my space fills up. How to
>>> free this space again?
>>>
>>> Used command?
>>> rados -p data bench 60 write -t 16
>>
>> What does this show:
>>
>> $ rados -p data ls|wc -l
>
> ~# rados -p data ls|wc -l
> 46631

That is weird, I thought the bench tool cleaned up it's "mess".

Imho it should cleanup after it's done, but there might be a reason why 
it's not. Did you abort the benchmark or did you let it do the whole run?

>
> I do not use the data pool so it is seperate ;-) i only use the rbd pool
> for block devices.
>
> So i will free the space with:
> for i in `rados -p data ls`; do echo $i; rados -p data rm $i; done

rados -p data ls|xargs -n 1 rados -p data rm

I love shorter commands ;)

Wido

>
> Thanks!
>
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: how to free space from rados bench comman?

[Stefan Priebe - Profihost AG]

Am 24.05.2012 10:22, schrieb Wido den Hollander:
> On 24-05-12 09:38, Stefan Priebe - Profihost AG wrote:
>>
>> ~# rados -p data ls|wc -l
>> 46631
> 
> That is weird, I thought the bench tool cleaned up it's "mess".
> 
> Imho it should cleanup after it's done, but there might be a reason why
> it's not. Did you abort the benchmark or did you let it do the whole run?
No it doesn't BUG?

~# rados -p data ls
~#
~# rados -p data bench 20 write -t 16
...
~# rados -p data ls| wc -l
589

>> I do not use the data pool so it is seperate ;-) i only use the rbd pool
>> for block devices.
>>
>> So i will free the space with:
>> for i in `rados -p data ls`; do echo $i; rados -p data rm $i; done
> 
> rados -p data ls|xargs -n 1 rados -p data rm
> 
> I love shorter commands ;)
me too i just tried it without -n and hoped that this works but rados
didn't support more than 1 file per command and i didn't remembered -n1 ;)

Stefan

Re: how to free space from rados bench comman?

[Greg Farnum]

On Thursday, May 24, 2012 at 1:51 AM, Stefan Priebe - Profihost AG wrote:
> Am 24.05.2012 10:22, schrieb Wido den Hollander:
> > On 24-05-12 09:38, Stefan Priebe - Profihost AG wrote:
> > > 
> > > ~# rados -p data ls|wc -l
> > > 46631
> > 
> > 
> > 
> > That is weird, I thought the bench tool cleaned up it's "mess".
> > 
> > Imho it should cleanup after it's done, but there might be a reason why
> > it's not. Did you abort the benchmark or did you let it do the whole run?
> 
> 
> No it doesn't BUG?
It doesn't because you might want to leave around the data for read benchmarking (or so that your cluster is full of data).
There should probably be an option to clean up bench data, though! I've created a bug: http://tracker.newdream.net/issues/2477
 
> 
> ~# rados -p data ls
> ~#
> ~# rados -p data bench 20 write -t 16
> ...
> ~# rados -p data ls| wc -l
> 589
> 
> > > I do not use the data pool so it is seperate ;-) i only use the rbd pool
> > > for block devices.
> > > 
> > > So i will free the space with:
> > > for i in `rados -p data ls`; do echo $i; rados -p data rm $i; done
> > 
> > 
> > 
> > rados -p data ls|xargs -n 1 rados -p data rm
> > 
> > I love shorter commands ;)
> me too i just tried it without -n and hoped that this works but rados
> didn't support more than 1 file per command and i didn't remembered -n1 ;)
> 
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org (mailto:majordomo@vger.kernel.org)
> More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: how to free space from rados bench comman?

[Josh Durgin]

On 05/24/2012 10:55 AM, Greg Farnum wrote:
> On Thursday, May 24, 2012 at 1:51 AM, Stefan Priebe - Profihost AG wrote:
>> Am 24.05.2012 10:22, schrieb Wido den Hollander:
>>> On 24-05-12 09:38, Stefan Priebe - Profihost AG wrote:
>>>>
>>>> ~# rados -p data ls|wc -l
>>>> 46631
>>>
>>>
>>>
>>> That is weird, I thought the bench tool cleaned up it's "mess".
>>>
>>> Imho it should cleanup after it's done, but there might be a reason why
>>> it's not. Did you abort the benchmark or did you let it do the whole run?
>>
>>
>> No it doesn't BUG?
> It doesn't because you might want to leave around the data for read benchmarking (or so that your cluster is full of data).
> There should probably be an option to clean up bench data, though! I've created a bug: http://tracker.newdream.net/issues/2477

Why not have the read benchmark write data itself, and then benchmark
reading? Then both read and write benchmarks can clean up after
themselves.

It's a bit odd to have the read benchmark depend on you running a write
benchmark first.

Josh

Re: how to free space from rados bench comman?

[Greg Farnum]

On Thursday, May 24, 2012 at 11:05 AM, Josh Durgin wrote:
> Why not have the read benchmark write data itself, and then benchmark
> reading? Then both read and write benchmarks can clean up after
> themselves.
> 
> It's a bit odd to have the read benchmark depend on you running a write
> benchmark first.
> 
> Josh 
We've talked about that and decided we didn't like it. I think it was about being able to repeat large read benchmarks without having to wait for all the data to get written out first, and also (although this was never implemented) being able to implement random read benchmarks and things in ways that allowed you to make the cache cold first.
Which is not to say that changing it is a bad idea; I could be talked into that or somebody else could do it. :)
-Greg