stuff for v0.56.4

stuff for v0.56.4

[Sage Weil]

There have been a few important bug fixes that people are hitting or 
want:

- the journal replay bug (5d54ab154ca790688a6a1a2ad5f869c17a23980a)
- the - _ pool name vs cap parsing thing that is biting openstack users
- ceph-disk-* changes to support latest ceph-deploy

If there are other things that we want to include in 0.56.4, lets get them 
into the bobtial branch sooner rather than later.

Possible items:

- pg log trimming (probably a conservative subset) to avoid memory bloat
- omap scrub?
- pg temp collection removal?
- buffer::cmp fix from loic?

Are there other items that we are missing?

sage

Re: stuff for v0.56.4

[Wido den Hollander]

On 03/06/2013 12:10 AM, Sage Weil wrote:
> There have been a few important bug fixes that people are hitting or
> want:
>
> - the journal replay bug (5d54ab154ca790688a6a1a2ad5f869c17a23980a)
> - the - _ pool name vs cap parsing thing that is biting openstack users
> - ceph-disk-* changes to support latest ceph-deploy
>
> If there are other things that we want to include in 0.56.4, lets get them
> into the bobtial branch sooner rather than later.
>
> Possible items:
>
> - pg log trimming (probably a conservative subset) to avoid memory bloat
> - omap scrub?
> - pg temp collection removal?
> - buffer::cmp fix from loic?
>
> Are there other items that we are missing?
>

I'm still seeing #3816 on my systems. The fix in wip-3816 did not 
resolve it for me.

Wido

> sage
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


-- 
Wido den Hollander
42on B.V.

Phone: +31 (0)20 700 9902
Skype: contact42on

Re: stuff for v0.56.4

[Travis Rhoden]

As long as the fix for...

osdc/ObjectCacher.cc: In function 'void
ObjectCacher::bh_write_commit(int64_t, sobject_t, loff_t, uint64_t,
tid_t, int)' thread 7fd316a50700 time 2013-03-07 15:03:21.641190
osdc/ObjectCacher.cc: 834: FAILED assert(ob->last_commit_tid < tid)

...s in there (which you already put on the bobtail branch, I believe)
I will be happy.  This particular bug crashes several VMs a day for
me.

 - Travis

On Wed, Mar 6, 2013 at 3:37 AM, Wido den Hollander <wido@42on.com> wrote:
> On 03/06/2013 12:10 AM, Sage Weil wrote:
>>
>> There have been a few important bug fixes that people are hitting or
>> want:
>>
>> - the journal replay bug (5d54ab154ca790688a6a1a2ad5f869c17a23980a)
>> - the - _ pool name vs cap parsing thing that is biting openstack users
>> - ceph-disk-* changes to support latest ceph-deploy
>>
>> If there are other things that we want to include in 0.56.4, lets get them
>> into the bobtial branch sooner rather than later.
>>
>> Possible items:
>>
>> - pg log trimming (probably a conservative subset) to avoid memory bloat
>> - omap scrub?
>> - pg temp collection removal?
>> - buffer::cmp fix from loic?
>>
>> Are there other items that we are missing?
>>
>
> I'm still seeing #3816 on my systems. The fix in wip-3816 did not resolve it
> for me.
>
> Wido
>
>
>> sage
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>
> --
> Wido den Hollander
> 42on B.V.
>
> Phone: +31 (0)20 700 9902
> Skype: contact42on
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: stuff for v0.56.4

[Yehuda Sadeh]

On Tue, Mar 5, 2013 at 3:10 PM, Sage Weil <sage@inktank.com> wrote:
> There have been a few important bug fixes that people are hitting or
> want:
>
> - the journal replay bug (5d54ab154ca790688a6a1a2ad5f869c17a23980a)
> - the - _ pool name vs cap parsing thing that is biting openstack users
> - ceph-disk-* changes to support latest ceph-deploy
>
> If there are other things that we want to include in 0.56.4, lets get them
> into the bobtial branch sooner rather than later.
>
> Possible items:
>
> - pg log trimming (probably a conservative subset) to avoid memory bloat
> - omap scrub?
> - pg temp collection removal?
> - buffer::cmp fix from loic?
>
> Are there other items that we are missing?
>

wip-4247-bobtail (pending review)

Yehuda

Re: stuff for v0.56.4

[Bryan K. Wright]

sage@inktank.com said:
> - pg log trimming (probably a conservative subset) to avoid memory bloat 

Anything that reduces the size of OSD processes would be appreciated.

					Bryan
-- 
========================================================================
Bryan Wright              |"If you take cranberries and stew them like 
Physics Department        | applesauce, they taste much more like prunes
University of Virginia    | than rhubarb does."  --  Groucho 
Charlottesville, VA  22901|			
(434) 924-7218            |         bryan@virginia.edu
========================================================================

Re: stuff for v0.56.4

[Sage Weil]

On Thu, 7 Mar 2013, Bryan K. Wright wrote:
> 
> sage@inktank.com said:
> > - pg log trimming (probably a conservative subset) to avoid memory bloat 
> 
> Anything that reduces the size of OSD processes would be appreciated.

You can probably do this with just

 log max recent = 1000

By default it's keeping 100k lines of logs in memory, which can eat a lot 
of ram (but is great when debugging issues).

s


> 
> 					Bryan
> -- 
> ========================================================================
> Bryan Wright              |"If you take cranberries and stew them like 
> Physics Department        | applesauce, they taste much more like prunes
> University of Virginia    | than rhubarb does."  --  Groucho 
> Charlottesville, VA  22901|			
> (434) 924-7218            |         bryan@virginia.edu
> ========================================================================
> 
> 
> 

Estimating OSD memory requirements (was Re: stuff for v0.56.4)

[Bryan K. Wright]

sage@inktank.com said:
> On Thu, 7 Mar 2013, Bryan K. Wright wrote:
> 
> sage@inktank.com said:
> > - pg log trimming (probably a conservative subset) to avoid memory bloat 
> 
> Anything that reduces the size of OSD processes would be appreciated.
> You can probably do this with just
>  log max recent = 1000
> By default it's keeping 100k lines of logs in memory, which can eat a lot  of
> ram (but is great when debugging issues).

	Thanks for the tip about "log max recent".  I've made this 
change, but it doesn't seem to significantly reduce the size of the 
OSD processes.

	In general, are there some rules of thumb for estimated the
memory requirements for OSDs?  I see processes blow up to 8gb of 
resident memory sometimes.  If I need to allow for that much memory
per OSD process, I may have to just walk away from ceph.

	Does the memory usage scale with the size of the disks?
I've been trying to run 12 OSDs with 12 2TB disks on a single box.
Would I be better off (memory-usage-wise) if I RAIDed the disks
together and used a single OSD process?

	Thanks for any advice.

					Bryan


-- 
========================================================================
Bryan Wright              |"If you take cranberries and stew them like 
Physics Department        | applesauce, they taste much more like prunes
University of Virginia    | than rhubarb does."  --  Groucho 
Charlottesville, VA  22901|			
(434) 924-7218            |         bryan@virginia.edu
========================================================================

Re: Estimating OSD memory requirements (was Re: stuff for v0.56.4)

[Greg Farnum]

On Monday, March 11, 2013 at 8:10 AM, Bryan K. Wright wrote:

> sage@inktank.com said:
> > On Thu, 7 Mar 2013, Bryan K. Wright wrote:
> > 
> > sage@inktank.com said:
> > > - pg log trimming (probably a conservative subset) to avoid memory bloat 
> > 
> > 
> > 
> > Anything that reduces the size of OSD processes would be appreciated.
> > You can probably do this with just
> > log max recent = 1000
> > By default it's keeping 100k lines of logs in memory, which can eat a lot of
> > ram (but is great when debugging issues).
> 
> 
> 
> Thanks for the tip about "log max recent". I've made this 
> change, but it doesn't seem to significantly reduce the size of the 
> OSD processes.
> 
> In general, are there some rules of thumb for estimated the
> memory requirements for OSDs? I see processes blow up to 8gb of 
> resident memory sometimes. If I need to allow for that much memory
> per OSD process, I may have to just walk away from ceph.
> 
> Does the memory usage scale with the size of the disks?
> I've been trying to run 12 OSDs with 12 2TB disks on a single box.
> Would I be better off (memory-usage-wise) if I RAIDed the disks
> together and used a single OSD process?
> 


Memory use depends on several things, but the most important are how many PGs the daemon is hosting, and whether it's undergoing recovery of some kind. (Absolute disk size is not involved.) If you're getting up to 8GB per, it sounds as if you may have a bit too many PGs.
You could try RAIDing some of your drives together instead, yes -- memory & CPU utilization is one of the trade offs there, balanced against larger discrete failure units and the loss of space or reliability (depending on the RAID chosen).
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com

Re: Estimating OSD memory requirements (was Re: stuff for v0.56.4)

[Jim Schutt]

Hi Bryan,

On 03/11/2013 09:10 AM, Bryan K. Wright wrote:
> sage@inktank.com said:
>> On Thu, 7 Mar 2013, Bryan K. Wright wrote:
>>
>> sage@inktank.com said:
>>> - pg log trimming (probably a conservative subset) to avoid memory bloat 
>>
>> Anything that reduces the size of OSD processes would be appreciated.
>> You can probably do this with just
>>  log max recent = 1000
>> By default it's keeping 100k lines of logs in memory, which can eat a lot  of
>> ram (but is great when debugging issues).
> 
> 	Thanks for the tip about "log max recent".  I've made this 
> change, but it doesn't seem to significantly reduce the size of the 
> OSD processes.
> 
> 	In general, are there some rules of thumb for estimated the
> memory requirements for OSDs?  I see processes blow up to 8gb of 
> resident memory sometimes.  If I need to allow for that much memory
> per OSD process, I may have to just walk away from ceph.
> 
> 	Does the memory usage scale with the size of the disks?
> I've been trying to run 12 OSDs with 12 2TB disks on a single box.
> Would I be better off (memory-usage-wise) if I RAIDed the disks
> together and used a single OSD process?
> 
> 	Thanks for any advice.

You might also try tuning "osd client message size cap"; its
current default is 500 MiB.

During the periods your aggregate applied write load is higher
than your OSD aggregate write bandwidth (taking into account
replicas), you'll be buffering up this amount of client data.

Since it only applies to incoming client messages, to figure
total memory use I believe you need to multiply that by the
number of replicas you're using.

FWIW, for sequential writes from lots of clients, I can
maintain full write bandwidth with "osd client message size
cap" tuned to 60 MiB.

-- Jim

> 
> 					Bryan
> 
>