* optimizing buffers, encode/decode
@ 2014-10-29 17:17 Sage Weil
2014-10-29 17:23 ` Matt W. Benjamin
2014-11-03 7:52 ` Aanchal Agrawal
0 siblings, 2 replies; 6+ messages in thread
From: Sage Weil @ 2014-10-29 17:17 UTC (permalink / raw)
To: ceph-devel
We talked a bit about improving the performance encode/decode
yesterday at CDS:
http://pad.ceph.com/p/hammer-buffer_encoding
I think the main takeaways were:
1- We need some up to date profiling information to see
- how much of it is buffer-related functions (e.g., append)
- which data types are slowest or most frequently encoded (or otherwise
show up in the profile)
2- For now we should probably focus on the efficiency of the encode/decode
paths. Possibilities include
- making more things inline
- improving the past path
3- Matt and the linuxbox folks have been playing with some general
optimizations for the buffer::list class. These include combining
some of the function of ptr and raw so that, for the common
single-reference case, we chain the raw pointers together directly from
list using the boost intrusive list type, and fall back to the current
list -> ptr -> raw strategy when there are additional refs.
For #2, one simple thought would be to cache a pointer and remaining bytes
or end pointer into the append_buffer directly in list so that we avoid
the duplicate asserts and size checks in the common append (encode) path.
Then a
::encode(myu64, bl);
would inline into something pretty quick, like
remaining -= 8;
if (remainining < 0) { // take slow path
} else {
*ptr = myu64;
ptr += 8;
}
Not sure if an end pointer would let us cut out the 2 arithmetic ops or
not. Or if it even matters on modern pipelining processors.
Anyway, any gains we make here will pay dividends across the entire
code base. And any profiling people want to do will help guide
things...
Thanks!
sage
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: optimizing buffers, encode/decode
2014-10-29 17:17 optimizing buffers, encode/decode Sage Weil
@ 2014-10-29 17:23 ` Matt W. Benjamin
2014-10-29 17:33 ` Haomai Wang
2014-11-03 7:52 ` Aanchal Agrawal
1 sibling, 1 reply; 6+ messages in thread
From: Matt W. Benjamin @ 2014-10-29 17:23 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-devel
Hi Sage,
We're starting a round of work on improving encode/decode workload profiling, which we'll
share as soon as we have something informative.
Matt
----- "Sage Weil" <sage@inktank.com> wrote:
> We talked a bit about improving the performance encode/decode
> yesterday at CDS:
>
> http://pad.ceph.com/p/hammer-buffer_encoding
>
> I think the main takeaways were:
>
> 1- We need some up to date profiling information to see
>
> - how much of it is buffer-related functions (e.g., append)
> - which data types are slowest or most frequently encoded (or
> otherwise
> show up in the profile)
>
> 2- For now we should probably focus on the efficiency of the
> encode/decode
> paths. Possibilities include
>
> - making more things inline
> - improving the past path
>
> 3- Matt and the linuxbox folks have been playing with some general
> optimizations for the buffer::list class. These include combining
> some of the function of ptr and raw so that, for the common
> single-reference case, we chain the raw pointers together directly
> from
> list using the boost intrusive list type, and fall back to the current
>
> list -> ptr -> raw strategy when there are additional refs.
>
>
> For #2, one simple thought would be to cache a pointer and remaining
> bytes
> or end pointer into the append_buffer directly in list so that we
> avoid
> the duplicate asserts and size checks in the common append (encode)
> path.
> Then a
>
> ::encode(myu64, bl);
>
> would inline into something pretty quick, like
>
> remaining -= 8;
> if (remainining < 0) { // take slow path
>
> } else {
> *ptr = myu64;
> ptr += 8;
> }
>
> Not sure if an end pointer would let us cut out the 2 arithmetic ops
> or
> not. Or if it even matters on modern pipelining processors.
>
> Anyway, any gains we make here will pay dividends across the entire
> code base. And any profiling people want to do will help guide
> things...
>
> Thanks!
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI 48104
http://linuxbox.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: optimizing buffers, encode/decode
2014-10-29 17:23 ` Matt W. Benjamin
@ 2014-10-29 17:33 ` Haomai Wang
2014-10-29 17:38 ` Matt W. Benjamin
2014-10-29 17:43 ` Sage Weil
0 siblings, 2 replies; 6+ messages in thread
From: Haomai Wang @ 2014-10-29 17:33 UTC (permalink / raw)
To: Matt W. Benjamin; +Cc: Sage Weil, ceph-devel@vger.kernel.org
I'm interested in bufferlist's own encode/decode performance. But as I
performed until now, I think we need to consider change caller's
behavior to get better performance.
Combined (https://wiki.ceph.com/Planning/Blueprints/Hammer/Fixed_memory_layout_for_Message%2F%2FOp_passing)
with bp(https://wiki.ceph.com/Planning/Blueprints/Hammer/osd%3A_update_Transaction_encoding),
I'm going to seeking a way to reduce encode/decodes calls.
Mainly, we have three points:
1. Make ObjectStore::Transaction's metadata and data separated
2. No copy for op's data from Messenger to ObjectStore
3. Avoid encode/decode for ObjectStore::OP instead of fixed-size memory layout
We have a simple perf test results and a overview design ppt. Hope we
can have a talk at 8:00(CST).
On Thu, Oct 30, 2014 at 1:23 AM, Matt W. Benjamin <matt@linuxbox.com> wrote:
> Hi Sage,
>
> We're starting a round of work on improving encode/decode workload profiling, which we'll
> share as soon as we have something informative.
>
> Matt
>
> ----- "Sage Weil" <sage@inktank.com> wrote:
>
>> We talked a bit about improving the performance encode/decode
>> yesterday at CDS:
>>
>> http://pad.ceph.com/p/hammer-buffer_encoding
>>
>> I think the main takeaways were:
>>
>> 1- We need some up to date profiling information to see
>>
>> - how much of it is buffer-related functions (e.g., append)
>> - which data types are slowest or most frequently encoded (or
>> otherwise
>> show up in the profile)
>>
>> 2- For now we should probably focus on the efficiency of the
>> encode/decode
>> paths. Possibilities include
>>
>> - making more things inline
>> - improving the past path
>>
>> 3- Matt and the linuxbox folks have been playing with some general
>> optimizations for the buffer::list class. These include combining
>> some of the function of ptr and raw so that, for the common
>> single-reference case, we chain the raw pointers together directly
>> from
>> list using the boost intrusive list type, and fall back to the current
>>
>> list -> ptr -> raw strategy when there are additional refs.
>>
>>
>> For #2, one simple thought would be to cache a pointer and remaining
>> bytes
>> or end pointer into the append_buffer directly in list so that we
>> avoid
>> the duplicate asserts and size checks in the common append (encode)
>> path.
>> Then a
>>
>> ::encode(myu64, bl);
>>
>> would inline into something pretty quick, like
>>
>> remaining -= 8;
>> if (remainining < 0) { // take slow path
>>
>> } else {
>> *ptr = myu64;
>> ptr += 8;
>> }
>>
>> Not sure if an end pointer would let us cut out the 2 arithmetic ops
>> or
>> not. Or if it even matters on modern pipelining processors.
>>
>> Anyway, any gains we make here will pay dividends across the entire
>> code base. And any profiling people want to do will help guide
>> things...
>>
>> Thanks!
>> sage
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> --
> Matt Benjamin
> The Linux Box
> 206 South Fifth Ave. Suite 150
> Ann Arbor, MI 48104
>
> http://linuxbox.com
>
> tel. 734-761-4689
> fax. 734-769-8938
> cel. 734-216-5309
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Best Regards,
Wheat
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: optimizing buffers, encode/decode
2014-10-29 17:33 ` Haomai Wang
@ 2014-10-29 17:38 ` Matt W. Benjamin
2014-10-29 17:43 ` Sage Weil
1 sibling, 0 replies; 6+ messages in thread
From: Matt W. Benjamin @ 2014-10-29 17:38 UTC (permalink / raw)
To: Haomai Wang; +Cc: Sage Weil, ceph-devel
Hi Haomai,
I agree, and we'll join to hear your talk.
Thanks,
Matt
----- "Haomai Wang" <haomaiwang@gmail.com> wrote:
> I'm interested in bufferlist's own encode/decode performance. But as
> I
> performed until now, I think we need to consider change caller's
> behavior to get better performance.
>
> Combined
> (https://wiki.ceph.com/Planning/Blueprints/Hammer/Fixed_memory_layout_for_Message%2F%2FOp_passing)
> with
> bp(https://wiki.ceph.com/Planning/Blueprints/Hammer/osd%3A_update_Transaction_encoding),
> I'm going to seeking a way to reduce encode/decodes calls.
>
> Mainly, we have three points:
> 1. Make ObjectStore::Transaction's metadata and data separated
> 2. No copy for op's data from Messenger to ObjectStore
> 3. Avoid encode/decode for ObjectStore::OP instead of fixed-size
> memory layout
>
> We have a simple perf test results and a overview design ppt. Hope we
> can have a talk at 8:00(CST).
>
> On Thu, Oct 30, 2014 at 1:23 AM, Matt W. Benjamin <matt@linuxbox.com>
> wrote:
> > Hi Sage,
> >
> > We're starting a round of work on improving encode/decode workload
> profiling, which we'll
> > share as soon as we have something informative.
> >
> > Matt
> >
> > ----- "Sage Weil" <sage@inktank.com> wrote:
> >
> >> We talked a bit about improving the performance encode/decode
> >> yesterday at CDS:
> >>
> >> http://pad.ceph.com/p/hammer-buffer_encoding
> >>
> >> I think the main takeaways were:
> >>
> >> 1- We need some up to date profiling information to see
> >>
> >> - how much of it is buffer-related functions (e.g., append)
> >> - which data types are slowest or most frequently encoded (or
> >> otherwise
> >> show up in the profile)
> >>
> >> 2- For now we should probably focus on the efficiency of the
> >> encode/decode
> >> paths. Possibilities include
> >>
> >> - making more things inline
> >> - improving the past path
> >>
> >> 3- Matt and the linuxbox folks have been playing with some general
> >> optimizations for the buffer::list class. These include combining
> >> some of the function of ptr and raw so that, for the common
> >> single-reference case, we chain the raw pointers together directly
> >> from
> >> list using the boost intrusive list type, and fall back to the
> current
> >>
> >> list -> ptr -> raw strategy when there are additional refs.
> >>
> >>
> >> For #2, one simple thought would be to cache a pointer and
> remaining
> >> bytes
> >> or end pointer into the append_buffer directly in list so that we
> >> avoid
> >> the duplicate asserts and size checks in the common append
> (encode)
> >> path.
> >> Then a
> >>
> >> ::encode(myu64, bl);
> >>
> >> would inline into something pretty quick, like
> >>
> >> remaining -= 8;
> >> if (remainining < 0) { // take slow path
> >>
> >> } else {
> >> *ptr = myu64;
> >> ptr += 8;
> >> }
> >>
> >> Not sure if an end pointer would let us cut out the 2 arithmetic
> ops
> >> or
> >> not. Or if it even matters on modern pipelining processors.
> >>
> >> Anyway, any gains we make here will pay dividends across the
> entire
> >> code base. And any profiling people want to do will help guide
> >> things...
> >>
> >> Thanks!
> >> sage
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe
> ceph-devel"
> >> in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> > --
> > Matt Benjamin
> > The Linux Box
> > 206 South Fifth Ave. Suite 150
> > Ann Arbor, MI 48104
> >
> > http://linuxbox.com
> >
> > tel. 734-761-4689
> > fax. 734-769-8938
> > cel. 734-216-5309
> > --
> > To unsubscribe from this list: send the line "unsubscribe
> ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> Best Regards,
>
> Wheat
--
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI 48104
http://linuxbox.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: optimizing buffers, encode/decode
2014-10-29 17:33 ` Haomai Wang
2014-10-29 17:38 ` Matt W. Benjamin
@ 2014-10-29 17:43 ` Sage Weil
1 sibling, 0 replies; 6+ messages in thread
From: Sage Weil @ 2014-10-29 17:43 UTC (permalink / raw)
To: Haomai Wang; +Cc: Matt W. Benjamin, ceph-devel@vger.kernel.org
On Thu, 30 Oct 2014, Haomai Wang wrote:
> I'm interested in bufferlist's own encode/decode performance. But as I
> performed until now, I think we need to consider change caller's
> behavior to get better performance.
>
> Combined (https://wiki.ceph.com/Planning/Blueprints/Hammer/Fixed_memory_layout_for_Message%2F%2FOp_passing)
> with bp(https://wiki.ceph.com/Planning/Blueprints/Hammer/osd%3A_update_Transaction_encoding),
> I'm going to seeking a way to reduce encode/decodes calls.
>
> Mainly, we have three points:
> 1. Make ObjectStore::Transaction's metadata and data separated
> 2. No copy for op's data from Messenger to ObjectStore
> 3. Avoid encode/decode for ObjectStore::OP instead of fixed-size memory layout
>
> We have a simple perf test results and a overview design ppt. Hope we
> can have a talk at 8:00(CST).
Yep! For reference, the blueprint I wrote is here:
https://wiki.ceph.com/Planning/Blueprints/Hammer/osd%3A_update_Transaction_encoding
but I think we'll want to discuss Haomai's approach as well.
This is at 17:15 PDT, 01:15 CET, 08:15 CST for those that want to join.
https://wiki.ceph.com/Planning/CDS/Hammer_(Oct_2014)
sage
>
> On Thu, Oct 30, 2014 at 1:23 AM, Matt W. Benjamin <matt@linuxbox.com> wrote:
> > Hi Sage,
> >
> > We're starting a round of work on improving encode/decode workload profiling, which we'll
> > share as soon as we have something informative.
> >
> > Matt
> >
> > ----- "Sage Weil" <sage@inktank.com> wrote:
> >
> >> We talked a bit about improving the performance encode/decode
> >> yesterday at CDS:
> >>
> >> http://pad.ceph.com/p/hammer-buffer_encoding
> >>
> >> I think the main takeaways were:
> >>
> >> 1- We need some up to date profiling information to see
> >>
> >> - how much of it is buffer-related functions (e.g., append)
> >> - which data types are slowest or most frequently encoded (or
> >> otherwise
> >> show up in the profile)
> >>
> >> 2- For now we should probably focus on the efficiency of the
> >> encode/decode
> >> paths. Possibilities include
> >>
> >> - making more things inline
> >> - improving the past path
> >>
> >> 3- Matt and the linuxbox folks have been playing with some general
> >> optimizations for the buffer::list class. These include combining
> >> some of the function of ptr and raw so that, for the common
> >> single-reference case, we chain the raw pointers together directly
> >> from
> >> list using the boost intrusive list type, and fall back to the current
> >>
> >> list -> ptr -> raw strategy when there are additional refs.
> >>
> >>
> >> For #2, one simple thought would be to cache a pointer and remaining
> >> bytes
> >> or end pointer into the append_buffer directly in list so that we
> >> avoid
> >> the duplicate asserts and size checks in the common append (encode)
> >> path.
> >> Then a
> >>
> >> ::encode(myu64, bl);
> >>
> >> would inline into something pretty quick, like
> >>
> >> remaining -= 8;
> >> if (remainining < 0) { // take slow path
> >>
> >> } else {
> >> *ptr = myu64;
> >> ptr += 8;
> >> }
> >>
> >> Not sure if an end pointer would let us cut out the 2 arithmetic ops
> >> or
> >> not. Or if it even matters on modern pipelining processors.
> >>
> >> Anyway, any gains we make here will pay dividends across the entire
> >> code base. And any profiling people want to do will help guide
> >> things...
> >>
> >> Thanks!
> >> sage
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> >> in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> > --
> > Matt Benjamin
> > The Linux Box
> > 206 South Fifth Ave. Suite 150
> > Ann Arbor, MI 48104
> >
> > http://linuxbox.com
> >
> > tel. 734-761-4689
> > fax. 734-769-8938
> > cel. 734-216-5309
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> Best Regards,
>
> Wheat
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: optimizing buffers, encode/decode
2014-10-29 17:17 optimizing buffers, encode/decode Sage Weil
2014-10-29 17:23 ` Matt W. Benjamin
@ 2014-11-03 7:52 ` Aanchal Agrawal
1 sibling, 0 replies; 6+ messages in thread
From: Aanchal Agrawal @ 2014-11-03 7:52 UTC (permalink / raw)
To: ceph-devel@vger.kernel.org; +Cc: Sage Weil, Matt W. Benjamin, Haomai Wang
Hi All,
I have used 'Vtune profiler' to profile 4k random write for hotspots, locks-and-waits and hotspots by thread concurrency.
Thought of sharing the results with you guys if it can be of any help.
But attachments are nearly of 1MB and mail-server seems to reject it.
Can anyone tell me how can I share these vtune results?
Thanks,
Aanchal
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
Sent: Wednesday, October 29, 2014 10:47 PM
To: ceph-devel@vger.kernel.org
Subject: optimizing buffers, encode/decode
We talked a bit about improving the performance encode/decode yesterday at CDS:
http://pad.ceph.com/p/hammer-buffer_encoding
I think the main takeaways were:
1- We need some up to date profiling information to see
- how much of it is buffer-related functions (e.g., append)
- which data types are slowest or most frequently encoded (or otherwise
show up in the profile)
2- For now we should probably focus on the efficiency of the encode/decode paths. Possibilities include
- making more things inline
- improving the past path
3- Matt and the linuxbox folks have been playing with some general optimizations for the buffer::list class. These include combining some of the function of ptr and raw so that, for the common single-reference case, we chain the raw pointers together directly from list using the boost intrusive list type, and fall back to the current list -> ptr -> raw strategy when there are additional refs.
For #2, one simple thought would be to cache a pointer and remaining bytes or end pointer into the append_buffer directly in list so that we avoid the duplicate asserts and size checks in the common append (encode) path.
Then a
::encode(myu64, bl);
would inline into something pretty quick, like
remaining -= 8;
if (remainining < 0) { // take slow path
} else {
*ptr = myu64;
ptr += 8;
}
Not sure if an end pointer would let us cut out the 2 arithmetic ops or not. Or if it even matters on modern pipelining processors.
Anyway, any gains we make here will pay dividends across the entire code base. And any profiling people want to do will help guide things...
Thanks!
sage
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-11-03 7:53 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-10-29 17:17 optimizing buffers, encode/decode Sage Weil
2014-10-29 17:23 ` Matt W. Benjamin
2014-10-29 17:33 ` Haomai Wang
2014-10-29 17:38 ` Matt W. Benjamin
2014-10-29 17:43 ` Sage Weil
2014-11-03 7:52 ` Aanchal Agrawal
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.