journal alignment

journal alignment

[changtao381]

HI All,

Why it is needed an entry of journal t is aligned  by  CEPH_PAGE_MASK ?  For
it causes the data of journal write are amplified by 2X for small io

For example write io size 4096 bytes, it may write 8192 bytes

prepare_single_write 2 will write 98304 : seq 24 len 4324 -> 8192 (head 40
pre_pad 0 ebl 4324 post_pad 3788 tail 40) (ebl alignment -1)

Thanks!

Re: journal alignment

[Haomai Wang]

On Fri, Nov 20, 2015 at 4:33 PM, changtao381 <changtao381@163.com> wrote:
> HI All,
>
> Why it is needed an entry of journal t is aligned  by  CEPH_PAGE_MASK ?  For
> it causes the data of journal write are amplified by 2X for small io
>

linux aio/dio required this

> For example write io size 4096 bytes, it may write 8192 bytes
>
> prepare_single_write 2 will write 98304 : seq 24 len 4324 -> 8192 (head 40
> pre_pad 0 ebl 4324 post_pad 3788 tail 40) (ebl alignment -1)
>
> Thanks!
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Best Regards,

Wheat

RE: journal alignment

[Piotr.Dalek]

> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> owner@vger.kernel.org] On Behalf Of changtao381
> Sent: Friday, November 20, 2015 9:33 AM
> 
> HI All,
> 
> Why it is needed an entry of journal t is aligned  by  CEPH_PAGE_MASK ?  For
> it causes the data of journal write are amplified by 2X for small io
> 
> For example write io size 4096 bytes, it may write 8192 bytes
> 
> prepare_single_write 2 will write 98304 : seq 24 len 4324 -> 8192 (head 40
> pre_pad 0 ebl 4324 post_pad 3788 tail 40) (ebl alignment -1)
> 
> Thanks!

It is a direct-IO+async IO requirement (writing on page-size boundary which is incidentally also a multiply of sector size).

With best regards / Pozdrawiam
Piotr Dałek

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

答复: journal alignment

[changtao381]

Hi All,

Thanks for you apply!

If directioIO + async IO requirement that alignment, it shouldn't aligned by PAGE for each journal entry.
For it may write many entries of journal once time

So it should do alignment after [prepare_multi_write] rather than do it each in [prepare_single_write] ? Am I right?

-----邮件原件-----
发件人: Piotr.Dalek@ts.fujitsu.com [mailto:Piotr.Dalek@ts.fujitsu.com] 
发送时间: 2015年11月20日 16:42
收件人: changtao381; ceph-devel@vger.kernel.org
主题: RE: journal alignment

> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> owner@vger.kernel.org] On Behalf Of changtao381
> Sent: Friday, November 20, 2015 9:33 AM
> 
> HI All,
> 
> Why it is needed an entry of journal t is aligned  by  CEPH_PAGE_MASK ?  For
> it causes the data of journal write are amplified by 2X for small io
> 
> For example write io size 4096 bytes, it may write 8192 bytes
> 
> prepare_single_write 2 will write 98304 : seq 24 len 4324 -> 8192 (head 40
> pre_pad 0 ebl 4324 post_pad 3788 tail 40) (ebl alignment -1)
> 
> Thanks!

It is a direct-IO+async IO requirement (writing on page-size boundary which is incidentally also a multiply of sector size).

With best regards / Pozdrawiam
Piotr Dałek


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 答复: journal alignment

[池信泽]

Yes, You are right. But in ceph master branch, we have already
prepare_entry(add padding) before sumibt_entry. If you have good idea
base on this,
It would be great.

2015-11-20 17:12 GMT+08:00 changtao381 <changtao381@163.com>:
> Hi All,
>
> Thanks for you apply!
>
> If directioIO + async IO requirement that alignment, it shouldn't aligned by PAGE for each journal entry.
> For it may write many entries of journal once time
>
> So it should do alignment after [prepare_multi_write] rather than do it each in [prepare_single_write] ? Am I right?
>
> -----邮件原件-----
> 发件人: Piotr.Dalek@ts.fujitsu.com [mailto:Piotr.Dalek@ts.fujitsu.com]
> 发送时间: 2015年11月20日 16:42
> 收件人: changtao381; ceph-devel@vger.kernel.org
> 主题: RE: journal alignment
>
>> -----Original Message-----
>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
>> owner@vger.kernel.org] On Behalf Of changtao381
>> Sent: Friday, November 20, 2015 9:33 AM
>>
>> HI All,
>>
>> Why it is needed an entry of journal t is aligned  by  CEPH_PAGE_MASK ?  For
>> it causes the data of journal write are amplified by 2X for small io
>>
>> For example write io size 4096 bytes, it may write 8192 bytes
>>
>> prepare_single_write 2 will write 98304 : seq 24 len 4324 -> 8192 (head 40
>> pre_pad 0 ebl 4324 post_pad 3788 tail 40) (ebl alignment -1)
>>
>> Thanks!
>
> It is a direct-IO+async IO requirement (writing on page-size boundary which is incidentally also a multiply of sector size).
>
> With best regards / Pozdrawiam
> Piotr Dałek
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Regards,
xinze
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 答复: journal alignment

[池信泽]

Because we should keep the logic of journal write thread simple. It
performs better in pcie ssd. But I think the strategy you mentioned
above is good for hdd or sata ssd.

2015-11-20 17:16 GMT+08:00 池信泽 <xmdxcxz@gmail.com>:
> Yes, You are right. But in ceph master branch, we have already
> prepare_entry(add padding) before sumibt_entry. If you have good idea
> base on this,
> It would be great.
>
> 2015-11-20 17:12 GMT+08:00 changtao381 <changtao381@163.com>:
>> Hi All,
>>
>> Thanks for you apply!
>>
>> If directioIO + async IO requirement that alignment, it shouldn't aligned by PAGE for each journal entry.
>> For it may write many entries of journal once time
>>
>> So it should do alignment after [prepare_multi_write] rather than do it each in [prepare_single_write] ? Am I right?
>>
>> -----邮件原件-----
>> 发件人: Piotr.Dalek@ts.fujitsu.com [mailto:Piotr.Dalek@ts.fujitsu.com]
>> 发送时间: 2015年11月20日 16:42
>> 收件人: changtao381; ceph-devel@vger.kernel.org
>> 主题: RE: journal alignment
>>
>>> -----Original Message-----
>>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
>>> owner@vger.kernel.org] On Behalf Of changtao381
>>> Sent: Friday, November 20, 2015 9:33 AM
>>>
>>> HI All,
>>>
>>> Why it is needed an entry of journal t is aligned  by  CEPH_PAGE_MASK ?  For
>>> it causes the data of journal write are amplified by 2X for small io
>>>
>>> For example write io size 4096 bytes, it may write 8192 bytes
>>>
>>> prepare_single_write 2 will write 98304 : seq 24 len 4324 -> 8192 (head 40
>>> pre_pad 0 ebl 4324 post_pad 3788 tail 40) (ebl alignment -1)
>>>
>>> Thanks!
>>
>> It is a direct-IO+async IO requirement (writing on page-size boundary which is incidentally also a multiply of sector size).
>>
>> With best regards / Pozdrawiam
>> Piotr Dałek
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> Regards,
> xinze



-- 
Regards,
xinze
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 答复: journal alignment

[Haomai Wang]

On Fri, Nov 20, 2015 at 7:41 PM, Sage Weil <sage@newdream.net> wrote:
> On Fri, 20 Nov 2015, changtao381 wrote:
>> Hi All,
>>
>> Thanks for you apply!
>>
>> If directioIO + async IO requirement that alignment, it shouldn't aligned by PAGE for each journal entry.
>> For it may write many entries of journal once time
>
> We also want to avoid copying the data around in memory to change the
> alignment.  The messenger takes care to read data off the wire into
> buffers with the correct alignment so that we can later use them for
> direct-io.
>
> If you're worried about the small io case, I think this is just a matter
> of setting a threshold for small ios so that we don't bother with all of
> the padding when the memory copy isn't that expensive.  But... given that
> we have a header *and* footer in the journal format and almost all IOs are
> 4k multiples I think it'd save you a single 4k block at most.
>
> (Also, I thought we already did something like this, but perhaps not!)

Hmm, based on our recently test, the data from messenger is aligned.
But the encoded data(pglog, transaction) will make thing worse, like
PR(https://github.com/ceph/ceph/pull/6368) solved, we even will get 14
ptr in the bufferlist which passed into filejournal before. So it make
we rebuild each time within filejournal thread. Like this
PR(https://github.com/ceph/ceph/pull/6484), we try to make it rebuild
not in filejournal thread which is single.



>
> sage



-- 
Best Regards,

Wheat

Re: 答复: journal alignment

[Haomai Wang]

On Fri, Nov 20, 2015 at 9:08 PM, Sage Weil <sage@newdream.net> wrote:
> On Fri, 20 Nov 2015, Haomai Wang wrote:
>> On Fri, Nov 20, 2015 at 7:41 PM, Sage Weil <sage@newdream.net> wrote:
>> > On Fri, 20 Nov 2015, changtao381 wrote:
>> >> Hi All,
>> >>
>> >> Thanks for you apply!
>> >>
>> >> If directioIO + async IO requirement that alignment, it shouldn't aligned by PAGE for each journal entry.
>> >> For it may write many entries of journal once time
>> >
>> > We also want to avoid copying the data around in memory to change the
>> > alignment.  The messenger takes care to read data off the wire into
>> > buffers with the correct alignment so that we can later use them for
>> > direct-io.
>> >
>> > If you're worried about the small io case, I think this is just a matter
>> > of setting a threshold for small ios so that we don't bother with all of
>> > the padding when the memory copy isn't that expensive.  But... given that
>> > we have a header *and* footer in the journal format and almost all IOs are
>> > 4k multiples I think it'd save you a single 4k block at most.
>> >
>> > (Also, I thought we already did something like this, but perhaps not!)
>>
>> Hmm, based on our recently test, the data from messenger is aligned.
>> But the encoded data(pglog, transaction) will make thing worse, like
>> PR(https://github.com/ceph/ceph/pull/6368) solved, we even will get 14
>> ptr in the bufferlist which passed into filejournal before. So it make
>> we rebuild each time within filejournal thread. Like this
>> PR(https://github.com/ceph/ceph/pull/6484), we try to make it rebuild
>> not in filejournal thread which is single.
>
> buffer::list::rebuild_page_aligned() should only copy/rebuild ptrs that
> are unaligned, and leave aligned ones untouched.  It looks like the
> journal code is already doing this?

Yes or not, for example we have a bufferlist contains 2 ptrs, the
first is unaligned, the second is aligned. But the current impl will
ignore the second alignment fact. Look at the code:

  void buffer::list::rebuild_aligned_size_and_memory(unsigned align_size,
    unsigned align_memory)
  {
........
      list unaligned;
      unsigned offset = 0;
      do {
        /*cout << " segment " << (void*)p->c_str()
               << " offset " << ((unsigned long)p->c_str() & (align - 1))
               << " length " << p->length() << " " << (p->length() &
(align - 1))
               << " overall offset " << offset << " " << (offset & (align - 1))
      << " not ok" << std::endl;
        */
        offset += p->length();
        unaligned.push_back(*p);
        _buffers.erase(p++);
      } while (p != _buffers.end() &&
      (!p->is_aligned(align_memory) ||
       !p->is_n_align_sized(align_size) ||
       (offset % align_size)));
((((((((((((( it will check offset alignment, so won't continues after
meeting the first unalign ptr ))))))))))))))

      if (!(unaligned.is_contiguous() &&
unaligned._buffers.front().is_aligned(align_memory))) {
        ptr nb(buffer::create_aligned(unaligned._len, align_memory));
        unaligned.rebuild(nb);
        _memcopy_count += unaligned._len;
      }
      _buffers.insert(p, unaligned._buffers.front());
    }
  }


>
> sage



-- 
Best Regards,

Wheat

Re: 答复: journal alignment

[Evgeniy Firsov]

Do you see any performance difference between direct io and non direct io
mode?
If it¹s disabled, you don¹t need any buffer alignment.

On 11/20/15, 5:29 AM, "ceph-devel-owner@vger.kernel.org on behalf of
Haomai Wang" <ceph-devel-owner@vger.kernel.org on behalf of
haomaiwang@gmail.com> wrote:

>On Fri, Nov 20, 2015 at 9:08 PM, Sage Weil <sage@newdream.net> wrote:
>> On Fri, 20 Nov 2015, Haomai Wang wrote:
>>> On Fri, Nov 20, 2015 at 7:41 PM, Sage Weil <sage@newdream.net> wrote:
>>> > On Fri, 20 Nov 2015, changtao381 wrote:
>>> >> Hi All,
>>> >>
>>> >> Thanks for you apply!
>>> >>
>>> >> If directioIO + async IO requirement that alignment, it shouldn't
>>>aligned by PAGE for each journal entry.
>>> >> For it may write many entries of journal once time
>>> >
>>> > We also want to avoid copying the data around in memory to change the
>>> > alignment.  The messenger takes care to read data off the wire into
>>> > buffers with the correct alignment so that we can later use them for
>>> > direct-io.
>>> >
>>> > If you're worried about the small io case, I think this is just a
>>>matter
>>> > of setting a threshold for small ios so that we don't bother with
>>>all of
>>> > the padding when the memory copy isn't that expensive.  But... given
>>>that
>>> > we have a header *and* footer in the journal format and almost all
>>>IOs are
>>> > 4k multiples I think it'd save you a single 4k block at most.
>>> >
>>> > (Also, I thought we already did something like this, but perhaps
>>>not!)
>>>
>>> Hmm, based on our recently test, the data from messenger is aligned.
>>> But the encoded data(pglog, transaction) will make thing worse, like
>>> PR(https://github.com/ceph/ceph/pull/6368) solved, we even will get 14
>>> ptr in the bufferlist which passed into filejournal before. So it make
>>> we rebuild each time within filejournal thread. Like this
>>> PR(https://github.com/ceph/ceph/pull/6484), we try to make it rebuild
>>> not in filejournal thread which is single.
>>
>> buffer::list::rebuild_page_aligned() should only copy/rebuild ptrs that
>> are unaligned, and leave aligned ones untouched.  It looks like the
>> journal code is already doing this?
>
>Yes or not, for example we have a bufferlist contains 2 ptrs, the
>first is unaligned, the second is aligned. But the current impl will
>ignore the second alignment fact. Look at the code:
>
>  void buffer::list::rebuild_aligned_size_and_memory(unsigned align_size,
>    unsigned align_memory)
>  {
>........
>      list unaligned;
>      unsigned offset = 0;
>      do {
>        /*cout << " segment " << (void*)p->c_str()
>               << " offset " << ((unsigned long)p->c_str() & (align - 1))
>               << " length " << p->length() << " " << (p->length() &
>(align - 1))
>               << " overall offset " << offset << " " << (offset & (align
>- 1))
>      << " not ok" << std::endl;
>        */
>        offset += p->length();
>        unaligned.push_back(*p);
>        _buffers.erase(p++);
>      } while (p != _buffers.end() &&
>      (!p->is_aligned(align_memory) ||
>       !p->is_n_align_sized(align_size) ||
>       (offset % align_size)));
>((((((((((((( it will check offset alignment, so won't continues after
>meeting the first unalign ptr ))))))))))))))
>
>      if (!(unaligned.is_contiguous() &&
>unaligned._buffers.front().is_aligned(align_memory))) {
>        ptr nb(buffer::create_aligned(unaligned._len, align_memory));
>        unaligned.rebuild(nb);
>        _memcopy_count += unaligned._len;
>      }
>      _buffers.insert(p, unaligned._buffers.front());
>    }
>  }
>
>
>>
>> sage
>
>
>
>-- 
>Best Regards,
>
>Wheat
>--
>To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 答复: journal alignment

[Haomai Wang]

Actually, the broken zero-copy for data is discouraging. I think the
way we could do is rearrange the layout of journal entry, just like
message we make journal header and payload(extra encoding bytes
without data), then data and footer. I think it should be a easy adapt
change?

On Fri, Nov 20, 2015 at 9:29 PM, Haomai Wang <haomaiwang@gmail.com> wrote:
> On Fri, Nov 20, 2015 at 9:08 PM, Sage Weil <sage@newdream.net> wrote:
>> On Fri, 20 Nov 2015, Haomai Wang wrote:
>>> On Fri, Nov 20, 2015 at 7:41 PM, Sage Weil <sage@newdream.net> wrote:
>>> > On Fri, 20 Nov 2015, changtao381 wrote:
>>> >> Hi All,
>>> >>
>>> >> Thanks for you apply!
>>> >>
>>> >> If directioIO + async IO requirement that alignment, it shouldn't aligned by PAGE for each journal entry.
>>> >> For it may write many entries of journal once time
>>> >
>>> > We also want to avoid copying the data around in memory to change the
>>> > alignment.  The messenger takes care to read data off the wire into
>>> > buffers with the correct alignment so that we can later use them for
>>> > direct-io.
>>> >
>>> > If you're worried about the small io case, I think this is just a matter
>>> > of setting a threshold for small ios so that we don't bother with all of
>>> > the padding when the memory copy isn't that expensive.  But... given that
>>> > we have a header *and* footer in the journal format and almost all IOs are
>>> > 4k multiples I think it'd save you a single 4k block at most.
>>> >
>>> > (Also, I thought we already did something like this, but perhaps not!)
>>>
>>> Hmm, based on our recently test, the data from messenger is aligned.
>>> But the encoded data(pglog, transaction) will make thing worse, like
>>> PR(https://github.com/ceph/ceph/pull/6368) solved, we even will get 14
>>> ptr in the bufferlist which passed into filejournal before. So it make
>>> we rebuild each time within filejournal thread. Like this
>>> PR(https://github.com/ceph/ceph/pull/6484), we try to make it rebuild
>>> not in filejournal thread which is single.
>>
>> buffer::list::rebuild_page_aligned() should only copy/rebuild ptrs that
>> are unaligned, and leave aligned ones untouched.  It looks like the
>> journal code is already doing this?
>
> Yes or not, for example we have a bufferlist contains 2 ptrs, the
> first is unaligned, the second is aligned. But the current impl will
> ignore the second alignment fact. Look at the code:
>
>   void buffer::list::rebuild_aligned_size_and_memory(unsigned align_size,
>     unsigned align_memory)
>   {
> ........
>       list unaligned;
>       unsigned offset = 0;
>       do {
>         /*cout << " segment " << (void*)p->c_str()
>                << " offset " << ((unsigned long)p->c_str() & (align - 1))
>                << " length " << p->length() << " " << (p->length() &
> (align - 1))
>                << " overall offset " << offset << " " << (offset & (align - 1))
>       << " not ok" << std::endl;
>         */
>         offset += p->length();
>         unaligned.push_back(*p);
>         _buffers.erase(p++);
>       } while (p != _buffers.end() &&
>       (!p->is_aligned(align_memory) ||
>        !p->is_n_align_sized(align_size) ||
>        (offset % align_size)));
> ((((((((((((( it will check offset alignment, so won't continues after
> meeting the first unalign ptr ))))))))))))))
>
>       if (!(unaligned.is_contiguous() &&
> unaligned._buffers.front().is_aligned(align_memory))) {
>         ptr nb(buffer::create_aligned(unaligned._len, align_memory));
>         unaligned.rebuild(nb);
>         _memcopy_count += unaligned._len;
>       }
>       _buffers.insert(p, unaligned._buffers.front());
>     }
>   }
>
>
>>
>> sage
>
>
>
> --
> Best Regards,
>
> Wheat



-- 
Best Regards,

Wheat