[GSoC2016] BlueStore SMR Support

[GSoC2016] BlueStore SMR Support

[Shehbaz Jaffer]

Hi All,

I am a 1st Year PhD student at University of Toronto. I am interested
in working on the SMR friendly allocator for Bluestore file system. I
am new to Ceph. But I am familiar with FS development in user space, and
SMR drives. I am currently going through Ceph codebase.
It looks like the allocator is very basic, hence the current name of
the block allocator:
os/bluestore/StupidAllocator.cc :-)

Few initial questions:

1) Will the project involve Drive managed SMR drives or Host Managed SMR drives?

2) Will scope of project be to build an allocator on a simulator for
SMR Drives? Or will I get actual HDDs to work with? In case of
simulators, will the project involve building a simulator on disksim?

3) Ideally, the benchmark should be to compare StupidAllocator scheme
with a SmartAllocator scheme in terms of the following performance
metrics:
a) speed
b) correctness
c) fragmentation
d) garbage collection efficiency

are there any other metrics that we would like to measure the allocator for?

Thanks and I look forward to working on this project. Please feel free
to send out your comments/suggestions.

Regards,

-- 
Shehbaz Jaffer
First Year Graduate Student
Sir Edward S Rogers Sr Department of Electrical and Computer Engineering
University of Toronto

Re: [GSoC2016] BlueStore SMR Support

[Sage Weil]

Hi Shehbaz,

Sorry for the slow reply!  I'm just catching up on GSoC queries.

On Fri, 4 Mar 2016, Shehbaz Jaffer wrote:
> Hi All,
> 
> I am a 1st Year PhD student at University of Toronto. I am interested
> in working on the SMR friendly allocator for Bluestore file system. I
> am new to Ceph. But I am familiar with FS development in user space, and
> SMR drives. I am currently going through Ceph codebase.
> It looks like the allocator is very basic, hence the current name of
> the block allocator:
> os/bluestore/StupidAllocator.cc :-)

:)

> Few initial questions:
> 
> 1) Will the project involve Drive managed SMR drives or Host Managed SMR 
> drives?

The goal is to make an allocator that will work with host managed SMR.  
Presumably that will work just as well on host-aware SMR.

A drive-managed SMR disk that doesn't tell us the zone layout is a bit of 
a lost cause, I think.

> 2) Will scope of project be to build an allocator on a simulator for
> SMR Drives? Or will I get actual HDDs to work with? In case of
> simulators, will the project involve building a simulator on disksim?

I seem to remember hearing something about a dm module that simulated SMR, 
but I'm not sure.  We may also be able to get some SMR disks to play 
with--I'll reach out to our drive vendor friends.

> 3) Ideally, the benchmark should be to compare StupidAllocator scheme
> with a SmartAllocator scheme in terms of the following performance
> metrics:
> a) speed
> b) correctness
> c) fragmentation
> d) garbage collection efficiency
> 
> are there any other metrics that we would like to measure the allocator for?

That sounds about right, (b) being the first and most important step.

Our only big idea here so far is that we probably want to put writes for 
each PG in a distinct zone, as I suspect PG is thing we have that is most 
closely correlated to object lifetime (because when the cluster does a 
rebalance or repair, and entire PG's worth of objects will typically get 
moved around).  That might mean a lot of open zones, though, and mean lots 
of seeks on writes if new objects are randomly spread across PGs (as they 
generally will be).  So we should probably explore other strategies as 
well.

> Thanks and I look forward to working on this project. Please feel free
> to send out your comments/suggestions.

Great!
sage

Re: [GSoC2016] BlueStore SMR Support

[Giridhar Yasa]

On Fri, Mar 11, 2016 at 8:20 PM, Sage Weil <sweil@redhat.com> wrote:
>
> On Fri, 4 Mar 2016, Shehbaz Jaffer wrote:
>
> > 2) Will scope of project be to build an allocator on a simulator for
> > SMR Drives? Or will I get actual HDDs to work with? In case of
> > simulators, will the project involve building a simulator on disksim?
>
> I seem to remember hearing something about a dm module that simulated SMR,
> but I'm not sure.  We may also be able to get some SMR disks to play
> with--I'll reach out to our drive vendor friends.

Would that be https://github.com/hgst/libzbc from HGST?

Giridhar

--
Giridhar Yasa | Flipkart Engineering | http://www.flipkart.com/

Re: [GSoC2016] BlueStore SMR Support

[Sage Weil]

On Sun, 13 Mar 2016, Giridhar Yasa wrote:
> On Fri, Mar 11, 2016 at 8:20 PM, Sage Weil <sweil@redhat.com> wrote:
> >
> > On Fri, 4 Mar 2016, Shehbaz Jaffer wrote:
> >
> > > 2) Will scope of project be to build an allocator on a simulator for
> > > SMR Drives? Or will I get actual HDDs to work with? In case of
> > > simulators, will the project involve building a simulator on disksim?
> >
> > I seem to remember hearing something about a dm module that simulated SMR,
> > but I'm not sure.  We may also be able to get some SMR disks to play
> > with--I'll reach out to our drive vendor friends.
> 
> Would that be https://github.com/hgst/libzbc from HGST?

Yes.

I'm not sure about a disk simulator or actual drive availability, 
though.

sage

Re: [GSoC2016] BlueStore SMR Support

[Giridhar Yasa]

On Mon, Mar 14, 2016 at 5:47 PM, Sage Weil <sweil@redhat.com> wrote:
> On Sun, 13 Mar 2016, Giridhar Yasa wrote:
>> On Fri, Mar 11, 2016 at 8:20 PM, Sage Weil <sweil@redhat.com> wrote:
>> >
>> > On Fri, 4 Mar 2016, Shehbaz Jaffer wrote:
>> >
>> > > 2) Will scope of project be to build an allocator on a simulator for
>> > > SMR Drives? Or will I get actual HDDs to work with? In case of
>> > > simulators, will the project involve building a simulator on disksim?
>> >
>> > I seem to remember hearing something about a dm module that simulated SMR,
>> > but I'm not sure.  We may also be able to get some SMR disks to play
>> > with--I'll reach out to our drive vendor friends.
>>
>> Would that be https://github.com/hgst/libzbc from HGST?
>
> Yes.
>
> I'm not sure about a disk simulator or actual drive availability,
> though.

libzbc has an emulation mode that can use files or block devices to
emulate SMR disk zones.

Giridhar

--
Giridhar Yasa | Flipkart Engineering | http://www.flipkart.com/

Re: [GSoC2016] BlueStore SMR Support

[Sage Weil]

On Mon, 14 Mar 2016, Giridhar Yasa wrote:
> On Mon, Mar 14, 2016 at 5:47 PM, Sage Weil <sweil@redhat.com> wrote:
> > On Sun, 13 Mar 2016, Giridhar Yasa wrote:
> >> On Fri, Mar 11, 2016 at 8:20 PM, Sage Weil <sweil@redhat.com> wrote:
> >> >
> >> > On Fri, 4 Mar 2016, Shehbaz Jaffer wrote:
> >> >
> >> > > 2) Will scope of project be to build an allocator on a simulator for
> >> > > SMR Drives? Or will I get actual HDDs to work with? In case of
> >> > > simulators, will the project involve building a simulator on disksim?
> >> >
> >> > I seem to remember hearing something about a dm module that simulated SMR,
> >> > but I'm not sure.  We may also be able to get some SMR disks to play
> >> > with--I'll reach out to our drive vendor friends.
> >>
> >> Would that be https://github.com/hgst/libzbc from HGST?
> >
> > Yes.
> >
> > I'm not sure about a disk simulator or actual drive availability,
> > though.
> 
> libzbc has an emulation mode that can use files or block devices to
> emulate SMR disk zones.

Oh, perfect!

sage

Re: [GSoC2016] BlueStore SMR Support

[Ric Wheeler]

On 03/14/2016 08:33 AM, Giridhar Yasa wrote:
> On Mon, Mar 14, 2016 at 5:47 PM, Sage Weil <sweil@redhat.com> wrote:
>> On Sun, 13 Mar 2016, Giridhar Yasa wrote:
>>> On Fri, Mar 11, 2016 at 8:20 PM, Sage Weil <sweil@redhat.com> wrote:
>>>> On Fri, 4 Mar 2016, Shehbaz Jaffer wrote:
>>>>
>>>>> 2) Will scope of project be to build an allocator on a simulator for
>>>>> SMR Drives? Or will I get actual HDDs to work with? In case of
>>>>> simulators, will the project involve building a simulator on disksim?
>>>> I seem to remember hearing something about a dm module that simulated SMR,
>>>> but I'm not sure.  We may also be able to get some SMR disks to play
>>>> with--I'll reach out to our drive vendor friends.
>>> Would that be https://github.com/hgst/libzbc from HGST?
>> Yes.
>>
>> I'm not sure about a disk simulator or actual drive availability,
>> though.
> libzbc has an emulation mode that can use files or block devices to
> emulate SMR disk zones.
>
> Giridhar
>

Just keep in mind that different vendors have different behaviors - you will 
need to tweak it I assume to make it act like the drive you are interested in.

Regards,

Ric

Re: [GSoC2016] BlueStore SMR Support

[Shehbaz Jaffer]

Sorry for the late reply.

>>>> Would that be https://github.com/hgst/libzbc from HGST?

Thanks Giridhar for the link.

Hi Rick,

> Just keep in mind that different vendors have different behaviors - you will
> need to tweak it I assume to make it act like the drive you are interested
> in.

There is a T10 and T13 standardization for interfaces (ZBC and ZAC
standard). libzbc also follows these standards. From libzbc README

"libzbc implemention is compliant with the latest drafts of
the ZBC and ZAC standards defined by INCITS technical committee T10 and
T13 (respectively)."

I plan to keep the scope of the project such that the allocator will
interact with the libzbc interface in emulated mode which is T10 and
T13 standard compliant. Once the allocator works on libzbc emulated
mode, it can run on disks that support ZBC/ZAC commands. Again from
README.

"In addition to supporting ZBC and ZAC disks, libzbc also implements an
emulation mode allowing emulating the behavior of a host-managed zoned
disks using a regular file or a raw standard block device as backing
storage."

 Is there any other vendor specific behaviour that I should consider?

Thanks,
Shehbaz

On Mon, Mar 14, 2016 at 9:22 AM, Ric Wheeler <rwheeler@redhat.com> wrote:
> On 03/14/2016 08:33 AM, Giridhar Yasa wrote:
>>
>> On Mon, Mar 14, 2016 at 5:47 PM, Sage Weil <sweil@redhat.com> wrote:
>>>
>>> On Sun, 13 Mar 2016, Giridhar Yasa wrote:
>>>>
>>>> On Fri, Mar 11, 2016 at 8:20 PM, Sage Weil <sweil@redhat.com> wrote:
>>>>>
>>>>> On Fri, 4 Mar 2016, Shehbaz Jaffer wrote:
>>>>>
>>>>>> 2) Will scope of project be to build an allocator on a simulator for
>>>>>> SMR Drives? Or will I get actual HDDs to work with? In case of
>>>>>> simulators, will the project involve building a simulator on disksim?
>>>>>
>>>>> I seem to remember hearing something about a dm module that simulated
>>>>> SMR,
>>>>> but I'm not sure.  We may also be able to get some SMR disks to play
>>>>> with--I'll reach out to our drive vendor friends.
>>>>
>>>> Would that be https://github.com/hgst/libzbc from HGST?
>>>
>>> Yes.
>>>
>>> I'm not sure about a disk simulator or actual drive availability,
>>> though.
>>
>> libzbc has an emulation mode that can use files or block devices to
>> emulate SMR disk zones.
>>
>> Giridhar
>>
>
> Just keep in mind that different vendors have different behaviors - you will
> need to tweak it I assume to make it act like the drive you are interested
> in.
>
> Regards,
>
> Ric
>



-- 
Shehbaz Jaffer
First Year Graduate Student
Sir Edward S Rogers Sr Department of Electrical and Computer Engineering
University of Toronto

Re: [GSoC2016] BlueStore SMR Support

[Ric Wheeler]

On 03/14/2016 12:42 PM, Shehbaz Jaffer wrote:
> Sorry for the late reply.
>
>>>>> Would that be https://github.com/hgst/libzbc from HGST?
> Thanks Giridhar for the link.
>
> Hi Rick,
>
>> Just keep in mind that different vendors have different behaviors - you will
>> need to tweak it I assume to make it act like the drive you are interested
>> in.
> There is a T10 and T13 standardization for interfaces (ZBC and ZAC
> standard). libzbc also follows these standards. From libzbc README
>
> "libzbc implemention is compliant with the latest drafts of
> the ZBC and ZAC standards defined by INCITS technical committee T10 and
> T13 (respectively)."
>
> I plan to keep the scope of the project such that the allocator will
> interact with the libzbc interface in emulated mode which is T10 and
> T13 standard compliant. Once the allocator works on libzbc emulated
> mode, it can run on disks that support ZBC/ZAC commands. Again from
> README.
>
> "In addition to supporting ZBC and ZAC disks, libzbc also implements an
> emulation mode allowing emulating the behavior of a host-managed zoned
> disks using a regular file or a raw standard block device as backing
> storage."
>
>   Is there any other vendor specific behaviour that I should consider?
>
> Thanks,
> Shehbaz
>
>

I think that following the spec is the right way to go. Last time we did a meet 
up with all of the various SMR vendors, they still had a lot of differences in 
how the devices actually worked though (number of zones, how to reset a zone, etc).

If the spec process works thought, this should all hopefully converge :)

Ric