TIER: combine SSDs and HDDs into a single block device

TIER: combine SSDs and HDDs into a single block device

[Tommi Virtanen]

Sounds like bcache in writeback mode. Assumes all underlying block
devices are RAIDed, or losing one will mean losing data; that is, for
example RAID1(SSD+SSD) & RAID5(8*HDD).

http://www.lessfs.com/wordpress/?p=776

Re: TIER: combine SSDs and HDDs into a single block device

[Mark Nelson]

On 8/2/12 11:13 AM, Tommi Virtanen wrote:
> Sounds like bcache in writeback mode. Assumes all underlying block
> devices are RAIDed, or losing one will mean losing data; that is, for
> example RAID1(SSD+SSD) & RAID5(8*HDD).
>
> http://www.lessfs.com/wordpress/?p=776
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Neat!  I'll try to play with it once the test hardware all makes it in.

Alex is also trying to bug the XFS guys (and Sage bugged the BTRFS guys) 
about ways to put metadata on SSD while keeping data on spinning disk. 
It sounds like there is a hack for XFS that would let us keep inodes in 
the lower portion of a volume up to some configurable boundary and then 
we could use lvm to assign that portion of the volume to an SSD.  The 
BTRFS guys have a SOC project in the works to separate out metadata onto 
another disk.

I think these kinds of things could really help our small request 
performance.

Mark

Re: TIER: combine SSDs and HDDs into a single block device

[Alex Elder]

On 08/02/2012 12:02 PM, Mark Nelson wrote:
> On 8/2/12 11:13 AM, Tommi Virtanen wrote:
>> Sounds like bcache in writeback mode. Assumes all underlying block
>> devices are RAIDed, or losing one will mean losing data; that is, for
>> example RAID1(SSD+SSD) & RAID5(8*HDD).
>>
>> http://www.lessfs.com/wordpress/?p=776
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> 
> Neat!  I'll try to play with it once the test hardware all makes it in.
> 
> Alex is also trying to bug the XFS guys (and Sage bugged the BTRFS guys)
> about ways to put metadata on SSD while keeping data on spinning disk.

I have the XFS patch.  It's based on pretty old kernel code.  I began
porting it forward yesterday but it was taking too long so I set
it aside.

I'll pick it up again soon to see if I can get through it.

					-Alex

> It sounds like there is a hack for XFS that would let us keep inodes in
> the lower portion of a volume up to some configurable boundary and then
> we could use lvm to assign that portion of the volume to an SSD.  The
> BTRFS guys have a SOC project in the works to separate out metadata onto
> another disk.
> 
> I think these kinds of things could really help our small request
> performance.
> 
> Mark
> 
> -- 
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: TIER: combine SSDs and HDDs into a single block device

[Christoph Hellwig]

On Thu, Aug 02, 2012 at 12:02:44PM -0500, Mark Nelson wrote:
> Alex is also trying to bug the XFS guys (and Sage bugged the BTRFS
> guys) about ways to put metadata on SSD while keeping data on
> spinning disk. It sounds like there is a hack for XFS that would let
> us keep inodes in the lower portion of a volume up to some
> configurable boundary and then we could use lvm to assign that
> portion of the volume to an SSD.  The BTRFS guys have a SOC project
> in the works to separate out metadata onto another disk.

Also with XFS you can use the realtime device for data and the main
device for all metadata.

Re: TIER: combine SSDs and HDDs into a single block device

[Mark Nelson]

On 08/02/2012 12:57 PM, Christoph Hellwig wrote:
> On Thu, Aug 02, 2012 at 12:02:44PM -0500, Mark Nelson wrote:
>> Alex is also trying to bug the XFS guys (and Sage bugged the BTRFS
>> guys) about ways to put metadata on SSD while keeping data on
>> spinning disk. It sounds like there is a hack for XFS that would let
>> us keep inodes in the lower portion of a volume up to some
>> configurable boundary and then we could use lvm to assign that
>> portion of the volume to an SSD.  The BTRFS guys have a SOC project
>> in the works to separate out metadata onto another disk.
>
> Also with XFS you can use the realtime device for data and the main
> device for all metadata.
>

I was thinking of doing that.  Is the realtime allocator a good fit for 
this kind of thing?  I think dchinner mentioned on the xfs mailing list 
last year that it's single threaded and not very well optimized (and 
maybe not production viable?)

Mark

-- 
Mark Nelson
Performance Engineer
Inktank

Re: TIER: combine SSDs and HDDs into a single block device

[Christoph Hellwig]

On Thu, Aug 02, 2012 at 04:49:11PM -0500, Mark Nelson wrote:
> I was thinking of doing that.  Is the realtime allocator a good fit
> for this kind of thing?  I think dchinner mentioned on the xfs
> mailing list last year that it's single threaded and not very well
> optimized (and maybe not production viable?)

It's generally a bit dated and bit rotting, and as Dave said doesn't
parallelize.  But for a setup where you have one OSD per disk and lots
of OSD that's not really quite as important.

Re: TIER: combine SSDs and HDDs into a single block device

[Calvin Morrow]

I played with TIER about a week ago.  Definitely a decent
implementation of HSM and seems to work well from my testing.
Assuming a three tier setup (we'll say SSD, 15K SAS, 7K SATA), the
code drops sequential I/O on tier 2 (the 15K) first, moving it down as
tier 2 fills up and/or blocks are unused.  If you do a lot of I/O on a
particular segment, it moves it up to the faster tiers.

Kent Overstreet (the bcache developer) also took a look at the code
and weighed in.  He said it was impressive what TIER managed to do in
~3K lines of code, although expressed his concern that TIER uses a lot
of rb trees which would limit random read I/O performance.

Current caveats:  TIER devices can't be expanded.  TIER devices also
can't have additional TIERs added after setup.  The developer said
he's working on adding those features and hopes to have them in the
next 2-3 months.

Calvin

On Thu, Aug 2, 2012 at 10:13 AM, Tommi Virtanen <tv@inktank.com> wrote:
> Sounds like bcache in writeback mode. Assumes all underlying block
> devices are RAIDed, or losing one will mean losing data; that is, for
> example RAID1(SSD+SSD) & RAID5(8*HDD).
>
> http://www.lessfs.com/wordpress/?p=776
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html