84% -> 60% cache hit ratio for same workflow, different conf

84% -> 60% cache hit ratio for same workflow, different conf

[Pradeep Jindal]

Hi,

I am facing a weird problem, I earlier had a configuration wherein my
backing device was a logical volume (LVM) backed by 3 software RAID0
arrays having 2x1TB disks each, so totally 6TB of storage for backing
device and 300GB logical volume (LVM) backed by SSD as a caching
device. Both were configured as a cache set and I was getting around
84% cache hit ratio.

Now, I have changed the backing device to a bare (no LVM, no raid) 6TB
disk while everything else stays same (caching device on LVM, 300GB)
and for exactly same workload I am only getting 58-60% cache hit
ratio.

I am confused, what could be the reason?

- Pradeep Jindal

Re: 84% -> 60% cache hit ratio for same workflow, different conf

[Vojtech Pavlik]

On Mon, Mar 28, 2016 at 11:47:08PM +0530, Pradeep Jindal wrote:
> Hi,
> 
> I am facing a weird problem, I earlier had a configuration wherein my
> backing device was a logical volume (LVM) backed by 3 software RAID0
> arrays having 2x1TB disks each, so totally 6TB of storage for backing
> device and 300GB logical volume (LVM) backed by SSD as a caching
> device. Both were configured as a cache set and I was getting around
> 84% cache hit ratio.
> 
> Now, I have changed the backing device to a bare (no LVM, no raid) 6TB
> disk while everything else stays same (caching device on LVM, 300GB)
> and for exactly same workload I am only getting 58-60% cache hit
> ratio.
> 
> I am confused, what could be the reason?

Full stripe size writes are configured according to the backing device,
for example.

-- 
Vojtech Pavlik
Director SuSE Labs

Re: 84% -> 60% cache hit ratio for same workflow, different conf

[Pradeep Jindal]

Didn't quite get it, could you please elaborate?

- Pradeep Jindal


On Tue, Mar 29, 2016 at 12:47 AM, Vojtech Pavlik <vojtech@suse.com> wrote:
> On Mon, Mar 28, 2016 at 11:47:08PM +0530, Pradeep Jindal wrote:
>> Hi,
>>
>> I am facing a weird problem, I earlier had a configuration wherein my
>> backing device was a logical volume (LVM) backed by 3 software RAID0
>> arrays having 2x1TB disks each, so totally 6TB of storage for backing
>> device and 300GB logical volume (LVM) backed by SSD as a caching
>> device. Both were configured as a cache set and I was getting around
>> 84% cache hit ratio.
>>
>> Now, I have changed the backing device to a bare (no LVM, no raid) 6TB
>> disk while everything else stays same (caching device on LVM, 300GB)
>> and for exactly same workload I am only getting 58-60% cache hit
>> ratio.
>>
>> I am confused, what could be the reason?
>
> Full stripe size writes are configured according to the backing device,
> for example.
>
> --
> Vojtech Pavlik
> Director SuSE Labs

Re: 84% -> 60% cache hit ratio for same workflow, different conf

[Vojtech Pavlik]

On Tue, Mar 29, 2016 at 01:35:32AM +0530, Pradeep Jindal wrote:

> > > I am facing a weird problem, I earlier had a configuration wherein my
> > > backing device was a logical volume (LVM) backed by 3 software RAID0
> > > arrays having 2x1TB disks each, so totally 6TB of storage for backing
> > > device and 300GB logical volume (LVM) backed by SSD as a caching
> > > device. Both were configured as a cache set and I was getting around
> > > 84% cache hit ratio.
> > >
> > > Now, I have changed the backing device to a bare (no LVM, no raid) 6TB
> > > disk while everything else stays same (caching device on LVM, 300GB)
> > > and for exactly same workload I am only getting 58-60% cache hit
> > > ratio.
> > >
> > > I am confused, what could be the reason?
> >
> > Full stripe size writes are configured according to the backing device,
> > for example.

> Didn't quite get it, could you please elaborate?

Backing devices that advertise a stripe size cause bcache to enable a
mode where it tries to always write a full stripe to the backing
devices, hence changing behavior of the cache.

-- 
Vojtech Pavlik
Director SuSE Labs

Re: 84% -> 60% cache hit ratio for same workflow, different conf

[Pradeep Jindal]

But in my new configuration backing device is a bare disk, no LVM,
caching device is on LVM though.

- Pradeep Jindal


On Tue, Mar 29, 2016 at 1:49 AM, Vojtech Pavlik <vojtech@suse.com> wrote:
> On Tue, Mar 29, 2016 at 01:35:32AM +0530, Pradeep Jindal wrote:
>
>> > > I am facing a weird problem, I earlier had a configuration wherein my
>> > > backing device was a logical volume (LVM) backed by 3 software RAID0
>> > > arrays having 2x1TB disks each, so totally 6TB of storage for backing
>> > > device and 300GB logical volume (LVM) backed by SSD as a caching
>> > > device. Both were configured as a cache set and I was getting around
>> > > 84% cache hit ratio.
>> > >
>> > > Now, I have changed the backing device to a bare (no LVM, no raid) 6TB
>> > > disk while everything else stays same (caching device on LVM, 300GB)
>> > > and for exactly same workload I am only getting 58-60% cache hit
>> > > ratio.
>> > >
>> > > I am confused, what could be the reason?
>> >
>> > Full stripe size writes are configured according to the backing device,
>> > for example.
>
>> Didn't quite get it, could you please elaborate?
>
> Backing devices that advertise a stripe size cause bcache to enable a
> mode where it tries to always write a full stripe to the backing
> devices, hence changing behavior of the cache.
>
> --
> Vojtech Pavlik
> Director SuSE Labs

Re: 84% -> 60% cache hit ratio for same workflow, different conf

[Pradeep Jindal]

I am using writearound cache mode and have xfs on bcache device
- Pradeep Jindal


On Tue, Mar 29, 2016 at 1:52 AM, Pradeep Jindal <praddyjindal@gmail.com> wrote:
> But in my new configuration backing device is a bare disk, no LVM, caching
> device is on LVM though.
>
> On Mar 29, 2016 1:49 AM, "Vojtech Pavlik" <vojtech@suse.com> wrote:
>>
>> On Tue, Mar 29, 2016 at 01:35:32AM +0530, Pradeep Jindal wrote:
>>
>> > > > I am facing a weird problem, I earlier had a configuration wherein
>> > > > my
>> > > > backing device was a logical volume (LVM) backed by 3 software RAID0
>> > > > arrays having 2x1TB disks each, so totally 6TB of storage for
>> > > > backing
>> > > > device and 300GB logical volume (LVM) backed by SSD as a caching
>> > > > device. Both were configured as a cache set and I was getting around
>> > > > 84% cache hit ratio.
>> > > >
>> > > > Now, I have changed the backing device to a bare (no LVM, no raid)
>> > > > 6TB
>> > > > disk while everything else stays same (caching device on LVM, 300GB)
>> > > > and for exactly same workload I am only getting 58-60% cache hit
>> > > > ratio.
>> > > >
>> > > > I am confused, what could be the reason?
>> > >
>> > > Full stripe size writes are configured according to the backing
>> > > device,
>> > > for example.
>>
>> > Didn't quite get it, could you please elaborate?
>>
>> Backing devices that advertise a stripe size cause bcache to enable a
>> mode where it tries to always write a full stripe to the backing
>> devices, hence changing behavior of the cache.
>>
>> --
>> Vojtech Pavlik
>> Director SuSE Labs

Re: 84% -> 60% cache hit ratio for same workflow, different conf

[Pradeep Jindal]

On my new configuration the bset unwritten key bytes are non-zero
whereas on older config that is zero:

btree nodes:            1192
written sets:           2377
unwritten sets:         1186
written key bytes:      204048256
unwritten key bytes:    115448
floats:                 68721067953
failed:                 367

What does that indicate?

Also, could you please clarify on how advertising stripe size would
affect writearound cache mode?
- Pradeep Jindal


On Tue, Mar 29, 2016 at 1:56 AM, Pradeep Jindal <praddyjindal@gmail.com> wrote:
> I am using writearound cache mode and have xfs on bcache device
> - Pradeep Jindal
>
>
> On Tue, Mar 29, 2016 at 1:52 AM, Pradeep Jindal <praddyjindal@gmail.com> wrote:
>> But in my new configuration backing device is a bare disk, no LVM, caching
>> device is on LVM though.
>>
>> On Mar 29, 2016 1:49 AM, "Vojtech Pavlik" <vojtech@suse.com> wrote:
>>>
>>> On Tue, Mar 29, 2016 at 01:35:32AM +0530, Pradeep Jindal wrote:
>>>
>>> > > > I am facing a weird problem, I earlier had a configuration wherein
>>> > > > my
>>> > > > backing device was a logical volume (LVM) backed by 3 software RAID0
>>> > > > arrays having 2x1TB disks each, so totally 6TB of storage for
>>> > > > backing
>>> > > > device and 300GB logical volume (LVM) backed by SSD as a caching
>>> > > > device. Both were configured as a cache set and I was getting around
>>> > > > 84% cache hit ratio.
>>> > > >
>>> > > > Now, I have changed the backing device to a bare (no LVM, no raid)
>>> > > > 6TB
>>> > > > disk while everything else stays same (caching device on LVM, 300GB)
>>> > > > and for exactly same workload I am only getting 58-60% cache hit
>>> > > > ratio.
>>> > > >
>>> > > > I am confused, what could be the reason?
>>> > >
>>> > > Full stripe size writes are configured according to the backing
>>> > > device,
>>> > > for example.
>>>
>>> > Didn't quite get it, could you please elaborate?
>>>
>>> Backing devices that advertise a stripe size cause bcache to enable a
>>> mode where it tries to always write a full stripe to the backing
>>> devices, hence changing behavior of the cache.
>>>
>>> --
>>> Vojtech Pavlik
>>> Director SuSE Labs