Disabling journal

Disabling journal

[Stefan Priebe - Profihost AG]

Hi list,

is there a way to disable journal completely? For fast ssd storage it doesn't make sense and I want to test how speed changes.

Greets
Stefan

Re: Disabling journal

[Sage Weil]

On Sun, 11 Nov 2012, Stefan Priebe - Profihost AG wrote:
> Hi list,
> 
> is there a way to disable journal completely? For fast ssd storage it 
> doesn't make sense and I want to test how speed changes.

With btrfs, yes, although this isn't something we have tested in a while.  
You'd need to play with 'filestore min sync interval' and 'filestore max 
sync interval' to basically pick your latency range.

sage

Re: Disabling journal

[Stefan Priebe]

Hi Sage,

> With btrfs, yes, although this isn't something we have tested in a while.
I'm not using btrfs as long as the devs claim it is not ready for prod.

> You'd need to play with 'filestore min sync interval' and 'filestore max
> sync interval' to basically pick your latency range.
Any suggestions? Right now i've 500MB tmpfs journal per SSD.

Stefan

Re: Disabling journal

[Sage Weil]

On Sun, 11 Nov 2012, Stefan Priebe wrote:
> Hi Sage,
> 
> > With btrfs, yes, although this isn't something we have tested in a while.
> I'm not using btrfs as long as the devs claim it is not ready for prod.

In that case, the journal is needed for consistency of the fs; we rely on 
writeahead journaling.  It can't be turned off.

Putting it on a ramdisk in this case is interesting for performance, but 
it means that a crash/reboot/powerloss event leaves the fs in an 
inconsistent and unusable state.

The only time tmpfs is potentially useful in production is when you're 
using btrfs *and* have independent backup power sources for replicas (and 
can thus avoid worrying about a site-wide power failure and loss of 
journal).  (Or have relaxed requirements for the durability of recent 
writes.)

sage

Re: Disabling journal

[Stefan Priebe - Profihost AG]

Am 12.11.2012 15:42, schrieb Sage Weil:
> On Sun, 11 Nov 2012, Stefan Priebe wrote:
>> Hi Sage,
>>
>>> With btrfs, yes, although this isn't something we have tested in a while.
>> I'm not using btrfs as long as the devs claim it is not ready for prod.
>
> In that case, the journal is needed for consistency of the fs; we rely on
> writeahead journaling.  It can't be turned off.
>
> Putting it on a ramdisk in this case is interesting for performance, but
> it means that a crash/reboot/powerloss event leaves the fs in an
> inconsistent and unusable state.

But only if for replicas 2 both nodes crash / have a powerloss?

> The only time tmpfs is potentially useful in production is when you're
> using btrfs *and* have independent backup power sources for replicas (and
> can thus avoid worrying about a site-wide power failure and loss of
> journal).  (Or have relaxed requirements for the durability of recent
> writes.)
What happens for XFS and replicas two and ONE host has a power loss? The 
other replica / journal should be still there.

I've no idea where to put the journal on.

I mean i've 8 SSDs per Host one per osd each with a write IOP/s speed of 
45.000 iops to whole IOP/s write speed of 360.000 IOP/s per Node.

Which journal device can handle this? And if i put the journal on the 
same disk as the OSD it has to copy the data around.

Greets,
Stefan

Re: Disabling journal

[Sage Weil]

On Mon, 12 Nov 2012, Stefan Priebe - Profihost AG wrote:
> Am 12.11.2012 15:42, schrieb Sage Weil:
> > On Sun, 11 Nov 2012, Stefan Priebe wrote:
> > > Hi Sage,
> > > 
> > > > With btrfs, yes, although this isn't something we have tested in a
> > > > while.
> > > I'm not using btrfs as long as the devs claim it is not ready for prod.
> > 
> > In that case, the journal is needed for consistency of the fs; we rely on
> > writeahead journaling.  It can't be turned off.
> > 
> > Putting it on a ramdisk in this case is interesting for performance, but
> > it means that a crash/reboot/powerloss event leaves the fs in an
> > inconsistent and unusable state.
> 
> But only if for replicas 2 both nodes crash / have a powerloss?

Then you're okay.. but the one that lost the journal effectively also lost 
the contents of the SSD.  Also, manual intervention is currently needed to 
reinitialize the osd (since this is not a normal failure mode).

> > The only time tmpfs is potentially useful in production is when you're
> > using btrfs *and* have independent backup power sources for replicas (and
> > can thus avoid worrying about a site-wide power failure and loss of
> > journal).  (Or have relaxed requirements for the durability of recent
> > writes.)
> What happens for XFS and replicas two and ONE host has a power loss? The other
> replica / journal should be still there.
> 
> I've no idea where to put the journal on.
> 
> I mean i've 8 SSDs per Host one per osd each with a write IOP/s speed of
> 45.000 iops to whole IOP/s write speed of 360.000 IOP/s per Node.
> 
> Which journal device can handle this? And if i put the journal on the same
> disk as the OSD it has to copy the data around.

I think you have two choices.  Either put the journal SSDs (perhaps a 
journal on an existing one), or use a higher-end NVRAM-based device.  
There are several of these out there, although I'm blanking on product 
names at the moment.  The best are probably the battery-backed DRAM ones 
with a bit of flash for when the battery gets low.  Lots of RAID 
controllers also have some onboard NVRAM that can often be finagled into 
being useful, at least with spinning disks; I'm not sure how they perform 
with SSDs.

sage