Re: understanding speculative preallocation

Jason Rosenberg
2013-07-26T17:40:25Z

[-- Attachment #1.1: Type: text/plain, Size: 6488 bytes --]

Hi Dave,

Thanks for your responses. I'm a bit confused, as I didn't see your
responses on the actual forum (only in my email inbox).

Anyway, I'm surprised that you don't have some list or other way to
correlate version history of XFS, with os release versions. I'm guessing
the version I have is not using the latest/greatest. We actually have
another system that uses an older version of the kernel (2.6.32-279), and
it behaves differently (it still preallocates space beyond what will ever
be used, but not by quite as much). When we rolled out our newer machines
to 2.6.32-358, we started seeing a marked increase in disk full problems.

If, say you tell me, the mainline xfs code has improved behavior, it would
be nice to have a way to know which version of CentOS might include that?
Telling me to read source code across multiple kernel versions sounds like
an interesting endeavor, but not something that is the most efficient use
of my time, unless there truly is no one who can easily tell me anything
about xfs version history.

Do you have any plans to have some sort of improved documentation story
around this? This speculative preallocation behavior is truly unexpected
and not transparent to the user. I can see that it's probably a great
performance boost (especially for something like kafka), but kafka does
have predictable log file rotation capped at fixed sizes, so it would be
great if that could be factored in.

I suppose using the allocsize setting might work in the short term. But I
probably don't want to set allocsize to 1Gb, since that would mean every
single file created would start with that size, is that right? Does the
allocsize setting basically work by always keeping the file size ahead of
consumed space by the allocsize amount?

Thanks,

Jason

On Fri, Jul 26, 2013 at 7:50 AM, Dave Chinner <david@fromorbit.com> wrote:

> On Fri, Jul 26, 2013 at 12:23:40AM -0700, jbr wrote:
> > Hello,
> >
> > I'm looking for general documentation/help with the speculative
> > preallocation feature in xfs. So far, I haven't really been able to find
> > any definitive, up to date documentation on it.
>
> Read the code - it's documented in the comments. ;)
>
> Or ask questions here, because the code changes and the only up to
> date reference is the code and/or the developers that work on it...
>
> > I'm wondering how I can find out definitively which version of xfs I am
> > using, and what the preallocation scheme in use is.
>
> Look at the kernel version, then look at the corresponding source
> code.
>
> > We are running apache kafka on our servers, and kafka uses sequential io
> to
> > write data log files. Kafka uses, by default, a maximum log file size of
> > 1Gb. However, most of the log files end up being 2Gb, and thus the disk
> > fills up twice as fast as it should.
> >
> > We are using xfs on CentOS 2.6.32-358. Is there a way I can know which
> > version of xfs is built into this version of the kernel?
>
> The XFS code is part of the kernel, so look at the kernel source
> code that CentOS ships.
>
> > We are using xfs (mounted with no allocsize specified). I've seen
> varying
> > info suggesting this means it either defaults to an allocsize of 64K
> (which
> > doesn't seem to match my observations), or that it will use dynamic
> > preallocation.
> >
> > I've also seen hints (but no actual canonical documentation) suggesting
> that
> > the dynamic preallocation works by progressively doubling the current
> file
> > size (which does match my observations).
>
> Well, it started off that way, but it has been refined since to
> handle many different cases where this behaviour is sub-optimal.
>
> > What I'm not clear on, is the scheduling for the preallocation. At what
> > point does it decide to preallocate the next doubling of space.
>
> Depends on the type of IO being done.
>
> > Is it when
> > the current preallocated space is used up,
>
> Usually.
>
> > or does it happen when the
> > current space is used up within some threshold.
>
> No.
>
> > What I'd like to do, is
> > keep the doubling behavior in tact, but have it capped so it never
> increases
> > the file beyond 1Gb. Is there a way to do that?
>
> No.
>
> > Can I trick the
> > preallocation to not do a final doubling, if I cap my kafka log files at
> > say, 900Mb (or some percentage under 1Gb)?
>
> No.
>
> > There are numerous references to an allocation schedule like this:
> >
> > freespace max prealloc size
> > >5% full extent (8GB)
> > 4-5% 2GB (8GB >> 2)
> > 3-4% 1GB (8GB >> 3)
> > 2-3% 512MB (8GB >> 4)
> > 1-2% 256MB (8GB >> 5)
> > <1% 128MB (8GB >> 6)
> >
> > I'm just not sure I understand what this is telling me. It seems to
> tell me
> > what the max prealloc size is, with being reduced if the disk is nearly
> > full.
>
> Yes, that's correct. Mainline also does this for quota exhaustion,
> too.
>
> > But it doesn't tell me about the progressive doubling in
> > preallocation (I assume up to a max of 8Gb). Is any of this
> configurable?
>
> No.
>
> > Can we specify a max prealloc size somewhere?
>
> Use the allocsize mount option. It turns off dynamic behaviour and
> fixes the pre-allocation size.
>
> > The other issue seems to be that after the files are closed (from within
> the
> > java jvm), they still don't seem to have their pre-allocated space
> > reclaimed. Are there known issues with closing the files in java not
> > properly causing a flush of the preallocated space?
>
> Possibly. There's a heuristic that turns of truncation at close - if
> your applicatin keeps doing "open-write-close" it will not truncate
> preallocation. Log files typically see this IO pattern from
> applications, and hence triggering that "no truncate" heuristic is
> exactly what you want to have happen to avoid severe fragmentation
> of the log files.
>
> > Any help pointing me to any documentation/user guides which accurately
> > describes this would be appreciated!
>
> The mechanism is not documented outside the code as it changes from
> kernel release to kernel release and supposed to be transparent to
> userspace. It's being refined and optimisaed as issues are reported.
> Indeed, I suspect that all your problems would disappear on mainline
> due to the background removal of preallocation that is no longer
> needed, and Centos doesn't have that...
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>

[-- Attachment #1.2: Type: text/html, Size: 7756 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]