[RFC PATCH 0/2] xfs: hard limit background CIL push size

[Dave Chinner <david@fromorbit.com>]

Hi folks,

As we've been discussing from the recent generic/530 hangs, the CIL
push work can get held off for an arbitrary amount of time and
result in the CIL checkpoint growing large enough that it can
consume the entire log. This is not allowed - the log recovery
algorithm for finding the head and tail of the log requires two
checkpoints in the log so it can find the last full checkpoint that
it needs to recover. That puts a hard limit on the size of the CIL
checkpoints of just under 50% of the log.

The CIL currently doesn't have any enforcement on that - log space
hangs due to CIL overruns are not something we've see in test or
production systems until the recent unlinked list modifications we
made. Hence this has laregely been a theoretical problem rather than
a practical problem up to this point.

While we've made changes that avoid the CIL hold-off vectors that
lead to the generic/530 hangs, we have enough data on the CIL
characteristics and performance to be able to put in place hard
limits without compromising performance.

The first patch limits the CIL size on large logs - we don't need to
aggregate hundreds of megabytes of metadata in the CIL to realise
the relogging benefits that the CIL provides. Measures show that the
point at which performance starts to be affected is somewhere
between 16MB and 32MB of aggregated changes in the CIL. Hence
the background push threshold is limited to be 32MB on large logs,
but remains at 12.5% of the log on small logs.

The second patch adds a hard limit on the CIL background push
threshold. This is set to double the size of the push threshold, so
at most a single CIL context will consume 25% of the log before
attempts to do background pushes will block waiting for the
background push of that context to start. This means that all
processes that commit to a push context that is over the hard limit
will sleep until the background CIL push work starts on that
context. At that point, they will be woken and their next
transaction commits will occur into the new CIL commit that the
background push will switch over to.

This provides a hard limit on the size of CIL checkpoint of ~1/4 of
the entire log, well inside the size limit that log recovery imposes
on us. For maximally sized logs, this hard limit ends up being about
3% of the entire log, so it serves to keep the logged objects moving
from the CIL to the AIL at a reasonable rate and spreads them over a
wider range of LSNs giving more graduated (less bursty) tail pushing
behaviour.

Comments welcome.

-Dave.