Re: RAID-6 and write hole with write-intent bitmap

Mukund Sivaraman
2020-11-24T18:50:19Z

Hi Wols

On Tue, Nov 24, 2020 at 10:10:32AM +0000, Wols Lists wrote:
> On 24/11/20 07:20, Mukund Sivaraman wrote:
> > Hi all
> >
> > I am trying to setup a MD RAID-6 array and use the ext4 filesystem in
> > ordered mode (default) on it. The data gets backed up periodically. I
> > want the array to be always available.
> >
> > I prefer not using a write-journal if it is sufficient for my usage. I
> > want to use the write-intent bitmap only. AIUI the write-hole problem
> > occurs when there is a crash or abrupt power off *and* disk failures.
>
> No, I don't think so. I'm not sure, but aiui, there is a critical point
> where the data is partially saved to disk, and should a power failure
> occur at that precise point you have a stripe incompletely saved, and
> therefore corrupt. This is why you need a log to fix it ...

I appreciate that you took time to reply. Thank you. I am also in the
"not sure" group, and we may be served well by an authoritative answer
from someone who is familiar with the code. I also didn't follow whether
you're saying there is a write hole or not. The answer may be
implementation specific too, so I am looking for an answer from someone
who knows the code.

The following may be incorrect as I am a RAID layperson, but AIUI:

(a) With RAID-5, assuming there are 4 member disks A, B, C, D, a write
operation with its data on disk A and stripe's parity on disk B may
involve:

1. a read of the stripe
2. update of data on A
3. computation and update of parity A^C^D on B

These are not atomic updates. If power is lost between steps 2 and 3,
upon recovery the mismatch between data and parity for the stripe would
be found and the parity can be updated on B. The data chunk written to A
may be incomplete if power is lost during step 2, but the ext4's journal
would return the FS to a consistent state. Moreover, there should not be
any modification/corruption of data in the stripe on disks C and D
(assuming the disks are OK).

(b) With RAID-6, assuming there are 5 member disks A, B, C, D, E, a
write operation with its data on disk A and stripe's parity on disks
B(p) and C(q) would involve:

1. a read of the stripe
2. update of data on A
3. computation and update of parity on B(p)
4. update of parity on C(q)

These are not atomic updates. If power is lost between steps 2 and 3,
upon recovery the mismatch of data on A would be found and the data
chunk can be updated on A. The data chunk written to A may be incomplete
if power is lost during step 2, but the ext4's journal would return the
FS to a consistent state. If power is lost between steps 3 and 4, upon
recovery the mismatch of parity would be found between B(p) and C(q) and
the parity can be updated on B(p) and C(q). Mainly, there should not be
any modification/corruption of data in the stripe on disks D and E
(assuming the disks are OK).

The above may be incorrect, so please indicate what happens, and if
there is a write hole, why there is one.

We don't mind if data in files being written to at the time of power
loss are partially written. It can happen with any abrupt power
loss. The concern is if other unrelated parts of the filesystem not
tracked by the filesystem's journal get corrupted because of other data
chunks of a stripe being updated during recovery.

> >
> > * After a crash or abrupt power off, the write-intent bitmap is used to
> > rewrite parity where necessary. If there is no disk failure during
> > this period, is the RAID-6 array guaranteed to recover without
> > corruption?
> >
> > With RAID-6, will recovery with write-intent bitmap succeed with 1
> > disk failure during the recovery period without a write-journal? i.e.,
> > is there a possibility of write hole with 1 disk failure in a RAID-6
> > array?
> >
> > * With RAID-6 with write-intent bitmap in use, ext4 in ordered mode, no
> > disk failures, and abrupt power loss, is there any chance of data loss
> > in files other than those being written to just before the power loss?
>
> Probably. Sod's law, you will have other files on the same stripe and
> things could go wrong ... Plus I believe some file systems (including
> ext4?) store small files in the directory, not as their own i-node, so
> there's a whole bunch of other complications possible, plus if you
> corrupt the directory ,,,
> >
> > (Apologies if these are silly questions, but I request answers.)
> >
> RULE 0: RAID IS NO SUBSTITUTE FOR BACKUPS.

The data is backed up periodically.

> And if you don't want to lose live data as it is being updated, you need
> a journal. Run the correct horse for the course :-)

It is important that the array is available and not in a failed state or
with an corrupted FS due to a power loss. We would also like to avoid
having to go to restoring from backups as much as possible. There is
power outage about once every week. The system is powered via an
inverter (a lead-acid battery backed UPS) which switches from mains to
battery power when there is power loss within a few tens of milliseconds
that the server's power supply tolerates. Rarely, the switchover time is
longer, or there is a dip, and the server powers off. So consider that
power outages are somewhat common and the array should survive it to
avoid extra work for us, regardless of backups.

The write-journal is a relatively new addition to MD and I feel
conservative about using it for now. I have come across failures
reported on the lists[1], it is not clear if others are using it in
production, and some things such as how to remove the write journal from
an array are not documented (there was a sequence of steps which was
mentioned in the commit log of a patch that introduced support[2], but a
step was missing in it as pointed out in a different mailing list
post[3]). Please don't take these things as criticism - it is just that
the feature appears to be relatively new. Adding an NVMe SSD to hold the
write journal would add another component to the mix which I want to
avoid. However if an authoritative answer indicates the write journal is
required in our case and the implementation is mature, we will try to
adopt it.

[1] https://www.spinics.net/lists/raid/msg62646.html
[2] https://marc.info/?l=linux-raid&m=149063896208043
[3] https://www.spinics.net/lists/raid/msg59940.html

Mukund