linux-lvm.redhat.com archive mirror
 help / color / mirror / Atom feed
From: Zdenek Kabelac <zdenek.kabelac@gmail.com>
To: "Matwey V. Kornilov" <matwey.kornilov@gmail.com>,
	linux-lvm@lists.linux.dev
Subject: Re: lvmthin man page is not clear on pmspare and RAID
Date: Wed, 18 Nov 2026 11:06:53 +0100	[thread overview]
Message-ID: <e7d5386a-aacb-4f21-b432-e1bbbc654a36@gmail.com> (raw)
In-Reply-To: <CAJs94EY01OBuV0hsOeP3sy=kMCamMRBO-6m4Zx3x72rLcNc46Q@mail.gmail.com>

Dne 15. 11. 26 v 11:12 Matwey V. Kornilov napsal(a):
> Dear LVM team,
> 
> Recently I came across the lvmthin man page. My interest is in having
> a thin pool on RAID1 (mirror). While there is information on this
> topic in the man page, I am not sure that I understand all the
> concepts correctly. I think that when we are talking about storing
> data, it is better to explain things explicitly.
> 
> The most problematic topic is how pmspare is related to RAID and vice
> versa. In the section "Thin Pool on LVM RAID", there is an example of
> a mirrored thin pool, where I can see that the pmspare volume is not
> mirrored. Why? What happens when the physical drive containing pmspare
> goes out of service? vgreduce won't let me remove the failed drive
> from the volume group due to the hidden pmspare volume allocated on
> "[unknown]".
> 
> My understanding, based on the section "Thin Pool Metadata check and
> repair", is that pmspare is really used only while "lvconvert
> --repair" is running, either manually or automatically, so the volume
> can be safely disposed of as long as there is enough free space in the
> volume group. And this means that each physical drive in the volume
> group needs to have enough free space for the pmspare volume in case
> of an emergency. Is this correct?
> 
> I would be particularly happy to see a new section, "Recovering Thin
> Pool on LVM RAID", in the lvmthin man page, explaining these specific
> concerns.

Hi,

Thanks for reading the page carefully and for writing this up. Your
reading is mostly right, and the pmspare/RAID relationship is indeed not
explained well in lvmthin(7). A "Recovering Thin Pool on LVM RAID"
section is a good idea.

Let me take your points in order.

What pmspare is

pmspare is a VG-level spare, not a per-pool volume. There is at most one
per VG, created with the first thin (or cache) pool unless
--poolmetadataspare n is used. It is sized to be at least as large as the
largest pool metadata LV in the VG, and is extended if a larger metadata
LV appears. It is a hidden LV with a single copy by default, allocated
from the VG's free space like any other LV, and it is not activated as
part of the pool.

It is not a copy of the pool metadata and it is not a backup. It is
reserved space. The option description says exactly that: "A spare
metadata LV is reserved space that can be used when repairing a pool." It
only contains valid metadata after thin_repair has written to it during
lvconvert --repair.

pmspare is a last-defence reserve, not a routine part of the recovery
workflow. The expectation is that an admin monitors the pools (dmeventd,
autoextend thresholds, whenfull, and free space in the VG) and does not
use the spare to compensate for a full or neglected VG. It is there for
the case where metadata is damaged despite that care, and it is intended
for an admin who understands the state of the VG. It also buys one
repair: lvconvert --repair writes the repaired metadata into the spare,
swaps it in as the new tmeta, and then tries to allocate a fresh spare
from free space (unless --poolmetadataspare n was used). If that
allocation fails, the VG is left without a pmspare, and a second pool
that needs repair has nowhere to write. One VG-wide spare cannot cover
several simultaneous pool failures; they have to be handled one at a
time, with enough free space for each repaired metadata LV and for the
next spare.

Why a second metadata volume is needed

Before the failure modes, it is worth being explicit about why metadata
matters so much. The data LV is just a pool of chunks, allocated on
demand in no particular order. The metadata contains the B-tree that
maps the logical blocks of each thin LV to chunks in the data LV.
Without that mapping, the data LV is not a filesystem or a set of LVs;
it is just a collection of thin-pool chunks, and there is no way to tell
which block belongs to which thin LV or file. The data blocks may still
be physically intact, but they are mostly useless without the mapping.
The metadata is small, but it is the difference between a usable pool
and a pile of chunks.

Two different failures can affect a thin pool. The device holding the
metadata LV can fail, in which case the metadata may be gone even if the
data LV is still readable. Or the metadata can be corrupted, for example
by a kernel bug, while the device itself is still usable. A physical
failure is handled by replacing the lost RAID image on a new device; if
the metadata was not redundant and the only copy is gone, the pmspare
cannot recreate it. A corrupted metadata LV is handled by writing a
repaired copy to a new metadata LV and swapping it in.

The dm-thin metadata format and tools do not support in-place repair.
Thin metadata was designed to be small, and supporting in-place repair
would add complexity and cost to the normal metadata path for a rare
operation. Instead, thin_repair reads the old metadata and writes a new,
reconstructed copy to a different LV. The old metadata LV is left
untouched, and the new copy is swapped in only after it has been
successfully written.

This is why a second metadata-sized LV is needed for the corruption
case, and why LVM keeps the pmspare LV: it is a preallocated destination
for that new copy. It also means the repair does not depend on finding
free space at the worst possible moment, when the old metadata LV is
still allocated and the VG may be full.

Why it is not mirrored

Because it is not part of the pool or of the RAID set. One pmspare serves
every thin and cache pool in the VG, and those pools can have different
layouts (linear, raid1, raid5, raid10, ...). A single spare cannot match
the layout of all of them. It is scratch space; redundancy of the live
metadata is the job of the tmeta LV itself. So the example output is
expected: [lvol0_pmspare] is a plain LV while pool0_tmeta is raid1.

Given that it is a single-use last-defence reserve rather than a live
copy, the current design does not try to mirror it; the trade-off is the
one you identified.

There is an important consequence, and this is the part I think the man
page must state explicitly: after lvconvert --repair, the pool metadata
LV is the former pmspare, so it becomes linear. The old RAID1 metadata LV
is kept read-only as ThinPool_metaN. The current code warns only that the
new metadata LV might use different PVs and may need pvmove; it does not
warn that the metadata is no longer RAID1. In practice, after a repair
the new tmeta is linear, while the saved old metadata LV still has its
original RAID layout.

So if you repair a RAID-backed pool this way, you must re-establish
metadata redundancy afterwards. If you want to keep RAID1 in the first
place, use the manual thin_repair plus lvconvert --thinpool --poolmetadata
path with a new RAID1 metadata LV instead of the built-in spare swap.

If the PV holding pmspare fails

Losing the pmspare itself does not affect the pool. It is not used during
normal operation and is not activated with the pool. You lose the
preallocated repair area, not data. If the same PV also held data or
metadata LVs, those failures are separate; with RAID they may leave the
pool degraded. The VG is marked partial because the hidden pmspare still
has extents on the missing PV.

You have found a real rough edge: vgreduce --removemissing will not
remove it. Without --force it only removes empty missing PVs; with
--force it skips invisible LVs, so the partial pmspare is left behind,
consolidation fails, and vgreduce exits with an error. The workaround is
to remove the spare explicitly first:

   # lvs -a -o name,devices,segtype VG
   # lvremove -y VG/lvol0_pmspare
   # vgreduce --removemissing --force VG

The exact name may be lvol1_pmspare, etc. lvremove is documented as
allowed for pmspare, and it will warn that removing it disables automatic
recovery attempts. After that, the next pool creation or conversion, or a
later lvconvert --repair that needs one, will create a new pmspare if
there is free space. There is currently no standalone "create pmspare"
command.

If the PV is still present and you simply want to remove it, vgreduce
also refuses while pmspare has extents there. In that case pvmove -n
VG/lvol0_pmspare <src> <dst> (or a full pvmove <src>) moves it out of the
way.

About free space

Your "as long as there is enough free space" is right in the narrow sense
that removing pmspare does not endanger the pool. But pmspare's whole
point is to reserve that space before you need it. At repair time the
damaged tmeta is still allocated, so without a spare LVM may have nowhere
to write the repaired copy if the VG is full. The spare is a last
defence, not a substitute for capacity planning: a successful repair
consumes it, and a fresh spare has to be allocated afterwards, which
again needs free space. Keeping one metadata-sized area free is cheap
insurance, but the primary defences are monitoring the pool and not
letting the VG fill up. An admin who treats pmspare as the normal
recovery path can end up with a full VG and no spare.

But "each physical drive needs enough free space for pmspare" is not
correct. pmspare is one LV with one copy. It does not need space on every
PV and it is not mirrored. What does need space is RAID reconstruction:
if a PV dies, each lost RAID image (a full copy of tdata or tmeta) needs
a new home on a surviving or replacement PV. That requirement is separate
from pmspare, and for tdata it is much larger. Plan spare capacity for
the largest image, not for a pmspare copy on every drive.

Also note that lvconvert --repair VG/pool on a thin pool is the
thin-metadata repair, not the RAID repair. For a failed RAID leg, repair
the RAID LV itself, e.g. lvconvert --repair VG/pool_tdata <newdev> and
lvconvert --repair VG/pool_tmeta <newdev>. For RAID LVs, dmeventd's
raid_fault_policy=allocate can replace failed images automatically using
free space; that path does not use pmspare. LVM does not run thin_repair
automatically; that is an admin action.

Recovery order for a RAID-backed thin pool

For the new section I would describe the cases separately:

1. Lost a PV, RAID legs missing but metadata content intact: after
    removing a partial pmspare, make the VG consistent and replace the
    missing legs:

      # lvconvert --repair VG/<pool>_tdata <newdev>
      # lvconvert --repair VG/<pool>_tmeta <newdev>

    This is RAID repair, not thin repair.

2. Metadata content corrupt: the pool must be inactive, then lvconvert
    --repair VG/<pool>. This consumes pmspare and leaves tmeta linear.
    Verify the pool and the thin LVs, then either restore redundancy or
    use the manual path:

      # lvchange -an VG/<pool>
      # lvcreate --type raid1 -m1 -L <meta size> -n newmeta VG <dev1> <dev2>
      # lvchange -ay VG/<pool>_tmeta VG/newmeta
      # thin_repair -i /dev/VG/<pool>_tmeta -o /dev/VG/newmeta
      # lvconvert --thinpool VG/<pool> --poolmetadata VG/newmeta

    The old metadata LV is kept as ThinPool_metaN until you have verified
    the new one.

3. Lost the pmspare PV: remove the partial pmspare as above, then
    vgreduce --removemissing. A new pmspare is created on demand if there
    is free space.

I will write this up for lvmthin(7), including that pmspare is a
single-use, VG-wide last defence and not a replacement for monitoring and
free-space planning. Thanks again for the report; the distinction between
"live metadata redundancy" and "reserved repair space" is exactly what
the page should make explicit.


Regards

Zdenek




      reply	other threads:[~2026-11-18 10:06 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-15 10:12 lvmthin man page is not clear on pmspare and RAID Matwey V. Kornilov
2026-11-18 10:06 ` Zdenek Kabelac [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=e7d5386a-aacb-4f21-b432-e1bbbc654a36@gmail.com \
    --to=zdenek.kabelac@gmail.com \
    --cc=linux-lvm@lists.linux.dev \
    --cc=matwey.kornilov@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).