DM_snapshot_cow filesystem (dmsetup create snapshot)

DM_snapshot_cow filesystem (dmsetup create snapshot)

[Mr Dash Four]

As part of my livecd data structure dmsetup is used to create a snapshot 
containing xz-compressed root combined with copy-on-write ram-based 
overlay.

The way as I understand it, the ram-based overlay contains only the 
"chunks" of data which is changed. Is there a way - in terms of a tool 
or by any other means - to look at this overlay to see what data has 
been changed exactly?

I have tried, rather naively as it turned out, to mount the ram-based 
overlay but received  "unknown filesystem type 'DM_snapshot_cow'" error 
message, which makes me thing that the data structure is unique somehow 
and private to the device mapper.

One other thing - my dmsetup man page is from 2006 (created some 5 years 
ago) even though I am using the latest device-mapper package (1.02.63 I 
believe) and this man page does not contain any information on dmsetup 
create snapshot format, which is very annoying. Is there an updated 
version of the man page I could look at?

Re: DM_snapshot_cow filesystem (dmsetup create snapshot)

[Alasdair G Kergon]

markmc did some code that shows how to read the format a few years ago here:
http://people.gnome.org/~markmc/code/merge-dm-snapshot.c

Otherwise look at dm-snap-persistent.c in the kernel tree.

Alasdair

Re: DM_snapshot_cow filesystem (dmsetup create snapshot)

[Frederick Grose]

[-- Attachment #1.1: Type: text/plain, Size: 606 bytes --]

On Sat, Oct 29, 2011 at 4:49 PM, Alasdair G Kergon <agk@redhat.com> wrote:

> markmc did some code that shows how to read the format a few years ago
> here:
> http://people.gnome.org/~markmc/code/merge-dm-snapshot.c
>
> Otherwise look at dm-snap-persistent.c in the kernel tree.
>
> Alasdair


Users can easily exhaust a LiveUSB snapshot overlay by writing
too many changes to the OS filesystem, such as by performing
a yum update.

Would such an 'exhausted' overlay be amenable to some sort of
data recovery or forensics?

If so, how so; if not, why not?

(or suitable references)

Thanks,         --Fred

[-- Attachment #1.2: Type: text/html, Size: 1369 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: DM_snapshot_cow filesystem (dmsetup create snapshot)

[Douglas McClendon]

On 11/27/2011 04:31 PM, Frederick Grose wrote:
> On Sat, Oct 29, 2011 at 4:49 PM, Alasdair G Kergon <agk@redhat.com
> <mailto:agk@redhat.com>> wrote:
>
>     markmc did some code that shows how to read the format a few years
>     ago here:
>     http://people.gnome.org/~markmc/code/merge-dm-snapshot.c
>     <http://people.gnome.org/%7Emarkmc/code/merge-dm-snapshot.c>
>
>     Otherwise look at dm-snap-persistent.c in the kernel tree.
>
>     Alasdair
>
>
> Users can easily exhaust a LiveUSB snapshot overlay by writing
> too many changes to the OS filesystem, such as by performing
> a yum update.
>
> Would such an 'exhausted' overlay be amenable to some sort of
> data recovery or forensics?
>
> If so, how so; if not, why not?

Can you not mount the exhausted dm snapshot?  If you mean data recovery 
or forensics beyond that, please elaborate.

I know I've had issues 'read-only' mounting ext3(2?3?4?) filesystems 
before (on actually read-only block devices).  Maybe some flags that 
I've forgotten get past that, but worst case you can always fake a 
writable device with a 2nd overlay/snapshot going to a writable device, 
i.e. if you wanted to let fsck repair things.

-dmc

Re: DM_snapshot_cow filesystem (dmsetup create snapshot)

[Frederick Grose]

[-- Attachment #1.1: Type: text/plain, Size: 3369 bytes --]

On Sun, Nov 27, 2011 at 8:08 PM, Douglas McClendon <
dmc.lists@filteredperception.org> wrote:

> On 11/27/2011 04:31 PM, Frederick Grose wrote:
>
>> On Sat, Oct 29, 2011 at 4:49 PM, Alasdair G Kergon <agk@redhat.com
>> <mailto:agk@redhat.com>> wrote:
>>
>>    markmc did some code that shows how to read the format a few years
>>    ago here:
>>    http://people.gnome.org/~**markmc/code/merge-dm-snapshot.**c<http://people.gnome.org/~markmc/code/merge-dm-snapshot.c>
>>    <http://people.gnome.org/%**7Emarkmc/code/merge-dm-**snapshot.c<http://people.gnome.org/%7Emarkmc/code/merge-dm-snapshot.c>
>> >
>>
>>
>>    Otherwise look at dm-snap-persistent.c in the kernel tree.
>>
>>    Alasdair
>>
>>
>> Users can easily exhaust a LiveUSB snapshot overlay by writing
>> too many changes to the OS filesystem, such as by performing
>> a yum update.
>>
>> Would such an 'exhausted' overlay be amenable to some sort of
>> data recovery or forensics?
>>
>> If so, how so; if not, why not?
>>
>
> Can you not mount the exhausted dm snapshot?  If you mean data recovery or
> forensics beyond that, please elaborate.
>
> I know I've had issues 'read-only' mounting ext3(2?3?4?) filesystems
> before (on actually read-only block devices).  Maybe some flags that I've
> forgotten get past that, but worst case you can always fake a writable
> device with a 2nd overlay/snapshot going to a writable device, i.e. if you
> wanted to let fsck repair things.
>
> -dmc


A typical scenario goes like this:

A LiveCD is loaded onto a USB device with a relatively small
persistent overlay and no separate home.img filesystem.  The user
proceeds to use the system and save some information.  They may
execute a df command and see that there is lots of space
available on the rootfs.  Then they decide to execute a yum
update or yum install when there is insufficient space available.
(They do not know about dmsetup status.)  Their session will now
not shutdown normally. dmsetup status now shows

live-rw: 0 8388608 snapshot Invalid.

The device fails to complete a subsequent boot attempt.  The
startup log shows

mount: /dev/mapper/live-rw: can't read superblock

e2fsck /dev/dm-0 shows

Buffer I/O error on device dm-0, logical block 0 (and 1, 2, 3, 0)
e2fsck: Attempt to read block from the filesystem resulted in
short read while trying to open /dev/dm-0
Could this be a zero-length partition?

I recreated the above scenario on Fedora-16-Live-Desktop
installed with an overlay-size-mb of 100.  After updating to ~70%
of the overlay, I wrote with
dd if=/dev/zero of=/boot/load bs=1M count=10
3 times to exhaust the overlay.

With the defective device mounted on a good system, and loop
devices set up for ext3fs.img and the overlay file,

dmsetup create item --table "0 8388608 snapshot 7:1 7:2 P 8"
reports
item: 0 8388608 snapshot Invalid

e2fsck /dev/dm-0 reports

e2fsck 1.41.14 (22-Dec-2010)
e2fsck: Attempt to read block from filesystem resulted in short
read while trying to open /dev/dm-0
Could this be a zero-length partition?

A hex editor view of the overlay file shows lots of content with
lots of zeros at the end of the file.

My questions are these:

Might there be a way to edit the overlay file to restore its
utility for the information written before the damaging write?

If so, how so; if not, why not?

Suitable references would do as well.

Thanks!       --Fred

[-- Attachment #1.2: Type: text/html, Size: 8034 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: DM_snapshot_cow filesystem (dmsetup create snapshot)

[Douglas McClendon]

On 11/29/2011 12:22 AM, Frederick Grose wrote:
> On Sun, Nov 27, 2011 at 8:08 PM, Douglas McClendon
> <dmc.lists@filteredperception.org
> <mailto:dmc.lists@filteredperception.org>> wrote:
>
>     On 11/27/2011 04:31 PM, Frederick Grose wrote:
>
>         On Sat, Oct 29, 2011 at 4:49 PM, Alasdair G Kergon
>         <agk@redhat.com <mailto:agk@redhat.com>
>         <mailto:agk@redhat.com <mailto:agk@redhat.com>>> wrote:
>
>             markmc did some code that shows how to read the format a few
>         years
>             ago here:
>         http://people.gnome.org/~__markmc/code/merge-dm-snapshot.__c
>         <http://people.gnome.org/~markmc/code/merge-dm-snapshot.c>
>         <http://people.gnome.org/%__7Emarkmc/code/merge-dm-__snapshot.c
>         <http://people.gnome.org/%7Emarkmc/code/merge-dm-snapshot.c>>
>
>
>             Otherwise look at dm-snap-persistent.c in the kernel tree.
>
>             Alasdair
>
>
>         Users can easily exhaust a LiveUSB snapshot overlay by writing
>         too many changes to the OS filesystem, such as by performing
>         a yum update.
>
>         Would such an 'exhausted' overlay be amenable to some sort of
>         data recovery or forensics?
>
>         If so, how so; if not, why not?
>
>
>     Can you not mount the exhausted dm snapshot?  If you mean data
>     recovery or forensics beyond that, please elaborate.
>
>     I know I've had issues 'read-only' mounting ext3(2?3?4?) filesystems
>     before (on actually read-only block devices).  Maybe some flags that
>     I've forgotten get past that, but worst case you can always fake a
>     writable device with a 2nd overlay/snapshot going to a writable
>     device, i.e. if you wanted to let fsck repair things.
>
>     -dmc
>
>
> A typical scenario goes like this:
>
> A LiveCD is loaded onto a USB device with a relatively small
> persistent overlay and no separate home.img filesystem.  The user
> proceeds to use the system and save some information.  They may
> execute a df command and see that there is lots of space
> available on the rootfs.  Then they decide to execute a yum
> update or yum install when there is insufficient space available.
> (They do not know about dmsetup status.)  Their session will now
> not shutdown normally. dmsetup status now shows
>
> live-rw: 0 8388608 snapshot Invalid.
>
> The device fails to complete a subsequent boot attempt.  The
> startup log shows

You probably know this, but note there is a reset_overlay (or similar) 
boot flag which I think causes the overlay/snapshot to be reset, to get 
a booting system.  So the rest of this is with the use-case in mind of 
trying to recover prior contents of the overlay/snapshot in a useful way.

>
> mount: /dev/mapper/live-rw: can't read superblock
>
> e2fsck /dev/dm-0 shows
>
> Buffer I/O error on device dm-0, logical block 0 (and 1, 2, 3, 0)
> e2fsck: Attempt to read block from the filesystem resulted in
> short read while trying to open /dev/dm-0
> Could this be a zero-length partition?
>
> I recreated the above scenario on Fedora-16-Live-Desktop
> installed with an overlay-size-mb of 100.  After updating to ~70%
> of the overlay, I wrote with
> dd if=/dev/zero of=/boot/load bs=1M count=10
> 3 times to exhaust the overlay.
>
> With the defective device mounted on a good system, and loop
> devices set up for ext3fs.img and the overlay file,
>
> dmsetup create item --table "0 8388608 snapshot 7:1 7:2 P 8"
> reports
> item: 0 8388608 snapshot Invalid

I don't know what that last line of output specifically means as far as 
subsequent expectations should go, though presumably others on this list 
can elaborate, or worst grep for the message in the code.

Maybe dmsetup is more cooperative with full overlays if they are created 
in read-only mode?(wild guess)

I.e. there is certainly the expectation that you won't be able to write 
to the device (or at a minimum to any chunk not already in the 
overlay(?)).  So as I mentioned, the way I'd go about it would be to 
recreate it (I'm taking for granted above cmdline is correct), but in 
read-only mode, then either just dup that to a new larger device, or add 
a second read-write snapshot layer on top (pointing to a looped tmpfile 
somewhere).  Then try to mount and/or fsck that.

But that presumes that the 'snapshot Invalid' isn't evidence that 
something is preventing even read-only access to the full snapshot.  For 
that I defer to the people more familiar with the relevent code.

>
> e2fsck /dev/dm-0 reports
>
> e2fsck 1.41.14 (22-Dec-2010)
> e2fsck: Attempt to read block from filesystem resulted in short
> read while trying to open /dev/dm-0
> Could this be a zero-length partition?
>
> A hex editor view of the overlay file shows lots of content with
> lots of zeros at the end of the file.
>
> My questions are these:
>
> Might there be a way to edit the overlay file to restore its
> utility for the information written before the damaging write?
>
> If so, how so; if not, why not?

Were it actually the case that full persistent snapshots become 
unusuable (short of 'editing'), that would seem like a bug needing to be 
fixed.

-dmc



>
> Suitable references would do as well.
>
> Thanks!       --Fred
>
>
>
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: DM_snapshot_cow filesystem (dmsetup create snapshot)

[Frederick Grose]

[-- Attachment #1.1: Type: text/plain, Size: 6547 bytes --]

On Tue, Nov 29, 2011 at 9:09 PM, Douglas McClendon <
dmc.lists@filteredperception.org> wrote:

> On 11/29/2011 12:22 AM, Frederick Grose wrote:
>
>> On Sun, Nov 27, 2011 at 8:08 PM, Douglas McClendon
>> <dmc.lists@filteredperception.**org <dmc.lists@filteredperception.org>
>> <mailto:dmc.lists@**filteredperception.org<dmc.lists@filteredperception.org>>>
>> wrote:
>>
>>    On 11/27/2011 04:31 PM, Frederick Grose wrote:
>>
>>        On Sat, Oct 29, 2011 at 4:49 PM, Alasdair G Kergon
>>        <agk@redhat.com <mailto:agk@redhat.com>
>>        <mailto:agk@redhat.com <mailto:agk@redhat.com>>> wrote:
>>
>>            markmc did some code that shows how to read the format a few
>>        years
>>            ago here:
>>        http://people.gnome.org/~__**markmc/code/merge-dm-snapshot.**__c<http://people.gnome.org/~__markmc/code/merge-dm-snapshot.__c>
>>        <http://people.gnome.org/~**markmc/code/merge-dm-snapshot.**c<http://people.gnome.org/~markmc/code/merge-dm-snapshot.c>
>> >
>>        <http://people.gnome.org/%__**7Emarkmc/code/merge-dm-__**
>> snapshot.c
>>
>>        <http://people.gnome.org/%**7Emarkmc/code/merge-dm-**snapshot.c<http://people.gnome.org/%7Emarkmc/code/merge-dm-snapshot.c>
>> >>
>>
>>
>>            Otherwise look at dm-snap-persistent.c in the kernel tree.
>>
>>            Alasdair
>>
>>
>>        Users can easily exhaust a LiveUSB snapshot overlay by writing
>>        too many changes to the OS filesystem, such as by performing
>>        a yum update.
>>
>>        Would such an 'exhausted' overlay be amenable to some sort of
>>        data recovery or forensics?
>>
>>        If so, how so; if not, why not?
>>
>>
>>    Can you not mount the exhausted dm snapshot?  If you mean data
>>    recovery or forensics beyond that, please elaborate.
>>
>>    I know I've had issues 'read-only' mounting ext3(2?3?4?) filesystems
>>    before (on actually read-only block devices).  Maybe some flags that
>>    I've forgotten get past that, but worst case you can always fake a
>>    writable device with a 2nd overlay/snapshot going to a writable
>>    device, i.e. if you wanted to let fsck repair things.
>>
>>    -dmc
>>
>>
>> A typical scenario goes like this:
>>
>> A LiveCD is loaded onto a USB device with a relatively small
>> persistent overlay and no separate home.img filesystem.  The user
>> proceeds to use the system and save some information.  They may
>> execute a df command and see that there is lots of space
>> available on the rootfs.  Then they decide to execute a yum
>> update or yum install when there is insufficient space available.
>> (They do not know about dmsetup status.)  Their session will now
>> not shutdown normally. dmsetup status now shows
>>
>> live-rw: 0 8388608 snapshot Invalid.
>>
>> The device fails to complete a subsequent boot attempt.  The
>> startup log shows
>>
>
> You probably know this, but note there is a reset_overlay (or similar)
> boot flag which I think causes the overlay/snapshot to be reset, to get a
> booting system.


Yes, the relevant code for Fedora is here,
http://git.kernel.org/?p=boot/dracut/dracut.git;a=blob;f=modules.d/90dmsquash-live/dmsquash-live-root;hb=HEAD#l101

dd if=/dev/zero of=$OVERLAY_LOOPDEV bs=64k count=1 2>/dev/null

although it seems to need an extra parameter, conv=notrunc
to prevent truncation of the overlay file.

 So the rest of this is with the use-case in mind of trying to recover
> prior contents of the overlay/snapshot in a useful way.
>
>
>
>> mount: /dev/mapper/live-rw: can't read superblock
>>
>> e2fsck /dev/dm-0 shows
>>
>> Buffer I/O error on device dm-0, logical block 0 (and 1, 2, 3, 0)
>> e2fsck: Attempt to read block from the filesystem resulted in
>> short read while trying to open /dev/dm-0
>> Could this be a zero-length partition?
>>
>> I recreated the above scenario on Fedora-16-Live-Desktop
>> installed with an overlay-size-mb of 100.  After updating to ~70%
>> of the overlay, I wrote with
>> dd if=/dev/zero of=/boot/load bs=1M count=10
>> 3 times to exhaust the overlay.
>>
>> With the defective device mounted on a good system, and loop
>> devices set up for ext3fs.img and the overlay file,
>>
>> dmsetup create item --table "0 8388608 snapshot 7:1 7:2 P 8"
>
> dmsetup status reports
>> item: 0 8388608 snapshot Invalid
>>
>
> I don't know what that last line of output specifically means as far as
> subsequent expectations should go, though presumably others on this list
> can elaborate, or worst grep for the message in the code.
>
> Maybe dmsetup is more cooperative with full overlays if they are created
> in read-only mode?(wild guess)
>

I tested this with --readonly added to the
dmsetup create command, both with and without a
-r parameter on the overlay loop device set up command.
Both resulted in the Invalid snapshot report from
dmsetup status.

I.e. there is certainly the expectation that you won't be able to write to
> the device (or at a minimum to any chunk not already in the overlay(?)).
>  So as I mentioned, the way I'd go about it would be to recreate it (I'm
> taking for granted above cmdline is correct), but in read-only mode, then
> either just dup that to a new larger device, or add a second read-write
> snapshot layer on top (pointing to a looped tmpfile somewhere).  Then try
> to mount and/or fsck that.
>

Attempting to mount the Invalid snapshot results in,
mount: /dev/mapper/item: can't read superblock

A second snapshot set up over the first also fails to mount
or e2fsck with the same errors.


> But that presumes that the 'snapshot Invalid' isn't evidence that
> something is preventing even read-only access to the full snapshot.


snapshot Invalid  does seem to prevent even read-only access.


> For that I defer to the people more familiar with the relevent code.
>
>
>
>> e2fsck /dev/dm-0 reports
>>
>> e2fsck 1.41.14 (22-Dec-2010)
>> e2fsck: Attempt to read block from filesystem resulted in short
>> read while trying to open /dev/dm-0
>> Could this be a zero-length partition?
>>
>> A hex editor view of the overlay file shows lots of content with
>> lots of zeros at the end of the file.
>>
>> My questions are these:
>>
>> Might there be a way to edit the overlay file to restore its
>> utility for the information written before the damaging write?
>>
>> If so, how so; if not, why not?
>>
>
> Were it actually the case that full persistent snapshots become unusuable
> (short of 'editing'), that would seem like a bug needing to be fixed.
>
> -dmc
>
> Suitable references would do as well.
>>
>> Thanks!       --Fred
>>
>

[-- Attachment #1.2: Type: text/html, Size: 10686 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: DM_snapshot_cow filesystem (dmsetup create snapshot)

[Alasdair G Kergon]

On Wed, Nov 30, 2011 at 04:48:05PM -0500, Frederick Grose wrote:
> snapshot Invalid  does seem to prevent even read-only access.

Indeed - once it's invalid, that's the end of it.
If we didn't do that, changes to areas of the origin that hadn't already
changed would start to appear in the 'snapshot' and the snapshot would become a
nonsensical hybrid of the two devices containing no information about which
parts of it got corrupted.

Don't allow your snapshots to run out of space!
If you want to see what you can recover, use that code I mentioned earlier.

Alasdair

Re: DM_snapshot_cow filesystem (dmsetup create snapshot)

[Douglas McClendon]

On 11/30/2011 04:03 PM, Alasdair G Kergon wrote:
> On Wed, Nov 30, 2011 at 04:48:05PM -0500, Frederick Grose wrote:
>> snapshot Invalid  does seem to prevent even read-only access.
>
> Indeed - once it's invalid, that's the end of it.
> If we didn't do that, changes to areas of the origin that hadn't already
> changed would start to appear in the 'snapshot' and the snapshot would become a
> nonsensical hybrid of the two devices containing no information about which
> parts of it got corrupted.
>
> Don't allow your snapshots to run out of space!

So wouldn't it be much better as far as the user is concerned if, the 
instant a snapshot reached capacity, it fell over into a read-only state 
(instead of being marked invalid?), such that further corruption beyond 
such plug-pulling variety could/would not occur?  If so, would it be 
straightforward to write such a patch and would it be beneficial enough 
to end upstream?

-dmc


> If you want to see what you can recover, use that code I mentioned earlier.
>
> Alasdair
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: DM_snapshot_cow filesystem (dmsetup create snapshot)

[Alasdair G Kergon]

On Wed, Nov 30, 2011 at 05:34:25PM -0600, Douglas McClendon wrote:
> So wouldn't it be much better as far as the user is concerned if, the  
> instant a snapshot reached capacity, it fell over into a read-only state  
> (instead of being marked invalid?), such that further corruption beyond  

That would mean your *origin* device becoming read-only as well.

It's been discussed before.  Nobody has been motivated to write the code,
as there's no need for it on a properly-managed system.

If you care enough about your snapshot contents, then surely you
would be monitoring it to make sure it doesn't fill up - and if you're
going to run out of space and can't expand, you would shut down your system
yourself in a *controlled* way *before* running out of space!

Alasdair

Re: DM_snapshot_cow filesystem (dmsetup create snapshot)

[Douglas McClendon]

On 12/01/2011 08:12 AM, Alasdair G Kergon wrote:
> On Wed, Nov 30, 2011 at 05:34:25PM -0600, Douglas McClendon wrote:
>> So wouldn't it be much better as far as the user is concerned if, the
>> instant a snapshot reached capacity, it fell over into a read-only state
>> (instead of being marked invalid?), such that further corruption beyond
>
> That would mean your *origin* device becoming read-only as well.

I apologize in that I know I may be getting some terminology confused, 
as devicemapper snapshots are used for many scenarios.  The fedora 
livecd/usb is perhaps only a subset of wider problem-constraints that 
I'm not keeping in mind.  That disclaimer said, in this particular 
situation, using the terminology I use in my own head, the base device 
here is a read-only ext3/4 filesystem image, necessarily read only 
because it is living on a squashfs filesystem, that may be on a 
read-only physical device like cdrom, or read-write (liveusb).

The overlay device in this situation is a read-write tmpfile in tmpfs, 
looped to be available as the dm-snapshot 'snapshot/overlay device'.

The resulting used device is then a read-write device used as a 
read-write ext3/4 filesystem for the rootfs.

When the snapshot fills up, keeping the snapshot device read-write would 
seem to serve no useful purpose.  And the 'base' device was read-only to 
begin with.

The real issue seems to be that the meaningfulness and usefulness of the 
overlay device seems to go from 'meaningful' to 'invalid/meaningless' 
the instant it fills up.  For no beneficial reason I can see (again 
maybe it complicates other dm-snapshot use-case scenarios that I'm not 
considering).


>
> It's been discussed before.  Nobody has been motivated to write the code,
> as there's no need for it on a properly-managed system.
>
> If you care enough about your snapshot contents, then surely you
> would be monitoring it to make sure it doesn't fill up - and if you're
> going to run out of space and can't expand, you would shut down your system
> yourself in a *controlled* way *before* running out of space!

In a typical livecd scenario, the amount of space in the snapshot can be 
very limited (say, half of ram).

In such a scenario, if the first thing I do is 'yum install lftp' I 
might be golden because that only takes a few MB.  If the next thing I 
do however is 'yum install emacs' or something else that brings in a ton 
of dependencies, that single command can exhaust the snapshot.

Are you suggesting some polling(or event driven?) method for the system 
to try to shutdown cleanly in the middle of that yum install?

I admit that would indeed solve the present use case of keeping the 
overlay device functionally usable.  In fact, going with a brutal 
virtual-plug-pulling halt triggered by snapshot full would be fine. 
Though I'd rather at that instant have the system go read-write-rootfs 
so that I can still poke around somewhat, rather than the screen go black.

But I really see no downside to keeping it usable without that 
pro-active infrastructure, by instead just making sure that when it 
fills up, it remains a 'valid, but full snapshot that can at least be 
utilized read-only'.  The alternate of letting it become invalid, such 
that the user can get the 'benefit' that if their current workload 
contains itself to chunks already managed in the snapshot, seems like no 
real benefit to me (maybe if a database was contained entirely within 
the snapshot chunks and the system was only fiddling with that).  Or 
perhaps I'm completely misunderstanding things (entirely possible)

-dmc



>
> Alasdair
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: DM_snapshot_cow filesystem (dmsetup create snapshot)

[Alasdair G Kergon]

On Thu, Dec 01, 2011 at 02:18:37PM -0600, Douglas McClendon wrote:
> The overlay device in this situation is a read-write tmpfile in tmpfs,  
> looped to be available as the dm-snapshot 'snapshot/overlay device'.

Ah - well if your base device is already read-only, then there is no
corruption and you should be able to edit the bit of the disk image
that says 'this image is invalid' and recover it read-only or extend it.

Alasdair

Re: DM_snapshot_cow filesystem (dmsetup create snapshot)

[Douglas McClendon]

On 12/01/2011 02:20 PM, Alasdair G Kergon wrote:
> On Thu, Dec 01, 2011 at 02:18:37PM -0600, Douglas McClendon wrote:
>> The overlay device in this situation is a read-write tmpfile in tmpfs,
>> looped to be available as the dm-snapshot 'snapshot/overlay device'.
>
> Ah - well if your base device is already read-only, then there is no
> corruption and you should be able to edit the bit of the disk image
> that says 'this image is invalid' and recover it read-only or extend it.

Can I get a commandline that summarizes how to do that 'edit'?

-dmc

Re: DM_snapshot_cow filesystem (dmsetup create snapshot)

[Douglas McClendon]

On 12/01/2011 02:20 PM, Alasdair G Kergon wrote:
> On Thu, Dec 01, 2011 at 02:18:37PM -0600, Douglas McClendon wrote:
>> The overlay device in this situation is a read-write tmpfile in tmpfs,
>> looped to be available as the dm-snapshot 'snapshot/overlay device'.
>
> Ah - well if your base device is already read-only, then there is no
> corruption and you should be able to edit the bit of the disk image
> that says 'this image is invalid' and recover it read-only or extend it.

The commandline to do the 'edit' was requested for users like Fred, and 
because I guess one day I might find myself wanting to do the same.

As per possibly avoiding the need for such a workaround, and thinking 
about the non-read-only base situation, does the following request sound 
at least logically consistent, and correct in its assumptions-

Currently in a dm-snapshot case such as the fedora liveusb, the instant 
a persistent snapshot becomes full, it is marked as invalid, and further 
accesses result in (fixme?).  But it would be preferable if instead of 
being marked invalid at the instant it fills up, it is instead marked as 
invalid at the first instant after it is full, and its base device gets 
written to.  And beyond that as a second change from the current state 
of things, the virtual device using the overlay falls into a read-only 
state the instant the overlay it is using reaches a full state. 
(instead of as happens now ?? maybe selective write failures based on 
whether or not the write destination is in a chunk within the full 
overlay ??)

Thus in read-only base cases, the overlay never gets marked as invalid, 
and no users have any need for a workaround tool or commandline to flip 
the invalid bit(?) to valid, in order to mount and use/recover the data 
present in the snapshot.

-dmc

Re: DM_snapshot_cow filesystem (dmsetup create snapshot)

[Alasdair G Kergon]

On Thu, Dec 01, 2011 at 09:44:53PM -0600, Douglas McClendon wrote:
> The commandline to do the 'edit' was requested for users like Fred, and  
> because I guess one day I might find myself wanting to do the same.

This is unsupported, but if you want to try:
You need to flip the 5th byte of the 'cow' device from 0 to 1 (when
the snapshot is not active).  (But you might have lost the most recent
changes if they weren't flushed out when it filled up.)

Something like this:

# lvs
  LV    VG   Attr   LSize  Origin Snap%  Move Log Copy%  Convert
  lvol0 vg2  owi-a- 10.00m
  lvol1 vg2  Swi-I- 10.00m lvol0  100.00

# dmsetup table vg2-lvol1-cow
0 20480 linear 7:0 22528

# lvchange -an vg2/lvol0

# dmsetup create tmp-vg2-lvol1-cow --table "0 20480 linear 7:0 22528"

# hexdump -C /dev/mapper/tmp-vg2-lvol1-cow | head -1
00000000  53 6e 41 70 00 00 00 00  01 00 00 00 08 00 00 00  |SnAp............|

Flip byte 5, e.g. copy to file, edit that bit, copy back.
  # dd if=/dev/mapper/tmp-vg2-lvol1-cow of=/tmp/cow-start bs=512 count=1
  # vim -b /tmp/cow-start   (and/or xxd)
  # dd if=/tmp/cow-start of=/dev/mapper/tmp-vg2-lvol1-cow  

# dmsetup remove tmp-vg2-lvol1-cow

# lvchange -ay vg2/lvol0

# lvs
  LV    VG   Attr   LSize  Origin Snap%  Move Log Copy%  Convert
  lvol0 vg2  owi-a- 10.00m                                      
  lvol1 vg2  swi-a- 10.00m lvol0   10.08                        

# hexdump -C /dev/mapper/vg2-lvol1-cow | head -1
00000000  53 6e 41 70 01 00 00 00  01 00 00 00 08 00 00 00  |SnAp............|

Alasdair

Re: DM_snapshot_cow filesystem (dmsetup create snapshot)

[Frederick Grose]

[-- Attachment #1.1: Type: text/plain, Size: 6113 bytes --]

On Fri, Dec 2, 2011 at 3:46 AM, Alasdair G Kergon <agk@redhat.com> wrote:

> On Thu, Dec 01, 2011 at 09:44:53PM -0600, Douglas McClendon wrote:
> > The commandline to do the 'edit' was requested for users like Fred, and
> > because I guess one day I might find myself wanting to do the same.
>
> This is unsupported, but if you want to try:
> You need to flip the 5th byte of the 'cow' device from 0 to 1 (when
> the snapshot is not active).  (But you might have lost the most recent
> changes if they weren't flushed out when it filled up.)
>
> Something like this:
>
> # lvs
>  LV    VG   Attr   LSize  Origin Snap%  Move Log Copy%  Convert
>  lvol0 vg2  owi-a- 10.00m
>  lvol1 vg2  Swi-I- 10.00m lvol0  100.00
>
> # dmsetup table vg2-lvol1-cow
> 0 20480 linear 7:0 22528
>
> # lvchange -an vg2/lvol0
>
> # dmsetup create tmp-vg2-lvol1-cow --table "0 20480 linear 7:0 22528"
>
> # hexdump -C /dev/mapper/tmp-vg2-lvol1-cow | head -1
> 00000000  53 6e 41 70 00 00 00 00  01 00 00 00 08 00 00 00
>  |SnAp............|
>
> Flip byte 5, e.g. copy to file, edit that bit, copy back.
>  # dd if=/dev/mapper/tmp-vg2-lvol1-cow of=/tmp/cow-start bs=512 count=1
>  # vim -b /tmp/cow-start   (and/or xxd)
>  # dd if=/tmp/cow-start of=/dev/mapper/tmp-vg2-lvol1-cow
>
> # dmsetup remove tmp-vg2-lvol1-cow
>
> # lvchange -ay vg2/lvol0
>
> # lvs
>  LV    VG   Attr   LSize  Origin Snap%  Move Log Copy%  Convert
>  lvol0 vg2  owi-a- 10.00m
>  lvol1 vg2  swi-a- 10.00m lvol0   10.08
>
> # hexdump -C /dev/mapper/vg2-lvol1-cow | head -1
> 00000000  53 6e 41 70 01 00 00 00  01 00 00 00 08 00 00 00
>  |SnAp............|
>
> Alasdair
>

Thank you Alasdair and Douglas for your attention to this matter,
the recovery succeeded on my simple test case.

Some reports follow:

I edited the 'exhausted', 100 MiB overlay file from my F16-Live test
case, which had been made Invalid by attempting to write beyond its
capacity.

After
# dmsetup create item --table "0 8388608 snapshot 7:1 7:2 P 8" --readonly

# dmsetup status
item: 0 8388608 snapshot 202336/204800 800

# mount /dev/dm-0 /mnt/b/
mount: block device /dev/mapper/item is write-protected, mounting read-only
mount: wrong fs type, bad option, bad superblock on /dev/mapper/item,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

# dmesg |tail
[  772.461469] EXT4-fs (dm-0): INFO: recovery required on readonly
filesystem
[  772.461476] EXT4-fs (dm-0): write access unavailable, cannot proceed

# e2fsck -n /dev/dm-0

e2fsck 1.41.14 (22-Dec-2010)
Warning: skipping journal recovery because doing a read-only filesystem
check.
_Fedora-16-x86_6 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inodes that were part of a corrupted orphan linked list found.  Fix? no

Inode 17442 was part of the orphaned inode list.  IGNORED.
Deleted inode 156288 has zero dtime.  Fix? no

Inode 162164 was part of the orphaned inode list.  IGNORED.
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  -(129696--129702) -(555921--557003)
-(557006--557016) -(557020--557050) -557058 -(557632--557663) -559125
-560586 -(565664--565759) -(566784--566911) -(566973--566991)
-(567016--567023) -(567032--567039) -(567044--567051) -(567068--567159)
-567164 -(567168--567183) -(567188--567189) -(567200--567279)
-(567284--567285) -(567288--567406) -(567408--567677) -(567680--567798)
-(567800--567807) -567839 -(567856--567912) -(567916--567919) -567960
-567967 -580765 -(580909--580911) -580920 -(580990--580991)
-(581024--581036) -(581040--581127) -592079 -606092 -(608256--608261)
-608264 -608290 -(608313--608314) -608317 -(608320--608322) -608361 -608376
-608400 -608421 -608767 -(608770--608773) -608775 -608777 -608779
-(608783--608784) -(608797--608807) -608809 -608813 -608816
-(608825--608831) -608895 -608914 -(608932--608943) -609054 -609077 -609147
-609167 -609195 -609203 -609223 -(609322--609323) -(609542--609543)
-(609558--609567) -(609584--609586) -(609666--609711) -609716
-(609718--609730) -(609736--609939) -609965 -(640485--640556)
Fix? no

Free blocks count wrong (381446, counted=375520).
Fix? no

Inode bitmap differences:  -17442 -156288 -162164
Fix? no

Free inodes count wrong (165564, counted=165454).
Fix? no


_Fedora-16-x86_6: ********** WARNING: Filesystem still has errors **********

_Fedora-16-x86_6: 96580/262144 files (0.3% non-contiguous), 667130/1048576
blocks


After removing and recreating the snapshot as read-write,

# e2fsck -f -y /dev/dm-0

e2fsck 1.41.14 (22-Dec-2010)
_Fedora-16-x86_6: recovering journal
Clearing orphaned inode 162164 (uid=0, gid=0, mode=0100644, size=10898776)
Clearing orphaned inode 17442 (uid=0, gid=0, mode=0100755, size=27520)
Clearing orphaned inode 156288 (uid=0, gid=0, mode=0100755, size=291672)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

_Fedora-16-x86_6: ***** FILE SYSTEM WAS MODIFIED *****
_Fedora-16-x86_6: 96688/262144 files (0.3% non-contiguous), 672875/1048576
blocks

# e2fsck -p /dev/dm-0

_Fedora-16-x86_6: clean, 96688/262144 files, 672875/1048576 blocks

# dmsetup status

item: 0 8388608 snapshot 202424/204800 800


Subsequent mounting (read-only for safety) succeeded in showing the
preexisting content.


I agree with Douglas that there is a significant class of non-technical
Live USB users who would benefit from a fail safe mechanism for their
content based in overlays of a read-only origin.

For example, students and teachers using the Sugar on a Stick Spin,

http://spins.fedoraproject.org/soas/

have been frustrated when their portfolio of work has been 'lost'
after a couple of months of classwork.

We are working to provide better file system support, viz.,

http://wiki.sugarlabs.org/go/LiveOS_image

and any Device-mapper support would be much appreciated!

       --Fred

[-- Attachment #1.2: Type: text/html, Size: 12172 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: DM_snapshot_cow filesystem (dmsetup create snapshot)

[Douglas McClendon]

correction- "go read-write-rootfs' below should have been "go 
read-only-rootfs"

-dmc

--------

I admit that would indeed solve the present use case of keeping the 
overlay device functionally usable.  In fact, going with a brutal 
virtual-plug-pulling halt triggered by snapshot full would be fine. 
Though I'd rather at that instant have the system go read-write-rootfs 
so that I can still poke around somewhat, rather than the screen go black.