[dm-devel] can we fix some dm snapshot crashes please? :)

[dm-devel] can we fix some dm snapshot crashes please? :)

[Michael Tokarev]

Hi!

I asked this 1.5 years ago, but go no replies.

The thing is that we have a few bugs in kernel in snapshot-related area
for years, maybe for decades, and the bugs are still there and it is still
trivial to crash the kernel is you're not very-very-very careful.

This might be due to user error ofcourse, but this still is definitely
NOT a valid reason for the crashes.

For example, here it goes for snapshot-origin dm target. Just create
a snapshot-origin and do some activity on it, it is a 2-line reproducer:

  # first create a test device: it can be anything
  # here we use a 100-MB loop device
  truncate --size=100M base
  losetup /dev/loop0 base
  # now create the snapshot-origin on it
  sz=$(blockdev --getsize /dev/loop0)
  dmsetup create base --table "0 $sz snapshot-origin /dev/loop0"
  # and now the crash
  mkfs.ext4 /dev/mapper/base

This crashes instantly. Different kernels crashed a bit differently,
I tried even some 3.x kernels. But the result is the same - crash.

Here's an example from 5.10 kernel:

[   89.661594] ------------[ cut here ]------------
[   89.663789] kernel BUG at block/bio.c:1473!
[   89.665624] invalid opcode: 0000 [#1] SMP PTI
[   89.669108] CPU: 0 PID: 264 Comm: mkfs.ext4 Not tainted 5.10.0-6-amd64 #1 Debian 5.10.28-1
[   89.672551] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-2 04/01/2014
[   89.675987] RIP: 0010:bio_split+0x74/0x80
[   89.677744] Code: 89 c7 e8 ff 5e 03 00 41 8b 74 24 28 48 89 ef e8 e2 f5 ff ff f6 45 15 01 74 08 66 41 81 4c 24 14 00 01 4c 89 e0 5b 5d 41 5c c3 
<0f> 0b 0f 0b 0f 0b 45 31 e4 eb ed 90 0f 1f 44 00 00 39 77 28 76 05
[   89.686181] RSP: 0018:ffffb248c026bb30 EFLAGS: 00010246
[   89.688347] RAX: 0000000000000008 RBX: 0000000000000000 RCX: ffff8bd5025f7d80
[   89.691201] RDX: 0000000000000c00 RSI: 0000000000000000 RDI: ffff8bd502031780
[   89.694089] RBP: 0000000000000000 R08: 00000019a1b717a8 R09: 0000000000000000
[   89.696975] R10: ffff8bd5341fc600 R11: ffff8bd5341fc658 R12: ffff8bd5024a0558
[   89.699935] R13: ffff8bd5024a0000 R14: ffff8bd502031780 R15: ffff8bd502383c80
[   89.704128] FS:  00007f0a33236780(0000) GS:ffff8bd53ea00000(0000) knlGS:0000000000000000
[   89.707641] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   89.710162] CR2: 00007f0a31db2000 CR3: 0000000002ab4000 CR4: 00000000000006f0
[   89.713071] Call Trace:
[   89.714247]  dm_submit_bio+0x35d/0x440 [dm_mod]
[   89.716230]  submit_bio_noacct+0xf8/0x420
[   89.719430]  ? bio_add_page+0x62/0x90
[   89.721073]  submit_bh_wbc+0x16a/0x190
[   89.722702]  __block_write_full_page+0x1fa/0x460
[   89.724767]  ? bdev_evict_inode+0xc0/0xc0
[   89.726523]  ? block_invalidatepage+0x150/0x150
[   89.728446]  __writepage+0x17/0x60
[   89.730010]  write_cache_pages+0x186/0x3d0
[   89.731789]  ? __wb_calc_thresh+0x120/0x120
[   89.734842]  generic_writepages+0x4c/0x80
[   89.736830]  do_writepages+0x34/0xc0
[   89.738390]  ? __fsnotify_parent+0xe7/0x2d0
[   89.740196]  __filemap_fdatawrite_range+0xc5/0x100
[   89.742245]  file_write_and_wait_range+0x61/0xb0
[   89.744219]  blkdev_fsync+0x17/0x40
[   89.745777]  __x64_sys_fsync+0x34/0x60
[   89.747446]  do_syscall_64+0x33/0x80
[   89.749341]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   89.753171] RIP: 0033:0x7f0a320b37a0
[   89.754777] Code: 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 83 3d 69 cf 2b 00 00 75 10 b8 4a 00 00 00 0f 05 
<48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 4e 3d 01 00 48 89 04 24
[   89.762134] RSP: 002b:00007ffdb7a55158 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
[   89.765245] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f0a320b37a0
[   89.770073] RDX: 0000000000000400 RSI: 000055797fd3e620 RDI: 0000000000000003
[   89.773025] RBP: 000055797fd3e510 R08: 0000000004800800 R09: 00007f0a32e08c40
[   89.775864] R10: 0000000004800800 R11: 0000000000000246 R12: 00007ffdb7a551c0
[   89.778680] R13: 00007ffdb7a551c8 R14: 000055797fd3e2e0 R15: 0000000000000000
[   89.781984] Modules linked in: dm_snapshot dm_bufio dm_mod loop hid_generic usbhid hid uhci_hcd ehci_hcd virtio_net net_failover ppdev failover 
joydev usbcore parport_pc psmouse evdev pcspkr serio_raw parport ata_generic floppy i2c_piix4 virtio_pci sg usb_common ata_piix virtio_ring virtio 
button qemu_fw_cfg ip_tables x_tables autofs4 crc32c_generic ext4 crc16 mbcache jbd2 sd_mod t10_pi crc_t10dif crct10dif_common ahci libahci libata 
scsi_mod
[   89.799301] ---[ end trace 1aa4a3cf509dc9b7 ]---

(block/bio.c:1473 is this:
struct bio *bio_split(struct bio *bio, int sectors,
                       gfp_t gfp, struct bio_set *bs)
{
         struct bio *split;

         BUG_ON(sectors <= 0);                   <==== here
         BUG_ON(sectors >= bio_sectors(bio));
)

I don't know what the problem with this is, but the kernel should
definitely not crash this easy, taking down whole system with it...

The same happens when at least one snapshot target is also associated
with this loop device, or when snapshot-origin is created on another
dm target instead of a loop device directly.

Can we finally, after so many years, fix at least the most grave
issues here, please? :)

Thanks!

/mjt

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] can we fix some dm snapshot crashes please? :)

[Zdenek Kabelac]

[-- Attachment #1: Type: text/plain, Size: 1407 bytes --]

Dne 07. 05. 21 v 12:31 Michael Tokarev napsal(a):
> Hi!
> 
> I asked this 1.5 years ago, but go no replies.
> 
> The thing is that we have a few bugs in kernel in snapshot-related area
> for years, maybe for decades, and the bugs are still there and it is still
> trivial to crash the kernel is you're not very-very-very careful.
> 
> This might be due to user error ofcourse, but this still is definitely
> NOT a valid reason for the crashes.
> 
> For example, here it goes for snapshot-origin dm target. Just create
> a snapshot-origin and do some activity on it, it is a 2-line reproducer:
> 
>   # first create a test device: it can be anything
>   # here we use a 100-MB loop device
>   truncate --size=100M base
>   losetup /dev/loop0 base
>   # now create the snapshot-origin on it
>   sz=$(blockdev --getsize /dev/loop0)
>   dmsetup create base --table "0 $sz snapshot-origin /dev/loop0"
>   # and now the crash
>   mkfs.ext4 /dev/mapper/base
> 
> This crashes instantly. Different kernels crashed a bit differently,
> I tried even some 3.x kernels. But the result is the same - crash.
> 
> Here's an example from 5.10 kernel:

Hi


Yes reproducible - can you please open BZ report here:

https://bugzilla.redhat.com/enter_bug.cgi?product=LVM%20and%20device-mapper

I assume you are aware you are trying to using snapshot target in wrong way,
but it should not be crashing kernel.

Regards

Zdeneek

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: zkabelac.vcf --]
[-- Type: text/x-vcard; name="zkabelac.vcf", Size: 4 bytes --]

null

[-- Attachment #3: Type: text/plain, Size: 97 bytes --]

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] can we fix some dm snapshot crashes please? :)

[Michael Tokarev]

07.05.2021 15:31, Zdenek Kabelac wrote:
> Dne 07. 05. 21 v 12:31 Michael Tokarev napsal(a):
...
>>   sz=$(blockdev --getsize /dev/loop0)
>>   dmsetup create base --table "0 $sz snapshot-origin /dev/loop0"
>>   # and now the crash
>>   mkfs.ext4 /dev/mapper/base

> Yes reproducible - can you please open BZ report here:
> 
> https://bugzilla.redhat.com/enter_bug.cgi?product=LVM%20and%20device-mapper

Ok, will do, thanks!

> I assume you are aware you are trying to using snapshot target in wrong way,
> but it should not be crashing kernel.

Actually I'm not aware about the wrong way. The documentation is a bit lacking
in this area. There are a few (maybe 2 or 3) guides describing the steps to do,
most involves first using an 1:1 linear table, next replacing it on-the-fly with
snapshot-origin table, but nowhere it is told that I can't create snapshot-origin
directly, or that I need at least one snapshot device, or that the origin of the
snapshot-origin table should be a dm device, or many many other variations.

I just tried to create a snapshot to be able to revert changes if something goes
wrong, and since I don't use lvm I thought that plain dmsetup should do, but
discovered that it doesn't quite work and it actually is a dangerous thing to use.. :)

Thanks!

/mjt


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] can we fix some dm snapshot crashes please? :)

[Mike Snitzer]

On Fri, May 07 2021 at 10:10P -0400,
Michael Tokarev <mjt@tls.msk.ru> wrote:

> 07.05.2021 15:31, Zdenek Kabelac wrote:
> > Dne 07. 05. 21 v 12:31 Michael Tokarev napsal(a):
> ...
> > >   sz=$(blockdev --getsize /dev/loop0)
> > >   dmsetup create base --table "0 $sz snapshot-origin /dev/loop0"
> > >   # and now the crash
> > >   mkfs.ext4 /dev/mapper/base
> 
> > Yes reproducible - can you please open BZ report here:
> > 
> > https://bugzilla.redhat.com/enter_bug.cgi?product=LVM%20and%20device-mapper
> 
> Ok, will do, thanks!

Thanks.  But when you do, please tone down the lamenting of how slow
snapshot crashes have been fixed -- or don't.  But that line of
rhetoric shows you're full of it.

> > I assume you are aware you are trying to using snapshot target in wrong way,
> > but it should not be crashing kernel.
> 
> Actually I'm not aware about the wrong way. The documentation is a bit lacking
> in this area. There are a few (maybe 2 or 3) guides describing the steps to do,
> most involves first using an 1:1 linear table, next replacing it on-the-fly with
> snapshot-origin table, but nowhere it is told that I can't create snapshot-origin
> directly, or that I need at least one snapshot device, or that the origin of the
> snapshot-origin table should be a dm device, or many many other variations.
> 
> I just tried to create a snapshot to be able to revert changes if something goes
> wrong, and since I don't use lvm I thought that plain dmsetup should do, but
> discovered that it doesn't quite work and it actually is a dangerous thing to use.. :)

Yes, it is dangerous to stab in the dark like you clearly are doing.

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] can we fix some dm snapshot crashes please? :)

[Michael Tokarev]

07.05.2021 18:20, Mike Snitzer wrote:
> On Fri, May 07 2021 at 10:10P -0400,
> Michael Tokarev <mjt@tls.msk.ru> wrote:
> 
>> 07.05.2021 15:31, Zdenek Kabelac wrote:
>>> Dne 07. 05. 21 v 12:31 Michael Tokarev napsal(a):
>> ...
>>>>    sz=$(blockdev --getsize /dev/loop0)
>>>>    dmsetup create base --table "0 $sz snapshot-origin /dev/loop0"
>>>>    # and now the crash
>>>>    mkfs.ext4 /dev/mapper/base
>>
>>> Yes reproducible - can you please open BZ report here:
>>>
>>> https://bugzilla.redhat.com/enter_bug.cgi?product=LVM%20and%20device-mapper
>>
>> Ok, will do, thanks!
> 
> Thanks.  But when you do, please tone down the lamenting of how slow
> snapshot crashes have been fixed -- or don't.  But that line of
> rhetoric shows you're full of it.

That's a joke really - note the smile at the end of my statement.

However this is true, - this particular crash has been here for a long time,
someone else asked about it earlier too, and I asked about it here 1.5 years
ago when trying snapshots the previous time (actually when just trying to
_understand_ how to do it properly).

Not really, - it is just a joke, and I know full well how it all is done.

..
> Yes, it is dangerous to stab in the dark like you clearly are doing.

I'd LOVE to know how to do it properly. Unfortunately I found no
information about it. There's one usage scenario outlined in a few
places which talks about the topic, all are doing it similar to how
LVM is doing it. But the thing is that there are actually numerous
usage scenarios for the feature, it definitely can be used numerous
ways, - the few mentions of dm snapshots offers one particular scenario
but does not show even the basic rules. There's an email (on this list
IIRC) from about a decade ago which defines a similar scenario and which -
apparently - is the only actual source of information about the whole
thing, and it again does not describe the game _rules_ :)

What you see as "I'm full of it" is a laugh actually - there's nether
a defined rules (for which many users asked in various places), and
there are crashes, and each time one points to a crash someone points
to "you're using a wrong rules". Which turns into a dead circle... :)

My actual intention is two-fold: - besides trying to make the thing
less dangerous, I'd *love* to make some more clear docs, describing
not only one particular scenario but the actual rules, so to say.

There's a significant gap between developers understanding how this
layer and this particular feature of this layer works, and the rest
of the world. Somehow this gap seems to be a bit more significant
than it is for many other areas (I'm a (positive! :) ) sysadmin and
system developer for 20+ years and know how things are working).

Just to give an example: I for one was puzzled by dmsetup reload -
why it is not actually replacing the table *and* returning success??
I started digging into kernel sources and doing some experiments,
and just by a chance discovered that the table *will* be replaced
by the next "dmsetup resume", - even for a non-suspended device!
This might be an obvious as an air thing for people who know some
internals, but it is non-obvious even for some users doing things
in this area for years. I asked several friends of mine who worked
with dmsetup and definitely knows it - no, they never *tried* to
reload a table without explicit suspend-resume, so the operation
is always 3-fold instead of single.  While the docs mentions it
somehow, one should have the question or a guess to notice or to
find this in the docs. Also the docs mentions, like, "should be
be suspended or else something can go wrong if the device is in
active use" - note the "if". So I assumed that nothing can go
wrong if the device is not in use. And finally, maybe this is
just a somewhat-uneasily designed software when it requires the
same 3 operations for a single operation?

Back to this snapshot-origin thing, to its *usage*. The usual
example given in a few places says:

  1. create a linear 1:1 mapping of the base volume
  2. mount it
  3. create a copy of the mapping from step 1
  4. suspend the mapping from step 1
  5. create snapshot device of the device from step 3
  6. *replace* the original mapping from step 1 with snapshot-origin
     device based on the mapping from step 3
  7. resume the snapshot-origin device
  8. the result is that the filesystem mounted in step 2
     is now based on the snapshot-origin instead of the linear mapping.

(I can have something backwards or slightly wrong but the concept
should be the same).

This gives tons of questions. The first question is why we need
to *replace* the table and why we need to mount it before replacing?
Why can't we mount (if it needs to be mounted) snapshot-origin device
(if that's the thing which gets mounted in the end) instead of using
snapshot-origin directly?

...hmm. After re-reading the docs in the gentoo wiki, I think I see
the issue in my (mis)understanding, even if it is not explicitly clear.
It *might* be the snapshot-origin is not a separate device which is useful
by its own, but is a "side-device", a "marker" to detect writes to the
base/origin device for one or more snapshots?? And the mount should be
done of the original linear mapping created in step 1?

If that's the case, please count me twice.. because 1.5 years ago
when I tried this for the first time, I made the same exact mistake.
Because it is still unclear to me how it is actually used...

I'll try it a bit later...

Maybe my confusion is not sufficient to refine the docs, I dunno..
But this is an example of an attempted usage after actually *trying*
to understand how it works... twice :)

Thanks!

/mjt


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel