Device-mapper for write monitoring

Device-mapper for write monitoring

[Michael Keller]

Hello,

I would like to ask the dm community whether our virtual block device
driver would be better implemented as a dm target; whether this is in-line
with the goals of the dm architecture, or constitutes abuse; and what
exactly would be the gain for us to do so.


For our block-based incremental backup software we need to monitor all
writes to a device, and report them to our user mode application for
further processing. Our plan is to add a virtual block device of our own
"on top" of every block device to be monitored. Our driver then intercepts
all writes to this virtual device, executes the appropriate logic, and
eventually passes it on to the underlying device.

Should we consider writing this as a dm target?

Some points that came up in discussions with other folks:
1. The "mapping" part itself is really trivial - it's an identity mapping.
Does this mean, we are really abusing the dm infrastructure?
2. What are the actual gains from writing a dm target over writing a
dedicated driver? The following comes to mind: writing less setup code, a
user space library, tools (dmsetup).
3. Are the interfaces of the device mapper relatively stable? (Assume that
our target is not general enough to be included into mainline, and we'll
have to do the API chasing ourselves.)

I was also hoping that the device-mapper can somehow help us with
monitoring the device hosting the root filesystem. Device-mapper apparently
comes with some special support, which allows it to, e.g., mount root on a
logical volume. Does this work for any device-mapper targets, or just for
inbuilt ones? Where do you store the configuration information (dm tables)
anyway, so that it's accessible before root is mounted?

Thank you very much,
Michael

Re: Device-mapper for write monitoring

[malahal]

Michael Keller [MKELLER@il.ibm.com] wrote:
> For our block-based incremental backup software we need to monitor all
> writes to a device, and report them to our user mode application for
> further processing. Our plan is to add a virtual block device of our own
> "on top" of every block device to be monitored. Our driver then intercepts
> all writes to this virtual device, executes the appropriate logic, and
> eventually passes it on to the underlying device.
> 
> Should we consider writing this as a dm target?
> 
> Some points that came up in discussions with other folks:
> 1. The "mapping" part itself is really trivial - it's an identity mapping.
> Does this mean, we are really abusing the dm infrastructure?

Don't think so. There are targets that have near 'identity mapping'
(dm-delay) and multipath target will have 'identity mapping' too, apart
from managing multiple paths!

In general, dm targets process the I/O without the help of any user land
code for performance.

> 2. What are the actual gains from writing a dm target over writing a
> dedicated driver? The following comes to mind: writing less setup code, a
> user space library, tools (dmsetup).

Yes. But then you will also get all the limitations/unimplemented
features of device-mapper framework too. E.g. barrier support comes to
mind! This is just an example, the simple barrier support might go to
upstream before you actually begin using your target though.

> 3. Are the interfaces of the device mapper relatively stable? (Assume that
> our target is not general enough to be included into mainline, and we'll
> have to do the API chasing ourselves.)

I don't think API changes, if any, give you a big trouble.

> I was also hoping that the device-mapper can somehow help us with
> monitoring the device hosting the root filesystem. Device-mapper apparently
> comes with some special support, which allows it to, e.g., mount root on a
> logical volume. Does this work for any device-mapper targets, or just for
> inbuilt ones? Where do you store the configuration information (dm tables)
> anyway, so that it's accessible before root is mounted?

As far as I know, "dm tables" are not stored persistently by the device
mapper tools. LVM is the one that stores and uses it from
initramfs/initrd. You will have to write your own tools to store the
configuration and use it. These tools will have to be included in the
initramfs/initrd for your thing to work on root device. You may need to
reserve some space on the device to store your configuration!

Re: Device-mapper for write monitoring

[Michael Keller]

> Re: [dm-devel] Device-mapper for write monitoring
> malahal
> to:
> dm-devel
> 19/02/2009 16:33

[snip] (and thanks for your answers!)

> > I was also hoping that the device-mapper can somehow help us with
> > monitoring the device hosting the root filesystem. Device-mapper
apparently
> > comes with some special support, which allows it to, e.g., mount root
on a
> > logical volume. Does this work for any device-mapper targets, or just
for
> > inbuilt ones? Where do you store the configuration information (dm
tables)
> > anyway, so that it's accessible before root is mounted?
>
> As far as I know, "dm tables" are not stored persistently by the device
> mapper tools. LVM is the one that stores and uses it from
> initramfs/initrd. You will have to write your own tools to store the
> configuration and use it. These tools will have to be included in the
> initramfs/initrd for your thing to work on root device. You may need to
> reserve some space on the device to store your configuration!

This got me a bit scared. To confirm, I unpacked my initrd and looked at
init. There is indeed no indication of the device-mapper trying to
reconstruct the correct tree, rather it is (as you say) all LVM:

lvm vgscan --ignorelockingfailure
lvm vgchange -ay --ignorelockingfailure  VolGroup00
resume /dev/VolGroup00/LogVol01
mkrootdev -t ext3 -o defaults,ro /dev/VolGroup00/LogVol00

But now suppose, I added a simple dm-linear device in between LVM and the
physical device - so to speak, a glass sheet on top of the physical volume.
What would happen? I tried something similar outside the boot process: I
put a dm-delay device on top of /dev/sda1 (which is a PV). The
resulting /dev/mapper/delayed has no users, and obviously the logical
volumes are not suddenly magically going through this device. On the other
hand, when I did a pvscan, it would now report to me a physical volume
called /dev/mapper/delayed, rather than /dev/sda1! I suppose from LVM's
point of view, it just finds two identical devices (two devices with
identical LVM metadata), and chooses to display ... well, one of them.

This means that in the boot process, to support a similar setup, I'd need
to create my device first, and then somehow instruct 'lvm vgscan' to make
the correct choice. I think /etc/lvm/lvm.conf (included in initrd) allows
me to do so. At this point this is clearly becoming an LVM question, rather
than a device-mapper question.

Unless I missed something?

Thanks,
Michael

Re: Device-mapper for write monitoring

[malahal]

Michael Keller [MKELLER@il.ibm.com] wrote:
> 
> > As far as I know, "dm tables" are not stored persistently by the device
> > mapper tools. LVM is the one that stores and uses it from
> > initramfs/initrd. You will have to write your own tools to store the
> > configuration and use it. These tools will have to be included in the
> > initramfs/initrd for your thing to work on root device. You may need to
> > reserve some space on the device to store your configuration!
> 
> But now suppose, I added a simple dm-linear device in between LVM and the
> physical device - so to speak, a glass sheet on top of the physical volume.
> What would happen? I tried something similar outside the boot process: I
> put a dm-delay device on top of /dev/sda1 (which is a PV). The
> resulting /dev/mapper/delayed has no users, and obviously the logical
> volumes are not suddenly magically going through this device. On the other
> hand, when I did a pvscan, it would now report to me a physical volume
> called /dev/mapper/delayed, rather than /dev/sda1! I suppose from LVM's
> point of view, it just finds two identical devices (two devices with
> identical LVM metadata), and chooses to display ... well, one of them.

If your LVM used /dev/mapper/delayed device as a PV, that amounts to
inserting dm-delayed between LVM and the "physical volume". By default
LVM scans all block devices in /dev. You should add "filter" rules so
that LVM doesn't use /dev/sda1 but /dev/mapper/delayed. You will need
code to make it persistent though -- either LVM changes or new tools!

> This means that in the boot process, to support a similar setup, I'd need
> to create my device first, and then somehow instruct 'lvm vgscan' to make
> the correct choice. I think /etc/lvm/lvm.conf (included in initrd) allows
> me to do so. At this point this is clearly becoming an LVM question, rather
> than a device-mapper question.
> 
> Unless I missed something?

You are right! Look at multipath tools. Multipath tools also use device
mapper framework. People use LVM on top of multipath devices. Multipath
tools use device identification and don't need to store any metadata on
the devices themselves. You could probably use similar method depending
on what you want to do.

Thanks, Malahal.

Re: Device-mapper for write monitoring

[Alasdair G Kergon]

On Thu, Feb 19, 2009 at 10:06:26AM +0200, Michael Keller wrote:
> For our block-based incremental backup software we need to monitor all
> writes to a device, and report them to our user mode application for
> further processing. 

Sounds like a variant of dm snapshot code.
What precisely do you need?
Notification of every block every time it is changed?
Or just a list of all the blocks that changed within a defined period (i.e.
since the last backup)?

> 3. Are the interfaces of the device mapper relatively stable? (Assume that
> our target is not general enough to be included into mainline, and we'll
> have to do the API chasing ourselves.)

If integrated nicely into dm, it'll go upstream.
 
Alasdair
-- 
agk@redhat.com

Re: Device-mapper for write monitoring

[Michael Keller]

dm-devel-bounces@redhat.com wrote on 19/02/2009 16:49:18:
> Re: [dm-devel] Device-mapper for write monitoring
> Alasdair G Kergon
> to:
> device-mapper development
> 19/02/2009 16:50
> Cc:
> Mikulas Patocka
>
> On Thu, Feb 19, 2009 at 10:06:26AM +0200, Michael Keller wrote:
> > For our block-based incremental backup software we need to monitor all
> > writes to a device, and report them to our user mode application for
> > further processing.
>
> Sounds like a variant of dm snapshot code.
> What precisely do you need?
> Notification of every block every time it is changed?
> Or just a list of all the blocks that changed within a defined period
(i.e.
> since the last backup)?

Our application is historically designed such that the bitmap is kept in
(and accessed only by) user mode.

If there is any general behavior in this (which would be worth including
into dm...) it would be a kind of "user-intercept": let a user application
register with the device, and get signaled (complete a pending read) for
each write (or read) to the device. The user then decides what to do about
it, and informs the kernel (e.g., filter, proceed, read-before-write, ...)

> > 3. Are the interfaces of the device mapper relatively stable? (Assume
that
> > our target is not general enough to be included into mainline, and
we'll
> > have to do the API chasing ourselves.)
>
> If integrated nicely into dm, it'll go upstream.

I'd wish. But as I said, I find it hard to see anything general in what
we're doing. If you have ideas, let me know!

Thanks,
Michael

>
> Alasdair
> --
> agk@redhat.com
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel