Is there a way to see updated contents of a DM target's underlying device while the DM target is in use?

Is there a way to see updated contents of a DM target's underlying device while the DM target is in use?

[Riches Jr, RobertX M]

[-- Attachment #1.1: Type: text/plain, Size: 2213 bytes --]

(New list member, just subscribed.  Apologies if this is not the right list for this question.)

Is there a way to see the updated contents of a DM target's underlying device while the DM target is in use?  I Googled and searched in documentation and existing code for an answer but was not able to find it.

Background:

I'm debugging a DM target that does change-tracking.  It's a combination of code from dm-linear, Alberto Bertogli's dm-csum, dm-delay, and some original code.  (The project architect does intend to properly release it under GPL per the license of the incorporated GPL pieces.)  The target reserves an early portion of the underlying device and exposes/creates/supports a device that is a linear mapping of the rest of the underlying device.  The idea is each write to the exposed device sets a corresponding bit in the reserved portion, using the write barrier trick from dm-csum's 'same' mode, to indicate which parts of the exposed device have been modified and which have not.  (I had earlier tried to use the userspace dirty-logging facility but found it to not meet our needs.)

The target is intended to be used with LVM snapshots to do block-device-level backups.  (I'm aware of filesystem-level issues with that, but the requirement is the requirement.)  The underlying device for the DM target will be an LVM logical volume.  The backup manager will send a dmsetup message to the DM target to hold all writes (and maybe reads) on indefinite delay, create an LVM snapshot of the DM target's underlying device, then flush the held bios and resume normal operation.

Problem:

The current problem I have is many updates are not visible in the DM target's underlying device until I do a 'dmsetup remove'.  When I look at the underlying device (an LVM LV) prior to 'dmsetup remove', it's not reliable whether I see what has been written by the DM target.  I have verified that the bios go out and the callbacks are called.

Any suggestions toward being able to reliably see (as in copy out) data from the underlying device or making an LVM snapshot of it while my change-tracking DM target module is still in use?

Thanks,

Robert Riches
Azad Consultant at Intel



[-- Attachment #1.2: Type: text/html, Size: 6155 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Is there a way to see updated contents of a DM target's underlying device while the DM target is in use?

[Alasdair G Kergon]

Sounds like a coding problem you need to debug...check your device stack - and the
code you are using to look for the change - carefully, check where you are
queueing/flushing/caching etc.

Alasdair

Re: Is there a way to see updated contents of a DM target's underlying device while the DM target is in use?

[Riches Jr, RobertX M]

[-- Attachment #1: Type: text/plain, Size: 1161 bytes --]

Thank you for the wise council to carefully check my code.  I certainly intend to do that.

However, I am able to reproduce the symptom using the standard dm-linear module supplied with the x86_64 kernel that comes with Ubuntu Karmic.  With the attached script, the first 'od' on /dev/vg1/lv1, done before 'dmsetup remove', shows the first part of the file contains 0x00.  A second 'od' after 'dmsetup remove' shows the file contains random data.

I didn't see any queues or caches in dm-linear that would need to be flushed.

Any other suggestions?

Thanks,

Robert Riches
Azad Consultant at Intel
 
-----Original Message-----
From: Alasdair G Kergon [mailto:agk@redhat.com] 
Sent: Wednesday, June 30, 2010 4:08 PM
To: Riches Jr, RobertX M
Cc: Dm-Devel Mailing List Send-To Address (dm-devel@redhat.com)
Subject: Re: [dm-devel] Is there a way to see updated contents of a DM target's underlying device while the DM target is in use?

Sounds like a coding problem you need to debug...check your device stack - and the
code you are using to look for the change - carefully, check where you are
queueing/flushing/caching etc.

Alasdair


[-- Attachment #2: test02linear --]
[-- Type: application/octet-stream, Size: 1013 bytes --]

#!/bin/csh -f

# C-shell script to run part A of test 02 but using the _LINEAR_ target.

echo 'creating new pv file'
dd if=/dev/zero of=dev01 bs=512 count=2105344
echo ''

echo 'setting up lv'
losetup /dev/loop0 dev01
pvcreate /dev/loop0
vgcreate vg1 /dev/loop0
lvcreate -L 1024M -n lv1 vg1

echo 'setting up linear mapping'
dmsetup create fred01 --table '0 2097152 linear /dev/vg1/lv1 0'

echo 'writing the whole thing'
sleep 3
dd if=/dev/urandom of=/dev/mapper/fred01 bs=8192
echo ''

echo 'making a copy'
sleep 3
cp -i /dev/vg1/lv1 zz.file
echo ''

echo 'doing a first diff'
sleep 3
diff /dev/vg1/lv1 zz.file
echo first diff status was $status
echo ''

echo 'showing the first contents'
od -A x -xc /dev/vg1/lv1 | head
echo ''

echo 'removing the linear mapping so the changes can be seen'
sleep 3
dmsetup remove fred01
echo ''

echo 'showing the second contents'
od -A x -xc /dev/vg1/lv1 | head
echo ''

echo 'doing a second diff'
sleep 3
diff /dev/vg1/lv1 zz.file
echo second diff status was $status
echo ''

[-- Attachment #3: Type: text/plain, Size: 0 bytes --]

Re: Is there a way to see updated contents of a DM target's underlying device while the DM target is in use?

[Alasdair G Kergon]

On Wed, Jun 30, 2010 at 04:53:23PM -0700, Riches Jr, RobertX M wrote:
> I didn't see any queues or caches in dm-linear that would need to be flushed.
 
It's your test script that is not behaving as you expect:)

Add conv=odirect to dd

Add blockdev --flushbufs after other writes.

Alasdair

Re: Is there a way to see updated contents of a DM target's underlying device while the DM target is in use?

[Alasdair G Kergon]

On Thu, Jul 01, 2010 at 01:51:57AM +0100, Alasdair G Kergon wrote:
> Add conv=odirect to dd
> Add blockdev --flushbufs after other writes.
 
Applications like lvm2 have those things built in to guarantee metadata
consistency at application level.

Alasdair

Re: Is there a way to see updated contents of a DM target's underlying device while the DM target is in use?

[Malahal Naineni]

Alasdair G Kergon [agk@redhat.com] wrote:
> On Wed, Jun 30, 2010 at 04:53:23PM -0700, Riches Jr, RobertX M wrote:
> > I didn't see any queues or caches in dm-linear that would need to be flushed.
> 
> It's your test script that is not behaving as you expect:)
> 
> Add conv=odirect to dd

You mean flags rather than conv? conv just converts ascii to ebcdic etc. 
oflag=direct would do direct I/O on the writes and iflag=direct should do
direct I/O on reads.

Thanks, Malahal.

Re: Is there a way to see updated contents of a DM target's underlying device while the DM target is in use?

[Alasdair G Kergon]

On Wed, Jun 30, 2010 at 06:03:09PM -0700, Malahal Naineni wrote:
> Alasdair G Kergon [agk@redhat.com] wrote:
> > Add conv=odirect to dd
 
> You mean flags 

Yeah - it's getting late here...

Alasdair

Re: Is there a way to see updated contents of a DM target's underlying device while the DM target is in use?

[Riches Jr, RobertX M]

Many thanks to those who responded with helpful suggestions.  A couple of the suggestions were sufficient to make my test script behave as I expect--to liberate the data that had been stuck in the page cache twilight zone.  :-)

For the benefit of anyone who might come across the archives with a similar problem, here's what worked and what didn't, with all actions taken between writing and reading:

  - Using 'dd -iflag=direct' to read from /dev/vg1/lv1 (the underlying device) worked.

  - Doing 'blockdev --flushbufs /dev/vg1/lv1' (on the underlying device) worked.

  - Doing 'blockdev --flushbufs /dev/mapper/fred01' (on the device exposed by dm-linear or my module) did not solve the problem.

  - Doing 'dmsetup suspend fred01' did not work.

  - Doing fsync() on /dev/mapper/fred01 did not work.

  - Doing fsync() on /dev/vg1/lv1 did not work.

Of those above that "worked", each was independently sufficient to solve the problematic symptoms. IOW, I only had to apply one solution, not all of them.

Again, thanks for all the solutions.

Robert Riches
Azad Consultant at Intel
 
-----Original Message-----
From: Alasdair G Kergon [mailto:agk@redhat.com] 
Sent: Wednesday, June 30, 2010 5:52 PM
To: Riches Jr, RobertX M
Cc: Dm-Devel Mailing List Send-To Address (dm-devel@redhat.com)
Subject: Re: [dm-devel] Is there a way to see updated contents of a DM target's underlying device while the DM target is in use?

On Wed, Jun 30, 2010 at 04:53:23PM -0700, Riches Jr, RobertX M wrote:
> I didn't see any queues or caches in dm-linear that would need to be flushed.
 
It's your test script that is not behaving as you expect:)

Add conv=odirect to dd

Add blockdev --flushbufs after other writes.

Alasdair

Re: Is there a way to see updated contents of a DM target's underlying device while the DM target is in use?

[Malahal Naineni]

Riches Jr, RobertX M [robertx.m.riches.jr@intel.com] wrote:
> Many thanks to those who responded with helpful suggestions.  A couple of the suggestions were sufficient to make my test script behave as I expect--to liberate the data that had been stuck in the page cache twilight zone.  :-)
> 
> For the benefit of anyone who might come across the archives with a similar problem, here's what worked and what didn't, with all actions taken between writing and reading:
> 
>   - Using 'dd -iflag=direct' to read from /dev/vg1/lv1 (the underlying device) worked.
> 
>   - Doing 'blockdev --flushbufs /dev/vg1/lv1' (on the underlying device) worked.
> 
>   - Doing 'blockdev --flushbufs /dev/mapper/fred01' (on the device exposed by dm-linear or my module) did not solve the problem.

I wonder why flushing the underlying device worked but not the actual
device itself. I expected the opposite! :-(

Re: Is there a way to see updated contents of a DM target's underlying device while the DM target is in use?

[Hannes Reinecke]

Malahal Naineni wrote:
> Riches Jr, RobertX M [robertx.m.riches.jr@intel.com] wrote:
>> Many thanks to those who responded with helpful suggestions.  A couple of the suggestions were sufficient to make my test script behave as I expect--to liberate the data that had been stuck in the page cache twilight zone.  :-)
>>
>> For the benefit of anyone who might come across the archives with a similar problem, here's what worked and what didn't, with all actions taken between writing and reading:
>>
>>   - Using 'dd -iflag=direct' to read from /dev/vg1/lv1 (the underlying device) worked.
>>
>>   - Doing 'blockdev --flushbufs /dev/vg1/lv1' (on the underlying device) worked.
>>
>>   - Doing 'blockdev --flushbufs /dev/mapper/fred01' (on the device exposed by dm-linear or my module) did not solve the problem.
> 
> I wonder why flushing the underlying device worked but not the actual
> device itself. I expected the opposite! :-(
> 
Why, no. That's exactly the point.
The device keeps the underlying block devices pinned in the cache, so any page invalidation can only be triggered
by the device itself, not the underlying ones. Any writes to the underlying devices won't trigger a page
invalidation, so they'll stay in the cache forever.
Hence you need to flush them explicitely.

Not that I'll recommend that. It'll play silly buggers with the page cache.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)

Re: Is there a way to see updated contents of a DM target's underlying device while the DM target is in use?

[Malahal Naineni]

Riches Jr, RobertX M [robertx.m.riches.jr@intel.com] wrote:
> However, I am able to reproduce the symptom using the standard dm-linear module supplied with the x86_64 kernel that comes with Ubuntu Karmic.  With the attached script, the first 'od' on /dev/vg1/lv1, done before 'dmsetup remove', shows the first part of the file contains 0x00.  A second 'od' after 'dmsetup remove' shows the file contains random data.
> 
> I didn't see any queues or caches in dm-linear that would need to be flushed.
> 
> Any other suggestions?

If you write to a block device devA and read from it on the same system,
you would get the expected data. In your case, you have devA and devB
(devB is a mapped device on devA, but that doesn't matter), you write to
devA and read from devB. You won't see what you wrote because of block
device caching. Try fsync() on the block device fred01 before copying or
try suspend on the linear dev (fred01 in your case).

Thanks, Malahal.

Re: Is there a way to see updated contents of a DM target's underlying device while the DM target is in use?

[Neil Brown]

On Wed, 30 Jun 2010 15:54:08 -0700
"Riches Jr, RobertX M" <robertx.m.riches.jr@intel.com> wrote:

> (New list member, just subscribed.  Apologies if this is not the right list for this question.)
> 
> Is there a way to see the updated contents of a DM target's underlying device while the DM target is in use?  I Googled and searched in documentation and existing code for an answer but was not able to find it.
> 
> Background:
> 
> I'm debugging a DM target that does change-tracking.  It's a combination of code from dm-linear, Alberto Bertogli's dm-csum, dm-delay, and some original code.  (The project architect does intend to properly release it under GPL per the license of the incorporated GPL pieces.)  The target reserves an early portion of the underlying device and exposes/creates/supports a device that is a linear mapping of the rest of the underlying device.  The idea is each write to the exposed device sets a corresponding bit in the reserved portion, using the write barrier trick from dm-csum's 'same' mode, to indicate which parts of the exposed device have been modified and which have not.  (I had earlier tried to use the userspace dirty-logging facility but found it to not meet our needs.)
> 
> The target is intended to be used with LVM snapshots to do block-device-level backups.  (I'm aware of filesystem-level issues with that, but the requirement is the requirement.)  The underlying device for the DM target will be an LVM logical volume.  The backup manager will send a dmsetup message to the DM target to hold all writes (and maybe reads) on indefinite delay, create an LVM snapshot of the DM target's underlying device, then flush the held bios and resume normal operation.
> 
> Problem:
> 
> The current problem I have is many updates are not visible in the DM target's underlying device until I do a 'dmsetup remove'.  When I look at the underlying device (an LVM LV) prior to 'dmsetup remove', it's not reliable whether I see what has been written by the DM target.  I have verified that the bios go out and the callbacks are called.

try blockdev --flushbufs
on the device before reading it.
Or issue the ioctl yourself.
Or use O_DIRECT to read.

I suspect you are reading from the page cache for the device, while dm-linear
reads/writes directly to the device avoiding the page cache.

NeilBrown


> 
> Any suggestions toward being able to reliably see (as in copy out) data from the underlying device or making an LVM snapshot of it while my change-tracking DM target module is still in use?
> 
> Thanks,
> 
> Robert Riches
> Azad Consultant at Intel
> 
> 

Re: Is there a way to see updated contents of a DM target's underlying device while the DM target is in use?

[Malahal Naineni]

Neil Brown [neilb@suse.de] wrote:
> try blockdev --flushbufs
> on the device before reading it.
> Or issue the ioctl yourself.

This should help.

> Or use O_DIRECT to read.

Can this help? He is reading from logical volume lv1 but the page cache
exists for linear device fred01.

Thanks, Malahal.