extracting thin mappings in real time

extracting thin mappings in real time

[Thanos Makatos]

[-- Attachment #1.1: Type: text/plain, Size: 1084 bytes --]

I have a kernel module that sits on top of a thin device mapper target that
receives block I/O requests and re-submits then to the thin target. I would
like to implement the following functionality: whenever I receive a write
completion from the thin target (assuming that it's the first time a block
written to) I would like to extract the newly-established mapping of that
virtual block.

I know that I can do this using thin_dump, however this involves:
(1) spawning a process
(2) reserving/releasing a metadata snapshot, and
(3) dumping _all_ the mappings.

In other words, it's far to heavyweight for my performance requirements.

Ideally I would like to be able to obtain the mapping in kernel space. I
had a look at thin_dump and from what I understand it directly reads the
B-tree from the disk? Is there some kernel function that already does this?
E.g. given a thin LBA return the physical block address.

Also, regarding having to have reserved a metadata snapshot, is this
necessary for obtaining mappings? Aren't mappings immutable once established
?

-- 
Thanos Makatos

[-- Attachment #1.2: Type: text/html, Size: 1787 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: extracting thin mappings in real time

[Joe Thornber]

On Wed, Oct 03, 2018 at 01:40:22PM +0100, Thanos Makatos wrote:
> I have a kernel module that sits on top of a thin device mapper target that
> receives block I/O requests and re-submits then to the thin target. I would
> like to implement the following functionality: whenever I receive a write
> completion from the thin target (assuming that it's the first time a block
> written to) I would like to extract the newly-established mapping of that
> virtual block.
> 
> I know that I can do this using thin_dump, however this involves:
> (1) spawning a process
> (2) reserving/releasing a metadata snapshot, and
> (3) dumping _all_ the mappings.
> 
> In other words, it's far to heavyweight for my performance requirements.
> 
> Ideally I would like to be able to obtain the mapping in kernel space. I
> had a look at thin_dump and from what I understand it directly reads the
> B-tree from the disk? Is there some kernel function that already does this?
> E.g. given a thin LBA return the physical block address.
> 
> Also, regarding having to have reserved a metadata snapshot, is this
> necessary for obtaining mappings? Aren't mappings immutable once established
> ?

Could you say more about why you want to do this?

Mappings don't change if you're not using snapshots.  If you are, then any write
to a shared block will trigger a copy on write operation, and subsequently a new
mapping.

So for backups etc. I made the metadata snapshot available, which gives you a
view of all the metadata frozen at a point in time.  If you're using the metadata
snap you still need to guarantee the mappings for the devices that you're interested
in are not changing.  eg, instead of backing up a live thin, take a snapshot of the thin
and back this up instead.

- Joe

Re: extracting thin mappings in real time

[Thanos Makatos]

[-- Attachment #1.1: Type: text/plain, Size: 1135 bytes --]

> Could you say more about why you want to do this?
>

So that I can directly read the data block without having to pass through
dm-thin, e.g. there might be a more direct datapath to the physical block
device.

Mappings don't change if you're not using snapshots.  If you are, then any
> write
> to a shared block will trigger a copy on write operation, and subsequently
> a new
> mapping.

So for backups etc. I made the metadata snapshot available, which gives you
> a
> view of all the metadata frozen at a point in time.  If you're using the
> metadata
> snap you still need to guarantee the mappings for the devices that you're
> interested
> in are not changing.  eg, instead of backing up a live thin, take a
> snapshot of the thin
> and back this up instead.
>

For now I can guarantee that these mappings don't change (no snapshots,
backups, etc), so it's safe to extract the mapping(s) without having
acquired the meradata snapshot, correct? If this is the case, then how can
I extract individual mappins? (Either from kernel space or from user
space.) Are there specific dm-thin kernel functions I should start looking
at?

[-- Attachment #1.2: Type: text/html, Size: 1631 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: extracting thin mappings in real time

[Joe Thornber]

On Wed, Oct 03, 2018 at 03:13:36PM +0100, Thanos Makatos wrote:
> > Could you say more about why you want to do this?
> >
> 
> So that I can directly read the data block without having to pass through
> dm-thin, e.g. there might be a more direct datapath to the physical block
> device.
> 
> Mappings don't change if you're not using snapshots.  If you are, then any
> > write
> > to a shared block will trigger a copy on write operation, and subsequently
> > a new
> > mapping.
> 
> So for backups etc. I made the metadata snapshot available, which gives you
> > a
> > view of all the metadata frozen at a point in time.  If you're using the
> > metadata
> > snap you still need to guarantee the mappings for the devices that you're
> > interested
> > in are not changing.  eg, instead of backing up a live thin, take a
> > snapshot of the thin
> > and back this up instead.
> >
> 
> For now I can guarantee that these mappings don't change (no snapshots,
> backups, etc), so it's safe to extract the mapping(s) without having
> acquired the meradata snapshot, correct? If this is the case, then how can
> I extract individual mappins? (Either from kernel space or from user
> space.) Are there specific dm-thin kernel functions I should start looking
> at?

Why is this thin-specific?  What happens if there are two thin-pools under
your code?  Do you want the final physical location?  The whole dm stack?

Efficiently reading the metadata requires a cache of metadata blocks.  Thinp uses dm-bufio
to provide this.  It seems very wasteful for you to duplicate this.  So your options are:


i) querying the pool metadata object directly.  Currently there is no way to get hold of the
   metadata object, you would need to make any queries off the io path, since they can block.

ii) or examining the completed bios, you'd probably need to add some per bio data to give context.  

- Joe

Re: extracting thin mappings in real time

[Thanos Makatos]

On Wed, Oct 3, 2018 at 3:47 PM Joe Thornber <thornber@redhat.com> wrote:
>
> On Wed, Oct 03, 2018 at 03:13:36PM +0100, Thanos Makatos wrote:
> > > Could you say more about why you want to do this?
> > >
> >
> > So that I can directly read the data block without having to pass through
> > dm-thin, e.g. there might be a more direct datapath to the physical block
> > device.
> >
> > Mappings don't change if you're not using snapshots.  If you are, then any
> > > write
> > > to a shared block will trigger a copy on write operation, and subsequently
> > > a new
> > > mapping.
> >
> > So for backups etc. I made the metadata snapshot available, which gives you
> > > a
> > > view of all the metadata frozen at a point in time.  If you're using the
> > > metadata
> > > snap you still need to guarantee the mappings for the devices that you're
> > > interested
> > > in are not changing.  eg, instead of backing up a live thin, take a
> > > snapshot of the thin
> > > and back this up instead.
> > >
> >
> > For now I can guarantee that these mappings don't change (no snapshots,
> > backups, etc), so it's safe to extract the mapping(s) without having
> > acquired the meradata snapshot, correct? If this is the case, then how can
> > I extract individual mappins? (Either from kernel space or from user
> > space.) Are there specific dm-thin kernel functions I should start looking
> > at?
>
> Why is this thin-specific?  What happens if there are two thin-pools under
> your code?  Do you want the final physical location?  The whole dm stack?

You're right, this isn't thin-specific.

Currently there will be only one thin pool per physical device so it's
simple enough.

Indeed I want the physical location, knowing the location in the data
LV would suffice.

>> Efficiently reading the metadata requires a cache of metadata blocks.  Thinp uses dm-bufio
> to provide this.  It seems very wasteful for you to duplicate this.  So your options are:
>
>
> i) querying the pool metadata object directly.  Currently there is no way to get hold of the
>    metadata object, you would need to make any queries off the io path, since they can block.

Do you mean that the pool metadata object is internal to dm-thin so
that other kernel modules cannot access it? If so then I suppose I can
add functions do dm-thin that return an opaque pointer to it (or
something like that). Is there some specific function that given the
poo metadata object can return this information? I've started looking
at  thin_bio_map(), is this the best place to start?

Also, is option (i) doable from userspace? Is there logic in thin_dump
to access the on-disk metadata? I suppose I will have to take care
when reading the metadata as they might not have been committed to
disk.

>> ii) or examining the completed bios, you'd probably need to add some per bio data to give context.

I think option (ii) sounds like the most efficient one but most invasive.

Re: extracting thin mappings in real time

[Joe Thornber]

On Wed, Oct 03, 2018 at 04:47:41PM +0100, Thanos Makatos wrote:
> poo metadata object can return this information? I've started looking
> at  thin_bio_map(), is this the best place to start?

See thin-metadata.h

- Joe

Re: extracting thin mappings in real time

[Thanos Makatos]

On Thu, 4 Oct 2018 at 13:56, Joe Thornber <thornber@redhat.com> wrote:
>
> On Wed, Oct 03, 2018 at 04:47:41PM +0100, Thanos Makatos wrote:
> > poo metadata object can return this information? I've started looking
> > at  thin_bio_map(), is this the best place to start?
>
> See thin-metadata.h
>
> - Joe

I started looking at piggybacking the mapping in bio->bi_pivate in
provision_block and it looks ugly. First, I'll have to introduce a new
bio flag specifically for this reason: whenever dm-thin sees this it
should store the mapping in the bio->bi_private. Second, the bio
passed to dm-thin is cloned so I'll have to find the original bio
(this requires making struct dm_io public). And third, the original
bio can be split to multiple cloned bios so the structure in
bio->bi_pivate should accomodate this.

Instead I gave a second thought to the solution of using dm-thin's
public functions to access it's metadata, namely dm_thin_find_block.
The problem is that if the mapping is not in memory I'll have to read
it from disk, outside the critical path, and this adds complexity.
However, since the mapping was just established it should be cached so
it should mostly work (not being able to obtain the mapping every now
and then is acceptable). Does this make sense?

Re: extracting thin mappings in real time

[Thanos Makatos]

> Instead I gave a second thought to the solution of using dm-thin's
> public functions to access it's metadata, namely dm_thin_find_block.
> The problem is that if the mapping is not in memory I'll have to read
> it from disk, outside the critical path, and this adds complexity.
> However, since the mapping was just established it should be cached so
> it should mostly work (not being able to obtain the mapping every now
> and then is acceptable). Does this make sense?

I call dm_thin_find_block right after I've received the write
completion and if I don't set can_issue_io to 1 it _always_ fails. If
I do set it to 1 then it always succeeds but reads 20 KB from the
metadata device, even when repeatedly called. I can see that
internally it uses dm-bufio, which is supposed to some caching, no? Am
I missing something?