Device mapper target driver for Write tracking or Write split

Device mapper target driver for Write tracking or Write split

[Suresh Babu Kandukuru]

[-- Attachment #1.1: Type: text/plain, Size: 308 bytes --]

Hi List,

 

I am mainly looking at write tracking Device Mapper device .  Do we  have any device mapper target driver for write tacking or write splitting ?. or any work  happening on this to leverage  ?. Please kindly share any web links or pointers on this . Thanks in advance .

 

/Suresh

 

[-- Attachment #1.2: Type: text/html, Size: 1994 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Device mapper target driver for Write tracking or Write split

[Vivek Goyal]

On Thu, Jan 08, 2015 at 09:23:21PM -0800, Suresh Babu Kandukuru wrote:
> Hi List,
> 
>  
> 
> I am mainly looking at write tracking Device Mapper device .  Do we  have any device mapper target driver for write tacking or write splitting ?. or any work  happening on this to leverage  ?. Please kindly share any web links or pointers on this . Thanks in advance .
> 

Hi Suresh,

Could you give more details. What is write tracking/splitting and what
are you planning to do with that.

Thanks
Vivek

Re: Device mapper target driver for Write tracking or Write split

[Suresh Babu Kandukuru]

Vivek, 
  1) Basically we map the LUN from storage arrary to one of linux host . Linux host sees that  LUN with mutlipath ( dm-multipath) abstact device /dev/mapper/LUNID . Linux host can mount that lun and do some read and write operation  on that . We want to cature those writes and send those writes  to another appliance . we will use that appilance for CDP( continues data protection)  of that LUN . 

2) My idea is To track writes  on the mapped lun  in Linux,  Can we  instantiate a write tracking Device Mapper device stacked on top of the mapped lun device ( /dev/mapper/LUIN) ?. Then, whenever the write tracking Device Mapper device receives  write request, it tracks the write. After that, it passes the  write request down to the underlying  lun device. In addition, to track writes to a Lun device, Linux host needs to be told to access the write tracking device instead of the LUN device itself. 

Kind of  Host side  write splitter in Storage domain,

Do we have any implemention which does  this ?. or  the work is in progress which we  can leverage ?. ?. Please kindly share any web links or pointers on this . Thanks in advance .

/Suresh

 
-----Original Message-----
From: Vivek Goyal [mailto:vgoyal@redhat.com] 
Sent: Friday, January 09, 2015 7:53 PM
To: device-mapper development
Subject: Re: [dm-devel] Device mapper target driver for Write tracking or Write split

On Thu, Jan 08, 2015 at 09:23:21PM -0800, Suresh Babu Kandukuru wrote:
> Hi List,
> 
>  
> 
> I am mainly looking at write tracking Device Mapper device .  Do we  have any device mapper target driver for write tacking or write splitting ?. or any work  happening on this to leverage  ?. Please kindly share any web links or pointers on this . Thanks in advance .
> 

Hi Suresh,

Could you give more details. What is write tracking/splitting and what are you planning to do with that.

Thanks
Vivek

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: Device mapper target driver for Write tracking or Write split

[Bryn M. Reeves]

On Fri, Jan 09, 2015 at 06:51:28AM -0800, Suresh Babu Kandukuru wrote:
>   1) Basically we map the LUN from storage arrary to one of linux host . Linux host sees that  LUN with mutlipath ( dm-multipath) abstact device /dev/mapper/LUNID . Linux host can mount that lun and do some read and write operation  on that . We want to cature those writes and send those writes  to another appliance . we will use that appilance for CDP( continues data protection)  of that LUN . 

The closest to this is probably dm-replicator:

  http://people.redhat.com/heinzm/sw/dm/dm-replicator/src/linux/Documentation/device-mapper/replicator.txt

However it's not upstream and is not currently under development.

In addition to the single device write replication you discuss it supports
a range of device and link counts and types (e.g. the README above shows
an example with multiple local and remote devices and inter site links).
 
> 2) My idea is To track writes  on the mapped lun  in Linux,  Can we  instantiate a write tracking Device Mapper device stacked on top of the mapped lun device ( /dev/mapper/LUIN) ?. Then, whenever the write tracking Device Mapper device receives  write request, it tracks the write. After that, it passes the  write request down to the underlying  lun device. In addition, to track writes to a Lun device, Linux host needs to be told to access the write tracking device instead of the LUN device itself. 

In terms of targets that are upstream and supported currently this has
some similarity to the mirroring modes now offered via the dm-raid target.

This exposes the Linux MD RAID personalities as device-mapper targets.
Some combination of RAID1 and write-behind behaviour for the replicated
device may give you what you're looking for.

Regards,
Bryn.

Re: Device mapper target driver for Write tracking or Write split

[Suresh Babu Kandukuru]

Bryn, Thanks for quick reply. Below items helps to do mirroring and replication . I am looking for something which  can track writes( device mapper based block level filter driver)   as explained in the below diagram in Linux open source community .

http://www.ryussi.com/ryucorp/uploads/2014/11/Block-level-filter-driver_jitu.png?723184


Thanks again

/Suresh


-----Original Message-----
From: Bryn M. Reeves [mailto:bmr@redhat.com] 
Sent: Friday, January 09, 2015 8:44 PM
To: device-mapper development
Cc: vgoyal@redhat.com
Subject: Re: [dm-devel] Device mapper target driver for Write tracking or Write split

On Fri, Jan 09, 2015 at 06:51:28AM -0800, Suresh Babu Kandukuru wrote:
>   1) Basically we map the LUN from storage arrary to one of linux host . Linux host sees that  LUN with mutlipath ( dm-multipath) abstact device /dev/mapper/LUNID . Linux host can mount that lun and do some read and write operation  on that . We want to cature those writes and send those writes  to another appliance . we will use that appilance for CDP( continues data protection)  of that LUN . 

The closest to this is probably dm-replicator:

  http://people.redhat.com/heinzm/sw/dm/dm-replicator/src/linux/Documentation/device-mapper/replicator.txt

However it's not upstream and is not currently under development.

In addition to the single device write replication you discuss it supports a range of device and link counts and types (e.g. the README above shows an example with multiple local and remote devices and inter site links).
 
> 2) My idea is To track writes  on the mapped lun  in Linux,  Can we  instantiate a write tracking Device Mapper device stacked on top of the mapped lun device ( /dev/mapper/LUIN) ?. Then, whenever the write tracking Device Mapper device receives  write request, it tracks the write. After that, it passes the  write request down to the underlying  lun device. In addition, to track writes to a Lun device, Linux host needs to be told to access the write tracking device instead of the LUN device itself. 

In terms of targets that are upstream and supported currently this has some similarity to the mirroring modes now offered via the dm-raid target.

This exposes the Linux MD RAID personalities as device-mapper targets.
Some combination of RAID1 and write-behind behaviour for the replicated device may give you what you're looking for.

Regards,
Bryn.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: Device mapper target driver for Write tracking or Write split

[Bryn M. Reeves]

On Fri, Jan 09, 2015 at 07:51:08AM -0800, Suresh Babu Kandukuru wrote:
> Bryn, Thanks for quick reply. Below items helps to do mirroring and replication . I am looking for something which  can track writes( device mapper based block level filter driver)   as explained in the below diagram in Linux open source community .

Anything that's doing mirroring or replication (or many other useful
target functions) is going to need to 'track' writes (normally by
copying them and keeping track of the copies in the target).

The device-mapper core provides infrastructure to help with this but if
none of the existing dm targets provide the functionality you need then
you will either need to create one from scratch or extend whatever
current target best matches your needs.

Aside from mentioning CDP and replicating data to another appliance you
haven't really stated what you're trying to do.

Regards,
Bryn.

Re: Device mapper target driver for Write tracking or Write split

[Suresh Babu Kandukuru]

Thanks Bryn.

>>Aside from mentioning CDP and replicating data to another appliance you
>>haven't really stated what you're trying to do.

[Suresh]  - >  Bascilly I would like tap/track the  all writes happening that mapped lun on the host   ( take it as any block device on the host ) , in the format of Offset, Length and Data  . and then send this data to CDP appliace . This CDP appliance has the target lun which is intially synced to source lun on the host .  after receving the writes happening  on  host , this cdp applicance apply writes to target lun . so both source lun and target luns are  in sync all the time .  Can you advise me anything better now ?. Thanks again 

-----Original Message-----
From: Bryn M. Reeves [mailto:bmr@redhat.com] 
Sent: Friday, January 09, 2015 10:29 PM
To: device-mapper development
Cc: vgoyal@redhat.com
Subject: Re: [dm-devel] Device mapper target driver for Write tracking or Write split

On Fri, Jan 09, 2015 at 07:51:08AM -0800, Suresh Babu Kandukuru wrote:
> Bryn, Thanks for quick reply. Below items helps to do mirroring and replication . I am looking for something which  can track writes( device mapper based block level filter driver)   as explained in the below diagram in Linux open source community .

Anything that's doing mirroring or replication (or many other useful
target functions) is going to need to 'track' writes (normally by
copying them and keeping track of the copies in the target).

The device-mapper core provides infrastructure to help with this but if
none of the existing dm targets provide the functionality you need then
you will either need to create one from scratch or extend whatever
current target best matches your needs.

Aside from mentioning CDP and replicating data to another appliance you
haven't really stated what you're trying to do.

Regards,
Bryn.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: Device mapper target driver for Write tracking or Write split

[Zdenek Kabelac]

Dne 9.1.2015 v 16:51 Suresh Babu Kandukuru napsal(a):
> Bryn, Thanks for quick reply. Below items helps to do mirroring and replication . I am looking for something which  can track writes( device mapper based block level filter driver)   as explained in the below diagram in Linux open source community .
>
> http://www.ryussi.com/ryucorp/uploads/2014/11/Block-level-filter-driver_jitu.png?723184
>
>

We do have a plan to use 'thin provisioning' eventually 'era' target to 
support 'remote' replication (having it as a future feature).

Basically to transmit 'snapshot' diffs between host's thin pool.

dm-replicator had some hard-to-fix performance and correctness problems so its 
development has been abandoned.

Regards

Zdenek

> Thanks again
>
> /Suresh
>
>
> -----Original Message-----
> From: Bryn M. Reeves [mailto:bmr@redhat.com]
> Sent: Friday, January 09, 2015 8:44 PM
> To: device-mapper development
> Cc: vgoyal@redhat.com
> Subject: Re: [dm-devel] Device mapper target driver for Write tracking or Write split
>
> On Fri, Jan 09, 2015 at 06:51:28AM -0800, Suresh Babu Kandukuru wrote:
>>    1) Basically we map the LUN from storage arrary to one of linux host . Linux host sees that  LUN with mutlipath ( dm-multipath) abstact device /dev/mapper/LUNID . Linux host can mount that lun and do some read and write operation  on that . We want to cature those writes and send those writes  to another appliance . we will use that appilance for CDP( continues data protection)  of that LUN .
>
> The closest to this is probably dm-replicator:
>
>    http://people.redhat.com/heinzm/sw/dm/dm-replicator/src/linux/Documentation/device-mapper/replicator.txt
>
> However it's not upstream and is not currently under development.
>
> In addition to the single device write replication you discuss it supports a range of device and link counts and types (e.g. the README above shows an example with multiple local and remote devices and inter site links).
>
>> 2) My idea is To track writes  on the mapped lun  in Linux,  Can we  instantiate a write tracking Device Mapper device stacked on top of the mapped lun device ( /dev/mapper/LUIN) ?. Then, whenever the write tracking Device Mapper device receives  write request, it tracks the write. After that, it passes the  write request down to the underlying  lun device. In addition, to track writes to a Lun device, Linux host needs to be told to access the write tracking device instead of the LUN device itself.
>
> In terms of targets that are upstream and supported currently this has some similarity to the mirroring modes now offered via the dm-raid target.
>
> This exposes the Linux MD RAID personalities as device-mapper targets.
> Some combination of RAID1 and write-behind behaviour for the replicated device may give you what you're looking for.
>
> Regards,
> Bryn.