[Drbd-dev] New Features

[Drbd-dev] New Features

[Morey Roof]

I have been using DRBD and really like but I have wanted to add some new 
features to it and I wanted to ask you guys what you need and 
requirements you may have on patches to DRBD.

Thanks,

Morey

Re: [Drbd-dev] New Features

[Lars Ellenberg]

On Thu, Sep 18, 2008 at 11:24:30PM -0600, Morey Roof wrote:
> I have been using DRBD and really like but I have wanted to add some new  
> features to it and I wanted to ask you guys what you need and  
> requirements you may have on patches to DRBD.

Could you tell us about your background, and prior coding experience,
and how much time you think you want to spend on this?

In the drbd module, we currently do not actively work on new features
that would be suitable to be implemented by anyone not very familiar
with the drbd code base.  If you think you are, let me know...

Finding and squashing bugs is always a wellcome contribution.

Userland could be improved, e.g. adding an "include" statement
to the drbd.conf grammar, which would allow to have the configuration
split out in /etc/drbd.conf.d/*.conf.

Error messages can be improved, in both kernel and userland.

The community would be happy to see some detailed walk-throughs from
scratch for drbd integration with pacemaker, including various typically
deployed services on top of drbd.

I'd be happy to see a recent and solid HA-DRBD-NFS howto,
so if you feel that would be a task for you,
set up an HA-DRBD-NFSv4 server, and document everything needed to make
it behave as expected (document the expectations as well) in face of
failover, switchover, node reboot, cluster reboot, replication link failure,
other component failures, various administrative tasks.
  An advanced variation on the theme: "load balancing" HA-DRBD-NFSv4
cluster, using multiple drbd, each with its own file system, fsid, and
HA-ip, and then switchover at will, without causing server nor client
hickups.

-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.

Re: [Drbd-dev] New Features

[Morey Roof]

I have been programming professionally for about 12 years with C/C++
in various system.  I have worked on various parts of Linux for
companies but I never submitted work to be included until recently.
So I am a bit new to the get stuff out there model, but I am quite
experienced working on very large and complex projects.

What has allowed me to start working on some open source projects
recently is the move to a job for a college where such work is
welcomed and encouraged instead of the traditional business models
where everything is a closely guarded secret.  We have been using DRBD
for a while and I have found that there are several itches I would
like to scratch, so to speak.

Let me know if you need more information.

Thanks,

Morey

On Fri, Sep 19, 2008 at 9:53 AM, Lars Ellenberg
<lars.ellenberg@linbit.com> wrote:
> On Thu, Sep 18, 2008 at 11:24:30PM -0600, Morey Roof wrote:
>> I have been using DRBD and really like but I have wanted to add some new
>> features to it and I wanted to ask you guys what you need and
>> requirements you may have on patches to DRBD.
>
> Could you tell us about your background, and prior coding experience,
> and how much time you think you want to spend on this?
>
> In the drbd module, we currently do not actively work on new features
> that would be suitable to be implemented by anyone not very familiar
> with the drbd code base.  If you think you are, let me know...
>
> Finding and squashing bugs is always a wellcome contribution.
>
> Userland could be improved, e.g. adding an "include" statement
> to the drbd.conf grammar, which would allow to have the configuration
> split out in /etc/drbd.conf.d/*.conf.
>
> Error messages can be improved, in both kernel and userland.
>
> The community would be happy to see some detailed walk-throughs from
> scratch for drbd integration with pacemaker, including various typically
> deployed services on top of drbd.
>
> I'd be happy to see a recent and solid HA-DRBD-NFS howto,
> so if you feel that would be a task for you,
> set up an HA-DRBD-NFSv4 server, and document everything needed to make
> it behave as expected (document the expectations as well) in face of
> failover, switchover, node reboot, cluster reboot, replication link failure,
> other component failures, various administrative tasks.
>  An advanced variation on the theme: "load balancing" HA-DRBD-NFSv4
> cluster, using multiple drbd, each with its own file system, fsid, and
> HA-ip, and then switchover at will, without causing server nor client
> hickups.
>
> --
> : Lars Ellenberg
> : LINBIT | Your Way to High Availability
> : DRBD/HA support and consulting http://www.linbit.com
>
> DRBD(R) and LINBIT(R) are registered trademarks of LINBIT, Austria.
> _______________________________________________
> drbd-dev mailing list
> drbd-dev@lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-dev
>

Re: [Drbd-dev] New Features

[Lars Ellenberg]

On Fri, Sep 19, 2008 at 10:19:03AM -0600, Morey Roof wrote:
> I have been programming professionally for about 12 years with C/C++
> in various system.  I have worked on various parts of Linux for
> companies but I never submitted work to be included until recently.
> So I am a bit new to the get stuff out there model, but I am quite
> experienced working on very large and complex projects.
> 
> What has allowed me to start working on some open source projects
> recently is the move to a job for a college where such work is
> welcomed and encouraged instead of the traditional business models
> where everything is a closely guarded secret.  We have been using DRBD
> for a while and I have found that there are several itches I would
> like to scratch, so to speak.
> 
> Let me know if you need more information.

sounds promising.

do you want to discuss some of the itches?

	Lars

Re: [Drbd-dev] New Features

[Morey Roof]

Lars Ellenberg wrote:
> On Fri, Sep 19, 2008 at 10:19:03AM -0600, Morey Roof wrote:
>   
>> I have been programming professionally for about 12 years with C/C++
>> in various system.  I have worked on various parts of Linux for
>> companies but I never submitted work to be included until recently.
>> So I am a bit new to the get stuff out there model, but I am quite
>> experienced working on very large and complex projects.
>>
>> What has allowed me to start working on some open source projects
>> recently is the move to a job for a college where such work is
>> welcomed and encouraged instead of the traditional business models
>> where everything is a closely guarded secret.  We have been using DRBD
>> for a while and I have found that there are several itches I would
>> like to scratch, so to speak.
>>
>> Let me know if you need more information.
>>     
>
> sounds promising.
>
> do you want to discuss some of the itches?
>
> 	Lars
> _______________________________________________
> drbd-dev mailing list
> drbd-dev@lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-dev
>   

The first item on the list is a write back cache.  Now don't run 
screaming to hills yet.  In our setup we have the two nodes in two 
different buildings which are feed by different power supplies from the 
city grid.  The machines themselves have dual power supplies which each 
power supply going to it's own UPS and circuit.  So, we have a good 
redundant power supply setup.  So, I was thinking about creating a 
caching system in DRBD to allow for dedicating some of the server's RAM 
to a large read/write-back cache.  In the event of communication failure 
between the peers I wanted it to flush the entire cache and then run in 
write-through mode and move back to write-back mode after communication 
is present again and the resync is finished.

So that is the basic idea.  I was thinking about having it use the 
ramdisk system so that you can specify a device that is used as the 
cache.  The reason I was thinking about this is that if you happen to 
purchase one of those Gigabyte ram HDD that is just like a large battery 
backed write cache you could set it as the cache device as well.

What are your thought?

-Morey

Re: [Drbd-dev] New Features

[Lars Ellenberg]

On Fri, Sep 19, 2008 at 04:28:19PM -0600, Morey Roof wrote:
> The first item on the list is a write back cache.  Now don't run  
> screaming to hills yet.  In our setup we have the two nodes in two  
> different buildings which are feed by different power supplies from the  
> city grid.  The machines themselves have dual power supplies which each  
> power supply going to it's own UPS and circuit.  So, we have a good  
> redundant power supply setup.  So, I was thinking about creating a  
> caching system in DRBD to allow for dedicating some of the server's RAM  
> to a large read/write-back cache.  In the event of communication failure  
> between the peers I wanted it to flush the entire cache and then run in  
> write-through mode and move back to write-back mode after communication  
> is present again and the resync is finished.
>
> So that is the basic idea.  I was thinking about having it use the  
> ramdisk system so that you can specify a device that is used as the  
> cache.  The reason I was thinking about this is that if you happen to  
> purchase one of those Gigabyte ram HDD that is just like a large battery  
> backed write cache you could set it as the cache device as well.
>
> What are your thought?

something I though of myself already,
but nothing for the curren drbd architechture.

I sent you my Linux-Kongress 2008 paper off list,
where I scetch what I have in mind for the future.
I'd put in a link here, if I had it online already.

though I did not mention it, a (possibly non-volatile) ram disk
would fit in pretty well as the log device.

let me know what you think.

-- 
: Lars Ellenberg                
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
__
please don't Cc me, but send to list   --   I'm subscribed

Re: [Drbd-dev] New Features

[Lars Ellenberg]

On Sat, Sep 20, 2008 at 03:18:21PM +0200, Lars Ellenberg wrote:

"Write-Back" cache:

some things to think of when introducing a write back cache,
 * need to do some cache coherency protocol
 * need to track which block is where, so we can read the correct
   version in case it has not yet been committed to final location
 * if using a ram buffer as log disk, we need to track the latest
   position for overwrites.
 * if we have stages, i.e. ram buffer first, then log disk, then real
   storage, we are the most flexible.
   if peers are sufficiently close, we can send_page from the ram
   buffer (and calculate checksums there, for data integrity).
   if we use it as ring buffer, we'd not have to worry about
   inconsistencies resulting from changes to in-flight buffers,
   as they are all private.
 * if we can use some efficient combination of digital tree,
   btree and hash table to track which block is where,
   we might be able to track a large, staged log device
   as a sort of log-structured block device, making snapshots after the
   fact for data generations still covered by the log very easy.
 * we need a good refcount scheme on the ram buffers.

of course we can start out "simple", and just provide a static cache, no
ring buffer or anything.

this should probably be implemented as a generic device-mapper target,
which also makes testing much easier.

which would make it possible to even add it to the current drbd
by just stacking it in front of the "lower-level device".

for the "write-back" to "write-through" change,
we only need a minimal change in the current drbd module, which we can
enable based on the type of the device directly below us.
we could detect whether its a device-mapper target,
if so, which one, and access its special methods if any.

this still sound a little quirky, so I'd suggest to introduce a special
BIO_RW_WRITE_THROUGH (to be defined) bit for the bi_flags.

when not using it, that is write back.
when using it, it would trigger a flush of any pending requests,
and a direct remapping to the lower level device.

BIO_RW_BARRIER requests would still need to trigger a flush as well,
and to go straight through.

in the new architecture, where "drbd" probably becomes just a special
implementation and collection of device-mapper targets, communicating
with other device mapper targets becomes more easy (I hope).

does that make sense?

-- 
: Lars Ellenberg                
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
__
please don't Cc me, but send to list   --   I'm subscribed

Re: [Drbd-dev] New Features

[Morey Roof]

This is pretty much what I was thinking.  For the btree, generations, 
and ref-count a good example to look at is how btrfs (This is the other 
project I have started to mess with) works.  The design is very 
efficient and I think we could use a very close match for our setup.  I 
haven't read the paper you sent yet but will get to that today.

Let me know how you would like to start and I can start working a proof 
of concept and we can see how to go from there.

-Morey

Lars Ellenberg wrote:
> On Sat, Sep 20, 2008 at 03:18:21PM +0200, Lars Ellenberg wrote:
>
> "Write-Back" cache:
>
> some things to think of when introducing a write back cache,
>  * need to do some cache coherency protocol
>  * need to track which block is where, so we can read the correct
>    version in case it has not yet been committed to final location
>  * if using a ram buffer as log disk, we need to track the latest
>    position for overwrites.
>  * if we have stages, i.e. ram buffer first, then log disk, then real
>    storage, we are the most flexible.
>    if peers are sufficiently close, we can send_page from the ram
>    buffer (and calculate checksums there, for data integrity).
>    if we use it as ring buffer, we'd not have to worry about
>    inconsistencies resulting from changes to in-flight buffers,
>    as they are all private.
>  * if we can use some efficient combination of digital tree,
>    btree and hash table to track which block is where,
>    we might be able to track a large, staged log device
>    as a sort of log-structured block device, making snapshots after the
>    fact for data generations still covered by the log very easy.
>  * we need a good refcount scheme on the ram buffers.
>
> of course we can start out "simple", and just provide a static cache, no
> ring buffer or anything.
>
> this should probably be implemented as a generic device-mapper target,
> which also makes testing much easier.
>
> which would make it possible to even add it to the current drbd
> by just stacking it in front of the "lower-level device".
>
> for the "write-back" to "write-through" change,
> we only need a minimal change in the current drbd module, which we can
> enable based on the type of the device directly below us.
> we could detect whether its a device-mapper target,
> if so, which one, and access its special methods if any.
>
> this still sound a little quirky, so I'd suggest to introduce a special
> BIO_RW_WRITE_THROUGH (to be defined) bit for the bi_flags.
>
> when not using it, that is write back.
> when using it, it would trigger a flush of any pending requests,
> and a direct remapping to the lower level device.
>
> BIO_RW_BARRIER requests would still need to trigger a flush as well,
> and to go straight through.
>
> in the new architecture, where "drbd" probably becomes just a special
> implementation and collection of device-mapper targets, communicating
> with other device mapper targets becomes more easy (I hope).
>
> does that make sense?
>
>   

Re: [Drbd-dev] New Features: write-back cache mode

[Lars Ellenberg]

On Sat, Sep 20, 2008 at 05:04:44PM -0600, Morey Roof wrote:
> This is pretty much what I was thinking.  For the btree, generations,  
> and ref-count a good example to look at is how btrfs (This is the other  
> project I have started to mess with) works.  The design is very  
> efficient and I think we could use a very close match for our setup.  I  
> haven't read the paper you sent yet but will get to that today.
>
> Let me know how you would like to start and I can start working a proof  
> of concept and we can see how to go from there.

I still think that should be implemented as a
generic device-mapper write-back target,
not as part of drbd.

there is already some caching target out there somewhere,
using small local low-latency disks as cache
for higher latency "remote" iSCSI or GNBD disks.
At least I read something like that
on one of the linux thin clients projects a while ago.
I did not look at the code, and never used it, though.
a 10 second google suggests that
http://www.acis.ufl.edu/~ming/dmcache/
is what I mean.

maybe you can use that as a starting point,
and point it to a ram disk as cache?


-- 
: Lars Ellenberg                
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
__
please don't Cc me, but send to list   --   I'm subscribed

Re: [Drbd-dev] New Features: write-back cache mode

[Lars Ellenberg]

On Sun, Sep 21, 2008 at 01:19:38AM +0200, Lars Ellenberg wrote:
> On Sat, Sep 20, 2008 at 05:04:44PM -0600, Morey Roof wrote:
> > This is pretty much what I was thinking.  For the btree, generations,  
> > and ref-count a good example to look at is how btrfs (This is the other  
> > project I have started to mess with) works.  The design is very  
> > efficient and I think we could use a very close match for our setup.  I  
> > haven't read the paper you sent yet but will get to that today.
> >
> > Let me know how you would like to start and I can start working a proof  
> > of concept and we can see how to go from there.
> 
> I still think that should be implemented as a
> generic device-mapper write-back target,
> not as part of drbd.

> http://www.acis.ufl.edu/~ming/dmcache/

> maybe you can use that as a starting point,
> and point it to a ram disk as cache?

any news on that?

	Lars