Status of Fencing

Status of Fencing

[Mandell Degerness]

I was wondering if there was an update on the RBD fencing question.
Either knowing when an RBD is mounted elsewhere or being able to
enforce a fence on an RBD would be really helpful.

-Mandell Degerness

Re: Status of Fencing

[Josh Durgin]

On 09/14/2012 05:14 PM, Mandell Degerness wrote:
> I was wondering if there was an update on the RBD fencing question.
> Either knowing when an RBD is mounted elsewhere or being able to
> enforce a fence on an RBD would be really helpful.
>
> -Mandell Degerness

Advisory locking of rbd images is ready for review in the 
wip-librbd-locking branch. This lets you lock/unlock images with the rbd 
cli tool,
so you could detect when it's being used, but fencing is not
implemented yet.

Josh

Re: Status of Fencing

[Sage Weil]

On Fri, 14 Sep 2012, Josh Durgin wrote:
> On 09/14/2012 05:14 PM, Mandell Degerness wrote:
> > I was wondering if there was an update on the RBD fencing question.
> > Either knowing when an RBD is mounted elsewhere or being able to
> > enforce a fence on an RBD would be really helpful.
> > 
> > -Mandell Degerness
> 
> Advisory locking of rbd images is ready for review in the wip-librbd-locking
> branch. This lets you lock/unlock images with the rbd cli tool,
> so you could detect when it's being used, but fencing is not
> implemented yet.

After talking it over with a few people, I have some confidence that the 
'incremental' fencing scheme I suggested a few weeks back will work... 
certainly with btrfs, and almost certainly with XFS.  Basically, this 
amounts to:

 - identify old rbd lock holder (rbd lock list <img>)
 - blacklist old owner (ceph osd blacklist add <addr>)
 - break old rbd lock (rbd lock remove <img> <lockid> <addr>)
 - lock rbd image on new host (rbd lock add <img> <lockid>)
 - map rbd image on new host

The trick is making sure that the new image owner knows about the osdmap 
that includes the new blacklist... this ensures that any rbd objects that 
client reads is no longer touchable by the old guy.  This is probably 
always the case if the mounting/mapping kernel is intracting with the 
cluster for the first time, but I'd like to have a positive confirmation 
of that, and/or a general way to make sure a client has at last a given 
osdmap even in the case where it already has a cluster session (because 
another image is mapped).

Anyway, enough pieces are already in place that you should be able to play 
with this.  The main piece of work we need for this scheme is a test that 
continually yanks a filesystem away from a VM/host, remounts, and 
continues, while verifying things don't go sour.

Moving forward, we also need to make the rbd clients take/release locks 
themselves (instead of the user doing so explicitly), and make the 
'takeover' process a bit more streamlined.

sage