[Cluster-devel] [PATCH 3/4 Revised] NLM - kernel lockd-statd changes

[Cluster-devel] [PATCH 3/4 Revised] NLM - kernel lockd-statd changes

[Wendy Cheng]

This kernel patch (based on 2.6.21-rc4) should be paired with nfs-utils 
user mode changes (patch 4-4, based on nfs-utils-1.1.0-rc1) that is 
optional. If changes made in patch 4-4 is not presented in nfs-utils, 
the rpc.statd will ignore whatever this kernel patch does.

The changes record the ip interface that accepts the lock requests and 
passes the correct "my_name" (in standard IPV4 dot notation) to user 
mode statd (instead of system_utsname.nodename). This enables rpc.statd 
to add the correct taken-over IPv4 address into the 3rd parameter of 
ha_callout program. Current nfs-utils always resets "my_name" into 
loopback address (127.0.0.1), regardless the statement made in rpc.statd 
man page. Check out "man rpc.statd" and "man sm-notify" for details.

-- Wendy

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 003_nlm_statd.patch
Type: text/x-patch
Size: 14486 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20070405/7f776025/attachment.bin>

[Cluster-devel] Re: [NFS] [PATCH 3/4 Revised] NLM - kernel lockd-statd changes

[Olaf Kirch]

On Thursday 05 April 2007 23:52, Wendy Cheng wrote:
> The changes record the ip interface that accepts the lock requests and 
> passes the correct "my_name" (in standard IPV4 dot notation) to user 
> mode statd (instead of system_utsname.nodename). This enables rpc.statd 
> to add the correct taken-over IPv4 address into the 3rd parameter of 
> ha_callout program. Current nfs-utils always resets "my_name" into 
> loopback address (127.0.0.1), regardless the statement made in rpc.statd 
> man page. Check out "man rpc.statd" and "man sm-notify" for details.

I don't think this is the right approach. For one, there's not enough
room in the SM_MON request to accomodate an additional IPv6
address, so you would have to come up with something entirely
different for IPv6 anyway. But more importantly, I think we should
move away from associating all sorts of network level addresses
with lockd state - addresses are just smoke and mirrors. Despite
all of NLM/NSMs shortcomings, there's a vehicle to convey identity,
and that's mon_name. IMHO the focus should be on making it work
properly if it doesn't do what you do.

But - why do you need to record the address on which the request was
received. at all? Don't you know beforehand on which IP addresses you
will be servicing NFS requests, and which will need to be migrated?

Side note: should we think about replacing SM_MON with some new
design altogether (think netlink)?

Olaf
-- 
Olaf Kirch  |  --- o --- Nous sommes du soleil we love when we play
okir at lst.de |    / | \   sol.dhoop.naytheet.ah kin.ir.samse.qurax

[Cluster-devel] Re: [NFS] [PATCH 3/4 Revised] NLM - kernel lockd-statd changes

[Lon Hohberger]

On Tue, Apr 10, 2007 at 11:09:43AM +0200, Olaf Kirch wrote:
> On Thursday 05 April 2007 23:52, Wendy Cheng wrote:
> > The changes record the ip interface that accepts the lock requests and 
> > passes the correct "my_name" (in standard IPV4 dot notation) to user 
> > mode statd (instead of system_utsname.nodename). This enables rpc.statd 
> > to add the correct taken-over IPv4 address into the 3rd parameter of 
> > ha_callout program. Current nfs-utils always resets "my_name" into 
> > loopback address (127.0.0.1), regardless the statement made in rpc.statd 
> > man page. Check out "man rpc.statd" and "man sm-notify" for details.
> 
> I don't think this is the right approach. For one, there's not enough
> room in the SM_MON request to accomodate an additional IPv6
> address, so you would have to come up with something entirely
> different for IPv6 anyway.

This is true.

> But more importantly, I think we should
> move away from associating all sorts of network level addresses
> with lockd state - addresses are just smoke and mirrors.

> Despite
> all of NLM/NSMs shortcomings, there's a vehicle to convey identity,
> and that's mon_name. IMHO the focus should be on making it work
> properly if it doesn't do what you do.

We'd have to give it an arbitrary name, completely disassociated from
all network addresses/hostnames/etc.  That could work.  The problems
don't go away, though - they just become different:

* multiple mon_names must be able to exist per-server, since
services in a cluster are not always advertised on the same node (and,
of course, multiple NFS services may exist and *MUST* operate
independently without affecting one-another)

* we have to tell clients our mon_name somehow; since it's not
associated with a server or an IP address

I guess the question is: How is mon_name determined currently by the
clients?


> But - why do you need to record the address on which the request was
> received. at all? Don't you know beforehand on which IP addresses you
> will be servicing NFS requests, and which will need to be migrated?

Here's an answer to the 'why'.  [Clearly, this is an IPv4-centric
example, but it's been implemented this way in the past, so we'll use
it.]

It matters if you have multiple virtual IPs servicing multiple file
systems.  Here's an overly complicated example, which indicates a 'why'
we might need per-address monitoring:

ip address 1
ip address 2
ip address 3
export 1 (file system 1)
export 2 (file system 1)
export 3 (file system 2)
export 4 (file system 2)
export 5 (file system 3)

client A mounts export 1 and 3 via IP address 1
client A mounts export 5 via IP address 2
client B mounts export 2 and 4 via IP address 2
client B mounts export 5 via IP address 1
client C mounts export 5 via IP address 3

Assume locks are taken in all cases.

The mapping we need to know becomes:

file system 1:
   client A via IP 1
   client B via IP 2
file system 2:
   client A via IP 1
   client B via IP 2
file system 3:
   client A via IP 2
   client B via IP 1
   client C via IP 3

For *correct* reclaim (no extraneous SM_NOTIFY requests to the wrong
clients, SM_NOTIFY correctly sent to each client), we must send the
following using the NSM design:

   SM_NOTIFY to client A via IP 1
   SM_NOTIFY to client A via IP 2
   SM_NOTIFY to client B via IP 1
   SM_NOTIFY to client B via IP 2
   SM_NOTIFY to client C via IP 3

Currently, if we do it by file system, fsid, etc - there is no
indication of the path SM_NOTIFY messages need to take.  I.e. If we
send to all clients via any old IP address for a specific file system,
rpc.statd on the remote host will drop the request, and locks won't get
reclaimed.  

One beautiful thing about the above (perhaps otherwise ugly) approach
(storing the entire fs/client/server-ip mapping) is that we maintain
compatibility with other NFS implementations.  We don't break anything
from the client's perspective; it works like it always did.

If we were able to use mon_name for the above set, it would be a single
message per client.  On the client side, there would need to be a
mapping of the mon_name to each network-level address.  When we send
SM_NOTIFY to each client, it can then reclaim the locks for all
associated server addresses (which may be different network protocols).

> Side note: should we think about replacing SM_MON with some new
> design altogether

Possibly.

> (think netlink)?

Redesign of the SM_MON messaging doesn't necessarily require rewriting
the underlying lockd->statd communication path.  That said, I have no
opinion about the merits (or not) of using netlink over the current
implementation.

We could also make a file system...

-- Lon

-- 
Lon Hohberger - Software Engineer - Red Hat, Inc.

[Cluster-devel] Re: [NFS] [PATCH 3/4 Revised] NLM - kernel lockd-statd changes

[Wendy Cheng]

Olaf Kirch wrote:
> On Thursday 05 April 2007 23:52, Wendy Cheng wrote:
>   
>> The changes record the ip interface that accepts the lock requests and 
>> passes the correct "my_name" (in standard IPV4 dot notation) to user 
>> mode statd (instead of system_utsname.nodename). This enables rpc.statd 
>> to add the correct taken-over IPv4 address into the 3rd parameter of 
>> ha_callout program. Current nfs-utils always resets "my_name" into 
>> loopback address (127.0.0.1), regardless the statement made in rpc.statd 
>> man page. Check out "man rpc.statd" and "man sm-notify" for details.
>>     
>
> I don't think this is the right approach. For one, there's not enough
> room in the SM_MON request to accomodate an additional IPv6
> address, so you would have to come up with something entirely
> different for IPv6 anyway. 

The original plan was to pass fsid instead of floating ip but it 
required some major restructures on host lookup and file lookup (in 
nlmsvc_retrieve_args). I have been hoping by the time IPV6 is really 
required, NFS V4 would be mature enough to get deployed (so this would 
be a non-issue anyway).

If people doesn't mind to restructure the sequence of host and file 
lookup, passing fsid can be one of the strong candidates to get this right.

> But more importantly, I think we should
> move away from associating all sorts of network level addresses
> with lockd state - addresses are just smoke and mirrors. Despite
> all of NLM/NSMs shortcomings, there's a vehicle to convey identity,
> and that's mon_name. IMHO the focus should be on making it work
> properly if it doesn't do what you do.
>
> But - why do you need to record the address on which the request was
> received. at all? Don't you know beforehand on which IP addresses you
> will be servicing NFS requests, and which will need to be migrated?
>
> Side note: should we think about replacing SM_MON with some new
> design altogether (think netlink)?
>
>   
Totally agree ! More on this later when I'm back to office.

-- Wendy

[Cluster-devel] Re: [NFS] [PATCH 3/4 Revised] NLM - kernel lockd-statd changes

[Wendy Cheng]

Olaf Kirch wrote:
> But - why do you need to record the address on which the request was
> received. at all? Don't you know beforehand on which IP addresses you
> will be servicing NFS requests, and which will need to be migrated?
>
>   

Sorry, somehow I missed this question ...

The intent is to avoid sending notifications to *all* clients.  The 
cluster user mode scripts will have to associate the floating IP(s) with 
fsid(s) and relies heavily on the information passed by rpc.statd's 
ha-callout program. My test ha-callout user mode program shows the 
following:

/mnt/lockd/ha-pg/hatest is invoked with 3 arguments.
    add-client
    192.168.24.146
    192.168.24.149
/mnt/lockd/ha-pg/hatest is invoked with 3 arguments.
    del-client
    192.168.24.145
    192.168.24.149

These info(s) will be used to construct the notification list when 
notification actually take places.

-- Wendy

[Cluster-devel] Re: [NFS] [PATCH 3/4 Revised] NLM - kernel lockd-statd changes

[Neil Brown]

On Tuesday April 10, okir at lst.de wrote:
> On Thursday 05 April 2007 23:52, Wendy Cheng wrote:
> > The changes record the ip interface that accepts the lock requests and 
> > passes the correct "my_name" (in standard IPV4 dot notation) to user 
> > mode statd (instead of system_utsname.nodename). This enables rpc.statd 
> > to add the correct taken-over IPv4 address into the 3rd parameter of 
> > ha_callout program. Current nfs-utils always resets "my_name" into 
> > loopback address (127.0.0.1), regardless the statement made in rpc.statd 
> > man page. Check out "man rpc.statd" and "man sm-notify" for details.
> 
> I don't think this is the right approach. For one, there's not enough
> room in the SM_MON request to accomodate an additional IPv6
> address, so you would have to come up with something entirely
> different for IPv6 anyway. But more importantly, I think we should
> move away from associating all sorts of network level addresses
> with lockd state - addresses are just smoke and mirrors. Despite
> all of NLM/NSMs shortcomings, there's a vehicle to convey identity,
> and that's mon_name. IMHO the focus should be on making it work
> properly if it doesn't do what you do.

I don't understand your complaint.
You say there's "not enough room", but the extra information is being
placed in the 'my_name' string which is up to 1024 bytes long, which
I think is long enough.

You say that "mon_name" is the vehicle to convey identity and while
that is true, I don't think it is relevant as it is the identity of
the *server* that is being conveyed, rather than the identity of the
client (this on an NFS server).

Think of it like this.  The goal is (appears to be) to make it
possible to implement multiple independent NFS servers on the one
Linux host.
As a simplification, each server serves precisely one filesystem which
no other server serves, and each server has precisely one network
address which no other server shares.

So the 'servers' can be identified either by the filesystem (or FSID)
or by the network address.

Most NFS operations are file-local or at most filesystem-local and so
they don't need to care that there are multiple servers.  But locking
and peer-restart in particular is not.  It is server-local.  So for
the peer monitoring/notification operations, we need to enhance to
model to make the server name explicit rather than implicit ('this
host').

To allow a 'server' to migrate from one host to another (active-active
failover) we need to synthesise a reboot which means knowing which
clients are using which server.

lockd knows which is which either based on the destination network
address of the lock request, or the filesystem on which the lock was
taken.   Somehow this information needs to get communicated to statd
so that different 'sm' directories can be used.  my_name seems a
sensible place to put the information.


However:  now that I think I actually understand what is happening, I
wonder if FSID and IPaddress are really the right handles to use.  It
would seem to make more sense to use the filesystem name (i.e. a
path).

So I'm suggesting writing a directory name to
    /proc/fs/nfsd/nlm_unlock
and maybe also to
    /proc/fs/nlm_restart_grace_for_fs

and have 'my_name' in the SM_MON request be the path name of the
export point rather the network address.

Thinking more,  lockd would need to know whether each filesystem is an
independent server so that it knows if independent nsm objects are
needed.

So maybe we want an export flag "active_failover".
If this is set then the filesystem has an independent grace period
that starts on first export (except that 'first export' isn't really a
well defined concept) and lockd treats clients using that filesystem
as different from the same client using any other filesystem.

I'm not sure we really need the 'nlm_unlock' interface either.  We can
just synthesis incoming SM_NOTIFYs from all clients of that filesystem
and the locks will go away.

Not that I'm saying we have to use this approach rather than the
current one.  I'm just exploring the issue and making sure that I
understand it.


> 
> But - why do you need to record the address on which the request was
> received. at all? Don't you know beforehand on which IP addresses you
> will be servicing NFS requests, and which will need to be migrated?
> 
> Side note: should we think about replacing SM_MON with some new
> design altogether (think netlink)?

Well, I want something new to support the various state that needs to
be recorded by NFSv4, and what ever gets created could probably be
used for lockd/statd too.
But given that we have SM_MON implemented, what is so broken that it
needs to be replaced?

NeilBrown

[Cluster-devel] Re: [NFS] [PATCH 3/4 Revised] NLM - kernel lockd-statd changes

[Lon Hohberger]

On Wed, Apr 11, 2007 at 02:50:04PM +1000, Neil Brown wrote:
> Think of it like this.  The goal is (appears to be) to make it
> possible to implement multiple independent NFS servers on the one
> Linux host.
> As a simplification, each server serves precisely one filesystem which
> no other server serves, and each server has precisely one network
> address which no other server shares.

The simplification here is a bit incorrect; we need to be able to handle
a given *set* of size >= 1 IPs which is paired with a *set* of size >= 1
of fsids.  The sets are not broken up (that is, we won't be moving 
file systems between IP addresses or vice-vers), but the number of
IPs:filesystems need not be 1:1.

   * 1 file system, 1 ip address
   * 2 fs, 1 ip
   * 1 ip, 2 fs
   * 18 ip, 42 fs.

... should all work without extraneous stuff being done.


> lockd knows which is which either based on the destination network
> address of the lock request, or the filesystem on which the lock was
> taken.   Somehow this information needs to get communicated to statd
> so that different 'sm' directories can be used.  my_name seems a
> sensible place to put the information.

Right.  Using the inbound-destination address instead of fsid or file
system will ensure the right address is used for sending the SM_NOTIFY
requests.  However, fsid, or export path - on a service with more than
one network address will not, I think...

Obviously, I agree with Olaf in that using all sorts of random network
protocol addresses is a hair ugly - but I think it's better than having
to send out multiple SM_NOTIFY messages on multiple net addrs just to
ensure that a particular client gets SM_NOTIFY from the *right* net
addr that it had in its monitor list...

If we could have a single 'name' per service that can get associated
with multiple network addresses, I'm all for it.

Just my opinion, of course.

> However:  now that I think I actually understand what is happening, I
> wonder if FSID and IPaddress are really the right handles to use.  It
> would seem to make more sense to use the filesystem name (i.e. a
> path).

This doesn't solve the 2 ip : 1 fs case very cleanly.  We need to send
SM_NOTIFY out on multiple IPs to ensure that each node gets it from the
right server address (that the client has on its monitor list)...?

Really, the goal is to have:

  * clients receive SM_NOTIFY from the ("virtual") host(s) they are
    monitoring when we start the service
  * No extraneous SM_NOTIFY requests are sent by server.
  * No extraneous SM_NOTIFY requests are received by clients.

-- 
Lon Hohberger - Software Engineer - Red Hat, Inc.

[Cluster-devel] Re: [NFS] [PATCH 3/4 Revised] NLM - kernel lockd-statd changes

[Wendy Cheng]

Lon Hohberger wrote:
> On Wed, Apr 11, 2007 at 02:50:04PM +1000, Neil Brown wrote:
>   
>> lockd knows which is which either based on the destination network
>> address of the lock request, or the filesystem on which the lock was
>> taken.   Somehow this information needs to get communicated to statd
>> so that different 'sm' directories can be used.  my_name seems a
>> sensible place to put the information.
>>     
>
> Right.  Using the inbound-destination address instead of fsid or file
> system will ensure the right address is used for sending the SM_NOTIFY
> requests.  However, fsid, or export path - on a service with more than
> one network address will not, I think...
>   

So I think we conclude here that inbound-destination address is required 
for SM_NOTIFY purpose.

Thanks for all the comments. Will summarize the issues on Monday and see 
what we should go from there.

-- Wendy

[Cluster-devel] Re: [NFS] [PATCH 3/4 Revised] NLM - kernel lockd-statd changes

[Olaf Kirch]

On Friday 13 April 2007 21:16, Lon Hohberger wrote:
> The simplification here is a bit incorrect; we need to be able to handle
> a given *set* of size >= 1 IPs which is paired with a *set* of size >= 1
> of fsids.  The sets are not broken up (that is, we won't be moving 
> file systems between IP addresses or vice-vers), but the number of
> IPs:filesystems need not be 1:1.
> 
>    * 1 file system, 1 ip address
>    * 2 fs, 1 ip
>    * 1 ip, 2 fs
>    * 18 ip, 42 fs.

So you move a bunch of file systems with a bunch of IPs, and that
information is recorded somewhere. So if you know you're moving
set X, which goes with IPs 1.2.3.4 and 1.2.3.5, you might as well
send notifications from both these IPs to all clients having established
locks on a file system from that set.

I think in term of correctness, it's better to send an SM_NOTIFY
for each IP associated with such a set, anyway.

To me, it seems the crucial point of information here when monitoring
a client, is which file system (or set of file systems) the client has
accessed.

Olaf
-- 
Olaf Kirch  |  --- o --- Nous sommes du soleil we love when we play
okir at lst.de |    / | \   sol.dhoop.naytheet.ah kin.ir.samse.qurax

[Cluster-devel] Re: [NFS] [PATCH 3/4 Revised] NLM - kernel lockd-statd changes

[Wendy Cheng]

Olaf Kirch wrote:
> On Friday 13 April 2007 21:16, Lon Hohberger wrote:
>   
>> The simplification here is a bit incorrect; we need to be able to handle
>> a given *set* of size >= 1 IPs which is paired with a *set* of size >= 1
>> of fsids.  The sets are not broken up (that is, we won't be moving 
>> file systems between IP addresses or vice-vers), but the number of
>> IPs:filesystems need not be 1:1.
>>
>>    * 1 file system, 1 ip address
>>    * 2 fs, 1 ip
>>    * 1 ip, 2 fs
>>    * 18 ip, 42 fs.
>>     
>
> So you move a bunch of file systems with a bunch of IPs, and that
> information is recorded somewhere. So if you know you're moving
> set X, which goes with IPs 1.2.3.4 and 1.2.3.5, you might as well
> send notifications from both these IPs to all clients having established
> locks on a file system from that set.
>
> I think in term of correctness, it's better to send an SM_NOTIFY
> for each IP associated with such a set, anyway.
>   

That's exactly what we have been proposing... :) .. We'll rely heavily 
on HA callout program to tell us which client uses which (server) 
floating IP. 
> To me, it seems the crucial point of information here when monitoring
> a client, is which file system (or set of file systems) the client has
> accessed.
>
>   
The summary will be sent out shortly ... sorry - it is a little bit 
behind schedules due to other issues.

-- Wendy

[Cluster-devel] Re: [NFS] [PATCH 3/4 Revised] NLM - kernel lockd-statd changes

[Olaf Kirch]

On Tuesday 17 April 2007 15:24, Wendy Cheng wrote:
> > I think in term of correctness, it's better to send an SM_NOTIFY
> > for each IP associated with such a set, anyway.
> >   
> 
> That's exactly what we have been proposing... :) .. We'll rely heavily 
> on HA callout program to tell us which client uses which (server) 
> floating IP. 

Correct me if I'm wrong, but it seems your patch records every IP
used by the client, rather than the *all* the IPs related to the set of
file systems being moved. So if there are several IPs for this set, you
will end up sending notifications only from those the client happened
to have talked to.

My point was, you move a set of file systems A, B and C, with
IPs X, Y, Z. You know what the addresses are, so from a
robustness point of view your best bet is to send SM_NOTIFY
messages from IPs X, Y, Z, regardless of whether the client has
been talking to all of them, or just one.

Olaf
-- 
Olaf Kirch  |  --- o --- Nous sommes du soleil we love when we play
okir at lst.de |    / | \   sol.dhoop.naytheet.ah kin.ir.samse.qurax

[Cluster-devel] Re: [NFS] [PATCH 3/4 Revised] NLM - kernel lockd-statd changes

[Wendy Cheng]

Olaf Kirch wrote:
> On Tuesday 17 April 2007 15:24, Wendy Cheng wrote:
>   
>>> I think in term of correctness, it's better to send an SM_NOTIFY
>>> for each IP associated with such a set, anyway.
>>>   
>>>       
>> That's exactly what we have been proposing... :) .. We'll rely heavily 
>> on HA callout program to tell us which client uses which (server) 
>> floating IP. 
>>     
>
> Correct me if I'm wrong, but it seems your patch records every IP
> used by the client, rather than the *all* the IPs related to the set of
> file systems being moved. So if there are several IPs for this set, you
> will end up sending notifications only from those the client happened
> to have talked to.
>   

Yes. Then why should we SM_NOTIFY the clients that do not have the 
associated locks (and introducing more possible reclaiming delay) ? Be 
aware that failover normally has timing constraints - it needs to get 
finished within a sensible time interval.

-- Wendy
> My point was, you move a set of file systems A, B and C, with
> IPs X, Y, Z. You know what the addresses are, so from a
> robustness point of view your best bet is to send SM_NOTIFY
> messages from IPs X, Y, Z, regardless of whether the client has
> been talking to all of them, or just one.
>
> Olaf
>