[RFC] After nfs restart, locks can't be recovered which record by lockd before

[RFC] After nfs restart, locks can't be recovered which record by lockd before

[Mi Jinlong]

Hi,

When testing the nfs's lock at NFSv3, I get a problem.
So I want someone help me, thanks!

Test Process:
  Step1, ClientA get an exclusive lock success.
  Step2, Using command "service nfs restart" to restart server's nfs se=
rvice.
  Step3. ClientB get lock should fail, but success.

I think after step2 (nfs service restart), clientA's lock should be rec=
overed.
But like above, clientA's lock doesn=E2=80=99t be recovered.

When tracing the kernel, I find nfsd will cause lockd stop when it stop=
=2E
When lockd stop, all locks will be release which is record before at lo=
ckd.

When nfsd start, the lockd will start also, but the statd don't know wh=
at happened
 at kernel, so after that, locks will be lost.

Is it right when nfs stop, the lockd will stop too?=20
If it's right, should locks be recovered after lockd start?

thanks,
Mi Jinlong

Re: [RFC] After nfs restart, locks can't be recovered which record by lockd before

[Jeff Layton]

On Wed, 13 Jan 2010 17:51:25 +0800
Mi Jinlong <mijinlong@cn.fujitsu.com> wrote:

> Hi,
>=20
> When testing the nfs's lock at NFSv3, I get a problem.
> So I want someone help me, thanks!
>=20
> Test Process:
>   Step1, ClientA get an exclusive lock success.
>   Step2, Using command "service nfs restart" to restart server's nfs =
service.

"service nfs restart" on an RH-derived distro is the same as running
"/etc/init.d/nfs restart". Init scripts vary between distros (and even
between releases on the same distro). Since you're asking this in a
more generic forum, you should probably be specific about what's
actually being restarted (and in what order). Understanding that may
also help you answer your own question here.

>   Step3. ClientB get lock should fail, but success.
>=20
> I think after step2 (nfs service restart), clientA's lock should be r=
ecovered.
> But like above, clientA's lock doesn=E2=80=99t be recovered.
>=20
> When tracing the kernel, I find nfsd will cause lockd stop when it st=
op.
> When lockd stop, all locks will be release which is record before at =
lockd.
>=20
> When nfsd start, the lockd will start also, but the statd don't know =
what happened
>  at kernel, so after that, locks will be lost.
>=20
> Is it right when nfs stop, the lockd will stop too?=20
> If it's right, should locks be recovered after lockd start?
>=20

Assuming you're using a RH-derived distro like Fedora or RHEL, then no.
statd is controlled by a separate init script (nfslock) and when you
run "service nfs restart" you're not restarting it. NSM notifications
are not sent and clients generally won't reclaim their locks.

IOW, "you're doing it wrong". If you want locks to be reclaimed then
you probably need to restart the nfslock service too.

--=20
Jeff Layton <jlayton@redhat.com>

Re: [RFC] After nfs restart, locks can't be recovered which record by lockd before

[Chuck Lever]

On 01/13/2010 07:51 AM, Jeff Layton wrote:
> On Wed, 13 Jan 2010 17:51:25 +0800
> Mi Jinlong<mijinlong@cn.fujitsu.com>  wrote:
>
>> Hi,
>>
>> When testing the nfs's lock at NFSv3, I get a problem.
>> So I want someone help me, thanks!
>>
>> Test Process:
>>    Step1, ClientA get an exclusive lock success.
>>    Step2, Using command "service nfs restart" to restart server's nf=
s service.
>
> "service nfs restart" on an RH-derived distro is the same as running
> "/etc/init.d/nfs restart". Init scripts vary between distros (and eve=
n
> between releases on the same distro). Since you're asking this in a
> more generic forum, you should probably be specific about what's
> actually being restarted (and in what order). Understanding that may
> also help you answer your own question here.
>
>>    Step3. ClientB get lock should fail, but success.
>>
>> I think after step2 (nfs service restart), clientA's lock should be =
recovered.
>> But like above, clientA's lock doesn=E2=80=99t be recovered.
>>
>> When tracing the kernel, I find nfsd will cause lockd stop when it s=
top.
>> When lockd stop, all locks will be release which is record before at=
 lockd.
>>
>> When nfsd start, the lockd will start also, but the statd don't know=
 what happened
>>   at kernel, so after that, locks will be lost.
>>
>> Is it right when nfs stop, the lockd will stop too?
>> If it's right, should locks be recovered after lockd start?
>>
>
> Assuming you're using a RH-derived distro like Fedora or RHEL, then n=
o.
> statd is controlled by a separate init script (nfslock) and when you
> run "service nfs restart" you're not restarting it. NSM notifications
> are not sent and clients generally won't reclaim their locks.
>
> IOW, "you're doing it wrong". If you want locks to be reclaimed then
> you probably need to restart the nfslock service too.

Mi Jinlong is exercising another case we know doesn't work right, but w=
e=20
don't expect admins will ever perform this kind of "down-up" on a norma=
l=20
production server.  In other words, we expect it to work this way, and=20
it's been good enough, so far.

As Jeff points out, the "nfs" and the "nfslock" services are separate.=20
This is because "nfslock" is required for both client and server side=20
NFS, but "nfs" is required only on the server.  This split also dictate=
s=20
the way sm-notify works, since it has to behave differently on NFS=20
clients and servers.

Two other points:

   + lockd would not restart itself in this case if there happened to b=
e=20
NFS mounts on that system

   + lockd doesn't currently poke statd when it restarts to tell it to=20
send reboot notifications, but it probably should

We know that lockd will start up when someone mounts the first NFS=20
share, or when the NFS server is started.  If lockd sent statd an=20
SM_SIMU_CRASH (or something like it) every time it cold started, statd=20
could send reboot notifications at the right time on both servers and=20
clients without extra logic in the init scripts, and we wouldn't need=20
that kludge in sm-notify to know when a machine has rebooted.

--=20
chuck[dot]lever[at]oracle[dot]com

Re: [RFC] After nfs restart, locks can't be recovered which record by lockd before

[Mi Jinlong]

Hi Jeff,

Jeff Layton =E5=86=99=E9=81=93:
> On Wed, 13 Jan 2010 17:51:25 +0800
> Mi Jinlong <mijinlong@cn.fujitsu.com> wrote:
>=20
>> Hi,
>>
>> When testing the nfs's lock at NFSv3, I get a problem.
>> So I want someone help me, thanks!
>>
>> Test Process:
>>   Step1, ClientA get an exclusive lock success.
>>   Step2, Using command "service nfs restart" to restart server's nfs=
 service.
>=20
> "service nfs restart" on an RH-derived distro is the same as running
> "/etc/init.d/nfs restart". Init scripts vary between distros (and eve=
n
> between releases on the same distro). Since you're asking this in a
> more generic forum, you should probably be specific about what's
> actually being restarted (and in what order). Understanding that may
> also help you answer your own question here.

  I test it on RHEL5(kernel 2.6.18, nfs-utils 1.0.9) and Fedora 12(kern=
el 2.6.33-rc3,
   nfs-utils 1.2.0).

>=20
>>   Step3. ClientB get lock should fail, but success.
>>
>> I think after step2 (nfs service restart), clientA's lock should be =
recovered.
>> But like above, clientA's lock doesn=E2=80=99t be recovered.
>>
>> When tracing the kernel, I find nfsd will cause lockd stop when it s=
top.
>> When lockd stop, all locks will be release which is record before at=
 lockd.
>>
>> When nfsd start, the lockd will start also, but the statd don't know=
 what happened
>>  at kernel, so after that, locks will be lost.
>>
>> Is it right when nfs stop, the lockd will stop too?=20
>> If it's right, should locks be recovered after lockd start?
>>
>=20
> Assuming you're using a RH-derived distro like Fedora or RHEL, then n=
o.
> statd is controlled by a separate init script (nfslock) and when you
> run "service nfs restart" you're not restarting it. NSM notifications
> are not sent and clients generally won't reclaim their locks.

  Yes, the statd don't restart when nfs restart. But, nfs restart will
  cause lockd restart. If lockd restart, locks which record before will=
 be lost.

> IOW, "you're doing it wrong". If you want locks to be reclaimed then
> you probably need to restart the nfslock service too.
>=20

  Don't you think it's more reasonable that locks should be recovered=20
  when nfs restart cause lockd restart?

--=20
Regards
Mi Jinlong

Re: [RFC] After nfs restart, locks can't be recovered which record by lockd before

[Mi Jinlong]

Hi Chuck,

Chuck Lever =E5=86=99=E9=81=93:
> On 01/13/2010 07:51 AM, Jeff Layton wrote:
>> On Wed, 13 Jan 2010 17:51:25 +0800
>> Mi Jinlong<mijinlong@cn.fujitsu.com>  wrote:
>>
>> Assuming you're using a RH-derived distro like Fedora or RHEL, then =
no.
>> statd is controlled by a separate init script (nfslock) and when you
>> run "service nfs restart" you're not restarting it. NSM notification=
s
>> are not sent and clients generally won't reclaim their locks.
>>
>> IOW, "you're doing it wrong". If you want locks to be reclaimed then
>> you probably need to restart the nfslock service too.
>=20
> Mi Jinlong is exercising another case we know doesn't work right, but=
 we
> don't expect admins will ever perform this kind of "down-up" on a nor=
mal
> production server.  In other words, we expect it to work this way, an=
d
> it's been good enough, so far.
>=20
> As Jeff points out, the "nfs" and the "nfslock" services are separate=
=2E
> This is because "nfslock" is required for both client and server side
> NFS, but "nfs" is required only on the server.  This split also dicta=
tes
> the way sm-notify works, since it has to behave differently on NFS
> clients and servers.
>=20
> Two other points:
>=20
>   + lockd would not restart itself in this case if there happened to =
be
> NFS mounts on that system

  When testing, i find nfs restart will cause lockd restart.
  I find some codes which cause the lock stop when nfs stop.

 At kernel 2.6.18, fs/lockd/svc.c
 ...
 354         if (nlmsvc_users) {
 355                 if (--nlmsvc_users)
 356                         goto out;
 357         } else
 358                 printk(KERN_WARNING "lockd_down: no users! pid=3D%=
d\n", nlmsvc_pid);
 ...
 366=20
 367         kill_proc(nlmsvc_pid, SIGKILL, 1);
 ...
=20
 At kernel 2.6.18, fs/lockd/svc.c
 ...
 344         if (nlmsvc_users) {
 345                 if (--nlmsvc_users)
 346                         goto out;
 347         } else {
 348                 printk(KERN_ERR "lockd_down: no users! task=3D%p\n=
",
 349                         nlmsvc_task);
 350                 BUG();
 351         }
 ....
 357         kthread_stop(nlmsvc_task);
 358         svc_exit_thread(nlmsvc_rqst);
 ...

 As above, when nlmsvc_users <=3D 1, the lockd will be killed.

>=20
>   + lockd doesn't currently poke statd when it restarts to tell it to
> send reboot notifications, but it probably should

  Yes, I agree with you. But now, when some reason cause lockd restart =
but
  statd not restart, locks which hold before will lost.

  Maybe, the kernel should fix this.

>=20
> We know that lockd will start up when someone mounts the first NFS
> share, or when the NFS server is started.  If lockd sent statd an
> SM_SIMU_CRASH (or something like it) every time it cold started, stat=
d
> could send reboot notifications at the right time on both servers and
> clients without extra logic in the init scripts, and we wouldn't need
> that kludge in sm-notify to know when a machine has rebooted.

  What's the meaning of cold start?? System reboot? Or statd reboot?

  I want to know when using cammond "service nfslock restart" restart t=
he=20
  nfslock service(means restart statd and lockd), will the statd call s=
m-notify=20
  to notify other client? Or don't?

  At RHEL5 and Fedora, the sm-notify will be call and send SM-NOTIFY ev=
ery time=20
  when nfslock restart.

thanks,
Mi Jinlong

Re: [RFC] After nfs restart, locks can't be recovered which record by lockd before

[Jeff Layton]

On Thu, 14 Jan 2010 17:41:13 +0800
Mi Jinlong <mijinlong@cn.fujitsu.com> wrote:

> > IOW, "you're doing it wrong". If you want locks to be reclaimed then
> > you probably need to restart the nfslock service too.
> > 
> 
>   Don't you think it's more reasonable that locks should be recovered 
>   when nfs restart cause lockd restart?
> 

Sure, that's the idea. That said, I'm leery of trying to make this too
automatic. Extra complexity in this sort of thing often means that we
get emergent behaviors from the system that we don't expect. I'm not
sure that risk is worth it to guard against admins restarting lockd
like this without forcing notifications to be sent.

-- 
Jeff Layton <jlayton@redhat.com>

Re: [RFC] After nfs restart, locks can't be recovered which record by lockd before

[Chuck Lever]

On Jan 14, 2010, at 5:06 AM, Mi Jinlong wrote:
> Hi Chuck,
>
> Chuck Lever =E5=86=99=E9=81=93:
>> On 01/13/2010 07:51 AM, Jeff Layton wrote:
>>> On Wed, 13 Jan 2010 17:51:25 +0800
>>> Mi Jinlong<mijinlong@cn.fujitsu.com>  wrote:
>>>
>>> Assuming you're using a RH-derived distro like Fedora or RHEL, =20
>>> then no.
>>> statd is controlled by a separate init script (nfslock) and when yo=
u
>>> run "service nfs restart" you're not restarting it. NSM =20
>>> notifications
>>> are not sent and clients generally won't reclaim their locks.
>>>
>>> IOW, "you're doing it wrong". If you want locks to be reclaimed the=
n
>>> you probably need to restart the nfslock service too.
>>
>> Mi Jinlong is exercising another case we know doesn't work right, =20
>> but we
>> don't expect admins will ever perform this kind of "down-up" on a =20
>> normal
>> production server.  In other words, we expect it to work this way, =20
>> and
>> it's been good enough, so far.
>>
>> As Jeff points out, the "nfs" and the "nfslock" services are =20
>> separate.
>> This is because "nfslock" is required for both client and server sid=
e
>> NFS, but "nfs" is required only on the server.  This split also =20
>> dictates
>> the way sm-notify works, since it has to behave differently on NFS
>> clients and servers.
>> Two other points:
>>
>>  + lockd would not restart itself in this case if there happened to =
=20
>> be
>> NFS mounts on that system
>
>  When testing, i find nfs restart will cause lockd restart.
>  I find some codes which cause the lock stop when nfs stop.
>
> At kernel 2.6.18, fs/lockd/svc.c
> ...
> 354         if (nlmsvc_users) {
> 355                 if (--nlmsvc_users)
> 356                         goto out;
> 357         } else
> 358                 printk(KERN_WARNING "lockd_down: no users! pid=3D=
%d=20
> \n", nlmsvc_pid);
> ...
> 366
> 367         kill_proc(nlmsvc_pid, SIGKILL, 1);
> ...
>
> At kernel 2.6.18, fs/lockd/svc.c
> ...
> 344         if (nlmsvc_users) {
> 345                 if (--nlmsvc_users)
> 346                         goto out;
> 347         } else {
> 348                 printk(KERN_ERR "lockd_down: no users! task=3D%p\=
n",
> 349                         nlmsvc_task);
> 350                 BUG();
> 351         }
> ....
> 357         kthread_stop(nlmsvc_task);
> 358         svc_exit_thread(nlmsvc_rqst);
> ...
>
> As above, when nlmsvc_users <=3D 1, the lockd will be killed.
>
>>
>>  + lockd doesn't currently poke statd when it restarts to tell it to
>> send reboot notifications, but it probably should
>
>  Yes, I agree with you. But now, when some reason cause lockd =20
> restart but
>  statd not restart, locks which hold before will lost.
>
>  Maybe, the kernel should fix this.

What did you have in mind?

>> We know that lockd will start up when someone mounts the first NFS
>> share, or when the NFS server is started.  If lockd sent statd an
>> SM_SIMU_CRASH (or something like it) every time it cold started, =20
>> statd
>> could send reboot notifications at the right time on both servers an=
d
>> clients without extra logic in the init scripts, and we wouldn't nee=
d
>> that kludge in sm-notify to know when a machine has rebooted.
>
>  What's the meaning of cold start?? System reboot? Or statd reboot?

cold start meaning that lockd is shutdown and rmmod'd, then started up =
=20
and re-loaded.

This can also happen on a client if all NFS mounts go away.  =20
lockd_down is invoked, and lockd.ko is removed.  On the next NFS =20
mount, lockd is loaded again.

>  I want to know when using cammond "service nfslock restart" restart =
=20
> the
>  nfslock service(means restart statd and lockd), will the statd call =
=20
> sm-notify
>  to notify other client? Or don't?

Currently "service nfslock restart" always causes a notification to be =
=20
sent.  Since "service nfslock restart" causes lockd to drop its locks =20
(I assume that's what that "killproc lockd" does) I guess we need to =20
force reboot notifications here.  (I still argue that removing the =20
pidfile in the "start" case is not correct).

It appears that both the nfs and nfslock start up scripts do something =
=20
to lockd (as well as the case when the number of NFS mounts goes to =20
zero).  However, only the nfslock script forces sm-notify to send =20
notifications.

I suppose a naive fix for your server restart issue might be to add an =
=20
"sm-notify -f" to the "restart" case in /etc/init.d/nfs.  This would =20
cause reboot notifications to be sent if the monitor list was not =20
empty during a server restart.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: [RFC] After nfs restart, locks can't be recovered which record by lockd before

[Mi Jinlong]

Hi Jeff,

Jeff Layton :
> On Thu, 14 Jan 2010 17:41:13 +0800
> Mi Jinlong <mijinlong@cn.fujitsu.com> wrote:
> 
>>> IOW, "you're doing it wrong". If you want locks to be reclaimed then
>>> you probably need to restart the nfslock service too.
>>>
>>   Don't you think it's more reasonable that locks should be recovered 
>>   when nfs restart cause lockd restart?
>>
> 
> Sure, that's the idea. That said, I'm leery of trying to make this too
> automatic. Extra complexity in this sort of thing often means that we
> get emergent behaviors from the system that we don't expect. I'm not
> sure that risk is worth it to guard against admins restarting lockd
> like this without forcing notifications to be sent.
> 

  IMO, when lockd restart without statd restart, lockd should recovered those locks.
  If the NFSv3's designers think it's not needed when the design it, we would respect
  theirs idea at first. But, I think this is indeed a problem!

-- 
Regards
Mi Jinlong

Re: [RFC] After nfs restart, locks can't be recovered which record by lockd before

[Mi Jinlong]

Hi Chuck,

Chuck Lever =E5=86=99=E9=81=93:
> On Jan 14, 2010, at 5:06 AM, Mi Jinlong wrote:
>> Hi Chuck,
>>
>> Chuck Lever =E5=86=99=E9=81=93:
>>> On 01/13/2010 07:51 AM, Jeff Layton wrote:
>>>> On Wed, 13 Jan 2010 17:51:25 +0800
>>>> Mi Jinlong<mijinlong@cn.fujitsu.com>  wrote:
>>>>
>>>> Assuming you're using a RH-derived distro like Fedora or RHEL, the=
n no.
>>>> statd is controlled by a separate init script (nfslock) and when y=
ou
>>>> run "service nfs restart" you're not restarting it. NSM notificati=
ons
>>>> are not sent and clients generally won't reclaim their locks.
>>>>
>>>> IOW, "you're doing it wrong". If you want locks to be reclaimed th=
en
>>>> you probably need to restart the nfslock service too.
>>>
>>> Mi Jinlong is exercising another case we know doesn't work right, b=
ut we
>>> don't expect admins will ever perform this kind of "down-up" on a n=
ormal
>>> production server.  In other words, we expect it to work this way, =
and
>>> it's been good enough, so far.
>>>
>>> As Jeff points out, the "nfs" and the "nfslock" services are separa=
te.
>>> This is because "nfslock" is required for both client and server si=
de
>>> NFS, but "nfs" is required only on the server.  This split also dic=
tates
>>> the way sm-notify works, since it has to behave differently on NFS
>>> clients and servers.
>>> Two other points:
>>>
>>>  + lockd would not restart itself in this case if there happened to=
 be
>>> NFS mounts on that system
>>
>>  When testing, i find nfs restart will cause lockd restart.
>>  I find some codes which cause the lock stop when nfs stop.
>>
>> At kernel 2.6.18, fs/lockd/svc.c
>> ...
>> 354         if (nlmsvc_users) {
>> 355                 if (--nlmsvc_users)
>> 356                         goto out;
>> 357         } else
>> 358                 printk(KERN_WARNING "lockd_down: no users!
>> pid=3D%d\n", nlmsvc_pid);
>> ...
>> 366
>> 367         kill_proc(nlmsvc_pid, SIGKILL, 1);
>> ...
>>
>> At kernel 2.6.18, fs/lockd/svc.c
>> ...
>> 344         if (nlmsvc_users) {
>> 345                 if (--nlmsvc_users)
>> 346                         goto out;
>> 347         } else {
>> 348                 printk(KERN_ERR "lockd_down: no users! task=3D%p=
\n",
>> 349                         nlmsvc_task);
>> 350                 BUG();
>> 351         }
>> ....
>> 357         kthread_stop(nlmsvc_task);
>> 358         svc_exit_thread(nlmsvc_rqst);
>> ...
>>
>> As above, when nlmsvc_users <=3D 1, the lockd will be killed.
>>
>>>
>>>  + lockd doesn't currently poke statd when it restarts to tell it t=
o
>>> send reboot notifications, but it probably should
>>
>>  Yes, I agree with you. But now, when some reason cause lockd restar=
t but
>>  statd not restart, locks which hold before will lost.
>>
>>  Maybe, the kernel should fix this.
>=20
> What did you have in mind?

  I think when lockd restart, statd should restart too and sent sm-noti=
fy to other client.
  But now, in kernel and nfs-uitls, it don't implemented.
  As the communication style between lockd and statd, this is indeed no=
t easy to implement it.

  So, I think it's should more easy to implement it through the mechani=
sm that exposes
  the kernel's nlm_host cache via /sys you show me before.

>=20
>>> We know that lockd will start up when someone mounts the first NFS
>>> share, or when the NFS server is started.  If lockd sent statd an
>>> SM_SIMU_CRASH (or something like it) every time it cold started, st=
atd
>>> could send reboot notifications at the right time on both servers a=
nd
>>> clients without extra logic in the init scripts, and we wouldn't ne=
ed
>>> that kludge in sm-notify to know when a machine has rebooted.
>>
>>  What's the meaning of cold start?? System reboot? Or statd reboot?
>=20
> cold start meaning that lockd is shutdown and rmmod'd, then started u=
p
> and re-loaded.
>=20
> This can also happen on a client if all NFS mounts go away.  lockd_do=
wn
> is invoked, and lockd.ko is removed.  On the next NFS mount, lockd is
> loaded again.

  Thanks.

>=20
>>  I want to know when using cammond "service nfslock restart" restart=
 the
>>  nfslock service(means restart statd and lockd), will the statd call
>> sm-notify
>>  to notify other client? Or don't?
>=20
> Currently "service nfslock restart" always causes a notification to b=
e
> sent.  Since "service nfslock restart" causes lockd to drop its locks=
 (I
> assume that's what that "killproc lockd" does) I guess we need to for=
ce
> reboot notifications here.  (I still argue that removing the pidfile =
in
> the "start" case is not correct).
>=20
> It appears that both the nfs and nfslock start up scripts do somethin=
g
> to lockd (as well as the case when the number of NFS mounts goes to
> zero).  However, only the nfslock script forces sm-notify to send
> notifications.

  But, at RHLE5 and Fedora, when using cammond "service nfslock restart=
" restart
  the nfslock service, the lockd isn't shutdown and rmmod'd.

  Is it a bug?

>=20
> I suppose a naive fix for your server restart issue might be to add a=
n
> "sm-notify -f" to the "restart" case in /etc/init.d/nfs.  This would
> cause reboot notifications to be sent if the monitor list was not emp=
ty
> during a server restart.

  Hehe. ^/^

thinks,
Mi Jinlong

Re: [RFC] After nfs restart, locks can't be recovered which record by lockd before

[Chuck Lever]

On Jan 15, 2010, at 4:35 AM, Mi Jinlong wrote:
> Hi Chuck,
>
> Chuck Lever =E5=86=99=E9=81=93:
>> On Jan 14, 2010, at 5:06 AM, Mi Jinlong wrote:
>>> Hi Chuck,
>>>
>>> Chuck Lever =E5=86=99=E9=81=93:
>>>> On 01/13/2010 07:51 AM, Jeff Layton wrote:
>>>>> On Wed, 13 Jan 2010 17:51:25 +0800
>>>>> Mi Jinlong<mijinlong@cn.fujitsu.com>  wrote:
>>>>>
>>>>> Assuming you're using a RH-derived distro like Fedora or RHEL, =20
>>>>> then no.
>>>>> statd is controlled by a separate init script (nfslock) and when =
=20
>>>>> you
>>>>> run "service nfs restart" you're not restarting it. NSM =20
>>>>> notifications
>>>>> are not sent and clients generally won't reclaim their locks.
>>>>>
>>>>> IOW, "you're doing it wrong". If you want locks to be reclaimed =20
>>>>> then
>>>>> you probably need to restart the nfslock service too.
>>>>
>>>> Mi Jinlong is exercising another case we know doesn't work right, =
=20
>>>> but we
>>>> don't expect admins will ever perform this kind of "down-up" on a =
=20
>>>> normal
>>>> production server.  In other words, we expect it to work this =20
>>>> way, and
>>>> it's been good enough, so far.
>>>>
>>>> As Jeff points out, the "nfs" and the "nfslock" services are =20
>>>> separate.
>>>> This is because "nfslock" is required for both client and server =20
>>>> side
>>>> NFS, but "nfs" is required only on the server.  This split also =20
>>>> dictates
>>>> the way sm-notify works, since it has to behave differently on NFS
>>>> clients and servers.
>>>> Two other points:
>>>>
>>>> + lockd would not restart itself in this case if there happened =20
>>>> to be
>>>> NFS mounts on that system
>>>
>>> When testing, i find nfs restart will cause lockd restart.
>>> I find some codes which cause the lock stop when nfs stop.
>>>
>>> At kernel 2.6.18, fs/lockd/svc.c
>>> ...
>>> 354         if (nlmsvc_users) {
>>> 355                 if (--nlmsvc_users)
>>> 356                         goto out;
>>> 357         } else
>>> 358                 printk(KERN_WARNING "lockd_down: no users!
>>> pid=3D%d\n", nlmsvc_pid);
>>> ...
>>> 366
>>> 367         kill_proc(nlmsvc_pid, SIGKILL, 1);
>>> ...
>>>
>>> At kernel 2.6.18, fs/lockd/svc.c
>>> ...
>>> 344         if (nlmsvc_users) {
>>> 345                 if (--nlmsvc_users)
>>> 346                         goto out;
>>> 347         } else {
>>> 348                 printk(KERN_ERR "lockd_down: no users! task=3D%=
p=20
>>> \n",
>>> 349                         nlmsvc_task);
>>> 350                 BUG();
>>> 351         }
>>> ....
>>> 357         kthread_stop(nlmsvc_task);
>>> 358         svc_exit_thread(nlmsvc_rqst);
>>> ...
>>>
>>> As above, when nlmsvc_users <=3D 1, the lockd will be killed.
>>>
>>>>
>>>> + lockd doesn't currently poke statd when it restarts to tell it t=
o
>>>> send reboot notifications, but it probably should
>>>
>>> Yes, I agree with you. But now, when some reason cause lockd =20
>>> restart but
>>> statd not restart, locks which hold before will lost.
>>>
>>> Maybe, the kernel should fix this.
>>
>> What did you have in mind?
>
>  I think when lockd restart, statd should restart too and sent sm-=20
> notify to other client.

Sending notifications is likely the correct thing to do if lockd is =20
restarted while there are active locks.  A statd restart isn't =20
necessarily required to send reboot notifications, however.  You can =20
do it with "sm-notify -f".

The problem with "sm-notify -f" is that it deletes the on-disk monitor =
=20
list while statd is still running.  This means the on-disk monitor =20
list and statd's in-memory monitor list will be out of sync.  I seem =20
to recall that sm-notify is run by itself by cluster scripts, and that =
=20
could be a real problem.

As implemented on RH, "service nfslock restart" will restart statd and =
=20
force an sm-notify anyway, so no real harm done, but that's pretty =20
heavyweight (and requires that admins do "service nfs stop; service =20
nfslock restart; service nfs start" or something like that if they =20
want to get proper lock recovery).

A simple restart of statd (outside of the nfslock script) probably =20
won't be adequate, though.  It will respect the sm-notify pidfile, and =
=20
not send notifications when started up.  I don't see a flag on statd =20
to force it to send notifications on restart (-N only sends =20
notifications; it doesn't also start the statd daemon).

In a perfect world, when lockd restarts, it would send up an =20
SM_SIMU_CRASH, and statd would do the right thing:  if there are =20
monitored peers, it would send reboot notifications, and adjust it's =20
monitor list accordingly;  if there were no monitored peers, it would =20
do nothing.  Thus no statd restart would be needed.

>  But now, in kernel and nfs-uitls, it don't implemented.
>  As the communication style between lockd and statd, this is indeed =20
> not easy to implement it.
>
>  So, I think it's should more easy to implement it through the =20
> mechanism that exposes
>  the kernel's nlm_host cache via /sys you show me before.

>>> I want to know when using cammond "service nfslock restart" =20
>>> restart the
>>> nfslock service(means restart statd and lockd), will the statd call
>>> sm-notify
>>> to notify other client? Or don't?
>>
>> Currently "service nfslock restart" always causes a notification to =
=20
>> be
>> sent.  Since "service nfslock restart" causes lockd to drop its =20
>> locks (I
>> assume that's what that "killproc lockd" does) I guess we need to =20
>> force
>> reboot notifications here.  (I still argue that removing the =20
>> pidfile in
>> the "start" case is not correct).
>>
>> It appears that both the nfs and nfslock start up scripts do =20
>> something
>> to lockd (as well as the case when the number of NFS mounts goes to
>> zero).  However, only the nfslock script forces sm-notify to send
>> notifications.
>
>  But, at RHLE5 and Fedora, when using cammond "service nfslock =20
> restart" restart
>  the nfslock service, the lockd isn't shutdown and rmmod'd.
>
>  Is it a bug?

=46or the "no more NFS mounts case" and the server shutdown case, the =20
NFS client or server, both being in the kernel, call lockd_down enough =
=20
times to make the user count go to zero.  lockd.ko can be removed at =20
that point.  I seem to recall there being some kind of automatic =20
mechanism for module removal after a period of zero module refcount.  =20
In other words, lockd.ko is removed as a side effect, afaict.

The nfslock script doesn't stop either the kernel client or server =20
code, so it doesn't really cause a lockd_down call.  But, nfslock does =
=20
do a "killproc lockd".  My assumption is that causes all locks to be =20
dropped.  So it's not a cold restart of lockd, but we still =20
potentially lose a lot of lock state here.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: [RFC] After nfs restart, locks can't be recovered which record by lockd before

[Mi Jinlong]

Hi Chuck,

Chuck Lever =E5=86=99=E9=81=93:
> On Jan 15, 2010, at 4:35 AM, Mi Jinlong wrote:

=2E..snip...

>>>>
>>>> Maybe, the kernel should fix this.
>>>
>>> What did you have in mind?
>>
>>  I think when lockd restart, statd should restart too and sent
>> sm-notify to other client.
>=20
> Sending notifications is likely the correct thing to do if lockd is
> restarted while there are active locks.  A statd restart isn't
> necessarily required to send reboot notifications, however.  You can =
do
> it with "sm-notify -f".
>=20
> The problem with "sm-notify -f" is that it deletes the on-disk monito=
r
> list while statd is still running.  This means the on-disk monitor li=
st
> and statd's in-memory monitor list will be out of sync.  I seem to
> recall that sm-notify is run by itself by cluster scripts, and that
> could be a real problem.
>=20
> As implemented on RH, "service nfslock restart" will restart statd an=
d
> force an sm-notify anyway, so no real harm done, but that's pretty
> heavyweight (and requires that admins do "service nfs stop; service
> nfslock restart; service nfs start" or something like that if they wa=
nt
> to get proper lock recovery).
>=20
> A simple restart of statd (outside of the nfslock script) probably wo=
n't
> be adequate, though.  It will respect the sm-notify pidfile, and not
> send notifications when started up.  I don't see a flag on statd to
> force it to send notifications on restart (-N only sends notification=
s;
> it doesn't also start the statd daemon).
>=20
> In a perfect world, when lockd restarts, it would send up an
> SM_SIMU_CRASH, and statd would do the right thing:  if there are
> monitored peers, it would send reboot notifications, and adjust it's
> monitor list accordingly;  if there were no monitored peers, it would=
 do
> nothing.  Thus no statd restart would be needed.

  Did this part have implemented at kernel?
  I don't find the codes about SM_SIMU_CRASH.

  IMO, if SM_SIMU_CRASH can work correctly, after lockd restart,
  the lockd and statd would work correctly (SYN) too.
  And the problem I asked above will not be happened.

>=20
>>  But now, in kernel and nfs-uitls, it don't implemented.
>>  As the communication style between lockd and statd, this is indeed
>> not easy to implement it.
>>
>>  So, I think it's should more easy to implement it through the
>> mechanism that exposes
>>  the kernel's nlm_host cache via /sys you show me before.
>=20
>>>> I want to know when using cammond "service nfslock restart" restar=
t the
>>>> nfslock service(means restart statd and lockd), will the statd cal=
l
>>>> sm-notify
>>>> to notify other client? Or don't?
>>>
>>> Currently "service nfslock restart" always causes a notification to=
 be
>>> sent.  Since "service nfslock restart" causes lockd to drop its loc=
ks (I
>>> assume that's what that "killproc lockd" does) I guess we need to f=
orce
>>> reboot notifications here.  (I still argue that removing the pidfil=
e in
>>> the "start" case is not correct).
>>>
>>> It appears that both the nfs and nfslock start up scripts do someth=
ing
>>> to lockd (as well as the case when the number of NFS mounts goes to
>>> zero).  However, only the nfslock script forces sm-notify to send
>>> notifications.
>>
>>  But, at RHLE5 and Fedora, when using cammond "service nfslock
>> restart" restart
>>  the nfslock service, the lockd isn't shutdown and rmmod'd.
>>
>>  Is it a bug?
>=20
> For the "no more NFS mounts case" and the server shutdown case, the N=
=46S
> client or server, both being in the kernel, call lockd_down enough ti=
mes
> to make the user count go to zero.  lockd.ko can be removed at that
> point.  I seem to recall there being some kind of automatic mechanism
> for module removal after a period of zero module refcount.  In other
> words, lockd.ko is removed as a side effect, afaict.
>=20
> The nfslock script doesn't stop either the kernel client or server co=
de,
> so it doesn't really cause a lockd_down call.  But, nfslock does do a
> "killproc lockd".  My assumption is that causes all locks to be
> dropped.  So it's not a cold restart of lockd, but we still potential=
ly
> lose a lot of lock state here.

 Thanks for your explain!

Thanks,
Mi Jinlong

Re: [RFC] After nfs restart, locks can't be recovered which record by lockd before

[Chuck Lever]

On Jan 18, 2010, at 5:51 AM, Mi Jinlong wrote:

> Hi Chuck,
>
> Chuck Lever =E5=86=99=E9=81=93:
>> On Jan 15, 2010, at 4:35 AM, Mi Jinlong wrote:
>
> ...snip...
>
>>>>>
>>>>> Maybe, the kernel should fix this.
>>>>
>>>> What did you have in mind?
>>>
>>> I think when lockd restart, statd should restart too and sent
>>> sm-notify to other client.
>>
>> Sending notifications is likely the correct thing to do if lockd is
>> restarted while there are active locks.  A statd restart isn't
>> necessarily required to send reboot notifications, however.  You =20
>> can do
>> it with "sm-notify -f".
>>
>> The problem with "sm-notify -f" is that it deletes the on-disk =20
>> monitor
>> list while statd is still running.  This means the on-disk monitor =20
>> list
>> and statd's in-memory monitor list will be out of sync.  I seem to
>> recall that sm-notify is run by itself by cluster scripts, and that
>> could be a real problem.
>>
>> As implemented on RH, "service nfslock restart" will restart statd =20
>> and
>> force an sm-notify anyway, so no real harm done, but that's pretty
>> heavyweight (and requires that admins do "service nfs stop; service
>> nfslock restart; service nfs start" or something like that if they =20
>> want
>> to get proper lock recovery).
>>
>> A simple restart of statd (outside of the nfslock script) probably =20
>> won't
>> be adequate, though.  It will respect the sm-notify pidfile, and not
>> send notifications when started up.  I don't see a flag on statd to
>> force it to send notifications on restart (-N only sends =20
>> notifications;
>> it doesn't also start the statd daemon).
>>
>> In a perfect world, when lockd restarts, it would send up an
>> SM_SIMU_CRASH, and statd would do the right thing:  if there are
>> monitored peers, it would send reboot notifications, and adjust it's
>> monitor list accordingly;  if there were no monitored peers, it =20
>> would do
>> nothing.  Thus no statd restart would be needed.
>
>  Did this part have implemented at kernel?
>  I don't find the codes about SM_SIMU_CRASH.

There isn't such code in the current kernel.  I'm simply suggesting a =20
possible solution.

I'm coding up a patch that does this so we can experiment with it.  =20
I'm a little worried that the current statd code won't do the right =20
thing, or even worse, it would crash.  That would make it difficult to =
=20
introduce such a patch without triggering regressions.

--=20
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: [RFC] After nfs restart, locks can't be recovered which record by lockd before

[Mi Jinlong]

Chuck Lever =E5=86=99=E9=81=93:
>=20
> On Jan 18, 2010, at 5:51 AM, Mi Jinlong wrote:
>=20
>> Hi Chuck,
>>
>> Chuck Lever =E5=86=99=E9=81=93:
>>> On Jan 15, 2010, at 4:35 AM, Mi Jinlong wrote:
>>
>> ...snip...
>>
>>>>>>
>>>>>> Maybe, the kernel should fix this.
>>>>>
>>>>> What did you have in mind?
>>>>
>>>> I think when lockd restart, statd should restart too and sent
>>>> sm-notify to other client.
>>>
>>> Sending notifications is likely the correct thing to do if lockd is
>>> restarted while there are active locks.  A statd restart isn't
>>> necessarily required to send reboot notifications, however.  You ca=
n do
>>> it with "sm-notify -f".
>>>
>>> The problem with "sm-notify -f" is that it deletes the on-disk moni=
tor
>>> list while statd is still running.  This means the on-disk monitor =
list
>>> and statd's in-memory monitor list will be out of sync.  I seem to
>>> recall that sm-notify is run by itself by cluster scripts, and that
>>> could be a real problem.
>>>
>>> As implemented on RH, "service nfslock restart" will restart statd =
and
>>> force an sm-notify anyway, so no real harm done, but that's pretty
>>> heavyweight (and requires that admins do "service nfs stop; service
>>> nfslock restart; service nfs start" or something like that if they =
want
>>> to get proper lock recovery).
>>>
>>> A simple restart of statd (outside of the nfslock script) probably =
won't
>>> be adequate, though.  It will respect the sm-notify pidfile, and no=
t
>>> send notifications when started up.  I don't see a flag on statd to
>>> force it to send notifications on restart (-N only sends notificati=
ons;
>>> it doesn't also start the statd daemon).
>>>
>>> In a perfect world, when lockd restarts, it would send up an
>>> SM_SIMU_CRASH, and statd would do the right thing:  if there are
>>> monitored peers, it would send reboot notifications, and adjust it'=
s
>>> monitor list accordingly;  if there were no monitored peers, it wou=
ld do
>>> nothing.  Thus no statd restart would be needed.
>>
>>  Did this part have implemented at kernel?
>>  I don't find the codes about SM_SIMU_CRASH.
>=20
> There isn't such code in the current kernel.  I'm simply suggesting a
> possible solution.
>=20
> I'm coding up a patch that does this so we can experiment with it.  I=
'm
> a little worried that the current statd code won't do the right thing=
,
> or even worse, it would crash.  That would make it difficult to
> introduce such a patch without triggering regressions.
>=20

  That's why I don't find the codes! Thanks!
  The statd code does have some rough, but, we can modify it when we ne=
ed.

  Do you have implemented those codes? If have, please give me a copy,=20
  or a git commit.

thanks,
Mi Jinlong