flock(1): working with fcntl locks

flock(1): working with fcntl locks

[Kjetil Torgrim Homme]

I was a bit surprised to find that flock(2) specifically ignores fcntl 
locks.  from its manual page:

        Since  kernel  2.0,  flock() is implemented as a system call in 
its own
        right rather than being emulated in the GNU C  library  as a  
call  to
        fcntl(2).   This  yields  true  BSD  semantics: there is no 
interaction
        between the types of lock placed by flock() and fcntl(2), and  
flock()
        does not detect deadlock.

I was trying to check if dpkg or apt-get was holding its lock and skip 
running my cron job if so, but unfortunately it uses fcntl (F_SETLK), 
and flock(1) will happily call flock(2) which succeeds.

it's a bit sad to have to write the lock testing in C or Perl rather 
than use the nice little flock(1), so I wonder if we could "fix" 
flock(1) somehow.  I think I'm not alone to be surprised that flock(1) 
is so ineffective against locking done by other utilities, so my 
prefered solution would be to switch to using fcntl(2).

the chance of a problematic regression is small, I think.  my *guess* is 
that most flock(1) usage is only interacting with other usage of 
flock(1) (not flock(2)).  also relying on flock(1) succeeding on a 
fcntl-locked file would be just Wrong(tm).

the "safe" solution is to add a flag, --fcntl, but isn't that just cruft?

I can provide patches when I hear what the mailing list wants.

-- 
Kjetil T. Homme
Redpill Linpro - Changing the game

Re: flock(1): working with fcntl locks

[Karel Zak]

On Fri, Jan 03, 2014 at 02:59:26PM +0100, Kjetil Torgrim Homme wrote:
> I was a bit surprised to find that flock(2) specifically ignores fcntl
> locks.  from its manual page:
> 
>        Since  kernel  2.0,  flock() is implemented as a system call in its
> own
>        right rather than being emulated in the GNU C  library  as a  call
> to
>        fcntl(2).   This  yields  true  BSD  semantics: there is no
> interaction
>        between the types of lock placed by flock() and fcntl(2), and
> flock()
>        does not detect deadlock.

 Welcome to POSIX/Linux locking... read nice Lennart's summary: 
 http://0pointer.de/blog/projects/locking.html
 http://0pointer.de/blog/projects/locking2

> I was trying to check if dpkg or apt-get was holding its lock and skip
> running my cron job if so, but unfortunately it uses fcntl (F_SETLK), and
> flock(1) will happily call flock(2) which succeeds.
> 
> it's a bit sad to have to write the lock testing in C or Perl rather than
> use the nice little flock(1), so I wonder if we could "fix" flock(1)
> somehow.  I think I'm not alone to be surprised that flock(1) is so
> ineffective against locking done by other utilities, so my prefered solution
> would be to switch to using fcntl(2).

 Sorry, but today is not 1st Apr ;-)  
 
 And process based fcntl(2) sucks more than flock(2) and for things like 
 flock(1) it's probably completely useless.

> the chance of a problematic regression is small, I think.  my *guess* is
> that most flock(1) usage is only interacting with other usage of flock(1)
> (not flock(2)).  also relying on flock(1) succeeding on a fcntl-locked file
> would be just Wrong(tm).
> 
> the "safe" solution is to add a flag, --fcntl, but isn't that just cruft?
> 
> I can provide patches when I hear what the mailing list wants.

 No please, flock(1) is based on flock(2), that's all. The semantic
 and all possible limitations are well known. I don't think we want to
 make things more complicated.

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: flock(1): working with fcntl locks

[Kjetil Torgrim Homme]

On 03/01/2014 15:40, Karel Zak wrote:
> On Fri, Jan 03, 2014 at 02:59:26PM +0100, Kjetil Torgrim Homme wrote:
>> I was a bit surprised to find that flock(2) specifically ignores fcntl
>> locks.  from its manual page:
>>
>>         Since  kernel  2.0,  flock() is implemented as a system call in its
>> own
>>         right rather than being emulated in the GNU C  library  as a  call
>> to
>>         fcntl(2).   This  yields  true  BSD  semantics: there is no
>> interaction
>>         between the types of lock placed by flock() and fcntl(2), and
>> flock()
>>         does not detect deadlock.
>   Welcome to POSIX/Linux locking... read nice Lennart's summary:
>   http://0pointer.de/blog/projects/locking.html
>   http://0pointer.de/blog/projects/locking2

thanks!  doesn't seem relevant for flock(1), though, since there is no 
threading involved.  flock(1) should acquire the lock, fork the child 
and wait for it before returning the lock.  no pitfalls there?

>> I was trying to check if dpkg or apt-get was holding its lock and skip
>> running my cron job if so, but unfortunately it uses fcntl (F_SETLK), and
>> flock(1) will happily call flock(2) which succeeds.
>>
>> it's a bit sad to have to write the lock testing in C or Perl rather than
>> use the nice little flock(1), so I wonder if we could "fix" flock(1)
>> somehow.  I think I'm not alone to be surprised that flock(1) is so
>> ineffective against locking done by other utilities, so my prefered solution
>> would be to switch to using fcntl(2).
>   Sorry, but today is not 1st Apr ;-)
>   
>   And process based fcntl(2) sucks more than flock(2) and for things like
>   flock(1) it's probably completely useless.

I don't see why you think fcntl(2) sucks more.  it is more portable and 
more versatile, and therefore most applications use that instead of 
flock(2).  as mentioned earlier, flock(2) is a relatively new invention, 
it used to be flock(3) which called fcntl(2) via a compatibility layer.

>> the chance of a problematic regression is small, I think.  my *guess* is
>> that most flock(1) usage is only interacting with other usage of flock(1)
>> (not flock(2)).  also relying on flock(1) succeeding on a fcntl-locked file
>> would be just Wrong(tm).
>>
>> the "safe" solution is to add a flag, --fcntl, but isn't that just cruft?
>>
>> I can provide patches when I hear what the mailing list wants.
>   No please, flock(1) is based on flock(2), that's all. The semantic
>   and all possible limitations are well known. I don't think we want to
>   make things more complicated.

do you think we should have a posixlock(1)?  (if so, perhaps it would 
fit better in coreutils rather than util-linux ...)

-- 
Kjetil T. Homme
Redpill Linpro - Changing the game

Re: flock(1): working with fcntl locks

[Karel Zak]

On Fri, Jan 03, 2014 at 04:12:37PM +0100, Kjetil Torgrim Homme wrote:
> >  Welcome to POSIX/Linux locking... read nice Lennart's summary:
> >  http://0pointer.de/blog/projects/locking.html
> >  http://0pointer.de/blog/projects/locking2
> 
> thanks!  doesn't seem relevant for flock(1), though, since there is no
> threading involved.  flock(1) should acquire the lock, fork the child and
> wait for it before returning the lock.  no pitfalls there?

       (
          flock -n 9 || exit 1
          # ... commands executed under lock ...
       ) 9>/var/lock/mylockfile

this is way how people use flock in scripts and it works because it's 
based on file descriptors and independent on original process.

> I don't see why you think fcntl(2) sucks more.

 see Lennart's summary, the problem is that the lock is based on
 process and it's useless for system files (due to open/close 
 in libraries), etc.

> >  No please, flock(1) is based on flock(2), that's all. The semantic
> >  and all possible limitations are well known. I don't think we want to
> >  make things more complicated.
> 
> do you think we should have a posixlock(1)?  (if so, perhaps it would fit
> better in coreutils rather than util-linux ...)

 Yep.

 Frankly, reliable fcntl locking requires a lot of code and extra lock
 files (we use it for example in original mount for /etc/mtab).

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: flock(1): working with fcntl locks

[Andy Lutomirski]

On 01/04/2014 12:31 AM, Karel Zak wrote:
> On Fri, Jan 03, 2014 at 04:12:37PM +0100, Kjetil Torgrim Homme wrote:
>>>  Welcome to POSIX/Linux locking... read nice Lennart's summary:
>>>  http://0pointer.de/blog/projects/locking.html
>>>  http://0pointer.de/blog/projects/locking2
>>
>> thanks!  doesn't seem relevant for flock(1), though, since there is no
>> threading involved.  flock(1) should acquire the lock, fork the child and
>> wait for it before returning the lock.  no pitfalls there?
> 
>        (
>           flock -n 9 || exit 1
>           # ... commands executed under lock ...
>        ) 9>/var/lock/mylockfile
> 
> this is way how people use flock in scripts and it works because it's 
> based on file descriptors and independent on original process.
> 
>> I don't see why you think fcntl(2) sucks more.
> 
>  see Lennart's summary, the problem is that the lock is based on
>  process and it's useless for system files (due to open/close 
>  in libraries), etc.
> 
>>>  No please, flock(1) is based on flock(2), that's all. The semantic
>>>  and all possible limitations are well known. I don't think we want to
>>>  make things more complicated.
>>
>> do you think we should have a posixlock(1)?  (if so, perhaps it would fit
>> better in coreutils rather than util-linux ...)
> 
>  Yep.
> 
>  Frankly, reliable fcntl locking requires a lot of code and extra lock
>  files (we use it for example in original mount for /etc/mtab).

FWIW, there are patches floating around on LKML (my pathetic crystal
ball says they'll be merged for 3.14 or 3.15 and maybe even make it into
POSIX) to add a new F_SETLKP64 that creates an fcntl lock that's
attached to the file descriptor.

Once that goes in, it might pay to add a --fcntl flag to flock(1) that
fails on older kernels.


--Andy