permission denied with >= ~2.6.25

permission denied with >= ~2.6.25

[Martin Vogt]

Hello list,

recently I updated my client to a new vendor kernel(SLES11), but the
problem is the same if I use a 2.6.33.3 as a client.

If I'm doing svn checkouts in an endless loop,
or running make -j X builds sometimes it happens
that I get a "Permision denied" error.

If I strace it, for svn, the following sequence appears:


5262  rename("Kernel.ERWQPsWCA5/.svn/tmp/entries",
"Kernel.ERWQPsWCA5/.svn/entries") = 0
5262  lstat("Kernel.ERWQPsWCA5/.svn/entries", {st_mode=S_IFREG|0666,
st_size=205, ...}) = 0
5262  chmod("Kernel.ERWQPsWCA5/.svn/entries", 0444) = 0
5262  open("Kernel.ERWQPsWCA5/.svn/format.tmp", O_RDWR|O_CREAT|O_EXCL,
0666) = 3
5262  write(3, "9\n", 2)                = 2
5262  close(3)                          = 0


5262  rename("Kernel.ERWQPsWCA5/.svn/format.tmp",
"Kernel.ERWQPsWCA5/.svn/format") = 0
5262  lstat("Kernel.ERWQPsWCA5/.svn/format", 0x7fff79949350) = -1 EACCES
(Permission denied)
^^^^^^^^^^^^ Here

5262  write(2, "svn: Can't change perms of file "..., 83) = 83
5262  lstat("Kernel.ERWQPsWCA5/.svn/log", 0x7fff79949650) = -1 ENOENT
(No such file or directory)
5262  unlink("Kernel.ERWQPsWCA5/.svn/lock") = 0


After a rename, the lstat call fails. I tried to isolate it,
but without any luck so far.

I dont have the problem if I go back to my vendor kernel <= 2.6.22.
I wonder if its more a NFS Server problem or a problem on the client
side so I tried writing a C programm to trigger this behaviour.
But without any success so far.

Maybe this problem is already known?


regards,

Martin

Re: permission denied with >= ~2.6.25

[Martin Vogt]

Martin Vogt wrote:
> Hello list,
> 
> recently I updated my client to a new vendor kernel(SLES11), but the
> problem is the same if I use a 2.6.33.3 as a client.
> 
> If I'm doing svn checkouts in an endless loop,
> or running make -j X builds sometimes it happens
> that I get a "Permision denied" error.
> 

Hello,

I tested 2.6.34-rc7 now and it has the problem too.
Then I checked 2.6.24.7 which has the problem,
but 2.6.23.1 looks fine.

>svn: In directory 'Kernel.9TKDNMQTDm/arch/arm/mach-realview'
>svn: Error processing command 'readonly' in
>'Kernel.9TKDNMQTDm/arch/arm/mach-realview'
>svn: Can't change perms of file
>'Kernel.9TKDNMQTDm/arch/arm/mach-realview/.svn/text-base/Makefile.boot.svn-base':
>Permission denied
>Tue May 11 12:57:51 CEST 2010,1
>runcounter: 1
>vogt@pxe3[vogt_nfs]>uname -a
>Linux pxe3.itwm.fhg.de 2.6.34-rc7-0.1-default #2 SMP Tue May 11
>10:28:41 CEST 2010 x86_64 x86_64 x86_64 GNU/Linux


Usually the "permission denied" happens very early. In this case the
first run.

regards,

Martin

Re: permission denied with >= ~2.6.25

[Martin Vogt]

> Hello,
>
> I tested 2.6.34-rc7 now and it has the problem too.
> Then I checked 2.6.24.7 which has the problem,
> but 2.6.23.1 looks fine.

Hello,

I searched a bit more in the git tree, when the "Permission denied"
problems does occur first(for me).
I'm no expert in git. I used git bisect visualize to find a commit
which should build and then tested if it works/works not.

The first commit which introduced the "Permission denied" problem,
was:

1-run:
>commit 77a55a1fe8f26f7d022986a599b68002e21d968a
>NFS: Eliminate nfs_renew_times()
>svn: Can't change perms of file
>'Kernel-pxe3-1.h2cuuZBQlF/arch/h8300/platform/h8
>s/.svn/log.1': Permission denied

When I go back one commit, the "Permission denied" disappears, but
now I get an "Input/output error".

5-run:
>commit 92f6c178250170222f6d80c8ae725400765aa7a4
>"Don't call nfs_renew_times() in nfs_dentry_iput()"
>svn: Can't read file
>'Kernel-pxe3-5.HQikkhUNHk/drivers/usb/storage/.svn/log': Input/output error

Thus it looks that the problem is now "I/O Error" instead of
permission denied. And that I have to search when the I/O Error
disappears?

Maybe someone has an idea what went wrong here?

All I can say that I have reproduced a similar svn problem with
2.6.33 as server/client and with 2.6.34-rc7 as client.
And that on my system does not go away with sync, tcp, udp, noac,
nosharecache.

But I cannot say that the problem in the 2.6.33 (server/client)
combination is the same.
Because with this the svn error is "No such file or directory".
The problem never existed on my 2.6.16 kernel from SLES10, but now
every kernel >= 2.6.24 seems to be affected.


regards,

Martin

Re: permission denied with >= ~2.6.25

[Martin Vogt]

On Fri, May 14, 2010 at 11:56:20AM +0200, Martin Vogt wrote:
> 
> > Hello,
> >
> > I tested 2.6.34-rc7 now and it has the problem too.
> > Then I checked 2.6.24.7 which has the problem,
> > but 2.6.23.1 looks fine.
>

Hello,

2.6.23.1 down to 2.6.20.1 has the same problem and 2.6.16.<vendor> too.

But its seems that, the older the kernel gets, the
longer it takes that the EPERM or EIO appears. 

For example: 2.6.34-rc7 is really quick ;-) 

But now I have no more ideas for debugging.

regards,

Martin

PS: here a two more straces:


1:
==
13037 open("Kernel-pxe2-1.JGD9NGERWJ/arch/ppc/syslib/pci_auto.c.tmp",  
O_RDWR|O_CREAT|O_EXCL, 0666) = 6
13037 read(5, "/*\n * arch/ppc/syslib/pci_auto.c"..., 16384) = 12365
13037 write(6, "/*\n * arch/ppc/syslib/pci_auto.c"..., 4096) = 4096
13037 write(6, ",\n\t\t\tcurrent_bus,\n\t\t\tpci_devfn,\n"..., 4096) = 4096
13037 write(6, "e behind a\n\t * P2P bridge in a s"..., 4096) = 4096
13037 read(5, "", 16384)                = 0  
13037 close(5)                          = 0
13037 write(6, "_devfn,\n\t\t\t\t\t\tPCI_LATENCY_TIMER,"..., 77) = 77
13037 close(6)                          = 0
13037 rename("Kernel-pxe2-1.JGD9NGERWJ/arch/ppc/syslib/pci_auto.c.tmp",
"Kernel-pxe2-1.JGD9NGERWJ/arch/ppc/syslib/pci_auto.c") = 0
13037 rename("Kernel-pxe2-1.JGD9NGERWJ/arch/ppc/syslib/.svn/tmp/text-base/pci_auto.c.svn-base",
"Kernel-pxe2-1.JGD9NGERWJ/arch/ppc/sysli
b/.svn/text-base/pci_auto.c.svn-base") = 0
13037 lstat("Kernel-pxe2-1.JGD9NGERWJ/arch/ppc/syslib/.svn/text-base/pci_auto.c.svn-base",
{st_mode=S_IFREG|0666, st_size=12365, ...}) =
 0
13037 chmod("Kernel-pxe2-1.JGD9NGERWJ/arch/ppc/syslib/.svn/text-base/pci_auto.c.svn-base",
0444) = -1 EACCES (Permission denied)
^^^^^^^^^^^^^^^^^
Here

13037 fcntl(3, F_GETFL)                 = 0x2 (flags O_RDWR)
13037 fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
13037 write(3, "( failure ( ( 155009 55:In direc"..., 423) = 423

2:  
==
13048 open("Kernel-pxe2-1.5MzYLADYdV/arch/ppc/kernel/traps.c.tmp",
O_RDWR|O_CREAT|O_EXCL, 0666) = 6
13048 read(5, "/*\n *  arch/ppc/kernel/traps.c\n "..., 16384) = 16384
13048 write(6, "/*\n *  arch/ppc/kernel/traps.c\n "..., 4096) = 4096
13048 write(6, "ble.\n * Note that the 601 only t"..., 4096) = 4096
13048 write(6, "CSR_IMPE)\n\t\t\tprintk(\"Machine Che"..., 4096) = 4096
13048 read(5, ", bug->file,\n\t\t       bug->line "..., 16384) = 9123
13048 write(6, "\t\t0x7c0007fe\n\n#define INST_STRIN"..., 4096) = 4096
13048 write(6, ", bug->file,\n\t\t       bug->line "..., 4096) = 4096
13048 write(6, "if (!user_mode(regs)) {\n\t\tdebugg"..., 4096) = 4096
13048 read(5, "", 16384)                = 0
13048 close(5)                          = 0
13048 write(6, "UND)) {\n\t\tcode = FPE_FLTUND;\n\t\ts"..., 931) = 931 
13048 close(6)                          = 0
13048 rename("Kernel-pxe2-1.5MzYLADYdV/arch/ppc/kernel/traps.c.tmp",  
"Kernel-pxe2-1.5MzYLADYdV/arch/ppc/kernel/traps.c") = 0
13048 rename("Kernel-pxe2-1.5MzYLADYdV/arch/ppc/kernel/.svn/tmp/text-base/traps.c.svn-base",
"Kernel-pxe2-1.5MzYLADYdV/arch/ppc/kernel/.svn/text-base/traps.c.svn-base") = 0
13048 lstat("Kernel-pxe2-1.5MzYLADYdV/arch/ppc/kernel/.svn/text-base/traps.c.svn-base",
0x7fff57f65820) = -1 EACCES (Permission denied)
^^^^^^^^^^^^^^^
Here
13048 fcntl(3, F_GETFL)                 = 0x2 (flags O_RDWR)
13048 fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
13048 write(3, "( failure ( ( 155009 55:In direc"..., 420) = 420

Re: permission denied with >= ~2.6.25

[Martin Vogt]

Hello,


today I tried 2.6.34 on server/client with standard
hardware. After around the 69th checkout I had an
"Invalid cross-device link" Error.
(Before that it was EIO/EPERM)

Test ran for around 9 hours before it failed.

>svn: In directory 'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc'
>svn: Can't move
'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h.tmp'
>to 'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h': Invalid
>cross-device link
>Mon May 17 23:46:32 CEST 2010,1
>runcounter: 69

Martin

Re: permission denied with >= ~2.6.25

[Martin Vogt]

Martin Vogt wrote:
> Hello,
> 
> 
> today I tried 2.6.34 on server/client with standard
> hardware. After around the 69th checkout I had an
> "Invalid cross-device link" Error.
> (Before that it was EIO/EPERM)
> 
> Test ran for around 9 hours before it failed.
> 
>> svn: In directory 'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc'
>> svn: Can't move
> 'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h.tmp'
>> to 'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h': Invalid
>> cross-device link
>> Mon May 17 23:46:32 CEST 2010,1
>> runcounter: 69
> 

Ok.
RTFM: no_subtree_check :-(

This option made the checkouts reliable.

Martin

Re: permission denied with >= ~2.6.25

[J. Bruce Fields]

On Wed, May 19, 2010 at 01:12:36PM +0200, Martin Vogt wrote:
> Martin Vogt wrote:
> > Hello,
> > 
> > 
> > today I tried 2.6.34 on server/client with standard
> > hardware. After around the 69th checkout I had an
> > "Invalid cross-device link" Error.
> > (Before that it was EIO/EPERM)
> > 
> > Test ran for around 9 hours before it failed.
> > 
> >> svn: In directory 'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc'
> >> svn: Can't move
> > 'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h.tmp'
> >> to 'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h': Invalid
> >> cross-device link
> >> Mon May 17 23:46:32 CEST 2010,1
> >> runcounter: 69
> > 
> 
> Ok.
> RTFM: no_subtree_check :-(
> 
> This option made the checkouts reliable.

Erp.  Should have thought of that.  It's no longer the default in recent
nfs-utils, for this sort of reason.

(But, note: for anyone exporting directories that aren't mountpoints,
that may expose more of their filesystem than intended.)

--b.

Re: permission denied with >= ~2.6.25

[Martin Vogt]

[-- Attachment #1: Type: text/plain, Size: 2202 bytes --]

J. Bruce Fields wrote:
> On Wed, May 19, 2010 at 01:12:36PM +0200, Martin Vogt wrote:
>> Martin Vogt wrote:
>>> Hello,
>>>
>>>
>>> today I tried 2.6.34 on server/client with standard
>>> hardware. After around the 69th checkout I had an
>>> "Invalid cross-device link" Error.
>>> (Before that it was EIO/EPERM)
>>>
>>> Test ran for around 9 hours before it failed.
>>>
>>>> svn: In directory 'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc'
>>>> svn: Can't move
>>> 'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h.tmp'
>>>> to 'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h': Invalid
>>>> cross-device link
>>>> Mon May 17 23:46:32 CEST 2010,1
>>>> runcounter: 69
>> Ok.
>> RTFM: no_subtree_check :-(
>>
>> This option made the checkouts reliable.
> 
> Erp.  Should have thought of that.  It's no longer the default in recent
> nfs-utils, for this sort of reason.
> 
> (But, note: for anyone exporting directories that aren't mountpoints,
> that may expose more of their filesystem than intended.)
> 

Hello,

the report for the latest kernel was done with "no_subtree_check"

-9 hours client/server both 2.6.34 before "Invalid cross-device link"

2.6.34 still shows some svn problem with "no_subtree_check" on.
But only after 9 hours (vs 10 minutes, with subtree_check)

It was done with:

>log:Assuming default behaviour ('no_subtree_check').
exports:
>/var/tmp/nfs_export
>192.168.9.0/255.255.255.0(rw,async,wdelay,acl,no_root_squash,fsid=6666)

Using "no_subtree_check" made it only "much more" stable
than before, but not "stable".

Server was 4 core (2 sockets) Xeon 5148@2.33GHz,
8GB mem.
Exported fs was ext3 from a Standard Sata drive,
and the client was a dual core E8400.

Mount client:
mount -o rw,nosuid,nodev,rsize=8192,wsize=8192,vers=3 -t nfs
pxe2:/var/tmp/nfs_export  /home/local


I'm attaching my test script. The script needs to be run two times
in parallel on the machine. (In screen for example).
If someone wants to reproduce this.A single checkout on
the machine would even takes longer to reproduce the problem.

I imported a normal kernel tree into a svn repository
on a dedicated server only for this and started svnserve -d
as normal user.


regards,

Martin


[-- Attachment #2: svn_run.sh --]
[-- Type: application/x-shellscript, Size: 1189 bytes --]

Re: permission denied with >= ~2.6.25

[J. Bruce Fields]

On Thu, May 20, 2010 at 11:28:41AM +0200, Martin Vogt wrote:
> J. Bruce Fields wrote:
> > On Wed, May 19, 2010 at 01:12:36PM +0200, Martin Vogt wrote:
> >> Martin Vogt wrote:
> >>> Hello,
> >>>
> >>>
> >>> today I tried 2.6.34 on server/client with standard
> >>> hardware. After around the 69th checkout I had an
> >>> "Invalid cross-device link" Error.
> >>> (Before that it was EIO/EPERM)
> >>>
> >>> Test ran for around 9 hours before it failed.
> >>>
> >>>> svn: In directory 'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc'
> >>>> svn: Can't move
> >>> 'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h.tmp'
> >>>> to 'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h': Invalid
> >>>> cross-device link
> >>>> Mon May 17 23:46:32 CEST 2010,1
> >>>> runcounter: 69
> >> Ok.
> >> RTFM: no_subtree_check :-(
> >>
> >> This option made the checkouts reliable.
> > 
> > Erp.  Should have thought of that.  It's no longer the default in recent
> > nfs-utils, for this sort of reason.
> > 
> > (But, note: for anyone exporting directories that aren't mountpoints,
> > that may expose more of their filesystem than intended.)
> > 
> 
> Hello,
> 
> the report for the latest kernel was done with "no_subtree_check"

OK, sorry, I don't know.

You've been varying both client and server, but from your initial
reports it sounded like this was something that first started happening
on a client upgrade?

Have you been able to keep the server fixed while doing a binary search
through the client version for the first non-working client?

--b.

> 
> -9 hours client/server both 2.6.34 before "Invalid cross-device link"
> 
> 2.6.34 still shows some svn problem with "no_subtree_check" on.
> But only after 9 hours (vs 10 minutes, with subtree_check)
> 
> It was done with:
> 
> >log:Assuming default behaviour ('no_subtree_check').
> exports:
> >/var/tmp/nfs_export
> >192.168.9.0/255.255.255.0(rw,async,wdelay,acl,no_root_squash,fsid=6666)
> 
> Using "no_subtree_check" made it only "much more" stable
> than before, but not "stable".
> 
> Server was 4 core (2 sockets) Xeon 5148@2.33GHz,
> 8GB mem.
> Exported fs was ext3 from a Standard Sata drive,
> and the client was a dual core E8400.
> 
> Mount client:
> mount -o rw,nosuid,nodev,rsize=8192,wsize=8192,vers=3 -t nfs
> pxe2:/var/tmp/nfs_export  /home/local
> 
> 
> I'm attaching my test script. The script needs to be run two times
> in parallel on the machine. (In screen for example).
> If someone wants to reproduce this.A single checkout on
> the machine would even takes longer to reproduce the problem.
> 
> I imported a normal kernel tree into a svn repository
> on a dedicated server only for this and started svnserve -d
> as normal user.
> 
> 
> regards,
> 
> Martin
> 

Re: permission denied with >= ~2.6.25

[Martin Vogt]

J. Bruce Fields wrote:
> On Thu, May 20, 2010 at 11:28:41AM +0200, Martin Vogt wrote:
>> J. Bruce Fields wrote:
>>> On Wed, May 19, 2010 at 01:12:36PM +0200, Martin Vogt wrote:
>>>> Martin Vogt wrote:
>>>>> Hello,
>>>>>
>>>>>
>>>>> today I tried 2.6.34 on server/client with standard
>>>>> hardware. After around the 69th checkout I had an
>>>>> "Invalid cross-device link" Error.
>>>>> (Before that it was EIO/EPERM)
>>>>>
>>>>> Test ran for around 9 hours before it failed.
>>>>>
>>>>>> svn: In directory 'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc'
>>>>>> svn: Can't move
>>>>> 'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h.tmp'
>>>>>> to 'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h': Invalid
>>>>>> cross-device link
>>>>>> Mon May 17 23:46:32 CEST 2010,1
>>>>>> runcounter: 69
>>>> Ok.
>>>> RTFM: no_subtree_check :-(
>>>>
>>>> This option made the checkouts reliable.
>>> Erp.  Should have thought of that.  It's no longer the default in recent
>>> nfs-utils, for this sort of reason.
>>>
>>> (But, note: for anyone exporting directories that aren't mountpoints,
>>> that may expose more of their filesystem than intended.)
>>>
>> Hello,
>>
>> the report for the latest kernel was done with "no_subtree_check"
> 
> OK, sorry, I don't know.
> 
> You've been varying both client and server, but from your initial
> reports it sounded like this was something that first started happening
> on a client upgrade?
> 

Yes. I tried to go back with the kernels on the client.
(and keep the server fixed)
But up to 2.6.16 I had this svn error. Up to this time
I didn't know that no_subtree_check would increase the
stability that much.
All my tests were done with "subtree_check", except
the tests with 2.6.34 kernel on server/client.

On this server no_subtree_check was the default because of newer nfs-utils.

With this kernel it took ~9hours until I got the "Invalid cross-device
link".

> Have you been able to keep the server fixed while doing a binary search
> through the client version for the first non-working client?

No, test were done with "subtree_check" and as first
I thought that older clients are stable, but that's not true.
It only takes more time until svn aborts.


But the abort after 9hours was only one tests.
I will start the test again and see how long it takes
until it aborts.


Martin