Page alloc error on nfs4 via automount

Page alloc error on nfs4 via automount

[Michael Guntsche]

Good morning list,

After sorting out my Kerberos problem I noticed another problem on my
setup here when mounting a NFS4 export with automount.

Both, client and server, are running 2.6.35 and debian SID. sec is set
to sys. The timeframe varies but after not being mounted for some time,
upon first access I see

[512416.780875] mount.nfs4: page allocation failure. order:4,
mode:0xc0d0
[512416.788166] Call Trace:
[512416.790900] [c32e3c00] [c00082d4] show_stack+0x48/0x168 (unreliable)
[512416.798440] [c32e3c40] [c0062a28] __alloc_pages_nodemask+0x3d4/0x52c
[512416.806468] [c32e3ce0] [c0062ba0] __get_free_pages+0x20/0x50
[512416.813106] [c32e3cf0] [c9c82eb8] nfs_idmap_new+0x38/0xf4 [nfs]
[512416.820290] [c32e3d10] [c9c519b4] nfs4_set_client+0xe0/0x2f0 [nfs]
[512416.828388] [c32e3d60] [c9c521ec] nfs4_create_server+0xa8/0x31c
[nfs]
[512416.835842] [c32e3db0] [c9c5c618] nfs4_remote_get_sb+0x6c/0x184
[nfs]
[512416.843145] [c32e3de0] [c008dc90] vfs_kern_mount+0x68/0x128
[512416.849909] [c32e3e00] [c9c5c988] nfs_do_root_mount+0x6c/0x90 [nfs]
[512416.857047] [c32e3e30] [c9c5cdbc] nfs4_try_mount+0x58/0xe4 [nfs]
[512416.863825] [c32e3e50] [c9c5cfe8] nfs4_get_sb+0x1a0/0x438 [nfs]
[512416.869942] [c32e3e80] [c008dc90] vfs_kern_mount+0x68/0x128
[512416.875751] [c32e3ea0] [c008dda0] do_kern_mount+0x40/0x100
[512416.881402] [c32e3ec0] [c00a7130] do_mount+0x184/0x79c
[512416.886674] [c32e3f10] [c00a77f8] sys_mount+0xb0/0xdc
[512416.891884] [c32e3f40] [c001130c] ret_from_syscall+0x0/0x38
[512416.897634] --- Exception: c01 at 0xff5dd94
[512416.897645]     LR = 0x1000af4c
[512416.905323] Mem-Info:
[512416.907703] DMA per-cpu:
[512416.910384] CPU    0: hi:   42, btch:   7 usd:   0
[512416.915318] active_anon:479 inactive_anon:1888 isolated_anon:0
[512416.915329]  active_file:16867 inactive_file:5620 isolated_file:43
[512416.915341]  unevictable:493 dirty:16 writeback:0 unstable:0
[512416.915352]  free:2208 slab_reclaimable:742 slab_unreclaimable:2324
[512416.915364]  mapped:1683 shmem:40 pagetables:216 bounce:0
[512416.945259] DMA free:8832kB min:1440kB low:1800kB high:2160kB
active_anon:1916kB inactive_anon:7552kB active_file:67468kB
inactive_file:22480kB unevictable:1972kB isolated(anon):0kB
isolated(file):172kB present:130048kB mlocked:1972kB dirty:64kB
writeback:0kB mapped:6732kB shmem:160kB slab_reclaimable:2968kB
slab_unreclaimable:9296kB kernel_stack:608kB pagetables:864kB
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
[512416.985146] lowmem_reserve[]: 0 0 0
[512416.988783] DMA: 1964*4kB 92*8kB 3*16kB 4*32kB 1*64kB 0*128kB
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 8832kB
[512416.999289] 22954 total pagecache pages
[512417.003266] 0 pages in swap cache
[512417.006732] Swap cache stats: add 0, delete 0, find 0/0
[512417.012856] Free swap  = 0kB
[512417.016362] Total swap = 0kB
[512417.027596] 32768 pages RAM
[512417.031153] 1234 pages reserved
[512417.034933] 23334 pages shared
[512417.038613] 11052 pages non-shared

on the client. The filesystem is obviously not mounted then. A second
try succeeds on the other hand.

Since I just recently started using this nfs mounts on the client I
cannot say if this is a .35 only problem or was already happening with
older kernels.

Kind regards,
Michael Guntsche

Re: page alloc error on nfs4 via automount

[Jan Sanislo]

We have also noted this behavior locally.  And there is an outstanding
Red Hat bugzilla entry at:

 https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=593035

Page alloc error on nfs4 via automount

[Michael Guntsche]

Good morning,

I hit this problem once again, the backtrace is completely the same.

The server in question is exporting the same filesystem (a
subdirectory) as nfs3 and a second client is using this with automount
too. I do not see any problems there, it is an older kernel version
though.

I get this backtrace on first access to the autofs directory. Subsequent
access works and the filesystem gets mounted ok.

[1033591.057007] mount.nfs4: page allocation failure. order:4, mode:0xc0d0
[1033591.064574] Call Trace:
[1033591.067584] [c3183c00] [c00082d4] show_stack+0x48/0x168 (unreliable)
[1033591.076962] [c3183c40] [c0062a28] __alloc_pages_nodemask+0x3d4/0x52c
[1033591.085536] [c3183ce0] [c0062ba0] __get_free_pages+0x20/0x50
[1033591.093511] [c3183cf0] [c9c82eb8] nfs_idmap_new+0x38/0xf4 [nfs]
[1033591.101687] [c3183d10] [c9c519b4] nfs4_set_client+0xe0/0x2f0 [nfs]
[1033591.111524] [c3183d60] [c9c521ec] nfs4_create_server+0xa8/0x31c [nfs]
[1033591.118803] [c3183db0] [c9c5c618] nfs4_remote_get_sb+0x6c/0x184 [nfs]
[1033591.125775] [c3183de0] [c008dc90] vfs_kern_mount+0x68/0x128
[1033591.132075] [c3183e00] [c9c5c988] nfs_do_root_mount+0x6c/0x90 [nfs]
[1033591.139059] [c3183e30] [c9c5cdbc] nfs4_try_mount+0x58/0xe4 [nfs]
[1033591.145566] [c3183e50] [c9c5cfe8] nfs4_get_sb+0x1a0/0x438 [nfs]
[1033591.152019] [c3183e80] [c008dc90] vfs_kern_mount+0x68/0x128
[1033591.158030] [c3183ea0] [c008dda0] do_kern_mount+0x40/0x100
[1033591.163821] [c3183ec0] [c00a7130] do_mount+0x184/0x79c
[1033591.169199] [c3183f10] [c00a77f8] sys_mount+0xb0/0xdc
[1033591.174487] [c3183f40] [c001130c] ret_from_syscall+0x0/0x38
[1033591.180455] --- Exception: c01 at 0xff5dd94
[1033591.180469]     LR = 0x1000af4c
[1033591.188524] Mem-Info:
[1033591.191240] DMA per-cpu:
[1033591.194342] CPU    0: hi:   42, btch:   7 usd:   0
[1033591.199848] active_anon:125 inactive_anon:2084 isolated_anon:0
[1033591.199864]  active_file:13430 inactive_file:8648 isolated_file:39
[1033591.199876]  unevictable:493 dirty:2 writeback:0 unstable:0
[1033591.199887]  free:2460 slab_reclaimable:798 slab_unreclaimable:2595
[1033591.199899]  mapped:1510 shmem:40 pagetables:201 bounce:0
[1033591.232165] DMA free:9840kB min:1440kB low:1800kB high:2160kB active_anon:500kB inactive_anon:8336kB active_file:53720kB inactive_file:34592kB unevictable:1972kB isolated(anon):0kB isolated(file):156kB present:130048kB mlocked:1972kB dirty:8kB writeback:0kB mapped:6040kB shmem:160kB slab_reclaimable:3192kB slab_unreclaimable:10380kB kernel_stack:584kB pagetables:804kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[1033591.273677] lowmem_reserve[]: 0 0 0
[1033591.277820] DMA: 998*4kB 469*8kB 13*16kB 57*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 9840kB
[1033591.289330] 22552 total pagecache pages
[1033591.293474] 0 pages in swap cache
[1033591.297054] Swap cache stats: add 0, delete 0, find 0/0
[1033591.302497] Free swap  = 0kB
[1033591.305571] Total swap = 0kB
[1033591.316097] 32768 pages RAM
[1033591.319299] 1234 pages reserved
[1033591.322687] 22622 pages shared
[1033591.326074] 11104 pages non-shared

Kind regards,
Michael Guntsche

Re: Page alloc error on nfs4 via automount

[Trond Myklebust]

On Mon, 2010-08-16 at 08:16 +0200, Michael Guntsche wrote:
> Good morning,
> 
> I hit this problem once again, the backtrace is completely the same.
> 
> The server in question is exporting the same filesystem (a
> subdirectory) as nfs3 and a second client is using this with automount
> too. I do not see any problems there, it is an older kernel version
> though.
> 
> I get this backtrace on first access to the autofs directory. Subsequent
> access works and the filesystem gets mounted ok.
> 
> [1033591.057007] mount.nfs4: page allocation failure. order:4, mode:0xc0d0
> [1033591.064574] Call Trace:
> [1033591.067584] [c3183c00] [c00082d4] show_stack+0x48/0x168 (unreliable)
> [1033591.076962] [c3183c40] [c0062a28] __alloc_pages_nodemask+0x3d4/0x52c
> [1033591.085536] [c3183ce0] [c0062ba0] __get_free_pages+0x20/0x50
> [1033591.093511] [c3183cf0] [c9c82eb8] nfs_idmap_new+0x38/0xf4 [nfs]
> [1033591.101687] [c3183d10] [c9c519b4] nfs4_set_client+0xe0/0x2f0 [nfs]
> [1033591.111524] [c3183d60] [c9c521ec] nfs4_create_server+0xa8/0x31c [nfs]
> [1033591.118803] [c3183db0] [c9c5c618] nfs4_remote_get_sb+0x6c/0x184 [nfs]
> [1033591.125775] [c3183de0] [c008dc90] vfs_kern_mount+0x68/0x128
> [1033591.132075] [c3183e00] [c9c5c988] nfs_do_root_mount+0x6c/0x90 [nfs]
> [1033591.139059] [c3183e30] [c9c5cdbc] nfs4_try_mount+0x58/0xe4 [nfs]
> [1033591.145566] [c3183e50] [c9c5cfe8] nfs4_get_sb+0x1a0/0x438 [nfs]
> [1033591.152019] [c3183e80] [c008dc90] vfs_kern_mount+0x68/0x128
> [1033591.158030] [c3183ea0] [c008dda0] do_kern_mount+0x40/0x100
> [1033591.163821] [c3183ec0] [c00a7130] do_mount+0x184/0x79c
> [1033591.169199] [c3183f10] [c00a77f8] sys_mount+0xb0/0xdc
> [1033591.174487] [c3183f40] [c001130c] ret_from_syscall+0x0/0x38
> [1033591.180455] --- Exception: c01 at 0xff5dd94
> [1033591.180469]     LR = 0x1000af4c
> [1033591.188524] Mem-Info:
> [1033591.191240] DMA per-cpu:
> [1033591.194342] CPU    0: hi:   42, btch:   7 usd:   0
> [1033591.199848] active_anon:125 inactive_anon:2084 isolated_anon:0
> [1033591.199864]  active_file:13430 inactive_file:8648 isolated_file:39
> [1033591.199876]  unevictable:493 dirty:2 writeback:0 unstable:0
> [1033591.199887]  free:2460 slab_reclaimable:798 slab_unreclaimable:2595
> [1033591.199899]  mapped:1510 shmem:40 pagetables:201 bounce:0
> [1033591.232165] DMA free:9840kB min:1440kB low:1800kB high:2160kB active_anon:500kB inactive_anon:8336kB active_file:53720kB inactive_file:34592kB unevictable:1972kB isolated(anon):0kB isolated(file):156kB present:130048kB mlocked:1972kB dirty:8kB writeback:0kB mapped:6040kB shmem:160kB slab_reclaimable:3192kB slab_unreclaimable:10380kB kernel_stack:584kB pagetables:804kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> [1033591.273677] lowmem_reserve[]: 0 0 0
> [1033591.277820] DMA: 998*4kB 469*8kB 13*16kB 57*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 9840kB
> [1033591.289330] 22552 total pagecache pages
> [1033591.293474] 0 pages in swap cache
> [1033591.297054] Swap cache stats: add 0, delete 0, find 0/0
> [1033591.302497] Free swap  = 0kB
> [1033591.305571] Total swap = 0kB
> [1033591.316097] 32768 pages RAM
> [1033591.319299] 1234 pages reserved
> [1033591.322687] 22622 pages shared
> [1033591.326074] 11104 pages non-shared

It looks like a regression in the Linux kernel memory management.

I agree that order 4 allocations are always going to be unreliable, but
if it was consistently working before, and is now consistently not
working, then something must have changed, and I know that part of the
NFS code has not changed...

Cheers
  Trond

Re: Page alloc error on nfs4 via automount

[Michael Guntsche]

On 17 Aug 10 19:21, Trond Myklebust wrote:
> It looks like a regression in the Linux kernel memory management.
> 
> I agree that order 4 allocations are always going to be unreliable, but
> if it was consistently working before, and is now consistently not
> working, then something must have changed, and I know that part of the
> NFS code has not changed...

Hello,

Just to clarify this. The setup with automount adn NFS4 is new so I
while the NFS3 setup is working for years now. I never experience this
with NFS3 so it may be a NFS4 only problem. As you can see from the
mails I am not the only one having the problem with version 4 in this
regard. I think setups with automount and version 3 are much more common
so a regression there would have been noticed a lot faster.

Kind regards,
Michael

Re: Page alloc error on nfs4 via automount

[Michael Guntsche]

On 17 Aug 10 19:21, Trond Myklebust wrote:
> On Mon, 2010-08-16 at 08:16 +0200, Michael Guntsche wrote:
> > Good morning,
> > 
> > I hit this problem once again, the backtrace is completely the same.
> > 
> > The server in question is exporting the same filesystem (a
> > subdirectory) as nfs3 and a second client is using this with automount
> > too. I do not see any problems there, it is an older kernel version
> > though.
> > 
> > I get this backtrace on first access to the autofs directory. Subsequent
> > access works and the filesystem gets mounted ok.
> > 
This is still happening with 2.6.38-rc5+ on the client side.

[506826.480330] mount.nfs4: page allocation failure. order:4, mode:0xc0d0
[506826.488114] Call Trace:
[506826.492244] [c799fbc0] [c0008908] show_stack+0x70/0x1bc (unreliable)
[506826.501169] [c799fc00] [c006aa08] __alloc_pages_nodemask+0x5f0/0x6fc
[506826.509391] [c799fcb0] [c006ab34] __get_free_pages+0x20/0x50
[506826.517247] [c799fcc0] [c9dd8538] nfs_idmap_new+0x40/0x100 [nfs]
[506826.525327] [c799fce0] [c9da2dd0] nfs4_set_client+0x128/0x2d8 [nfs]
[506826.533309] [c799fd30] [c9da35e4] nfs4_create_server+0xc8/0x37c [nfs]
[506826.540360] [c799fd80] [c9daec9c] nfs4_remote_mount+0x50/0x1bc [nfs]
[506826.547240] [c799fdb0] [c009cccc] vfs_kern_mount+0x6c/0x15c
[506826.553367] [c799fde0] [c9daf0c0] nfs_do_root_mount+0x84/0xc0 [nfs]
[506826.560283] [c799fe10] [c9daf5a4] nfs4_try_mount+0x64/0x104 [nfs]
[506826.566596] [c799fe40] [c9daf700] nfs4_get_sb+0xbc/0x3dc [nfs]
[506826.572928] [c799fe70] [c009cd80] vfs_kern_mount+0x120/0x15c
[506826.579071] [c799fea0] [c009ce20] do_kern_mount+0x54/0x118
[506826.585077] [c799fec0] [c00b9934] do_mount+0x410/0x7f0
[506826.590388] [c799ff10] [c00b9db4] sys_mount+0xa0/0xf8
[506826.595891] [c799ff40] [c0010184] ret_from_syscall+0x0/0x38
[506826.602029] --- Exception: c01 at 0xff5de64
[506826.602043]     LR = 0x1000af4c
[506826.609918] Mem-Info:
[506826.612299] DMA per-cpu:
[506826.615042] CPU    0: hi:   42, btch:   7 usd:   0
[506826.620449] active_anon:2420 inactive_anon:20 isolated_anon:6
[506826.620465]  active_file:12609 inactive_file:11601 isolated_file:25
[506826.620476]  unevictable:493 dirty:0 writeback:0 unstable:0
[506826.620486]  free:612 slab_reclaimable:802 slab_unreclaimable:1992
[506826.620497]  mapped:1438 shmem:32 pagetables:200 bounce:0
[506826.650982] DMA free:2448kB min:1440kB low:1800kB high:2160kB active_anon:9680kB inactive_anon:80kB active_file:50436kB inactive_file:46404kB unevictable:1972kB isolated(anon):24kB isolated(file):100kB present:130048kB mlocked:1972kB dirty:0kB writeback:0kB mapped:5752kB shmem:128kB slab_reclaimable:3208kB slab_unreclaimable:7968kB kernel_stack:568kB pagetables:800kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[506826.691851] lowmem_reserve[]: 0 0 0
[506826.695867] DMA: 408*4kB 30*8kB 14*16kB 9*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2448kB
[506826.707258] 24680 total pagecache pages
[506826.711533] 0 pages in swap cache
[506826.715150] Swap cache stats: add 0, delete 0, find 0/0
[506826.720858] Free swap  = 0kB
[506826.724007] Total swap = 0kB
[506826.733886] 32768 pages RAM
[506826.737069] 1203 pages reserved
[506826.740597] 22349 pages shared
[506826.744047] 13330 pages non-shared

The setup is still the same NFS4 automount. It is not happening
repeatedly though. Is it possible that memory gets fragmented over time
and therefore this error is occuring? As always subsequent access
succeeds.

Kind regards,
Michael