[Cluster-devel] [GFS2 PATCH v2 14/15] GFS2: Hold onto iopen glock longer when dinode creation fails

[Bob Peterson <rpeterso@redhat.com>]

----- Original Message -----
> 
> Bob,
> 
>      Unsure if this is related to my other issues, but I should probably
>    at least pass this along:
> 
> void gfs2_glock_put(struct gfs2_glock *gl)
> {
>  ...
>  GLOCK_BUG_ON(gl, !list_empty(&gl->gl_holders)); <- this line
>  ...
> }
> 
> > @@ -883,6 +880,14 @@ fail_free_acls:
> >  		posix_acl_release(acl);
> >  fail_free_vfs_inode:
> >  	free_vfs_inode = 1;
> > +	/* We hold off until the very end to release the iopen glock. That
> > +	 * keeps other processes from acquiring it in EX mode and deleting
> > +	 * it while we're still using it. Since gfs2_delete_inode already
> > +	 * handles the iopen vs. inode glocks in any order, the lock order
> > +	 * does not matter. It must be done before iput, though, otherwise
> > +	 * we might get a segfault trying to dereference it. */
> > +	if (ip && ip->i_iopen_gh.gh_gl) /* if holder is linked to the glock */
> 
> via this line:
> 
> > +		gfs2_glock_put(ip->i_iopen_gh.gh_gl);
> 
> 
> [209071.114484] gfs2: G:  s:SH n:5/396fcce f:Iqob t:SH d:EX/0 a:0 v:0 r:-128
> m:200
> [209071.114493] gfs2:  H: s:SH f:EH e:0 p:40735 [nfsd]
> gfs2_glock_nq_init+0x11/0x40 [gfs2]
> [209071.114529] ------------[ cut here ]------------
> [209071.114530] kernel BUG at fs/gfs2/glock.c:208!
> [209071.114531] invalid opcode: 0000 [#1] SMP
> [209071.114555] Modules linked in: gfs2 dlm sctp drbd(OE) cts rpcsec_gss_krb5
> nfsv4 dns_resolver nfs fscache dm_service_time iscsi_tcp libiscsi_tcp
> libiscsi scsi_transport_iscsi 8021q garp mrp stp llc bonding nf_log_ipv4
> nf_log_common xt_LOG ipt_REJECT nf_reject_ipv4 xt_conntrack iptable_filter
> nf_conntrack_ftp nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack ip_tables
> dm_multipath x86_pkg_temp_thermal coretemp kvm_intel kvm crct10dif_pclmul
> crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel lrw gf128mul
> glue_helper ablk_helper cryptd iTCO_wdt ipmi_devintf iTCO_vendor_support
> nfsd sb_edac ipmi_si auth_rpcgss nfs_acl lpc_ich ipmi_msghandler dcdbas
> mfd_core edac_core pcspkr mei_me lockd wmi mei grace shpchp acpi_power_meter
> acpi_pad sunrpc binfmt_misc xfs mgag200 sr_mod cdrom syscopyarea sysfillrect
> [209071.114561]  sysimgblt i2c_algo_bit drm_kms_helper ttm sd_mod ahci ixgbe
> drm tg3 libahci mdio dca libata ptp megaraid_sas i2c_core pps_core dm_mirror
> dm_region_hash dm_log dm_mod
> [209071.114563] CPU: 4 PID: 40735 Comm: nfsd Tainted: G        W  OE
> 4.1.10_5 #1
> [209071.114563] Hardware name: Dell Inc. PowerEdge R720/0X3D66, BIOS 2.2.2
> 01/16/2014
> [209071.114564] task: ffff880035480d90 ti: ffff881f5c76c000 task.ti:
> ffff881f5c76c000
> [209071.114568] RIP: 0010:[<ffffffffa086ef49>]  [<ffffffffa086ef49>]
> gfs2_glock_put+0x139/0x160 [gfs2]
> [209071.114569] RSP: 0018:ffff881f5c76fa98  EFLAGS: 00010296
> [209071.114569] RAX: 0000000000000000 RBX: ffff88144026a940 RCX:
> 0000000000005298
> [209071.114570] RDX: 0000000052985298 RSI: 0000000000000286 RDI:
> 0000000000000286
> [209071.114585] RBP: ffff881f5c76fab8 R08: 000000000000004a R09:
> ffffffff81dbd15e
> [209071.114585] R10: 0000000000001cb8 R11: 0000000000000001 R12:
> 0000000000000000
> [209071.114585] R13: ffff883fed457000 R14: ffff88144026a970 R15:
> ffff881f1cb3c590
> [209071.114586] FS:  0000000000000000(0000) GS:ffff881fff880000(0000)
> knlGS:0000000000000000
> [209071.114587] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [209071.114587] CR2: 00007f83fbd67000 CR3: 000000000197e000 CR4:
> 00000000001406e0
> [209071.114588] Stack:
> [209071.114589]  ffff8804501a98c0 ffff883fed457000 0000000000000000
> ffff883fed457000
> [209071.114590]  ffff881f5c76fc08 ffffffffa087d04e ffff881ed6512470
> ffff8804501a98e0
> [209071.114590]  ffff881ed6512470 ffff881effffff86 ffff881ed6512470
> ffff8000fd5b5c80
> [209071.114591] Call Trace:
> [209071.114596]  [<ffffffffa087d04e>] gfs2_create_inode+0x77e/0x11b0 [gfs2]
> [209071.114600]  [<ffffffffa087c9a6>] ? gfs2_create_inode+0xd6/0x11b0 [gfs2]
> [209071.114603]  [<ffffffffa087dbcb>] gfs2_create+0x3b/0x40 [gfs2]
> [209071.114607]  [<ffffffff81284f5f>] ? security_inode_create+0x1f/0x30
> [209071.114609]  [<ffffffff811ed465>] vfs_create+0xd5/0x140
> [209071.114618]  [<ffffffffa049f141>] do_nfsd_create+0x481/0x600 [nfsd]
> [209071.114623]  [<ffffffffa04ab29a>] nfsd4_open+0x24a/0x830 [nfsd]
> [209071.114628]  [<ffffffffa04abd57>] nfsd4_proc_compound+0x4d7/0x7e0 [nfsd]
> [209071.114632]  [<ffffffffa0497f83>] nfsd_dispatch+0xc3/0x210 [nfsd]
> [209071.114658]  [<ffffffffa02ad352>] ? svc_tcp_adjust_wspace+0x12/0x30
> [sunrpc]
> [209071.114666]  [<ffffffffa02ac260>] svc_process_common+0x440/0x6d0 [sunrpc]
> [209071.114673]  [<ffffffffa02ac603>] svc_process+0x113/0x1b0 [sunrpc]
> [209071.114676]  [<ffffffffa049798f>] nfsd+0xff/0x170 [nfsd]
> [209071.114680]  [<ffffffffa0497890>] ? nfsd_destroy+0x80/0x80 [nfsd]
> [209071.114682]  [<ffffffff81096989>] kthread+0xc9/0xe0
> [209071.114683]  [<ffffffff810968c0>] ? kthread_create_on_node+0x180/0x180
> [209071.114685]  [<ffffffff81660462>] ret_from_fork+0x42/0x70
> [209071.114687]  [<ffffffff810968c0>] ? kthread_create_on_node+0x180/0x180
> [209071.114696] Code: 49 8b 04 24 48 85 c0 75 e9 eb b8 0f 1f 80 00 00 00 00
> f3 90 48 8b 10 83 e2 01 75 f6 e9 44 ff ff ff 48 89 de 31 ff e8 17 fb ff ff
> <0f> 0b 49 83 7c 24 50 00 74 89 48 89 de 31 ff e8 03 fb ff ff 0f
> [209071.114699] RIP  [<ffffffffa086ef49>] gfs2_glock_put+0x139/0x160 [gfs2]
> [209071.114699]  RSP <ffff881f5c76fa98>
> 
> 
> Thanks,
> 
> Andy
> 
> --
> Andrew W. Elble
> aweits at discipline.rit.edu
> Infrastructure Engineer, Communications Technical Lead
> Rochester Institute of Technology
> PGP: BFAD 8461 4CCF DC95 DA2C B0EB 965B 082E 863E C912
> 
Hi Andy,

Thanks. I'll investigate it.

BTW, I haven't found any more blatant bugs during testing, however I'm
debugging another issue. It seems as if I still have a reference counter
issue somewhere because if I slam GFS2 hard enough, I can get it to
accumulate millions of glocks that are never freed (unless memory pressure
causes the glock shrinker to be called). Having the extra glocks is causing
undue strain on the dlm, to the point where dlm can't keep up.
So it seems likely that I'll do another revision here anyway.
I'll keep you posted.

Regards,

Bob Peterson
Red Hat File Systems