pnfs: WRITE operation will not send LAYOUTCOMMI OP to MDS if use filelayout

pnfs: WRITE operation will not send LAYOUTCOMMI OP to MDS if use filelayout

[chenditang]

In the recent pnfs kernel versions, WRITE operation will not send
LAYOUTCOMMI OP to MDS if use filelayout.

In nfs_write_rpcsetup function, the value of the variable 'stable' will
always be NFS_FILE_SYNC(ncommit is 0). so NFS_INO_LAYOUTCOMMIT 
flag will not be set.

The value of the variable ncommit' will be ++ in the later process
(nfs_writeback_release_common function).


Thanks!

Re: pnfs: WRITE operation will not send LAYOUTCOMMI OP to MDS if use filelayout

[Benny Halevy]

On 2013-03-18 09:11, chenditang wrote:
> In the recent pnfs kernel versions, WRITE operation will not send
> LAYOUTCOMMI OP to MDS if use filelayout.

I presume that the file server sets NFL4_UFLG_COMMIT_THRU_MDS
and that the DS is returning how=FILE_SYNC and that the client
is implementing the behavior described in  Errata ID: 2751
http://www.rfc-editor.org/errata_search.php?rfc=5661

Do you have a tcp dump of client, MDS, and DS traffic?

> 
> In nfs_write_rpcsetup function, the value of the variable 'stable' will
> always be NFS_FILE_SYNC(ncommit is 0). so NFS_INO_LAYOUTCOMMIT 
> flag will not be set.

What's the value of the "how" arg to nfs_write_rpcsetup that you see?
At any rate, if you look at filelayout_set_layoutcommit, it will not
call pnfs_set_layoutcommit in the following cases:

        if (FILELAYOUT_LSEG(hdr->lseg)->commit_through_mds ||
            wdata->res.verf->committed == NFS_FILE_SYNC)
                return;

Benny

> 
> The value of the variable ncommit' will be ++ in the later process
> (nfs_writeback_release_common function).
> 
> 
> Thanks!
> 

Re: pnfs: WRITE operation will not send LAYOUTCOMMI OP to MDS if use filelayout

[chenditang]

于 2013年03月18日 17:01, Benny Halevy 写道:
> On 2013-03-18 09:11, chenditang wrote:
>> In the recent pnfs kernel versions, WRITE operation will not send
>> LAYOUTCOMMI OP to MDS if use filelayout.
> I presume that the file server sets NFL4_UFLG_COMMIT_THRU_MDS
> and that the DS is returning how=FILE_SYNC and that the client
> is implementing the behavior described in  Errata ID: 2751
> http://www.rfc-editor.org/errata_search.php?rfc=5661
>
> Do you have a tcp dump of client, MDS, and DS traffic?
NFL4_UFLG_COMMIT_THRU_MDS is not set, "nfl_util" is 8192 and "commit_through_mds" is 0 to filelayout_decode_layout.

>> In nfs_write_rpcsetup function, the value of the variable 'stable' will
>> always be NFS_FILE_SYNC(ncommit is 0). so NFS_INO_LAYOUTCOMMIT
>> flag will not be set.
> What's the value of the "how" arg to nfs_write_rpcsetup that you see?
> At any rate, if you look at filelayout_set_layoutcommit, it will not
> call pnfs_set_layoutcommit in the following cases:
>
>          if (FILELAYOUT_LSEG(hdr->lseg)->commit_through_mds ||
>              wdata->res.verf->committed == NFS_FILE_SYNC)
>                  return;
"how" is 32, and "cinfo->mds->ncommit" is 0. pnfs_set_layoutcommit has 
not beeb called.
I think "cinfo->mds->ncommit" can not be used here.

I dump the stack, and the initial value of "ncommit" is 0, then it to be 
used in nfs_write_rpcsetup function,
but "ncommit++" in nfs_writeback_release_common function(if 
NFS_FILE_SYNC is not set).

static const struct rpc_call_ops nfs_write_common_ops = {
.rpc_call_prepare = nfs_write_prepare,
.rpc_call_done = nfs_writeback_done_common,
.rpc_release = nfs_writeback_release_common,
};

The code processes:
step1: init_once --> nfsi->commit_info.ncommit = 0;
setp2: nfs_flush_one --> nfs_write_rpcsetup
setp3: nfs_writeback_done --> nfs_writeback_done --> 
filelayout_write_done_cb --> filelayout_set_layoutcommit
setp4: nfs_writeback_release_common--> ...-->nfs_write_completion --> 
.... --> cinfo->mds->ncommit++;
>
>
> Benny
>
>> The value of the variable ncommit' will be ++ in the later process
>> (nfs_writeback_release_common function).
>>
>>
>> Thanks!
>>

RE: pnfs: WRITE operation will not send LAYOUTCOMMI OP to MDS if use filelayout

[Myklebust, Trond]

PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBsaW51eC1uZnMtb3duZXJAdmdl
ci5rZXJuZWwub3JnIFttYWlsdG86bGludXgtbmZzLQ0KPiBvd25lckB2Z2VyLmtlcm5lbC5vcmdd
IE9uIEJlaGFsZiBPZiBjaGVuZGl0YW5nDQo+IFNlbnQ6IE1vbmRheSwgTWFyY2ggMTgsIDIwMTMg
MzoxMiBBTQ0KPiBUbzogYmhhbGV2eUB0b25pYW4uY29tDQo+IENjOiBsaW51eC1uZnNAdmdlci5r
ZXJuZWwub3JnOyBMSU5FDQo+IFN1YmplY3Q6IHBuZnM6IFdSSVRFIG9wZXJhdGlvbiB3aWxsIG5v
dCBzZW5kIExBWU9VVENPTU1JIE9QIHRvIE1EUyBpZg0KPiB1c2UgZmlsZWxheW91dA0KPiANCj4g
SW4gdGhlIHJlY2VudCBwbmZzIGtlcm5lbCB2ZXJzaW9ucywgV1JJVEUgb3BlcmF0aW9uIHdpbGwg
bm90IHNlbmQNCj4gTEFZT1VUQ09NTUkgT1AgdG8gTURTIGlmIHVzZSBmaWxlbGF5b3V0Lg0KPiAN
Cj4gSW4gbmZzX3dyaXRlX3JwY3NldHVwIGZ1bmN0aW9uLCB0aGUgdmFsdWUgb2YgdGhlIHZhcmlh
YmxlICdzdGFibGUnIHdpbGwgYWx3YXlzDQo+IGJlIE5GU19GSUxFX1NZTkMobmNvbW1pdCBpcyAw
KS4gc28gTkZTX0lOT19MQVlPVVRDT01NSVQgZmxhZyB3aWxsIG5vdA0KPiBiZSBzZXQuDQoNCklm
IHRoZXNlIGFyZSBzbWFsbCBmaWxlcywgdGhlbiB0aGF0IGlzIGV4cGVjdGVkIGJlaGF2aW91ci4g
VGhlIHBORlMgc3BlYyBkb2VzIG5vdCBleGVtcHQgdGhlIERTIGZyb20gaGF2aW5nIHRvIGhvbm91
ciB0aGUgJ3N0YWJsZScgZmxhZy4NCg0KQ2hlZXJzDQogIFRyb25kDQo=