nfs root problem in linux-next

nfs root problem in linux-next

[Mark Salter]

As of Friday's linux-next I've been unable to successfully boot using an
nfs root with a c6x board. I bisected it to this patch:

  commit cc3b296fd9a01c9e654bcb9373af70dd6e32fb24
  Author: Trond Myklebust <Trond.Myklebust-HgOvQuBEEgTQT0dZR+AlfA@public.gmane.org>
  Date:   Thu Jun 21 14:53:10 2012 -0400

      SUNRPC: Clean up xdr_read_pages and xdr_enter_page

I placed "good-boot" and "bad-boot" packet capture files at:

  http://people.redhat.com/~msalter/linux-next/

The only difference in the kernels used is the above patch. The
bad-boot mounted the nfs root but was unable to find the init
file which caused a panic. The nfsroot= parameter uses v3,tcp
flags and the server side is running rhel6.

I'm not at all familiar with the nfs code, but can help with
testing or providing any addtional info needed.

--Mark


--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

nfs root problem in linux-next

[Mark Salter]

As of Friday's linux-next I've been unable to successfully boot using an
nfs root with a c6x board. I bisected it to this patch:

  commit cc3b296fd9a01c9e654bcb9373af70dd6e32fb24
  Author: Trond Myklebust <Trond.Myklebust@netapp.com>
  Date:   Thu Jun 21 14:53:10 2012 -0400

      SUNRPC: Clean up xdr_read_pages and xdr_enter_page

I placed "good-boot" and "bad-boot" packet capture files at:

  http://people.redhat.com/~msalter/linux-next/

The only difference in the kernels used is the above patch. The
bad-boot mounted the nfs root but was unable to find the init
file which caused a panic. The nfsroot= parameter uses v3,tcp
flags and the server side is running rhel6.

I'm not at all familiar with the nfs code, but can help with
testing or providing any addtional info needed.

--Mark

Re: nfs root problem in linux-next

[Myklebust, Trond]

On Mon, 2012-06-25 at 16:26 -0400, Mark Salter wrote:
> As of Friday's linux-next I've been unable to successfully boot using an
> nfs root with a c6x board. I bisected it to this patch:
> 
>   commit cc3b296fd9a01c9e654bcb9373af70dd6e32fb24
>   Author: Trond Myklebust <Trond.Myklebust@netapp.com>
>   Date:   Thu Jun 21 14:53:10 2012 -0400
> 
>       SUNRPC: Clean up xdr_read_pages and xdr_enter_page
> 
> I placed "good-boot" and "bad-boot" packet capture files at:
> 
>   http://people.redhat.com/~msalter/linux-next/
> 
> The only difference in the kernels used is the above patch. The
> bad-boot mounted the nfs root but was unable to find the init
> file which caused a panic. The nfsroot= parameter uses v3,tcp
> flags and the server side is running rhel6.
> 
> I'm not at all familiar with the nfs code, but can help with
> testing or providing any addtional info needed.

Hi Mark,

So you are saying that reverting that patch alone suffices to fix the
problem that you are seeing?

Firstly, a big "Thank You!!!" for testing the linux-next branch. I very
much appreciate that you report the bugs before they get merged
upstream.

I'll have a look to see if I can find out what is the problem. The only
significant change that patch should be introducing would be the use of
xdr_stream_pos().

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

Re: nfs root problem in linux-next

[Myklebust, Trond]

T24gTW9uLCAyMDEyLTA2LTI1IGF0IDE2OjI2IC0wNDAwLCBNYXJrIFNhbHRlciB3cm90ZToNCj4g
QXMgb2YgRnJpZGF5J3MgbGludXgtbmV4dCBJJ3ZlIGJlZW4gdW5hYmxlIHRvIHN1Y2Nlc3NmdWxs
eSBib290IHVzaW5nIGFuDQo+IG5mcyByb290IHdpdGggYSBjNnggYm9hcmQuIEkgYmlzZWN0ZWQg
aXQgdG8gdGhpcyBwYXRjaDoNCj4gDQo+ICAgY29tbWl0IGNjM2IyOTZmZDlhMDFjOWU2NTRiY2I5
MzczYWY3MGRkNmUzMmZiMjQNCj4gICBBdXRob3I6IFRyb25kIE15a2xlYnVzdCA8VHJvbmQuTXlr
bGVidXN0QG5ldGFwcC5jb20+DQo+ICAgRGF0ZTogICBUaHUgSnVuIDIxIDE0OjUzOjEwIDIwMTIg
LTA0MDANCj4gDQo+ICAgICAgIFNVTlJQQzogQ2xlYW4gdXAgeGRyX3JlYWRfcGFnZXMgYW5kIHhk
cl9lbnRlcl9wYWdlDQo+IA0KPiBJIHBsYWNlZCAiZ29vZC1ib290IiBhbmQgImJhZC1ib290IiBw
YWNrZXQgY2FwdHVyZSBmaWxlcyBhdDoNCj4gDQo+ICAgaHR0cDovL3Blb3BsZS5yZWRoYXQuY29t
L35tc2FsdGVyL2xpbnV4LW5leHQvDQo+IA0KPiBUaGUgb25seSBkaWZmZXJlbmNlIGluIHRoZSBr
ZXJuZWxzIHVzZWQgaXMgdGhlIGFib3ZlIHBhdGNoLiBUaGUNCj4gYmFkLWJvb3QgbW91bnRlZCB0
aGUgbmZzIHJvb3QgYnV0IHdhcyB1bmFibGUgdG8gZmluZCB0aGUgaW5pdA0KPiBmaWxlIHdoaWNo
IGNhdXNlZCBhIHBhbmljLiBUaGUgbmZzcm9vdD0gcGFyYW1ldGVyIHVzZXMgdjMsdGNwDQo+IGZs
YWdzIGFuZCB0aGUgc2VydmVyIHNpZGUgaXMgcnVubmluZyByaGVsNi4NCj4gDQo+IEknbSBub3Qg
YXQgYWxsIGZhbWlsaWFyIHdpdGggdGhlIG5mcyBjb2RlLCBidXQgY2FuIGhlbHAgd2l0aA0KPiB0
ZXN0aW5nIG9yIHByb3ZpZGluZyBhbnkgYWRkdGlvbmFsIGluZm8gbmVlZGVkLg0KDQpIaSBNYXJr
LA0KDQpTbyB5b3UgYXJlIHNheWluZyB0aGF0IHJldmVydGluZyB0aGF0IHBhdGNoIGFsb25lIHN1
ZmZpY2VzIHRvIGZpeCB0aGUNCnByb2JsZW0gdGhhdCB5b3UgYXJlIHNlZWluZz8NCg0KRmlyc3Rs
eSwgYSBiaWcgIlRoYW5rIFlvdSEhISIgZm9yIHRlc3RpbmcgdGhlIGxpbnV4LW5leHQgYnJhbmNo
LiBJIHZlcnkNCm11Y2ggYXBwcmVjaWF0ZSB0aGF0IHlvdSByZXBvcnQgdGhlIGJ1Z3MgYmVmb3Jl
IHRoZXkgZ2V0IG1lcmdlZA0KdXBzdHJlYW0uDQoNCkknbGwgaGF2ZSBhIGxvb2sgdG8gc2VlIGlm
IEkgY2FuIGZpbmQgb3V0IHdoYXQgaXMgdGhlIHByb2JsZW0uIFRoZSBvbmx5DQpzaWduaWZpY2Fu
dCBjaGFuZ2UgdGhhdCBwYXRjaCBzaG91bGQgYmUgaW50cm9kdWNpbmcgd291bGQgYmUgdGhlIHVz
ZSBvZg0KeGRyX3N0cmVhbV9wb3MoKS4NCg0KLS0gDQpUcm9uZCBNeWtsZWJ1c3QNCkxpbnV4IE5G
UyBjbGllbnQgbWFpbnRhaW5lcg0KDQpOZXRBcHANClRyb25kLk15a2xlYnVzdEBuZXRhcHAuY29t
DQp3d3cubmV0YXBwLmNvbQ0KDQo=

Re: nfs root problem in linux-next

[Mark Salter]

On Mon, 2012-06-25 at 23:14 +0000, Myklebust, Trond wrote:
> On Mon, 2012-06-25 at 16:26 -0400, Mark Salter wrote:
> > As of Friday's linux-next I've been unable to successfully boot using an
> > nfs root with a c6x board. I bisected it to this patch:
> > 
> >   commit cc3b296fd9a01c9e654bcb9373af70dd6e32fb24
> >   Author: Trond Myklebust <Trond.Myklebust-HgOvQuBEEgTQT0dZR+AlfA@public.gmane.org>
> >   Date:   Thu Jun 21 14:53:10 2012 -0400
> > 
> >       SUNRPC: Clean up xdr_read_pages and xdr_enter_page
> > 
> > I placed "good-boot" and "bad-boot" packet capture files at:
> > 
> >   http://people.redhat.com/~msalter/linux-next/
> > 
> > The only difference in the kernels used is the above patch. The
> > bad-boot mounted the nfs root but was unable to find the init
> > file which caused a panic. The nfsroot= parameter uses v3,tcp
> > flags and the server side is running rhel6.
> > 
> > I'm not at all familiar with the nfs code, but can help with
> > testing or providing any addtional info needed.
> 
> Hi Mark,
> 
> So you are saying that reverting that patch alone suffices to fix the
> problem that you are seeing?

Yes. If I check out a tree at commit cc3b296fd9a, the resulting kernel
does not boot (it panics that it cannot find init). If I check out a
tree at  cc3b296fd9a~1, it boots the nfs root successfully.

...
> I'll have a look to see if I can find out what is the problem. The only
> significant change that patch should be introducing would be the use of
> xdr_stream_pos().

Okay. I am happy to help with any testing if needed to help characterize
the problem better.

--Mark




--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nfs root problem in linux-next

[Mark Salter]

On Mon, 2012-06-25 at 23:14 +0000, Myklebust, Trond wrote:
> On Mon, 2012-06-25 at 16:26 -0400, Mark Salter wrote:
> > As of Friday's linux-next I've been unable to successfully boot using an
> > nfs root with a c6x board. I bisected it to this patch:
> > 
> >   commit cc3b296fd9a01c9e654bcb9373af70dd6e32fb24
> >   Author: Trond Myklebust <Trond.Myklebust@netapp.com>
> >   Date:   Thu Jun 21 14:53:10 2012 -0400
> > 
> >       SUNRPC: Clean up xdr_read_pages and xdr_enter_page
> > 
> > I placed "good-boot" and "bad-boot" packet capture files at:
> > 
> >   http://people.redhat.com/~msalter/linux-next/
> > 
> > The only difference in the kernels used is the above patch. The
> > bad-boot mounted the nfs root but was unable to find the init
> > file which caused a panic. The nfsroot= parameter uses v3,tcp
> > flags and the server side is running rhel6.
> > 
> > I'm not at all familiar with the nfs code, but can help with
> > testing or providing any addtional info needed.
> 
> Hi Mark,
> 
> So you are saying that reverting that patch alone suffices to fix the
> problem that you are seeing?

Yes. If I check out a tree at commit cc3b296fd9a, the resulting kernel
does not boot (it panics that it cannot find init). If I check out a
tree at  cc3b296fd9a~1, it boots the nfs root successfully.

...
> I'll have a look to see if I can find out what is the problem. The only
> significant change that patch should be introducing would be the use of
> xdr_stream_pos().

Okay. I am happy to help with any testing if needed to help characterize
the problem better.

--Mark

Re: nfs root problem in linux-next

[Myklebust, Trond]

On Mon, 2012-06-25 at 23:17 -0400, Mark Salter wrote:
> On Mon, 2012-06-25 at 23:14 +0000, Myklebust, Trond wrote:
> > On Mon, 2012-06-25 at 16:26 -0400, Mark Salter wrote:
> > > As of Friday's linux-next I've been unable to successfully boot using an
> > > nfs root with a c6x board. I bisected it to this patch:
> > > 
> > >   commit cc3b296fd9a01c9e654bcb9373af70dd6e32fb24
> > >   Author: Trond Myklebust <Trond.Myklebust@netapp.com>
> > >   Date:   Thu Jun 21 14:53:10 2012 -0400
> > > 
> > >       SUNRPC: Clean up xdr_read_pages and xdr_enter_page
> > > 
> > > I placed "good-boot" and "bad-boot" packet capture files at:
> > > 
> > >   http://people.redhat.com/~msalter/linux-next/
> > > 
> > > The only difference in the kernels used is the above patch. The
> > > bad-boot mounted the nfs root but was unable to find the init
> > > file which caused a panic. The nfsroot= parameter uses v3,tcp
> > > flags and the server side is running rhel6.
> > > 
> > > I'm not at all familiar with the nfs code, but can help with
> > > testing or providing any addtional info needed.
> > 
> > Hi Mark,
> > 
> > So you are saying that reverting that patch alone suffices to fix the
> > problem that you are seeing?
> 
> Yes. If I check out a tree at commit cc3b296fd9a, the resulting kernel
> does not boot (it panics that it cannot find init). If I check out a
> tree at  cc3b296fd9a~1, it boots the nfs root successfully.
> 
> ...
> > I'll have a look to see if I can find out what is the problem. The only
> > significant change that patch should be introducing would be the use of
> > xdr_stream_pos().
> 
> Okay. I am happy to help with any testing if needed to help characterize
> the problem better.

Hi Mark and Fengguang,

I believe that I've found the bug that was being triggered in the above
patch. It was a mistake in the earlier patch "SUNRPC: Don't decode
beyond the end of the RPC reply message", which was failing to correctly
initialise the xdr->nwords field.

I've now fixed it and updated the linux-next branch.

Cheers
  Trond
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

Re: nfs root problem in linux-next

[Myklebust, Trond]

T24gTW9uLCAyMDEyLTA2LTI1IGF0IDIzOjE3IC0wNDAwLCBNYXJrIFNhbHRlciB3cm90ZToNCj4g
T24gTW9uLCAyMDEyLTA2LTI1IGF0IDIzOjE0ICswMDAwLCBNeWtsZWJ1c3QsIFRyb25kIHdyb3Rl
Og0KPiA+IE9uIE1vbiwgMjAxMi0wNi0yNSBhdCAxNjoyNiAtMDQwMCwgTWFyayBTYWx0ZXIgd3Jv
dGU6DQo+ID4gPiBBcyBvZiBGcmlkYXkncyBsaW51eC1uZXh0IEkndmUgYmVlbiB1bmFibGUgdG8g
c3VjY2Vzc2Z1bGx5IGJvb3QgdXNpbmcgYW4NCj4gPiA+IG5mcyByb290IHdpdGggYSBjNnggYm9h
cmQuIEkgYmlzZWN0ZWQgaXQgdG8gdGhpcyBwYXRjaDoNCj4gPiA+IA0KPiA+ID4gICBjb21taXQg
Y2MzYjI5NmZkOWEwMWM5ZTY1NGJjYjkzNzNhZjcwZGQ2ZTMyZmIyNA0KPiA+ID4gICBBdXRob3I6
IFRyb25kIE15a2xlYnVzdCA8VHJvbmQuTXlrbGVidXN0QG5ldGFwcC5jb20+DQo+ID4gPiAgIERh
dGU6ICAgVGh1IEp1biAyMSAxNDo1MzoxMCAyMDEyIC0wNDAwDQo+ID4gPiANCj4gPiA+ICAgICAg
IFNVTlJQQzogQ2xlYW4gdXAgeGRyX3JlYWRfcGFnZXMgYW5kIHhkcl9lbnRlcl9wYWdlDQo+ID4g
PiANCj4gPiA+IEkgcGxhY2VkICJnb29kLWJvb3QiIGFuZCAiYmFkLWJvb3QiIHBhY2tldCBjYXB0
dXJlIGZpbGVzIGF0Og0KPiA+ID4gDQo+ID4gPiAgIGh0dHA6Ly9wZW9wbGUucmVkaGF0LmNvbS9+
bXNhbHRlci9saW51eC1uZXh0Lw0KPiA+ID4gDQo+ID4gPiBUaGUgb25seSBkaWZmZXJlbmNlIGlu
IHRoZSBrZXJuZWxzIHVzZWQgaXMgdGhlIGFib3ZlIHBhdGNoLiBUaGUNCj4gPiA+IGJhZC1ib290
IG1vdW50ZWQgdGhlIG5mcyByb290IGJ1dCB3YXMgdW5hYmxlIHRvIGZpbmQgdGhlIGluaXQNCj4g
PiA+IGZpbGUgd2hpY2ggY2F1c2VkIGEgcGFuaWMuIFRoZSBuZnNyb290PSBwYXJhbWV0ZXIgdXNl
cyB2Myx0Y3ANCj4gPiA+IGZsYWdzIGFuZCB0aGUgc2VydmVyIHNpZGUgaXMgcnVubmluZyByaGVs
Ni4NCj4gPiA+IA0KPiA+ID4gSSdtIG5vdCBhdCBhbGwgZmFtaWxpYXIgd2l0aCB0aGUgbmZzIGNv
ZGUsIGJ1dCBjYW4gaGVscCB3aXRoDQo+ID4gPiB0ZXN0aW5nIG9yIHByb3ZpZGluZyBhbnkgYWRk
dGlvbmFsIGluZm8gbmVlZGVkLg0KPiA+IA0KPiA+IEhpIE1hcmssDQo+ID4gDQo+ID4gU28geW91
IGFyZSBzYXlpbmcgdGhhdCByZXZlcnRpbmcgdGhhdCBwYXRjaCBhbG9uZSBzdWZmaWNlcyB0byBm
aXggdGhlDQo+ID4gcHJvYmxlbSB0aGF0IHlvdSBhcmUgc2VlaW5nPw0KPiANCj4gWWVzLiBJZiBJ
IGNoZWNrIG91dCBhIHRyZWUgYXQgY29tbWl0IGNjM2IyOTZmZDlhLCB0aGUgcmVzdWx0aW5nIGtl
cm5lbA0KPiBkb2VzIG5vdCBib290IChpdCBwYW5pY3MgdGhhdCBpdCBjYW5ub3QgZmluZCBpbml0
KS4gSWYgSSBjaGVjayBvdXQgYQ0KPiB0cmVlIGF0ICBjYzNiMjk2ZmQ5YX4xLCBpdCBib290cyB0
aGUgbmZzIHJvb3Qgc3VjY2Vzc2Z1bGx5Lg0KPiANCj4gLi4uDQo+ID4gSSdsbCBoYXZlIGEgbG9v
ayB0byBzZWUgaWYgSSBjYW4gZmluZCBvdXQgd2hhdCBpcyB0aGUgcHJvYmxlbS4gVGhlIG9ubHkN
Cj4gPiBzaWduaWZpY2FudCBjaGFuZ2UgdGhhdCBwYXRjaCBzaG91bGQgYmUgaW50cm9kdWNpbmcg
d291bGQgYmUgdGhlIHVzZSBvZg0KPiA+IHhkcl9zdHJlYW1fcG9zKCkuDQo+IA0KPiBPa2F5LiBJ
IGFtIGhhcHB5IHRvIGhlbHAgd2l0aCBhbnkgdGVzdGluZyBpZiBuZWVkZWQgdG8gaGVscCBjaGFy
YWN0ZXJpemUNCj4gdGhlIHByb2JsZW0gYmV0dGVyLg0KDQpIaSBNYXJrIGFuZCBGZW5nZ3Vhbmcs
DQoNCkkgYmVsaWV2ZSB0aGF0IEkndmUgZm91bmQgdGhlIGJ1ZyB0aGF0IHdhcyBiZWluZyB0cmln
Z2VyZWQgaW4gdGhlIGFib3ZlDQpwYXRjaC4gSXQgd2FzIGEgbWlzdGFrZSBpbiB0aGUgZWFybGll
ciBwYXRjaCAiU1VOUlBDOiBEb24ndCBkZWNvZGUNCmJleW9uZCB0aGUgZW5kIG9mIHRoZSBSUEMg
cmVwbHkgbWVzc2FnZSIsIHdoaWNoIHdhcyBmYWlsaW5nIHRvIGNvcnJlY3RseQ0KaW5pdGlhbGlz
ZSB0aGUgeGRyLT5ud29yZHMgZmllbGQuDQoNCkkndmUgbm93IGZpeGVkIGl0IGFuZCB1cGRhdGVk
IHRoZSBsaW51eC1uZXh0IGJyYW5jaC4NCg0KQ2hlZXJzDQogIFRyb25kDQotLSANClRyb25kIE15
a2xlYnVzdA0KTGludXggTkZTIGNsaWVudCBtYWludGFpbmVyDQoNCk5ldEFwcA0KVHJvbmQuTXlr
bGVidXN0QG5ldGFwcC5jb20NCnd3dy5uZXRhcHAuY29tDQoNCg==

Re: nfs root problem in linux-next

[Mark Salter]

On Fri, 2012-06-29 at 13:55 +0000, Myklebust, Trond wrote:
> Hi Mark and Fengguang,
> 
> I believe that I've found the bug that was being triggered in the
> above
> patch. It was a mistake in the earlier patch "SUNRPC: Don't decode
> beyond the end of the RPC reply message", which was failing to
> correctly
> initialise the xdr->nwords field.
> 
> I've now fixed it and updated the linux-next branch.

Yes, I can confirm that today's linux-next is booting correctly
for me on an nfsroot.

Thanks!

Re: nfs root problem in linux-next

[Fengguang Wu]

On Fri, Jun 29, 2012 at 10:48:48AM -0400, Mark Salter wrote:
> On Fri, 2012-06-29 at 13:55 +0000, Myklebust, Trond wrote:
> > Hi Mark and Fengguang,
> > 
> > I believe that I've found the bug that was being triggered in the
> > above
> > patch. It was a mistake in the earlier patch "SUNRPC: Don't decode
> > beyond the end of the RPC reply message", which was failing to
> > correctly
> > initialise the xdr->nwords field.
> > 
> > I've now fixed it and updated the linux-next branch.
> 
> Yes, I can confirm that today's linux-next is booting correctly
> for me on an nfsroot.
> 
> Thanks!
 
I can confirm it, too. Thanks!