* Re: Re: Regression caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server")
@ 2013-09-26 4:22 Jongman Heo
0 siblings, 0 replies; 6+ messages in thread
From: Jongman Heo @ 2013-09-26 4:22 UTC (permalink / raw)
To: J. Bruce Fields, linux-nfs@vger.kernel.org,
linux-kernel@vger.kernel.org
[-- Attachment #1: Type: text/plain, Size: 6418 bytes --]
Hi,
>
>------- Original Message -------
>Sender : J. Bruce Fields<bfields@fieldses.org>
>Date : 2013-09-25 23:05 (GMT+09:00)
>Title : Re: Regression caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server")
>
>On Wed, Sep 25, 2013 at 05:19:50AM +0000, Jongman Heo wrote:
>> My embedded development box fails to NFS-boot with NFS server which uses recent kernel.
>>
>> Using git bisect, I found it is caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server").
>>
>>
>> 1. dmesg (NFS boot failure case)
>>
>> ...
>> [ 2.040893] ADDRCONF(NETDEV_UP): eth0: link is not ready
>> [ 2.046207] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
>> [ 2.053570] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>> [ 3.055023] IP-Config: Guessing netmask 255.255.0.0
>> [ 3.059979] IP-Config: Gateway not on directly connected network.
>> [ 3.066330] Looking up port of RPC 100003/2 on 165.213.88.249
>> [ 3.074001] Looking up port of RPC 100005/1 on 165.213.88.249
>> [ 3.122878] VFS: Unable to mount root fs via NFS, trying floppy.
>> [ 3.129134] VFS: Cannot open root device "nfs" or unknown-block(2,0)
>> [ 3.135478] Please append a correct "root=" boot option; here are the available partitions:
>> [ 3.143831] 1f00 3072 mtdblock0 (driver?)
>> [ 3.148798] 1f01 64 mtdblock1 (driver?)
>> [ 3.153758] 1f02 64 mtdblock2 (driver?)
>> [ 3.158719] 1f03 64 mtdblock3 (driver?)
>> [ 3.163682] 1f04 64 mtdblock4 (driver?)
>> [ 3.168644] 1f05 64 mtdblock5 (driver?)
>> [ 3.173607] 1f06 64 mtdblock6 (driver?)
>> [ 3.178568] 0800 488386584 sda driver: sd
>> [ 3.183099] 0801 506016 sda1
>> [ 3.186927] 0802 4008217 sda2
>> [ 3.190755] 0803 483869767 sda3
>> [ 3.194584] b300 1880064 mmcblk0 driver: mmcblk
>> [ 3.199802] b301 4096 mmcblk0p1
>> [ 3.204063] b302 102400 mmcblk0p2
>> [ 3.208330] b303 4096 mmcblk0p3
>> [ 3.212594] b304 1 mmcblk0p4
>> [ 3.216855] b305 2048 mmcblk0p5
>> [ 3.221116] b306 2048 mmcblk0p6
>> [ 3.225382] b307 2048 mmcblk0p7
>> [ 3.229644] b308 4096 mmcblk0p8
>> [ 3.233906] b309 12288 mmcblk0p9
>> [ 3.238176] b30a 16384 mmcblk0p10
>> [ 3.242524] b30b 142336 mmcblk0p11
>> [ 3.246869] b30c 1572864 mmcblk0p12
>> [ 3.251219] b320 12288 mmcblk0gp1 (driver?)
>> [ 3.256272] b310 12288 mmcblk0gp0 (driver?)
>> [ 3.261320] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
>> [ 3.269566] Pid: 1, comm: swapper Not tainted 2.6.35 #1
>> [ 3.274776] Call Trace:
>> [ 3.277232] [<80d0db5b>] ? printk+0x1e/0x20
>> [ 3.281492] [<80d0dad1>] panic+0x65/0xd1
>> [ 3.285495] [<80eb9ce3>] mount_block_root+0x125/0x1be
>> [ 3.290631] [<809d1f6d>] ? sys_mknod+0x2d/0x30
>> [ 3.295156] [<80eb9f6d>] mount_root+0xd0/0xf2
>> [ 3.299591] [<80eba0d9>] prepare_namespace+0x14a/0x184
>> [ 3.304803] [<809c44f6>] ? sys_access+0x26/0x30
>> [ 3.309411] [<80eb9a4e>] kernel_init+0x25e/0x26e
>> [ 3.314105] [<80eb97f0>] ? kernel_init+0x0/0x26e
>> [ 3.318800] [<80903242>] kernel_thread_helper+0x6/0x10
>>
>>
>> 2. Client (my embedded box) configuration
>> It's kernel 2.6.35 based, and has following NFS kernel configs.
>>
>> # grep NFS .config
>> CONFIG_NFS_FS=y
>> CONFIG_NFS_V3=y
>> CONFIG_NFS_V3_ACL=y
>> CONFIG_NFS_V4=y
>> # CONFIG_NFS_V4_1 is not set
>> CONFIG_ROOT_NFS=y
>> # CONFIG_NFSD is not set
>> CONFIG_NFS_ACL_SUPPORT=y
>> CONFIG_NFS_COMMON=y
>>
>>
>> 3. Server (NFSD) configuration
>> Fedora 19 + latest linus git kernel 3.12.0-rc2+ (commit 22356f44, mm: Place preemption point in do_mlockall() loop)
>>
>>
>> 4. workaround
>>
>> Reverting the commit 4bdc33ed resolves my issue, NFS boot is working then.
>> I've done git bisect, but lost the resulting bisect log due to sudden power loss :(.
>
>So when you say you revert that commit, you mean you revert it on your
>*server*, right? You're not changing the client at all throughout these
>tests?
Right. I reverted the commit on my server, while client is same throughout the tests.
>
>A network trace might be interesting: so, on the server, run
>
>tcpdump -s0 -wtmp.pcap -ieth0
>
>(replace eth0 by the right network interface), then try booting the
>client and after the client fails, kill tcpdump and send us a copy of
>tmp.pcap.
>
>(And also you might want to fire up "wireshark tmp.pcap" and take a look
>yourself--you'll probably see something like a version mismatch error in
>the network traffic.)
>
>--b.
I've attached two tcpdump files.
In the dump, 165.213.88.238 is IP address for NFS client (embedded box with 2.6.35 kernel), and 192.168.64.128 is for NFS server (running latest git kernel with and without the commit revert)
* tmp_good_filtered.pcap
- latest linus git tree + commit 4bdc33ed reverted
- NFS boot is working
* tmp_bad_filtered.pcap
- latest linus git tree
- NFS boot doesn't work
In error case, I can see following message from wireshark packet window ;
Accept State: remote can't support version # (2)
Program Version (Minimum): 3
Program Version (Maximum): 4
And I forgot to attach my config of NFS server. Here it is.
# grep NFS .config
CONFIG_NFS_FS=y
CONFIG_NFS_V2=y
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
# CONFIG_NFS_SWAP is not set
CONFIG_NFS_V4_1=y
# CONFIG_NFS_V4_2 is not set
CONFIG_PNFS_FILE_LAYOUT=m
CONFIG_PNFS_BLOCK=m
CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org"
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFSD=y
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
# CONFIG_NFSD_V4_SECURITY_LABEL is not set
# CONFIG_NFSD_FAULT_INJECTION is not set
CONFIG_NFS_ACL_SUPPORT=y
CONFIG_NFS_COMMON=y
# rpm -qa|grep nfs
nfs-utils-1.2.8-4.0.fc19.i686
libnfsidmap-0.25-5.fc19.i686
# cat /etc/exports
/home/NFSROOT_mmc/ 165.213.88.238(rw,no_root_squash,sync)
Thanks,
Jongman Heo.
[-- Attachment #2: tmp_good_filtered.pcap --]
[-- Type: application/octet-stream, Size: 7448 bytes --]
[-- Attachment #3: tmp_bad_filtered.pcap --]
[-- Type: application/octet-stream, Size: 1140 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Re: Regression caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server")
@ 2013-09-26 4:22 Jongman Heo
0 siblings, 0 replies; 6+ messages in thread
From: Jongman Heo @ 2013-09-26 4:22 UTC (permalink / raw)
To: J. Bruce Fields, linux-nfs@vger.kernel.org,
linux-kernel@vger.kernel.org
[-- Attachment #1: Type: text/plain, Size: 6418 bytes --]
Hi,
>
>------- Original Message -------
>Sender : J. Bruce Fields<bfields@fieldses.org>
>Date : 2013-09-25 23:05 (GMT+09:00)
>Title : Re: Regression caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server")
>
>On Wed, Sep 25, 2013 at 05:19:50AM +0000, Jongman Heo wrote:
>> My embedded development box fails to NFS-boot with NFS server which uses recent kernel.
>>
>> Using git bisect, I found it is caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server").
>>
>>
>> 1. dmesg (NFS boot failure case)
>>
>> ...
>> [ 2.040893] ADDRCONF(NETDEV_UP): eth0: link is not ready
>> [ 2.046207] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
>> [ 2.053570] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>> [ 3.055023] IP-Config: Guessing netmask 255.255.0.0
>> [ 3.059979] IP-Config: Gateway not on directly connected network.
>> [ 3.066330] Looking up port of RPC 100003/2 on 165.213.88.249
>> [ 3.074001] Looking up port of RPC 100005/1 on 165.213.88.249
>> [ 3.122878] VFS: Unable to mount root fs via NFS, trying floppy.
>> [ 3.129134] VFS: Cannot open root device "nfs" or unknown-block(2,0)
>> [ 3.135478] Please append a correct "root=" boot option; here are the available partitions:
>> [ 3.143831] 1f00 3072 mtdblock0 (driver?)
>> [ 3.148798] 1f01 64 mtdblock1 (driver?)
>> [ 3.153758] 1f02 64 mtdblock2 (driver?)
>> [ 3.158719] 1f03 64 mtdblock3 (driver?)
>> [ 3.163682] 1f04 64 mtdblock4 (driver?)
>> [ 3.168644] 1f05 64 mtdblock5 (driver?)
>> [ 3.173607] 1f06 64 mtdblock6 (driver?)
>> [ 3.178568] 0800 488386584 sda driver: sd
>> [ 3.183099] 0801 506016 sda1
>> [ 3.186927] 0802 4008217 sda2
>> [ 3.190755] 0803 483869767 sda3
>> [ 3.194584] b300 1880064 mmcblk0 driver: mmcblk
>> [ 3.199802] b301 4096 mmcblk0p1
>> [ 3.204063] b302 102400 mmcblk0p2
>> [ 3.208330] b303 4096 mmcblk0p3
>> [ 3.212594] b304 1 mmcblk0p4
>> [ 3.216855] b305 2048 mmcblk0p5
>> [ 3.221116] b306 2048 mmcblk0p6
>> [ 3.225382] b307 2048 mmcblk0p7
>> [ 3.229644] b308 4096 mmcblk0p8
>> [ 3.233906] b309 12288 mmcblk0p9
>> [ 3.238176] b30a 16384 mmcblk0p10
>> [ 3.242524] b30b 142336 mmcblk0p11
>> [ 3.246869] b30c 1572864 mmcblk0p12
>> [ 3.251219] b320 12288 mmcblk0gp1 (driver?)
>> [ 3.256272] b310 12288 mmcblk0gp0 (driver?)
>> [ 3.261320] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
>> [ 3.269566] Pid: 1, comm: swapper Not tainted 2.6.35 #1
>> [ 3.274776] Call Trace:
>> [ 3.277232] [<80d0db5b>] ? printk+0x1e/0x20
>> [ 3.281492] [<80d0dad1>] panic+0x65/0xd1
>> [ 3.285495] [<80eb9ce3>] mount_block_root+0x125/0x1be
>> [ 3.290631] [<809d1f6d>] ? sys_mknod+0x2d/0x30
>> [ 3.295156] [<80eb9f6d>] mount_root+0xd0/0xf2
>> [ 3.299591] [<80eba0d9>] prepare_namespace+0x14a/0x184
>> [ 3.304803] [<809c44f6>] ? sys_access+0x26/0x30
>> [ 3.309411] [<80eb9a4e>] kernel_init+0x25e/0x26e
>> [ 3.314105] [<80eb97f0>] ? kernel_init+0x0/0x26e
>> [ 3.318800] [<80903242>] kernel_thread_helper+0x6/0x10
>>
>>
>> 2. Client (my embedded box) configuration
>> It's kernel 2.6.35 based, and has following NFS kernel configs.
>>
>> # grep NFS .config
>> CONFIG_NFS_FS=y
>> CONFIG_NFS_V3=y
>> CONFIG_NFS_V3_ACL=y
>> CONFIG_NFS_V4=y
>> # CONFIG_NFS_V4_1 is not set
>> CONFIG_ROOT_NFS=y
>> # CONFIG_NFSD is not set
>> CONFIG_NFS_ACL_SUPPORT=y
>> CONFIG_NFS_COMMON=y
>>
>>
>> 3. Server (NFSD) configuration
>> Fedora 19 + latest linus git kernel 3.12.0-rc2+ (commit 22356f44, mm: Place preemption point in do_mlockall() loop)
>>
>>
>> 4. workaround
>>
>> Reverting the commit 4bdc33ed resolves my issue, NFS boot is working then.
>> I've done git bisect, but lost the resulting bisect log due to sudden power loss :(.
>
>So when you say you revert that commit, you mean you revert it on your
>*server*, right? You're not changing the client at all throughout these
>tests?
Right. I reverted the commit on my server, while client is same throughout the tests.
>
>A network trace might be interesting: so, on the server, run
>
>tcpdump -s0 -wtmp.pcap -ieth0
>
>(replace eth0 by the right network interface), then try booting the
>client and after the client fails, kill tcpdump and send us a copy of
>tmp.pcap.
>
>(And also you might want to fire up "wireshark tmp.pcap" and take a look
>yourself--you'll probably see something like a version mismatch error in
>the network traffic.)
>
>--b.
I've attached two tcpdump files.
In the dump, 165.213.88.238 is IP address for NFS client (embedded box with 2.6.35 kernel), and 192.168.64.128 is for NFS server (running latest git kernel with and without the commit revert)
* tmp_good_filtered.pcap
- latest linus git tree + commit 4bdc33ed reverted
- NFS boot is working
* tmp_bad_filtered.pcap
- latest linus git tree
- NFS boot doesn't work
In error case, I can see following message from wireshark packet window ;
Accept State: remote can't support version # (2)
Program Version (Minimum): 3
Program Version (Maximum): 4
And I forgot to attach my config of NFS server. Here it is.
# grep NFS .config
CONFIG_NFS_FS=y
CONFIG_NFS_V2=y
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
# CONFIG_NFS_SWAP is not set
CONFIG_NFS_V4_1=y
# CONFIG_NFS_V4_2 is not set
CONFIG_PNFS_FILE_LAYOUT=m
CONFIG_PNFS_BLOCK=m
CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org"
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFSD=y
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
# CONFIG_NFSD_V4_SECURITY_LABEL is not set
# CONFIG_NFSD_FAULT_INJECTION is not set
CONFIG_NFS_ACL_SUPPORT=y
CONFIG_NFS_COMMON=y
# rpm -qa|grep nfs
nfs-utils-1.2.8-4.0.fc19.i686
libnfsidmap-0.25-5.fc19.i686
# cat /etc/exports
/home/NFSROOT_mmc/ 165.213.88.238(rw,no_root_squash,sync)
Thanks,
Jongman Heo.
[-- Attachment #2: tmp_good_filtered.pcap --]
[-- Type: application/octet-stream, Size: 7448 bytes --]
[-- Attachment #3: tmp_bad_filtered.pcap --]
[-- Type: application/octet-stream, Size: 1140 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Re: Regression caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server")
@ 2013-09-26 4:27 Jongman Heo
0 siblings, 0 replies; 6+ messages in thread
From: Jongman Heo @ 2013-09-26 4:27 UTC (permalink / raw)
To: Anna Schumaker, linux-nfs@vger.kernel.org,
linux-kernel@vger.kernel.org
Pg0KPi0tLS0tLS0gT3JpZ2luYWwgTWVzc2FnZSAtLS0tLS0tDQo+DQo+U2VuZGVyIDogQW5uYSBT
Y2h1bWFrZXI8c2NodW1ha2VyLmFubmFAZ21haWwuY29tPg0KPg0KPkRhdGUgOiAyMDEzLTA5LTI1
IDIyOjUyIChHTVQrMDk6MDApDQo+DQo+VGl0bGUgOiBSZTogUmVncmVzc2lvbiBjYXVzZWQgYnkg
Y29tbWl0IDRiZGMzM2VkICgiTkZTRHY0LjI6IEFkZCBORlMgdjQuMiBzdXBwb3J0IHRvIHRoZSBO
RlMgc2VydmVyIikNCj4NCj4NCj4NCj5IaSBKb25nbWFuLA0KPg0KPklzIHRoZSBwYW5pYyBvbiB5
b3VyIGNsaWVudCBvciBzZXJ2ZXI/ICBJIGRvbid0IHNlZSBob3cgdGhlIHBhdGNoIHlvdXINCj5i
aXNlY3QgbGVkIHlvdSB0byBjb3VsZCBjYXVzZSB0aGUgcHJvYmxlbSwgc2luY2UgYWxsIGl0IGRv
ZXMgaXMgZXhwYW5kDQo+dGhlIG1pbm9yIHZlcnNpb24gYXJyYXkgb24gdGhlIHNlcnZlci4gIFlv
dXIgY2xpZW50IGRvZXNuJ3QgaGF2ZSBORlNEDQo+ZW5hYmxlZCwgc28gdGhpcyBjb2RlIHNob3Vs
ZG4ndCBldmVuIGJlIGFmZmVjdGluZyBpdC4NCj4NCj5BIGZldyBxdWVzdGlvbnM6ICB3aGF0IGlz
IHlvdXIgL2V0Yy9leHBvcnRzIG9uIHRoZSBzZXJ2ZXI/ICBXaGF0DQo+dmVyc2lvbiBvZiBORlMg
YXJlIHlvdSB1c2luZyBmb3IgbmZzcm9vdD8NCj4NCj5UaGFua3MhDQo+QW5uYQ0KPg0KPk9uIFdl
ZCwgU2VwIDI1LCAyMDEzIGF0IDE6MTkgQU0sIEpvbmdtYW4gSGVvIHdyb3RlOg0KPj4NCj4+IEhp
IGFsbCwNCj4+DQo+PiBNeSBlbWJlZGRlZCBkZXZlbG9wbWVudCBib3ggZmFpbHMgdG8gTkZTLWJv
b3Qgd2l0aCBORlMgc2VydmVyIHdoaWNoIHVzZXMgcmVjZW50IGtlcm5lbC4NCj4+DQo+PiBVc2lu
ZyBnaXQgYmlzZWN0LCBJIGZvdW5kIGl0IGlzIGNhdXNlZCBieSBjb21taXQgNGJkYzMzZWQgKCJO
RlNEdjQuMjogQWRkIE5GUyB2NC4yIHN1cHBvcnQgdG8gdGhlIE5GUyBzZXJ2ZXIiKS4NCj4+DQo+
Pg0KPj4gMS4gZG1lc2cgKE5GUyBib290IGZhaWx1cmUgY2FzZSkNCj4+DQo+PiAuLi4NCj4+IFsg
ICAgMi4wNDA4OTNdIEFERFJDT05GKE5FVERFVl9VUCk6IGV0aDA6IGxpbmsgaXMgbm90IHJlYWR5
DQo+PiBbICAgIDIuMDQ2MjA3XSBlMTAwMDogZXRoMCBOSUMgTGluayBpcyBVcCAxMDAwIE1icHMg
RnVsbCBEdXBsZXgsIEZsb3cgQ29udHJvbDogUlgNCj4+IFsgICAgMi4wNTM1NzBdIEFERFJDT05G
KE5FVERFVl9DSEFOR0UpOiBldGgwOiBsaW5rIGJlY29tZXMgcmVhZHkNCj4+IFsgICAgMy4wNTUw
MjNdIElQLUNvbmZpZzogR3Vlc3NpbmcgbmV0bWFzayAyNTUuMjU1LjAuMA0KPj4gWyAgICAzLjA1
OTk3OV0gSVAtQ29uZmlnOiBHYXRld2F5IG5vdCBvbiBkaXJlY3RseSBjb25uZWN0ZWQgbmV0d29y
ay4NCj4+IFsgICAgMy4wNjYzMzBdIExvb2tpbmcgdXAgcG9ydCBvZiBSUEMgMTAwMDAzLzIgb24g
MTY1LjIxMy44OC4yNDkNCj4+IFsgICAgMy4wNzQwMDFdIExvb2tpbmcgdXAgcG9ydCBvZiBSUEMg
MTAwMDA1LzEgb24gMTY1LjIxMy44OC4yNDkNCj4+IFsgICAgMy4xMjI4NzhdIFZGUzogVW5hYmxl
IHRvIG1vdW50IHJvb3QgZnMgdmlhIE5GUywgdHJ5aW5nIGZsb3BweS4NCj4+IFsgICAgMy4xMjkx
MzRdIFZGUzogQ2Fubm90IG9wZW4gcm9vdCBkZXZpY2UgIm5mcyIgb3IgdW5rbm93bi1ibG9jaygy
LDApDQo+PiBbICAgIDMuMTM1NDc4XSBQbGVhc2UgYXBwZW5kIGEgY29ycmVjdCAicm9vdD0iIGJv
b3Qgb3B0aW9uOyBoZXJlIGFyZSB0aGUgYXZhaWxhYmxlIHBhcnRpdGlvbnM6DQo+PiBbICAgIDMu
MTQzODMxXSAxZjAwICAgICAgICAgICAgMzA3MiBtdGRibG9jazAgKGRyaXZlcj8pDQo+PiBbICAg
IDMuMTQ4Nzk4XSAxZjAxICAgICAgICAgICAgICA2NCBtdGRibG9jazEgKGRyaXZlcj8pDQo+PiBb
ICAgIDMuMTUzNzU4XSAxZjAyICAgICAgICAgICAgICA2NCBtdGRibG9jazIgKGRyaXZlcj8pDQo+
PiBbICAgIDMuMTU4NzE5XSAxZjAzICAgICAgICAgICAgICA2NCBtdGRibG9jazMgKGRyaXZlcj8p
DQo+PiBbICAgIDMuMTYzNjgyXSAxZjA0ICAgICAgICAgICAgICA2NCBtdGRibG9jazQgKGRyaXZl
cj8pDQo+PiBbICAgIDMuMTY4NjQ0XSAxZjA1ICAgICAgICAgICAgICA2NCBtdGRibG9jazUgKGRy
aXZlcj8pDQo+PiBbICAgIDMuMTczNjA3XSAxZjA2ICAgICAgICAgICAgICA2NCBtdGRibG9jazYg
KGRyaXZlcj8pDQo+PiBbICAgIDMuMTc4NTY4XSAwODAwICAgICAgIDQ4ODM4NjU4NCBzZGEgZHJp
dmVyOiBzZA0KPj4gWyAgICAzLjE4MzA5OV0gICAwODAxICAgICAgICAgIDUwNjAxNiBzZGExDQo+
PiBbICAgIDMuMTg2OTI3XSAgIDA4MDIgICAgICAgICA0MDA4MjE3IHNkYTINCj4+IFsgICAgMy4x
OTA3NTVdICAgMDgwMyAgICAgICA0ODM4Njk3Njcgc2RhMw0KPj4gWyAgICAzLjE5NDU4NF0gYjMw
MCAgICAgICAgIDE4ODAwNjQgbW1jYmxrMCBkcml2ZXI6IG1tY2Jsaw0KPj4gWyAgICAzLjE5OTgw
Ml0gICBiMzAxICAgICAgICAgICAgNDA5NiBtbWNibGswcDENCj4+IFsgICAgMy4yMDQwNjNdICAg
YjMwMiAgICAgICAgICAxMDI0MDAgbW1jYmxrMHAyDQo+PiBbICAgIDMuMjA4MzMwXSAgIGIzMDMg
ICAgICAgICAgICA0MDk2IG1tY2JsazBwMw0KPj4gWyAgICAzLjIxMjU5NF0gICBiMzA0ICAgICAg
ICAgICAgICAgMSBtbWNibGswcDQNCj4+IFsgICAgMy4yMTY4NTVdICAgYjMwNSAgICAgICAgICAg
IDIwNDggbW1jYmxrMHA1DQo+PiBbICAgIDMuMjIxMTE2XSAgIGIzMDYgICAgICAgICAgICAyMDQ4
IG1tY2JsazBwNg0KPj4gWyAgICAzLjIyNTM4Ml0gICBiMzA3ICAgICAgICAgICAgMjA0OCBtbWNi
bGswcDcNCj4+IFsgICAgMy4yMjk2NDRdICAgYjMwOCAgICAgICAgICAgIDQwOTYgbW1jYmxrMHA4
DQo+PiBbICAgIDMuMjMzOTA2XSAgIGIzMDkgICAgICAgICAgIDEyMjg4IG1tY2JsazBwOQ0KPj4g
WyAgICAzLjIzODE3Nl0gICBiMzBhICAgICAgICAgICAxNjM4NCBtbWNibGswcDEwDQo+PiBbICAg
IDMuMjQyNTI0XSAgIGIzMGIgICAgICAgICAgMTQyMzM2IG1tY2JsazBwMTENCj4+IFsgICAgMy4y
NDY4NjldICAgYjMwYyAgICAgICAgIDE1NzI4NjQgbW1jYmxrMHAxMg0KPj4gWyAgICAzLjI1MTIx
OV0gYjMyMCAgICAgICAgICAgMTIyODggbW1jYmxrMGdwMSAoZHJpdmVyPykNCj4+IFsgICAgMy4y
NTYyNzJdIGIzMTAgICAgICAgICAgIDEyMjg4IG1tY2JsazBncDAgKGRyaXZlcj8pDQo+PiBbICAg
IDMuMjYxMzIwXSBLZXJuZWwgcGFuaWMgLSBub3Qgc3luY2luZzogVkZTOiBVbmFibGUgdG8gbW91
bnQgcm9vdCBmcyBvbiB1bmtub3duLWJsb2NrKDIsMCkNCj4+IFsgICAgMy4yNjk1NjZdIFBpZDog
MSwgY29tbTogc3dhcHBlciBOb3QgdGFpbnRlZCAyLjYuMzUgIzENCj4+IFsgICAgMy4yNzQ3NzZd
IENhbGwgVHJhY2U6DQo+PiBbICAgIDMuMjc3MjMyXSAgWzw4MGQwZGI1Yj5dID8gcHJpbnRrKzB4
MWUvMHgyMA0KPj4gWyAgICAzLjI4MTQ5Ml0gIFs8ODBkMGRhZDE+XSBwYW5pYysweDY1LzB4ZDEN
Cj4+IFsgICAgMy4yODU0OTVdICBbPDgwZWI5Y2UzPl0gbW91bnRfYmxvY2tfcm9vdCsweDEyNS8w
eDFiZQ0KPj4gWyAgICAzLjI5MDYzMV0gIFs8ODA5ZDFmNmQ+XSA/IHN5c19ta25vZCsweDJkLzB4
MzANCj4+IFsgICAgMy4yOTUxNTZdICBbPDgwZWI5ZjZkPl0gbW91bnRfcm9vdCsweGQwLzB4ZjIN
Cj4+IFsgICAgMy4yOTk1OTFdICBbPDgwZWJhMGQ5Pl0gcHJlcGFyZV9uYW1lc3BhY2UrMHgxNGEv
MHgxODQNCj4+IFsgICAgMy4zMDQ4MDNdICBbPDgwOWM0NGY2Pl0gPyBzeXNfYWNjZXNzKzB4MjYv
MHgzMA0KPj4gWyAgICAzLjMwOTQxMV0gIFs8ODBlYjlhNGU+XSBrZXJuZWxfaW5pdCsweDI1ZS8w
eDI2ZQ0KPj4gWyAgICAzLjMxNDEwNV0gIFs8ODBlYjk3ZjA+XSA/IGtlcm5lbF9pbml0KzB4MC8w
eDI2ZQ0KPj4gWyAgICAzLjMxODgwMF0gIFs8ODA5MDMyNDI+XSBrZXJuZWxfdGhyZWFkX2hlbHBl
cisweDYvMHgxMA0KPj4NCj4+DQo+PiAyLiBDbGllbnQgKG15IGVtYmVkZGVkIGJveCkgY29uZmln
dXJhdGlvbg0KPj4gICBJdCdzIGtlcm5lbCAyLjYuMzUgYmFzZWQsIGFuZCBoYXMgZm9sbG93aW5n
IE5GUyBrZXJuZWwgY29uZmlncy4NCj4+DQo+PiAjIGdyZXAgTkZTIC5jb25maWcNCj4+IENPTkZJ
R19ORlNfRlM9eQ0KPj4gQ09ORklHX05GU19WMz15DQo+PiBDT05GSUdfTkZTX1YzX0FDTD15DQo+
PiBDT05GSUdfTkZTX1Y0PXkNCj4+ICMgQ09ORklHX05GU19WNF8xIGlzIG5vdCBzZXQNCj4+IENP
TkZJR19ST09UX05GUz15DQo+PiAjIENPTkZJR19ORlNEIGlzIG5vdCBzZXQNCj4+IENPTkZJR19O
RlNfQUNMX1NVUFBPUlQ9eQ0KPj4gQ09ORklHX05GU19DT01NT049eQ0KPj4NCj4+DQo+PiAzLiBT
ZXJ2ZXIgKE5GU0QpIGNvbmZpZ3VyYXRpb24NCj4+ICAgIEZlZG9yYSAxOSArIGxhdGVzdCBsaW51
cyBnaXQga2VybmVsIDMuMTIuMC1yYzIrIChjb21taXQgMjIzNTZmNDQsIG1tOiBQbGFjZSBwcmVl
bXB0aW9uIHBvaW50IGluIGRvX21sb2NrYWxsKCkgbG9vcCkNCj4+DQo+Pg0KPj4gNC4gd29ya2Fy
b3VuZA0KPj4NCj4+IFJldmVydGluZyB0aGUgY29tbWl0IDRiZGMzM2VkIHJlc29sdmVzIG15IGlz
c3VlLCBORlMgYm9vdCBpcyB3b3JraW5nIHRoZW4uDQo+PiBJJ3ZlIGRvbmUgZ2l0IGJpc2VjdCwg
YnV0IGxvc3QgdGhlIHJlc3VsdGluZyBiaXNlY3QgbG9nIGR1ZSB0byBzdWRkZW4gcG93ZXIgbG9z
cyA6KC4NCj4+DQo+PiBCZXN0IHJlZ2FyZHMsDQo+PiBKb25nbWFuIEhlbw0KPg0KPg0KPg0KDQpI
aSwNCg0KUGxlYXNlIHNlZSBteSBlLW1haWwgcmVwbHkgdG8gSi4gQnJ1Y2UgRmllbGRzIGZvciB0
aGUgZGV0YWlsLg0KDQpUaGFua3MsDQpKb25nbWFuIEhlby4=
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Re: Regression caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server")
[not found] <42.19.15034.896B3425@epcpsbgx2.samsung.com>
@ 2013-09-26 17:47 ` J. Bruce Fields
2013-09-26 18:35 ` Wendy Cheng
0 siblings, 1 reply; 6+ messages in thread
From: J. Bruce Fields @ 2013-09-26 17:47 UTC (permalink / raw)
To: Jongman Heo; +Cc: linux-nfs@vger.kernel.org, linux-kernel@vger.kernel.org
On Thu, Sep 26, 2013 at 04:22:48AM +0000, Jongman Heo wrote:
>
> Hi,
>
> >
> >------- Original Message -------
> >Sender : J. Bruce Fields<bfields@fieldses.org>
> >Date : 2013-09-25 23:05 (GMT+09:00)
> >Title : Re: Regression caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server")
> >
> >On Wed, Sep 25, 2013 at 05:19:50AM +0000, Jongman Heo wrote:
> >> My embedded development box fails to NFS-boot with NFS server which uses recent kernel.
> >>
> >> Using git bisect, I found it is caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server").
> >>
> >>
> >> 1. dmesg (NFS boot failure case)
> >>
> >> ...
> >> [ 2.040893] ADDRCONF(NETDEV_UP): eth0: link is not ready
> >> [ 2.046207] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
> >> [ 2.053570] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> >> [ 3.055023] IP-Config: Guessing netmask 255.255.0.0
> >> [ 3.059979] IP-Config: Gateway not on directly connected network.
> >> [ 3.066330] Looking up port of RPC 100003/2 on 165.213.88.249
> >> [ 3.074001] Looking up port of RPC 100005/1 on 165.213.88.249
> >> [ 3.122878] VFS: Unable to mount root fs via NFS, trying floppy.
> >> [ 3.129134] VFS: Cannot open root device "nfs" or unknown-block(2,0)
> >> [ 3.135478] Please append a correct "root=" boot option; here are the available partitions:
> >> [ 3.143831] 1f00 3072 mtdblock0 (driver?)
> >> [ 3.148798] 1f01 64 mtdblock1 (driver?)
> >> [ 3.153758] 1f02 64 mtdblock2 (driver?)
> >> [ 3.158719] 1f03 64 mtdblock3 (driver?)
> >> [ 3.163682] 1f04 64 mtdblock4 (driver?)
> >> [ 3.168644] 1f05 64 mtdblock5 (driver?)
> >> [ 3.173607] 1f06 64 mtdblock6 (driver?)
> >> [ 3.178568] 0800 488386584 sda driver: sd
> >> [ 3.183099] 0801 506016 sda1
> >> [ 3.186927] 0802 4008217 sda2
> >> [ 3.190755] 0803 483869767 sda3
> >> [ 3.194584] b300 1880064 mmcblk0 driver: mmcblk
> >> [ 3.199802] b301 4096 mmcblk0p1
> >> [ 3.204063] b302 102400 mmcblk0p2
> >> [ 3.208330] b303 4096 mmcblk0p3
> >> [ 3.212594] b304 1 mmcblk0p4
> >> [ 3.216855] b305 2048 mmcblk0p5
> >> [ 3.221116] b306 2048 mmcblk0p6
> >> [ 3.225382] b307 2048 mmcblk0p7
> >> [ 3.229644] b308 4096 mmcblk0p8
> >> [ 3.233906] b309 12288 mmcblk0p9
> >> [ 3.238176] b30a 16384 mmcblk0p10
> >> [ 3.242524] b30b 142336 mmcblk0p11
> >> [ 3.246869] b30c 1572864 mmcblk0p12
> >> [ 3.251219] b320 12288 mmcblk0gp1 (driver?)
> >> [ 3.256272] b310 12288 mmcblk0gp0 (driver?)
> >> [ 3.261320] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
> >> [ 3.269566] Pid: 1, comm: swapper Not tainted 2.6.35 #1
> >> [ 3.274776] Call Trace:
> >> [ 3.277232] [<80d0db5b>] ? printk+0x1e/0x20
> >> [ 3.281492] [<80d0dad1>] panic+0x65/0xd1
> >> [ 3.285495] [<80eb9ce3>] mount_block_root+0x125/0x1be
> >> [ 3.290631] [<809d1f6d>] ? sys_mknod+0x2d/0x30
> >> [ 3.295156] [<80eb9f6d>] mount_root+0xd0/0xf2
> >> [ 3.299591] [<80eba0d9>] prepare_namespace+0x14a/0x184
> >> [ 3.304803] [<809c44f6>] ? sys_access+0x26/0x30
> >> [ 3.309411] [<80eb9a4e>] kernel_init+0x25e/0x26e
> >> [ 3.314105] [<80eb97f0>] ? kernel_init+0x0/0x26e
> >> [ 3.318800] [<80903242>] kernel_thread_helper+0x6/0x10
> >>
> >>
> >> 2. Client (my embedded box) configuration
> >> It's kernel 2.6.35 based, and has following NFS kernel configs.
> >>
> >> # grep NFS .config
> >> CONFIG_NFS_FS=y
> >> CONFIG_NFS_V3=y
> >> CONFIG_NFS_V3_ACL=y
> >> CONFIG_NFS_V4=y
> >> # CONFIG_NFS_V4_1 is not set
> >> CONFIG_ROOT_NFS=y
> >> # CONFIG_NFSD is not set
> >> CONFIG_NFS_ACL_SUPPORT=y
> >> CONFIG_NFS_COMMON=y
> >>
> >>
> >> 3. Server (NFSD) configuration
> >> Fedora 19 + latest linus git kernel 3.12.0-rc2+ (commit 22356f44, mm: Place preemption point in do_mlockall() loop)
> >>
> >>
> >> 4. workaround
> >>
> >> Reverting the commit 4bdc33ed resolves my issue, NFS boot is working then.
> >> I've done git bisect, but lost the resulting bisect log due to sudden power loss :(.
> >
> >So when you say you revert that commit, you mean you revert it on your
> >*server*, right? You're not changing the client at all throughout these
> >tests?
>
> Right. I reverted the commit on my server, while client is same throughout the tests.
>
> >
> >A network trace might be interesting: so, on the server, run
> >
> >tcpdump -s0 -wtmp.pcap -ieth0
> >
> >(replace eth0 by the right network interface), then try booting the
> >client and after the client fails, kill tcpdump and send us a copy of
> >tmp.pcap.
> >
> >(And also you might want to fire up "wireshark tmp.pcap" and take a look
> >yourself--you'll probably see something like a version mismatch error in
> >the network traffic.)
> >
> >--b.
>
> I've attached two tcpdump files.
> In the dump, 165.213.88.238 is IP address for NFS client (embedded box with 2.6.35 kernel), and 192.168.64.128 is for NFS server (running latest git kernel with and without the commit revert)
>
> * tmp_good_filtered.pcap
> - latest linus git tree + commit 4bdc33ed reverted
> - NFS boot is working
>
> * tmp_bad_filtered.pcap
> - latest linus git tree
> - NFS boot doesn't work
>
> In error case, I can see following message from wireshark packet window ;
>
> Accept State: remote can't support version # (2)
> Program Version (Minimum): 3
> Program Version (Maximum): 4
This is pretty weird--it's not at all obvious how that patch would
affect this.
You're absolutely positive that the *only* thing you're changing on the
server between the "good" and "bad" cases is that one kernel patch?
You're not changing anything in userspace?
What does "cat /proc/fs/nfsd/versions" report in the good and bad cases?
(BTW, out of curiosity: what kind of client is this that only supports
NFSv2 and NFSv3? Even for an embedded system that's a bit surprising.)
--b.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Re: Regression caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server")
2013-09-26 17:47 ` Re: Regression caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server") J. Bruce Fields
@ 2013-09-26 18:35 ` Wendy Cheng
2013-10-01 14:47 ` J. Bruce Fields
0 siblings, 1 reply; 6+ messages in thread
From: Wendy Cheng @ 2013-09-26 18:35 UTC (permalink / raw)
To: J. Bruce Fields; +Cc: Jongman Heo, linux-nfs@vger.kernel.org
On Thu, Sep 26, 2013 at 10:47 AM, J. Bruce Fields <bfields@fieldses.org> wrote:
> (BTW, out of curiosity: what kind of client is this that only supports
> NFSv2 and NFSv3? Even for an embedded system that's a bit surprising.)
>
There are few floating around - at least I'm working on one at this moment.
Look to me the upgrade issue is with user space packages (e.g. where
"mount" lives). Embedded systems can take their user space packages
from certain "distribution"s that are not necessarily sync well with
individual linux upstream tool distribution, even the kernel itself is
reasonably updated.
And don't shoot a messenger :) .. I'm just describing the practices I
have been observed.
-- Wendy
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Re: Regression caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server")
2013-09-26 18:35 ` Wendy Cheng
@ 2013-10-01 14:47 ` J. Bruce Fields
0 siblings, 0 replies; 6+ messages in thread
From: J. Bruce Fields @ 2013-10-01 14:47 UTC (permalink / raw)
To: Wendy Cheng; +Cc: Jongman Heo, linux-nfs@vger.kernel.org
On Thu, Sep 26, 2013 at 11:35:46AM -0700, Wendy Cheng wrote:
> On Thu, Sep 26, 2013 at 10:47 AM, J. Bruce Fields <bfields@fieldses.org> wrote:
>
> > (BTW, out of curiosity: what kind of client is this that only supports
> > NFSv2 and NFSv3? Even for an embedded system that's a bit surprising.)
> >
>
> There are few floating around - at least I'm working on one at this moment.
>
> Look to me the upgrade issue is with user space packages (e.g. where
> "mount" lives). Embedded systems can take their user space packages
> from certain "distribution"s that are not necessarily sync well with
> individual linux upstream tool distribution, even the kernel itself is
> reasonably updated.
>
> And don't shoot a messenger :) .. I'm just describing the practices I
> have been observed.
OK. I'm not sure what standard to hold this sort of change to. I'm
inclined to use the same "it's OK if and only if nobody notices"
standard as for normal kernel interfaces. Which means we've already got
enough evidence to put off dropping NFSv2 support for a few more years,
at least on the server side....
--b.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-10-01 14:47 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <42.19.15034.896B3425@epcpsbgx2.samsung.com>
2013-09-26 17:47 ` Re: Regression caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server") J. Bruce Fields
2013-09-26 18:35 ` Wendy Cheng
2013-10-01 14:47 ` J. Bruce Fields
2013-09-26 4:27 Jongman Heo
-- strict thread matches above, loose matches on Subject: below --
2013-09-26 4:22 Jongman Heo
2013-09-26 4:22 Jongman Heo
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).