attempt to access beyond end of device

attempt to access beyond end of device

[Piotr Szymaniak]

[-- Attachment #1: Type: text/plain, Size: 5007 bytes --]

Hello,

I'm running nilfs2 on Raspberry Pi and some time ago it got some
problem.

Connected the SD card to my PC today and mounted the fs, the cleaner
started and few minutes later it remounted the fs readonly. Here's the
dmesg part:

[ 1109.889697] mmcblk0: mmc0:b368 SMI   7.51 GiB 
[ 1109.891261]  mmcblk0: p1 p2 p3
[ 1129.467450] NILFS warning: mounting unchecked fs
[ 1129.956859] NILFS: recovery complete.
[ 1129.960064] segctord starting. Construction interval = 300 seconds, CP frequency < 30 seconds
[ 1129.979344] NILFS warning: mounting fs with errors
[ 1283.005874] attempt to access beyond end of device
[ 1283.005882] mmcblk0p3: rw=8192, want=145241775180196072, limit=15120384
[ 1283.005891] attempt to access beyond end of device
[ 1283.005896] mmcblk0p3: rw=8192, want=36678440, limit=15120384
[ 1283.512451] init_special_inode: bogus i_mode (160444) for inode mmcblk0p3:59824
[ 1283.667864] attempt to access beyond end of device
[ 1283.667869] mmcblk0p3: rw=8192, want=5764893396078298024, limit=15120384
[ 1284.492368] init_special_inode: bogus i_mode (4620) for inode mmcblk0p3:17013
[ 1284.707910] mmcblk0: error -110 transferring data, sector 5236904, nr 24, cmd response 0x900, card status 0xb00
[ 1284.707926] mmcblk0: retrying using single block read
[ 1284.792489] mmcblk0: error -110 transferring data, sector 5236904, nr 24, cmd response 0x900, card status 0x0
[ 1284.792498] end_request: I/O error, dev mmcblk0, sector 5236904
[ 1284.877077] mmcblk0: error -110 transferring data, sector 5236905, nr 23, cmd response 0x900, card status 0x0
[ 1284.877091] end_request: I/O error, dev mmcblk0, sector 5236905
[ 1284.961660] mmcblk0: error -110 transferring data, sector 5236906, nr 22, cmd response 0x900, card status 0x0
[ 1284.961674] end_request: I/O error, dev mmcblk0, sector 5236906
[ 1285.046235] mmcblk0: error -110 transferring data, sector 5236907, nr 21, cmd response 0x900, card status 0x0
[ 1285.046246] end_request: I/O error, dev mmcblk0, sector 5236907
[ 1285.130754] mmcblk0: error -110 transferring data, sector 5236908, nr 20, cmd response 0x900, card status 0x0
[ 1285.130764] end_request: I/O error, dev mmcblk0, sector 5236908
[ 1285.215320] mmcblk0: error -110 transferring data, sector 5236909, nr 19, cmd response 0x900, card status 0x0
[ 1285.215332] end_request: I/O error, dev mmcblk0, sector 5236909
[ 1285.299926] mmcblk0: error -110 transferring data, sector 5236910, nr 18, cmd response 0x900, card status 0x0
[ 1285.299945] end_request: I/O error, dev mmcblk0, sector 5236910
[ 1285.384537] mmcblk0: error -110 transferring data, sector 5236911, nr 17, cmd response 0x900, card status 0x0
[ 1285.384554] end_request: I/O error, dev mmcblk0, sector 5236911
[ 1285.615295] init_special_inode: bogus i_mode (2024) for inode mmcblk0p3:2218
[ 1288.601672] attempt to access beyond end of device
[ 1288.601677] mmcblk0p3: rw=8192, want=145241775180196072, limit=15120384
[ 1288.601681] attempt to access beyond end of device
[ 1288.601683] mmcblk0p3: rw=8192, want=36678440, limit=15120384
[ 1290.442474] init_special_inode: bogus i_mode (160545) for inode mmcblk0p3:59808
[ 1290.677312] attempt to access beyond end of device
[ 1290.677317] mmcblk0p3: rw=0, want=12305576757786212880, limit=15120384
[ 1290.677319] attempt to access beyond end of device
[ 1290.677321] mmcblk0p3: rw=0, want=12305576757786212880, limit=15120384
[ 1290.677324] NILFS error (device mmcblk0p3): nilfs_readdir: bad page in #38030
[ 1290.677325] Remounting filesystem read-only
[ 1290.681553] attempt to access beyond end of device
[ 1290.681557] mmcblk0p3: rw=0, want=1231588818889969824, limit=15120384
[ 1290.681559] attempt to access beyond end of device
[ 1290.681560] mmcblk0p3: rw=0, want=1231588818889969824, limit=15120384
[ 1290.681563] NILFS error (device mmcblk0p3): nilfs_readdir: bad page in #38033
[ 1290.681578] attempt to access beyond end of device
[ 1290.681579] mmcblk0p3: rw=0, want=6682469067434422488, limit=15120384
[ 1290.681581] attempt to access beyond end of device
[ 1290.681582] mmcblk0p3: rw=0, want=6682469067434422488, limit=15120384
[ 1290.681584] NILFS error (device mmcblk0p3): nilfs_readdir: bad page in #38036
[ 1290.734000] init_special_inode: bogus i_mode (1752) for inode mmcblk0p3:38170
[ 1290.736411] init_special_inode: bogus i_mode (160545) for inode mmcblk0p3:59816
[ 1290.736419] init_special_inode: bogus i_mode (1752) for inode mmcblk0p3:59818
[ 1290.736424] init_special_inode: bogus i_mode (160545) for inode mmcblk0p3:38168
[ 1291.005227] init_special_inode: bogus i_mode (50004) for inode mmcblk0p3:2042
[ 1291.013041] init_special_inode: bogus i_mode (50444) for inode mmcblk0p3:2815


Should I check the card in the first place?


Piotr Szymaniak.
-- 
<kow`> "There are 10 types of people in the world... those who
understand binary and those who don't."
<SpaceRain> That's only 2 types of people, kow.
<SpaceRain> STUPID
  -- bash.org

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: attempt to access beyond end of device

[Vyacheslav Dubeyko]

Hi Piotr,

On Mon, 2014-03-24 at 08:13 +0100, Piotr Szymaniak wrote:
> Hello,
> 
> I'm running nilfs2 on Raspberry Pi and some time ago it got some
> problem.
> 
> Connected the SD card to my PC today and mounted the fs, the cleaner
> started and few minutes later it remounted the fs readonly. Here's the
> dmesg part:
> 
> [ 1109.889697] mmcblk0: mmc0:b368 SMI   7.51 GiB 
> [ 1109.891261]  mmcblk0: p1 p2 p3
> [ 1129.467450] NILFS warning: mounting unchecked fs
> [ 1129.956859] NILFS: recovery complete.
> [ 1129.960064] segctord starting. Construction interval = 300 seconds, CP frequency < 30 seconds
> [ 1129.979344] NILFS warning: mounting fs with errors
> [ 1283.005874] attempt to access beyond end of device
> [ 1283.005882] mmcblk0p3: rw=8192, want=145241775180196072, limit=15120384

I see that it was requested 8 KB block. How did you create file system?
Could you share the superblock content?

> 
> Should I check the card in the first place?
> 

Currently, I suspect that you have bad SD card. But, maybe, you have
something different.

With the best regards,
Vyacheslav Dubeyko.


--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: attempt to access beyond end of device

[Piotr Szymaniak]

[-- Attachment #1.1: Type: text/plain, Size: 3110 bytes --]

On Mon, Mar 24, 2014 at 11:37:35AM +0400, Vyacheslav Dubeyko wrote:
> Hi Piotr,
> 
> On Mon, 2014-03-24 at 08:13 +0100, Piotr Szymaniak wrote:
> > Hello,
> > 
> > I'm running nilfs2 on Raspberry Pi and some time ago it got some
> > problem.
> > 
> > Connected the SD card to my PC today and mounted the fs, the cleaner
> > started and few minutes later it remounted the fs readonly. Here's the
> > dmesg part:
> > 
> > [ 1109.889697] mmcblk0: mmc0:b368 SMI   7.51 GiB 
> > [ 1109.891261]  mmcblk0: p1 p2 p3
> > [ 1129.467450] NILFS warning: mounting unchecked fs
> > [ 1129.956859] NILFS: recovery complete.
> > [ 1129.960064] segctord starting. Construction interval = 300 seconds, CP frequency < 30 seconds
> > [ 1129.979344] NILFS warning: mounting fs with errors
> > [ 1283.005874] attempt to access beyond end of device
> > [ 1283.005882] mmcblk0p3: rw=8192, want=145241775180196072, limit=15120384
> 
> I see that it was requested 8 KB block. How did you create file system?

That was some time ago, so I'm unable to answer this question. But
nilfs-tune shows 4KB blocks:

~ # nilfs-tune -l /dev/sdf3
nilfs-tune 2.1.6
Filesystem volume name:   (none)
Filesystem UUID:          53760664-f9ed-4c8d-af42-c6ee2f16d956
Filesystem magic number:  0x3434
Filesystem revision #:    2.0
Filesystem features:      (none)
Filesystem state:         invalid or mounted,error
Filesystem OS type:       Linux
Block size:               4096
Filesystem created:       Fri Aug  3 08:37:06 2012
Last mount time:          Mon Mar 24 07:55:53 2014
Last write time:          Mon Mar 24 07:58:04 2014
Mount count:              100
Maximum mount count:      50
Reserve blocks uid:       0 (user root)
Reserve blocks gid:       0 (group root)
First inode:              11
Inode size:               128
DAT entry size:           32
Checkpoint size:          192
Segment usage size:       16
Number of segments:       922
Device size:              7741636608
First data block:         1
# of blocks per segment:  2048
Reserved segments %:      5
Last checkpoint #:        770589
Last block address:       595617
Last sequence #:          41763
Free blocks count:        395264
Commit interval:          300
# of blks to create seg:  0
CRC seed:                 0x3e0bea06
CRC check sum:            0x2596b671
CRC check data size:      0x00000118



> Could you share the superblock content?

If my calculations are correct (it's the second superblock, right?), it should
be:
dd if=/dev/nilfs of=dump bs=Block size skip=Device size / Block size - 1
count=1

Attached this one.


> > Should I check the card in the first place?
> > 
> 
> Currently, I suspect that you have bad SD card. But, maybe, you have
> something different.

Maybe I will check it.


Piotr Szymaniak.
-- 
(...) wszystko to sprawilo,  iz przekroczyl owa umowna  granice,  ktora
jego przyjaciel,  prawnik Dan Tabares,  nazywal linia MTWD. Gdy raz juz
przekroczyles linie  MTWD  cokolwiek by sie zdarzylo, ty po prostu Masz
To W Dupie.
  -- Graham Masterton, "The Burning"

[-- Attachment #1.2: dump.1890048.gz --]
[-- Type: application/octet-stream, Size: 179 bytes --]

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: attempt to access beyond end of device

[Piotr Szymaniak]

[-- Attachment #1: Type: text/plain, Size: 512 bytes --]

On Mon, Mar 24, 2014 at 11:37:35AM +0400, Vyacheslav Dubeyko wrote:
> Currently, I suspect that you have bad SD card. But, maybe, you have
> something different.

FYI nondestructive read/write badblocks test passed with 0 errors.

Piotr Szymaniak.
-- 
 Relastatyka  czasowa utrzymywała ją w kuli nicości,  której nie należy
mylić z próżnią.  Kula składała się z absolutnej nicości,  w jakiej nie
mogłaby zaistnieć próżnia.
  -- Douglas Adams, "Restaurant at The End of The Universe"

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: attempt to access beyond end of device

[Vyacheslav Dubeyko]

On Mon, 2014-03-24 at 21:49 +0100, Piotr Szymaniak wrote:

> That was some time ago, so I'm unable to answer this question. But
> nilfs-tune shows 4KB blocks:
> 
> ~ # nilfs-tune -l /dev/sdf3
> nilfs-tune 2.1.6
> Filesystem volume name:   (none)
> Filesystem UUID:          53760664-f9ed-4c8d-af42-c6ee2f16d956
> Filesystem magic number:  0x3434
> Filesystem revision #:    2.0
> Filesystem features:      (none)
> Filesystem state:         invalid or mounted,error
> Filesystem OS type:       Linux
> Block size:               4096
> Filesystem created:       Fri Aug  3 08:37:06 2012

I suspect that you used this NILFS2 volume with kernel version that it
was trouble with segments creation. So, you can have volume corruption
because of this bug that it was fixed.

(1) What Linux kernel version do use currently?
(2) How long did you use old kernel versions?
(3) What workloads did you use for this volume?
(4) Had you situations of simultaneous working of several threads with
intensive operations with volume (for example, simultaneous compilation
and checking for updates)?

Thanks,
Vyacheslav Dubeyko.


--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: attempt to access beyond end of device

[Piotr Szymaniak]

[-- Attachment #1: Type: text/plain, Size: 1883 bytes --]

On Tue, Mar 25, 2014 at 01:18:10PM +0400, Vyacheslav Dubeyko wrote:
> On Mon, 2014-03-24 at 21:49 +0100, Piotr Szymaniak wrote:
> 
> > That was some time ago, so I'm unable to answer this question. But
> > nilfs-tune shows 4KB blocks:
> > 
> > ~ # nilfs-tune -l /dev/sdf3
> > nilfs-tune 2.1.6
> > Filesystem volume name:   (none)
> > Filesystem UUID:          53760664-f9ed-4c8d-af42-c6ee2f16d956
> > Filesystem magic number:  0x3434
> > Filesystem revision #:    2.0
> > Filesystem features:      (none)
> > Filesystem state:         invalid or mounted,error
> > Filesystem OS type:       Linux
> > Block size:               4096
> > Filesystem created:       Fri Aug  3 08:37:06 2012
> 
> I suspect that you used this NILFS2 volume with kernel version that it
> was trouble with segments creation. So, you can have volume corruption
> because of this bug that it was fixed.
> 
> (1) What Linux kernel version do use currently?

I think it was 3.6.y or 3.10.y


> (2) How long did you use old kernel versions?

I suppose at least since 2012-10.08 [1] as this is the same Raspberry Pi
and, afair, the same fs. So it was 3.2.27 and then some of (some of them
was unstable) 3.{6,10,11}.y


> (3) What workloads did you use for this volume?

It was almost idle except once a day or two accepting backup from
another machine (but that was kept on external drive/fs).


> (4) Had you situations of simultaneous working of several threads with
> intensive operations with volume (for example, simultaneous compilation
> and checking for updates)?

Could be, but compilation was made on external drive. nilfs holds the
rootfs and almost nothing more.


[1] http://marc.info/?l=linux-nilfs&m=134973563927736&w=2


Piotr Szymaniak.
-- 
Czekajcie,  czekajcie.  Ktos cos do mnie mowil,  ale nie wiem kto i nie
wiem co.
  -- Rafal Solecki

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: attempt to access beyond end of device

[Vyacheslav Dubeyko]

On Thu, 2014-03-27 at 13:39 +0100, Piotr Szymaniak wrote:

> [ 1109.889697] mmcblk0: mmc0:b368 SMI   7.51 GiB 
> [ 1109.891261]  mmcblk0: p1 p2 p3
> [ 1129.467450] NILFS warning: mounting unchecked fs
> [ 1129.956859] NILFS: recovery complete.
> [ 1129.960064] segctord starting. Construction interval = 300 seconds, CP frequency < 30 seconds
> [ 1129.979344] NILFS warning: mounting fs with errors
> [ 1283.005874] attempt to access beyond end of device
> [ 1283.005882] mmcblk0p3: rw=8192, want=145241775180196072, limit=15120384

> > ~ # nilfs-tune -l /dev/sdf3
> > > nilfs-tune 2.1.6
> > > Filesystem volume name:   (none)
> > > Filesystem UUID:          53760664-f9ed-4c8d-af42-c6ee2f16d956
> > > Filesystem magic number:  0x3434
> > > Filesystem revision #:    2.0
> > > Filesystem features:      (none)
> > > Filesystem state:         invalid or mounted,error
> > > Filesystem OS type:       Linux
> > > Block size:               4096

I suppose that you have corrupted file system. Because it is possible to
see that file system has 4 KB block size but the read request is 8 KB in
size (moreover, this request is out of device). You are using this
NILFS2 volume from 2012 year. So, the reason of the issue can be only
corrupted block on the volume.

I assume that there is significant probability of occurrence the issue
with competition of segments for dirty blocks during segment
construction. I've reproduced this issue very stably when it takes place
simultaneous checking for updates and some intensive activity with
volume (for example, compilation). So, such issue results with
corruption of file system. But you can discover such corruption far away
from place of origin.

Anyway, I'll happy to investigate your issue if you can describe a
reproducing path. I need in stable reproducing of the issue for
investigation and fix. Could you describe it?

Thanks,
Vyacheslav Dubeyko.


--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html