nilfs_cleanerd cannot open nilfs on device that exists

nilfs_cleanerd cannot open nilfs on device that exists

[Piotr Szymaniak]

[-- Attachment #1: Type: text/plain, Size: 1080 bytes --]

Hello,

I got a power failure today and after power restore there's a weird
issue with nilfs_cleanerd:

wloczykij ~ # ll /dev/sda3
brw-rw---- 1 root disk 8, 3 Dec 12 08:35 /dev/sda3

wloczykij ~ # mount | grep nilfs
/dev/root on / type nilfs2 (rw,relatime)
/dev/mapper/backup on /mnt/home_backup type nilfs2 (rw,gcpid=2995)
/dev/sda3 on / type nilfs2 (rw,noatime,nodiratime,gcpid=5333)

But when I try to run the cleaner:
Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: start
Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: cannot open nilfs on
/dev/sda3: No such file or directory
Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: cannot create cleanerd
on /dev/sda3: No such file or directory
Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: shutdown

Never happend before. Shutdown should be nice (got UPS monitored by
apcupsd, so I hope it worked - was in the middle of the night, so can't
really tell if was superclean shutdown).


Piotr Szymaniak.
-- 
Smiejmy sie! Kto wie czy swiat potrwa jeszcze trzy tygodnie?
  -- Beaumarchais Pierre Augustin

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: nilfs_cleanerd cannot open nilfs on device that exists

[Piotr Szymaniak]

[-- Attachment #1: Type: text/plain, Size: 1559 bytes --]

On Wed, Dec 12, 2012 at 03:51:16PM +0100, Piotr Szymaniak wrote:
> Hello,
> 
> I got a power failure today and after power restore there's a weird
> issue with nilfs_cleanerd:
> 
> wloczykij ~ # ll /dev/sda3
> brw-rw---- 1 root disk 8, 3 Dec 12 08:35 /dev/sda3
> 
> wloczykij ~ # mount | grep nilfs
> /dev/root on / type nilfs2 (rw,relatime)
> /dev/mapper/backup on /mnt/home_backup type nilfs2 (rw,gcpid=2995)
> /dev/sda3 on / type nilfs2 (rw,noatime,nodiratime,gcpid=5333)
> 
> But when I try to run the cleaner:
> Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: start
> Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: cannot open nilfs on
> /dev/sda3: No such file or directory
> Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: cannot create cleanerd
> on /dev/sda3: No such file or directory
> Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: shutdown
> 
> Never happend before. Shutdown should be nice (got UPS monitored by
> apcupsd, so I hope it worked - was in the middle of the night, so can't
> really tell if was superclean shutdown).

I should also notice, that other cleaner of different drive is working
without issues:
wloczykij ~ # ps aux | grep nilfs
root      2995  0.0  0.0   2348   808 ?        Ss   12:19   0:00
/sbin/nilfs_cleanerd -n /dev/mapper/backup /mnt/backup


Piotr Szymaniak.
-- 
Przypomnialem sobie, jak powiedzial, ze po raz pierwszy w zyciu boi sie
piekla,  poniewaz  szykujemy sie do zniszczenia daru niebios.  Poczulem
nagle, ze chce mi sie rzygac.
  -- Stephen King, "The Green Mile"

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: nilfs_cleanerd cannot open nilfs on device that exists

[Piotr Szymaniak]

[-- Attachment #1.1: Type: text/plain, Size: 1696 bytes --]

On Wed, Dec 12, 2012 at 03:54:55PM +0100, Piotr Szymaniak wrote:
> On Wed, Dec 12, 2012 at 03:51:16PM +0100, Piotr Szymaniak wrote:
> > Hello,
> > 
> > I got a power failure today and after power restore there's a weird
> > issue with nilfs_cleanerd:
> > 
> > wloczykij ~ # ll /dev/sda3
> > brw-rw---- 1 root disk 8, 3 Dec 12 08:35 /dev/sda3
> > 
> > wloczykij ~ # mount | grep nilfs
> > /dev/root on / type nilfs2 (rw,relatime)
> > /dev/mapper/backup on /mnt/home_backup type nilfs2 (rw,gcpid=2995)
> > /dev/sda3 on / type nilfs2 (rw,noatime,nodiratime,gcpid=5333)
> > 
> > But when I try to run the cleaner:
> > Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: start
> > Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: cannot open nilfs on
> > /dev/sda3: No such file or directory
> > Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: cannot create cleanerd
> > on /dev/sda3: No such file or directory
> > Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: shutdown
> > 
> > Never happend before. Shutdown should be nice (got UPS monitored by
> > apcupsd, so I hope it worked - was in the middle of the night, so can't
> > really tell if was superclean shutdown).
> 
> I should also notice, that other cleaner of different drive is working
> without issues:
> wloczykij ~ # ps aux | grep nilfs
> root      2995  0.0  0.0   2348   808 ?        Ss   12:19   0:00
> /sbin/nilfs_cleanerd -n /dev/mapper/backup /mnt/backup

Sorry for answering my own msg again.

Attached strace output.


Piotr Szymaniak.
-- 
Gdzie  w  tym  logika? Po co w ogole sie rodzic,  jesli jedyne,  co cie
czeka u kresu tego wszystkiego, to smierc?
  -- Graham Masterton, "The Burning"

[-- Attachment #1.2: strace-nilfs_cleanerd.dontexist.out --]
[-- Type: text/plain, Size: 5137 bytes --]

execve("/sbin/nilfs_cleanerd", ["/sbin/nilfs_cleanerd", "-c", "/etc/nilfs_cleanerd.conf", "/dev/sda3"], [/* 28 vars */]) = 0
brk(0)                                  = 0x865c000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7727000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=23848, ...}) = 0
mmap2(NULL, 23848, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7721000
close(3)                                = 0
open("/lib/libnilfs.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\240\24\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=18744, ...}) = 0
mmap2(NULL, 21660, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb771b000
mmap2(0xb771f000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3) = 0xb771f000
close(3)                                = 0
open("/lib/libnilfsgc.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\220\20\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=13652, ...}) = 0
mmap2(NULL, 16568, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7716000
mmap2(0xb7719000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2) = 0xb7719000
close(3)                                = 0
open("/lib/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\0\36\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=30552, ...}) = 0
mmap2(NULL, 33364, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb770d000
mmap2(0xb7714000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6) = 0xb7714000
close(3)                                = 0
open("/lib/libuuid.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\0\21\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=13808, ...}) = 0
mmap2(NULL, 16588, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7708000
mmap2(0xb770b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2) = 0xb770b000
close(3)                                = 0
open("/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\320\314\1\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1709408, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7707000
mmap2(NULL, 1719044, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7563000
mmap2(0xb7701000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19e) = 0xb7701000
mmap2(0xb7704000, 11012, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7704000
close(3)                                = 0
open("/lib/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\260k\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=124345, ...}) = 0
mmap2(NULL, 107016, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7548000
mmap2(0xb755f000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16) = 0xb755f000
mmap2(0xb7561000, 4616, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7561000
close(3)                                = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7547000
set_thread_area({entry_number:-1 -> 6, base_addr:0xb7547940, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0xb7701000, 8192, PROT_READ)   = 0
mprotect(0xb755f000, 4096, PROT_READ)   = 0
mprotect(0xb770b000, 4096, PROT_READ)   = 0
mprotect(0xb7714000, 4096, PROT_READ)   = 0
mprotect(0xb771f000, 4096, PROT_READ)   = 0
mprotect(0xb7719000, 4096, PROT_READ)   = 0
mprotect(0x804e000, 4096, PROT_READ)    = 0
mprotect(0xb7749000, 4096, PROT_READ)   = 0
munmap(0xb7721000, 23848)               = 0
set_tid_address(0xb75479a8)             = 9341
set_robust_list(0xb75479b0, 12)         = 0
futex(0xbff37f24, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0xbff37f24, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, b7547940) = -1 EAGAIN (Resource temporarily unavailable)
rt_sigaction(SIGRTMIN, {0xb754e540, [], SA_SIGINFO}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0xb754e5c0, [], SA_RESTART|SA_SIGINFO}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
uname({sys="Linux", node="wloczykij", ...}) = 0
readlink("/dev", 0xbff35baf, 4096)      = -1 EINVAL (Invalid argument)
readlink("/dev/sda3", 0xbff35baf, 4096) = -1 EINVAL (Invalid argument)
brk(0)                                  = 0x865c000
brk(0x867d000)                          = 0x867d000
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb75479a8) = 9342
exit_group(0)                           = ?
+++ exited with 0 +++

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: nilfs_cleanerd cannot open nilfs on device that exists

[Vyacheslav Dubeyko]

On Dec 12, 2012, at 7:10 PM, Piotr Szymaniak wrote:

> On Wed, Dec 12, 2012 at 03:54:55PM +0100, Piotr Szymaniak wrote:
>> On Wed, Dec 12, 2012 at 03:51:16PM +0100, Piotr Szymaniak wrote:
>>> Hello,
>>> 
>>> I got a power failure today and after power restore there's a weird
>>> issue with nilfs_cleanerd:
>>> 
>>> wloczykij ~ # ll /dev/sda3
>>> brw-rw---- 1 root disk 8, 3 Dec 12 08:35 /dev/sda3
>>> 
>>> wloczykij ~ # mount | grep nilfs
>>> /dev/root on / type nilfs2 (rw,relatime)
>>> /dev/mapper/backup on /mnt/home_backup type nilfs2 (rw,gcpid=2995)
>>> /dev/sda3 on / type nilfs2 (rw,noatime,nodiratime,gcpid=5333)
>>> 
>>> But when I try to run the cleaner:

Sorry, I slightly confused. As I understand, mount output shows that you have mounted /dev/sda3 with started nilfs_cleanerd (gcpid=5333). Why do you try to start nilfs_cleanerd by hands? Or I misunderstand something?

As I can see from strace, there is such error message:

futex(0xbff37f24, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, b7547940) = -1 EAGAIN (Resource temporarily unavailable)

I think that it is the reason of failing nilfs_cleanerd start.

Maybe, do you have working or failed nilfs_cleanerd for /dev/sda3 in background? Do you have any NILFS2-related messages in system log?

With the best regards,
Vyacheslav Dubeyko.


>>> Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: start
>>> Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: cannot open nilfs on
>>> /dev/sda3: No such file or directory
>>> Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: cannot create cleanerd
>>> on /dev/sda3: No such file or directory
>>> Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: shutdown
>>> 
>>> Never happend before. Shutdown should be nice (got UPS monitored by
>>> apcupsd, so I hope it worked - was in the middle of the night, so can't
>>> really tell if was superclean shutdown).
>> 
>> I should also notice, that other cleaner of different drive is working
>> without issues:
>> wloczykij ~ # ps aux | grep nilfs
>> root      2995  0.0  0.0   2348   808 ?        Ss   12:19   0:00
>> /sbin/nilfs_cleanerd -n /dev/mapper/backup /mnt/backup
> 
> Sorry for answering my own msg again.
> 
> Attached strace output.
> 
> 
> Piotr Szymaniak.
> -- 
> Gdzie  w  tym  logika? Po co w ogole sie rodzic,  jesli jedyne,  co cie
> czeka u kresu tego wszystkiego, to smierc?
>  -- Graham Masterton, "The Burning"
> <strace-nilfs_cleanerd.dontexist.out>

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nilfs_cleanerd cannot open nilfs on device that exists

[Piotr Szymaniak]

[-- Attachment #1: Type: text/plain, Size: 2078 bytes --]

On Wed, Dec 12, 2012 at 11:43:40PM +0300, Vyacheslav Dubeyko wrote:
> 
> On Dec 12, 2012, at 7:10 PM, Piotr Szymaniak wrote:
> 
> > On Wed, Dec 12, 2012 at 03:54:55PM +0100, Piotr Szymaniak wrote:
> >> On Wed, Dec 12, 2012 at 03:51:16PM +0100, Piotr Szymaniak wrote:
> >>> Hello,
> >>> 
> >>> I got a power failure today and after power restore there's a weird
> >>> issue with nilfs_cleanerd:
> >>> 
> >>> wloczykij ~ # ll /dev/sda3
> >>> brw-rw---- 1 root disk 8, 3 Dec 12 08:35 /dev/sda3
> >>> 
> >>> wloczykij ~ # mount | grep nilfs
> >>> /dev/root on / type nilfs2 (rw,relatime)
> >>> /dev/mapper/backup on /mnt/home_backup type nilfs2 (rw,gcpid=2995)
> >>> /dev/sda3 on / type nilfs2 (rw,noatime,nodiratime,gcpid=5333)
> >>> 
> >>> But when I try to run the cleaner:
> 
> Sorry, I slightly confused. As I understand, mount output shows that you have mounted /dev/sda3 with started nilfs_cleanerd (gcpid=5333). Why do you try to start nilfs_cleanerd by hands? Or I misunderstand something?

It's not running. I got similar message on boot:
Dec 12 08:35:23 wloczykij nilfs_cleanerd[1689]: start
Dec 12 08:35:23 wloczykij nilfs_cleanerd[1689]: cannot open nilfs on
/dev/sda3: No such file or directory
Dec 12 08:35:23 wloczykij nilfs_cleanerd[1689]: cannot create cleanerd
on /dev/sda3: No such file or directory
Dec 12 08:35:23 wloczykij nilfs_cleanerd[1689]: shutdown



> As I can see from strace, there is such error message:
> 
> futex(0xbff37f24, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, b7547940) = -1 EAGAIN (Resource temporarily unavailable)
> 
> I think that it is the reason of failing nilfs_cleanerd start.
> 
> Maybe, do you have working or failed nilfs_cleanerd for /dev/sda3 in background? Do you have any NILFS2-related messages in system log?

ps ax shows only one nilfs_cleanerd process (at different drive).


Piotr Szymaniak.
-- 
 Podejrzane!  No,  yhm!  Takem  sobie  pomyślał!  Jezus  jakby  się  o tym
zwiedział, toby wtrząchnął puszkę fasoli!
  -- Stephen King, "The Dark Half"

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: nilfs_cleanerd cannot open nilfs on device that exists

[Piotr Szymaniak]

[-- Attachment #1: Type: text/plain, Size: 729 bytes --]

On Wed, Dec 12, 2012 at 03:51:16PM +0100, Piotr Szymaniak wrote:
> Hello,

Sorry, it looks like I need more memory [1]. The issue is _again_
related to missing /dev/root that should point to my /dev/sda3 [2].
I was sure it was fixed meanwhile (had ~200 days uptime, so I just
forgot about the issue…), but it looks like it isn't (at least in Gentoo
Linux [3]).

Sorry for wasting your time. New udev, blargh. /-:

[1] http://article.gmane.org/gmane.comp.file-systems.nilfs.user/2437
[2] http://article.gmane.org/gmane.comp.file-systems.nilfs.user/2438
[3] https://bugs.gentoo.org/438380


Piotr Szymaniak.
-- 
 - You know nothing about woman!
 - Of course not, I'm a man, so how could I know?
  -- Berserk

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: nilfs_cleanerd cannot open nilfs on device that exists

[Vyacheslav Dubeyko]

Hi,

On Jan 3, 2013, at 1:51 AM, sheng qiu wrote:

> Hi all,
> 
> i am new on nilfs2. i want to collect the information about the segment cleaning overhead. for example, how many valid data copied during segment cleaning process. Can anyone give me some hint where i can add these stats codes within the nilfs2 source code? or which source files are related with segment cleaning? 
> 

The segments' cleaning code is distributed between nilfs-cleanerd daemon and NILFS2 driver itself. The nilfs-cleanerd daemon is a part of nilfs-utils. It collects information about dirty segments and initiates cleaning activity by means of ioctl on the driver side. So, you can see nilfs-cleanerd source code and linux/fs/nilfs2/ioctl.c source code as starting points. It is used in cleaning activity, as minimum, nilfs_ioctl_get_bdescs() and nilfs_ioctl_clean_segments(). I think that, maybe, it will be more easy and useful to use user-space nilfs-cleanerd side. Moreover, maybe, it make sense to use blktrace utility for your research as additional tool.

What information do you want to get for your research? How do you plan to analyze and interpret segments cleaning overhead?

But, currently, I think that such research can be a very difficult task because of unpredictability of cleaning overhead nature. It depends from filesystem aging state and history of previous filesystem operations very much, from my point of view. What the purpose of your research activity? Could you share results of your research in the direction of segments cleaning overhead?

With the best regards,
Vyacheslav Dubeyko.

> I will be very appreciated for your help. 
> i tried to send email through the linux-nilfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, however it fails to deliver, i have to write my email by replying this. hope you won't mind.
> 
> Thanks,
> Sheng
> 
> -- 
> Sheng Qiu
> Texas A & M University
> Room 332B Wisenbaker    
> email: herbert1984106-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
> College Station, TX 77843-3259

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html