IOzone testing

IOzone testing

[Jeff Layton]

Good morning,

I'm running IOzone on a NILFS2 partition and it's running alot
slower that I had expected. I'm pretty sure it's because of the
checkpoints. I can change the protection_period to something
very small (or zero). Would that improve the performance?

In addition, is there a way to turn off checkpointing?

Thanks!

Jeff

Re: IOzone testing

[Jérôme Poulin]

On Sat, Sep 26, 2009 at 8:13 AM, Jeff Layton <laytonjb-fOdFMYwuEsI@public.gmane.org> wrote:
> Good morning,
>
> I'm running IOzone on a NILFS2 partition and it's running alot
> slower that I had expected. I'm pretty sure it's because of the
> checkpoints. I can change the protection_period to something
> very small (or zero). Would that improve the performance?
>
The protection period only affects how the cleaner daemon will clean
the old checkpoints, if 0 I guess it will just clean them up as soon
as it can but it won't stop creating them. You could, just for
benchmark purpose, kill nilfs_cleanerd which will stop checkpoint
related activities.

> In addition, is there a way to turn off checkpointing?
>
The whole filesystem is based on those checkpoint, the fact that it is
a log filesystem makes it so all writes are made at the end of the
device, and when a write is complete, a checkpoint is made.

> Thanks!
>
> Jeff

Re: IOzone testing

[Jeff Layton]

Jérôme Poulin wrote:
> On Sat, Sep 26, 2009 at 8:13 AM, Jeff Layton <laytonjb-fOdFMYwuEsI@public.gmane.org> wrote:
>   
>> Good morning,
>>
>> I'm running IOzone on a NILFS2 partition and it's running alot
>> slower that I had expected. I'm pretty sure it's because of the
>> checkpoints. I can change the protection_period to something
>> very small (or zero). Would that improve the performance?
>>
>>     
> The protection period only affects how the cleaner daemon will clean
> the old checkpoints, if 0 I guess it will just clean them up as soon
> as it can but it won't stop creating them. You could, just for
> benchmark purpose, kill nilfs_cleanerd which will stop checkpoint
> related activities.
>   
It will only stop the clean up of checkpoints outside of policy - correct?
So it won't stop them from being created but it will stop the cleaning 
process.

>> In addition, is there a way to turn off checkpointing?
>>
>>     
> The whole filesystem is based on those checkpoint, the fact that it is
> a log filesystem makes it so all writes are made at the end of the
> device, and when a write is complete, a checkpoint is made.
>   

Understood. But for benchmarks like IOzone where you are creating
files and most likely syncing, this will trigger a checkpoint. I haven't
looked too much at IOzone but it might even be causing a sync after
each record right, again forcing a checkpoint.

So I was looking for some way to control how often checkpoints are
created - something like the protection_period option for the GC. But
I understand that the checkpoint is a part of the design of the file system.

Thanks!

Jeff

Re: IOzone testing

[Jérôme Poulin]

On Sat, Sep 26, 2009 at 8:13 AM, Jeff Layton <laytonjb-fOdFMYwuEsI@public.gmane.org> wrote:
> It will only stop the clean up of checkpoints outside of policy - correct?
> So it won't stop them from being created but it will stop the cleaning
> process.
>
It will completely stop cleaning checkpoints, which is what is
consuming drive I/O. Checkpoints in themselve don't use much
ressources, cleaning them however does.

>
> Understood. But for benchmarks like IOzone where you are creating
> files and most likely syncing, this will trigger a checkpoint. I haven't
> looked too much at IOzone but it might even be causing a sync after
> each record right, again forcing a checkpoint.
>
> So I was looking for some way to control how often checkpoints are
> created - something like the protection_period option for the GC. But
> I understand that the checkpoint is a part of the design of the file system.
>
Checkpoints are created each time a change is commited to disk right,
but creating a checkpoint does not take much system ressources, try
time'ing mkcp, it won't use any ressources, it just writes a marker at
the end of the disk to say that all that is written behind it is good
to use if a crash happens or the user wants to mount the partition in
the past. Increasing checkpoint period would just increases the risk
of losing more data on crash with minimal or no performance gain.

I don't know if you ever worked with shadow copies in Windows but
creating a shadow copie would consume lots of drive I/O and CPU, that
is because NTFS wasn't conceived as a log filesystem like NILFS and
must write structures and files to the disk for whatever changes
happened to the disk, it must also freeze the filesystem and make sure
all writes are atomic before creating the snapshot. While NILFS just
write a checkpoint each time it knows the last write is complete,
converting a checkpoint to a snapshot then is just a matter of
changing a flag so the GC won't remove it afterward.

Re: IOzone testing

[Jeff Layton]

Jérôme Poulin wrote:
> On Sat, Sep 26, 2009 at 8:13 AM, Jeff Layton <laytonjb-fOdFMYwuEsI@public.gmane.org> wrote:
>   
>> It will only stop the clean up of checkpoints outside of policy - correct?
>> So it won't stop them from being created but it will stop the cleaning
>> process.
>>
>>     
> It will completely stop cleaning checkpoints, which is what is
> consuming drive I/O. Checkpoints in themselve don't use much
> ressources, cleaning them however does.
>   

Got it - thanks.

At the same time, I also hit an out of space message. Why would I
get this message? If the checkpoints are just creating a small maker
then it shouldn't take much space. Unless of course, a huge number
of checkpoints are created.

Thanks!

Re: IOzone testing

[Jérôme Poulin]

On Sat, Sep 26, 2009 at 9:14 AM, Jeff Layton <laytonjb-fOdFMYwuEsI@public.gmane.org> wrote:
> Got it - thanks.
>
> At the same time, I also hit an out of space message. Why would I
> get this message? If the checkpoints are just creating a small maker
> then it shouldn't take much space. Unless of course, a huge number
> of checkpoints are created.
>
> Thanks!
>
If nilfs_cleanerd is shut down, then the filesystem will fill up until
there's no room for more data. When you delete data, only the
nilfs_cleanerd can reclaim the space, which is why you set up a
protection_period, deleted data will never be overwritten unless
reclaimed by the cleanerd.
Here is a program I made to represent the disk in a graphical form,
you'll see the cleanerd in action:
http://ticpu.net:8080/~jerome/lssu-gtk.py

You need python-2.4+ with pygtk and nilfs-utils to run the program.

Re: IOzone testing

[Jeff Layton]

Jérôme Poulin wrote:
> On Sat, Sep 26, 2009 at 9:14 AM, Jeff Layton <laytonjb-fOdFMYwuEsI@public.gmane.org> wrote:
>   
>> Got it - thanks.
>>
>> At the same time, I also hit an out of space message. Why would I
>> get this message? If the checkpoints are just creating a small maker
>> then it shouldn't take much space. Unless of course, a huge number
>> of checkpoints are created.
>>
>> Thanks!
>>
>>     
> If nilfs_cleanerd is shut down, then the filesystem will fill up until
> there's no room for more data. When you delete data, only the
> nilfs_cleanerd can reclaim the space, which is why you set up a
> protection_period, deleted data will never be overwritten unless
> reclaimed by the cleanerd.
> Here is a program I made to represent the disk in a graphical form,
> you'll see the cleanerd in action:
> http://ticpu.net:8080/~jerome/lssu-gtk.py
>
> You need python-2.4+ with pygtk and nilfs-utils to run the program.
>   

Cool program - I will definitely try this.

So, let me summarize what I think I should do to test nilfs with
IOzone without the impact of the checkpoints and without filling
up the space.

- Run IOzone without cleanerd running to minimize the
impact of checkpoints on IO performance.
- Assuming I don't fill up the file system, after a run of
IOzone, I need to run cleanerd but put the protection
period to something very small (e.g. 1 second) to reclaim the
space.
- Shut down cleanerd again and run another IOzone test.
(rinse and repeat)

Does this sound about right?

Thanks!

Jeff

Re: IOzone testing

[Jérôme Poulin]

On Sat, Sep 26, 2009 at 10:36 AM, Jeff Layton <laytonjb-fOdFMYwuEsI@public.gmane.org> wrote:
> Jérôme Poulin wrote:
>
> Cool program - I will definitely try this.
>
> So, let me summarize what I think I should do to test nilfs with
> IOzone without the impact of the checkpoints and without filling
> up the space.
>
> - Run IOzone without cleanerd running to minimize the
> impact of checkpoints on IO performance.
> - Assuming I don't fill up the file system, after a run of
> IOzone, I need to run cleanerd but put the protection
> period to something very small (e.g. 1 second) to reclaim the
> space.
> - Shut down cleanerd again and run another IOzone test.
> (rinse and repeat)
>
> Does this sound about right?
>
> Thanks!
>
> Jeff
>
That sounds right, either run cleanerd or mkfs the partition again for
faster results ;)

Re: IOzone testing

[Jeff Layton]

Jérôme Poulin wrote:
> On Sat, Sep 26, 2009 at 10:36 AM, Jeff Layton <laytonjb-fOdFMYwuEsI@public.gmane.org> wrote:
>   
>> Jérôme Poulin wrote:
>>
>> Cool program - I will definitely try this.
>>
>> So, let me summarize what I think I should do to test nilfs with
>> IOzone without the impact of the checkpoints and without filling
>> up the space.
>>
>> - Run IOzone without cleanerd running to minimize the
>> impact of checkpoints on IO performance.
>> - Assuming I don't fill up the file system, after a run of
>> IOzone, I need to run cleanerd but put the protection
>> period to something very small (e.g. 1 second) to reclaim the
>> space.
>> - Shut down cleanerd again and run another IOzone test.
>> (rinse and repeat)
>>
>> Does this sound about right?
>>
>> Thanks!
>>
>> Jeff
>>
>>     
> That sounds right, either run cleanerd or mkfs the partition again for
> faster results ;)
>   

Good point about using mkfs. :)

I think I'm going to run two test: (1) using the process above, (2) setting
the protection period to 1 (maybe 0). Let's see what the differences are.

Thanks!

Jeff

Re: IOzone testing

[Byron Guernsey]

Be aware that garbage collection will take quite a while on a full  
drive to go back to 0 unless you also increase the segments cleaned  
per run and if you accidentally leave it high it will impact  
performance.

This is why I had asked about a way to force a full GC with a  
protection period other than that in the conf file. It seems like  
quick forced cleanup when a user deletes a lot of files is a common  
need...

Byron

On Sep 26, 2009, at 11:08 AM, Jeff Layton <laytonjb@att.net> wrote:

> Jérôme Poulin wrote:
>> On Sat, Sep 26, 2009 at 10:36 AM, Jeff Layton <laytonjb@att.net>  
>> wrote:
>>
>>> Jérôme Poulin wrote:
>>>
>>> Cool program - I will definitely try this.
>>>
>>> So, let me summarize what I think I should do to test nilfs with
>>> IOzone without the impact of the checkpoints and without filling
>>> up the space.
>>>
>>> - Run IOzone without cleanerd running to minimize the
>>> impact of checkpoints on IO performance.
>>> - Assuming I don't fill up the file system, after a run of
>>> IOzone, I need to run cleanerd but put the protection
>>> period to something very small (e.g. 1 second) to reclaim the
>>> space.
>>> - Shut down cleanerd again and run another IOzone test.
>>> (rinse and repeat)
>>>
>>> Does this sound about right?
>>>
>>> Thanks!
>>>
>>> Jeff
>>>
>>>
>> That sounds right, either run cleanerd or mkfs the partition again  
>> for
>> faster results ;)
>>
>
> Good point about using mkfs. :)
>
> I think I'm going to run two test: (1) using the process above, (2)  
> setting
> the protection period to 1 (maybe 0). Let's see what the differences  
> are.
>
> Thanks!
>
> Jeff
>
> _______________________________________________
> users mailing list
> users@nilfs.org
> https://www.nilfs.org/mailman/listinfo/users
_______________________________________________
users mailing list
users@nilfs.org
https://www.nilfs.org/mailman/listinfo/users

Re: IOzone testing

[Jeff Layton]

Byron Guernsey wrote:
> Be aware that garbage collection will take quite a while on a full  
> drive to go back to 0 unless you also increase the segments cleaned  
> per run and if you accidentally leave it high it will impact  
> performance.
>
> This is why I had asked about a way to force a full GC with a  
> protection period other than that in the conf file. It seems like  
> quick forced cleanup when a user deletes a lot of files is a common  
> need...
>   

That is a good feature to add - quick full GC with a different
protection period. +1 on that option.

I think I might just to a mkfs in between runs to make sure that
the checkpoints are eliminated. That works for my testing but
it doesn't work for real, live, working file systems.

Jeff

NILFS hanging SLES 11 - advise on diagnosis needed

[Barham, David]

Hi
I'm running SLES 11, 2.6.27.19-5-default with NILFS2 nilfs-2.0.16. I have a 1.5Tb NILFS2 partition which I am setting up with the intention of using Robocopy from various PCs via samba. The robocopy scripts run nightly and a checkpoint is taken once night. A script stops samba, unmounts the previous weeks checkpoint, deletes the checkpoint, creates a new one and then mounts it and restarts samba. This should mean that at any time the user can go back to 'snapshot_{DAY}' to get their files back.

So far so good.

However as I copy the previously backed up files from the previous linux machine where I was doing this (only giving a 'current' copy with reiserfs). I'm finding that the new machine is occasionally hanging. The OS just locks up, screen on console frozen but host still responds to ping. 

I'm trying to work out what is causing the hang, I'm getting various messages in the log from smartd relating to the disk which houses the NILFS along the lines of:

 Oct  2 09:56:59 cpli6008 syslog-ng[1933]: Log statistics; dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0', processed='center(queued)=947', processed='center(received)=478', processed='destination(newsnotice)=0', processed='destination(acpid)=0', processed='destination(firewall)=0', processed='destination(mail)=12', processed='destination(mailinfo)=12', processed='destination(console)=151', processed='destination(newserr)=0', processed='destination(newscrit)=0', processed='destination(messages)=466', processed='destination(mailwarn)=0', processed='destination(localmessages)=0', processed='destination(netmgm)=0', processed='destination(mailerr)=0', processed='destination(xconsole)=151', processed='destination(warn)=155', processed='source(src)=478'
Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 110 to 112
Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 115 to 117
Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Usage Attribute: 189 High_Fly_Writes changed from 88 to 87
Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61
Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 39
Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Usage Attribute: 195 Hardware_ECC_Recovered changed from 50 to 51

{machine stops responding and gets power cycled}

Oct  2 10:10:58 cpli6008 syslog-ng[1948]: syslog-ng starting up; version='2.0.9'

Do folks think that the hang is NILFS or dodgy hardware/reporting from smartd? Is there any advise on getting some debug or status information from NILFS to help show it isn't the cause of the problem. I would have expected that if it went bang I'd have seen something 'worrying' in the log. 

For information the hardware is a Dell Precision 380.

Many thanks
David Barham
Siemens PLM Software

Re: NILFS hanging SLES 11 - advise on diagnosis needed

[Ryusuke Konishi]

Hi,
On Fri, 2 Oct 2009 12:11:14 +0200, "Barham, David" wrote:
> Hi
> I'm running SLES 11, 2.6.27.19-5-default with NILFS2 nilfs-2.0.16. I have a 1.5Tb NILFS2 partition which I am setting up with the intention of using Robocopy from various PCs via samba. The robocopy scripts run nightly and a checkpoint is taken once night. A script stops samba, unmounts the previous weeks checkpoint, deletes the checkpoint, creates a new one and then mounts it and restarts samba. This should mean that at any time the user can go back to 'snapshot_{DAY}' to get their files back.
> 
> So far so good.
> 
> However as I copy the previously backed up files from the previous linux machine where I was doing this (only giving a 'current' copy with reiserfs). I'm finding that the new machine is occasionally hanging. The OS just locks up, screen on console frozen but host still responds to ping. 
> 
> I'm trying to work out what is causing the hang, I'm getting various messages in the log from smartd relating to the disk which houses the NILFS along the lines of:
> 
>  Oct  2 09:56:59 cpli6008 syslog-ng[1933]: Log statistics; dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0', processed='center(queued)=947', processed='center(received)=478', processed='destination(newsnotice)=0', processed='destination(acpid)=0', processed='destination(firewall)=0', processed='destination(mail)=12', processed='destination(mailinfo)=12', processed='destination(console)=151', processed='destination(newserr)=0', processed='destination(newscrit)=0', processed='destination(messages)=466', processed='destination(mailwarn)=0', processed='destination(localmessages)=0', processed='destination(netmgm)=0', processed='destination(mailerr)=0', processed='destination(xconsole)=151', processed='destination(warn)=155', processed='source(src)=478'
> Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 110 to 112
> Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 115 to 117
> Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Usage Attribute: 189 High_Fly_Writes changed from 88 to 87
> Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61
> Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 39
> Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Usage Attribute: 195 Hardware_ECC_Recovered changed from 50 to 51
> 
> {machine stops responding and gets power cycled}
>
> Oct  2 10:10:58 cpli6008 syslog-ng[1948]: syslog-ng starting up; version='2.0.9'
> 
> Do folks think that the hang is NILFS or dodgy hardware/reporting
> from smartd? Is there any advise on getting some debug or status
> information from NILFS to help show it isn't the cause of the
> problem. I would have expected that if it went bang I'd have seen
> something 'worrying' in the log.

The nilfs2 standalone module has a debug mode.  You can enable it by
commenting out the following line (i.e. CONFIG_NILFS_DEBUG=y) in
nilfs2-module/fs/Makefile before compiling:

ifndef CONFIG_NILFS
  EXTERNAL_BUILD=y
  CONFIG_NILFS=m
  # Uncomment below to do debug build.
  CONFIG_NILFS_DEBUG=y
  # Uncomment below to enable bmap validity check.
  #CONFIG_NILFS_BMAP_DEBUG=y
endif

By the way, I'm planning to release nilfs-2.0.17 tomorrow in order to
solve file system corruption problems which infrequently happen and
were reported on this list.

The bugfix was already merged in the mainline and also sent to -stable
trees for 3.6.30 and 3.6.31, but not yet done.

Your problem looks hardware problem to me, but I think the new version
is worth a try.

Cheers,
Ryusuke Konishi

Re: NILFS hanging SLES 11 - advise on diagnosis needed

[Fernando Jiménez Solano]

Ryusuke Konishi escribió:
>> Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 110 to 112
>
> Your problem looks hardware problem to me, but I think the new version
> is worth a try.
>
>   
I concur. Please, open the computer and touch the hard disk to check 
whether that reading is correct or not. If it is, your disk is dead or 
about to die.

Re: NILFS hanging SLES 11 - advise on diagnosis needed

[Barham, David]

I checked the hardware today as the machine keeps falling over, but only as I'm doing data copies into the nilfs2 filesystem via NFS. The last few times I've not had any errors in the logs at all except those the clearerd which is in debug mode and the startup after I power cycle the machine.

I tried to build a debug version of 2.0.17 but I'm getting 

/usr/local/sources/nilfs-2.0.17 # make
make -C fs
make[1]: Entering directory `/usr/local/sources/nilfs-2.0.17/fs'
make -C /lib/modules/2.6.27.19-5-default/build SUBDIRS=/usr/local/sources/nilfs-2.0.17/fs BUILD_DIR=/usr/local/sources/nilfs-2.0.17/fs modules
make[2]: Entering directory `/usr/src/linux-2.6.27.19-5-obj/x86_64/default'
make -C ../../../linux-2.6.27.19-5 O=/usr/src/linux-2.6.27.19-5-obj/x86_64/default/. modules
  CC [M]  /usr/local/sources/nilfs-2.0.17/fs/debug.o
/usr/local/sources/nilfs-2.0.17/fs/debug.c: In function âsnprint_page_flagsâ:
/usr/local/sources/nilfs-2.0.17/fs/debug.c:408: error: implicit declaration of function âPagePinnedâ
make[5]: *** [/usr/local/sources/nilfs-2.0.17/fs/debug.o] Error 1
make[4]: *** [_module_/usr/local/sources/nilfs-2.0.17/fs] Error 2
make[3]: *** [sub-make] Error 2
make[2]: *** [all] Error 2
make[2]: Leaving directory `/usr/src/linux-2.6.27.19-5-obj/x86_64/default'
make[1]: *** [default] Error 2
make[1]: Leaving directory `/usr/local/sources/nilfs-2.0.17/fs'
make: *** [fs] Error 2

I get the same error trying to make 2.0.16.

I've had a look at the C but I'm out of my depth.

Any help would be great.

Thanks
David Barham
Siemens PLM Software


-----Original Message-----
From: Ryusuke Konishi [mailto:ryusuke-sG5X7nlA6pw@public.gmane.org] 
Sent: 02 October 2009 12:42
To:; Barham, David
Subject: Re: [NILFS users] NILFS hanging SLES 11 - advise on diagnosis needed

Hi,
On Fri, 2 Oct 2009 12:11:14 +0200, "Barham, David" wrote:
> Hi
> I'm running SLES 11, 2.6.27.19-5-default with NILFS2 nilfs-2.0.16. I have a 1.5Tb NILFS2 partition which I am setting up with the intention of using Robocopy from various PCs via samba. The robocopy scripts run nightly and a checkpoint is taken once night. A script stops samba, unmounts the previous weeks checkpoint, deletes the checkpoint, creates a new one and then mounts it and restarts samba. This should mean that at any time the user can go back to 'snapshot_{DAY}' to get their files back.
> 
> So far so good.
> 
> However as I copy the previously backed up files from the previous linux machine where I was doing this (only giving a 'current' copy with reiserfs). I'm finding that the new machine is occasionally hanging. The OS just locks up, screen on console frozen but host still responds to ping. 
> 
> I'm trying to work out what is causing the hang, I'm getting various messages in the log from smartd relating to the disk which houses the NILFS along the lines of:
> 
>  Oct  2 09:56:59 cpli6008 syslog-ng[1933]: Log statistics; dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0', processed='center(queued)=947', processed='center(received)=478', processed='destination(newsnotice)=0', processed='destination(acpid)=0', processed='destination(firewall)=0', processed='destination(mail)=12', processed='destination(mailinfo)=12', processed='destination(console)=151', processed='destination(newserr)=0', processed='destination(newscrit)=0', processed='destination(messages)=466', processed='destination(mailwarn)=0', processed='destination(localmessages)=0', processed='destination(netmgm)=0', processed='destination(mailerr)=0', processed='destination(xconsole)=151', processed='destination(warn)=155', processed='source(src)=478'
> Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 110 to 112
> Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 115 to 117
> Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Usage Attribute: 189 High_Fly_Writes changed from 88 to 87
> Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61
> Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 39
> Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Usage Attribute: 195 Hardware_ECC_Recovered changed from 50 to 51
> 
> {machine stops responding and gets power cycled}
>
> Oct  2 10:10:58 cpli6008 syslog-ng[1948]: syslog-ng starting up; version='2.0.9'
> 
> Do folks think that the hang is NILFS or dodgy hardware/reporting
> from smartd? Is there any advise on getting some debug or status
> information from NILFS to help show it isn't the cause of the
> problem. I would have expected that if it went bang I'd have seen
> something 'worrying' in the log.

The nilfs2 standalone module has a debug mode.  You can enable it by
commenting out the following line (i.e. CONFIG_NILFS_DEBUG=y) in
nilfs2-module/fs/Makefile before compiling:

ifndef CONFIG_NILFS
  EXTERNAL_BUILD=y
  CONFIG_NILFS=m
  # Uncomment below to do debug build.
  CONFIG_NILFS_DEBUG=y
  # Uncomment below to enable bmap validity check.
  #CONFIG_NILFS_BMAP_DEBUG=y
endif

By the way, I'm planning to release nilfs-2.0.17 tomorrow in order to
solve file system corruption problems which infrequently happen and
were reported on this list.

The bugfix was already merged in the mainline and also sent to -stable
trees for 3.6.30 and 3.6.31, but not yet done.

Your problem looks hardware problem to me, but I think the new version
is worth a try.

Cheers,
Ryusuke Konishi

Re: NILFS hanging SLES 11 - advise on diagnosis needed

[Barham, David]

I managed to get the debug option to compile but only after I commented out the lines in debug.c

/*#if (LINUX_VERSION_CODE > KERNEL_VERSION(2, 6, 22))
        TEST_PAGE_FLAG(page, Pinned, b, size, n, len);
        TEST_PAGE_FLAG(page, Readahead, b, size, n, len);
#endif
/*

Is there perhaps something different about 2.6.27?

Do I know need to update the mount options with a 'v' ?

Thanks
David


-----Original Message-----
From: users-bounces-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org [mailto:users-bounces-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org] On Behalf Of Barham, David
Sent: 06 October 2009 15:24
To: users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
Subject: Re: [NILFS users] NILFS hanging SLES 11 - advise on diagnosis needed

I checked the hardware today as the machine keeps falling over, but only as I'm doing data copies into the nilfs2 filesystem via NFS. The last few times I've not had any errors in the logs at all except those the clearerd which is in debug mode and the startup after I power cycle the machine.

I tried to build a debug version of 2.0.17 but I'm getting 

/usr/local/sources/nilfs-2.0.17 # make
make -C fs
make[1]: Entering directory `/usr/local/sources/nilfs-2.0.17/fs'
make -C /lib/modules/2.6.27.19-5-default/build SUBDIRS=/usr/local/sources/nilfs-2.0.17/fs BUILD_DIR=/usr/local/sources/nilfs-2.0.17/fs modules
make[2]: Entering directory `/usr/src/linux-2.6.27.19-5-obj/x86_64/default'
make -C ../../../linux-2.6.27.19-5 O=/usr/src/linux-2.6.27.19-5-obj/x86_64/default/. modules
  CC [M]  /usr/local/sources/nilfs-2.0.17/fs/debug.o
/usr/local/sources/nilfs-2.0.17/fs/debug.c: In function âsnprint_page_flagsâ:
/usr/local/sources/nilfs-2.0.17/fs/debug.c:408: error: implicit declaration of function âPagePinnedâ
make[5]: *** [/usr/local/sources/nilfs-2.0.17/fs/debug.o] Error 1
make[4]: *** [_module_/usr/local/sources/nilfs-2.0.17/fs] Error 2
make[3]: *** [sub-make] Error 2
make[2]: *** [all] Error 2
make[2]: Leaving directory `/usr/src/linux-2.6.27.19-5-obj/x86_64/default'
make[1]: *** [default] Error 2
make[1]: Leaving directory `/usr/local/sources/nilfs-2.0.17/fs'
make: *** [fs] Error 2

I get the same error trying to make 2.0.16.

I've had a look at the C but I'm out of my depth.

Any help would be great.

Thanks
David Barham
Siemens PLM Software


-----Original Message-----
From: Ryusuke Konishi [mailto:ryusuke-sG5X7nlA6pw@public.gmane.org] 
Sent: 02 October 2009 12:42
To:; Barham, David
Subject: Re: [NILFS users] NILFS hanging SLES 11 - advise on diagnosis needed

Hi,
On Fri, 2 Oct 2009 12:11:14 +0200, "Barham, David" wrote:
> Hi
> I'm running SLES 11, 2.6.27.19-5-default with NILFS2 nilfs-2.0.16. I have a 1.5Tb NILFS2 partition which I am setting up with the intention of using Robocopy from various PCs via samba. The robocopy scripts run nightly and a checkpoint is taken once night. A script stops samba, unmounts the previous weeks checkpoint, deletes the checkpoint, creates a new one and then mounts it and restarts samba. This should mean that at any time the user can go back to 'snapshot_{DAY}' to get their files back.
> 
> So far so good.
> 
> However as I copy the previously backed up files from the previous linux machine where I was doing this (only giving a 'current' copy with reiserfs). I'm finding that the new machine is occasionally hanging. The OS just locks up, screen on console frozen but host still responds to ping. 
> 
> I'm trying to work out what is causing the hang, I'm getting various messages in the log from smartd relating to the disk which houses the NILFS along the lines of:
> 
>  Oct  2 09:56:59 cpli6008 syslog-ng[1933]: Log statistics; dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0', processed='center(queued)=947', processed='center(received)=478', processed='destination(newsnotice)=0', processed='destination(acpid)=0', processed='destination(firewall)=0', processed='destination(mail)=12', processed='destination(mailinfo)=12', processed='destination(console)=151', processed='destination(newserr)=0', processed='destination(newscrit)=0', processed='destination(messages)=466', processed='destination(mailwarn)=0', processed='destination(localmessages)=0', processed='destination(netmgm)=0', processed='destination(mailerr)=0', processed='destination(xconsole)=151', processed='destination(warn)=155', processed='source(src)=478'
> Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 110 to 112
> Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 115 to 117
> Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Usage Attribute: 189 High_Fly_Writes changed from 88 to 87
> Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61
> Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 39
> Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Usage Attribute: 195 Hardware_ECC_Recovered changed from 50 to 51
> 
> {machine stops responding and gets power cycled}
>
> Oct  2 10:10:58 cpli6008 syslog-ng[1948]: syslog-ng starting up; version='2.0.9'
> 
> Do folks think that the hang is NILFS or dodgy hardware/reporting
> from smartd? Is there any advise on getting some debug or status
> information from NILFS to help show it isn't the cause of the
> problem. I would have expected that if it went bang I'd have seen
> something 'worrying' in the log.

The nilfs2 standalone module has a debug mode.  You can enable it by
commenting out the following line (i.e. CONFIG_NILFS_DEBUG=y) in
nilfs2-module/fs/Makefile before compiling:

ifndef CONFIG_NILFS
  EXTERNAL_BUILD=y
  CONFIG_NILFS=m
  # Uncomment below to do debug build.
  CONFIG_NILFS_DEBUG=y
  # Uncomment below to enable bmap validity check.
  #CONFIG_NILFS_BMAP_DEBUG=y
endif

By the way, I'm planning to release nilfs-2.0.17 tomorrow in order to
solve file system corruption problems which infrequently happen and
were reported on this list.

The bugfix was already merged in the mainline and also sent to -stable
trees for 3.6.30 and 3.6.31, but not yet done.

Your problem looks hardware problem to me, but I think the new version
is worth a try.

Cheers,
Ryusuke Konishi
_______________________________________________
users mailing list
users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
https://www.nilfs.org/mailman/listinfo/users

Re: NILFS space usage

[Barham, David]

As per my previous question...

I'm running SLES 11, 2.6.27.19-5-default with NILFS2 nilfs-2.0.16. I
have a 1.5Tb NILFS2 partition which I am setting up with the intention
of using Robocopy from various PCs via samba. The robocopy scripts run
nightly and a checkpoint is taken once night. A script stops samba,
unmounts the previous weeks checkpoint, deletes the checkpoint, creates
a new one and then mounts it and restarts samba. This should mean that
at any time the user can go back to 'snapshot_{DAY}' to get their files
back.

So far so good.

I'm really confused by the space utilisation I'm seeing and what
cleanerd actually does.

The drive is now up to 98% full but should only have about 500Gb on it.
I have removed the few checkpoint I made so no all there is are the
snapshots. Turning the option in nilfs_cleanerd.conf to debug shows.

Oct  2 16:37:44 cpli6008 nilfs_cleanerd[3611]: start
Oct  2 16:37:44 cpli6008 nilfs_cleanerd[3611]: ncleansegs = 13850
Oct  2 16:38:50 cpli6008 nilfs_cleanerd[3611]: 2 segments selected to be
cleaned

I just used the default options on the mkfs and haven't changed any
setting (other than debug) in nilfs_cleanerd.conf.

I'm guessing that my cpio -pdumv from a different machine into the nilfs
file system has resulted masses of snapshots and these are taking the
space.

Is there a way to clean up or change the preferences to create fewer
snapshots?



Many thanks
David Barham
Siemens PLM Software


_______________________________________________
users mailing list
users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
https://www.nilfs.org/mailman/listinfo/users

Re: NILFS space usage

[Fernando Jiménez Solano]

On Fri, Oct 02, 2009 at 06:20:41PM +0200, Barham, David wrote:
> I just used the default options on the mkfs and haven't changed any
> setting (other than debug) in nilfs_cleanerd.conf.

There's a parameter on nilfs_cleanerd.conf called protection_period.
It means checkpoints are kept intact at least that many seconds. You
may want to lower that. In addition, you can lower cleaning_interval
to make nilfs_cleanerd act more often, and raise nsegments_per_clean
to perform a more aggressive (and in my experience, CPU intensive)
cleaning.

Unless I'm doing insanely heavy writes or I have very little space
left, I find the defaults to work just fine, but yes, some kind of
autotuning feature in nilfs_cleanerd would be great.

Re: IOzone testing

[Reinoud Zandijk]

Hi folks,

On Sat, Sep 26, 2009 at 10:36:52AM -0400, Jeff Layton wrote:
> So, let me summarize what I think I should do to test nilfs with
> IOzone without the impact of the checkpoints and without filling
> up the space.

I've dug into the iozone code and found it littered with sync(2) calls to
flush the buffers. This is what creates the checkpoints. If you take a big
enough disc it wouldn't matter otherwise you'll get into dynamic behaviour
IOzone is not really build for.

With regards,
Reinoud