reiser4 resize

reiser4 resize

[Jack Byer]

Short summary: Will a resize program for reiser4 be available within the
next six months?

Long explanation:

I use a 3ware SATA raid card for the main storage on my home network. I
currently have 8 250 GB drives in a raid 5 + hot spare configuration. I
 chose this card because it allows for online capacity expansion. My
plan was to wait until a few generations of hard drives emerged then
upgrade them one at a time in cycles. This way my storage will
periodically expand without any major downtime.

When I first created the filesystem, there was a reiser4 resize program.
This is no longer the case.

I've began my next upgrade cycle with the purchase of two 750 GB drives.
I plan to buy one each month until all eight are replaced. Now I need to
make a decision. Do I backup my raid onto the new drives and reformat my
raid with another filesystem (xfs, reiser3, jfs), or do I put these new
drives into the array and assume that when the time comes to resize the
filesystem that a working program will exist?

Re: reiser4 resize

[Vladimir V. Saveliev]

Hello

On Tuesday 19 September 2006 05:12, Jack Byer wrote:
> Short summary: Will a resize program for reiser4 be available within the
> next six months?
> 

Currently nobody works on that. So, I guess it is not very likely that reiser4.resize will be created within next six months.

> Long explanation:
> 
> I use a 3ware SATA raid card for the main storage on my home network. I
> currently have 8 250 GB drives in a raid 5 + hot spare configuration. I
>  chose this card because it allows for online capacity expansion. My
> plan was to wait until a few generations of hard drives emerged then
> upgrade them one at a time in cycles. This way my storage will
> periodically expand without any major downtime.
> 
> When I first created the filesystem, there was a reiser4 resize program.
> This is no longer the case.

that was not a working program.

> 
> I've began my next upgrade cycle with the purchase of two 750 GB drives.
> I plan to buy one each month until all eight are replaced. Now I need to
> make a decision. Do I backup my raid onto the new drives and reformat my
> raid with another filesystem (xfs, reiser3, jfs), or do I put these new
> drives into the array and assume that when the time comes to resize the
> filesystem that a working program will exist?
> 
I think you should change to a filesystem which has resize.

Re: reiser4 resize

[David Masover]

Vladimir V. Saveliev wrote:
> Hello
> 
> On Tuesday 19 September 2006 05:12, Jack Byer wrote:
>> Short summary: Will a resize program for reiser4 be available within the
>> next six months?
>>
> 
> Currently nobody works on that. So, I guess it is not very likely that reiser4.resize will be created within next six months.

Not even an expand?  I know a shrink depends on a working repacker (even 
an offline one), but I'd think expanding it would be simple enough, so 
long as there's a big warning of "You cannot undo this (can't shrink)!"

>> When I first created the filesystem, there was a reiser4 resize program.
>> This is no longer the case.
> 
> that was not a working program.

Yes, I remember that, it was a stub.

> I think you should change to a filesystem which has resize.

Alternately, how much would it cost to implement basic resizefs.reiser4?

There are other reasons that make me wish I'd stayed away from reiser4 
for awhile.  Mainly, right now, I need a repacker, and the system seems 
to have become absurdly slow when it's fragmented.  When I have over a 
gig of RAM free (not even buffer/cache, but _free_), and am trying to 
download anything over BitTorrent, even if it's less than 200 megs, the 
disk thrashes so badly that the system is really only usable for web and 
email.  Even movies will occasionally stall when this is happening, and 
by "occasionally", I mean every minute or so.

I believe there was a patch to address the thrashing, so I'm eagerly 
awaiting 2.6.18, but the lack of a repacker bothers me.

Re: reiser4 resize

[Jack Byer]

> that was not a working program.
> 
I never looked too far into that issue, because I didn't need it back then.

> I think you should change to a filesystem which has resize.
> 

I guess this means that I'll won't use reiser4 again until 2 TB drives
come out and I upgrade. Maybe by then reiser4 will be included in the
kernel and files-as-directory will work :)

Re: reiser4 resize

[Alexey Polyakov]

On 9/19/06, David Masover <ninja@slaphack.com> wrote:

> When I have over a
> gig of RAM free (not even buffer/cache, but _free_), and am trying to
> download anything over BitTorrent, even if it's less than 200 megs, the
> disk thrashes so badly that the system is really only usable for web and
> email.  Even movies will occasionally stall when this is happening, and
> by "occasionally", I mean every minute or so.

Do you have this problem on plain vanilla + reiser4? I think some
out-of-tree patches (like adaptive readahead?) may result in such
behaviour, independent of file system.
With a free gig of RAM reiser4 shouldn't access disk a lot, only during flushes.

-- 
Alexey Polyakov

Re: reiser4 resize

[Alexey Polyakov]

On 9/20/06, £ukasz Mierzwa <prymitive@pcserwis.net> wrote:

> It's been proven that flushes are doing much more job then they should.
> Not so long ago someone send a trace of block device io accesess during
> reiser4 work and someone anylized it and said that some files or parts of
> file where written over and over 200 times or so.  Bittottent downloads on
> a few months old filesystem that had been used often just shows a week
> spot in reiser4, while downloading files with azureus with only 64KB of
> data per second I got disk lid on almost all the time, swithcing to
> rtorrent helped as it does not seem to call fsync ( I think I disabled
> fsync in azureus).

Ah, I see, if bittorrent calls fsync often, it's no wonder that
reiser4 behaves badly. I had to preload libnosync for some of my
programs that do fsync to avoid this.

-- 
Alexey Polyakov

Re: reiser4 resize

[David Masover]

Alexey Polyakov wrote:
> On 9/20/06, £ukasz Mierzwa <prymitive@pcserwis.net> wrote:
> 
>> It's been proven that flushes are doing much more job then they should.
>> Not so long ago someone send a trace of block device io accesess during
>> reiser4 work and someone anylized it and said that some files or parts of
>> file where written over and over 200 times or so.

Wow.  I should go back and read that -- I assume this is being worked on?

>> a few months old filesystem that had been used often just shows a week
>> spot in reiser4, while downloading files with azureus with only 64KB of
>> data per second I got disk lid on almost all the time, swithcing to
>> rtorrent helped as it does not seem to call fsync ( I think I disabled
>> fsync in azureus).

Hmm, strange.  I am using Azureus, but I don't think it's fsync.  I can 
try rtorrent, but there are several things I like about Azureus that 
nothing else seems to do yet.

But also, Azureus didn't always do this.  In fact, I used it for several 
months before I started having this problem.

> Ah, I see, if bittorrent calls fsync often, it's no wonder that
> reiser4 behaves badly. I had to preload libnosync for some of my
> programs that do fsync to avoid this.

Way ahead of you.  I noticed how much fsync performance sucked when 
using vim, and I was sick of waiting 10 seconds every time I hit :w -- a 
LOT of stuff can pile up in 2 gigs of disk buffer, and at the time, 
Reiser4 fsync effectively just called sync.

I didn't know about libnosync (or it didn't exist yet, or didn't work, 
I'm not entirely sure), so I was faced with either patching vim, which 
had just been patched to _add_ fsync'ing, not to mention all the other 
programs that might fsync too much;  patching glibc (huge, I don't 
update it often, and I'd have no idea where to start);  or patching the 
kernel.

I now keep backups, and I maintain and apply the following (STUPID, 
DON'T TRY THIS AT HOME) patch to my kernel:

--- linux/fs/buffer.c   2006-08-15 20:40:36.504608696 -0500
+++ linux/fs/buffer.c.new       2006-08-15 20:42:35.877461264 -0500
@@ -366,12 +366,12 @@

  asmlinkage long sys_fsync(unsigned int fd)
  {
-       return __do_fsync(fd, 0);
+       return 0;
  }

  asmlinkage long sys_fdatasync(unsigned int fd)
  {
-       return __do_fsync(fd, 1);
+       return 0;
  }

  /*

Re: reiser4 resize

[David Masover]

Alexey Polyakov wrote:
> On 9/19/06, David Masover <ninja@slaphack.com> wrote:
> 
>> When I have over a
>> gig of RAM free (not even buffer/cache, but _free_), and am trying to
>> download anything over BitTorrent, even if it's less than 200 megs, the
>> disk thrashes so badly that the system is really only usable for web and
>> email.  Even movies will occasionally stall when this is happening, and
>> by "occasionally", I mean every minute or so.
> 
> Do you have this problem on plain vanilla + reiser4?

Yes.

Well, no.  My kernel is:

vanilla 2.6.17.13 on amd64

patches:
sk98lin 8.36, latest from the manufacturer
reiser4-for-2.6.17-3
my own patch that disables fsync and fdatasync

external modules, installed via Portage:
ALSA 1.0.11 driver, using snd_emu10k1 and all sorts of support stuff 
(OSS emulation, synth, etc)
nvidia driver, 1.0.8762

I've also been having a bit of instability issues, but only very rarely 
do these seem at all FS-related.  I'm overclocked a bit, and I can 
reliably crash my system by playing Neverball, Doom 3, or Quake 4 for 
several hours.  I strongly suspect this is either my overclocking or the 
nvidia drivers here.

However, I doubt anything I've done beyond vanilla+reiser4 is affecting 
this disk access issue, and I'm pretty much rock solid when I'm not 
playing a game.  I also have a close-to-identical machine nearby which 
is not overclocked, same kernel, same modules, everything except the 
nvidia driver, been rock solid for a year, no performance issues to 
speak of.  The main difference, other than graphics, is that the stable 
machine is using 21 gigs out of 72, whereas the unstable one (the one 
that's sluggish for BitTorrent) is using 279 gigs out of 350, and has 
been up to 320 or 330 at least before I started cleaning things out.

So I think we're down to two possibilities:  Either an update to Azureus 
has found a way to sync that I'm not aware of, or this is the behavior 
someone described where Reiser4 will attempt to find contiguous space to 
allocate, and continue searching and re-searching the same areas of the 
disk almost every write.  To be honest, I hope it's about syncing, 
somehow, because I'd much rather believe my disk isn't horrendously 
fragmented...

Re: reiser4 resize

[Łukasz Mierzwa]

Dnia Thu, 21 Sep 2006 09:30:09 +0200, David Masover <ninja@slaphack.com>  
napisa³:

> --- linux/fs/buffer.c   2006-08-15 20:40:36.504608696 -0500
> +++ linux/fs/buffer.c.new       2006-08-15 20:42:35.877461264 -0500
> @@ -366,12 +366,12 @@
>
>   asmlinkage long sys_fsync(unsigned int fd)
>   {
> -       return __do_fsync(fd, 0);
> +       return 0;
>   }
>
>   asmlinkage long sys_fdatasync(unsigned int fd)
>   {
> -       return __do_fsync(fd, 1);
> +       return 0;
>   }
>
>   /*

I remember that I played a little with disabling sync in reiser4 sources,  
it helped only for amarok (it's uses sqlite for storing statistic data and  
it writes to it on song change, sqlite calls sync and it ends up with  
writeing to disk instead of playing a song, at least on my fs),  
bittorrents clients were generating as lot of disk as previously so I gues  
it's more likely coused by data/tree fragmentation.
Just my 0,6374526$ ;)

Re: reiser4 resize

[Łukasz Mierzwa]

Dnia Thu, 21 Sep 2006 09:30:09 +0200, David Masover <ninja@slaphack.com>  
napisa³:

> On 9/20/06, £ukasz Mierzwa <prymitive@pcserwis.net> wrote:
>  It's been proven that flushes are doing much more job then they should.
> Not so long ago someone send a trace of block device io accesess during
> reiser4 work and someone anylized it and said that some files or parts of
> file where written over and over 200 times or so.
>  Wow.  I should go back and read that -- I assume this is being worked  
> on?

Well it was not a file but a block, but the effect is the same.

http://marc.theaimsgroup.com/?l=reiserfs&m=115488109712570&w=2

Reiser4 resize

[Oleg Osovitskiy]

Hello!

  Does Reiser4 fs support resize as reiserfs ?
If yes which tool I have to use to enlarge my reiser4 fs ?

---
Best regards, Oleg O. Osovitskiy
mailto:o.osovitskiy@rambler.ru, icq# 33366588

Re: Reiser4 resize

[Edward Shishkin]

Oleg Osovitskiy wrote:

>Hello!
>  
>

Hi

>  Does Reiser4 fs support resize as reiserfs ?
>  
>

nop, sorry..

>If yes which tool I have to use to enlarge my reiser4 fs ?
>
>---
>Best regards, Oleg O. Osovitskiy
>mailto:o.osovitskiy@rambler.ru, icq# 33366588
>
>-
>To unsubscribe from this list: send the line "unsubscribe reiserfs-devel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>  
>

No space left on rfs4

[John]

Hello people,

I've been following this list for some time but this is my first e-mail here since at least 2 years.
(thank you Edwards for your work, you've fixed many of my issues lately)


Today I am mailing you about an issue that you probably know about, but that I still believe needs to be fixed: filling the FS till it corrupts itself.

How many times have I almost killed by computer by filling my rfs4, copying data on the wrong partition or doing something else...

The FS needs to be able to handle that, but instead some files get corrupted and when I try to delete them my computer is kind of dead and needs to reboot.  Of course, I cannot reboot and have to shut it down myself which might corrupt even more data... If I tried to delete some files after filling the partition, at the reboot they're back and so I'm again with a couple of Kb free. Usually a fsck fixes this problem, but when I do that on / the fsck may not be usable anymore...

Don't we have like 5% of space blocked for this case like on other FS? or something else that would prevent this problem from happening?


I am using the latest rfs4 patch from here with a 2.6.24 kernel and a ccreg40 partition, but this issue has been here for years so nothing new I believe.

Thank you,

John

Re: No space left on rfs4

[Edward Shishkin]

Hello.

How long ago did it happen?
I can not reproduce such problems, so let's start with full bugreport.
Root partition can be checked after booting with gentoo-based live
CD which contains  reiser4progs-1.0.6

Thanks,
Edward.

John wrote:

>Hello people,
>
>I've been following this list for some time but this is my first e-mail here since at least 2 years.
>(thank you Edwards for your work, you've fixed many of my issues lately)
>
>
>Today I am mailing you about an issue that you probably know about, but that I still believe needs to be fixed: filling the FS till it corrupts itself.
>
>How many times have I almost killed by computer by filling my rfs4, copying data on the wrong partition or doing something else...
>
>The FS needs to be able to handle that, but instead some files get corrupted and when I try to delete them my computer is kind of dead and needs to reboot.  Of course, I cannot reboot and have to shut it down myself which might corrupt even more data... If I tried to delete some files after filling the partition, at the reboot they're back and so I'm again with a couple of Kb free. Usually a fsck fixes this problem, but when I do that on / the fsck may not be usable anymore...
>
>Don't we have like 5% of space blocked for this case like on other FS? or something else that would prevent this problem from happening?
>
>
>I am using the latest rfs4 patch from here with a 2.6.24 kernel and a ccreg40 partition, but this issue has been here for years so nothing new I believe.
>
>Thank you,
>
>John
>
>--
>To unsubscribe from this list: send the line "unsubscribe reiserfs-devel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>  
>

Re: No space left on rfs4

[John]

Hello,

Last time, it happened 2 days ago.

What would be a full bugreport? I do not really know what you mean by that. Do you want me to do it again and to send you the error message or something?

As for the live-cd yes I know, but this computer does not have a CD any-more so my only way out is my other /. But yes there is always a way to fix it, it is just bothersome...


Thinking about it those partitions were created with reiser4progs-1.0.5 if I remember correctly. Maybe that would make a difference?

Thank you,

John

On Sun, 02 Mar 2008 00:27:33 +0300, Edward Shishkin <edward.shishkin@gmail.com> wrote:
> Hello.
> 
> How long ago did it happen?
> I can not reproduce such problems, so let's start with full bugreport.
> Root partition can be checked after booting with gentoo-based live
> CD which contains  reiser4progs-1.0.6
> 
> Thanks,
> Edward.
> 
> John wrote:
> 
>>Hello people,
>>
>>I've been following this list for some time but this is my first e-mail
> here since at least 2 years.
>>(thank you Edwards for your work, you've fixed many of my issues lately)
>>
>>
>>Today I am mailing you about an issue that you probably know about, but
> that I still believe needs to be fixed: filling the FS till it corrupts
> itself.
>>
>>How many times have I almost killed by computer by filling my rfs4,
> copying data on the wrong partition or doing something else...
>>
>>The FS needs to be able to handle that, but instead some files get
> corrupted and when I try to delete them my computer is kind of dead and
> needs to reboot.  Of course, I cannot reboot and have to shut it down
> myself which might corrupt even more data... If I tried to delete some
> files after filling the partition, at the reboot they're back and so I'm
> again with a couple of Kb free. Usually a fsck fixes this problem, but when
> I do that on / the fsck may not be usable anymore...
>>
>>Don't we have like 5% of space blocked for this case like on other FS? or
> something else that would prevent this problem from happening?
>>
>>
>>I am using the latest rfs4 patch from here with a 2.6.24 kernel and a
> ccreg40 partition, but this issue has been here for years so nothing new I
> believe.
>>
>>Thank you,
>>
>>John
>>
>>--
>>To unsubscribe from this list: send the line "unsubscribe reiserfs-devel"
> in
>>the body of a message to majordomo@vger.kernel.org
>>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>>
> 
> --
> To unsubscribe from this list: send the line "unsubscribe reiserfs-devel"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: No space left on rfs4

[Edward Shishkin]

John wrote:

>Hello,
>
>Last time, it happened 2 days ago.
>  
>

So it was with 2.6.24, right?

>What would be a full bugreport? 
>

What exactly happens when you get no space left on device?
Is it a kernel oops, or just a system a freeze?
Are there any related kernel messages?
What kind of workload do you use to exhaust disk space?
What exactly happens when you try to delete files on the
device with no space left?
Is your device formatted with reg40 (default), or ccreg40 (I am
a bit confused: reiser4progs-1.0.5 are not aware of ccreg40).

Thanks,
Edward.

>I do not really know what you mean by that. Do you want me to do it again and to send you the error message or something?
>
>As for the live-cd yes I know, but this computer does not have a CD any-more so my only way out is my other /. But yes there is always a way to fix it, it is just bothersome...
>
>
>Thinking about it those partitions were created with reiser4progs-1.0.5 if I remember correctly. Maybe that would make a difference?
>
>Thank you,
>
>John
>
>On Sun, 02 Mar 2008 00:27:33 +0300, Edward Shishkin <edward.shishkin@gmail.com> wrote:
>  
>
>>Hello.
>>
>>How long ago did it happen?
>>I can not reproduce such problems, so let's start with full bugreport.
>>Root partition can be checked after booting with gentoo-based live
>>CD which contains  reiser4progs-1.0.6
>>
>>Thanks,
>>Edward.
>>
>>John wrote:
>>
>>    
>>
>>>Hello people,
>>>
>>>I've been following this list for some time but this is my first e-mail
>>>      
>>>
>>here since at least 2 years.
>>    
>>
>>>(thank you Edwards for your work, you've fixed many of my issues lately)
>>>
>>>
>>>Today I am mailing you about an issue that you probably know about, but
>>>      
>>>
>>that I still believe needs to be fixed: filling the FS till it corrupts
>>itself.
>>    
>>
>>>How many times have I almost killed by computer by filling my rfs4,
>>>      
>>>
>>copying data on the wrong partition or doing something else...
>>    
>>
>>>The FS needs to be able to handle that, but instead some files get
>>>      
>>>
>>corrupted and when I try to delete them my computer is kind of dead and
>>needs to reboot.  Of course, I cannot reboot and have to shut it down
>>myself which might corrupt even more data... If I tried to delete some
>>files after filling the partition, at the reboot they're back and so I'm
>>again with a couple of Kb free. Usually a fsck fixes this problem, but when
>>I do that on / the fsck may not be usable anymore...
>>    
>>
>>>Don't we have like 5% of space blocked for this case like on other FS? or
>>>      
>>>
>>something else that would prevent this problem from happening?
>>    
>>
>>>I am using the latest rfs4 patch from here with a 2.6.24 kernel and a
>>>      
>>>
>>ccreg40 partition, but this issue has been here for years so nothing new I
>>believe.
>>    
>>
>>>Thank you,
>>>
>>>John
>>>
>>>--
>>>To unsubscribe from this list: send the line "unsubscribe reiserfs-devel"
>>>      
>>>
>>in
>>    
>>
>>>the body of a message to majordomo@vger.kernel.org
>>>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>>
>>>      
>>>
>>--
>>To unsubscribe from this list: send the line "unsubscribe reiserfs-devel"
>>in
>>the body of a message to majordomo@vger.kernel.org
>>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>    
>>
>
>
>  
>

Re: No space left on rfs4

[John]

On Sun, 02 Mar 2008 01:54:02 +0300, Edward Shishkin <edward.shishkin@gmail.com> wrote:
> John wrote:
> 
>>Hello,
>>
>>Last time, it happened 2 days ago.
>>
>>
> 
> So it was with 2.6.24, right?
Yep.
Also happened with 2.6.23 a few weeks ago.
> 
>>What would be a full bugreport?
>>
> 
> What exactly happens when you get no space left on device?
> Is it a kernel oops, or just a system a freeze?
> Are there any related kernel messages?
I'm getting an error message about it, and that is about all.
The system can still be used if it is not the / partition.

> What kind of workload do you use to exhaust disk space?
Just extracting files from a tar.bz2 to the wrong partition.

> What exactly happens when you try to delete files on the
> device with no space left?
The issue starts when I try to remove the files that caused that problem.
I'm guessing that the files freezing my shell when I try to delete them are the ones that could not be fully copied: some files can be removed without any issue, and others just freeze my shell (not my system).
Once my shell has been frozen rebooting will not be possible. Probably the partition cannot be unmounted or something like that.

> Is your device formatted with reg40 (default), or ccreg40 (I am
> a bit confused: reiser4progs-1.0.5 are not aware of ccreg40).
Right, last time it happened with a reg40 partition.
Before that I thought it was with a ccreg40 but maybe not. I ain't sure anymore.
For now my / is ccreg40 and my /home is reg40. The issue this week came with /home being full.


Would you like me to recreate that partition with 1.0.6?
Actually I have another partition created this week with 1.0.6 in ccreg40, I can easily fill it and see what happens then.



Thank you,

John

Re: No space left on rfs4

[Edward Shishkin]

Hello.

Yeah, indeed, I have reproduced it for reg40 (default plugin):
tar process is in permanent "+D" state. After reboot all files
were successfully deleted, although there is some leak of free
disk space there.

Ok, I'll take a look at this more carefully (I guess -ENOSPC
error is handled incorrectly somewhere).

If you have a problems with deleting files on reg40 partition,
then please pack your metadata by
debugfs.reiser4 -P /dev/xxx | gzip > meta.gz
and let me download the file meta.gz

I don' t see such problems with ccreg40 (compression plugin).
Please, let me know, if something goes wrong here..

Thanks!
Edward.

John wrote:

>On Sun, 02 Mar 2008 01:54:02 +0300, Edward Shishkin <edward.shishkin@gmail.com> wrote:
>  
>
>>John wrote:
>>
>>    
>>
>>>Hello,
>>>
>>>Last time, it happened 2 days ago.
>>>
>>>
>>>      
>>>
>>So it was with 2.6.24, right?
>>    
>>
>Yep.
>Also happened with 2.6.23 a few weeks ago.
>  
>
>>>What would be a full bugreport?
>>>
>>>      
>>>
>>What exactly happens when you get no space left on device?
>>Is it a kernel oops, or just a system a freeze?
>>Are there any related kernel messages?
>>    
>>
>I'm getting an error message about it, and that is about all.
>The system can still be used if it is not the / partition.
>
>  
>
>>What kind of workload do you use to exhaust disk space?
>>    
>>
>Just extracting files from a tar.bz2 to the wrong partition.
>
>  
>
>>What exactly happens when you try to delete files on the
>>device with no space left?
>>    
>>
>The issue starts when I try to remove the files that caused that problem.
>I'm guessing that the files freezing my shell when I try to delete them are the ones that could not be fully copied: some files can be removed without any issue, and others just freeze my shell (not my system).
>Once my shell has been frozen rebooting will not be possible. Probably the partition cannot be unmounted or something like that.
>
>  
>
>>Is your device formatted with reg40 (default), or ccreg40 (I am
>>a bit confused: reiser4progs-1.0.5 are not aware of ccreg40).
>>    
>>
>Right, last time it happened with a reg40 partition.
>Before that I thought it was with a ccreg40 but maybe not. I ain't sure anymore.
>For now my / is ccreg40 and my /home is reg40. The issue this week came with /home being full.
>
>
>Would you like me to recreate that partition with 1.0.6?
>Actually I have another partition created this week with 1.0.6 in ccreg40, I can easily fill it and see what happens then.
>
>
>
>Thank you,
>
>John
>
>
>  
>

Re: No space left on rfs4

[John]

On Sun, 02 Mar 2008 13:17:12 +0300, Edward Shishkin <edward.shishkin@gmail.com> wrote:
> Hello.
> 
> Yeah, indeed, I have reproduced it for reg40 (default plugin):
> tar process is in permanent "+D" state. After reboot all files
> were successfully deleted, although there is some leak of free
> disk space there.
> 
> Ok, I'll take a look at this more carefully (I guess -ENOSPC
> error is handled incorrectly somewhere).
> 
> If you have a problems with deleting files on reg40 partition,
> then please pack your metadata by
> debugfs.reiser4 -P /dev/xxx | gzip > meta.gz
> and let me download the file meta.gz
My FS is currently no full and with no bug, so I hope these are the metadata you wanted.
If not I'll fill it again and send it to you again

http://www.megaupload.com/?d=XVITV1CU

(sorry I cannot send a file that big by e-mail)
> 
> I don' t see such problems with ccreg40 (compression plugin).
> Please, let me know, if something goes wrong here..
I just tried and had the same issue with a ccreg40 partition.
I could not remove all the files, and when I tried it just froze my shell.
After killing my shell I was able to reboot but not to umount the partition.
This is the debugfs output:

http://www.megaupload.com/?d=CBEGIFDL


I also wanted to join an fsck output but I don't know how to do that.
Basically the results were in checking the semantic tree
FSCK: obj40_repair.c 223: obj40_stat_unix_check: Node (XYZ), item (a), [fgfdgf:fgfdggdf:fhgfhgf] (stat40): wrong bytes (ABCDE), fixed to (EFC).

(obviously this is some sort of templated message, they were all like that...)


Thank you,

John

Re: No space left on rfs4

[Christopher Sawtell]

My own experience with RR4 when the filesystem is full, is that it
works perfectly if only a very few files are being written to, as in
/var/tmp/ overflowing when it's being used to build a package in
Gentoo, but that if there are many files open, as in an rsync session,
or a single file being written to in many places, as in the bittorrent
situation, then major breakage occurs. Perhaps I should say occurred
because the overfill of the RR4 partition during a Gentoo package
build happened 3 days ago, whereas the others many months.



-- 
Sincerely etc.
Christopher Sawtell

Re: Reiser4 resize

[John]

Edward Shishkin wrote:
>Oleg Osovitskiy wrote:

>>Hello!
>>  
>>

>Hi

>>  Does Reiser4 fs support resize as reiserfs ?
>>  
>>

>nop, sorry..

>>If yes which tool I have to use to enlarge my reiser4 fs ?
>>
>>---
>>Best regards, Oleg O. Osovitskiy
>>mailto:o.osovitskiy@rambler.ru, icq# 33366588


Hi there,

While playing with LVM today I was wondering about that too.
How hard would that be?
(What will follow are just assumptions from my memories of writing FS, so sorry if I'm way off with R4)

- Growing should not be too hard as only the main table would need to be changed no?
- As for shrinking I see 2 cases:
   a- There is no data beyond my desired new end, it looks like I only need to modify the table
   b- There is some data bothering us, now we need to pull it the same way as a repacker would do I believe.


The first 2 cases look easy. For the last it would need something close to a defrag.

So again, how hard would it be to write a dirty resizer? I have no Linux or R4 dev knowledge, but I have a lot spare time and I'm willing to put some of my data at risk. C is not an issue either. Do you think I can try something if you give me some pointers or am I just day-dreaming?

Thanks,

John

Re: No space left on rfs4

[Edward Shishkin]

[-- Attachment #1: Type: text/plain, Size: 2090 bytes --]

On 3/3/08, John <geearf@free.fr> wrote:
>
>
>  On Sun, 02 Mar 2008 13:17:12 +0300, Edward Shishkin <edward.shishkin@gmail.com> wrote:
>  > Hello.
>  >
>  > Yeah, indeed, I have reproduced it for reg40 (default plugin):
>  > tar process is in permanent "+D" state. After reboot all files
>  > were successfully deleted, although there is some leak of free
>  > disk space there.
>  >
>  > Ok, I'll take a look at this more carefully (I guess -ENOSPC
>  > error is handled incorrectly somewhere).
>  >
>  > If you have a problems with deleting files on reg40 partition,
>  > then please pack your metadata by
>  > debugfs.reiser4 -P /dev/xxx | gzip > meta.gz
>  > and let me download the file meta.gz
>
> My FS is currently no full and with no bug, so I hope these are the metadata you wanted.
>  If not I'll fill it again and send it to you again
>
>  http://www.megaupload.com/?d=XVITV1CU
>

Eventually I have not downloaded this:
It said "No free slots available for your country" ;)

Well, don't bother with this for a while:
I have caught a mutex leak in cryptcompress plugin,
(fixup is attached) this explains undeletable files in ccreg40.

I'll try fo fix unix-file plugin a bit later.

Thanks,
Edward.

>  (sorry I cannot send a file that big by e-mail)
>
> >
>  > I don' t see such problems with ccreg40 (compression plugin).
>  > Please, let me know, if something goes wrong here..
>
> I just tried and had the same issue with a ccreg40 partition.
>  I could not remove all the files, and when I tried it just froze my shell.
>  After killing my shell I was able to reboot but not to umount the partition.
>  This is the debugfs output:
>
>  http://www.megaupload.com/?d=CBEGIFDL
>
>
>  I also wanted to join an fsck output but I don't know how to do that.
>  Basically the results were in checking the semantic tree
>  FSCK: obj40_repair.c 223: obj40_stat_unix_check: Node (XYZ), item (a), [fgfdgf:fgfdggdf:fhgfhgf] (stat40): wrong bytes (ABCDE), fixed to (EFC).
>
>  (obviously this is some sort of templated message, they were all like that...)
>
>
>  Thank you,
>
>
>  John
>
>

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: reiser4-handle-enospc-fixup.patch --]
[-- Type: text/x-patch; name=reiser4-handle-enospc-fixup.patch, Size: 825 bytes --]

---
 linux-2.6.23-mm1/fs/reiser4/plugin/file/cryptcompress.c |    7 ++++---
 1 files changed, 4 insertions(+), 3 deletions(-)

--- linux-2.6.23-mm1/fs/reiser4/plugin/file/cryptcompress.c.orig
+++ linux-2.6.23-mm1/fs/reiser4/plugin/file/cryptcompress.c
@@ -2721,7 +2721,8 @@
 		if (result)
 			goto out;
 		if (cont->state == PSCHED_ASSIGNED_NEW)
-			goto out_no_release;
+			/* done_lh was called in write_pschedule_hook */
+			goto out_no_longterm_lock;
 
 		result = prepare_logical_cluster(inode, pos, count, &clust,
 						 LC_APPOV);
@@ -2793,9 +2794,9 @@
 	} while (count);
  out:
 	done_lh(&hint->lh);
-	mutex_unlock(&info->checkin_mutex);
 	save_file_hint(file, hint);
- out_no_release:
+ out_no_longterm_lock:
+	mutex_unlock(&info->checkin_mutex);
 	kfree(hint);
 	put_cluster_handle(&clust);
 	assert("edward-195",

Re: No space left on rfs4

[Edward Shishkin]

[-- Attachment #1: Type: text/plain, Size: 3847 bytes --]

Edward Shishkin wrote:

>On 3/3/08, John <geearf@free.fr> wrote:
>  
>
>> On Sun, 02 Mar 2008 13:17:12 +0300, Edward Shishkin <edward.shishkin@gmail.com> wrote:
>> > Hello.
>> >
>> > Yeah, indeed, I have reproduced it for reg40 (default plugin):
>> > tar process is in permanent "+D" state. After reboot all files
>> > were successfully deleted, although there is some leak of free
>> > disk space there.
>> >
>> > Ok, I'll take a look at this more carefully (I guess -ENOSPC
>> > error is handled incorrectly somewhere).
>> >
>> > If you have a problems with deleting files on reg40 partition,
>> > then please pack your metadata by
>> > debugfs.reiser4 -P /dev/xxx | gzip > meta.gz
>> > and let me download the file meta.gz
>>
>>My FS is currently no full and with no bug, so I hope these are the metadata you wanted.
>> If not I'll fill it again and send it to you again
>>
>> http://www.megaupload.com/?d=XVITV1CU
>>
>>    
>>
>
>Eventually I have not downloaded this:
>It said "No free slots available for your country" ;)
>
>Well, don't bother with this for a while:
>I have caught a mutex leak in cryptcompress plugin,
>(fixup is attached) this explains undeletable files in ccreg40.
>
>I'll try fo fix unix-file plugin a bit later.
>
>  
>

I have found some ancient bugs related to tail conversion.
In particular, it take place when application writes by chunks
< 20K in no-space-left-on-device situation (for example, tar,
which uses 10240 bytes chunks by default).
The bugs are:
. leak of per-inode exclusive access;
. leak of per-inode flag REISER4_PART_IN_CONV
All of them are responsible for reported deadlocks , and, perhaps
can lead to data corruptions.

The fixup is attached.

There still takes place a silent leak of free disk space, when
applications runs in no-space-left-on-device situation. This is also
related only to (default) unix-file plugin with (default) "smart"
formatting policy. Hope to address it soon..

Thanks for reports,
Edward.

>> (sorry I cannot send a file that big by e-mail)
>>
>>    
>>
>> > I don' t see such problems with ccreg40 (compression plugin).
>> > Please, let me know, if something goes wrong here..
>>
>>I just tried and had the same issue with a ccreg40 partition.
>> I could not remove all the files, and when I tried it just froze my shell.
>> After killing my shell I was able to reboot but not to umount the partition.
>> This is the debugfs output:
>>
>> http://www.megaupload.com/?d=CBEGIFDL
>>
>>
>> I also wanted to join an fsck output but I don't know how to do that.
>> Basically the results were in checking the semantic tree
>> FSCK: obj40_repair.c 223: obj40_stat_unix_check: Node (XYZ), item (a), [fgfdgf:fgfdggdf:fhgfhgf] (stat40): wrong bytes (ABCDE), fixed to (EFC).
>>
>> (obviously this is some sort of templated message, they were all like that...)
>>
>>
>> Thank you,
>>
>>
>> John
>>
>>
>>    
>>
>>------------------------------------------------------------------------
>>
>>---
>> linux-2.6.23-mm1/fs/reiser4/plugin/file/cryptcompress.c |    7 ++++---
>> 1 files changed, 4 insertions(+), 3 deletions(-)
>>
>>--- linux-2.6.23-mm1/fs/reiser4/plugin/file/cryptcompress.c.orig
>>+++ linux-2.6.23-mm1/fs/reiser4/plugin/file/cryptcompress.c
>>@@ -2721,7 +2721,8 @@
>> 		if (result)
>> 			goto out;
>> 		if (cont->state == PSCHED_ASSIGNED_NEW)
>>-			goto out_no_release;
>>+			/* done_lh was called in write_pschedule_hook */
>>+			goto out_no_longterm_lock;
>> 
>> 		result = prepare_logical_cluster(inode, pos, count, &clust,
>> 						 LC_APPOV);
>>@@ -2793,9 +2794,9 @@
>> 	} while (count);
>>  out:
>> 	done_lh(&hint->lh);
>>-	mutex_unlock(&info->checkin_mutex);
>> 	save_file_hint(file, hint);
>>- out_no_release:
>>+ out_no_longterm_lock:
>>+	mutex_unlock(&info->checkin_mutex);
>> 	kfree(hint);
>> 	put_cluster_handle(&clust);
>> 	assert("edward-195",
>>    
>>


[-- Attachment #2: reiser4-handle-enospc-fixup.patch --]
[-- Type: text/x-patch, Size: 5552 bytes --]

---
 linux-2.6.23-mm1/fs/reiser4/plugin/file/cryptcompress.c   |    7 +--
 linux-2.6.23-mm1/fs/reiser4/plugin/file/file.c            |    8 ++-
 linux-2.6.23-mm1/fs/reiser4/plugin/file/tail_conversion.c |   29 +++++++++-----
 linux-2.6.23-mm1/fs/reiser4/plugin/item/internal.c        |   18 ++++++--
 4 files changed, 43 insertions(+), 19 deletions(-)

--- linux-2.6.23-mm1/fs/reiser4/plugin/file/cryptcompress.c.orig
+++ linux-2.6.23-mm1/fs/reiser4/plugin/file/cryptcompress.c
@@ -2721,7 +2721,8 @@
 		if (result)
 			goto out;
 		if (cont->state == PSCHED_ASSIGNED_NEW)
-			goto out_no_release;
+			/* done_lh was called in write_pschedule_hook */
+			goto out_no_longterm_lock;
 
 		result = prepare_logical_cluster(inode, pos, count, &clust,
 						 LC_APPOV);
@@ -2793,9 +2794,9 @@
 	} while (count);
  out:
 	done_lh(&hint->lh);
-	mutex_unlock(&info->checkin_mutex);
 	save_file_hint(file, hint);
- out_no_release:
+ out_no_longterm_lock:
+	mutex_unlock(&info->checkin_mutex);
 	kfree(hint);
 	put_cluster_handle(&clust);
 	assert("edward-195",
--- linux-2.6.23-mm1/fs/reiser4/plugin/file/file.c.orig
+++ linux-2.6.23-mm1/fs/reiser4/plugin/file/file.c
@@ -2201,8 +2201,11 @@
 					}
 					if (uf_info->container ==  UF_CONTAINER_TAILS) {
 						result = tail2extent(uf_info);
-						if (result)
+						if (result) {
+							drop_exclusive_access(uf_info);
+							context_set_commit_async(ctx);
 							break;
+						}
 					}
 				}
 				drop_exclusive_access(uf_info);
@@ -2244,7 +2247,7 @@
 			current->backing_dev_info = NULL;
 			drop_access(uf_info);
 			context_set_commit_async(ctx);
-			return result;
+			break;
 		}
 		drop_access(uf_info);
 		ea = NEITHER_OBTAINED;
@@ -2315,6 +2318,7 @@
 		    !rofs_inode(inode)) {
 			result = extent2tail(file, uf_info);
 			if (result != 0) {
+				context_set_commit_async(ctx);
 				warning("nikita-3233",
 					"Failed (%d) to convert in %s (%llu)",
 					result, __FUNCTION__,
--- linux-2.6.23-mm1/fs/reiser4/plugin/file/tail_conversion.c.orig
+++ linux-2.6.23-mm1/fs/reiser4/plugin/file/tail_conversion.c
@@ -133,9 +133,11 @@
 
 	for (i = 0; i < nr_pages; i++) {
 		if (pages[i] == NULL) {
+#if REISER4_DEBUG
 			unsigned j;
 			for (j = i + 1; j < nr_pages; j++)
 				assert("vs-1620", pages[j] == NULL);
+#endif
 			break;
 		}
 		page_cache_release(pages[i]);
@@ -348,8 +350,10 @@
 	while (done == 0) {
 		memset(pages, 0, sizeof(pages));
 		result = reserve_tail2extent_iteration(inode);
-		if (result != 0)
+		if (result != 0) {
+			reiser4_inode_clr_flag(inode, REISER4_PART_IN_CONV);
 			goto out;
+		}
 		if (first_iteration) {
 			reiser4_inode_set_flag(inode, REISER4_PART_MIXED);
 			reiser4_update_sd(inode);
@@ -494,11 +498,9 @@
 							  REISER4_PART_MIXED));
 		}
 	}
-
-	reiser4_inode_clr_flag(inode, REISER4_PART_IN_CONV);
-
 	if (result == 0) {
 		/* file is converted to extent items */
+		reiser4_inode_clr_flag(inode, REISER4_PART_IN_CONV);
 		assert("vs-1697", reiser4_inode_get_flag(inode,
 							 REISER4_PART_MIXED));
 
@@ -507,16 +509,21 @@
 	} else {
 		/*
 		 * conversion is not complete. Inode was already marked as
-		 * REISER4_PART_CONV and stat-data were updated at the first
+		 * REISER4_PART_MIXED and stat-data were updated at the first
 		 * iteration of the loop above.
 		 */
-	      error:
+	error:
 		release_all_pages(pages, sizeof_array(pages));
-		warning("nikita-2282", "Partial conversion of %llu: %i",
+		reiser4_inode_clr_flag(inode, REISER4_PART_IN_CONV);
+		warning("edward-1548", "Partial conversion of %llu: %i",
 			(unsigned long long)get_inode_oid(inode), result);
 	}
 
-      out:
+ out:
+	/* this assertion is to make sure get_exclusive_access_careful()
+	   won't fall into deadlock loop */
+	assert("edward-1549", !reiser4_inode_get_flag(inode,
+						      REISER4_PART_IN_CONV));
 	return result;
 }
 
@@ -703,7 +710,7 @@
 	}
 	/*
 	 * conversion is not complete. Inode was already marked as
-	 * REISER4_PART_MIXED and stat-data were updated at the first *
+	 * REISER4_PART_MIXED and stat-data were updated at the first
 	 * iteration of the loop above.
 	 */
 	warning("nikita-2282",
@@ -711,6 +718,10 @@
 		(unsigned long long)get_inode_oid(inode), i,
 		num_pages, result);
 
+	/* this assertion is to make sure get_exclusive_access_careful()
+	   won't fall into deadlock loop */
+	assert("edward-1550", !reiser4_inode_get_flag(inode,
+						      REISER4_PART_IN_CONV));
 	return result;
 }
 
--- linux-2.6.23-mm1/fs/reiser4/plugin/item/internal.c.orig
+++ linux-2.6.23-mm1/fs/reiser4/plugin/item/internal.c
@@ -308,15 +308,23 @@
 		       struct carry_kill_data *p UNUSED_ARG)
 {
 	znode *child;
+	int result = 0;
 
 	assert("nikita-1222", item != NULL);
 	assert("nikita-1224", from == 0);
 	assert("nikita-1225", count == 1);
 
 	child = znode_at(item, item->node);
+	if (child == NULL)
+		return 0;
 	if (IS_ERR(child))
 		return PTR_ERR(child);
-	else if (node_is_empty(child)) {
+	result = zload(child);
+	if (result) {
+		zput(child);
+		return result;
+	}
+	if (node_is_empty(child)) {
 		reiser4_tree *tree;
 
 		assert("nikita-1397", znode_is_write_locked(child));
@@ -328,14 +336,14 @@
 		init_parent_coord(&child->in_parent, NULL);
 		--item->node->c_count;
 		write_unlock_tree(tree);
-		zput(child);
-		return 0;
 	} else {
 		warning("nikita-1223",
 			"Cowardly refuse to remove link to non-empty node");
-		zput(child);
-		return RETERR(-EIO);
+		result = RETERR(-EIO);
 	}
+	zrelse(child);
+	zput(child);
+	return result;
 }
 
 /* hook called by ->shift() node plugin method when iternal item was just

Re: No space left on rfs4

[John]

On Wed, 12 Mar 2008 00:39:08 +0300, Edward Shishkin <edward.shishkin@gmail.com> wrote:
> Edward Shishkin wrote:
> 
>>On 3/3/08, John <geearf@free.fr> wrote:
>>
>>
>>> On Sun, 02 Mar 2008 13:17:12 +0300, Edward Shishkin
> <edward.shishkin@gmail.com> wrote:
>>> > Hello.
>>> >
>>> > Yeah, indeed, I have reproduced it for reg40 (default plugin):
>>> > tar process is in permanent "+D" state. After reboot all files
>>> > were successfully deleted, although there is some leak of free
>>> > disk space there.
>>> >
>>> > Ok, I'll take a look at this more carefully (I guess -ENOSPC
>>> > error is handled incorrectly somewhere).
>>> >
>>> > If you have a problems with deleting files on reg40 partition,
>>> > then please pack your metadata by
>>> > debugfs.reiser4 -P /dev/xxx | gzip > meta.gz
>>> > and let me download the file meta.gz
>>>
>>>My FS is currently no full and with no bug, so I hope these are the
> metadata you wanted.
>>> If not I'll fill it again and send it to you again
>>>
>>> http://www.megaupload.com/?d=XVITV1CU
>>>
>>>
>>>
>>
>>Eventually I have not downloaded this:
>>It said "No free slots available for your country" ;)
>>
>>Well, don't bother with this for a while:
>>I have caught a mutex leak in cryptcompress plugin,
>>(fixup is attached) this explains undeletable files in ccreg40.
>>
>>I'll try fo fix unix-file plugin a bit later.
>>
>>
>>
> 
> I have found some ancient bugs related to tail conversion.
> In particular, it take place when application writes by chunks
> < 20K in no-space-left-on-device situation (for example, tar,
> which uses 10240 bytes chunks by default).
> The bugs are:
> . leak of per-inode exclusive access;
> . leak of per-inode flag REISER4_PART_IN_CONV
> All of them are responsible for reported deadlocks , and, perhaps
> can lead to data corruptions.
> 
> The fixup is attached.
> 
> There still takes place a silent leak of free disk space, when
> applications runs in no-space-left-on-device situation. This is also
> related only to (default) unix-file plugin with (default) "smart"
> formatting policy. Hope to address it soon..
> 
> Thanks for reports,
> Edward.
> 


Hello Edward,

Would you like me to test something particular now? 
I was waiting for 2.6.25 to compile again but I can do it before if you want.
My configuration changed a little bit, now all my reiser4 partitions are ccreg40, but I can replace swap for a reg40 partition to test it...


Thank you for your help,

John

Re: No space left on rfs4

[Edward Shishkin]

John wrote:

>On Wed, 12 Mar 2008 00:39:08 +0300, Edward Shishkin <edward.shishkin@gmail.com> wrote:
>  
>
>>Edward Shishkin wrote:
>>
>>    
>>
>>>On 3/3/08, John <geearf@free.fr> wrote:
>>>
>>>
>>>      
>>>
>>>>On Sun, 02 Mar 2008 13:17:12 +0300, Edward Shishkin
>>>>        
>>>>
>><edward.shishkin@gmail.com> wrote:
>>    
>>
>>>>>Hello.
>>>>>
>>>>>Yeah, indeed, I have reproduced it for reg40 (default plugin):
>>>>>tar process is in permanent "+D" state. After reboot all files
>>>>>were successfully deleted, although there is some leak of free
>>>>>disk space there.
>>>>>
>>>>>Ok, I'll take a look at this more carefully (I guess -ENOSPC
>>>>>error is handled incorrectly somewhere).
>>>>>
>>>>>If you have a problems with deleting files on reg40 partition,
>>>>>then please pack your metadata by
>>>>>debugfs.reiser4 -P /dev/xxx | gzip > meta.gz
>>>>>and let me download the file meta.gz
>>>>>          
>>>>>
>>>>My FS is currently no full and with no bug, so I hope these are the
>>>>        
>>>>
>>metadata you wanted.
>>    
>>
>>>>If not I'll fill it again and send it to you again
>>>>
>>>>http://www.megaupload.com/?d=XVITV1CU
>>>>
>>>>
>>>>
>>>>        
>>>>
>>>Eventually I have not downloaded this:
>>>It said "No free slots available for your country" ;)
>>>
>>>Well, don't bother with this for a while:
>>>I have caught a mutex leak in cryptcompress plugin,
>>>(fixup is attached) this explains undeletable files in ccreg40.
>>>
>>>I'll try fo fix unix-file plugin a bit later.
>>>
>>>
>>>
>>>      
>>>
>>I have found some ancient bugs related to tail conversion.
>>In particular, it take place when application writes by chunks
>>< 20K in no-space-left-on-device situation (for example, tar,
>>which uses 10240 bytes chunks by default).
>>The bugs are:
>>. leak of per-inode exclusive access;
>>. leak of per-inode flag REISER4_PART_IN_CONV
>>All of them are responsible for reported deadlocks , and, perhaps
>>can lead to data corruptions.
>>
>>The fixup is attached.
>>
>>There still takes place a silent leak of free disk space, when
>>applications runs in no-space-left-on-device situation. This is also
>>related only to (default) unix-file plugin with (default) "smart"
>>formatting policy. Hope to address it soon..
>>
>>Thanks for reports,
>>Edward.
>>
>>    
>>
>
>
>Hello Edward,
>
>Would you like me to test something particular now? 
>I was waiting for 2.6.25 to compile again but I can do it before if you want.
>My configuration changed a little bit, now all my reiser4 partitions are ccreg40, but I can replace swap for a reg40 partition to test it...
>
>  
>

Not in urgent need of this.
Just make sure you have previous patch applied (for ccreg40 to work 
properly).

Thanks,
Edward.