reiserfs3, rsync and hardlinks

reiserfs3, rsync and hardlinks

[Pierre Etchemaite]

	Hi all,

I'm using rsync to make a full backup of one box to another. Both are using
reiserfs 3.6.
The problem is that at the end of the backup, some pairs of unrelated files
(usually 3 or 4), are hardlinked together.

From what I've read in rsync mailing list archives, it probably means that
inode numbers are being reused while the backup is in progress.

Does that look plausible to you ?

And if it is, what can be done to significantly lower the probability of
inode number reuse over a short time ? The backup shouldn't take over 1
hour, the server stores around 570k files, but it's only a quite low traffic
IMAP server, so even with 32 bits inode numbers there's probably some room
for improvement.

I know a definitive answer would be to backup filesystem snapshots, but the
source server isn't currently using LVM1, and as far as I know snapshots
haven't been reimplemented yet for LVM2 :(

Best regards,
Pierre.

Re: reiserfs3, rsync and hardlinks

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Pierre Etchemaite wrote:
| 	Hi all,
|
| I'm using rsync to make a full backup of one box to another. Both are
using
| reiserfs 3.6.

Go to XFS or Reiser4.  Unless someone is about to correct me, reiserfs3
still has problems with hash collisions.  The rest of it *should* be
stable.... ;-)

If it's mission-critical, you probably want XFS -- reiser4 still has at
least one performance issue to iron out, and it doesn't have the years
of public testing that XFS does.

I don't take the above advice -- I'm adventurous enough to use reiser4
on all my boxes, and I'm reading code as a bedtime story.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQgZtm3gHNmZLgCUhAQLgbBAAjAPSoC52ZfFC6w7rxBVFHVmU/drVHBtn
H/YH+qa4rGVTI63aEiYuQXEln8xjnantikiQDdYb7zEHD0sOLiWgEGEjVf8FW942
HhtLGEZFxgjj6e65/jRDo/+FAobern+k6CktoxyDJawyySbAjnvvjLZwInwnkqPi
tvtlSZjm0P7Y0AwzNg4cxCycKGJI2ZaUJ2wsy2CypZkDqSf6PAedt0Za/qhU34/8
tXHXxkte0hDOjp1SZ30Mh96WsXOu8WfH8wiCX71MZwwZEioY7+apMvQx4LuCXR6b
W64Ug7zu7RygIPWNB61x0kjS9GRF3YoB1kfyQzPMYBi5BaIB33mejiZv1vaTGIuE
kdMwooF0tZ2tY2N8dwELcRP4B4HCF1anmAig+bqElB6VBwLUozFe8DEmQZv3yhrQ
9iPWOm9HGfR/sQVm5M7VGmBK6WeHPC2h747aOAurlsrUSwQNFAdHosVfatVmcFVn
WdqbBi8ryUu1CX/y5sln4E8B/qDciboIoyWPOzDPT44lBkSnYhT9VrYhrYW5lp6U
SfhNHIdT+IkPVpJjCEN08dBPA6Lgw9nn7Caxp7r4cmREJOpWxBOgZoHh4YQBQbZ9
BzgFoVnEZsCuUo1mr3S3ODf6qZ7YZf12ssdKcdflgIDyZYks53zi/E38NLrYWmn0
GkVFUgMiLw4=
=qaQk
-----END PGP SIGNATURE-----

Re: reiserfs3, rsync and hardlinks

[Pierre Etchemaite]

Le dim 06 fév 2005 13:18:52 CET, David Masover <ninja@slaphack.com> a écrit
:

> Go to XFS or Reiser4.  Unless someone is about to correct me, reiserfs3
> still has problems with hash collisions.  The rest of it *should* be
> stable.... ;-)

I read this recently, but never had problems with hash collisions since I
use reiserfs 3 (don't remember the exact time, but probably around '98), as
far as I can tell.

> If it's mission-critical, you probably want XFS -- reiser4 still has at
> least one performance issue to iron out, and it doesn't have the years
> of public testing that XFS does.

I use reiser4 on my home box, it's fine, but I won't use it for that
server... yet, at least.

Because trustiness isn't the only parameter. The problem with either XFS or
Reiser4, is that they're probably more CPU intensive than reiserfs 3, and on
that server that's starting to age, it could be a problem.

So, no solution with reiserfs 3 ?

Re: reiserfs3, rsync and hardlinks

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Pierre Etchemaite wrote:
| Le dim 06 fév 2005 13:18:52 CET, David Masover <ninja@slaphack.com> a
écrit
| :
|
|
|>Go to XFS or Reiser4.  Unless someone is about to correct me, reiserfs3
|>still has problems with hash collisions.  The rest of it *should* be
|>stable.... ;-)
|
|
| I read this recently, but never had problems with hash collisions since I
| use reiserfs 3 (don't remember the exact time, but probably around
'98), as
| far as I can tell.

There was a debian package which caused a hash collision.  Every time.
Not even something obscure; I think it was some piece of X.

|>If it's mission-critical, you probably want XFS -- reiser4 still has at
|>least one performance issue to iron out, and it doesn't have the years
|>of public testing that XFS does.
|
|
| I use reiser4 on my home box, it's fine, but I won't use it for that
| server... yet, at least.
|
| Because trustiness isn't the only parameter. The problem with either
XFS or
| Reiser4, is that they're probably more CPU intensive than reiserfs 3,
and on
| that server that's starting to age, it could be a problem.

Think I remember some other filesystem being really lightweight CPU.
UFS or something.  But XFS shouldn't be too bad.  Where does your
"probably" come from?

| So, no solution with reiserfs 3 ?

Don't know.  I don't work here :P
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQgb9w3gHNmZLgCUhAQL9ZA/8DzKOqj+4xcNmp6xaqdLZOounaaO1dImK
9YSyE7VBtQkE23CpRS/SdFe17ZjVmlgbho93r7BGnuMr4BrctInd+B7V4GwBXPvx
MUiC08lJztwL2JMY2yeN5dzkYbSUjnt4jhDmy8/XHH5MUTFzwH5vtZzXzhaQYbZ5
gWurzJdMj8l0pHUlCZUTLUO7i6yiuD4lh20QZaX2pe3tbQ1jfPKJTGw+o7OFm8GY
vcQCe8JUCTkfpt+bROqcnPWT15Al1cxzRzEocuBvToDfiJlBjuWK9m+QYnuBMaCC
gmy5z5MHE3Jak3bM+kADvtdmnuP2OAZ+SFWTDXQOVuD94dGa7Qpsa2HjpQj8cMsk
w8FNO1nqEYtqWghA0k/97U2cfhkXUlSoxZ+Ud5xyIqy4R1463NziQwIHXryF32cN
alg2ja6yO+4d4UuhgaMxw6uqWIk7ZBDsnrq93qEBi0J+MWXzLlQRKQZyg0dVEdyZ
nRoDMYKP2JkKcx8/2RmJjNlKa9uiyoy0/jb8F+wquAObBzUZOpMxuPVogsvMlo69
95mu2OgCLfr+NY7GWL/iIqbKj9R8nvX2NiSdN+nlrPu4ODCrb1ce2PbtGEuIEbhg
KlcgyZiFDY90Uq/YOOyx4yacdYOykTjQKzDfeAwbUkgEdd8kllYSzkC2rmA1m+wu
xkHMdj3UHu4=
=8vLa
-----END PGP SIGNATURE-----

Re: reiserfs3, rsync and hardlinks

[Vladimir Saveliev]

[-- Attachment #1: Type: text/plain, Size: 1467 bytes --]

Hello

On Sun, 2005-02-06 at 05:41, Pierre Etchemaite wrote:
> 	Hi all,
> 
> I'm using rsync to make a full backup of one box to another. Both are using
> reiserfs 3.6.
> The problem is that at the end of the backup, some pairs of unrelated files
> (usually 3 or 4), are hardlinked together.
> 
> >From what I've read in rsync mailing list archives, it probably means that
> inode numbers are being reused while the backup is in progress.
> 
> Does that look plausible to you ?
> 

yes, reiserfs reuses inode number of removed files for newly created
files. However, ext2 also does that. Have  you ever noticed this problem
on other filesystems?

You can try to make reiserfs to not free inode numbers of removed files
with the attached patch and check whether it helps. It decreases number
of files which can be created on a filesystem to ~2^^32.
I am not sure if it is enough for low traffic IMPA server.
 

> And if it is, what can be done to significantly lower the probability of
> inode number reuse over a short time ? The backup shouldn't take over 1
> hour, the server stores around 570k files, but it's only a quite low traffic
> IMAP server, so even with 32 bits inode numbers there's probably some room
> for improvement.
> 
> I know a definitive answer would be to backup filesystem snapshots, but the
> source server isn't currently using LVM1, and as far as I know snapshots
> haven't been reimplemented yet for LVM2 :(
> 
> Best regards,
> Pierre.
> 

[-- Attachment #2: reiserfs-dont-free-objectid.patch --]
[-- Type: text/plain, Size: 597 bytes --]

 fs/reiserfs/objectid.c |    2 +-
 1 files changed, 1 insertion(+), 1 deletion(-)

diff -puN fs/reiserfs/objectid.c~reiserfs-dont-free-objectid fs/reiserfs/objectid.c
--- linux-2.6.9/fs/reiserfs/objectid.c~reiserfs-dont-free-objectid	2005-02-07 13:15:25.967303557 +0300
+++ linux-2.6.9-vs/fs/reiserfs/objectid.c	2005-02-07 13:15:42.576600790 +0300
@@ -99,7 +99,7 @@ void reiserfs_release_objectid (struct r
     __u32 * map = objectid_map (s, rs);
     int i = 0;
 
-    //return;
+    return;
     check_objectid_map (s, map);
 
     reiserfs_prepare_for_journal(s, SB_BUFFER_WITH_SB(s), 1) ;

_

Re: reiserfs3, rsync and hardlinks

[Hans Reiser]

Pierre Etchemaite wrote:

>	Hi all,
>
>I'm using rsync to make a full backup of one box to another. Both are using
>reiserfs 3.6.
>The problem is that at the end of the backup, some pairs of unrelated files
>(usually 3 or 4), are hardlinked together.
>
>>From what I've read in rsync mailing list archives, it probably means that
>inode numbers are being reused while the backup is in progress.
>
>Does that look plausible to you ?
>
>And if it is, what can be done to significantly lower the probability of
>inode number reuse over a short time ? The backup shouldn't take over 1
>hour, the server stores around 570k files, but it's only a quite low traffic
>IMAP server, so even with 32 bits inode numbers there's probably some room
>for improvement.
>
>I know a definitive answer would be to backup filesystem snapshots, but the
>source server isn't currently using LVM1, and as far as I know snapshots
>haven't been reimplemented yet for LVM2 :(
>
>Best regards,
>Pierre.
>
>
>  
>
This really sounds like an rsync bug, maybe you could cc the rsync 
mailing list or tridgell?

Re: reiserfs3, rsync and hardlinks

[Pierre Etchemaite]

Le lun 07 fév 2005 13:22:51 CET, Vladimir Saveliev <vs@namesys.com> a écrit
:

> Hello

	Hi,

> yes, reiserfs reuses inode number of removed files for newly created
> files. However, ext2 also does that. Have  you ever noticed this problem
> on other filesystems?

No, but I'm only using rsync -H for a few weeks. The problem may also exist
with tar, but unnoticed (unless tar detects hardlinks in a different way,
or does more checks, like checking the consistency with references counters,
whatever, to avoid it). rsync handles hardlinks in a final pass, so as soon
as the verbosity level is raised, problems are easy to detect.

I have only one server left that uses ext2. It's also saved with rsync, no
problem seen so far (a few weeks only, as I said).
But the filesystem used isn't the only difference. Usage pattern probably
matters a lot. On the system where it happens, hardlinked files are often
Maildir files (unsurprizingly) and mrtg log files (which are rotated every 5
minutes). inodes are probably freed by mrtg, and one reused for a new email.

> You can try to make reiserfs to not free inode numbers of removed files
> with the attached patch and check whether it helps. It decreases number
> of files which can be created on a filesystem to ~2^^32.
> I am not sure if it is enough for low traffic IMPA server.

Ok, I can probably try this hack to verify the hypothesis. But what are the
drawbacks, on the long term ? Lost disk space ? What happens if all inode
numbers get allocated ? mkreiserfs ?

Best regards,
Pierre.

Re: reiserfs3, rsync and hardlinks

[Chris Mason]

On Monday 07 February 2005 15:50, Pierre Etchemaite wrote:
> Le lun 07 fév 2005 13:22:51 CET, Vladimir Saveliev <vs@namesys.com> a écrit
>
> > Hello
>
> 	Hi,
>
> > yes, reiserfs reuses inode number of removed files for newly created
> > files. However, ext2 also does that. Have  you ever noticed this problem
> > on other filesystems?
>
> No, but I'm only using rsync -H for a few weeks. The problem may also exist
> with tar, but unnoticed (unless tar detects hardlinks in a different way,
> or does more checks, like checking the consistency with references
> counters, whatever, to avoid it). rsync handles hardlinks in a final pass,
> so as soon as the verbosity level is raised, problems are easy to detect.
>
> I have only one server left that uses ext2. It's also saved with rsync, no
> problem seen so far (a few weeks only, as I said).
> But the filesystem used isn't the only difference. Usage pattern probably
> matters a lot. On the system where it happens, hardlinked files are often
> Maildir files (unsurprizingly) and mrtg log files (which are rotated every
> 5 minutes). inodes are probably freed by mrtg, and one reused for a new
> email.
>
If you've got files being deleted in the middle of the backup,  then it is 
extremely difficult for rsync (or any tool) to get the hard link detection 
correct.  You've got a few choices:

1) put everything on lvm and backup snapshots instead of the live filesystem.  
This has a number of benefits.

2) link everyfile into some temp directory before the backup starts.  This 
will prevent that particular inode number from being reused during the 
backup, but won't help if new files are added during the rsync (since those 
new files could also be deleted).

for each file in backup list
    ln file tmpdir/counter
    counter++
rsync
rm -rf  tmpdir

-chris

Re: reiserfs3, rsync and hardlinks

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Chris Mason wrote:
| On Monday 07 February 2005 15:50, Pierre Etchemaite wrote:
|
|>Le lun 07 fév 2005 13:22:51 CET, Vladimir Saveliev <vs@namesys.com> a
écrit
|>
|>
|>>Hello
|>
|>	Hi,
|>
|>
|>>yes, reiserfs reuses inode number of removed files for newly created
|>>files. However, ext2 also does that. Have  you ever noticed this problem
|>>on other filesystems?
|>
|>No, but I'm only using rsync -H for a few weeks. The problem may also
exist
|>with tar, but unnoticed (unless tar detects hardlinks in a different way,
|>or does more checks, like checking the consistency with references
|>counters, whatever, to avoid it). rsync handles hardlinks in a final pass,
|>so as soon as the verbosity level is raised, problems are easy to detect.
|>
|>I have only one server left that uses ext2. It's also saved with rsync, no
|>problem seen so far (a few weeks only, as I said).
|>But the filesystem used isn't the only difference. Usage pattern probably
|>matters a lot. On the system where it happens, hardlinked files are often
|>Maildir files (unsurprizingly) and mrtg log files (which are rotated every
|>5 minutes). inodes are probably freed by mrtg, and one reused for a new
|>email.
|>
|
| If you've got files being deleted in the middle of the backup,  then
it is
| extremely difficult for rsync (or any tool) to get the hard link
detection
| correct.  You've got a few choices:
|
| 1) put everything on lvm and backup snapshots instead of the live
filesystem.
| This has a number of benefits.

Isn't snapshot support planned for reiser4?  It would be saner than
lvm/device-mapper -- no dedicated partition needed...

| 2) link everyfile into some temp directory before the backup starts.
This
| will prevent that particular inode number from being reused during the
| backup, but won't help if new files are added during the rsync (since
those
| new files could also be deleted).

Is this just backup?  You could use (or writing) a backup utility which
indexes by a hash of each file.  Wouldn't be that much slower, and would
avoid any issues with hardlink detection.

Don't know of any existing utility that does that.  I think BackupPC was
close (backuppc.sf.net)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQghTzXgHNmZLgCUhAQIy5xAAiT/ip19yy169PLqDJhMB1vtI1bEWiZsl
CWvljdxP4keoaWd32pXzZ1PJ1c3GoCb6W3TgxwlLgW+dvx87cvCQDK3pTFIpqOnL
0EGBwEkYVDEpmBdbkizuL744LSP+eubHFRfzJa5VFuZMROGIU2M5GbXlNM1uTimG
3q9rZSTmJL1pdAJqqZjNX0SRmg7z26KFKQL/E1Sm7E/VUAKrZSubqxEif4wG7JZ4
llNEitQlW61CG2nX8qgtta8ufrmmiAm37Vt6G7+fRyrKLbCIgsOuPOr//XL3j8aR
Qg4KrlWgjH/Fu/CR+TB+PSKeSBFbh47gvHChY3eqZaBg7UJ/ftPfZ/hz/m4S6MXK
Ic/Ww91cF/t6vMiErlj2GObggEgzRMq4ayitwxz19PucmSbv1gwVgmzVBX9uRvuZ
r/qE2r38qhzveiik3Y/MO8I5/R8H4FaF7JIhxNcujYqyqx6H4l4qfVksOG4Ox3OH
/sh3h7Ev+266w9WVbshD+gmFThvzDAKvjcntDFXLXDXMFGMKkrwbNVONczNPDkm3
G+ZiPjWyuKCaiGVbRPdvdQ4Jlpw4559E4APiHD7I4mhmdmmUVJSUbiTpVCQUA2gQ
F3dtiLTp4zshvG8yRqHS/CtWPMQHGo9TbjIKv4ynSOwXuAETiCtze7w9tZwZgV5W
UFAQ8tBHS24=
=ogtF
-----END PGP SIGNATURE-----

Re: reiserfs3, rsync and hardlinks

[mjt]

On Sun, Feb 06, 2005 at 11:33:56PM -0600, David Masover wrote:
>There was a debian package which caused a hash collision.  Every time.
>Not even something obscure; I think it was some piece of X.

Which, btw, has been around for a looong time and a LOT of people have
it installed. Without a hash collision. Not once.

Now I haven't been reading these messages as carefully as I should have,
after some time of being away, but seems to me no-one could explain
why that hash collision came to be for that one guy.

>Think I remember some other filesystem being really lightweight CPU.
>UFS or something.  But XFS shouldn't be too bad.  Where does your
>"probably" come from?

UFS as in Unix File System? Eww..

Dunno about CPU usage, but I tested XFS once and it was painfully slow.
Fortunately it got better, so maybe my test was somewhat flawed.

Maybe the Namesys guys would want to release some new and up-to-date
benchmarks now that the AMD64 panic bug is fixed?

-- 
mjt

Re: reiserfs3, rsync and hardlinks

[Vladimir Saveliev]

Hello

On Mon, 2005-02-07 at 23:50, Pierre Etchemaite wrote:
> Le lun 07 fév 2005 13:22:51 CET, Vladimir Saveliev <vs@namesys.com> a écrit
> :
> 
> > Hello
> 
> 	Hi,
> 
> > yes, reiserfs reuses inode number of removed files for newly created
> > files. However, ext2 also does that. Have  you ever noticed this problem
> > on other filesystems?
> 
> No, but I'm only using rsync -H for a few weeks. The problem may also exist
> with tar, but unnoticed (unless tar detects hardlinks in a different way,
> or does more checks, like checking the consistency with references counters,
> whatever, to avoid it). rsync handles hardlinks in a final pass, so as soon
> as the verbosity level is raised, problems are easy to detect.
> 
> I have only one server left that uses ext2. It's also saved with rsync, no
> problem seen so far (a few weeks only, as I said).
> But the filesystem used isn't the only difference. Usage pattern probably
> matters a lot. On the system where it happens, hardlinked files are often
> Maildir files (unsurprizingly) and mrtg log files (which are rotated every 5
> minutes). inodes are probably freed by mrtg, and one reused for a new email.
> 
> > You can try to make reiserfs to not free inode numbers of removed files
> > with the attached patch and check whether it helps. It decreases number
> > of files which can be created on a filesystem to ~2^^32.
> > I am not sure if it is enough for low traffic IMPA server.
> 
> Ok, I can probably try this hack to verify the hypothesis. 

Yes, please do that.

> But what are the
> drawbacks, on the long term ? Lost disk space ? What happens if all inode
> numbers get allocated ? 

open(O_CREAT), mkdir, etc will refuse to create files. ENOMEM will be
returned.


> mkreiserfs ?
> 
> Best regards,
> Pierre.
> 

Re: reiserfs3, rsync and hardlinks

[Hans Reiser]

Chris Mason wrote:

>
>
>1) put everything on lvm and backup snapshots instead of the live filesystem.  
>This has a number of benefits.
>
>  
>
I recommend this one.

Re: reiserfs3, rsync and hardlinks

[Hans Reiser]

Reiser4 CPU usage is reasonable now I think.  If you experience that it 
is too high, use the extents only mount option, and it should be pretty 
good.  Maybe somebody has some time to create some numbers?  I could be 
wrong, but let us look at numbers.....

Hans

Markus Törnqvist wrote:

>
>  
>
>>Think I remember some other filesystem being really lightweight CPU.
>>UFS or something.  But XFS shouldn't be too bad.  Where does your
>>"probably" come from?
>>    
>>
>
>UFS as in Unix File System? Eww..
>
>Dunno about CPU usage, but I tested XFS once and it was painfully slow.
>Fortunately it got better, so maybe my test was somewhat flawed.
>
>Maybe the Namesys guys would want to release some new and up-to-date
>benchmarks now that the AMD64 panic bug is fixed?
>
>  
>

Re: reiserfs3, rsync and hardlinks

[Pierre Etchemaite]

Le mar 08 fév 2005 14:16:43 CET, Vladimir Saveliev <vs@namesys.com> a écrit
:

> > > You can try to make reiserfs to not free inode numbers of removed
> > > files with the attached patch and check whether it helps. It decreases
> > > number of files which can be created on a filesystem to ~2^^32.
> > > I am not sure if it is enough for low traffic IMPA server.
> > 
> > Ok, I can probably try this hack to verify the hypothesis. 
> 
> Yes, please do that.

Worked fine this night, but I will keep it running some more (they were many
mrtg test processes stuck for a reason I couldn't determine, rebooting
cleaned them up, I hope this didn't influence the test result...)

Best regards,
Pierre.

Re: reiserfs3, rsync and hardlinks

[Pierre Etchemaite]

Le mar 08 fév 2005 14:14:08 CET, Hans Reiser <reiser@namesys.com> a écrit :

> Chris Mason wrote:
> >1) put everything on lvm and backup snapshots instead of the live
> >filesystem.  This has a number of benefits.
> >
> I recommend this one.

Yes, my favorite too. And since my first message, I learned that the
snapshot feature is under way (alpha stage ?) for LVM2, which is a great
news.

Best regards,
Pierre.