severe hardlink bug

severe hardlink bug

[Michael Niederle]

I'm using btrfs with a kernel 2.6.32.2 (builtin) as the root file system of a
Gentoo Linux installation.

While attempting to install the plt-scheme package a strange error about link
counts occurred ([Error 31] Too many Links).

I was able to create a simple scenario to reproduce the error:

Create an empty directory and run the following script within the directory:

#!/bin/bash

echo "abc" >.tab-snip-class_scrbl.dep._portage_merge_.9766
for x in $(seq 80)
do echo $x; ln .tab-snip-class_scrbl.dep._portage_merge_.9766 \
.tab-snip-class_scrbl.dep._portage_merge_.9766.$x; done


(The names are taken from the original Gentoo package.)

When I run this script the first 66 links are created properly. All further
link creations fail:

...
63
64
65
66
67
ln: creating hard link to `.tab-snip-class_scrbl.dep._portage_merge_.9766': Too
many links 68
ln: creating hard link to `.tab-snip-class_scrbl.dep._portage_merge_.9766': Too
many links 69
ln: creating hard link to `.tab-snip-class_scrbl.dep._portage_merge_.9766': Too
many links 70
ln: creating hard link to `.tab-snip-class_scrbl.dep._portage_merge_.9766': Too
many links
...


I was able to reproduce the bug independently on two different machines.

During earlier tests I determined that filename length has an effect as well as
the link count.

Greetings, Michael

Re: severe hardlink bug

[Goffredo Baroncelli]

On Sunday 24 January 2010, Michael Niederle wrote:
> I'm using btrfs with a kernel 2.6.32.2 (builtin) as the root file system of 
a
> Gentoo Linux installation.
> 
> While attempting to install the plt-scheme package a strange error about 
link
> counts occurred ([Error 31] Too many Links).

See this thread:

Mass-Hardlinking Oops - http://thread.gmane.org/gmane.comp.file-
systems.btrfs/3427

There is a limit of the number of hardlink for a file. The maximum number of 
the link depends by  name length.

BR
Goffredo


-- 
gpg key@ keyserver.linux.it: Goffredo Baroncelli (ghigo) <kreijackATinwind.it>
Key fingerprint = 4769 7E51 5293 D36C 814E  C054 BF04 F161 3DC5 0512

Re: severe hardlink bug

[Stephan von Krawczynski]

On Sun, 24 Jan 2010 09:09:44 +0100
Goffredo Baroncelli <kreijack@gmail.com> wrote:

> On Sunday 24 January 2010, Michael Niederle wrote:
> > I'm using btrfs with a kernel 2.6.32.2 (builtin) as the root file system of 
> a
> > Gentoo Linux installation.
> > 
> > While attempting to install the plt-scheme package a strange error about 
> link
> > counts occurred ([Error 31] Too many Links).
> 
> See this thread:
> 
> Mass-Hardlinking Oops - http://thread.gmane.org/gmane.comp.file-
> systems.btrfs/3427
> 
> There is a limit of the number of hardlink for a file. The maximum number of 
> the link depends by  name length.

Honestly, this dependency is braindead. How do the fs authors think an
application programmer should judge how many hardlinks are possible for a
certain fs ? This is a really bad design issue. Are we really in year 2010 ?

> BR
> Goffredo

-- 
Regards,
Stephan

Re: severe hardlink bug

[Chris Mason]

On Sun, Jan 24, 2010 at 12:04:32PM +0100, Stephan von Krawczynski wrote:
> On Sun, 24 Jan 2010 09:09:44 +0100
> Goffredo Baroncelli <kreijack@gmail.com> wrote:
> 
> > On Sunday 24 January 2010, Michael Niederle wrote:
> > > I'm using btrfs with a kernel 2.6.32.2 (builtin) as the root file system of 
> > a
> > > Gentoo Linux installation.
> > > 
> > > While attempting to install the plt-scheme package a strange error about 
> > link
> > > counts occurred ([Error 31] Too many Links).
> > 
> > See this thread:
> > 
> > Mass-Hardlinking Oops - http://thread.gmane.org/gmane.comp.file-
> > systems.btrfs/3427
> > 
> > There is a limit of the number of hardlink for a file. The maximum number of 
> > the link depends by  name length.
> 
> Honestly, this dependency is braindead. How do the fs authors think an
> application programmer should judge how many hardlinks are possible for a
> certain fs ? This is a really bad design issue. Are we really in year 2010 ?

Yes, this was a design trade off to maintain the back links for the
file.  Keep in mind the limit is only per-directory, as long as you're
working between different directories you can have any many back links
as you want.

But, it's on the list to get fixed once raid5/6 are out of the way.

-chris

Re: severe hardlink bug

[Dipl.-Ing. Michael Niederle]

Hi, Goffredo!

> See this thread:

> Mass-Hardlinking Oops - http://thread.gmane.org/gmane.comp.file-
> systems.btrfs/3427

Sorry, I didn't find the thread myself - despite of massive googeling. :-(

The gentoo scenario however is a real world application - no artificial
situation, so it would be really nice to fix the hard link problem somewhere in
the future!

The problem will happen in all busybox-like scenarios - and long filenames are
not so rarely used.

Greetings, Michael

Re: severe hardlink bug

[Goffredo Baroncelli]

On Sunday 24 January 2010, you (Dipl.-Ing. Michael Niederle) wrote:
> Hi, Goffredo!
> 
> > See this thread:
> 
> > Mass-Hardlinking Oops - 

May be that the link was broken in two line in the previous mail.
I copied again the ling. please give a try to see the thread.

http://thread.gmane.org/gmane.comp.file-systems.btrfs/3427

> 
> Sorry, I didn't find the thread myself - despite of massive googeling. :-(
> 
> The gentoo scenario however is a real world application - no artificial
> situation, so it would be really nice to fix the hard link problem somewhere 
in
> the future!

At the time of the thread, some peoples raised concerns about this limit. But 
at the time nobody highlighted a real case which required a lot of hard links.

Pay attention that in order to reach the limits the hard links have to be in 
the same directory.

If there is a real case in gentoo/bosybox, please send all the info. But in 
order solve this problem, it was told that a disk format change is required.

BR
Goffredo

 
> The problem will happen in all busybox-like scenarios - and long filenames 
are
> not so rarely used.
> 
> Greetings, Michael
> 


-- 
gpg key@ keyserver.linux.it: Goffredo Baroncelli (ghigo) <kreijackATinwind.it>
Key fingerprint = 4769 7E51 5293 D36C 814E  C054 BF04 F161 3DC5 0512

Re: severe hardlink bug

[Dipl.-Ing. Michael Niederle]

Hi, Goffredo!

I understood that a disk layout change will be necessary to accomodate for a
bigger total filename-space per file. So it is best done at once when more
issues arise that will need layout changes.

To recreate the error situation you need a Gentoo Linux with a btrfs root file
system and then try to install the plt-scheme package:

> emerge plt-scheme

Compilation works quite fine, but the installation aborts with a "Too many
links" error.

I reinstalled over 700 packages - plt-scheme beeing the only one failing due to
the btrfs link restriction.

Greetings, Michael

Re: severe hardlink bug

[Konstantin Dmitriev]

Dipl.-Ing. Michael Niederle <mniederle <at> gmx.at> writes:

> I reinstalled over 700 packages - plt-scheme beeing the only one failing due to
> the btrfs link restriction.
> 

I have hit the same issue - tried to run BackupPC with a pool on btrfs
filesystem. After some time the error of "too many links (31)" appeared to me.
Now I'm forced to migrate to some other filesystem...

Re: severe hardlink bug

[C Anthony Risinger]

On Sun, Jul 29, 2012 at 2:02 PM, Konstantin Dmitriev
<ksee.zelgadis@gmail.com> wrote:
> Dipl.-Ing. Michael Niederle <mniederle <at> gmx.at> writes:
>
>> I reinstalled over 700 packages - plt-scheme beeing the only one failing due to
>> the btrfs link restriction.
>>
>
> I have hit the same issue - tried to run BackupPC with a pool on btrfs
> filesystem. After some time the error of "too many links (31)" appeared to me.
> Now I'm forced to migrate to some other filesystem...

btrfs only fails when you have hundreds of hardlinks to the same file
in the *same* directory ... certainly not a standard use case.

use snapshots to your advantage:
 - snap source
 - rsync --inplace source to target (with some other opts that have
been discussed on list)
 - snap target
 - {rinse-and-repeat-in-24-hrs}

-- 

C Anthony

Re: severe hardlink bug

[Konstantin Dmitriev]

C Anthony Risinger <anthony <at> xtfx.me> writes:

> btrfs only fails when you have hundreds of hardlinks to the same file
> in the *same* directory ... certainly not a standard use case.
> 
> use snapshots to your advantage:
>  - snap source
>  - rsync --inplace source to target (with some other opts that have
> been discussed on list)
>  - snap target
>  - {rinse-and-repeat-in-24-hrs}

I understand that the case is only for *same* directory.
You can claim that it's not a standard use case, but first Michael hit that,
now me. There's at least one more case  -
https://lists.samba.org/archive/rsync/2011-December/027117.html
The count of such cases will be increasing and the sooner it will be fixed - 
the less pain it will bring to the users. I know fixing that is a big 
structural change, but it will become worse with time.
If it's not going to be fixed - I don't care. Right now I'm forced 
to migrate to old mdadm raid-1 or ZFS. The sad thing is that 
I really LOVED btrfs. Only that. ^__^

K.

Re: severe hardlink bug

[Arnd Hannemann]

Am 29.07.2012 21:13, schrieb C Anthony Risinger:
> On Sun, Jul 29, 2012 at 2:02 PM, Konstantin Dmitriev
> <ksee.zelgadis@gmail.com> wrote:
>> Dipl.-Ing. Michael Niederle <mniederle <at> gmx.at> writes:
>>
>>> I reinstalled over 700 packages - plt-scheme beeing the only one failing due to
>>> the btrfs link restriction.
>>>
>>
>> I have hit the same issue - tried to run BackupPC with a pool on btrfs
>> filesystem. After some time the error of "too many links (31)" appeared to me.
>> Now I'm forced to migrate to some other filesystem...
> 
> btrfs only fails when you have hundreds of hardlinks to the same file
> in the *same* directory ... certainly not a standard use case.

Actually, "hundreds of hardlinks" is certainly over optimistic.
In my testing 15 links in the same directory were enough to get
the "Too many links" error. It depends on the length of the file
name of the hardlinks.

Best regards
Arnd

Re: severe hardlink bug

[C Anthony Risinger]

On Mon, Jul 30, 2012 at 4:36 AM, Arnd Hannemann <arnd@arndnet.de> wrote:
> Am 29.07.2012 21:13, schrieb C Anthony Risinger:
>> On Sun, Jul 29, 2012 at 2:02 PM, Konstantin Dmitriev
>> <ksee.zelgadis@gmail.com> wrote:
>>> Dipl.-Ing. Michael Niederle <mniederle <at> gmx.at> writes:
>>>
>>>> I reinstalled over 700 packages - plt-scheme beeing the only one failing due to
>>>> the btrfs link restriction.
>>>>
>>>
>>> I have hit the same issue - tried to run BackupPC with a pool on btrfs
>>> filesystem. After some time the error of "too many links (31)" appeared to me.
>>> Now I'm forced to migrate to some other filesystem...
>>
>> btrfs only fails when you have hundreds of hardlinks to the same file
>> in the *same* directory ... certainly not a standard use case.
>
> Actually, "hundreds of hardlinks" is certainly over optimistic.
> In my testing 15 links in the same directory were enough to get
> the "Too many links" error. It depends on the length of the file
> name of the hardlinks.

Yes, per the linked patch it states 4k as the limit ... I thought I
recalled a limit of 256 but it seems I may have been mistaken.

The purpose of my initial response was to suggest an alternative
strategy -- one complementing btrfs's strengths -- a simple rsync +
snapshot is much more effective than BackupPC IMO ... but then again,
I'm bias, because I generally think BackupPC is junk.

--

C Anthony

Re: severe hardlink bug

[Helmut Hullen]

Hallo, Arnd,

Du meintest am 30.07.12:

>> btrfs only fails when you have hundreds of hardlinks to the same
>> file in the *same* directory ... certainly not a standard use case.

> Actually, "hundreds of hardlinks" is certainly over optimistic.
> In my testing 15 links in the same directory were enough to get
> the "Too many links" error. It depends on the length of the file
> name of the hardlinks.

And then I may get problems with my favourite backup program  
"rsnapshot"; it uses hard links, and my backups often have more than 20  
hard links to 1 file. Sometimes about 80 characters long.

Viele Gruesse!
Helmut

Re: severe hardlink bug

[Jan Schmidt]

Hi Konstantin,

On Sun, July 29, 2012 at 21:02 (+0200), Konstantin Dmitriev wrote:
> Dipl.-Ing. Michael Niederle <mniederle <at> gmx.at> writes:
> 
>> I reinstalled over 700 packages - plt-scheme beeing the only one failing due to
>> the btrfs link restriction.
>>
> 
> I have hit the same issue - tried to run BackupPC with a pool on btrfs
> filesystem. After some time the error of "too many links (31)" appeared to me.
> Now I'm forced to migrate to some other filesystem...

Please give the patch set "btrfs: extended inode refs" by Mark Fasheh a try
(http://lwn.net/Articles/498226/). It eliminates the hard links per directory
limit (introducing a rather random, artificial limit of 64k instead).

-Jan

Re: severe hardlink bug

[Konstantin Dmitriev]

Jan Schmidt <list.btrfs <at> jan-o-sch.net> writes:
> Please give the patch set "btrfs: extended inode refs" by Mark Fasheh a try
> (http://lwn.net/Articles/498226/). It eliminates the hard links per directory
> limit (introducing a rather random, artificial limit of 64k instead).

Hi, Jan!
I'm happy to see that there is something done on fixing that issue.
Unfortunately I cannot afford to have unstable patched kernel on the production
server. Probably I'll give another try to btrfs in 2013. ^__^
K.

Re: severe hardlink bug

[Chris Samuel]

On 08/08/2012 04:37 PM, Konstantin Dmitriev wrote:

> Unfortunately I cannot afford to have unstable patched kernel on the production
> server.

I'd suggest that using btrfs for production servers is probably a bit 
brave (no matter what Oracle say), and what you've found illustrates the 
point rather well I'm afraid..

cheers,
Chris
-- 
  Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC

Re: severe hardlink bug

[Martin Steigerwald]

Am Sonntag, 24. Januar 2010 schrieb Michael Niederle:
> I'm using btrfs with a kernel 2.6.32.2 (builtin) as the root file
> system of a Gentoo Linux installation.

Upgrade your kernel!

This kernel is waaaaaaaaaaaaaaaaaaaaaaaay to old for any production use of 
BTRFS. Heck, upstream still did not mark BTRFS as stable although Oracle 
and SUSE do support it. Current kernel have a many fixes.

Use at least 3.2. And have backups anyway.

The hard-link issue is known since ages and being worked on (see link from 
Jan in this thread). If you want to help, test the patch.

So, folks, can we get over it?

Thanks, 
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7